Package: word2ket
word2ketword2ketis an space-efficient embedding layer that can reduce the space required to store the embeddings by up to 100,000x.This is a PyTorch implementaion of the embedding layer that is proposed in the paperword2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement.word2ketInstallationDependenciesInstall word2ketGetting StartedRunning examplesDependenciesDownload DatasetsRun text summarizationRun German-English machine translationReferenceAPABibTexLicenseInstallationDependencies# Install PyTorchcondainstallpytorchtorchvision-cpytorch# Install GPyTorchcondainstallgpytorch-cgpytorchInstall word2ketpipinstallword2ketGetting StartedYou can directly useEmbeddingKetandEmbeddingKetXSlayers in your model definition or you can use theketifyfunction to automatically replace all thenn.Embeddinglayers in your model. Below you can see a model that is using7,680,000parameters for its embedding could beketifyto just use448parameters.fromword2ketimportEmbeddingKet,EmbeddingKetXS,ketify,summaryfromtorchimportnnimportlogginglogging.basicConfig(level=logging.INFO)# Word2Ket Embedding Layerw2v_embedding_layer=EmbeddingKetXS(num_embeddings=30000,embedding_dim=256,order=4,rank=1)summary(w2v_embedding_layer)"""INFO:root:EmbeddingKetXS num_embeddings_leaf: 14INFO:root:EmbeddingKetXS embedding_dim_leaf: 4INFO:root:EmbeddingKetXS weight_leafs shape: torch.Size([4, 1, 14, 4])Module Name                                                                           Total Parameters  Trainable Parameters # Elements in Trainable ParametrsEmbeddingKetXS(30000, 256)                                                            1                 1                    224Total number of trainable parameters elements 224"""# PyTorch Embedding LayerclassMyModel(nn.Module):def__init__(self,vocab_size,embedding_dim):super(MyModel,self).__init__()self.embedding=nn.Embedding(num_embeddings=vocab_size,embedding_dim=embedding_dim)defforward(self,x):self.embedding()model=MyModel(vocab_size=30000,embedding_dim=256)print("embedding.weight.shape: ",model.embedding.weight.shape)"""embedding.weight.shape:  torch.Size([30000, 256])"""summary(model)"""Module Name                                                                           Total Parameters  Trainable Parameters # Elements in Trainable ParametrsEmbedding(30000, 256)                                                                 1                 1                    7,680,000Total number of trainable parameters elements 7,680,000"""# Replace the nn.Embedding to EmbeddingKetXS automatically using the ketify function.ketify(model,order=4,rank=2,use_EmbeddingKetXS=True)summary(model)"""INFO:root:EmbeddingKetXS num_embeddings_leaf: 14INFO:root:EmbeddingKetXS embedding_dim_leaf: 4INFO:root:EmbeddingKetXS weight_leafs shape: torch.Size([4, 2, 14, 4])INFO:root:Replaced embedding in MyModelModule Name                                                                           Total Parameters  Trainable Parameters # Elements in Trainable ParametrsEmbeddingKetXS(30000, 256)                                                            1                 1                    448Total number of trainable parameters elements 448"""Running examplesDependencies# Install Texar-PyTorchpipinstalltexar-pytorch# Install RougepipinstallrougeDownload Datasetscd./examples/texar/
python./prepare_data.py--dataiwslt14
python./prepare_data.py--datagigaRun text summarizationUsingGIGAWORDdataset.# Using Pytorch nn.Embedding Layerpythonseq2seq_attn.py--embedding_typenn.Embedding--gpu4--runNameG_000--config-modelconfig_model--config-dataconfig_giga# Using EmbeddingKet Layerpythonseq2seq_attn.py--embedding_typeEmbeddingKet--gpu0--runNameV2K_G_000--config-modelconfig_model--config-dataconfig_giga--order4--rank1# Using EmbeddingKetXS Layerpythonseq2seq_attn.py--embedding_typeEmbeddingKetXS--gpu0--runNameV2K_XS_G_000--config-modelconfig_model--config-dataconfig_giga--order4--rank1Run German-English machine translationUsingIWSLT2014(DE-EN) dataset# Using Pytorch nn.Embedding Layerpythonseq2seq_attn.py--embedding_typenn.Embedding--gpu4--runNameI_000--config-modelconfig_model--config-dataconfig_iwslt14# Using EmbeddingKet Layerpythonseq2seq_attn.py--embedding_typeEmbeddingKet--gpu0--runNameV2K_I_000--config-modelconfig_model--config-dataconfig_iwslt14--order4--rank1# Using EmbeddingKetXS Layerpythonseq2seq_attn.py--embedding_typeEmbeddingKetXS--gpu0--runNameV2K_XS_I_000--config-modelconfig_model--config-dataconfig_iwslt14--order4--rank1ReferenceIf you useword2ket, please cite the tech report with the following BibTex entry:APAPanahi, A., Saeedi, S., & Arodz, T. (2020). word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement. In International Conference on Learning Representations. Retrieved from https://openreview.net/forum?id=HkxARkrFwBBibTex@inproceedings{panahi2020wordket,
  title={word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement},
  author={Aliakbar Panahi and Seyran Saeedi and Tom Arodz},
  booktitle={International Conference on Learning Representations},
  year={2020},
  url={https://openreview.net/forum?id=HkxARkrFwB}
}LicenseBSD-3-Clause