word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement

1 INTRODUCTION . Modern deep learning approaches for natural language processing ( NLP ) often rely on vector representation of words to convert discrete space of human language into continuous space best suited for further processing through a neural network . For a language with vocabulary of size d , a simple way to achieve this mapping is to use one-hot representation – each word is mapped to its own row of a d× d identity matrix . There is no need to actually store the identity matrix in memory , it is trivial to reconstruct the row from the word identifier . Word embedding approaches such as word2vec ( Mikolov et al. , 2013 ) or GloVe ( Pennington et al. , 2014 ) use instead vectors of dimensionality p much smaller than d to represent words , but the vectors are not necessarily extremely sparse nor mutually orthogonal . This has two benefits : the embeddings can be trained on large text corpora to capture the semantic relationship between words , and the downstream neural network layers only need to be of width proportional to p , not d , to accept a word or a sentence . We do , however , need to explicitly store the d× p embedding matrix in GPU memory for efficient access during training and inference . Vocabulary sizes can reach d = 105 or 106 ( Pennington et al. , 2014 ) , and dimensionality of the embeddings used in current systems ranges from p = 300 ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) to p = 1024 ( Devlin et al. , 2018 ) . The d × p embedding matrix thus becomes a substantial , often dominating , part of the parameter space of a learning model . In classical computing , information is stored in bits – a single bit represents an element from the set B = { 0 , 1 } , it can be in one of two possible states . A quantum equivalent of a bit , a qubit , is ∗Corresponding author . 1PyTorch implementation available at : https : //github.com/panaali/word2ket fully described by a single two-dimensional complex unit-norm vector , that is , an element from the set C2 . A state of an n-qubit quantum register corresponds to a vector in C2n . To have exponential dimensionality of the state space , though , the qubits in the register have to be interconnected so that their states can become entangled ; a set of all possible states of n completely separated , independent qubits can be fully represented by C2n instead of C2n . Entanglement is a purely quantum phenomenon – we can make quantum bits interconnected , so that a state of a two-qubit system can not be decomposed into states of individual qubits . We do not see entanglement in classical bits , which are always independent – we can describe a byte by separately listing the state of each of the eight bits . We can , however , approximate quantum register classically - store vectors of size m using O ( logm ) space , at the cost of losing the ability to express all possible m-dimensional vectors that an actual O ( logm ) -qubit quantum register would be able to represent . As we show in this paper , the loss of representation power does not have a significant impact on NLP machine learning algorithms that use the approximation approaches to store and manipulate the high-dimensional word embedding matrix . 1.1 OUR CONTRIBUTION . Here , we used approaches inspired by quantum computing to propose two related methods , word2ket and word2ketXS , for storing word embedding matrix during training and inference in a highly efficient way2 . The first method operates independently on the embedding of each word , allowing for more efficient processing , while the second method operates jointly on all word embeddings , offering even higher efficiency in storing the embedding matrix , at the cost of more complex processing . Empirical evidence from three NLP tasks shows that the new word2ket embeddings offer high space saving rate at little cost in terms of accuracy of the downstream NLP model . 2 FROM TENSOR PRODUCT SPACES TO word2ket EMBEDDINGS 2.1 TENSOR PRODUCT SPACE . Consider two separable3 Hilbert spaces V and W . A tensor product space of V and W , denoted as V ⊗ W , is a separable Hilbert space H constructed using ordered pairs v ⊗ w , where v ∈ V and w ∈ W . In the tensor product space , the addition and multiplication in H have the following properties c { v ⊗ w } = { cv } ⊗ w = v ⊗ { cw } , ( 1 ) v ⊗ w + v′ ⊗ w = { v + v′ } ⊗ w , v ⊗ w + v ⊗ w′ = v ⊗ { w + w′ } . The inner product between v ⊗ w and v′ ⊗ w′ is defined as a product of individual inner products 〈v ⊗ w , v′ ⊗ w′〉 = 〈v , v′〉 〈w , w′〉 . ( 2 ) It immediately follows that ||v ⊗ w|| = ||v|| ||w|| ; in particular , a tensor product of two unit-norm vectors , from V andW , respectively , is a unit norm vector in V ⊗W . The Hilbert space V ⊗W is a space of equivalence classes of pairs v ⊗ w ; for example { cv } ⊗ w and v ⊗ { cw } are equivalent ways to write the same vector . A vector in a tensor product space is often simply called a tensor . Let { ψj } and { φk } be orthonormal basis sets in V andW , respectively . From eq . 1 and 2 we can see that ∑ j cjψj ⊗ { ∑ k dkφk } = ∑ j ∑ k cjdkψj ⊗ φk , 〈ψj ⊗ φk , ψj′ ⊗ φk′〉 = δj−j′δk−k′ , where δz is the Kronecker delta , equal to one at z = 0 and to null elsewhere . That is , the set { ψj ⊗ φk } jk forms an orthonormal basis in V ⊗ W , with coefficients indexed by pairs jk and 2In Dirac notation popular in quantum mechanics and quantum computing , a vector u ∈ C2 n is written as |u〉 , and called a ket . 3That is , with countable orthonormal basis . numerically equal to the products of the corresponding coefficients in V and W . We can add any pairs of vectors in the new spaces by adding the coefficients . The dimensionality of V ⊗W is the product of dimensionalities of V andW . We can create tensor product spaces by more than one application of tensor product , H = U⊗V⊗W , with arbitrary bracketing , since tensor product is associative . Tensor product space of the form n⊗ j=1 Hj = H1 ⊗H2 ⊗ . . .⊗Hn is said to have tensor order4 of n . 2.2 ENTANGLED TENSORS . Consider H = V ⊗ W . We have seen the addition property v ⊗ w + v′ ⊗ w = { v + v′ } ⊗ w and similar property with linearity in the first argument – tensor product is bilinear . We have not , however , seen how to express v ⊗ w + v′ ⊗ w′ as φ ⊗ ψ for some φ ∈ V , ψ ∈ W . In many cases , while the left side is a proper vector from the tensor product space , it is not possible to find such φ and ψ . The tensor product space contains not only vectors of the form v ⊗ w , but also their linear combinations , some of which can not be expressed as φ⊗ ψ . For example , ∑1 j=0 ∑1 k=1 ψj⊗φk√ 4 can be decomposed as { ∑1 j=0 1√ 2 ψj } ⊗ { ∑1 k=1 1√ 2 φk } . On the other hand , ψ0⊗φ0+ψ1⊗φ1√ 2 can not ; no matter what we choose as coefficients a , b , c , d , we have 1√ 2 ψ0 ⊗ φ0 + 1√ 2 ψ1 ⊗ φ1 6= ( aψ0 + bψ1 ) ⊗ ( cφ0 + dφ1 ) = acψ0 ⊗ φ0 + bdψ1 ⊗ φ1 + adψ0 ⊗ φ1 + bcψ1 ⊗ φ0 , since we require ac = 1/ √ 2 , that is , a 6= 0 , c 6= 0 , and similarly bd = 1/ √ 2 , that is , b 6= 0 , c 6= 0 , yet we also require bd = ad = 0 , which is incompatible with a , b , c , d 6= 0 . For tensor product spaces of order n , that is , ⊗n j=1Hj , tensors of the form v = ⊗n j=1 vj , where vj ∈ Hj , are called simple . Tensor rank5 of a tensor v is the smallest number of simple tensors that sum up to v ; for example , ψ0⊗φ0+ψ1⊗φ1√ 2 is a tensor of rank 2 . Tensors with rank greater than one are called entangled . Maximum rank of a tensor in a tensor product space of order higher than two is not known in general ( Buczyński & Landsberg , 2013 ) . 2.3 THE word2ket EMBEDDINGS A p-dimensional word embedding model involving a d-token vocabulary is6 a mapping f : [ d ] → Rp , that is , it maps word identifiers into a p-dimensional real Hilbert space , an inner product space with the standard inner product 〈· , ·〉 leading to the L2 norm . Function f is trained to capture semantic information from the language corpus it is trained on , for example , two words i , j with 〈f ( i ) , f ( j ) 〉 ∼ 0 are expected to be semantically unrelated . In practical implementations , we represent f as a collection of vectors fi ∈ Rp indexed by i , typically in the form of d× p matrix M , with embeddings of individual words as rows . We propose to represent an embedding v ∈ Rp of each a single word as an entangled tensor . Specifically , in word2ket , we use tensor of rank r and order n of the form v = r∑ k=1 n⊗ j=1 vjk , ( 3 ) 4Note that some sources alternatively call n a degree or a rank of a tensor . Here , we use tensor rank to refer to a property similar to matrix rank , see below . 5Note that some authors use rank to denote what we above called order . In the nomenclature used here , a vector space of n×m matrices is isomorphic to a tensor product space of order 2 and dimensionality mn , and individual tensors in that space can have rank of up to min ( m , n ) . 6We write [ d ] = { 0 , ... , d } . where vjk ∈ Rq . The resulting vector v has dimension p = qn , but takes rnq = O ( rq log p/q ) space . We use q ≥ 4 ; it does not make sense to reduce it to q = 2 since a tensor product of two vectors in R2 takes the same space as a vector in R4 , but not every vector in R4 can be expressed as a rank-one tensor in R2 ⊗ R2 . If the downstream computation involving the word embedding vectors is limited to inner products of embedding vectors , there is no need to explicitly calculate the qn-dimensional vectors . Indeed , we have ( see eq . 2 ) 〈v , w〉 = 〈 r∑ k=1 n⊗ j=1 vjk , r∑ k′=1 n⊗ j=1 wjk′ 〉 = r , r∑ k , k′=1 n∏ j=1 〈vjk , wjk′〉 . Thus , the calculation of inner product between two p-dimensional word embeddings , v and w , represented via word2ket takes O ( r2q log p/q ) time and O ( 1 ) additional space . In most applications , a small number of embedding vectors do need to be made available for processing through subsequent neural network layers – for example , embeddings of all words in all sentences in a batch . For a batch consisting of b words , the total space requirement is O ( bp+ rq log p/q ) , instead of O ( dp ) in traditional word embeddings . Reconstructing a b-word batch of p-dimensional word embedding vectors from tensors of rank r and order n takesO ( brpn ) arithmetic operations . To facilitate parallel processing , we arrange the ordern tensor product space into a balanced tensor product tree ( see Figure 1 ) , with the underlying vectors vjk as leaves , and v as root . For example , for n = 4 , instead of v = ∑ k ( ( v1k ⊗ v2k ) ⊗ v3k ) ⊗ v4k we use v = ∑ k ( v1k ⊗ v2k ) ⊗ ( v3k ⊗ v4k ) . Instead of performing n multiplications sequentially , we can perform them in parallel along branches of the tree , reducing the length of the sequential processing to O ( log n ) . Typically , word embeddings are trained using gradient descent . The proposed embedding representation involves only differentiable arithmetic operations , so gradients with respect to individual elements of vectors vjk can always be defined . With the balanced tree structure , word2ket representation can be seen as a sequence of O ( log n ) linear layers with linear activation functions , where n is already small . Still , the gradient of the embedding vector v with respect to an underlying tunable parameters vlk involves products ∂ ( ∑ k ∏n j=1 vjk ) /∂vlk = ∏ j 6=l vjk , leading to potentially high Lipschitz constant of the gradient , which may harm training . To alleviate this problem , at each node in the balanced tensor product tree we use LayerNorm ( Ba et al. , 2016 ) . 3 LINEAR OPERATORS IN TENSOR PRODUCT SPACES AND word2ketXS

Summary: This paper proposes word2ket - a space-efficient form of storing word embeddings through tensor products. The idea is to factorize each d-dimensional vector into a tensor product of much smaller vectors (either with or without linear operators). While this results in a time cost for each word lookup, the space savings are enormous and can potentially impact several applications where the vocabulary size is too large to fit into processor memory (CPU or GPU). The experimental evaluation is done on several tasks like summarization, machine translation and question answering and convincingly demonstrates that one can achieve close to original model performance with very few parameters!