word2ket: Space-efficient Word Embeddings inspired by Quantum Entanglement

1 INTRODUCTION . Modern deep learning approaches for natural language processing ( NLP ) often rely on vector representation of words to convert discrete space of human language into continuous space best suited for further processing through a neural network . For a language with vocabulary of size d , a simple way to achieve this mapping is to use one-hot representation – each word is mapped to its own row of a d× d identity matrix . There is no need to actually store the identity matrix in memory , it is trivial to reconstruct the row from the word identifier . Word embedding approaches such as word2vec ( Mikolov et al. , 2013 ) or GloVe ( Pennington et al. , 2014 ) use instead vectors of dimensionality p much smaller than d to represent words , but the vectors are not necessarily extremely sparse nor mutually orthogonal . This has two benefits : the embeddings can be trained on large text corpora to capture the semantic relationship between words , and the downstream neural network layers only need to be of width proportional to p , not d , to accept a word or a sentence . We do , however , need to explicitly store the d× p embedding matrix in GPU memory for efficient access during training and inference . Vocabulary sizes can reach d = 105 or 106 ( Pennington et al. , 2014 ) , and dimensionality of the embeddings used in current systems ranges from p = 300 ( Mikolov et al. , 2013 ; Pennington et al. , 2014 ) to p = 1024 ( Devlin et al. , 2018 ) . The d × p embedding matrix thus becomes a substantial , often dominating , part of the parameter space of a learning model . In classical computing , information is stored in bits – a single bit represents an element from the set B = { 0 , 1 } , it can be in one of two possible states . A quantum equivalent of a bit , a qubit , is ∗Corresponding author . 1PyTorch implementation available at : https : //github.com/panaali/word2ket fully described by a single two-dimensional complex unit-norm vector , that is , an element from the set C2 . A state of an n-qubit quantum register corresponds to a vector in C2n . To have exponential dimensionality of the state space , though , the qubits in the register have to be interconnected so that their states can become entangled ; a set of all possible states of n completely separated , independent qubits can be fully represented by C2n instead of C2n . Entanglement is a purely quantum phenomenon – we can make quantum bits interconnected , so that a state of a two-qubit system can not be decomposed into states of individual qubits . We do not see entanglement in classical bits , which are always independent – we can describe a byte by separately listing the state of each of the eight bits . We can , however , approximate quantum register classically - store vectors of size m using O ( logm ) space , at the cost of losing the ability to express all possible m-dimensional vectors that an actual O ( logm ) -qubit quantum register would be able to represent . As we show in this paper , the loss of representation power does not have a significant impact on NLP machine learning algorithms that use the approximation approaches to store and manipulate the high-dimensional word embedding matrix . 1.1 OUR CONTRIBUTION . Here , we used approaches inspired by quantum computing to propose two related methods , word2ket and word2ketXS , for storing word embedding matrix during training and inference in a highly efficient way2 . The first method operates independently on the embedding of each word , allowing for more efficient processing , while the second method operates jointly on all word embeddings , offering even higher efficiency in storing the embedding matrix , at the cost of more complex processing . Empirical evidence from three NLP tasks shows that the new word2ket embeddings offer high space saving rate at little cost in terms of accuracy of the downstream NLP model . 2 FROM TENSOR PRODUCT SPACES TO word2ket EMBEDDINGS 2.1 TENSOR PRODUCT SPACE . Consider two separable3 Hilbert spaces V and W . A tensor product space of V and W , denoted as V ⊗ W , is a separable Hilbert space H constructed using ordered pairs v ⊗ w , where v ∈ V and w ∈ W . In the tensor product space , the addition and multiplication in H have the following properties c { v ⊗ w } = { cv } ⊗ w = v ⊗ { cw } , ( 1 ) v ⊗ w + v′ ⊗ w = { v + v′ } ⊗ w , v ⊗ w + v ⊗ w′ = v ⊗ { w + w′ } . The inner product between v ⊗ w and v′ ⊗ w′ is defined as a product of individual inner products 〈v ⊗ w , v′ ⊗ w′〉 = 〈v , v′〉 〈w , w′〉 . ( 2 ) It immediately follows that ||v ⊗ w|| = ||v|| ||w|| ; in particular , a tensor product of two unit-norm vectors , from V andW , respectively , is a unit norm vector in V ⊗W . The Hilbert space V ⊗W is a space of equivalence classes of pairs v ⊗ w ; for example { cv } ⊗ w and v ⊗ { cw } are equivalent ways to write the same vector . A vector in a tensor product space is often simply called a tensor . Let { ψj } and { φk } be orthonormal basis sets in V andW , respectively . From eq . 1 and 2 we can see that ∑ j cjψj ⊗ { ∑ k dkφk } = ∑ j ∑ k cjdkψj ⊗ φk , 〈ψj ⊗ φk , ψj′ ⊗ φk′〉 = δj−j′δk−k′ , where δz is the Kronecker delta , equal to one at z = 0 and to null elsewhere . That is , the set { ψj ⊗ φk } jk forms an orthonormal basis in V ⊗ W , with coefficients indexed by pairs jk and 2In Dirac notation popular in quantum mechanics and quantum computing , a vector u ∈ C2 n is written as |u〉 , and called a ket . 3That is , with countable orthonormal basis . numerically equal to the products of the corresponding coefficients in V and W . We can add any pairs of vectors in the new spaces by adding the coefficients . The dimensionality of V ⊗W is the product of dimensionalities of V andW . We can create tensor product spaces by more than one application of tensor product , H = U⊗V⊗W , with arbitrary bracketing , since tensor product is associative . Tensor product space of the form n⊗ j=1 Hj = H1 ⊗H2 ⊗ . . .⊗Hn is said to have tensor order4 of n . 2.2 ENTANGLED TENSORS . Consider H = V ⊗ W . We have seen the addition property v ⊗ w + v′ ⊗ w = { v + v′ } ⊗ w and similar property with linearity in the first argument – tensor product is bilinear . We have not , however , seen how to express v ⊗ w + v′ ⊗ w′ as φ ⊗ ψ for some φ ∈ V , ψ ∈ W . In many cases , while the left side is a proper vector from the tensor product space , it is not possible to find such φ and ψ . The tensor product space contains not only vectors of the form v ⊗ w , but also their linear combinations , some of which can not be expressed as φ⊗ ψ . For example , ∑1 j=0 ∑1 k=1 ψj⊗φk√ 4 can be decomposed as { ∑1 j=0 1√ 2 ψj } ⊗ { ∑1 k=1 1√ 2 φk } . On the other hand , ψ0⊗φ0+ψ1⊗φ1√ 2 can not ; no matter what we choose as coefficients a , b , c , d , we have 1√ 2 ψ0 ⊗ φ0 + 1√ 2 ψ1 ⊗ φ1 6= ( aψ0 + bψ1 ) ⊗ ( cφ0 + dφ1 ) = acψ0 ⊗ φ0 + bdψ1 ⊗ φ1 + adψ0 ⊗ φ1 + bcψ1 ⊗ φ0 , since we require ac = 1/ √ 2 , that is , a 6= 0 , c 6= 0 , and similarly bd = 1/ √ 2 , that is , b 6= 0 , c 6= 0 , yet we also require bd = ad = 0 , which is incompatible with a , b , c , d 6= 0 . For tensor product spaces of order n , that is , ⊗n j=1Hj , tensors of the form v = ⊗n j=1 vj , where vj ∈ Hj , are called simple . Tensor rank5 of a tensor v is the smallest number of simple tensors that sum up to v ; for example , ψ0⊗φ0+ψ1⊗φ1√ 2 is a tensor of rank 2 . Tensors with rank greater than one are called entangled . Maximum rank of a tensor in a tensor product space of order higher than two is not known in general ( Buczyński & Landsberg , 2013 ) . 2.3 THE word2ket EMBEDDINGS A p-dimensional word embedding model involving a d-token vocabulary is6 a mapping f : [ d ] → Rp , that is , it maps word identifiers into a p-dimensional real Hilbert space , an inner product space with the standard inner product 〈· , ·〉 leading to the L2 norm . Function f is trained to capture semantic information from the language corpus it is trained on , for example , two words i , j with 〈f ( i ) , f ( j ) 〉 ∼ 0 are expected to be semantically unrelated . In practical implementations , we represent f as a collection of vectors fi ∈ Rp indexed by i , typically in the form of d× p matrix M , with embeddings of individual words as rows . We propose to represent an embedding v ∈ Rp of each a single word as an entangled tensor . Specifically , in word2ket , we use tensor of rank r and order n of the form v = r∑ k=1 n⊗ j=1 vjk , ( 3 ) 4Note that some sources alternatively call n a degree or a rank of a tensor . Here , we use tensor rank to refer to a property similar to matrix rank , see below . 5Note that some authors use rank to denote what we above called order . In the nomenclature used here , a vector space of n×m matrices is isomorphic to a tensor product space of order 2 and dimensionality mn , and individual tensors in that space can have rank of up to min ( m , n ) . 6We write [ d ] = { 0 , ... , d } . where vjk ∈ Rq . The resulting vector v has dimension p = qn , but takes rnq = O ( rq log p/q ) space . We use q ≥ 4 ; it does not make sense to reduce it to q = 2 since a tensor product of two vectors in R2 takes the same space as a vector in R4 , but not every vector in R4 can be expressed as a rank-one tensor in R2 ⊗ R2 . If the downstream computation involving the word embedding vectors is limited to inner products of embedding vectors , there is no need to explicitly calculate the qn-dimensional vectors . Indeed , we have ( see eq . 2 ) 〈v , w〉 = 〈 r∑ k=1 n⊗ j=1 vjk , r∑ k′=1 n⊗ j=1 wjk′ 〉 = r , r∑ k , k′=1 n∏ j=1 〈vjk , wjk′〉 . Thus , the calculation of inner product between two p-dimensional word embeddings , v and w , represented via word2ket takes O ( r2q log p/q ) time and O ( 1 ) additional space . In most applications , a small number of embedding vectors do need to be made available for processing through subsequent neural network layers – for example , embeddings of all words in all sentences in a batch . For a batch consisting of b words , the total space requirement is O ( bp+ rq log p/q ) , instead of O ( dp ) in traditional word embeddings . Reconstructing a b-word batch of p-dimensional word embedding vectors from tensors of rank r and order n takesO ( brpn ) arithmetic operations . To facilitate parallel processing , we arrange the ordern tensor product space into a balanced tensor product tree ( see Figure 1 ) , with the underlying vectors vjk as leaves , and v as root . For example , for n = 4 , instead of v = ∑ k ( ( v1k ⊗ v2k ) ⊗ v3k ) ⊗ v4k we use v = ∑ k ( v1k ⊗ v2k ) ⊗ ( v3k ⊗ v4k ) . Instead of performing n multiplications sequentially , we can perform them in parallel along branches of the tree , reducing the length of the sequential processing to O ( log n ) . Typically , word embeddings are trained using gradient descent . The proposed embedding representation involves only differentiable arithmetic operations , so gradients with respect to individual elements of vectors vjk can always be defined . With the balanced tree structure , word2ket representation can be seen as a sequence of O ( log n ) linear layers with linear activation functions , where n is already small . Still , the gradient of the embedding vector v with respect to an underlying tunable parameters vlk involves products ∂ ( ∑ k ∏n j=1 vjk ) /∂vlk = ∏ j 6=l vjk , leading to potentially high Lipschitz constant of the gradient , which may harm training . To alleviate this problem , at each node in the balanced tensor product tree we use LayerNorm ( Ba et al. , 2016 ) . 3 LINEAR OPERATORS IN TENSOR PRODUCT SPACES AND word2ketXS

Summary: The paper presents two methods to learn word embedding matrices that can be stored in much less space compared to traditional d x p embedding matrices, where d is the vocabulary size and p is the embedding size. Two methods are proposed: the first method estimates a p-dimensional embedding for a word as a sum of r tensor products of order n (tensor product of n q-dimensional embeddings).  This representation takes rnq parameters which can be much less than p, since p = q^n. The second method factorizes a full d x p embedding matrix jointly as a tensor product of much smaller t x q matrices and can obtain even larger space savings. Algorithms for efficiently computing full p-dimensional representations are also included. When only dot products are needed, the p-dimensional representations do not need to be explicitly constructed.