Massively Multilingual Sparse Word Representations

1 INTRODUCTION . Cross-lingual transferability of natural language processing models is of paramount importance for low-resource languages which often lack a sufficient amount of training data for various NLP tasks . A series of attempts have been made to remedy the shortage of labeled training data . Both part-ofspeech tagging ( Fang & Cohn , 2017 ; Zhang et al. , 2016 ; Haghighi & Klein , 2006 ; Gouws & Søgaard , 2015 ; Kim et al. , 2015 ; Agić et al. , 2015 ) and dependency parsing ( Guo et al. , 2015 ; Agić et al. , 2016 ) have been investigated from that perspective . The mapping of distributed word representations of low-resource languages to the embedding space of a resource-rich language makes NLP models trained on the source language directly applicable for texts in resource-scarce languages . Overcomplete word representations aim at expressing low dimensional distributed word representations as a sparse linear combination of an overcomplete set of basis vectors . Using sparse word representations has not only been reported to give better performance compared to dense representations for certain problems ( Yogatama et al. , 2015 ; Faruqui et al. , 2015 ; Sun et al. , 2016 ; Berend , 2017 ) , but it is also argued to provide increased interpretability ( Murphy et al. , 2012 ; Vyas & Carpuat , 2016 ; Park et al. , 2017 ; Subramanian et al. , 2018 ) . Such bilingual sparse representations have straightforward benefits as any machine learning model that is trained on the labeled training data using the sparse representation of some resource-rich source language can be used directly and interchangeably for texts written in some target language . This way we can enjoy the benefits of sparsity such as smaller and more interpretable models . In this work , we propose a new algorithm for determining multilingual sparse word representations in which words with similar meaning across different languages are described with similar sparse vectors . Our work has multiple advantageous properties compared to previous similar attempts ( Vyas & Carpuat , 2016 ; Upadhyay et al. , 2018 ) . Firstly , our algorithm naturally fits multilingual setting , i.e . it is not limited to pairs of languages , but is capable of determining cross-lingual sparse embeddings for multiple languages . Our algorithm differs from previous approaches in that we obtain target language sparse word representations via a series of convex optimization problems with a substantially reduced parameter set , whereas previous solutions perform non-convex optimization for orders of more parameters . The model formulation we propose not only speeds up computation drastically , but our empirical results also suggest that the proposed sparse word vectors perform better when employed in various extrinsic tasks , i.e . dependency parsing , document classification and natural language inference . We make our sparse embeddings for 27 languages and the source code that we used to obtain them publicly available at https : //github.com/begab/mamus . 2 MAMUS – THE PROPOSED ALGORITHM . In order to facilitate the introduction of the MAssively MUltilingual Sparse ( dubbed as MAMUS ) word representations , we first introduce our notation . In what follows , we denote dense word embedding matrices of the source and target languages as S ∈ Rm×|Vs| and T ∈ Rn×|Vt| , respectively , with Vs and Vt indicating the vocabulary of the source and target languages . The m = n condition does not have to hold in general , however , it is going to be the case throughout the paper . We shall denote some symbolic word form as x and its corresponding distributed vectorial representation in boldface , i.e . x . 2.1 PREPROCESSING OF INPUT WORD EMBEDDINGS . Our algorithm takes as input a pair of “ traditionally ” trained distributed dense word embeddings such as Glove ( Pennington et al. , 2014 ) , word2vec ( Mikolov et al. , 2013a ) or fasttext ( Bojanowski et al. , 2017 ) . It is important to note that we do not assume the existence of parallel text , meaning that we can train the input word embeddings in total isolation from each other . We transform the dense input embedding matrices so that the word vectors comprising it have unit norm , as it is frequently met in the literature ( Xing et al. , 2015 ; Artetxe et al. , 2016 ; 2017 ) . This preprocessing step ensures that the dot product of word embeddings equals their cosine similarity . Unit normalization of word embeddings also makes cross-lingual comparison more natural , since all embeddings have identical length , irrespective of the language they belong to . 2.2 MAPPING OF WORD EMBEDDINGS . In order to align independently trained word representations , we learn a linear mapping W . The mapping is expected to bring target language word vectors close to their semantically equivalent counterparts in the source embedding space . As proposed by Mikolov et al . ( 2013b ) suchW can be simply defined by minimizing l∑ i=1 ‖si−W ti‖ , with { ( si , ti ) } li=1 being a set of word pairs which are cross-lingual equivalents of each other . If we construct matrices So = [ si ] li=1 and To = [ ti ] l i=1 by stacking the embeddings of translation pairs , we can express the solution as SoT+o , with T + o denoting the Moore-Penrose pseudoinverse of To . Multiple studies have argued that ensuring W to be orthonormal can significantly improve the quality of the mapping of word embeddings across languages ( Smith et al. , 2017 ; Xing et al. , 2015 ; Artetxe et al. , 2016 ; Hamilton et al. , 2016 ) . Finding the optimal orthonormal W can be viewed as an instance of the orthogonal Procrustes problem ( Schönemann , 1966 ) which can be solved by W⊥ = UV , with U and V coming from the singular value decomposition of the matrix product SᵀoTo . By applying an isometric transformation , we ensure that transformed embeddings preserve their norm . 2.3 SPARSE CODING OF THE EMBEDDING SPACES . Sparse coding decomposes a X ∈ Rm×v matrix of signals ( word embeddings in our case ) into the product of a dictionary matrix D ∈ Rm×k and a matrix of sparse coefficients α ∈ Rk×v , where k denotes the number of basis vectors to be employed . The columns of D form an overcomplete set of basis vectors and the sparse nonzero coefficients in the ith column of α indicate which column vectors from D should be incorporated in the reconstruction of xi . We minimize the objective min D∈C , α∈Rk×|v|≥0 1 2 ‖X −Dα‖2F + λ‖α‖1 , ( 1 ) where C denotes the convex set of matrices with column norm at most 1 and the sparse coefficients in α are required to be non-negative . Ensuring that all the coefficients in α are non-negative makes their cross-lingual comparison more natural , as the signs of sparse word coefficients can not mismatch . Additionally , non-negativity has been reported to provide increased interpretability ( Murphy et al. , 2012 ) . We used the SPAMS library ( Mairal et al. , 2009 ) for calculating D and α . We can perform sparse coding as defined in ( 1 ) for the independently created source and target embedding matrices S and T , obtaining decompositions S ≈ Dsαs and T ≈ Dtαt . If we do so , however , sparse word representations extractable from matrices αs and αt are not comparable to any extent due to the fact that S and T are decomposed independently of each other . We propose to first solve a single instance of the non-convex dictionary learning problem as defined in ( 1 ) for the source language alone as if our goal was to obtain monolingual sparse word representations . After determining Ds , we can apply this dictionary matrix to find sparse coefficient matrices for the isometrically transformed embeddings of all the target languages , hence solving a much simpler complex optimization problem with reasonably fewer parameters in the case of target languages of the form min α∈Rk×|v|≥0 1 2 ‖W⊥X −Dsα‖2F + λ‖α‖1 , ( 2 ) with W⊥ denoting the isometric transformation specific to a given source-target language pair . We summarize the working mechanisms of MAMUS in Algorithm 1 . Algorithm 1 Pseudocode of MAMUS Require : source and target embeddings S ∈ Rm×|Vs| , T1 ∈ Rm×|Vt1 | , . . . , TN ∈ Rm×|VtN | semantically equivalent word pairs { ( s ( i ) 1 , t ( i ) 1 ) } l1 i=1 , . . . , { ( s ( i ) N , t ( i ) N ) } lN i=1 Ensure : Sparse representation matrices ( αs , αt1 , . . . , αtN ) procedure MAMUS ( S , T ) S ← UNITNORMALIZE ( S ) D∗s , α ∗ s ← argmin α , D ‖S −Dα‖F + λ‖α‖1 for k ← 1 to N do Tk ← UNITNORMALIZE ( Tk ) W ∗⊥ ← argmin WᵀW=I lk∑ i=1 ‖s ( i ) k −W t ( i ) k ‖ αtk ← argmin α∈R≥0 ‖W ∗⊥Tk −D∗sα‖F + λ‖α‖1 end for return αs , αt1 , . . . , αtN end procedure 2.4 DECISIVE DIFFERENCES TO BISPARSE . The goal of the BISPARSE ( Vyas & Carpuat , 2016 ) is to determine such sparse word representations that behave similarly for a pair of languages . We subsequently juxtapose the conceptual differences between BISPARSE and MAMUS both from theoretical and practical perspectives . BISPARSE ( Vyas & Carpuat , 2016 ) determines sparse bilingual word representations by solving min Ds , Dt∈Rm×k , αs∈Rk×|Vs|≥0 , αt∈Rk×|Vt|≥0 1 2 ( ‖Xs−Dsαs‖2F+‖Xt−Dtαt‖2F+λx |Vs|∑ i=1 |Vt|∑ j=1 Mij‖α ( i ) s −α ( j ) t ‖22 ) +λt‖αt‖1+λs‖αs‖1 , ( 3 ) where M is a pairwise similarity matrix between the source and target language word pairs , λs , λt and λx denotes the regularization coefficients for the source , target language and the cross-lingual loss , respectively . Finally , α ( i ) s and α ( j ) t refers to the i th and jth columns of matrices αs and αt , respectively . As a first notable difference , BISPARSE operates by employing an explicit |Vs| × |Vt| pairwise similarity matrix , whereas MAMUS relies on a list of word translation pairs , which is used to rotate all the target language embeddings into the embedding space of the source language . BISPARSE hence requires an additional hyperparameter which controls for the strength of the cross-lingual regularization ( cf . λx ) . According to our experiments choosing this hyperparameter is a crucial factor in employing BISPARSE . The objective in ( 3 ) reveals that the original BISPARSE model optimizes the source and target word representations jointly , i.e . a separate source language representation is created whenever applied to a new target . This behavior of BISPARSE limits its application for a single pair of languages , instead of the more general multilingual setting , when our goal is to represent more than two languages at a time . As the default behavior of BISPARSE hampers truly multilingual application , we adopt it similar to MAMUS , i.e . we first train sparse word representations for the source language by solving min Ds∈Rm×k , αs∈Rk×|Vs|≥0 1 2 ‖Xs −Dsαs‖2F + λs‖αs‖1 , ( 4 ) with the source language being chosen as English . Once the dictionary matrix of semantic atoms ( D∗s ) and the sparse coefficients ( α ∗ s ) are calculated for the source language , we next solve min Dt∈Rm×k , αt∈Rk×|Vt|≥0 1 2 ‖Xt −Dtαt‖2F + λt‖αt‖1 + 1 2 λx |Vs|∑ i=1 |Vt|∑ j=1 Mij‖α∗ ( i ) s −α ( j ) t ‖22 . ( 5 ) Identical to BISPARSE ( Vyas & Carpuat , 2016 ) , we solved the modified optimization problems in ( 4 ) and ( 5 ) relying on the efficient Forward-Backward Splitting Solver implementation of the FASTA framework ( Goldstein et al. , 2014 ; 2015 ) . To summarize , the way BISPARSE works in its original formulation is that it creates sparse word representations for pairs of languages , both for the source and target language . One compelling characteristic of MAMUS , however , is that it learns the Ds and αs parameters independent from the target embedding space , thus the same decomposition of source language embeddings can be utilized for multiple target languages . This makes our algorithm conveniently applicable to an arbitrary number of languages . Furthermore , by using the dictionary matrix D∗s for all the target languages , MAMUS not only involves optimization over a substantially reduced set of parameters , it additionally enjoys the ease of solving a series a convex optimization problems . The model formulation of BISPARSE on the other hand is such , that it requires to solve a series of highly non-convex optimization problems for each of the target languages , since it requires the optimization over Dt as well . This property is also true for the modified version of BISPARSE which is adapted for multilingual usage . We will refer to this adapted version of BISPARSE as MULTISPARSE for the rest of the paper .

Summary: This paper describes a method to build sparse multilingual word vectors that is designed to scale easily to many languages. The key idea is to pick one language to be the source language, and then to build word embeddings and then a sparse dictionary + sparse coefficients  for that source language monolingually. Other target languages then first align their embedding to the source using a seed list of translations and standard techniques, and then determine their sparse coefficients based on the fixed source sparse dictionary. This latter process is a convex optimization, which improves efficiency and stability. The method is tested with an established correlation-based intrinsic metric (QVEC-CCA) as well as by using the multilingual embeddings to project systems for cross-lingual document classification, dependency parsing and natural language inference.