Unsupervised Hyper-alignment for Multilingual Word Embeddings

We consider the problem of aligning continuous word representations , learned in multiple languages , to a common space . It was recently shown that , in the case of two languages , it is possible to learn such a mapping without supervision . This paper extends this line of work to the problem of aligning multiple languages to a common space . A solution is to independently map all languages to a pivot language . Unfortunately , this degrades the quality of indirect word translation . We thus propose a novel formulation that ensures composable mappings , leading to better alignments . We evaluate our method by jointly aligning word vectors in eleven languages , showing consistent improvement with indirect mappings while maintaining competitive performance on direct word translation . 1 INTRODUCTION . Pre-trained continuous representations of words are standard building blocks of many natural language processing and machine learning systems ( Mikolov et al. , 2013b ) . Word vectors are designed to summarize and quantify semantic nuances through a few hundred coordinates . Such representations are typically used in downstream tasks to improve generalization when the amount of data is scarce ( Collobert et al. , 2011 ) . The distributional information used to learn these word vectors derives from statistical properties of word co-occurrence found in large corpora ( Deerwester et al. , 1990 ) . Such corpora are , by design , monolingual ( Mikolov et al. , 2013b ; Bojanowski et al. , 2016 ) , resulting in the independent learning of word embeddings for each language . A limitation of these monolingual embeddings is that it is impossible to compare words across languages . It is thus natural to try to combine all these word representations into a common multilingual space , where every language could be mapped . Mikolov et al . ( 2013a ) observed that word vectors learned on different languages share a similar structure . More precisely , two sets of pre-trained vectors in different languages can be aligned to some extent : a linear mapping between the two sets of embeddings is enough to produce decent word translations . Recently , there has been an increasing interest in mapping these pre-trained vectors in a common space ( Xing et al. , 2015b ; Artetxe et al. , 2017 ) , resulting in many publicly available embeddings in many languages mapped into a single common vector space ( Smith et al. , 2017 ; Conneau et al. , 2017 ; Joulin et al. , 2018 ) . The quality of these multilingual embeddings can be tested by composing mappings between languages and looking at the resulting translations . As an example , learning a direct mapping between Italian and Portuguese leads to a word translation accuracy of 78.1 % with a nearest neighbor ( NN ) criterion , while composing the mapping between Italian and English and Portuguese and English leads to a word translation accuracy of 70.7 % only . Practically speaking , it is not surprising to see such a degradation since these bilingual alignments are trained separately , without enforcing transitivity . In this paper , we propose a novel approach to align multiple languages simultaneously in a common space in a way that enforces transitive translations . Our method relies on constraining word translations to be coherent between languages when mapped to the common space . Nakashole and Flauger ( 2017 ) has recently shown that similar constraints over a well chosen triplet of languages improve supervised bilingual alignment . Our work extends their conclusions to the unsupervised case . We show that our approach achieves competitive performance while enforcing composition . 2 PRELIMINARIES ON BILINGUAL ALIGNMENT . In this section , we provide a brief overview of bilingual alignment methods to learn a mapping between two sets of embeddings , and discuss their limits when used in multilingual settings . 2.1 SUPERVISED BILINGUAL ALIGNMENT . Mikolov et al . ( 2013a ) formulate the problem of word vector alignment as a quadratic problem . Given two sets of n word vectors stacked in two n × d matrices X and Y and a n × n assignment matrix P ∈ { 0 , 1 } n×n built using a bilingual lexicon , the mapping matrix Q ∈ Rd×d is the solution of the least-square problem : min Q∈Rd×d ‖XQ − PY‖22 , which admits a closed form solution . Restraining Q to the set of orthogonal matrices Od , improves the alignments ( Xing et al. , 2015b ) . The resulting problem , known as Orthogonal Procrustes , still admits a closed form solution through a singular value decomposition ( Schnemann , 1966 ) . Alternative loss function . The ` 2 loss is intrinsically associated with the nearest neighbor ( NN ) criterion . This criterion suffers from the existence of “ hubs ” , which are data points that are nearest neighbors to many other data points ( Dinu et al. , 2014 ) . Alternative criterions have been suggested , such as the inverted softmax ( Smith et al. , 2017 ) and CSLS ( Conneau et al. , 2017 ) . Recently , Joulin et al . ( 2018 ) has shown that directly minimizing a loss inspired by the CSLS criterion significantly improve the quality of the retrieved word translations . Their loss function , called RCSLS , is defined as : RCSLS ( x , y ) = −2x > y + 1 k ∑ y∈NY ( x ) x > y + 1 k ∑ x∈NX ( y ) x > y . ( 1 ) This loss is a tight convex relaxation of the CSLS cristerion for normalized word vectors , and can be efficiently minimized with a subgradient method . 2.2 UNSUPERVISED BILINGUAL ALIGNMENT : WASSERSTEIN-PROCRUSTES . In the setting of unsupervised bilingual alignment , the assignment matrix P is unknown and must be learned jointly with the mapping Q . An assignment matrix represents a one-to-one correspondence between the two sets of words , i.e. , is a bi-stochastic matrix with binary entries . The set of assignment matrices Pn is thus defined as : Pn = Bn ∩ { 0 , 1 } n×n , where Bn = { P ∈ Rn×n+ , P1n = 1n , P > 1n = 1n } . The resulting approach , called Wasserstein-Procrustes ( Zhang et al. , 2017a ; Grave et al. , 2018 ) , jointly learns both matrices by solving the following problem : min Q∈Od min P∈Pn ‖XQ−PY‖22 . ( 2 ) This problem is not convex since neither of the sets Pn and Od are convex . Minimizing over each variable separately leads , however , to well understood optimization problems : when P is fixed , minimizing over Q involves solving the orthogonal Procrustes problem . When Q is fixed , an optimal permutation matrix P can be obtained with the Hungarian algorithm . A simple heuristic to address Eq . ( 2 ) is thus to use an alternate optimization . Both algorithms have a cubic complexity but on different quantities : Procrustes involves the dimension of the vectors , i.e. , O ( d3 ) ( with d = 300 ) , whereas the Hungarian algorithm involves the size of the sets , i.e. , O ( n3 ) ( with n = 20k–200k ) . Figure 2 : Plain black lines indicate pairs of languages for which a translation matrix is learned . Dashed red lines indicate pairs used in the loss functions . Languages are aligned onto a common pivot language , i.e. , English . Directly applying the Hungarian algorithm is computationally prohibitive , but efficient alternatives exist . Cuturi ( 2013 ) shows that regularizing this problem with a negative entropy leads to a Sinkhorn algorithm , with a complexity of O ( n2 ) up to logarithmic factors ( Altschuler et al. , 2017 ) . As for many non-convex problems , a good initial guess helps converge to better local minima . Grave et al . ( 2018 ) compute an initial P with a convex relaxation of the quadratic assignment problem . We found , however , that the entropic regularization of the Gromov-Wasserstein ( GW ) problem ( Mémoli , 2011 ) worked well in practice and was significantly faster ( Solomon et al. , 2016 ; Peyré et al. , 2016 ) : min P∈Pn ∑ i , j , i′ , j′ ( ‖xi − xi′‖2 − ‖yj − yj′‖2 ) 2 P ( i , j ) P ( i′ , j′ ) + ∑ i , j P ( i , j ) logP ( i , j ) . The case ε = 0 corresponds to Mémoli ’ s initial proposal . Optimizing the regularized version ( > 0 ) leads to a local minimum P̂ that can be used as an initialization to solve Eq . ( 2 ) . Note that a similar formulation was recently used in the same context by Alvarez-Melis and Jaakkola ( 2018 ) . 3 COMPOSABLE MULTILINGUAL ALIGNMENTS . In this section , we propose an unsupervised approach to jointly align N sets of vectors to a unique common space while preserving the quality of word translation between every pair of languages . 3.1 MULTILINGUAL ALIGNMENT TO A COMMON SPACE . Given N sets of word vectors , we are interested in aligning these to a common target space T . For simplicity , we assume that this target space coincide with one of the word vector set . The language associated with this vector set is denoted as “ pivot ” and is indexed by i = 0 . A typical choice for the pivot , used in publicly available aligned vectors , is English ( Smith et al. , 2017 ; Conneau et al. , 2017 ; Joulin et al. , 2018 ) . Aligning multiple languages to a common space consists in learning a mapping Qi for each language i , such that its vectors are aligned with the vectors of the pivot language up to a permutation matrix Pi : min Qi∈Od , Pi∈Pn ∑ i ` ( XiQi , PiX0 ) , ( 3 ) This objective function decomposes over each language and does not guarantee good indirect word translation between pairs of languages that do not include the pivot . A solution would be to directly enforce compositionality by adding constraints on the mappings . However , this would require to introduce mappings between all pairs of languages , leading to the estimation of O ( N2 ) mappings simultaneously . Instead we leverage the fact that all the vector sets are mapped to a common space to enforce good alignments within this space . With the convention that Q0 is equal to the identity , this leads to the following problem : min Qi∈Od , Pij∈Pn ∑ i , j αij ` ( XiQi , PijXjQj ) , ( 4 ) where αij > 0 weights the importance of the alignment quality between the languages i and j . This formulation does not introduce any unnecessary mapping . It constrains all pairs of vector sets to be well aligned , instead of directly constraining the mappings . Constraining the mappings would encourage coherence over the entire space , while we are interested in well aligned data ; that is coherent mapping within the span of the word vectors . Our approach takes its inspiration from the hyperalignment of multiple graphs ( Goodall , 1991 ) . We refer to our approach as Unsupervised Multilingual Hyperalignment ( UMH ) . Choice of weights . The weights on the alignments can be chosen to reflect prior knowledge about the relations between languages . For example , these weights can be a function of a rough language similarity measure extracted from the initial alignment . It is not clear however how the weights should depend on these similarities : constraining similar languages with higher weights may be unnecessary , since they are already well aligned . On the other hand , increasing weights of distant languages may lead to a problem where alignments are harder to learn . In practice , we find that simple weighting schemes work best with little assumptions . For instance , uniform weights work well but tends to degrade the performance of translations to and from the pivot if the number of languages is large . Instead , we choose to use larger weights for direct alignments to the pivot , to insure that our multilingual objective does not degrade bilingual alignments to the pivot . In practice , αij is set to N if i or j is equal to 0 , and 1 otherwise . Choice of loss function . We consider the RCSLS loss of Joulin et al . ( 2018 ) . This loss is computationally expensive and wasteful to minimize it from scratch . We thus optimize a ` 2 for the first couple of epochs before switching to the RCSLS loss . This two-step procedure shares some similarities with the use of a refinement step after learning a first rough alignment ( Artetxe et al. , 2017 ; Conneau et al. , 2017 ) . The ` 2 loss between two sets of normalized vectors is equivalent to a linear function , up to an additive constant C : ` 2 ( QX , PY ) = −2tr ( QTXTPY ) + C ( 5 ) Where Q = QiQTj if X and Y are the vectors from languages i and j . We adapt the RCSLS loss of Eq . ( 1 ) to the unsupervised case by applying an assignment matrix to the target vectors : RCSLS ( XQ , PY ) = −2tr ( Q > X > PY ) + 1 k ∑ x∈X , z∈NPY ( Q > x ) z > Q > x+ ∑ y∈PY , Q > z∈NX ( y ) z > Qy . Efficient optimization . Directly optimizing Eq . ( 4 ) is computationally prohibitive sinceN2 terms are involved . We use a stochastic procedure where N pairs ( i , j ) of languages are sampled at each iteration and updated . We sample pairs according to the weights αij . The RCSLS loss being slower to compute than the ` 2 loss , we first optimize the latter for a couple of epochs before switching to the former . The ` 2 loss is optimized with the same procedure as in Grave et al . ( 2018 ) : alternate minimization with a Sinkhorn algorithm for the assignment matrix on small batches . We then switch to a cruder optimization scheme when minimizing the RCSLS loss : we use a greedy assignment algorithm by taking the maximum per row . We also subsample the number of elements to compute the K nearest neighbors from . We restrict each set of vectors to its first 20k elements . UMH runs on a CPU with 10 threads in less than 10 minutes for a pair of languages and in 2 hours for 6 languages .

Summary: The authors present a method for unsupervised alignment of word across multiple languages. In particular, they extend an existing unsupervised bilingual alignment to the case of multiple languages by adding constraints to the optimization problem. The main aim is to ensure that the embeddings can now be composed and the performance (alignment quality) does not degrade across multiple compositions.