BLISS in Non-Isometric Embedding Spaces

1 INTRODUCTION . Bilingual lexicon induction ( BLI ) , the task of finding corresponding words in two languages from comparable corpora ( Haghighi et al. , 2008 ; Xing et al. , 2015 ; Zhang et al. , 2017b ; Artetxe et al. , 2017 ; Lample et al. , 2018 ) , finds use in numerous NLP tasks like POS tagging ( Zhang et al. , 2016 ) , parsing ( Xiao & Guo , 2014 ) , document classification ( Klementiev et al. , 2012 ) , and machine translation ( Irvine & Callison-Burch , 2013 ; Qi et al. , 2018 ) . Most work on BLI uses methods that learn a mapping between two word embedding spaces ( Ruder , 2017 ) , which makes it possible to leverage pre-trained embeddings learned on large monolingual corpora . A commonly used method for BLI , which also empirically works well , involves learning an orthogonal mapping between the two embedding spaces ( Mikolov et al . ( 2013a ) , Xing et al . ( 2015 ) , Artetxe et al . ( 2016 ) , Smith et al . ( 2017 ) ) . However , learning an orthogonal mapping inherently assumes that the embedding spaces for the two languages are isometric ( subsequently referred to as the orthogonality assumption ) . This is a particularly strong assumption that may not necessarily hold true , and consequently we can expect methods relying on this assumption to provide sub-optimal results . In this work , we examine this assumption , identify where it breaks down , and propose a method to alleviate this problem . We first formally describe this orthogonality assumption , and then present a theoretically motivated approach based on the Gromov-Hausdroff ( GH ) distance to check the validity and extent of the orthogonality assumption ( §2 ) . We show that the constraint indeed does not hold , particularly for etymologically distant language pairs . SpeMotivated by the above observation , we propose a framework for Bilingual Lexicon Induction with Semi-Supervision ( BLISS ) ( §3.2 ) that alleviates the aforementioned issues . Moreover , besides addressing the limitations of the orthogonality assumption , the semi-supervised framework also addresses the shortcomings of purely supervised and purely unsupervised methods for BLI ( §3.1 ) . Our framework jointly optimizes for supervised embedding alignment , unsupervised distribution matching , and a weak orthogonality constraint in the form of a back-translation loss . Our results show that the different losses work in tandem to learn a better mapping than any one could on its own ( §4.3 ) . We also show that the proposed framework improves performance over strong baselines on two datasets , particularly for the case of embedding spaces where the orthogonal assumption is not valid . Our analysis ( §4.3 ) demonstrates that adding supervision to the learning objective , even in the form of a small seed dictionary , significantly improves the stability of the learning procedure . In particular , for cases where either the embedding spaces are far apart according to GH distance or the quality of the original embeddings is poor , our framework converges where the unsupervised baselines fail to . We also show that for the same amount of available supervised data , leveraging unsupervised learning allows us to obtain superior performance over baseline supervised and unsupervised methods using a comparable amount of data . All our code can be found at www.toaddhereifaccepted.com . 2 ISOMETRY OF EMBEDDING SPACES . Both supervised and unsupervised BLI often rely on the assumption that the word embedding spaces are isometric to each other . Thus , they learn an orthogonal mapping matrix to map one space to another . For example , for the case of supervised Bilingual Lexicon Induction , Xing et al . ( 2015 ) learn an orthogonal mapping matrix to minimize the distance between the projected source and the target embeddings ; while in the unsupervised case , Lample et al . ( 2018 ) propose learning a matrix near the manifold of orthogonal matrices to match the distributions of the projected source and target word embeddings . We hypothesize that this assumption might not always hold , in particular for the cases when the language pairs in consideration are etymologically distant — Zhang et al . ( 2017a ) and Søgaard et al . ( 2018 ) provide evidence of this by observing a higher Earth Mover ’ s distance and eigenvector similarity metric respectively between etymologically distant languages . In order to test this hypothesis , we propose a novel way of a-priori analyzing the validity of the orthogonality assumption using the Gromov Hausdorff ( also referred as GH ) distance . In order to analyze the validity of the orthogonality assumption , we quantitatively check how well the metric spaces of the word vectors of two languages can be aligned under an isometric transform using the GH distance.1 The Hausdorff distance between two metric spaces is a measure of the worst case or the diametric distance between the spaces . Intuitively , it measures the distance between the nearest neighbours that are the farthest apart . Concretely , given two metric spaces X , and Y with a distance function d ( . , . ) , the Hausdorff distance is defined as : H ( X , Y ) = max { sup x∈X inf y∈Y d ( x , y ) , sup y∈Y inf x∈X d ( x , y ) } . ( 1 ) The Gromov-Hausdorff distance minimizes the Hausdorff distance over all isometric transforms between X and Y , thereby providing a quantitative estimate of the isometry of two spaces H ( X , Y ) = inf f , g H ( f ( X ) , g ( Y ) ) , ( 2 ) where f , g belong to set of isometric transforms . Computing the Gromov-Hausdorff distance involves solving hard combinatorial problems , but can be tractably approximated using the Bottleneck distance ( Chazal et al. , 2009 ) . In order to compute the Bottleneck distance between two metric spaces , we compute the first order Vietoris-Rips complex ( first order for computational efficiency ) at t for both spaces : a graph containing an edge between two points iff they lie within a Euclidean distance t from each other in the metric space . As t is varied , the Vietoris-Rips complex goes from the individual points ( at t = 0 ) to a single cluster ( at t = ∞ ) . As t increases , clusters are formed ( birth ) and eventually merge together ( death ) . The persistence diagram is a 2D plot of the ( tbirth , tdeath ) of each cluster , where tbirth and tdeath are the values of t at which the cluster was born and died respectively . Given two persistence diagrams f , g , let γ be a bijective map from the points of f to the points of g. The bottleneck distance ( B ) is then defined as : B ( f , g ) = inf γ ( sup u∈f ||u− γ ( u ) ||∞ ) ( 3 ) 1Note that since we mean center the embeddings , the orthogonal transforms are equivalent to isometric transforms . Source→ Target Incorrect Predicted aunt→ тетя бабушка ( Grandmother ) uruguay→ уругвая аргентины ( Argentina ) regiments→ полков кавалерийские ( Cavalry ) comedian→ комик актёр ( Actor ) Table 1 : Words for which semi-supervised method predicts correctly , but unsupervised method doesn ’ t . The unsupervised method is able to guess the general family but fails to pinpoint exact match Chazal et al . ( 2009 ) showed that the Gromov-Hausdorff distance can be lower bounded by the Bottleneck Distance between the Persistence Diagrams of the Vietoris-Rips Filtration of the two spaces . We compute this lower bound for the top frequency words of different language pairs . As can be observed from Figure 1 , the GH distances are higher for distant language pairs . 3 SEMI-SUPERVISED FRAMEWORK . In this section , we motivate and define our semi-supervised framework for BLI . First we describe issues with purely supervised and unsupervised methods , and then lay the framework for tackling them along with orthogonality constraints . 3.1 DRAWBACKS OF PURELY SUPERVISED AND UNSUPERVISED METHODS . Purely supervised methods for aligning word vectors do not utilize the rich information present in the topology of the word vectors . Purely unsupervised methods , on the other hand , can suffer from poor performance if the distribution of the embedding spaces of the two languages are very different from each other . Moreover , unsupervised methods can successfully align clusters of words together , but miss out on fine grained alignment within the clusters . We explicitly show the aforementioned problem of purely unsupervised methods with the help of the toy dataset shown in 2a , and 2b . In this dataset , due to the density difference between the two large blue clusters , unsupervised matching is consistently able to align them properly , but has trouble aligning the smaller embedded green and red sub-clusters . The correct transformation of the source space is a clockwise 90◦ rotation followed by reflection along the x-axis . Unsupervised matching converges to this correct transformation only half of the time ; in rest of the cases , it ignores the alignment of the sub-clusters and converges to a 90◦ counter-clockwise transformation as shown in 2c . We also find evidence of this problem in the real datasets used in our experiments as shown in Table 1 . It can be seen that the unsupervised method aligns clusters of similar words , but is poor at the fine grained alignment . We hypothesize that this problem can be resolved by giving it a some supervision in the form of matching anchor points inside these sub-clusters , which correctly aligns them . Analogously , for the task of BLI , generating a small supervised seed lexicon is generally feasible for most language pairs , through either bilingual speakers , existing dictionary resources , or Wikipedia language links . This can provide the requisite supervision . 3.2 A SEMI-SUPERVISED FRAMEWORK . In order to alleviate the problems with the orthogonality constraints , the purely unsupervised and supervised approaches , we propose a semi-supervised framework , described below . Let X = { x1 . . . xn } and Y = { y1 . . . ym } , xi , yi ∈ Rd be two sets of word embeddings from the source and target language respectively and let S = { ( xs1 , ys1 ) . . . ( xsk , ysk ) } denote the bilingual aligned word embeddings . Define W to be the mapping matrix . For learning W , we leverage unsupervised distribution matching , aligning known word pairs and a data-driven weak orthogonality constraint . Unsupervised Distribution Matching : Given all word embeddings X and Y , the unsupervised loss LW |D aims to match the distribution of both embedding spaces . In particular , for our formulation , we use an adversarial distribution matching objective , similar to the work of Lample et al . ( 2018 ) . Specifically , a mapping matrix W from the source to the target is learned to fool a discriminator D , which is trained to distinguish between the mapped source embeddings WX = { Wx1 . . .Wxn } and Y . The corresponding objectives are defined as : LD|W = − 1 n ∑ xi∈X log ( 1−D ( Wxi ) ) − 1 m ∑ xi∈Y logD ( xi ) ( 4 ) LW |D = − 1 n ∑ xi∈X logD ( Wxi ) ( 5 ) Aligning Known Word Pairs : Given aligned bilingual word embeddings S , we aim to minimize a similarity function ( fs ) which maximizes the similarity between the corresponding matched pairs of words . Specifically , the loss is defined as : LW |S = − 1 |S| ∑ ( xsi , y s i ) ∈S fs ( Wx s i , y s i ) ( 6 ) Weak Orthogonality Constraint : Given an embedding space X , we define a consistency loss that maximizes the similarity fa between x and WTWx , x ∈ X . This cyclic consistency loss LW|O encourages orthogonality of the W matrix based on the joint optimization : LW |O = − 1 |X | ∑ xi∈X fa ( xi , W TWxi ) ( 7 ) The above loss term allows the model adjust the trade-off between orthogonality for accuracy , based on the joint optimization . This is particularly helpful for cases where the orthogonality constraint is violated , where the embedding spaces are not isometric , specifically for etymologically distant language pairs , as we show in ( 4.3 ) . The final loss function for the mapping matrix is : L = LW |D + LW |S + LW |O ( 8 ) LW |D enables the model to leverage the distributional information available from the two embedding spaces , thereby using all available monolingual data . On the other hand , LW |S allows for the correct alignment of labeled pairs when available in the form of a small seed dictionary . Finally , LW |O encourages orthogonality . One can think of LW |O and LW |S as working against each other when the spaces are not isometric . Jointly optimizing both helps the model to strike a balance between them in a data driven manner , encouraging orthogonality but still allowing for flexible mapping .

Summary: The paper propose a semi-supervised algorithm for bilingual lexicon induction (BLI) problem. Prior works on BLI problem usually impose orthogonality constraint on the linear transformation in order to obtain a "reversible" mapping and to preserve the monolingual performance. However, from both modeling and generalization perspective, recent works do not impose this constraint while learning the mapping (Doval et al 2018, Jawanpuria et al 2018, Joulin et al 2018, Sogaard et al 2018, among others). The present work argues for the removal of the orthogonality constraint when language spaces are non-isometric, and proposes to employ the Gromov Hausdroff (GH) distance to validate this condition. Overall, the paper employs  an objective function which is the sum of the (unsupervised) adversarial distribution matching objective (Lample et al 2018b), (supervised) the BLI loss function (typically the square loss), and a consistency loss (Hoshen and Wolf 2018). Empirically, the proposed method shows better results than unsupervised method of Lample et al (2018b) and the Procrustes solution.