Constrained Density Matching and Modeling for Effective Contextualized Alignment

1 INTRODUCTION . Multilingual text encoders such as m-BERT ( Devlin et al. , 2019 ) and XLM-R ( Conneau et al. , 2020 ) have been profiled as the de facto solutions to modeling languages at scale . However , such encoders pre-trained with monolingual data often produce very different representations for parallel data , particularly in low-resource and distant languages settings ( Zhao et al. , 2020a ) as the induced multilingual subspaces are mis-aligned . As remedies , supervised alignment techniques have received large attention . Researchers use them to rectify multilingual representations post-hoc with crosslingual supervision ( Aldarmaki & Diab , 2019 ; Cao et al. , 2020 ; Zhao et al. , 2020b ; Chi et al. , 2021 ) , but their utility is limited to high-resource languages with large parallel data . While unsupervised techniques remove the need of parallel data and thus promise unlimited applicability for all language pairs , they have surprisingly been little studied in contextualized , multilingual representations . In this work , we particularly address the scenarios of limited and no parallel data . In supervised settings , we tackle two limitations leading to the ineffectiveness of previous alignments , namely , ( i ) the inability to sufficiently leverage data , i.e. , to model data density with limited parallel data and ( ii ) that alignments are not properly trained due to a lack of validation criteria1 . To address them , we model density based alignments as follows : First , we frame the problem of aligning multilingual representations as density matching . Next , we use Normalizing Flows ( Dinh et al. , 2017 ) to perform density estimation , by which we leverage data ( be it parallel or not ) . Finally , we introduce two validation criteria accounting for task performances , to prevent models from overfitting . In unsupervised settings , we reuse the above paradigm that combines density based approaches and validation criteria . However , we identify a statistical issue of density matching in the unsupervised case , i.e. , that it only leads to a weak notion of equality of multilingual subspaces , viz. , equality in distribution . To address this , we introduce boostrapping procedures promoting equality of multilingual subspaces , from which we derive cross-lingual supervision from non-parallel data ( see §2 ) . We thoroughly evaluate 16 alignment techniques ( including ours ) across 6 language pairs , synthetic and 4 NLP tasks . Our major findings are summarized as follows : 1Alignments such as those of Wu & Dredze ( 2020 ) and Cao et al . ( 2020 ) have been trained for several epochs without the use of any criteria for model selection ; this comes at the risk of mistraining , given that task performances are not taken into account . • Combining density matching and modeling leads to the best alignment design in both supervised and unsupervised settings . Unsupervised alignments integrated in boostrapping procedures rival supervised counterparts . However , as exposed in simulation setups , we find that such alignments have trouble in generalization to unseen words . • Our validation criteria are better alternatives to intrinsic data for model selection , as the latter ’ s results do not correlate with extrinsic task results . Further , we find intra-task results correlate negatively in about 30 % of setups , showing that validation criteria should address a performance trade-off between tasks by taking all these tasks performances into account . 2 RELATED WORK . Recent advances in multilingual representations ( e.g. , m-BERT and XLM-R ) boost performances of cross-lingual systems in NLP ; however , such systems exhibit weak ( er ) performances for distant languages ( Pires et al. , 2019 ; Wu & Dredze , 2019 ) and for languages with resource disparity ( Lauscher et al. , 2020 ; Zhao et al. , 2020a ) . To address this , contextualized alignments have been proposed in the literature : Aldarmaki & Diab ( 2019 ) show that language-independent rotation can linearly rectify m-BERT representations , while Cao et al . ( 2020 ) find that jointly aligning multiple languages performs better . Zhao et al . ( 2020b ) show that removing language bias in multilingual representations mitigates misalignments between languages . Recently , Mengzhou et al . ( 2021 ) suggest to use gradient based alignments , showing that they are effective for unseen languages in XLM-R. Alqahtani et al . ( 2021 ) use optimal transport to finetune multilingual representations , while other works ( Hu et al. , 2021 ; Chi et al. , 2021 ) finetune them with translation language modeling . However , these efforts have predominantly focused on supervised , resource-intensive alignments that use 250k-2M parallel sentences per language pair for considerable improvements . As early attempts to remove the use of parallel data , Libovický et al . ( 2020 ) and Zhao et al . ( 2020a ) find that applying vector space normalizations alone is helpful to yield language-neural representations . However , there still lacks a systematic study on unsupervised alignments for contextualized multilingual representations , while these have been long researched in the context of static embeddings . The latter often uses iterative procedures to derive bilingual lexicons ( as cross-lingual supervision ) from monolingual data in two steps : ( i ) inducing seed dictionaries using different approaches such as adversarial learning ( Lample et al. , 2018 ) , similarity based heuristics ( Artetxe et al. , 2018b ) and identical strings ( Smith et al. , 2017 ; Artetxe et al. , 2017 ) and ( ii ) applying Procrustes to augment induced lexicons ( Artetxe et al. , 2018b ; Lample et al. , 2018 ) in an iterative fashion . In this work , we present a principled , iterative procedure for unsupervised , contextualized alignments : First , we use density based approaches to induce bilingual lexicons . Next , we apply our boostrapping procedures that theoretically lead to equality of multilingual subspaces , to iteratively augment lexicons . Finally , we complement the first two steps with our validation criteria accounting for all tasks that we investigate . 3 CONTEXTUALIZED ALIGNMENT . Let two random variables X and Y with densities PX and PY describe respective contextual word embeddings in two languages ` 1 and ` 2 , with Ω ` 1 and Ω ` 2 as two lexicons . Each occurrence of a word is associated to a separate entry in the lexicons . X maps all entries in Ω ` 1 to real-valued m-dimensional embedding vectors , denoted by X : Ω ` 1 →Rm , and similarly for Y . A bilingual lexicon Ω describes a set of translations between Ω ` 1 and Ω ` 2 . Empirical inference . Assume a function f : Rm→ Rm perfectly maps m-dimensional embedding vectors from X to Y . As standard in machine learning , a mapping function fθ can be empirically inferred from data , with θ as model parameters . To do so , we assume data M ` 1 ∈ Rn×m and M ` 2 ∈ Rn×m are given , which are two sets of contextual word embeddings with a common size of n for simplicity . Let a permutation matrix P ∈ { 0,1 } n×n ( P1n = 1n and P > 1n = 1n ) be a realization of Ω , serving as cross-lingual supervision when available . A random variable Ỹ with the density PỸ is a prediction of Y given X , namely Ỹ = fX→Y ( X ) ( we use subscripts to denote the mapping direction ) . 3.1 SUPERVISED ALIGNMENT . When parallel data is available , one can induce a permutation matrix P from parallel data either by bilingual experts or by word alignment tools such as FastAlign ( Dyer et al. , 2013 ) or SimAlign ( Jalili Sabet et al. , 2020 ) . Here , we introduce a density based mapping function . As an alternative , we frame learning objectives of contextualized alignment as density matching between PỸ and PY : KL ( PỸ , PY ) = CE ( PỸ , PY ) −Ey∼PY [ logPY ( y ) ] = ‖ fX→Y ( M ` 1 ) −PM ` 2‖ 2−Ey∼PY [ logPX ( f −1 X→Y ( y ) ) |det ( Oθ f −1 X→Y ( y ) ) | ] ( 1 ) where fX→Y is the trainable mapping function from X to Y . As PY is intractable to compute , previous work in supervised alignments always minimizes the cross-entropy term alone by solving the least squares problem . Note that the density PY can be rewritten to PX ( f−1X→Ỹ ( y ) ) |det ( Oθ f −1 X→Ỹ ( y ) ) | by using the change-of-variable rule ( assuming f is an invertible function ) , but the density PY is still intractable to compute given the unknown density PX . To address this issue , we introduce a generative model , namely Real-NVP ( Dinh et al. , 2017 ) as use case of Normalizing Flows ( Dinh et al. , 2015 ; Rezende & Mohamed , 2015 ) , which learns a mapping between data points with an unknown distribution and with the normalized Gaussian distribution through the change-of-variable rule . From this , one can infer data likelihood of an unknown complex distribution using a simple Gaussian distribution . This technique has been shown useful in many cross-domain applications , e.g. , image-to-image translation ( Grover et al. , 2020 ; Gong et al. , 2019 ) and machine translation ( Setiawan et al. , 2020 ) . In particular , we introduce a latent variable Z with the normalized density PZ , namely N ( 0 , I ) , and we let Z share a common vector space with X and Y . We use Real-NVP to infer the likelihood of data from Y using PZ , denoted by PY ( y ) = PZ ( f−1Z→Y ( y ) ) |det ( Oθ f −1 Z→Y ( y ) ) | with fZ→Y as a trainable mapping function from Z to Y . Finally , we rewrite the entropy term in Eq . 1 to : Ey∼PY [ logPY ( y ) ] = Ey∼PY [ logN ( f −1 Z→Y ( y ) ,0 , I ) ] +Ey∼PY [ log |det ( Oθ f −1 Z→Y ( y ) ) | ] ( 2 ) In this work , we perform a dual form of density matching based on JS divergence , from which we take into account density estimation ( modeling ) for both PX and PY . We omit the definition for simplicity . 3.2 UNSUPERVISED ALIGNMENT . When P is not given due to a lack of parallel data , we apply adversarial learning to align two densities PỸ and PY . As standard in adversarial training , we involve a min-max game between two components to perform density matching : ( a ) a discriminator that distinguishes source and target word embeddings after mapping them and ( b ) a mapping function that aligns source and target word embeddings and fools the discriminator . Here , we use a popular adversarial approach , the Wasserstein GAN ( Arjovsky et al. , 2017 ) . It minimizes the Earth Mover distance ( EMD ) between Ỹ and Y to align their densities PỸ and PY . To better leverage data , we include density estimation based on Real-NVP in the procedure of adversarial training . By doing so , we maximize likelihood of data from X and Y using estimated densities PX and PY ( see Eq . 2 ) . Taken together , we denote our density based learning objectives by : EMD ( PỸ , PY ) = minfX→Y max hφ Ey∼PY [ hφ ( y ) ] −Eỹ∼PỸ [ hφ ( ỹ ) ) ] +Ey∼PY [ logPY ( y ) ] ( 3 ) where hφ is a 1-Lipschitz constrained discriminator , ỹ = fX→Y ( x ) maps embeddings from X to Y . Note that fX→Y is the composition of fX→Z and fZ→Y , and the last entropy term is to maximize log-likelihood of data from Y . As in the supervised case , we use a dual form of Eq . 3 . Bootstrapping procedure . After adversarial learning Y and Ỹ are ideally equal in distribution , denoted by Y dist= Ỹ . However , this is not sufficient . For instance , let Y ∼Uniform ( −1,1 ) and Ỹ =−Y . Then , Y and Y are equal in distribution , but they are identical only at the origin . As remedies , we derive two conditions to further improve alignments , promoting the equality of Y and Ỹ . Proposition 1 . Given Y dist= Ỹ , Ỹ and Y are equal if one of the following conditions is met : ( i ) Y = UỸ , where U is invertible and Ui j ≥ 0 ∀i , j . ( ii ) cor ( Ỹi , Yi ) = 1 for ∀i , where Ỹi and Yi represent the i-th component in Ỹ and Y . Proof . ( i ) P ( Y ≤ y ) = P ( Ỹ ≤ y ) for all y , due to Y dist= Ỹ . If Y = UỸ , then P ( Ỹ ≤ y ) = P ( Y ≤ y ) = P ( UỸ ≤ y ) . If U ≥ 0 , then P ( UỸ ≤ y ) = P ( Ỹ ≤ U−1y ) . Thus , P ( Ỹ ≤ U−1y ) = P ( Ỹ ≤ y ) for all y . This implies that U = I . Thus , Ỹ = Y . ( ii ) If cor ( Ỹi , Yi ) = 1 for ∀i , then Var [ ( Ỹiσỹi − Yi σyi ) ] = 0 , thus E [ ( Ỹiσỹi − Yi σyi ) 2 ] −E [ ( Ỹiσỹi − Yi σyi ) ] 2 = 0 . However , the second term equals to 0 by using E [ ( Ỹiσỹi − Yi σyi ) ] = E [ Ỹi ] σỹi − E [ Yi ] σyi = 0 due to Ỹi dist = Yi . Thus , E [ ( Ỹiσỹi − Yi σyi ) 2 ] = 0 , and this implies Ỹiσỹi = Yiσyi since the non-negative ( Ỹiσỹi − Yiσyi ) 2 must be zero if its expectation is 0 . Note that σỹi = σyi due to Ỹi dist = Yi . This implies that Ỹi = Yi for ∀i . To realize the conditions in Prop . 1 , we need additional notation and a lemma . Let MX , MY be embeddings from X and Y , and MỸ = fθ ( MX ) . Lemma 2 . If MỸ M ᵀ Ỹ = MY M ᵀ Y and MỸ is invertible , then Y = UỸ . Proof . If MỸ M ᵀ Ỹ = MY M ᵀ Y and MỸ is invertible , then MY = MỸ M −1 Ỹ MY . Let U = M −1 Ỹ MY . Then , MY = MỸ U . If this holds for all MỸ and MY , then Y = UỸ . In the following , we refer to our two realizations of Prop . 1 as constraints added to to Eq . 3 , from which we promote the equality of Y and Ỹ . Moreover , we discuss their connections with canonical correlation and with language isomorphism ( see §A.1 ( appendix ) ) . Graph structure . We describe MỸ and MY as m-dimensional vertices in two graphs , and denote their corresponding ( weighted ) adjacency matrices by MỸ M ᵀ Ỹ and MY M ᵀ Y . Then , we minimize the difference between them . We use this to realize Prop.1 ( i ) 2 according to Lemm . 2 . The objective becomes : EMD ( PỸ , PY ) +‖MỸ M ᵀ Ỹ −MY M ᵀ Y‖ 2 ( 4 ) Cross-correlation . We maximize Pearson cross-correlation between ( demeaned ) MỸ and MY , and use this to realize Prop.1 ( ii ) . The objective becomes : EMD ( PỸ , PY ) +‖ diag ( MᵀỸ MY ) diag ( MᵀỸ MỸ ) diag ( M ᵀ Y MY ) − # » 1 ‖2 ( 5 ) Regarding MY , MỸ , we use CSLS ( Lample et al. , 2018 ) to induce them , and take them as input arguments of our realizations to guide learning objectives . We refer to this as boostrapping procedures3 ( see Algorithm 1 ) . Algorithm 1 Bootstrapping Procedures 1 : MX , MY ← population word embeddings of X and Y 2 : n← number of boostrapping iterations 3 : fX→Y ← an identity matrix as initial mapping function 4 : for each of n iterations do 5 : MỸ ← fX→Y ( MX ) 6 : P← CSLS ( MY , MỸ ) . induce permutation matrix 7 : fX→Y ← EMD ( PỸ , PY ) + g ( MY , PMỸ ) . g is a realization ( see Eq . 4+5 ) 8 : end for 9 : return fX→Y 2Prop.1 ( i ) is not strictly met , as it is not trivial to guarantee U≥ 0 , i.e. , M−1 Ỹ MY ≥ 0 . This might explain why graph structure performs worse than cross-correlation in our experiments . 3Our initial study finds taking the difference between MY and PMỸ as a realization leads to poor results , as it is very sensitive to noises in induced permutation matrices . We set n to 10 in simulation and to 3 on real data .

Summary: This work presents a new method for aligning contextualized embedding using density matching. Prior work on embedding alignment relies on non-contextualized embedding representation, thus they have limited capability when aligning representations in two languages. This work introduces density matching via normalizing flows so that contextualized representations are matched in two languages. Two methods are investigated: one employ permutation matrix to align representation in a supervised manner. The second one employs Earth Mover distance under Wasserstein GAN for matching densities in an unsupervised manner. A carefully designed bootstrapping procedure is introduced for stabler training for the unsupervised method. Experiments show gains in various benchmarks both for synthetic and real World datasets.