Year: 2021

Abstract:
Multilingual representations pre-trained with monolingual data offer task performances considerably unequal between languages. While this has been tackled through the lens of contextualized alignments, these techniques require large parallel data, thereby leaving under-represented language communities behind. In this work, we analyze the limitations according to which previous alignments become very resource-intensive, viz., (i) the inability to sufficiently leverage data and (ii) that alignments are not trained properly. To address them, we present density based approaches to perform alignments, and we complement them with our validation criteria accounting for both intrinsic and extrinsic task performances. Our experiments encompass 16 alignments (including ours) evaluated across 6 language pairs, synthetic and 4 NLP tasks. We demonstrate that our solutions are particularly effective in the scenarios of limited and no parallel data. More importantly, we show, both theoretically and empirically, the advantages of our boostrapping procedures, by which unsupervised approaches rival supervised counterparts.

heading: 1 INTRODUCTION
text: Multilingual text encoders such as m-BERT (Devlin et al., 2019) and XLM-R (Conneau et al., 2020) have been profiled as the de facto solutions to modeling languages at scale. However, such encoders pre-trained with monolingual data often produce very different representations for parallel data, particularly in low-resource and distant languages settings (Zhao et al., 2020a) as the induced multilingual subspaces are mis-aligned. As remedies, supervised alignment techniques have received large attention. Researchers use them to rectify multilingual representations post-hoc with crosslingual supervision (Aldarmaki & Diab, 2019; Cao et al., 2020; Zhao et al., 2020b; Chi et al., 2021), but their utility is limited to high-resource languages with large parallel data. While unsupervised techniques remove the need of parallel data and thus promise unlimited applicability for all language pairs, they have surprisingly been little studied in contextualized, multilingual representations.
In this work, we particularly address the scenarios of limited and no parallel data. In supervised settings, we tackle two limitations leading to the ineffectiveness of previous alignments, namely, (i) the inability to sufficiently leverage data, i.e., to model data density with limited parallel data and (ii) that alignments are not properly trained due to a lack of validation criteria1. To address them, we model density based alignments as follows: First, we frame the problem of aligning multilingual representations as density matching. Next, we use Normalizing Flows (Dinh et al., 2017) to perform density estimation, by which we leverage data (be it parallel or not). Finally, we introduce two validation criteria accounting for task performances, to prevent models from overfitting.
In unsupervised settings, we reuse the above paradigm that combines density based approaches and validation criteria. However, we identify a statistical issue of density matching in the unsupervised case, i.e., that it only leads to a weak notion of equality of multilingual subspaces, viz., equality in distribution. To address this, we introduce boostrapping procedures promoting equality of multilingual subspaces, from which we derive cross-lingual supervision from non-parallel data (see §2).
We thoroughly evaluate 16 alignment techniques (including ours) across 6 language pairs, synthetic and 4 NLP tasks. Our major findings are summarized as follows:
1Alignments such as those of Wu & Dredze (2020) and Cao et al. (2020) have been trained for several epochs without the use of any criteria for model selection; this comes at the risk of mistraining, given that task performances are not taken into account.
• Combining density matching and modeling leads to the best alignment design in both supervised and unsupervised settings. Unsupervised alignments integrated in boostrapping procedures rival supervised counterparts. However, as exposed in simulation setups, we find that such alignments have trouble in generalization to unseen words.
• Our validation criteria are better alternatives to intrinsic data for model selection, as the latter’s results do not correlate with extrinsic task results. Further, we find intra-task results correlate negatively in about 30% of setups, showing that validation criteria should address a performance trade-off between tasks by taking all these tasks performances into account.

heading: 2 RELATED WORK
text: Recent advances in multilingual representations (e.g., m-BERT and XLM-R) boost performances of cross-lingual systems in NLP; however, such systems exhibit weak(er) performances for distant languages (Pires et al., 2019; Wu & Dredze, 2019) and for languages with resource disparity (Lauscher et al., 2020; Zhao et al., 2020a). To address this, contextualized alignments have been proposed in the literature: Aldarmaki & Diab (2019) show that language-independent rotation can linearly rectify m-BERT representations, while Cao et al. (2020) find that jointly aligning multiple languages performs better. Zhao et al. (2020b) show that removing language bias in multilingual representations mitigates misalignments between languages. Recently, Mengzhou et al. (2021) suggest to use gradient based alignments, showing that they are effective for unseen languages in XLM-R. Alqahtani et al. (2021) use optimal transport to finetune multilingual representations, while other works (Hu et al., 2021; Chi et al., 2021) finetune them with translation language modeling. However, these efforts have predominantly focused on supervised, resource-intensive alignments that use 250k-2M parallel sentences per language pair for considerable improvements. As early attempts to remove the use of parallel data, Libovický et al. (2020) and Zhao et al. (2020a) find that applying vector space normalizations alone is helpful to yield language-neural representations.
However, there still lacks a systematic study on unsupervised alignments for contextualized multilingual representations, while these have been long researched in the context of static embeddings. The latter often uses iterative procedures to derive bilingual lexicons (as cross-lingual supervision) from monolingual data in two steps: (i) inducing seed dictionaries using different approaches such as adversarial learning (Lample et al., 2018), similarity based heuristics (Artetxe et al., 2018b) and identical strings (Smith et al., 2017; Artetxe et al., 2017) and (ii) applying Procrustes to augment induced lexicons (Artetxe et al., 2018b; Lample et al., 2018) in an iterative fashion. In this work, we present a principled, iterative procedure for unsupervised, contextualized alignments: First, we use density based approaches to induce bilingual lexicons. Next, we apply our boostrapping procedures that theoretically lead to equality of multilingual subspaces, to iteratively augment lexicons. Finally, we complement the first two steps with our validation criteria accounting for all tasks that we investigate.

heading: 3 CONTEXTUALIZED ALIGNMENT
text: Let two random variables X and Y with densities PX and PY describe respective contextual word embeddings in two languages `1 and `2, with Ω`1 and Ω`2 as two lexicons. Each occurrence of a word is associated to a separate entry in the lexicons. X maps all entries in Ω`1 to real-valued m-dimensional embedding vectors, denoted by X : Ω`1 →Rm, and similarly for Y . A bilingual lexicon Ω describes a set of translations between Ω`1 and Ω`2 .
Empirical inference. Assume a function f : Rm→ Rm perfectly maps m-dimensional embedding vectors from X to Y . As standard in machine learning, a mapping function fθ can be empirically inferred from data, with θ as model parameters. To do so, we assume data M`1 ∈ Rn×m and M`2 ∈ Rn×m are given, which are two sets of contextual word embeddings with a common size of n for simplicity. Let a permutation matrix P ∈ {0,1}n×n (P1n = 1n and P>1n = 1n) be a realization of Ω, serving as cross-lingual supervision when available. A random variable Ỹ with the density PỸ is a prediction of Y given X , namely Ỹ = fX→Y (X) (we use subscripts to denote the mapping direction).

heading: 3.1 SUPERVISED ALIGNMENT
text: When parallel data is available, one can induce a permutation matrix P from parallel data either by bilingual experts or by word alignment tools such as FastAlign (Dyer et al., 2013) or SimAlign (Jalili Sabet et al., 2020). Here, we introduce a density based mapping function. As an alternative, we frame learning objectives of contextualized alignment as density matching between PỸ and PY :
KL(PỸ ,PY ) = CE(PỸ ,PY )−Ey∼PY [logPY (y)] = ‖ fX→Y (M`1)−PM`2‖ 2−Ey∼PY [logPX ( f −1 X→Y (y))|det(Oθ f −1 X→Y (y))|] (1)
where fX→Y is the trainable mapping function from X to Y . As PY is intractable to compute, previous work in supervised alignments always minimizes the cross-entropy term alone by solving the least squares problem. Note that the density PY can be rewritten to PX ( f−1X→Ỹ (y))|det(Oθ f −1 X→Ỹ (y))| by using the change-of-variable rule (assuming f is an invertible function), but the density PY is still intractable to compute given the unknown density PX . To address this issue, we introduce a generative model, namely Real-NVP (Dinh et al., 2017) as use case of Normalizing Flows (Dinh et al., 2015; Rezende & Mohamed, 2015), which learns a mapping between data points with an unknown distribution and with the normalized Gaussian distribution through the change-of-variable rule. From this, one can infer data likelihood of an unknown complex distribution using a simple Gaussian distribution. This technique has been shown useful in many cross-domain applications, e.g., image-to-image translation (Grover et al., 2020; Gong et al., 2019) and machine translation (Setiawan et al., 2020).
In particular, we introduce a latent variable Z with the normalized density PZ , namely N (0,I), and we let Z share a common vector space with X and Y . We use Real-NVP to infer the likelihood of data from Y using PZ , denoted by PY (y) = PZ( f−1Z→Y (y))|det(Oθ f −1 Z→Y (y))| with fZ→Y as a trainable mapping function from Z to Y . Finally, we rewrite the entropy term in Eq. 1 to:
Ey∼PY [logPY (y)] = Ey∼PY [logN ( f −1 Z→Y (y),0,I)]+Ey∼PY [log |det(Oθ f −1 Z→Y (y))|] (2)
In this work, we perform a dual form of density matching based on JS divergence, from which we take into account density estimation (modeling) for both PX and PY . We omit the definition for simplicity.

heading: 3.2 UNSUPERVISED ALIGNMENT
text: When P is not given due to a lack of parallel data, we apply adversarial learning to align two densities PỸ and PY . As standard in adversarial training, we involve a min-max game between two components to perform density matching: (a) a discriminator that distinguishes source and target word embeddings after mapping them and (b) a mapping function that aligns source and target word embeddings and fools the discriminator. Here, we use a popular adversarial approach, the Wasserstein GAN (Arjovsky et al., 2017). It minimizes the Earth Mover distance (EMD) between Ỹ and Y to align their densities PỸ and PY . To better leverage data, we include density estimation based on Real-NVP in the procedure of adversarial training. By doing so, we maximize likelihood of data from X and Y using estimated densities PX and PY (see Eq. 2). Taken together, we denote our density based learning objectives by:
EMD(PỸ ,PY ) = minfX→Y max hφ Ey∼PY [hφ(y)]−Eỹ∼PỸ [hφ(ỹ))]+Ey∼PY [logPY (y)] (3)
where hφ is a 1-Lipschitz constrained discriminator, ỹ = fX→Y (x) maps embeddings from X to Y . Note that fX→Y is the composition of fX→Z and fZ→Y , and the last entropy term is to maximize log-likelihood of data from Y . As in the supervised case, we use a dual form of Eq. 3.
Bootstrapping procedure. After adversarial learning Y and Ỹ are ideally equal in distribution, denoted by Y dist= Ỹ . However, this is not sufficient. For instance, let Y ∼Uniform(−1,1) and Ỹ =−Y . Then, Y and Y are equal in distribution, but they are identical only at the origin. As remedies, we derive two conditions to further improve alignments, promoting the equality of Y and Ỹ .
Proposition 1. Given Y dist= Ỹ , Ỹ and Y are equal if one of the following conditions is met:
(i) Y = UỸ , where U is invertible and Ui j ≥ 0 ∀i, j.
(ii) cor(Ỹi,Yi) = 1 for ∀i, where Ỹi and Yi represent the i-th component in Ỹ and Y .
Proof. (i) P(Y ≤ y) = P(Ỹ ≤ y) for all y, due to Y dist= Ỹ . If Y = UỸ , then P(Ỹ ≤ y) = P(Y ≤ y) = P(UỸ ≤ y). If U ≥ 0, then P(UỸ ≤ y) = P(Ỹ ≤ U−1y). Thus, P(Ỹ ≤ U−1y) = P(Ỹ ≤ y) for all y. This implies that U = I. Thus, Ỹ = Y .
(ii) If cor(Ỹi,Yi) = 1 for ∀i, then Var[( Ỹiσỹi − Yi σyi )] = 0, thus E[( Ỹiσỹi − Yi σyi )2]−E[( Ỹiσỹi − Yi σyi
)] 2 = 0.
However, the second term equals to 0 by using E[( Ỹiσỹi − Yi σyi )] = E[Ỹi]σỹi − E[Yi]σyi = 0 due to Ỹi dist = Yi. Thus, E[( Ỹiσỹi − Yi σyi )2] = 0, and this implies Ỹiσỹi = Yiσyi since the non-negative ( Ỹiσỹi − Yiσyi ) 2 must be zero if its expectation is 0. Note that σỹi = σyi due to Ỹi dist = Yi. This implies that Ỹi = Yi for ∀i.
To realize the conditions in Prop. 1, we need additional notation and a lemma. Let MX , MY be embeddings from X and Y , and MỸ = fθ(MX ). Lemma 2. If MỸ M ᵀ Ỹ = MY M ᵀ Y and MỸ is invertible, then Y = UỸ .
Proof. If MỸ M ᵀ Ỹ = MY M ᵀ Y and MỸ is invertible, then MY = MỸ M −1 Ỹ MY . Let U = M −1 Ỹ MY . Then, MY = MỸ U. If this holds for all MỸ and MY , then Y = UỸ .
In the following, we refer to our two realizations of Prop. 1 as constraints added to to Eq. 3, from which we promote the equality of Y and Ỹ . Moreover, we discuss their connections with canonical correlation and with language isomorphism (see §A.1 (appendix)).
Graph structure. We describe MỸ and MY as m-dimensional vertices in two graphs, and denote their corresponding (weighted) adjacency matrices by MỸ M ᵀ Ỹ and MY M ᵀ Y . Then, we minimize the difference between them. We use this to realize Prop.1(i)2 according to Lemm. 2. The objective becomes:
EMD(PỸ ,PY )+‖MỸ M ᵀ Ỹ −MY M ᵀ Y‖ 2 (4)
Cross-correlation. We maximize Pearson cross-correlation between (demeaned) MỸ and MY , and use this to realize Prop.1(ii). The objective becomes:
EMD(PỸ ,PY )+‖ diag(MᵀỸ MY )
diag(MᵀỸ MỸ )diag(M ᵀ Y MY )
− #»1 ‖2 (5)
Regarding MY ,MỸ , we use CSLS (Lample et al., 2018) to induce them, and take them as input arguments of our realizations to guide learning objectives. We refer to this as boostrapping procedures3 (see Algorithm 1).
Algorithm 1 Bootstrapping Procedures 1: MX ,MY ← population word embeddings of X and Y 2: n← number of boostrapping iterations 3: fX→Y ← an identity matrix as initial mapping function 4: for each of n iterations do 5: MỸ ← fX→Y (MX ) 6: P← CSLS(MY ,MỸ ) . induce permutation matrix 7: fX→Y ← EMD(PỸ ,PY )+ g(MY ,PMỸ ) . g is a realization (see Eq. 4+5) 8: end for 9: return fX→Y
2Prop.1(i) is not strictly met, as it is not trivial to guarantee U≥ 0, i.e., M−1 Ỹ MY ≥ 0. This might explain why graph structure performs worse than cross-correlation in our experiments.
3Our initial study finds taking the difference between MY and PMỸ as a realization leads to poor results, as it is very sensitive to noises in induced permutation matrices. We set n to 10 in simulation and to 3 on real data.

heading: 4 EXPERIMENTS
text: Unlike static embeddings, contextual embeddings have not yet been investigated in intrinsic tasks such as Bilingual Lexicon Induction (BLI). We address this by creating simulated data in §4.3, on which we contrast findings from static and contextual embeddings and expose limitations of alignment techniques (including ours). In §4.4, we compare the alignment techniques across 6 language pairs and 4 real cross-lingual NLP tasks.

heading: 4.1 BASELINES AND OURS
text: Supervised alignments. (a) Rotation (Aldarmaki & Diab, 2019; Zhao et al., 2020a): a linear orthogonal-constrained transformation; (b) GBDD (Zhao et al., 2020b): subtracting a global language bias vector from multilingual representations; (c) FCNN: an architecture that contains three fullyconnected layers followed by a tanh activation function each; (d) a popular approach Joint-Align (Cao et al., 2020): jointly aligning multiple languages using fine-tuning; (e) recent InfoXLM (Chi et al., 2021): finetuning multilingual representations with translation language modeling and contrastive learning; and (f) our density based approach Real-NVP. Except for Joint-Align and InfoXLM, all other approaches have been trained separately for different language pairs.
Unsupervised alignments. (a) MUSE: an unsupervised variant of Rotation (Lample et al., 2018); (b) VecMap: a heuristic unsupervised approach that assumes word translations have similar distributions of word similarities (Artetxe et al., 2018a); (c) vector space normalization (Zhao et al., 2020a): removing language-specific means and variances of multilingual representations and (d) our GAN-RealNVP: an unsupervised variant of Real-NVP. For boostrapping based unsupervised approaches, we denote them by [Method]+Cross-Correlation, [Method]+Graph-Structure and [Method]+Procrustes, where [Method] is MUSE or GAN-Real-NVP; Procrustes is a popular bootstrapping procedure commonly used for static embeddings. Both MUSE and VecMap are popular unsupervised alignments in static embeddings.

heading: 4.2 VALIDATION CRITERION
text: We examine two unsupervised validation criteria for model selection during training, and compare them with no-criteria (i.e., training for several epochs) in both supervised and unsupervised settings. Our criteria are measured based on the 30k most confident word translations induced by CSLS.
• Semantic criterion has been established in static embeddings. Lample et al. (2018) select the 10k most frequent source words and use them to generate target translations. Next, they average cosine similarities on word translations and use it as a validation criterion, and find it correlates strongly with results on validation sets.
• We measure the difference between two ordered lists of singular values obtained from source and target word embeddings. While Dubossarsky et al. (2020) use this to measure language isomorphism, we refer to it as a structural criterion.

heading: 4.3 SIMULATION
text: BLI is a popular intrinsic task used to evaluate static embeddings, as it covers over 100 language pairs and evaluates the word alignment itself other than its impact on extrinsic tasks. BLI evaluations induce bilingual lexicons according to similarities of word embeddings and compare them with gold lexicons. However, contextual embeddings lack such evaluations. As argued in Artetxe et al. (2020), when not evaluated under similar conditions, the lessons learned from static embeddings cannot transfer to contextual ones. To address this, we use data simulation to create synthetic data, a form of contextual extension of BLI (CBLI), to intrinsically evaluate contextual embeddings. We split data to train, validation and test sets, on which we compare induced bilingual lexicons with gold ones. Task performances are measured in Precision@K as in BLI evaluation. Our creation procedure is two-fold: First, we produce source embeddings sampled from a two-dimensional Gaussian mixture distribution. Next, we perform linear and non-linear transformations on them, to produce target embeddings, by which we mimic different typological language pairs (see example in Fig. 4 (appendix)).
When constructing simulation setups, we adjust three parameters: (a) the number of occurrences k for a word, k ∈ {5,10, . . . ,100}; we use 20 words in all setups, (b) the degree of language isomorphism
t ∈ {1, . . . ,10}—which mimics different language pairs and (c) the distance between embeddings in train and test sets, ∈ {0,0.2, . . . ,5}—which reflects different similarities between train and test domains. For the i-th word and its occurrences, their embeddings are sampled from a Gaussian distribution N (µi,I) for train sets and from a N (µi + ,I) for validation and test sets. µi denotes a mean vector sampled uniformly from [−5,5] for each component. For isomorphic languages (t = 1), we transform source to target embeddings using an orthogonal matrix. For non-isomorphic languages (t > 1), we combine translation and rotation to perform non-linear transformations.
Generalization to unseen words. While Czarnowska et al. (2019) finds that word frequency has a big impact on task performance for static embeddings, Fig. 1 (a)+(b) show that, in the contextual case, task performance often does not correlate with word frequency but strongly correlates with domain similarities between train and test sets.
In isomorphic settings, Fig. 1 (c) shows that linear alignments are the winner, both in supervised (Rotation) or unsupervised (MUSE) settings. It means a simple linear transformation perfectly aligns vector spaces of isomorphic languages, especially for Rotation. We mark this as a sanity test, as most of languages seem non-isomorphic (Søgaard et al., 2018).
In non-isomorphic settings, Fig. 1 (d) shows that non-linear alignments, both supervised (Real-NVP) and unsupervised (GAN-Real-NVP), are better than linear counterparts when train and test domains are similar; however, linear alignments are indeed better in terms of generalization when train and test domains are dissimilar. One may not forget: many works (Mikolov et al., 2013; Glavaš et al., 2019) conclude that non-linear alignments for static embeddings are always worse than linear ones. We complement this analysis with ours, showing in which cases non-linear alignments based on contextual embeddings succeed and fail. Besides, our findings hint at real scenarios in which linear alignments might be a good choice. For instance, one aligns vector spaces of Modern-English and Simplified-Chinese but evaluates on Middle-English and Traditional-Chinese.
Overall, we show that alignment for static and contextual embeddings have different conclusions. By contrasting them, we offer better understanding of each. For instance, non-linear alignments based on contextual embeddings are good in general, but they have trouble in generalization.
Maximum capacities of unsupervised approaches. Fig. 2 (left) shows how well two languages are aligned when train and test domains are similar. Real-NVP and GAN-Real-NVP perform best according to task performances in corresponding supervised and unsupervised settings, and their resulting vector spaces are better overlapped than others. This confirms the effect of our density based approaches. However, there are big performance gaps between supervised and unsupervised approaches, especially for Real-NVP (73.3) vs. GAN-Real-NVP (41.0), notwithstanding small subspaces overlaps and small differences in model architectures. This confirms that density matching alone is not sufficient. Fig. 2 (right) shows that after bootstrapping GAN-Real-NVP rivals (sometimes exceeds) its supervised variant Real-NVP. We also see similar results from Rotation vs. MUSE. Cross-correlation helps best in all cases, while graph-structure and Procrustes lead to less consistent gains across approaches.
Overall, these experiments show that bootstrapping procedures are crucial for unsupervised alignments to compete with their supervised variants.

heading: 4.4 EXPERIMENTS ON REAL DATA
text: XTREME (Hu et al., 2020) and many others are popular benchmarks for evaluating multilingual representations; however, they do not address alignments themselves, but rather focus on how multilingual representations impact cross-lingual systems—as such, the impact of better alignments is indirect and may be lost in complex supervised downstream task classifiers. In this work, we systematically evaluate inherent strengths of alignments through the lens of (i) intrinsic alignment results and (ii) how alignments extrinsically impact multilingual representations without supervised classifiers on top. We leave performances in supervised downstream tasks to future work. Our tasks are listed below:
• CBLI4 is the contextualized extension of BLI. Both contain a bilingual lexicon per language pair, but CBLI marks each occurrence of a word as an entry in lexicons. For each language pair, we extract 10k word translations from parallel sentences using FastAlign (Dyer et al., 2013). We report Precision@1.
• Alignment (Align) is a bilingual word retrieval task. Each language pair contains gold standard 2.5k word translations annotated from parallel sentences. We use a word alignment tool SimAlign (Jalili Sabet et al., 2020) to retrieve word translations from parallel sentences based on their embeddings. We report F-score that combines precision and recall.
• Reference-free evaluation (RFEval) measures correlations between human and automatic judgments of translation quality. We use XMover (Zhao et al., 2019; 2020b) (excluding a target-side language model) to rate translation quality. It compares system translations with source sentences based on their embeddings. We report Pearson correlations between XMover and human scores. Each language pair covers 3k source sentences, each containing multiple system translations and human judgments.
• Tatoeba is a bilingual sentence retrieval task taken from XTREME. Each language pair contains 1k sentence translations. Given a source sentence, we find its nearest translation from a pool of candidates based on cosine similarities of embeddings. We report Precision@1.
Tatoeba, CBLI and RFEval contain 6 languages German, Czech, Latvian, Finnish, Russian and Turkish, paired to English, while Align contains German/Czech-to-English, as the remaining ones are not available. We investigate two popular multilingual representations: m-BERT and XLM-R. We mark CBLI as an intrinsic task and the others as extrinsic tasks, as CBLI is the only one following the same data creation procedure of our train data (see Setup below).
4We provide two complementary CBLI data: one is gold but simulated; another is real but contains noises.
Setup. To contrast supervised and unsupervised approaches, we consider two data scenarios: (i) very limited parallel data and (ii) no parallel data. In case (i), we use 20k (compared to 250k often used in previous work) parallel sentences sampled from News-Commentary (Tiedemann, 2012) for Russian/Turkish-to-English and from EuroParl (Koehn, 2005) for other languages. We extract word translations from parallel sentences using FastAlign, from which we induce permutation matrices as cross-lingual supervision. In case (ii), we unpair word translations obtained from (i) by removing the use of permutation matrices. Thus, we compare supervised and unsupervised approaches under similar conditions, viz., with similar scale of data.
m-BERT XLM-R Alignments RFEval Tatoeba CBLI Align RFEval Tatoeba CBLI Align
How to select the best model. We compare two choices for model selection during training: using (i) intrinsic data (CBLI) and (ii) our unsupervised validation criteria. Fig. 3 (left) shows that CBLI
results often correlate poorly (or even negatively) with results on other tasks in both supervised and unsupervised settings. This means intrinsic data is a bad choice for model selection. Tab. 2 (appendix) reports correlation statistics across approaches, showing that task performances correlate negatively in about 30% setups. This means when selecting the best model, one should take all these task results into account. We take Real-NVP as use case, to show how to choose our criteria. Fig. 3 (middle)+(right) shows that semantic criterion correlates better than structural criterion with task performances on average (0.62 vs. 0.15). This suggests to use semantic criterion for Real-NVP. We also see similar results for other approaches. Our initial study finds linearly combining two criteria leads to small gains. Thus, we use the semantic criterion for simplicity in all setups.
Overall, these results show that we cannot use intrinsic data for model selection, and when choosing a validation criterion, we should take into account all task performances.
Results. Tab. 1 shows results of approaches on real data in supervised and unsupervised settings. In supervised settings, FCNN and Real-NVP training for 5 epochs are worse than their variants that use our criteria for model selection in almost all tasks, especially on RFEval. This confirms the importance of validation criteria. Joint-Align that uses 100k parallel data performs best on intrinsic CBLI; however, in the remaining tasks, it is worse than Real-NVP-20k that uses merely 20k data and our criteria accounting for all task results; we see similar results in unsupervised settings (VecMap vs. GAN-Real-NVP). This means our validation criteria prevent approaches from overfitting intrinsic data. Note that gains on Align are much smaller than on others. Thus, either the word alignment tool SimAlign or the dataset is not sensitive to improved embeddings. This indicates that one should improve evaluation setups such as tools and datasets. Our approach Real-NVP is the strongest approach. It helps for both m-BERT and XLM-R. When using validation criteria, Real-NVP performs best in 7 out of 8 tasks in supervised settings. Tab. 3 (appendix) also confirms the effect of Real-NVP when comparing InfoXLM with our approaches under fair conditions—evaluated across three language pairs for which InfoXLM use parallel data.
In unsupervised settings, our criteria are also crucial, as MUSE and GAN-Real-NVP that use our criteria largely outperform training for 5 epochs. Unlike in simulation, there are small performance gaps between supervised and unsupervised approaches, especially the gap between Real-NVP and GAN-Real-NVP (9.0 vs. 32.3 points in simulation). Therefore, while boostrapping procedures offer less help in our real data setups, cross-correlation based boostrapping still helps best for both unsupervised approaches. Procrustes results are similar as in simulation—it only helps for MUSE and harms GAN-Real-NVP. Our GAN-Real-NVP that uses our criteria and cross-correlation based boostrapping performs best in all cases. It rivals the best supervised approach Real-NVP.
Overall, these results show our semantic criterion is helpful; density based approaches are effective in both supervised and unsupervised settings. Unsupervised approaches after bootstrapping based on cross-correlation rival supervised ones. Full results are provided in Tab. 4 (appendix).

heading: 5 CONCLUSION
text: Resource and typology disparities are primary characteristics of languages, leading to multilingual representations with capabilities unequal across languages. While recent developments on contextualized alignments allow for producing language-agnostic representations, their applications are limited in scope due to data scarcity in low-resource languages. In this work, we demonstrated that our solutions are particularly effective to mitigate the data scarcity issue. With 20k parallel data, our alignments outperformed others that have been trained on 100k parallel data. Further, we showed that parallel data could be removed when integrating unsupervised alignments in boostrapping procedures.
As a form of domain adaptation techniques, density based approaches have been applied in many cross-domain applications, such as image-captioning (Mahajan et al., 2020), image-to-image translation (Grover et al., 2020; Gong et al., 2019), static embeddings (Zhou et al., 2019) and machine translation (Setiawan et al., 2020). In this work, we showed that density based approaches overfit intrinsic data when not trained properly; simple linear techniques indeed perform better when train and test domains are dissimilar. Further, we showed that boostrapping procedures and validation criteria are crucial for density based approaches to improve their effectiveness. While these analyses have been provided in contextualized alignments as use case of cross-domain applications, we believe our findings provide strong cues towards effective domain adaptation techniques in other applications.

heading: A APPENDIX
text: A.1 CONNECTION WITH CANONICAL CORRELATION AND LANGUAGE ISOMORPHISM.
Cross-correlation between random vectors are often measured via Canonical Correlation Analysis (CCA). While Faruqui & Dyer (2014) find CCA is useful to improve static embeddings, it requires finding k primary canonical variables. However, our solution is simple. It measures cross-correlation using random vectors per se.
In graph theory, two graphs are called isomorphic if corresponding adjacency matrices A and B are permutation similar, denoted by PAP−1 = B with P as a permutation matrix. According to Eq. 4, A and B are denoted by MỸ M ᵀ Ỹ and MY M ᵀ Y ; minimizing the difference between A and B ideally leads to P as an identity matrix when converged. As such, applying graph-structure approach meets the condition of graph isomorphism—which has been profiled as a form of language isomorphism (Søgaard et al., 2018). Therefore, our approach theoretically yields isomorphic multilingual subspaces for non-isomorphic (distant) languages.
A.2 TRAINING DETAILS
For Real-NVP and GAN-Real-NVP, the Real-NVP component is stacked with two FCNN networks with a dimension size of 512 for each. We optimize them with RMSprop and 32 batch size, learning rate 1e-4 for Real-NVP and learning rate 1e-6 for GAN-Real-NVP. The coefficient for maximum likelihood estimation is set to 1e-4. For FCNN, we set the network with 1024 as a dimension size. We optimize it with Adam, learning rate 1e-4 and 32 batch size. For MUSE, we optimize it with SGD, learning rate 1e-1 and 32 batch size. For Joint-Align, we follow the parameters in Cao et al. (2020). We run all the baselines and our models on a single GTX 1070 for synthetic data, but run them on a single Tesla V100 for real data. We align the 8-th layer of multilingual representations such as m-BERT and XLM-R for individual language pairs, as previous works (Ethayarajh, 2019; Zhao et al., 2020a) show that the representations obtained from upper layers often have poor discriminativeness, i.e., a failure to distinguish mutual word translations from random word pairs.
Under review as a conference paper at ICLR 2022 L an g O ri gi na l R ot at io n FC N N N V P G B D D Jo in tA lig n N O R M M U SE M U SE + C M U SE + G M U SE + P G N V P G N V P+ C G N V P+ G G N V P+ P in fo X L M V ec M ap R ef er en ce -f re e E va lu at io n (m -B E R T) cs -e n 23 .6 32 .5 41 .5 30 .2 24 .4 32 .5 22 .5 24 .8 29 .7 26 .7 30 .4 24 .9 25 .3 25 .2 30 .5 - 25 .0 de -e n 29 .8 31 .6 34 .1 34 .2 30 .0 39 .9 31 .4 29 .9 31 .3 31 .1 31 .2 32 .8 33 .0 32 .8 31 .3 - 31 .8 fien 30 .9 48 .5 58 .3 51 .8 32 .5 48 .6 32 .3 37 .9 44 .0 40 .4 46 .0 37 .2 39 .5 37 .9 46 .1 - 39 .1 lv -e n 23 .0 44 .0 55 .1 52 .1 25 .7 46 .1 31 .3 33 .4 35 .9 33 .5 40 .9 38 .0 40 .7 38 .8 41 .0 - 24 .9 ru -e n 19 .4 23 .4 30 .5 33 .4 21 .4 31 .7 23 .9 19 .8 22 .0 21 .3 22 .9 26 .5 27 .0 26 .3 22 .9 - 18 .7 tren 36 .7 52 .4 60 .8 52 .2 38 .6 48 .6 39 .1 41 .6 48 .6 46 .3 49 .7 44 .0 46 .5 44 .9 49 .8 - 44 .5 R ef er en ce -f re e E va lu at io n (X LM -R ) cs -e n 20 .6 26 .3 26 .1 32 .9 22 .4 - 28 .0 20 .4 24 .2 22 .3 24 .6 29 .2 30 .1 29 .1 25 .1 26 .6 de -e n 25 .1 27 .4 28 .2 40 .5 25 .4 - 39 .5 25 .2 26 .5 26 .0 26 .6 40 .4 40 .4 40 .5 27 .2 37 .3 fien 26 .6 39 .9 44 .6 47 .7 26 .0 - 41 .6 29 .0 35 .7 32 .3 37 .8 44 .3 47 .2 44 .5 38 .2 36 .1 lv -e n 28 .3 42 .3 48 .6 51 .2 29 .6 - 44 .1 30 .8 37 .9 32 .7 40 .7 47 .3 48 .7 47 .4 40 .9 38 .3 ru -e n 21 .4 25 .6 27 .7 38 .0 21 .5 - 34 .3 21 .5 23 .3 22 .4 24 .6 35 .9 36 .3 36 .1 25 .1 34 .9 tren 36 .5 46 .5 44 .8 54 .7 37 .9 - 49 .6 38 .4 44 .7 41 .2 45 .9 53 .2 52 .9 51 .7 46 .2 52 .6 Ta to eb a (m -B E R T) cs -e n 47 .5 53 .1 61 .4 61 .3 48 .3 51 .1 60 .6 48 .2 50 .8 48 .7 51 .6 60 .6 61 .3 60 .7 51 .5 - 47 .5 de -e n 76 .6 79 .5 84 .5 87 .3 78 .6 89 .6 86 .0 77 .7 79 .3 78 .7 78 .9 86 .2 86 .5 86 .0 78 .8 - 81 .2 fien 41 .5 50 .6 57 .2 54 .3 46 .4 70 .4 53 .5 44 .7 46 .5 45 .2 47 .8 53 .9 56 .1 54 .6 47 .5 - 53 .5 lv -e n 31 .7 39 .8 47 .4 51 .2 35 .0 31 .1 44 .0 35 .3 35 .0 33 .8 37 .1 44 .1 44 .7 43 .8 37 .0 - 33 .5 ru -e n 63 .0 65 .7 70 .3 72 .6 64 .8 74 .2 75 .4 63 .5 64 .9 64 .7 66 .3 75 .1 76 .0 75 .1 66 .3 - 66 .5 tren 35 .8 43 .0 46 .3 50 .5 40 .6 38 .4 48 .2 39 .1 40 .9 38 .8 43 .1 47 .3 49 .3 48 .1 42 .6 - 47 .6 Ta to eb a (X LM -R ) cs -e n 49 .8 57 .5 74 .6 72 .9 59 .2 - 72 .5 53 .8 58 .9 60 .6 59 .1 72 .8 74 .0 73 .3 57 .6 69 .6 de -e n 89 .0 91 .0 94 .5 94 .9 90 .1 - 95 .2 88 .7 91 .0 89 .6 90 .8 94 .6 95 .2 94 .8 90 .5 95 .5 fien 63 .8 68 .6 81 .3 80 .8 69 .3 - 79 .6 65 .6 69 .8 67 .2 68 .8 80 .6 81 .2 80 .6 67 .7 68 .4 lv -e n 52 .2 61 .3 72 .6 72 .4 58 .9 - 71 .9 56 .4 63 .1 59 .2 61 .0 71 .5 71 .5 71 .7 59 .7 51 .2 ru -e n 70 .2 71 .5 83 .1 83 .9 73 .2 - 83 .9 71 .2 70 .6 70 .2 72 .2 83 .4 84 .2 84 .0 70 .9 85 .7 tren 55 .4 61 .7 75 .1 75 .6 62 .4 - 75 .4 58 .6 62 .0 63 .2 61 .0 75 .1 76 .9 75 .7 61 .3 86 .1 C on te xt ua lB LI (m -B E R T) cs -e n 47 .3 52 .6 49 .2 52 .3 47 .7 57 .0 50 .1 47 .7 50 .0 48 .4 50 .8 51 .8 52 .0 51 .5 51 .0 - 55 .1 de -e n 61 .0 64 .6 59 .0 63 .8 60 .9 79 .4 63 .3 61 .8 61 .8 62 .1 62 .5 64 .8 64 .8 64 .7 63 .0 - 71 .7 fien 36 .8 48 .9 45 .8 45 .8 37 .9 74 .2 43 .0 40 .7 40 .6 40 .7 43 .4 45 .0 47 .6 44 .7 44 .6 - 56 .7 lv -e n 42 .7 51 .8 49 .5 50 .6 43 .5 49 .8 47 .9 45 .5 44 .5 45 .9 48 .7 49 .9 50 .5 49 .7 49 .5 - 53 .9 ru -e n 66 .5 70 .9 67 .6 71 .4 67 .0 67 .4 69 .2 67 .2 67 .5 67 .8 69 .5 70 .5 71 .0 70 .6 69 .9 - 76 .9 tren 51 .1 61 .9 60 .7 61 .8 51 .5 60 .2 55 .8 52 .0 52 .8 53 .7 56 .4 60 .2 62 .1 58 .7 58 .3 - 70 .4 C on te xt ua lB LI (X LM -R ) cs -e n 40 .2 45 .1 43 .6 51 .1 39 .3 - 48 .2 41 .6 42 .0 41 .5 42 .3 49 .8 50 .0 49 .6 43 .3 50 .7 de -e n 54 .1 57 .9 53 .2 64 .6 54 .3 - 62 .7 55 .2 55 .1 55 .3 55 .4 65 .1 65 .3 64 .9 56 .3 67 .3 fien 44 .2 51 .0 46 .1 60 .9 44 .0 - 58 .1 46 .8 45 .7 47 .2 46 .6 60 .2 60 .5 59 .9 48 .3 57 .1 lv -e n 44 .2 50 .0 48 .3 57 .6 44 .2 - 54 .8 46 .4 45 .7 47 .3 46 .5 56 .9 57 56 .7 47 .8 53 .5 ru -e n 60 .9 64 .4 60 .3 71 .7 60 .7 - 70 .3 61 .4 61 .8 61 .2 62 .2 71 .5 71 .5 71 .4 63 .0 72 .4 tren 47 .9 53 .6 53 .8 63 .9 48 .0 - 60 .1 48 .6 48 .5 48 .6 49 .2 62 .5 63 .1 62 .5 50 .9 64 .0 W or d A lig nm en t( m -B E R T) cs -e n 44 .0 45 .0 43 .4 45 .7 44 .1 44 .2 44 .6 44 .0 45 .0 44 .5 45 .0 44 .6 45 .1 44 .9 45 .1 - 45 .0 de -e n 79 .1 80 .6 79 .6 79 .5 79 .3 80 .2 80 .5 79 .3 80 .2 79 .9 80 .4 80 .0 80 .3 80 .2 80 .5 - 80 .0 W or d A lig nm en t( X LM -R ) cs -e n 41 .1 42 .1 41 .8 45 .2 40 .9 - 44 .4 41 .2 42 .1 41 .8 42 .1 44 .6 44 .9 44 .8 42 .0 43 .3 de -e n 78 .5 79 .6 77 .9 80 .5 78 .7 - 80 .8 78 .8 78 .9 79 .2 79 .1 81 .0 81 .4 81 .2 79 .0 82 .6
Ta bl
e 4:
Fu ll
re su
lts of
ba se
lin es
an d
ou ra
lig nm
en ts
.R ea
lN
V P
an d
G A
N -R
ea l-
N V
P ar
e de
no te
d by
N V
P an
d G
N V
P. [M
et ho
d] +
P/ C
/G de
no te
s th
e in
te gr
at io
n in
th e
Pr oc
ru st
es re
fin em
en t(
P) ,o
ri n
ou rb
oo ts
tr ap
pi ng
pr oc
ed ur
e co
ns tr
ai ne
d w
ith cr
os s-
co rr
el at
io n
(C )a
nd w
ith gr
ap h
st ru
ct ur
e (G
).

Review summaries:
This paper deals with alignment between embeddings of different languages. The embeddings are contextualized representations produced by multilingual text encoders. Although alignment between static embeddings have been thoroughly studied, alignment between contextualized representations have not, and existing works use large parallel data. This work proposes a density matching approach to this task, using limited or no supervision. The authors contribute ideas of normalizing flows, EMD, constraints like cross-correlation, validation criteria, etc. to this task, and verify their approach with empirical results.

This paper considers density based supervised and unsupervised alignment of contextual embeddings with Real-NVP and Wasserstein GAN, respectively. For unsupervised alignment, an additional bootstrap step is considered. Additionally, it considers two model selection criteria. The experiment focuses on intrinsic (as opposed to its extrinsic impact on cross-lingual systems) evaluation on representation alignment. 

This work presents a new method for aligning contextualized embedding using density matching. Prior work on embedding alignment relies on non-contextualized embedding representation, thus they have limited capability when aligning representations in two languages. This work introduces density matching via normalizing flows so that contextualized representations are matched in two languages. Two methods are investigated: one employ permutation matrix to align representation in a supervised manner. The second one employs Earth Mover distance under Wasserstein GAN for matching densities in an unsupervised manner. A carefully designed bootstrapping procedure is introduced for stabler training for the unsupervised method. Experiments show gains in various benchmarks both for synthetic and real World datasets.