Constrained Density Matching and Modeling for Effective Contextualized Alignment

Abstract: Multilingual representations pre-trained with monolingual data offer unmatched task performances between languages. While this has been tackled through the lens of contextualized alignments, these techniques require large parallel data, thereby leaving under-represented language communities behind. In this work, we analyze the limitations according to which previous alignments become very resource-intensive, \emph{viz.,} (i) the inability to sufficiently leverage data and (ii) that alignments are not trained properly. To address them, we present density based approaches to perform alignments, and we complement them with our validation criteria accounting for downstream task performances. Our experiments encompass 16 alignment techniques (including ours), evaluated across 6 language pairs, synthetic and 4 NLP tasks. We demonstrate that our solutions are particularly effective in the scenarios of limited and no parallel data. More importantly, we show, both theoretically and empirically, the advantages of our boostrapping procedures, by which unsupervised approaches rival supervised counterparts. 



Reviews and author responses:


This paper propose methods for contextualized vector alignment, using normalizing flow and earth mover distance for supervised and unsupervised alignment respsectively. 
Experiments demonstrate the effectiveness of the proposed approaches.  ### Strengths 
- This is a well-written paper in general, with reasonable approaches for both supervised and unsupervised settings.

- The paper includes rigorous proofs that provide clear intuitions on the algorithm design.

- The paper includes comparisons to a wide range of baselines that require significant effort.

### Weakness

- The major downside, in my opinion, is that the paper lacks comparison to the popular unsupervised "supervised" approach [1, 2]. In summary, [1] is an unsupervised machine translation model, whereas [2] proposes using the bitext built by [1] to perform multilingual NLP tasks. In terms of supervision, the unsupervised bitext built by [1] or similar techniques can be used as a free lunch.

- According to Table 4 (supp. ), the majority of natural languages involved in experiments have no significant typological difference from English, while the English-Turkish results are usually lower than the overall best by a significant margin—-this makes me concerned that it may be overfitting nearby languages (to English).

- Presentation of results could be improved.

### Other comments and questions

- Abstract: I don't think the paper has theoretically shown the advantage of the bootstrapping method. While I agree that the proofs are correct, reasonable and valuable for algorithm design, they do not theoretically compare to existing methods. I would suggest the authors to weaken their argument slightly.

- Footnote 1: I agree that bringing the dev criteria issue into folks' attention is very valuable. After reading the rest of the paper, I don't think the argument here is very strong. On the one hand, any unsupervised criterion cannot guarantee that it works for all downstream tasks (we can always construct counter-example tasks, though maybe unrealistic); on the other hand, loss on a held-out development set can always be a natural criteria for model selection when training Wu & Dredze or Cao et al. (or even more) models, but was not examined. 

- Sec. 3.1: Although I believe that most appropriate readers could work out the "dual form of density matching based on JS divergence" based on Eq (2), it is always good to write it out explicitly. Referring to supplementary material is way better than omitting it. 

- Sec 4.3: I am confused by the settings of CBLI. How do the training, dev and test sets look like? \
Also, methods such as [2] have been proposed to perform BLI using modern large pretrained models, though not necessarily perform vector alignment. Why would we need the CBLI task given this?

- Lemma 2: I am confused about the last sentence in the proof. Different $\mathbf{M}_\tilde{Y}$ and $\mathbf{M}_Y$, if I understood correctly, should correspond to different $\mathbf{U}$.
I assume that you would like to show that for any pair of $\mathbf{M}_\tilde{Y}$ and $\mathbf{M}_Y$, there exists a $\mathbf{U}$ that satisfies the desirable condition, but the last sentence has a different meaning. Can you elaborate a bit on this? 

- Personally, I think extrinsic downstream tasks may strengthen this paper. 
Some relevant references [3, 4] are listed below, where they propose similar methods for contextualized multilingual alignment, and evaluate cross-lingual alignment by cross-lingual dependency parsing. 

- As said above, the presentation of results needs improve: the current Table 1 is very dense and difficult to digest. Some potential ideas are: 
    - Pointing out the results of the new method proposed in the paper, e.g., by "(ours)", and make it the first or the last rows. 
    - Breaking out into several tables, and only include results relevant to a specific argument. 

### Missing Reference
[1] https://arxiv.org/pdf/2006.09526.pdf

[2] https://aclanthology.org/2021.acl-long.67/ 

[3] https://aclanthology.org/N19-1162.pdf 

[4] https://aclanthology.org/D19-1575.pdf In summary, I think this paper has a strong theoretical motivation, and the proposed approach is reasonable and valuable. Empirically, the paper also emphasizes the importance of development criteria compared to simply training for arbitrary epochs. 

As for weaknesses, I am mainly concerned about the missing of unsupervised baselines, lack of extrinsic evaluation, and the potential overfit towards English and nearby language pairs. 


This paper deals with alignment between embeddings of different languages. The embeddings are contextualized representations produced by multilingual text encoders. Although alignment between static embeddings have been thoroughly studied, alignment between contextualized representations have not, and existing works use large parallel data. This work proposes a density matching approach to this task, using limited or no supervision. The authors contribute ideas of normalizing flows, EMD, constraints like cross-correlation, validation criteria, etc. to this task, and verify their approach with empirical results. The authors formulate the task as density matching, and use various techniques for supervised and unsupervised settings. Although most of the ideas have been used for other tasks, combining them for this task is interesting.
My main complaint is the clarity of the simulation experiment, especially the generation of simulation data. For example: “...their embeddings are sampled from a Gaussian distribution... \mu_i denotes a mean vector sampled uniformly from [-5;5] for each component.” I guess this is “each component of the Gaussian mixture” because there is a previous statement “embeddings sampled from a two-dimensional Gaussian mixture distribution”? How do k and t affect the generation of simulation data? I guess a word is sampled k times, but I cannot understand how t could reflect the degree of language isomorphism.
 This paper presents interesting techniques and empirical insights for the alignment of multilingual contextualized representations. The clarity of the paper could be improved.

This paper considers density based supervised and unsupervised alignment of contextual embeddings with Real-NVP and Wasserstein GAN, respectively. For unsupervised alignment, an additional bootstrap step is considered. Additionally, it considers two model selection criteria. The experiment focuses on intrinsic (as opposed to its extrinsic impact on cross-lingual systems) evaluation on representation alignment.
 
# Strengths

This paper addresses unsupervised alignment of contextual embeddings with density matching and model selection criteria, and evaluates various methods on intrinsic representation evaluation. Ablation study of the proposed method is also presented.

# Weaknesses
* The evaluation does not consider any extrinsic evaluation on cross-lingual transfer. While the intrinsic evaluation (word retrieval: CBLI and Align) offer some insight into the representation, only Tatoeba evaluation offers a glimpse at its practical implication. As most contextual embeddings are used for cross-lingual transfer (parallel sentence retrieval/bitext mining could use sentence embeddings instead), not evaluating on cross-lingual transfer undermines the impact of the proposed method, casting doubts on its effectiveness for downstream systems. Based on the current experiment, the reader is not sure of the impact of the proposed alignment method on downstream systems. 
* While this paper shows that the proposed method works well with 20k parallel sentences, it is unclear whether the proposed method is more data efficient. Comparison against additional bitext should be presented to support the claim of addressing “the inability to sufficiently leverage data”. Additionally, the limited data setup is synthetic as the 20k bitext is sampled from a much larger dataset. To show that the proposed method works well with limited data, experiments with low resource language pairs should be presented. With the current design, it is unclear how well the proposed method works in real world setup: both high resource language pairs (using all available data instead of subsampling to create low resource language pairs) and truly low resource language pairs.
 This paper considers density matching and model selection criteria for supervised and unsupervised contextual embeddings alignment. However, the experiment does not support the claim with limited evaluation and experiment design.


This work presents a new method for aligning contextualized embedding using density matching. Prior work on embedding alignment relies on non-contextualized embedding representation, thus they have limited capability when aligning representations in two languages. This work introduces density matching via normalizing flows so that contextualized representations are matched in two languages. Two methods are investigated: one employ permutation matrix to align representation in a supervised manner. The second one employs Earth Mover distance under Wasserstein GAN for matching densities in an unsupervised manner. A carefully designed bootstrapping procedure is introduced for stabler training for the unsupervised method. Experiments show gains in various benchmarks both for synthetic and real World datasets. # Strength

- The proposed method is novel in that the representation alignment in two languages are treated as density matching by transferring representations by normalizing flows. The use of KL divergence as objective function in the supervised setting is sound, and it is very intuitive to extend the idea to the unsupervised setting by introducing the Earth Mover distance in Wasserstein GAN.

- Experiments are well designed with solid results both on synthetic datasets and real datasets used as standard benchmarks.

# Weakness

- The performance is measured by averages, but I'd like to see individual results given that the languages are quite differs in various aspects, e.g., syntax, and thus, might have large impact by employing contextual representations.

- This work could experiment other variants of normalizing flows. At least, it would be good to mention why Real-NVP was employed in the proposed method. This work is a novel contribution for unsupervised alignment of embedding representations. There exists minor weakness, though, the proposed method might have an impact for the future research, and I'd suggest acceptance to this submission.

Meta-review: The paper investigates various methods for cross-lingual alignment of contextual word embeddings. It also introduces a new method based on density matching via normalizing flows to align contextual representations in two languages. The paper has many strengths, but also reviewers identify several major weaknesses including the lack of strong baselines and the lack of extrinsic evaluation. These concerns were not addressed by the authors during the discussion period.

Decision: Reject