Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval
Year: 2023

Abstract:
Contrastive learning has been successfully used for retrieval of semantically aligned sentences, but it often requires large batch sizes and carefully engineered heuristics to work well. In this paper, we instead propose a generative model for learning multilingual text embeddings which can be used to retrieve or score sentence pairs. Our model operates on parallel data in N languages and, through an approximation we introduce, efficiently encourages source separation in this multilingual setting, separating semantic information that is shared between translations from stylistic or language-specific variation. We show careful large-scale comparisons between contrastive and generation-based approaches for learning multilingual text embeddings, a comparison that has not been done to the best of our knowledge despite the popularity of these approaches. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval—the last of which we introduce in this paper. Overall, our Variational Multilingual Source-Separation Transformer (VMSST) model outperforms both a strong contrastive and generative baseline on these tasks.1

heading: 1 Introduction
text: Contrastive learning is the dominant paradigm for learning text representations from parallel text (Hermann and Blunsom, 2014; Singla et al., 2018; Guo et al., 2018; Wieting et al., 2019; Feng et al., 2022). However, contrastive learning requires strong negative examples in the data and finding these negatives can be expensive in terms of compute or manual effort. In this paper, we propose a generative2 model for learning multilingual text embeddings 1Code and Flax-based T5X model checkpoint available at https://github.com/google-research/ google-research/tree/master/vmsst.
2We mean generative both in terms of text generation and as a statistical model of the joint probability distribution.
which encourages source separation, separating semantic information that is shared between translations from stylistic or language-specific variation. We find that by filtering this variation into separate variables, performance of the remaining representations, that encode shared semantic information, increases across all downstream tasks.
Through an approximation that greatly reduces the memory footprint of our model, we scale our model and train on 92 languages. We systematically compare our model, the Variational Multilingual Source-Separation Transformer (VMSST) to strong contrastive and generative baselines on a suite of tasks including semantic similarity, bitext mining, and question retrieval, which we introduce for the cross-lingual setting, using the same training data and architecture. We show that our model outperforms these models and is also competitive with the state-of-the-art.
We analyze VMSST with careful ablations, showing the contribution of each aspect of the model to performance. We also show that even at large batch sizes, the advantage over contrastive learning remains, especially for large models. Furthermore, we also find the learned embedding space of our model to be smoother, making it less affected by the “hubness problem” (Radovanovic et al., 2010; Radovanović et al., 2010) in representation learning, and more suitable for large-scale retrieval than the baseline methods.
To the best of our knowledge, this is the first work to systematically compare generative and contrastive models for learning multilingual embeddings on a large parallel corpus containing many languages in a carefully controlled experimental setup—despite the popularity of these approaches (Artetxe and Schwenk, 2019b; Yang et al., 2020). We carry out these experiments with both pretrained and randomly initialized models. The comparison of objective functions is an important research question due to the large amounts of multi-
12044
lingual text available to train models and the many uses of these models in downstream tasks. To that end, another contribution of this paper is showing these comparisons and the surprising result that contrastive objectives do not provide the overall best accuracy on downstream tasks. Moreover, our generative VMSST increasingly outperforms the contrastive model when more layers are added and when training with larger batches and more training data, suggesting that as models continue to scale in the future, this performance gap may continue to increase further motivating the use of generative approaches for learning multilingual text embeddings.

heading: 2 Related Work
text: There has been a number of approaches proposed for learning bilingual and multilingual text embeddings. One popular approach is contrastive learning (Hermann and Blunsom, 2014; Singla et al., 2018; Guo et al., 2018; Wieting et al., 2019; Feng et al., 2022) where translation pairs are positive examples and text from other pairs are used as negative examples. An alternative approach is to use a neural machine translation objective, where the representation from the hidden states of the encoder is used as the sentence embedding (Espana-Bonet et al., 2017; Schwenk and Douze, 2017; Artetxe and Schwenk, 2019b). Other approaches include multi-task learning approaches which often use some type of contrastive learning of parallel text to align representations among languages (Yang et al., 2020; Goswami et al., 2021), cross-lingual pretraining (Chi et al., 2022), and model distillation from a large pretrained multilingual model (Reimers and Gurevych, 2020).
An alternative approach that is more closely related to our work is generative models that separate the linguistic variation from the shared semantic information in translation pairs. Wieting et al. (2020) considered this for bitext, with each language having its own encoder and decoder parameters. This approach however does not scale, since it is not feasible to have thousands of encoders and decoders if one wants to model all of the more than 7,000 languages in the world.

heading: 3 Model
text: The generative process of our underlying probabilistic model and the computation graph of our training objective procedure are depicted in Fig-
ure 1 and Figure 2 respectively. In the generative story for VMSST, we first sample a semantic variable zsem for the sentence. Then for each of the N languages, we sample a language-specific variable zli . Each latent variable z is sampled from a multivariate Gaussian prior N(0, Ik). These variables are then fed into a decoder that samples each of the N sentences in the translation set. Each observed translation xli , is sampled conditioned on zsem and its language variable zli . Because zsem will be used to generate the sampled sentences in all languages, we expect that this variable will encode semantic, syntactic, or stylistic information that is shared in all of the translations. Conversely, the language variables zli will handle languagespecific peculiarities or specific style differences that are not central to the meaning of the translation and are therefore not contained in many of the sentences. Concretely, the likelihood function of our model can be written for a single N -way tuple of translations x = (x1, ..., xN ):
p(x|zsem, zl1 , ..., zlN ) = N∏
i
p(xi|zsem, zli)
In the next section, we discuss how this separation of information is encouraged during learning.

heading: 4 Learning and Inference
text: We would like to train our model on a set of parallel sentences X consisting of M examples in N languages and a collection of latent variables Z. However, N -way parallel corpora are not available at the scale of bilingual text, and so we therefore approximate an N -way parallel corpus by sampling translation pairs from a large pool of pairs containing text in N languages. Therefore in our model, X = {⟨x1li , x 1 lj ⟩, . . . , ⟨xMli , x M lj ⟩} and Z = (⟨z1li , z 1 lj , z1sem⟩, . . . , ⟨zMli , z M lj , zMsem⟩).
We aim to maximize the likelihood of the observed X with respect to the parameters of the decoder θ, marginalizing over the latent variables Z. We follow established procedures for this optimization problem from related latent variable models like variational autoencoders (VAEs; Kingma and Welling (2013)). Specifically, we optimize a variational lower bound on the log marginal likelihood, the evidence lower bound (ELBO). ELBO introduces a variational approximation q(zsem, zli , zlj |xli , xlj ;ϕ) to the true posterior of the model. The q distribution is parameterized by encoders or inference networks with parameters ϕ. ELBO can be optimized by gradient ascent by using the reparameterization trick (Kingma and Welling, 2013), which allows for the expectation under q to be approximated through sampling in a way that preserves backpropagation. The decoders and encoders are discussed in further detail in Section 5.
In contrast to variational autoencoders, which have only a single latent variable for each example, we have three in our model for each example. To encourage source separation, we make several independence assumptions for q and factor it into three terms:
q(zsem, zli , zlj |xli , xlj ;ϕ) = q(zsem|xli , xlj ;ϕ)q(zli |xli ;ϕ)q(zlj |xlj ;ϕ)
Lastly, we note that the ELBO contains a KL term that acts to regularize the latent variables. In our model, the KL term encourages zsem, zli , and zlj to be close to a zero-centered Gaussian prior. The KL term thus encourages source separation, as encoding information shared by the translation pair in the shared variable results in only a single penalty from the KL loss, while encoding the information separately in the language-specific variables
unnecessarily doubles the overall cost. In effect, we can view these language-specific latent variables as collecting information that cannot be captured in a common semantic space, separating it out from the variables collecting shared semantic information that we use for downstream tasks.
Objective Function. The overall objective function for VMSST consists of consists of two terms, the first being ELBO as described earlier:
ELBO = Eq(ZS ,ZL|X;ϕ)[log p(X|ZS , ZL; θ)]− KL(q(ZS , ZL|X;ϕ)||p(ZS ; θ)p(ZL; θ))
where ZS is the collection of semantic variables, while ZL is the collection of language variables.
The second term, which we found necessary for strong performance, is the sum of p(xli |µsemlj ) and p(xlj |µsemli ) which can be interpreted as samples from the mean of the posterior distribution using semantic variables generated from both input sentences. When training variational objectives, where the model ignores the latent variables and the learned posterior remains close to the prior. Examples of other approaches to address these issues include: (Yang et al., 2017; Kim et al., 2018; Xu and Durrett, 2018; He et al., 2019). We weight the ELBO by λ giving the total objective as:
∑
(xli ,xlj )∈X p(xli |µsemlj ) + p(xlj |µsemli ) + λELBO
Therefore, our objective resembles translation with a weighted source-separation term. We show the effectiveness of this formulation compared to a pure translation objective in our experiments in Section 6.

heading: 5 Architecture
text: Our architecture is an encoder-decoder model, where the encoder produces a single representation that is fed into the decoder. Cross-attention between the encoder and decoder is not used, therefore the decoder has no full sequence visibility and more pressure is applied on the encoder to create a semantically meaningful representation. Specifically, we follow the approach of Wieting et al. (2020) which uses a Transformer (Vaswani et al., 2017) encoder-decoder model, where the sentence embeddings are used in two places: at each layer of the decoder in place of cross-attention and in the computation of the logits.
Decoder Architecture. The decoder models p(xli |zsem, zli ; θ) for each language i (see right side of Figure 2). The inputs to the decoder are the language-specific variable zli and the semantic variable zsem, which are concatenated and used to condition the decoder to generate the reconstruction of the observed text xli . We use a single decoder for all languages.
Encoder Architecture. The encoders play an important role in the source separation as well as inference as detailed below.
In order to motivate the separation of the linguistic and semantic information we split the encoder into two parts, only sharing the embedding table. We use one of these encoders to be the semantic inference network, which produces the semantic variable. The other encoder represents the N language inference networks and produces the language variables for each language. These inference networks are shown on the left side of Figure 2. We mean-pool the hidden states followed by a linear projection to produce each variable from the encoders.
The semantic inference network, which models q(zsem|xli , xlj ;ϕ), is a multilingual encoder that encodes each language. For each translation pair, we alternate which of the two parallel sentences is fed into the semantic encoder within a batch for the ELBO term in the objective. Since the semantic encoder is meant to capture language agnostic semantic information, its outputs for a translation
pair should be similar regardless of the language of the input sentence. We use the mean of the semantic encoder as the sentence representation for downstream tasks.

heading: 6 Experiments

heading: 6.1 Constructing the Training Data
text: We follow Artetxe and Schwenk (2019b) in constructing our training data. However, since the exact data is not publicly available, we expect their may be small differences due to random sampling and different dataset versions. More specifically we sample our data from Europarl,3 United Nations (Rafalovitch and Dale, 2009),4 OpenSubtitles2018 (Lison et al., 2018),5 Global Voices,6 Tanzil,7 and Tatoeba v2021-07-22.8
We sample the same amount of data as was done in Artetxe and Schwenk (2019b), detailed in Appendix C. The only deviation being that we take care to not include any Tatoeba test data in our training data. Our final corpus has nearly 216 million training examples, slightly less than 220 million reported in Artetxe and Schwenk (2019b). We use both English and Spanish as pivot languages, so each pair includes at least one English or Spanish sentence, and we use approximately the same
3http://opus.nlpl.eu/Europarl.php 4https://opus.nlpl.eu/UN.php 5http://opus.nlpl.eu/OpenSubtitles.php 6https://opus.nlpl.eu/GlobalVoices.php 7https://opus.nlpl.eu/Tanzil.php 8https://opus.nlpl.eu/Tatoeba.php
amount of data for each language. We note that we only have training data for 92 languages instead of the 93 in Artetxe and Schwenk (2019b) due to not having training data for Aymara (ay).

heading: 6.2 Evaluation
text: We evaluate on three tasks: semantic similarity, bitext mining and question retrieval. While the first two are commonly used to evaluate multilingual sentence embeddings, we introduce question retrieval in this paper. As can be seen by our results, we found question retrieval to be somewhat uncorrelated to either of the latter two. For each task, we use a collection of different datasets, detailed below.
Semantic Textual Similarity The goal of the semantic textual similarity tasks is to predict the degree to which sentences have the same meaning as measured by human judges. The evaluation metric is Pearson’s r × 100 with the gold labels, which is convention for these tasks.
We make a distinction between two semantic similarity evaluations, English-only and crosslingual. For the English-only evaluation, we follow Wieting et al. (2016) by averaging the yearly performance on 2012–2016 SemEval Semantic Textual Similarity (STS) shared tasks (Agirre et al., 2012, 2013, 2014, 2015, 2016). More specifically, for each year of the competition, we average the Pearson’s r × 100 for each dataset in that year, and then finally average this result for each year of the competition. For the cross-lingual evaluation we use the cross-lingual STS tasks from SemEval 2017 (Cer et al., 2017). This evaluation contains Arabic-Arabic, Arabic-English, SpanishSpanish, Spanish-English, and Turkish-English STS datasets. These datasets were created by translating one or both pairs of an English STS pair into Arabic (ar), Spanish (es), or Turkish (tr). We average Pearson’s r × 100 for these datasets.
Bitext Mining For bitext mining, we use the Tatoeba dataset introduced in Artetxe and Schwenk (2019b) and the 2018 Building and Using Parallel Corpora (BUCC) shared bitext mining task (Zweigenbaum et al., 2018).
The Tatoeba dataset consists of 100–1000 pairs of data aligned to English for 112 languages. The accuracy for Tatoeba can be computed in two ways, depending if English is the target language or source language. We compute accuracy using
cosine similarity in both directions for all 112 languages (19 are unseen in the training data) and average this score for all languages.
The goal of the BUCC task is to find the gold aligned parallel sentences given two corpora (one being very large) in two distinct languages. Languages are aligned with English and consist of German (de), French (fr), Russian (ru), and Chinese (zh). Typically, only about 2.5% of the sentences are aligned. Following Schwenk (2018), we evaluate on the publicly available BUCC data. This involves scoring all pairs between the source target sentences and finding the optimal threshold that separates the data. Using the threshold, we can compute the precision, recall, and F1 of the alignments. We report F1 × 100 in our results.
We compare two different approaches for finding the sentence alignments. In the first, BUCC (cosine), we compute the cosine similarity between the non-English source sentences and the English target sentences, selecting the highest scoring English sentence as the match. In the second, BUCC (margin), we follow Artetxe and Schwenk (2019a) and use a margin-based scoring approach, where the final score of a sentence pair is both a function of the score between the pair and the scores of each sentence with its nearest neighbors. To compute this margin score, we divide the cosine similarity for source sentence si and target sentence ti by the sum of the scores of the four nearest neighbors of si with the target sentences and the sum of the scores of the four nearest neighbors of ti with the source sentences.
Margin-based scoring is designed to alleviate the “hubness problem” (Radovanovic et al., 2010; Radovanović et al., 2010) where the neighborhood around embeddings in a high-dimensional space, like in sentence embeddings, have many neighbors in common. These neighbors can displace the correct mapping in the ordering, hurting performance.
Question Retrieval For our question retrieval evaluation, we report the accuracy (R@1) on the test sets of Natural Questions (NQ) (Kwiatkowski et al., 2019) and the Multilingual Knowledge Questions and Answers (MKQA) (Longpre et al., 2021). We use the the Probably Asked Questions dataset (PAQ) (Lewis et al., 2021) as a knowledge base from which we look up the nearest neighbor of each question in the NQ and MKQA test sets using cosine similarity. PAQ is a very large resource of 65 million automatically generated question-answer
pairs. This is a zero-shot evaluation without any NQ supervised data.9
Overall Score We consolidate all of these evaluations into a score, as a way to get a sense of overall performance since different models favor different evaluations. While we are averaging different metrics (accuracy, Pearson’s r, and F1), we justify this as they do have the same scale,10 and a simple average gives a way for us to see overall performance. Our score is the average of six subtasks, two subtasks for each of semantic similarity, bitext mining, and question retrieval: English semantic similarity, cross-lingual semantic similarity, Tatoeba, BUCC (we average performance of the cosine and margin based scoring), NQ, and MKQA.

heading: 6.3 Baselines
text: We compare VMSST against two strong baselines, which have been used extensively in the literature.
The first baseline is CONTRASTIVE, where we use contrastive learning with the other sentences in the batch (“in-batch negative sampling”) as negative examples (Sohn, 2016). CONTRASTIVE is computed as the average of computing p(si|ti) and p(ti|si) for source sentence si and target sentence ti, and their respective representations si and ti where the first term uses all the other targets as negatives and the second use all of the other source sentence as negatives. Specifically,
p(si|ti) = exp(si · ti) / ∑
j∈B exp si · tj
p(ti|si) = exp(si · ti) / ∑
j∈B exp ti · sj
loss = − 1 2|B|
∑
(si,ti)∈B log p(si|ti) + log p(ti|si)
where B is a minibatch. This version of contrastive learning has been used in representation learning for retrieval (DPR, Karpukhin et al., 2020), visual tasks (SimCLR, Chen et al., 2020) and image/text tasks (CLIP, Radford et al., 2021). There are other variations of this loss (Qian et al., 2019), and other
9This is opposed to the formulation in the original paper where a model based on BART Large (Lewis et al., 2020a) was fine-tuned using a RAG-like objective (Lewis et al., 2020b) on the NQ training data in a model the authors call RePAQ. RePAQ, without using a reranker achieves an accuracy of 41.2 on NQ.
10Technically Pearson’s r can be negative, but this does not happen in our evaluations.
contrastive losses like triplet loss (Weston et al., 2010) which has been used for learning text embeddings, but we leave a comparison of contrastive objectives for learning multilingual text embeddings for future work.
The second baseline is BITRANSLATION, where we use a translation objective to learn the representation Espana-Bonet et al. (2017); Schwenk and Douze (2017); Artetxe and Schwenk (2019b).
We also explore an alternative to the VMSST, VMSST CONTRASTIVE, by incorporating a contrastive loss to use in a multitask setting. Again, we weight the contribution of the VMSST loss by λ.

heading: 6.4 Experimental Settings
text: We explore three different settings for each of four objective functions we consider. We use the Transformer architecture for all settings. Specifically, we explore a 6 layer encoder-decoder model, a 24 layer encoder-decoder model, and a 24 layer encoderdecoder initialized with the Multilingual T5 (mT5) Large (Xue et al., 2021). We set the dimension of the embeddings and hidden states for the encoders and decoders to 1024. The mT5 Large model inherently has embedding and hidden state dimensions of 1024. For all models, we use the mT5 vocabulary, which is derived from sentencepiece (Kudo and Richardson, 2018). The vocabulary consists of 250,000 tokens and was learned from multilingual variant of the C4 dataset called mC4 which includes 101 languages.
For optimization, we use Adafactor (Shazeer and Stern, 2018). We use the same learning rate schedule as Vaswani et al. (2017), i.e., the learning rate increases linearly for 4,000 steps, after which it is decayed proportionally to the inverse square root of the number of steps. We set the peak learning rate to be 0.001, and we train our models for 100,000 steps total. We use a batch size of 2048 and set the maximum sequence length of our model to 32 for all experiments.
We use a dropout rate of 0.1 for CONTRASTIVE models and no dropout for BITRANSLATION, VMSST CONTRASTIVE (with the exception of the randomly initialized 24 layer model which used 0.1), and VMSST. For VMSST, we anneal the KL term so that it increased linearly for 1,000,000 updates.
For VMSST, we set λ, the weight on the VMSST ELBO loss term, to be 0.025 for the pre-
trained models, and 0.1 when training from randomly initialized parameters. For VMSST CONTRASTIVE, we set it to .0005 for the pretrained and 6 layer settings and 0.001 for the randomly initialized 24 layer setting.

heading: 6.5 Results
text: The results of our experiments are shown in Table 1. Overall, VMSST has the best performance for all three experimental settings and the best performance on each task on average, with the exception of Tatoeba. In fact, for NQ question retrieval with a pretrained model, it performs nearly to that of the model trained specifically for this task on NQ data from Lewis et al. (2021) which has an accuracy of 41.2. VMSST and BITRANSLATION are especially strong when using more layers, which is not the case for CONTRASTIVE which declines in performance when moving from 6 to 24 layers. In fact at 24 layers, BITRANSLATION performs better on average than CONTRASTIVE. Perhaps for even larger models, the gap between contrastive and generative models will increase. We also see that CONTRASTIVE seems to benefit more from pretraining than VMSST and BITRANSLATION, which could possibly be due to VMSST re-purposing and adding additional randomly initialized param-
eters to the decoder. Perhaps different pretraining strategies using this modified decoder would resolve these differences. We also see that VMSST CONTRASTIVE has negligible improvement over CONTRASTIVE which was unexpected —that is, a traditional contrastive loss does not improve further on top of generative loss of VMSST. We leave the exploration of different strategies of combining these approaches to future work.
It is also interesting to observe the stark performance difference for different tasks. Bitext mining tasks like Tatoeba, and BUCC (m.) for the pretrained 24 layer model, favor CONTRASTIVE, while semantic similarity, BUCC (c.) and question retrieval favor VMSST, suggesting some fundamental difference in these tasks favoring CONTRASTIVE. An examination of the Tatoeba and BUCC data shows that there are paraphrases in the test set, but accounting for these does not seem to meaningfully explain this performance difference.
Lastly, we see that VMSST outperforms CONTRASTIVE on the BUCC task with cosine similarity, though the results between the two models are closer when using margin. This suggests that the “hubness problem” (Radovanovic et al., 2010; Radovanović et al., 2010) where the neighborhood around embeddings in a high-dimensional spaces
have many neighbors in common, is less of an issue when learning embeddings with VMSST. This smoother embedding space may also contribute to the stronger results VMSST has on the question retrieval tasks.

heading: 6.6 Comparison to Related Work
text: Prior work on learning multilingual embeddings has explored a variety of models utilizing different strategies and using difference source and types of training data. However, comparing approaches is difficult as they differ in many factors that are crucial to performance: training data, model size, architecture, vocabulary, training time, and evaluation datasets. Complicating matters further, even the metric used in evaluation for the same dataset, the distance measure used between embeddings for the same dataset, and the specific subsets of the evaluation datasets used can be different.
The main goal of this paper is to compare contrastive and generative losses systematically and uniformly, on the same data, metrics and underlying architecture. However, we also emphasize that the best systems we compare are competitive with the current state-of-the-art. Hence, in this section we compare VMSST to published results of other models on semantic similarity and the Tatoeba and BUCC bitext mining tasks. We primarily compare against five models which have the strongest multilingual results in the literature: mUSE (Yang et al., 2020), LASER (Artetxe and Schwenk, 2019b), XLM-R (NLI/STS-B) and XLM (Para.) (Reimers and Gurevych, 2020), and LaBSE (Feng et al., 2022).
For semantic similarity, we include Spearman’s ρ in order to compare to work that solely uses this correlation metric. We use cosine as the similarity measure for all models in these evaluations.11 The results are shown in Table 2.
For Tatoeba, we compare to methods that have evaluated on all 112 languages, which excludes mUSE as it was only trained on 16 language pairs. The results are shown in Table 3. Baselines results are taken from Reimers and Gurevych (2020).
For BUCC, we include results on the training sets using the margin retrieval methods from Artetxe and Schwenk (2019b). The results are
11Note that mUSE and LaBSE report results using the angle as the metric instead of its cosine for semantic similarity tasks, but as they do not evaluate on these specific datasets, we include the results from Reimers and Gurevych (2020) for comparison which uses cosine similarity.
shown in Table 5. Baselines results are taken from Artetxe and Schwenk (2019b); Reimers and Gurevych (2020).
While VMSST does not have the best performance relative to models from the literature on any single task, it does have the best overall performance if one averages the results for each task.12 While these models share much in common, namely using parallel text and some type of pretraining or pretrained model, there are differences in the exact data and models used, among other confounding variables. For instance, LaBSE used training data consisting of six billion parallel pairs across languages and was also trained on monolingual text using a masked language modelling objective. XLM-R (Para.) makes use of a 50 million example paraphrase corpus for distillation. In contrast, our setup most closely follows LASER, using an approximation of the 220M example parallel data used to train their model.
12The average performance for VMSST is 84.3, versus 82.6 for LaBSE, and 79.3 for XLM-R (Para.)

heading: 7 Analysis
text: In this section, we analyze VMSST with additional experiments. We give a high-level overview in this section and put details and results in Appendix A.
We first investigate different ablations of the model. We analyzed aspects such as factorizing the projection layer, using weaker decoders, using 4 language-specific encoders instead of 1, removing the KL term, using a single encoder for the semantic and language embeddings, and removing the language embeddings from the encoder and from the decoder.
Secondly, we analyze the effect of the parameter sharing approximation in VMSST, where we train a full model with separate encoders and decoders for each language. This experiment uses data in 4 languages to make this experiment tractable. We found the performance to be similar enough that we can say the approximation holds, but there does remain a small gap. We hypothesize however, that this performance gap will shrink as the number of layers of the model increases.
Thirdly, we evaluate the performance of zeroshot bitext mining on languages that were unseen in the training data. We find significant improvement in this setting over the baseline BITRANSLATION. Since BITRANSLATION can be seen as an ablation of VMSST, we see that the source-separation loss especially helps with generalization to new languages.
Lastly, we investigate the impact of batch size on performance, comparing VMSST with CONTRASTIVE. It is common knowledge that contrastive models learn better representations when given harder negative examples, and bigger batch sizes increases the chances of finding these harder negatives. We experiment with batch sizes of 4096 and 8192, for both the 6 layer and 24 layer randomly initialized versions of CONTRASTIVE and VMSST. We find that both models improve when trained with larger batches, with the very best model being the 24 layer VMSST.

heading: 8 Conclusion
text: We present VMSST, a generative massively multilingual text embedding model trained to separate semantic information from language-specific information. VMSST also outperforms strong contrastive and generative baselines on a variety of tasks. There are several avenues for future work including alternative pretraining objectives that better fit the use case of the decoder, explore incorporating monolingual data into the generative objective, investigate synergy between VMSST and contrastive methods as they seem to specialize in different tasks, and lastly scale up to bigger models, more data, and languages to further investigate VMSST versus contrastive methods.

heading: Limitations
text: Some of our experiments, specifically those in the ablations with large batch sizes, required significant computational resources. We trained these models on Google Cloud TPUv3 Pod slice with 128 chips for a few days. This experiment is important, as otherwise there would be questions on how the models compare at large batch sizes where contrastive models are known to work better. Due to training costs and in the interest of open research, we will open source our code and model checkpoints for the community to use and build upon.
Secondly, VMSST and BITRANSLATION require decoding which which means they need more memory for the decoder and are slower during training. However one advantage of these models is that they can be trained with gradient checkpointing greatly reducing their memory requirements, which cannot be used for the contrastive models as that would reduce the effective batch size for finding negative examples. Moreover, during inference, there is no difference in the memory or speed requirements in CONTRASTIVE, BITRANSLATION, or VMSST as only a single encoder is used in inference and there is no decoding.

heading: Acknowledgements
text: We are grateful to Livio Baldini-Soares, Wenhu Chen, Zhuyun Dai, Tom Kwiatkowski, Jianmo Ni, Slav Petrov, Jason Riesa, and Pat Verga for useful discussions during the course of the project.

heading: Appendices accompanying “Beyond Contrastive Learning: A Variational Generative Model for Multilingual
text: Retrieval”

heading: A Analysis

heading: A.1 Model Ablations
text: In this section, we investigate different ablations of VMSST. The ablations are shown in Table 4. We start from the 24 layer randomly initialized VMSST, and change it to see how certain hyperparameters and model choices affect performance.
Our first experiment, VMSST (fact.) investigates what happens if we simply factor the final projection layer of the decoder. This can save a lot of memory in the model, as that projection layer is 3 × d × V where d is the hidden state dimension size and V is the size of the vocabulary.13 If we factor the projection layer, we can reduce the space to d × V + 3d × d. In practice, this saves about 509 million parameters for our 24 layer models. However from the first row in Table 4, we see that this small change has a significant effect on performance, weakening results on semantic similarity and question retrieval tasks and strengthening results on bitext mining tasks.
In our second ablation, VMSST (4 enc.), we spread the model capacity of the language-specific encoder to 4 encoders, instead of the single encoder in our previous experiments. We allocate the languages randomly to the different encoders. We find that this doesn’t improve results, perhaps because the 24 layer model has sufficient capacity to model all of the languages in one shared encoder. We could allocate languages to encoders based on language families, and perhaps this could fare better, but we leave that for future work.
Prior work (Wieting et al., 2020) shows that, a decoder that is weaker (i.e. less layers) can lead to stronger embeddings. This effect is presumably because there is more pressure on the sentence embedding to fully and clearly capture the semantics since it cannot rely on a strong decoder to fill in gaps. We found that that using a weaker single layer decoder (1L dec.), does indeed seem to improve performance. We also tried a 12 layer ablation (12L dec.), but that seemed to not have a significant improvement in the results.
13We multiply by 3 because we have three embeddings, the hidden state, the language-specific vector, and the semantic vector.
The last four ablations investigate different modelling choices. In the first we eliminate the KL term (no KL), which has the most significant effect on performance, especially on cross-lingual tasks. In the second ablation, we use a single encoder instead of the twin encoders (1 enc.), one for semantic embeddings and one for language embeddings, we find that this has a modest overall effect on performance. Lastly, we eliminate the language embeddings. First we remove the language embedding inputs to the decoder (no enc. l.e.), then we experiment by removing the input language embeddings to the language-specific encoder (no dec. l.e.). We find these language embeddings have a smaller than expected impact on performance, perhaps because the large capacity of the decoder can ascertain the language being input or decoded.

heading: A.2 Testing the Parameter Sharing in VMSST
text: Parameter sharing was needed in order efficiently perform source separation on N languages. Specifically we collapsed the language encoders into a single encoder and we collapsed the decoders into a single decoder. The VMSST approximates having N language encoders by using an input embedding to indicate the language being considered. The same strategy is applied with the decoders as well, with the first input token to the decoder indicating the language to be generated.
In this section, we investigate what effect this parameter sharing has on VMSST by using N encoders and decoders (full enc, full dec.). We experiment with 6 layer Transformer encoders and 4 languages Spanish, English, Arabic, and Turkish in order to keep the experiments tractable as in this setting we have 5 encoders and 4 decoders. The results are shown in Table 6.
The results indicate that the approximation appears to hold, as VMSST is much closer to the full model than BITRANSLATION, which is an ablation of VMSST without the source separation. However, there is still a gap between the full encoder/decoder of VMSST and VMSST. We hypothesize however, that as the number of layers of the model increases, this performance gap also shrinks. The extra capacity of these layers will allow for the model to separate language-specific variations without having separate parameters for each language. Evidence for this hypothesis is in Table 4 where having the language variation shared
amongst 4 encoders instead of 1 actually appears to weaken performance overall.

heading: A.3 Zero-Shot Bitext Mining
text: The Tatoeba dataset contains parallel sentence pairs of English with 112 languages. Our model is trained using 93 of these languages, and therefore there are 19 languages we can use for a zeroshot evaluation of bitext mining. Table 8 summarizes the results of this zero-shot evaluation for the two generation objectives, BITRANSLATION and VMSST considered in this paper. The results are shown in Table 8. We also compute ∆ which is the difference between the performance gap of VMSST and BITRANSLATION on the seen and unseen languages. From the results, we see that VMSST does even better than BITRANSLATION on unseen languages than unseen languages. Since BITRANSLATION can be seen as an ablation of VMSST, i.e. VMSST without the sourceseparation loss, we see that the source-separation loss especially helps with generalization to new languages.

heading: A.4 Effects of Batch Size
text: Lastly, we investigate how VMSST compares to CONTRASTIVE as batch size increases. It is common knowledge that contrastive models learn better representations when given harder negative examples. Since we are using in-batch negatives in our contrastive baseline, the increased batch size increases the chances of encountering harder negative examples and will generally increase performance up to the point where the negatives become false. Furthermore. bigger batch sizes are known to also improve results in models using the Transformer architecture, presumably due to less noisy gradients, which would improve the results of both CONTRASTIVE and VMSST. It is important to note that using bigger batch sizes, means seeing
more examples (100,000 steps at a batch size of 2048 is about 1 pass through the data). However, parallel data is so numerous that training to convergence on the available data is not very practical. Therefore, these experiments do not separate out the gains from using a bigger batch size versus seeing more training data, but we argue that is not an important distinction to make due to the sheer amount (billions of pairs) of parallel data available.
We experiment with batch sizes of 4096 and 8192, double and quadruple the 2048 used in all experiments up to this point, for both the 6 layer and 24 layer randomly initialized versions of CONTRASTIVE and VMSST. All models are trained again for 100,000 steps. The results are shown in Table 7.
From the results, we see that for the 6 layer model, increasing the batch size equalizes VMSST and CONTRASTIVE overall, however each performs better at different tasks. CONTRASTIVE has better performance on Tatoeba, XL semantic similarity, and BUCC with margin (Artetxe and Schwenk, 2019a), where VMSST has better performance on English semantic similarity, BUCC with cosine similarity, and the retrieval tasks. For the 24 layer variations, VMSST is better at every task, with the exception of Tatoeba, and has the highest overall score of any model in the table. The 24 layer CONTRASTIVE variation does not perform as well as the 6 layer version at any batch size, in contrast to VMSST where the 24 layer model always outperforms the 6 layer variation.

heading: B Full Experimental Results
text: We include full results for our models using the pre-trained mT5 large checkpoint. We evaluate on English semantic similarity, Cross-lingual semantic similarity, question retrieval, and bitext mining.

heading: B.1 Semantic Similarity
text: For English semantic similarity, we use the SemEval semantic textual similarity (STS) tasks from 2012 to 2016 (Agirre et al., 2012, 2013, 2014, 2015, 2016) as was done initially for sentence embeddings in (Wieting et al., 2016). As our test set, we report the average Pearson’s r over each year of the STS tasks from 2012-2016 as is convention in the top part of Table 9. However, some recent work, like Reimers and Gurevych (2019) computed Spearman’s ρ over concatenated datasets for each year of the STS competition. To be consistent with these works, we also include evaluations using this approach in the bottom part of Table 9. One other difference between these two ways of calculating the results is the inclusion of the SMT dataset of the 2013 task. When computing the results using Pearson’s r, this dataset is included, but when
computing the results using Spearman’s ρ, it is not included.
For cross-lingual semantic similarity and semantic similarity in non-English languages, we evaluate on the STS tasks from SemEval 2017. This evaluation contains Arabic-Arabic, Arabic-English, Spanish-Spanish, Spanish-English, and TurkishEnglish datasets. The datasets were created by translating one or both pairs of an English STS pair into Arabic (ar), Spanish (es), or Turkish (tr). Following convention, we report results with Pearson’s r for all systems, but also include results in Spearman’s ρ in Table 10.

heading: B.2 Question Retrieval
text: For our question retrieval evaluation, we report the accuracy (R@1) on the test sets of Natural Questions (NQ) (Kwiatkowski et al., 2019) shown in Table 11 and the Multilingual Knowledge Questions and Answers (MKQA) (Longpre et al., 2021) shown in Table 12. We use the the Probably Asked Questions dataset (PAQ) (Lewis et al., 2021) as a knowledge base from which we look up the nearest neighbor of each question in the NQ and MKQA test sets using cosine similarity.

heading: B.3 Bitext Mining
text: For bitext mining, we use the Tatoeba dataset introduced in Artetxe and Schwenk (2019b) and the 2018 Building and Using Parallel Corpora (BUCC) shared bitext mining task (Zweigenbaum et al., 2018).
The Tatoeba dataset consists of 100-1000 pairs
of data aligned to English for 112 languages. The accuracy for Tatoeba can be computed in two ways, depending if English is the target language or source language. We compute accuracy using cosine similarity in both directions for all 112 languages (19 are unseen in the training data) and average this score for all languages.
The goal of the BUCC task is to find the gold aligned parallel sentences given two corpora (one being very large) in two distinct languages. Languages are aligned with English and consist of German (de), French (fr), Russian (ru), and Chinese (zh). Following Schwenk (2018), we evaluate on the publicly available BUCC data. This involves scoring all pairs between the source target
sentences and finding the optimal threshold that separates the data. Using the threshold, we can compute the precision, recall, and F1 of the alignments. We report F1 × 100 in our results.
We compare two different approaches for finding the sentence alignments. In the first, BUCC (cosine), we compute the cosine similarity between the non-English source sentences and the English target sentences, selecting the highest scoring English sentence as the match. In the second, BUCC (margin), we follow Artetxe and Schwenk (2019a) and use a margin-based scoring approach.

heading: C Full Training Data
text: We follow Artetxe and Schwenk (2019b) in constructing our training data, sampling data from Europarl,14, United Nations (Rafalovitch and Dale, 2009),15 OpenSubtitles2018 (Lison et al., 2018),16, Global Voices,17 Tanzil,18 and Tatoeba v2021-07-
14http://opus.nlpl.eu/Europarl.php 15https://opus.nlpl.eu/UN.php 16http://opus.nlpl.eu/OpenSubtitles.php 17https://opus.nlpl.eu/GlobalVoices.php 18https://opus.nlpl.eu/Tanzil.php
22.19
The only deviation from their data sampling approach is that we take care to not include any Tatoeba test data in our training data. Our final corpus has nearly 216 million training examples, slightly less than 220 million reported in Artetxe and Schwenk (2019b). We use both English and Spanish as pivot languages, so each pair includes at least one English or Spanish sentence, and attempt to use approximately the same amount of data for each language if possible. We note that we only have training data for 92 languages instead of the 93 in Artetxe and Schwenk (2019b) due to not having training data for Aymara (ay). The full amount of English and Spanish parallel data used for each of the 92 languages is reported in Table 15.
19https://opus.nlpl.eu/Tatoeba.php
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
In the Limitations section 8.
7 A2. Did you discuss any potential risks of your work? This is textual similarity model that does not generate text, trained on publicly available bitext. It does not carry any novel risks from previous generic textual embedding models.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Introduction is Section 1.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Yes, Appendix H.
3 B1. Did you cite the creators of artifacts you used? Yes, Appendix H.
7 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? They all have licenses applicable to academic publishing and model release.
7 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? No need for discussion as our use (academic research) is consistent with the implied intended use of these datasets.
7 B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? These are well-studied standard academic translation datasets - there is no PII info and little offensitve content (possibly in the subtitles data for the latter).
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Appendix H
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section 6.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
C 3 Did you run computational experiments? Section 6.
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Section 6 and Section 8.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Appendix B
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Appendix B
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Appendix B
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? No response.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? No response.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? No response.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? No response.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? No response.

heading: Future Work
text: We could allocate languages to encoders based on language families, and perhaps this could fare better, but we leave that for future work. Prior work (Wieting et al. , 2020) shows that, a decoder that is weaker (i. e. less layers) can lead to stronger embeddings. This effect is presumably because there is more pressure on the sentence embedding to fully and clearly capture the semantics since it cannot rely on a strong decoder to fill in gaps. We found that that using a weaker single layer decoder (1L dec. ), does indeed seem to improve performance. We also tried a 12 layer ablation (12L dec. ), but that seemed to not have a significant improvement in the results. 13We multiply by 3 because we have three embeddings, the hidden state, the language-specific vector, and the semantic vector. The last four ablations investigate different modelling choices. In the first we eliminate the KL term (no KL), which has the most significant effect on performance, especially on cross-lingual tasks. In the second ablation, we use a single encoder instead of the twin encoders (1 enc. ), one for semantic embeddings and one for language embeddings, we find that this has a modest overall effect on performance. Lastly, we eliminate the language embeddings. First we remove the language embedding inputs to the decoder (no enc. l. e. ), then we experiment by removing the input language embeddings to the language-specific encoder (no dec. l. e. ). We find these language embeddings have a smaller than expected impact on performance, perhaps because the large capacity of the decoder can ascertain the language being input or decoded