Improving Zero-shot Multilingual Neural Machine Translation by Leveraging Cross-lingual Consistency Regularization
Year: 2023

Abstract:
The multilingual neural machine translation (NMT) model has a promising capability of zero-shot translation, where it could directly translate between language pairs unseen during training. For good transfer performance from supervised directions to zero-shot directions, the multilingual NMT model is expected to learn universal representations across different languages. This paper introduces a cross-lingual consistency regularization, CrossConST, to bridge the representation gap among different languages and boost zero-shot translation performance. The theoretical analysis shows that CrossConST implicitly maximizes the probability distribution for zero-shot translation, and the experimental results on both low-resource and high-resource benchmarks show that CrossConST consistently improves the translation performance. The experimental analysis also proves that CrossConST could close the sentence representation gap and better align the representation space. Given the universality and simplicity of CrossConST, we believe it can serve as a strong baseline for future multilingual NMT research.

heading: 1 Introduction
text: The objective of multilingual neural machine translation (NMT) is to construct a single, comprehensive model capable of translating between any pair of languages (Firat et al., 2016; Ha et al., 2016; Gu et al., 2018; Zhang et al., 2020; Fan et al., 2021). This not only benefits low-resource translation (Aharoni et al., 2019), but also enables zeroshot translation (Gu et al., 2019). The success of zero-shot translation depends on the capability of the model to learn language-agnostic representations. The conventional multilingual NMT model (Johnson et al., 2017), however, often struggles with learning the universal representations among different languages (Figure 1 (a)), which leads to poor zero-shot translation performance, particu-
larly compared to the pivot-based methods (Cheng et al., 2017).
Several methods have been proposed to improve the zero-shot translation performance by learning language-agnostic representations and maximizing cross-lingual transfer. Some approaches modify the model architecture to achieve universal representations (Lu et al., 2018; Ji et al., 2020; Liu et al., 2021; Chen et al., 2021), while others utilize auxiliary training objectives to encourage similarity between the representations of different languages (Arivazhagan et al., 2019; Al-Shedivat and Parikh, 2019; Pham et al., 2019; Pan et al., 2021). Specifically, Gu and Feng (2022) introduce an agreementbased training approach to help the multilingual NMT model make consistent predictions based on the semantics-equivalent sentences. However, most existing methods are far from being widely used due to the degraded supervised translation performance, complicated algorithm implementation, and tedious hyperparameter search.
In this paper, our primary goal is to provide a simple, easy-to-reproduce, yet effective strategy
12103
for learning multilingual NMT. Inspired by Gao et al. (2022), which boost the NMT performance by leveraging intra-lingual consistency regularization, we here propose a cross-lingual consistency regularization method, CrossConST, to learn the universal representations across different languages (Figure 1 (b)) for boosting the zero-shot translation performance, where we introduce the explicit constraints to the semantic-equivalent sentence pairs by leveraging Kullback-Leibler (KL) regularization. The contributions of this paper can be summarized as follows:
• We propose CrossConST, a simple but effective method with only one hyperparameter for improving the generalization of the multilingual NMT model, and theoretically prove that it implicitly maximizes the probability distribution for zero-shot translation.
• Our experimental results show that CrossConST achieves significant zero-shot translation improvements over the Transformer model on both low-resource and highresource multilingual translation benchmarks and outperforms the state-of-the-art (SOTA) methods OT & AT (Gu and Feng, 2022) and mRASP2 (Pan et al., 2021) on average.

heading: 2 Cross-lingual Consistency for Multilingual NMT
text: In this section, we formally propose CrossConST, a cross-lingual consistency regularization for learning multilingual NMT. We first review the multilingual neural machine translation (Section 2.1), then introduce our method in detail (Section 2.2). We theoretically analyze the regularization effect of CrossConST (Section 2.3) and propose a two-stage training strategy (Section 2.4).

heading: 2.1 Multilingual Neural Machine Translation
text: Define L = {L1, ..., LM}, where L is a collection of M languages. The multilingual NMT model refers to a neural network with an encoder-decoder architecture, which receives a sentence in language Li as input and returns a corresponding translated sentence in language Lj as output. Assume x = x1, ..., xI and y = y1, ..., yJ that correspond to the source and target sentences with lengths I and J , respectively. Note that x1 denotes the language identification token to indicate the target language the multilingual NMT model should translate to, and yJ denotes the special end-of-sentence
symbol ⟨eos⟩. The encoder first maps a source sentence x into a sequence of word embeddings e(x) = e(x1), ..., e(xI), where e(x) ∈ Rd×I , and d is the embedding dimension. The word embeddings are then encoded to the corresponding hidden representations h. Similarly, the decoder maps a shifted copy of the target sentence y, i.e., ⟨bos⟩, y1, ..., yJ−1, into a sequence of word embeddings e(y) = e(⟨bos⟩), e(y1), ..., e(yJ−1), where ⟨bos⟩ denotes a special beginning-of-sentence symbol, and e(y) ∈ Rd×J . The decoder then acts as a conditional language model that operates on the word embeddings e(y) and the hidden representations h generated by the encoder.
Let Si,j denote the parallel corpus of language pair (Li, Lj), and S denotes the entire training corpus. The standard training objective is to minimize the empirical risk:
Lce(θ) = E (x,y)∈S [ℓ(f(x,y; θ), ÿ)], (1)
where ℓ denotes the cross-entropy loss, θ is a set of model parameters, f(x,y; θ) is a sequence of probability predictions, i.e.,
fj(x,y; θ) = P (y|x,y<j ; θ), (2)
and ÿ is a sequence of one-hot label vectors for y.

heading: 2.2 CrossConST: A Cross-lingual Consistency Regularization for Multilingual NMT
text: Consider the multilingual NMT model as a function f(x,y; θ), which could be further decomposed as follows:
f(x,y; θ) := fdec(fenc(x; θenc),y; θdec), (3)
where fenc(·) and fdec(·) denote the encoder and decoder, and θenc and θdec are the sets of parameters for the encoder and decoder respectively. An ideal multilingual NMT model should have the following properties:
• The encoder should output universal representations which are language agnostic. Semantic-equivalent sentences in different languages should share similar representations in the encoder output.
• Given the target language to which the multilingual NMT model should translate to, the decoder should make consistent predictions based on the semantic-equivalent representations in the encoder output.
The main idea of our method is to close the representation gap among semantic-equivalent sentences in the encoder output and force the output distribution of the decoder to be consistent among different semantic-equivalent representations. During the training of multilingual NMT model, for each sentence pair (x,y), the training objective of CrossConST is defined as:
LCrossConST (θ) = Lce(θ) + αLkl(θ), (4)
where
Lkl(θ) = KL(f(x,y; θ)∥f(y,y; θ)), (5)
KL(·∥·) denotes the Kullback-Leibler (KL) divergence of two distributions, and α is a scalar hyperparameter that balances Lce(θ) and Lkl(θ). Note that the gradient could be backpropagated through both sides of the KL regularization in CrossConST. Figure 2 illustrates CrossConST regularization for learning multilingual NMT model.
Note that the constraint introduced by (5) forces the equivalence between f(x,y; θ) and f(y,y; θ), which implicitly leads to
fenc(x; θenc) = fenc(y; θenc). (6)
Semantic-equivalent sentences x and y then share similar representations in the encoder output, and the decoder makes consistent predictions based on the semantic-equivalent representations fenc(x; θenc) and fenc(y; θenc). The properties of the ideal multilingual NMT model implicitly hold.

heading: 2.3 Theoretical Analysis
text: Consider training a multilingual NMT model on the English-centric dataset, where x and y denote the sentences in two non-English languages, and
z denotes the English sentence. Let’s consider the zero-shot translation direction x→ y. Inspired by Ren et al. (2018) and Wang et al. (2021), we take a different approach to modeling the translation probability P (y|x; θ). We introduce language z as a bridge to connect x and y. Following Jensen’s Inequality, we could derive the lower bound of P (y|x; θ) over the parallel corpus S as follows:
L(θ) = ∑
(x,y)∈S logP (y|x; θ)
≥ ∑
(x,y)∈S
∑
z
Q(z; θ) log P (y|z; θ)P (z|x; θ)
Q(z; θ)
:=L̄(θ),
and the gap between L(θ) and L̄(θ) could be calculated as follows:
L(θ)− L̄(θ) = ∑
(x,y)∈S
∑
z
Q(z; θ) log Q(z; θ)
P (z|y; θ)
= ∑
(x,y)∈S KL(Q(z; θ)∥P (z|y; θ)),
where Q(z; θ) is an arbitrary posterior distribution of z. Note that we utilize the approximation that P (y|x, z; θ) ≈ P (y|z; θ) and P (z|x,y; θ) ≈ P (z|y; θ) due to the semantic equivalence of parallel sentences x and y.
We then introduce the autoencoding task of z by replacing Q(z; θ) with P (z|z; θ) such that
L̄(θ) = ∑
(x,y)∈S E z∼P (z|z;θ) logP (y|z; θ)
− KL(P (z|z; θ)∥P (z|x; θ)) (7)
and L(θ)− L̄(θ) = ∑
(x,y)∈S KL(P (z|z; θ)∥P (z|y; θ)).
(8)
To maximize L(θ), we should maximize the lower bound L̄(θ) and minimize the the gap betweenL(θ) and L̄(θ). By utilizing the cross-lingual consistency regularization, CrossConST helps minimize the KL terms in (7) and (8) and implicitly maximizes the probability distributions for zero-shot translation, which results in better translation performance in x→ y direction. The detailed proof can be found in Appendix A.

heading: 2.4 Training Strategy: Multilingual NMT Pretraining and CrossConST Finetuning
text: Inspired by Johnson et al. (2017) and Wu et al. (2021), we only use one language tag to indicate the target language the multilingual NMT model should translate to. For instance, the following English to German sentence pair “How are you? → Wie geht es dir?” is transformed to “<de> How are you? →Wie geht es dir?”. And Wu et al. (2021) demonstrate that such language tag strategy could enhance the consistency of semantic representations and alleviate the off-target issue in zero-shot translation directions.
To stabilize the multilingual NMT training procedure and accelerate the convergence of the multilingual NMT model, we adopt a two-stage training strategy. We first train a conventional multilingual NMT model as the pretrained model and then finetune the model with CrossConST objective function (4). It is worth mentioning that Pham et al. (2019) derive a similar problem formulation and training strategy. However, they do not demonstrate the effectiveness of their proposed method (KL Softmax) in Pham et al. (2019). To the best of our knowledge, we for the first time show the effectiveness of the simple cross-lingual consistency regularization for improving the translation performance of the multilingual NMT model. Note that while Pham et al. (2019) decouple the gradient path in the decoder from the KL divergence term, our design allows for backpropagation through both sides of the KL regularization in CrossConST. We do not decouple any gradient path in our model.

heading: 3 Low Resource Scenario
text: We here investigate the performance of CrossConST on the low-resource multilingual machine
translation benchmark. For fair comparisons, we keep our experimental settings consistent with the previous work (Gu and Feng, 2022).

heading: 3.1 Dataset Description
text: We conduct our experiments on the IWSLT17 benchmark (Cettolo et al., 2017), which releases a multilingual corpus in five languages: English (en), German (de), Dutch (nl), Romanian (ro), and Italian (it). We consider the English-centric scenario, where we collect the parallel sentences from/to English. The detailed information of the training dataset is summarized in Table 5 in Appendix B. There are eight supervised translation directions and twelve zero-shot translation directions, and we use the official validation and test sets in our experiments. Following the common practice, we tokenize each language by applying the Moses toolkit (Koehn et al., 2007) and build a shared dictionary with 32K byte-pair-encoding (BPE) (Sennrich et al., 2016) types.

heading: 3.2 Model Configuration
text: We implement our approach on top of the Transformer (Vaswani et al., 2017). We apply a standard base Transformer with 6 encoder and decoder layers, 8 attention heads, embedding size 512, and FFN layer dimension 2048. We apply cross-entropy loss with label smoothing rate 0.1 and set max tokens per batch to be 4096. We use the Adam optimizer with Beta (0.9, 0.98), 4000 warmup updates, and inverse square root learning rate scheduler with initial learning rates 7e−4. We use dropout rate 0.3 and beam search decoding with beam size 5 and length penalty 0.6. We apply the same training configurations in both pretraining and finetuning stages. We fix α to be 0.25 in (4) for CrossConST. We use case-sensitive sacreBLEU (Post, 2018) to evaluate the translation quality. We train all models until convergence on eight NVIDIA Tesla V100 GPUs. All reported BLEU scores are from a single model. For all the experiments below, we select the saved model state with the best validation performance.

heading: 3.3 Main Results
text: We compare our approach with the following methods on the IWSLT17 benchmark:
• m-Transformer (Johnson et al., 2017): A multilingual NMT model that directly learns the
many-to-many translation on the English-centric dataset.
• Pivot Translation (Cheng et al., 2017): mTransformer first translates the source language into English before generating the target language.
• Sentence Representation Alignment (SR Alignment) (Arivazhagan et al., 2019): An additional regularization loss is utilized to minimize the discrepancy of the source and target sentence representations.
• Softmax Forcing (KL-Softmax) (Pham et al., 2019): This method forces the decoder to generate the target sentence from itself by introducing a KL divergence loss.
• Contrastive Learning (mRASP2 w/o AA) (Pan et al., 2021): This method introduces a contrastive loss to minimize the representation gap between the similar sentences and maximize that between the irrelevant sentences. Note that the aligned augmentation (AA) method is not utilized.
• Disentangling Positional Information (DisPos) (Liu et al., 2021): This method drops the residual connections in a middle layer of the encoder to achieve the language-agnostic representations.
• Denosing Training (DAE Training) (Wang et al., 2021): This approach introduces a denoising autoencoding task during the multilingual NMT model training.
• Target Gradient Projection (TGP) (Yang et al., 2021b): This method guides the training with constructed oracle data, where the gradient is projected not to conflict with the oracle gradient.
• Language Model Pretraining (LM Pretraining) (Gu et al., 2019): This approach strengthens the decoder language model (LM) prior to NMT model training.
• Optimal Transport & Agreement-based Training (OT & AT) (Gu and Feng, 2022): This method proposes an optimal transport loss to bridge the gap between the semantic-equivalent representations and an agreement-based loss to force the decoder to make consistent predictions based on semantic-equivalent sentences. We set γ1 and γ2 in OT & AT to be 0.4 and 0.001 respectively in the experiments.
We report test BLEU scores of all comparison methods and our approach on the IWSLT17 dataset in Table 1. We can see that our multilingual NMT model achieves strong or SOTA BLEU scores in both supervised and zero-shot translation directions. Note that our approach outperforms OT & AT even though its implementation is much more complicated than ours. It is worth mentioning that CrossConST is the only method that can achieve a similar zero-shot translation performance compared with the pivot translation. Note that the BLEU scores of our m-Transformer, especially in the zero-shot translation directions, are higher than that reported in Gu and Feng (2022). Such gap
might be due to the different language tag strategies used in Gu and Feng (2022) and our experiments, which is in line with Wu et al. (2021).

heading: 3.4 Does CrossConST Still Work Beyond English-centric Scenario?
text: We here extend our experiments on the IWSLT17 benchmark beyond the English-centric scenario. Specifically, we gather the English-centric dataset used in Section 3.3 and supplement it with an additional 20K de ↔ it sentence pairs, which are subsampled from the IWSLT17 dataset. This experimental setup is highly practical because the size of the non-English datasets is usually an order less than that of the English-centric dataset.
translation benchmark. 1 denotes the English-centric dataset. 2 denotes the English-centric dataset + extra de ↔ it dataset. The detailed evaluation results are summarized in Table 9 in Appendix C.
We report test BLEU scores of the baseline and our approach on the IWSLT17 dataset in Table 2. By checking model performance under different combinations of dataset and training strategy, we have the following observations: 1) Adding beyond the English-centric dataset (de↔ it) could greatly improve the overall zero-shot translation performance. 2) The CrossConST is complementary to the data-based method and could further improve the performance of the zero-shot translation.

heading: 4 High Resource Scenario
text: We here investigate the performance of the CrossConST on the high-resource multilingual machine translation benchmark. For fair comparisons, we keep our experimental settings consistent with the previous works (Lin et al., 2020; Pan et al., 2021).

heading: 4.1 Dataset Description
text: We conduct our experiments on PC32, a multilingual parallel corpus of 32 English-centric language pairs. We collect the pre-processed PC32 dataset from Lin et al. (2020)’s release1. We also collect
1https://github.com/linzehui/mRASP
the pre-processed PC32 dataset after applying random aligned substitution (RAS) technique from Lin et al. (2020)’s release. The detailed statistics of all training datasets are summarized in Tables 6 and 7 in Appendix B.
For supervised directions, we collect testsets from WMT benchmarks, where four languages, Spanish (es), Finnish (fi), French (fr), and Turkish (tr), are selected, resulting in 8 translation directions. We use multi-bleu.pl2 for tokenized BLEU (Papineni et al., 2002) evaluation, where both reference and hypothesis are tokenized by Sacremoses3. For zero-shot directions, we collect OPUS-100 zero-shot testsets from Zhang et al. (2020)’s release4, where six languages, Arabic (ar), German (de), French (fr), Dutch (nl), Russian (ru), and Chinese (zh), are selected, resulting in 25 translation directions. Note that Dutch is not covered in our training dataset such that we only evaluate the zero-shot directions when Dutch is at the source side. We evaluate the multilingual NMT models by case-sensitive sacreBLEU (Post, 2018).

heading: 4.2 Model Configuration
text: We apply a Transformer with 12 encoder and decoder layers, 16 attention heads, embedding size 1024, and FFN layer dimension 4096. We use dropout rate 0.1, learning rate 3e−4 with polynomial decay scheduling and 10000 warmup updates. We use Adam optimizer with Beta (0.9, 0.98) and ϵ = 1e−6. We set the threshold of gradient norm to be 5.0. We apply cross-entropy loss with label smoothing rate 0.1 and set max tokens per batch to be 1536 with update frequency 50. We use beam search decoding with beam size 5 and length penalty 1.0. We apply the same training configurations in both pretraining and finetuning stages. We fix α to be 0.1 in (4) for CrossConST. We train all models until convergence on 8 × 4 NVIDIA Tesla V100 GPUs. All reported BLEU scores are from a single model. We select the saved model state with the best validation performance for all the experiments below.

heading: 4.3 Main Results
text: We compare our approach with the following methods on the PC32 benchmark:
2https://github.com/moses-smt/mosesdecoder/blob/ master/scripts/generic/multi-bleu.perl
3https://github.com/alvations/sacremoses 4https://opus.nlpl.eu/opus-100.php
except for mRASP2 in each column. 1 denotes PC32. 2 denotes PC32 + RAS. 3 denotes PC32 + RAS + MC24.
• mRASP (Lin et al., 2020): This method proposes a random aligned substitution (RAS) technique that builds code-switched sentence pairs for multilingual pretraining. Note that the results of mRASP reported in this paper are obtained without finetuning.
• mRASP2 (Pan et al., 2021): This method utilizes the RAS technique on both the bilingual dataset (PC32) and an additional monolingual dataset (MC24). It introduces a contrastive loss to minimize the representation gap between the similar sentences and maximize that between the irrelevant sentences. mRASP2 w/o AA only adopts the contrastive loss based on m-Transformer, and mRASP2 w/o MC24 excludes MC24 from mRASP2.
We report test BLEU scores of all comparison methods and our approach on WMT super-
vised translation directions in Table 3. With CrossConST regularization, our multilingual NMT model achieves strong or SOTA BLEU scores on the supervised translation directions. Note that all comparison methods and our approach share the same model architecture, and the only differences are the training dataset and the objective loss function. We report test BLEU scores of all comparison methods and our approach on OPUS-100 zero-shot translation directions in Table 4, which includes six languages and 25 translation directions in total. The detailed evaluation results are summarized in Table 10 in Appendix D. We also report the evaluation results of the pivot translation based on m-Transformer and mRASP. We can see that CrossConST greatly boosts the performance in the zero-shot translation directions and substantially narrows the performance gap with the pivot translation. It is worth mentioning that our approach could
improve zero-shot translation by a large margin and also benefit the supervised translation.
By checking model performance under different scenarios, we have the following observations: 1) Our language tag strategy works better than that in Pan et al. (2021) for learning the multilingual NMT model on the English-centric dataset, especially for the zero-shot translation, which is in line with Wu et al. (2021). 2) CrossConST is crucial for the performance improvement in the zero-shot translation directions and performs slightly better when combined with the code-switched training dataset. 3) Our approach could outperform mRASP2 on average in the absence of the MC24 dataset, which implies the effectiveness of CrossConST compared with the contrastive loss utilized in mRASP2.

heading: 4.4 Does CrossConST Really Learn A Better Latent Representation?
text: We conduct the experiments on the multi-way parallel testset newstest20125 from the WMT13 (Bojar et al., 2013) translation task, where 3003 sentences have translations in six languages: Czech (cs), Germany (de), English (en), Spanish (es), French (fr), and Russian (ru). We calculate the sentence representations by max-pooling the multilingual NMT encoder outputs.
Sentence Representation Visualization To verify whether CrossConst can better align different languages’ semantic space, we visualize the sentence representations of Germany (de), English (en), and French (fr). We apply dimension reduction on the 1024-dimensional sentence representations with T-SNE (Hinton and Roweis, 2002) and then depict the bivariate kernel density estimation based on the 2-dimensional representations in Figure 1. Figure 1 shows that m-Transformer cannot align these three languages well in the representation space, while CrossConST draws the sentence representations across different languages much closer. Please check Figures 4 and 5 in Appendix E for the visualization of the sentence representations in other languages.
Multilingual Similarity Search We conduct the multilingual similarity search experiment to verify that CrossConST indeed closes the latent representation gap among different languages. For each sentence in the source language, we find the closest sentence in the target language according to the
5https://www.statmt.org/wmt13/dev.tgz
cosine similarity of the corresponding sentence representations. The evaluation results are reported in Figure 3. By checking model performance on different language pairs, we have the following observations: 1) m-Transformer could achieve decent performance (94.71% on average) among nonEnglish directions. However, the similarity search accuracy degrades dramatically (81.03% on average) in the English-centric directions, which implies that English does not align well with nonEnglish languages in m-Transformer. We think such bad representation alignment between English and non-English languages is one of the critical reasons that m-Transformer underperforms in the zero-shot translation directions compared with the pivot-based method. 2) CrossConST significantly improves the similarity search performance in the English-centric direction (14.74% improvement on average) and further boosts the performance among non-English directions (1% improvement on aver-
age). We believe the improvement of similarity search accuracy could be regarded as an indicator of better cross-lingual representation alignment and confirm that CrossConST can learn effective universal representation across different languages.

heading: 5 Related Work
text: Early works on multilingual NMT demonstrate its zero-shot translation capability (Ha et al., 2016; Johnson et al., 2017). To further improve the zero-shot translation performance, one direction is to force the multilingual NMT encoder output to be language-agnostic via additional regularization constraints or training tasks (Pham et al., 2019; Arivazhagan et al., 2019; Wei et al., 2020; Liu et al., 2021; Wang et al., 2021; Yang et al., 2021b; Gu and Feng, 2022). For example, Gu and Feng (2022) introduce an agreement-based training approach to help the multilingual NMT model make consistent predictions based on the semantics-equivalent sentences. Our method follows this line but outperforms these methods by introducing a simple yet effective cross-lingual regularization constraint, which effectively reduces discrepancies in representations across languages.
Another direction is to utilize extra data such as generated pseudo sentence pairs, monolingual datasets, and pretrained models (Gu et al., 2019; Al-Shedivat and Parikh, 2019; Zhang et al., 2020; Chen et al., 2021; Yang et al., 2021a). For example, Al-Shedivat and Parikh (2019) encourages the multilingual NMT model to produce equivalent translations of parallel training sentence pairs into an auxiliary language. Zhang et al. (2020) proposes random online back-translation to enforce the translation of unseen training language pairs. Unlike these methods, CrossConST does not require additional data and is orthogonal to these methods. We could further boost the zero-shot translation performance by combining our method with these data-driven approaches.

heading: 6 Conclusion
text: In this paper, we propose CrossConST: a simple but effective cross-lingual consistency regularization method for learning multilingual NMT models. We theoretically analyze the regularization effect of CrossConST and verify its effectiveness for zero-shot translation. For the stable training of multilingual NMT, we propose a two-state training strategy that consists of multilingual NMT pre-
training and CrossConST finetuning. Experiments on low and high resource multilingual translation benchmarks demonstrate CrossConST’s capabilities to improve translation performance in both supervised and zero-shot directions. Further experimental analysis confirms that our method indeed leads to better cross-lingual representation alignment. Given its universality and simplicity, we anticipate that researchers could leverage the simplicity of CrossConST as a foundation to achieve new SOTA results in their own work. For future work, we will explore the effectiveness of CrossConST on more multilingual tasks, such as multilingual sentence embedding, multilingual word alignment, etc.
Limitations
In this paper, we mainly focus on evaluating our approach on two English-centric corpora, IWSLT17 and PC32. Future research could consider more multilingual machine translation benchmarks with different number of languages and training samples and conduct experiments on more challenging training scenarios such as chain configurations where we have multiple bridge languages and different zero-shot distances.

heading: Acknowledgements
text: We would like to thank the anonymous reviewers for their insightful comments.

heading: A Theoretical Discussion of CrossConST
text: We first discuss how to derive the lower bound of L(θ) as follows. Please note that we drop θ
in the following proofs for the simplicity of the expression.
L(θ) = ∑
(x,y)∈S logP (y|x)
= ∑
(x,y)∈S log
∑
z
P (y|x, z)P (z|x)
≈ ∑
(x,y)∈S log
∑
z
P (y|z)P (z|x)
= ∑
(x,y)∈S log
∑
z
P (z|z)P (y|z)P (z|x) P (z|z)
≥ ∑
(x,y)∈S
∑
z
P (z|z) log P (y|z)P (z|x) P (z|z)
= ∑
(x,y)∈S E z∼P (z|z) logP (y|z)
− KL(P (z|z)∥P (z|x)) := L̄(θ)
We then discuss how to derive the gap between L(θ) and L̄(θ) as follows.
L(θ)− L̄(θ)
= ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z)P (y|x) P (y|z)P (z|x)
= ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z)P (y|x)P (z|y) P (y|z)P (z|x)P (z|y)
≈ ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z)P (y|x)P (z|y) P (y|x, z)P (z|x)P (z|y)
= ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z)P (y|x)P (z|y) P (y, z|x)P (z|y)
= ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z)P (y|x)P (z|y) P (z|x,y)P (y|x)P (z|y)
≈ ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z)P (y|x)P (z|y) P (z|y)P (y|x)P (z|y)
= ∑
(x,y)∈S
∑
z
P (z|z) log P (z|z) P (z|y)
= ∑
(x,y)∈S KL(P (z|z)∥P (z|y)),
where we utilize two approximations as follows:
P (y|x, z) ≈ P (y|z) (9)
and P (z|x,y) ≈ P (z|y). (10)

heading: B Statistics of all training datasets

heading: C Details of Evaluation Results on IWSLT17

heading: D Details of Evaluation Results on
text: OPUS-100
E Sentence Representation Visualization
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
7
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? 1
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? 3 and 4
3 B1. Did you cite the creators of artifacts you used? 3 and 4
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
3 B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? 3 and 4
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Left blank.
C 3 Did you run computational experiments? 3 and 4
3 C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? 3 and 4
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? 3 and 4
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? 3 and 4
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? 3 and 4
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: For future work, we will explore the effectiveness of CrossConST on more multilingual tasks, such as multilingual sentence embedding, multilingual word alignment, etc. Limitations
In this paper, we mainly focus on evaluating our approach on two English-centric corpora, IWSLT17 and PC32. Future research could consider more multilingual machine translation benchmarks with different number of languages and training samples and conduct experiments on more challenging training scenarios such as chain configurations where we have multiple bridge languages and different zero-shot distances