Contrastive Clustering to Mine Pseudo Parallel Data for Unsupervised Translation

Modern unsupervised machine translation systems mostly train their models by generating synthetic parallel training data from large unlabeled monolingual corpora of different languages through various means , such as iterative backtranslation . However , there may exist small amount of actual parallel data hidden in the sea of unlabeled data , which has not been exploited . We develop a new fine-tuning objective , called Language-Agnostic Constraint for SwAV loss , or LAgSwAV , which enables a pre-trained model to extract such pseudo-parallel data from the monolingual corpora in a fully unsupervised manner . We then propose an effective strategy to utilize the obtained synthetic data to augment unsupervised machine translation . Our method achieves the state of the art in the WMT ’ 14 English-French , WMT ’ 16 German-English and English-Romanian bilingual unsupervised translation tasks , with 40.2 , 36.8 , and 37.0 BLEU , respectively . We also achieve substantial improvements in the FLoRes low-resource English-Nepali and English-Sinhala unsupervised tasks with 5.3 and 5.4 BLEU , respectively . 1 INTRODUCTION . The quest to build a fully unsupervised machine translation ( UMT ) system , where only unlabeled monolingual corpora are available , has received increasing attention in recent years . Profoundly , Lample et al . ( 2018a ; c ) introduced the general principles for UMT that include cross-lingual initialization , language modeling and iterative back-translation . Although various UMT variants ( Lample et al. , 2018a ; c ; Conneau & Lample , 2019 ; Song et al. , 2019 ; Liu et al. , 2020 ; Nguyen et al. , 2021 ) applied these principles differently , they ultimately train their models on noisy synthetic parallel data generated by the models themselves or some randomization processes , which may cause harm as the generated synthetic data is often noisy and low-quality , especially at the beginning of training . However , these methods may have missed out a potential that some parallel ( or comparable ) data may exist in the monolingual corpora , which can be effectively mined to augment the UMT training . This paper is motivated to explore mining of pseudo-parallel data for UMT tasks . While there have been limited research in unsupervised mining of such data ( Wu et al. , 2019a ; b ) , there have been several studies on bitext mining from sentence embeddings in semi-supervised translation or zeroshot transfer setups ( Zweigenbaum et al. , 2018 ; Schwenk , 2018 ; Artetxe & Schwenk , 2019a ; b ) . However , these methods require the models to be pre-trained on massive multilingual parallel data , which renders them inapplicable and incomparable to the fully unsupervised setup . Furthermore , such models may not behave well off-the-shell on low-resource languages that were not in the pretraining data , such as Nepali and Sinhala ( Guzmán et al. , 2019 ) . Meanwhile , applying these mining algorithms directly to sentence embeddings produced by fully self-supervised models ( Conneau & Lample , 2019 ) leads to a significant amount of misalignments , which hurts the UMT models as shown later in our experiments ( §4 ) . ∗Most of work done during an internship at Meta AI . In this paper , we propose a novel modification to SwAV loss ( Caron et al. , 2020 ) , called LanguageAgnostic Constraint for SwAV ( LAgSwAV ) 1 , which is used to fine-tune existing pre-trained selfsupervised models ( Conneau & Lample , 2019 ; Liu et al. , 2020 ) , so that they become semantically contrastive and language-agnostic . Specifically , semantic contrastiveness is defined , in this paper , as the model ’ s ability to cluster sentences of similar semantic contents into similar groups . Meanwhile , language-agnostic property means that the embedding space of sentences from different languages are indistinguishably mixed together . Figure 1 further illustrates the two properties . We argue that these two properties combined can help the model mine more accurate pseudo-parallel data , since language-agnosticism reduces the distances between cross-lingual embeddings while semantic contrastiveness helps bring sentences of similar contents closer together even if they are in different languages . To achieve the semantically contrastive property , we adopt the SwAV loss ( Caron et al. , 2020 ) , which is designed to classify input data into different clusters that reflect certain semantic characteristics . However , since SwAV is not designed to be language-agnostic , we introduce a new language-agnostic constraint for it to enable the latter property . Analysis experiments in §4 show that our constraint significantly outperforms vanilla SwAV loss as well as other baselines in mining pseudo-parallel data from monolingual corpora . After fine-tuning pre-trained self-supervised models with LAgSwAV loss , we sample sentences from both languages and pass them to the model to obtain the SwAV cluster assignment probabilities , which are treated as sentence embeddings . The soft cluster assignments are then matched by a margin-based algorithm ( Artetxe & Schwenk , 2019b ) with additional filtering criteria ( §3.2 ) . To augment UMT training with the obtained pseudo-parallel data , we propose a simple ranking-based weighted cross-entropy loss to train on this data along with the standard iterative back-translation ( §3.3 ) . We also adopt a dynamic weight to the augmentation loss to alleviate the overfitting effect due to the limited augmentation data . We tested our method on the standard bilingual WMT ’ 14 English-French , WMT ’ 16 GermanEnglish and English-Romanian unsupervised translation tasks and achieved the state of the art with 40.2 , 36.8 , and 37.0 BLEU , respectively ; which is up to 2 BLEU gain from the baselines . In the FLoRes low-resource unsupervised setups ( Guzmán et al. , 2019 ) , our method also outperforms mBART baseline ( Liu et al. , 2020 ) by up to 2.8 BLEU in the English-Nepali and English-Sinhala tasks . We conducted a series of analyses to demonstrate the outperforming ability of LAgSwAV in mining pseudo-parallel data . The method is also found to outperform supervised LASER ( Artetxe & Schwenk , 2019a ) , which can be attributed to LASER ’ s lack of language specifiers . 2 BACKGROUND . 2.1 UNSUPERVISED MACHINE TRANSLATION . Lample et al . ( 2018a ) and Artetxe et al . ( 2018 ) were among the first to employ iterative backtranslation in fully unsupervised machine translation ( UMT ) . Lample et al . ( 2018c ) later formulated the three-principles to UMT : initialization , language modeling and iterative back-translation ( Sennrich et al. , 2016a ) . Initialization can takes form of MUSE bilingual dictionary ( Lample et al. , 1Code : https : //github.com/nxphi47/fairseq/tree/swav umt 2018b ; a ) , cross-lingual masked language model ( XLM ) ( Conneau & Lample , 2019 ) or Seq2Seq model ( Song et al. , 2019 ) . Language modeling often takes form of denoising autoencoding ( Lample et al. , 2018a ) or is omitted entirely ( Song et al. , 2019 ) . Apart from these principles , Nguyen et al . ( 2021 ) use two distinct UMT teachers to distill the student in a two-stage back-translation process . Despite being different , most existing methods commonly employ the same iterative backtranslation process . Multilinguality is also proved to be useful as Liu et al . ( 2020 ) and Garcia et al . ( 2021 ) utilize extra training data of more than two languages . These methods , however , do not leverage the potential pseudo-parallel data that may exist in the monolingual data to improve UMT . 2.2 PSEUDO-PARALLEL DATA MINING . Parallel data mining for machine translation has been an active field of research ( Zweigenbaum et al. , 2018 ; Schwenk , 2018 ; Chinea-Rı́os et al. , 2017 ) . Artetxe & Schwenk ( 2019b ) suggested an effective margin-based algorithm to mine parallel data using sentence embeddings , which we also use in our method . More recently , LASER ( Artetxe & Schwenk , 2019a ) was trained on massive multilingual parallel data from 93 languages . Nonetheless , most of the effort has been invested largely in a supervised setup where the training data is parallel . In terms of UMT , Wu et al . ( 2019a ) proposed an extract-edit approach to mine relevant data and edit it for use , while Wu et al . ( 2019b ) mine weakly-paired documents . Tran et al . ( 2020 ) suggested to train on massive multilingual data from 25 languages and mine data from 180 translation directions , which may not be applicable in bilingual setup , where monolingual data only for the two languages under consideration are available . 2.3 SWAPPING ASSIGNMENTS BETWEEN VIEWS ( SWAV LOSS ) . Caron et al . ( 2020 ) introduced Swapping Assignments between Views ( SwAV ) loss as a means to conduct self-supervised training in computer vision because masking-based training methods applied in language modeling are intractable in images . Although SwAV is a type of contrastive learning , it differs from its sisters ( Chen et al. , 2020 ; He et al. , 2020 ; Wu et al. , 2018 ) in that it is an online clustering-based method . This means the model is trained to assign cluster labels , or “ code ” vector q , to an input image with encoded latent features z . In particular , SwAV loss seeks to enforce consistency between codes of different augmentations ( or views ) of the same image . In other words , different augmentations of the same image should have almost the same semantic content as the original , and thus the cluster assignments of them should also be consistently the same . Formally , let z1 , z2 ∈ Rd be latent features of two distinct views X1 , X2 of the same image X after passing them through an encoder fθ , we compute the corresponding codes q1 , q2 ∈ [ 0 , 1 ] K by passing z1 , z2 into the prototype layer , which consists of prototype weights C = [ c1 , ... , cK ] with ci ∈ Rd . We then proceed to swap the codes q1 , q2 and use them as labels for z2 , z1 , respectively : LSwAV ( z1 , z2 ) = l ( z1 , q2 ) + l ( z2 , q1 ) ( 1 ) The loss function l ( z , q ) in Equation ( 1 ) measures the similarity distance between features z and code q , and is defined as : l ( z , q ) = − ∑ k qk log pk , with pk = exp ( 1τ z T ck ) ∑ k′ exp ( 1 τ z T ck′ ) ( 2 ) where τ is a temperature hyper-parameter and k is the index for ck row in C. Figure 2a illustrates how SwAV loss works . While the above formulations involve two different augmentations X1 , X2 of the same image for brevity purpose , multiple augmentations are used in practice . We refer the reader to ( Caron et al. , 2020 ) for further details on SwAV loss . 3 METHOD . In this section , we present our three main contributions : the language-agnostic constraint for SwAV loss or LAgSwAV ( §3.1 ) , the pseudo-parallel data mining with filter suite ( §3.2 ) , and finally the rank-based cross-entropy loss for UMT training with augmented data ( §3.3 ) . 3.1 LANGUAGE-AGNOSTIC CONSTRAINT FOR SWAV LOSS IN LANGUAGE MODEL . The SwAV loss was developed to conduct self-supervised training in vision due to infeasibility of applying masked language modeling ( MLM ) to images ( Caron et al. , 2020 ) . Thus , for language model pre-training , we stick to MLM and use our proposed LAgSwAV to fine-tune a pre-trained LM to achieve semantic contrastiveness and language agnosticism . Specifically , in the presence of sentences from two languages , let Bs = [ X1 , ... , XB ] and Bt = [ Y1 , ... , YB ] be the batches of B sentences from language s and t , respectively . We proceed to augment them into Bs,1 = [ X11 , ... , X1B ] and Bs,2 = [ X21 , ... , X2B ] , and Bt,1 = [ Y 11 , ... , Y 1B ] and Bt,2 = [ Y 21 , ... , Y 2B ] by various noising techniques , such as token swapping , masking and deleting . Next , we denote Fθ as the Transformer encoder ( Conneau & Lample , 2019 ) , which outputs the sentence embedding z given an input sentence . We compute the respective latent features Zs,1 , Zs,2 , Zt,1 , Zt,2 ∈ RB×d for Bs,1 , Bs,2 , Bt,1 , Bt,2 using Fθ , as described in §2.3 . Subsequently , we compute the code cluster assignments Qs,1 , Qs,2 , Qt,1 , Qt,2 ∈ [ 0 , 1 ] B×K as : [ Qs,1 ; Qs,2 ; Qt,1 ; Qt,2 ] = sinkhorn ( exp ( [ Zs,1 ; Zs,2 ; Zt,1 ; Zt,2 ] CT ) ) ( 3 ) where [ • ; • ] is the row-wise concatenation function and sinkhorn ( • ) is the iterative Sinkhorn-Knopp algorithm ( Cuturi , 2013 ) , a batch-wise renormalization process to compute the cluster assignment probabilities , and is a hyper-parameter to control the smoothness of the distribution . The above formulation is in fact indifferent from the vanilla SwAV loss ( §2.3 ) . However , intuitively , if the model Fθ is not language-agnostic ( i.e. , s and t are distributionally separate ) , the resulting cluster assignments Qs , j and Qt , j are likely to be distributed unevenly for s and t. In other words , some clusters ci ⊂ C will be frequently assigned to the sentences of language s but rarely to the sentences of t , and vice versa . Figure 2b illustrates the imbalance phenomenon . We argue that it is possible to enforce the language-agnostic property indirectly by fixing this imbalance phenomenon . Therefore , the objective of our proposed LAgSwAV is to measure the stochastic imbalance in cluster assignments Qs , j , Qt , j , and rebalance them into V s , j , V t , j ∈ [ 0 , 1 ] B×K such that clusters in V s , j , V t , j are stochastically assigned evenly regardless of the language of the sentences . Then , we compute the total LAgSwAV loss as usual for both s and t as : LsAgSwAV , i = l ( z s,1 i , v s,2 i ) + l ( z s,2 i , v s,1 i ) ; L t AgSwAV , i = l ( z t,1 i , v t,2 i ) + l ( z t,2 i , v t,1 i ) ( 4 ) LLAgSwAV = ∑ i LsAgSwAV , i + ∑ i LtAgSwAV , i ( 5 ) We formalize the transformation from Q to V using the language-agnostic rebalancing function Ω , which operates at the batch level to capture the statistics on cluster assignments . Specifically , given the Q matrices specified above , we compute the per-language rebalance weight vectors ws , wt ∈ RK , such that when multiplying them with prototype outputs , they try to suppress the magnitudes of frequently-assigned clusters and promote those of rarely-assigned clusters , resulting in more balanced cluster assignments . More formally , the weight vectors are computed as : fs = 2B∑ i=1 onehot ( [ Qs,1 ; Qs,2 ] ) i ; f t = 2B∑ i=1 onehot ( [ Qt,1 ; Qt,2 ] ) i ( 6 ) ws = 1.0− ( fs/ K∑ j=1 fsj ) ; w t = 1.0− ( f t/ K∑ j=1 f tj ) ( 7 ) where onehot ( • ) is a row-wise one-hot function which returns a one-hot vector by turning the maximum entry in the row to 1.0 and the rest to 0.0 . We then use the weight vectors ws , wt to compute the rebalanced cluster assignments V s , j , V t , j ∈ RB×K by modifying Equation ( 3 ) as : [ V s,1 ; V s,2 ; V t,1 ; V t,2 ] = sinkhorn ( exp ( 1 [ ws [ Zs,1 ; Zs,2 ] ; wt [ Zt,1 ; Zt,2 ] ] CT ) ) ( 8 ) where a B is an element-wise multiplication operation of vector a applied to each row of B , and the rows of V s , j , V t , j represent the vs , ji , v s , j i vectors in Equation ( 4 ) . Note that there is no gradient flow for the computation of ws and wt , as well as the cluster assignments Q and V .

Summary: This work provides a nice extension to the SwAV proposed by Caron. Specifically, it proposes a variant that can be used to build meaningful cluster assignment for different languages at the same time in an unsupervised manner. The key to their modification to SwAv is the so called language-agnostic rebalancing function, which is a simple modification that encourages the cluster assignments to sentence in different languages balance. Experiments show that their modification improves mining quality (i.e. resulting higher BLEU score) than using SwAV. Despite that, the work shows that using their model is not enough, and we need extra steps to make the mined data usuable (i.e. using filter suite to filter data and rank-based cross entropy to train the model with a better use of the data).