Globetrotter: Unsupervised Multilingual Translation from Visual Alignment

1 INTRODUCTION Machine translation aims to learn a mapping between sentences of different languages while also maintaining the underlying semantics . In the last few years , sequenceto-sequence models have emerged as remarkably powerful methods for this task , leading to widespread applications in robust language translation . However , sequenceto-sequence models also require large data sets of parallel corpora for learning , which is expensive to collect and often impractical for rare language pairs . We propose to leverage the synchronization between language and vision in order to learn models for machine translation without parallel training corpora . Instead of learning a direct mapping between languages , we present a model that aligns them by first mapping through a visual representation . We show how vision creates a transitive closure across modalities , which we use to establish positive and negative pairs of sentences without supervision . Since the visual appearance of scenes and objects will remain relatively stable between different spoken languages , vision acts as a “ bridge ” between them . Our approach integrates these transitive relations into multi-modal contrastive learning . In our experiments and visualizations we show that the transitive relations through vision provide excellent self-supervision for learning neural machine translation . Although we train our approach without paired language data , our approach is able to translate between 52 different languages better than several baselines . While vision is necessary for our approach during learning , there is no dependence on vision during inference . After learning the language representation , our approach can translate both individual words and full sentences using retrieval . The contributions of this paper are three-fold . First , we propose a method that leverages crossmodal alignment between language and vision to train a multilingual translation system without any parallel corpora . Second , we show that our method outperforms previous work by a significant margin on both sentence and word translation , where we use retrieval to test translation . Finally , to evaluate and analyze our approach , we release a federated multi-modal dataset spanning 52 different languages . Overall , our work shows that grounding language in vision helps developing language processing tools that are robust across languages , even in cases where ground truth alignment across languages is not available . Code , data , and pre-trained models will be released . 2 RELATED WORK . Our unsupervised joint visual and multilingual model builds on recent progress in both the natural language processing and computer vision communities . We briefly summarize the prior work . Unsupervised language translation has been studied as a word representation alignment problem in Lample et al . ( 2018b ) , where the distribution of word embeddings for two unpaired languages is aligned to minimize a statistical distance between them . Lample et al . ( 2018a ) ; Artetxe et al . ( 2018 ) ; Lample et al . ( 2018c ) ; Lample & Conneau ( 2019 ) build on top of this idea , and train an encoderdecoder structure to enforce cycle-consistency when translating from one language to another and back to the first one . This method achieves strong unsupervised word translation results , but does not scale beyond two languages . It also does not leverage visual information in learning . Multi-language models are general language models that develop language-independent architectures that work equally well for any language ( Gerz et al. , 2018 ) . Lample & Conneau ( 2019 ) ; Conneau et al . ( 2020 ) ; Artetxe & Schwenk ( 2019 ) ; Devlin et al . ( 2019 ) ; Liu et al . ( 2020 ) ; Phang et al . ( 2020 ) share the same token embeddings across different languages , showing that this improves language modeling both for general downstream single-language NLP tasks and also for supervised language translation across multiple languages . Lample & Conneau ( 2019 ) ; Conneau et al . ( 2020 ) ; Artetxe & Schwenk ( 2019 ) use a shared Byte Pair Encoding ( BPE ) , which we use in our work . We loosely follow the architecture of Conneau et al . ( 2020 ) in that we train a transformer-based ( Vaswani et al. , 2017 ) masked language model with BPE . Vision as multi-modal bridge implies using vision as an interlingua between all languages . Using a third language as a pivot to translate between pairs of languages without source-target paired corpora has been studied for the past few years ( e.g . Firat et al. , 2016 ; Johnson et al. , 2017 ; Garcia et al. , 2020 ) . Harwath et al . ( 2018 ) ; Azuh et al . ( 2019 ) use vision for the same purpose , and they work directly on the speech signal instead of text . Chen et al . ( 2018 ) use images to help translate between languages in the text modality . Their model involves both generation and reinforcement learning , which makes optimization difficult , and they do not generalize to more than two languages . Sigurdsson et al . ( 2020 ) also use vision as a pivot for unsupervised translation . However , our approach works for multiple languages at once ( instead of just two ) and also obtains an explicit cross-lingual alignment . We share a single word embedding and language model for all languages , and use different training strategies . Our experiments quantitatively compare the two approaches , showing that our approach performs better both in word and sentence translation . Other work views the input image as extra information for translation ( e.g . Calixto & Liu , 2017 ; Su et al. , 2019 ) , and we refer readers to Specia et al . ( 2016 ) for an extensive overview on this topic . Instead of using images as a bridge , paired data between languages is used . There has also been research on training multilingual language representations for downstream vision tasks , in general leveraging visual-language correspondence , but without translation as a goal . Unlike this paper , they make use of ground truth language pairs ( Wehrmann et al. , 2019 ; Gella et al. , 2017 ; Kim et al. , 2020 ; Burns et al. , 2020 ) . Translation by retrieval . We evaluate the representations using retrieval-based machine translation ( Baldwin & Tanaka , 2000 ; Liu et al. , 2012 ) , which is often used in the context of example-based machine translation ( e.g . Brown , 1996 ; 2001 ; 1997 ; Cranias et al. , 1994 ; El-Shishtawy & El-Sammak , 2014 ) , analogy-based translation ( e.g . Nagao , 1984 ; Kimura et al. , 2014 ) , or translation memories ( e.g . Chatzitheodorou , 2015 ; Dong et al. , 2014 ; Wäschle & Riezler , 2015 ; Baldwin , 2001 ) . While there are also generative-based translation approaches , they are difficult to automatically evaluate . There is generally no well-defined metric for what consists of a good generative translation ( Callison-Burch et al. , 2006 ) . Instead , we evaluate our approach using translation-by-retrieval , allowing for rigorous experimental validation of the cross-lingual alignment in the representation . State-of-the-art cross-lingual retrieval approaches rely on supervised language pairs , and range from training the models in a standard contrastive learning setting ( Chi et al. , 2020 ) to more complex combinations of the language pairs such as using cross-attention ( Anonymous , 2021 ) or introducing custom fusion layers ( Fang et al. , 2020 ) . Our approach does not require supervised language pairs . Text Network . 3 METHOD . We present an approach that learns to map words and sentences from one language to semantically similar words and sentences from different languages , for a large number of languages simultaneously . Our approach does not require any paired data between languages , and instead only depends on image-language pairs . Fig . 2 provides an overview of our framework . 3.1 SENTENCE EMBEDDING . Our approach learns an aligned embedding space for sentences across languages . Let zli ∈ RD be the learned embedding of sentence i , obtained by processing the text through a language network Θl . Moreover , let βij be the similarity between sentences zli and z l j , for example through the cosine similarity . Our goal is to learn the parameters of the embedding z such that sentences with the same meaning are mapped to similar positions in the embedding space despite coming from different languages . After learning , we will have a sentence embedding zli that we can use for a variety of tasks , such as retrieving or generating sentences in different languages . We learn the parameters of the embedding space z by optimizing the contrastive learning problem : Lt = − ∑ i ∑ j 6=i αij log exp ( βij/τ ) ∑ k 6=i exp ( βik/τ ) with βij = sim ( zli , z l j ) ( 1 ) In contrastive learning , we need to define which pairs of examples should be close in the learned embedding space ( the positives ) , and which pairs of examples should not ( the negatives ) . In the above formulation , the scalar αij ∈ [ 0 , 1 ] indicates this assignment . However , since we are in an unsupervised translation setting , we do not have ground truth pairs . Our main idea , which we introduce in the next section , is that we can use the visual modality to discover these pairs . 3.2 TRANSITIVE RELATIONS . Estimating the similarity for sentences of different languages is challenging without labels . Unsupervised machine translation approaches typically rely on topological properties , such as distributional alignment or back-translation ( Lample et al. , 2018b ; Lample & Conneau , 2019 ) . However , these constraints provide a noisy gradient for learning , which makes large-scale optimization difficult . We propose to take advantage of a transitive relation through the visual modality in order to estimate the similarity in language space αij . Given a dataset of images and their corresponding captions , we estimate both a cross-modal ( sentence-image ) similarity as well as a cross-image ( image-image ) similarity . Let αxii be the cross-modal similarity , which indicates the alignment between image i and its corresponding caption i . We also let αvij be the cross-image similarity , indicating the perceptual similarity between image i and another image j . This provides the transitive relation as the product of similarities : αij = f ( α x ii · αvij · αxjj ) , where f ( x ) = max ( 0 , x−m ) / ( 1−m ) , ( 2 ) and m is a margin that we set to m = 0.4 , which prevents pairs with low similarity from being used as positives . Note that αij = αji . The transitive similarity causes two sentences from different languages to be similar if they appear in similar visual contexts . Since both αxii ∈ [ 0 , 1 ] and αvij ∈ [ 0 , 1 ] , the final similarity is in the same range , αij ∈ [ 0 , 1 ] . Only when there is a strong alignment between an image and its caption , and there is also another image with close perceptual similarity , will a transitive relation be formed . In realistic scenes , the correspondence for some image and caption pairs may be difficult to establish in the presence of noise , which our formulation handles by breaking the transitive relation . In other words , we only consider paths with high total similarity as positives for the contrastive objective , and discard those paths with low total similarity , since their sentences likely do not match .

Summary: The authors propose to leverage images to train an unsupervised machine translation (MT) model. Their main idea is that the similarity of images can be used as a proxy for the similarity of sentences describing the images. The sentences, in turn, can be in different languages, and knowledge about their similarity can be exploited as training signal for an unsupervised MT model, i.e., training without parallel sentences. Their model consists of a sentence encoder and an image encoder. For training and evaluation of the model, they use translations (multi-way for the test set) of image captioning datasets.