Globetrotter: Unsupervised Multilingual Translation from Visual Alignment
Authors: Didac Suris Coll-Vinent
Dave Epstein
Carl Vondrick
Venue:  (2021)
Keywords: cross-modal
multilingual
unsupervised translation
visual similarity

Abstract: Machine translation in a multi-language scenario requires large-scale parallel corpora  for  every  language  pair.   Unsupervised  translation  is  challenging  because there is no explicit connection between languages, and the existing methods have to rely on topological properties of the language representations.  We introduce a framework that leverages visual similarity to align multiple languages, using images as the bridge between them. We estimate the cross-modal alignment between language and images, and use this estimate to guide the learning of cross-lingual representations.   Our  language  representations  are  trained  jointly  in  one  model with a single stage.  Experiments with fifty-two languages show that our method outperforms prior work on unsupervised word-level and sentence-level translation using retrieval.

Under review as a conference paper at ICLR 2021
GLOBETROTTER: UNSUPERVISED MULTILINGUAL TRANSLATION FROM VISUAL ALIGNMENT
Anonymous authors Paper under double-blind review
ABSTRACT
Machine translation in multi-languuge scenario requires large-scale parallel cor- pora for every language pair. Unsupervised translation is challenging because there is no explicit connection between languages, and the existing methods have to rely on topological properties of the language representations. We introduce a framework that leverages visual similarity align multiple languages, using im- ages as the bridge between them. We estimate the cross-mode modal alignment between language and images, and use this estimate to guide the learning of cross-lingual representations. Our language representations are trained jointly in one model with a single stage. Experiments with fifty-two languages show that our method outperforms prior work unsupervised word-level and sentence-level translation using retrieval.
INTRODUCTION
Machine translation aims learn mapping between tences of different languages while also maintaining the underlying semantics. In the last few years, sequence- lo-sequence models have emerged as remarkably power- ful methods for this task, leading to widespread applica- tions in robust language translation. However, sequence- to sequence models also require large data sets of parallel corpora for learning, which is expensive to collect and of- ten impractical for rare language pairs.
E
vélo
bicycle
Figure 1: While each language repre- sents a bicycle with different word, the underlying visual representations re- mains consistent. A bicycle has simi- lar appearance in the UK, France, Japan and India. We leverage this natural property to learn models machine translation across multiple languages without paired training corpora.
We propose to leverage the synchronization between lan- guage and vision order to Iearn models for machine translation without parallel training corpora. Instead of learning direct mapping between languages, we present model that aligns them by first mapping through vi- sual representation. We show how vision creates a transi- establish
tive closure across modalities, which positive and negative pairs of sentences without supervi- sion. Since the visual appearance of scenes and objects
will remain relatively stable between different spoken languages, vision acts as a "bridge" between them. Our approach integrates these transitive relations into multi-modal contrastive learning.
In our experiments and visualizations we show that the transitive relations through vision provide excellent sup--vuperviioon for learning neural machine translation. Although we train our approach without paired language data, our approach is able to translate between 52 different languages better than several baselines. While vision is necessary for our approach during learning, there is no dependence vision during inference. After learning the language representation, approach can translate both individual words and full sentences using retrieval.
The contributions of this paper are three-fold. First, propose method that leverages cross- modal alignment between language and vision to train a multilingual translation system without any parallel corpora. Second, we show that method outperforms previous work by significant margin on both sentence and word translation, where we use retrieval to test translation. Finally, to evaluate and analyze approach, release federated multi-modal dataset spanning 52 different

---

Under review as a conference paper at ICLR 2021
languages. Overall, our work shows that grounding language in vision helps developing language processing tools that are robust across languages, even in cases where ground truth alignment across languages is not available. Code, data, and pre-trained models will be released.
RELATED WORK
Our unsupervised joint visual and multilingual model builds recent progress in both the natural language processing and computer vision communities. We briefly summarize the prior work.
Unsupervised language translation has been studied as word representation alignment problem in Lample et al. (2018b), where the distribution of word embeddings for two unpaired languages is aligned to minimize statistical distance between them. Lample al. (201 8a); Artetxe et al. (2018); Lample et al. (2018c); Lample & Conneau (2019) build on top of this idea, and train an encoder- decoder structure to enforce cycle-consistency when translating from one language to another and back to the first one. This method achieves strong unsupervised word translation results, but does not scale beyond two languages. It also does not leverage visual information in learning.
Multi-languaage models are general language models that develop Language-independenn architec- tures that work equally well for any language (Gerz et al., 2018). Lample & Conneau (2019); Con- neau et al. (2020); Artetxe & Schwenk (2019): Devlin et al. (2019); Liu et al. (2020); Phang et al.
(2020) share the same token embeddings across different languages, showing that this improves lan-
guage modeling both for general downstream single-aanguage NLP tasks and also for supervised language translation across multiple languages. [ample & Conneaul (2019); Conneau et al. (2020);
Artetxe & Schwenk (2019) use a shared Byte Pair Encoding (BPE), which we use in our work.
We loosely follow the architecture of Conneau et al. (2020) in that we train a transformer-
Vaswani al. 2017) masked language model with BPE.
Vision as multi-modal bridge implies using vision as an interlingua between all languages. Using third language
pivot translate between pairs of languages without source-target paired cor- pora has been studied for the past few years (e.g. Firat et al. [0166] Johnson et al. 2017; Garriia et al.) 2020 Harwath et al.] (2018); Azuh et al. (2019) use vision for the same purpose, and they work directly on the speech signal instead of text. Chen et al. (2018) use images to help translate between languages in the text modality. Their model involves both generation and reinforcement learning, which makes optimization difficult, and they do not generalize to more than two languages. [Sig.) urdsson al. (2020) also use vision as a pivot for unsupervised translation. However, our approach works for multiple languages at once (instead of just two) and also obtains an explicit cross-lingual alignment. We share a single word embedding and language model for all languages, and use dif- ferent training strategies. Our experiments quantitatively compare the two approaches, showing that our approach performs better both in word and sentence translation.
Other work views the input image extra information for translation (e.g. Calixto & Liu 2017 Su jet al. 2019). and we refer readers to Specia et al. (2016) for an extensive overview on this topic. Instead of using images as bridge, paired data between languages is used. There has also been research on training multilingual language representations for downstream vision tasks, in general leveraging visual- risual-language correspondence, but without translation as a goal. Unlike this paper, they make use of ground truth language pairs (Wehrmann et al. 20199: Gella et al. 2017 Kim et al. 2020
Burns et al. 2020).
Translation by retrieval. We evaluate the representations using retrieval-based machine translation
(Baldwin & Tanaka 2000 Liu al. 2012), which often used in the context of example-based ma-
chine translation (e.g. Brown 1996:
2001 1997 Cranias et al., 1994 El El-Shishtawy & El-Sammak,
2014), analogy-basse
translation (e.g. Nagao 1984 Kimura et al. (0144), or translation memories
(e.g. Chatzitheeodoouu 2015 Dong et al. 2014 Wäschle & Riezler 2015 Baldwin 2001). While there are also generative-based translation approaches, they are diffcult to automatically evalu- atc. There is generally well-defined metric for what consists of a good generative translation (Callison- Callison-Burchh et al., 2006). Instead, we evaluate our approach using translation-by by-rriivab-rttrrvaall al- lowing for rigorous experimental validation of the cross-lingual alignment in the representation.
State-of Sfate-of-the-ar the-art cross- loogs-lingual retrieval approaches rely on supervised language pairs, and range from training the models in standard contrastive learning setting (Chi et al., 2020) to more complex combinations of the language pairs such as using cross-attention (Anonymous) (021) or introducing
custom fusion layers (Fang al. 2020 Our approach does require supervised language pairs.
2

---

Under review as a conference paper at ICLR 2021
Alvi pril den car Aoat vundans calto antrecriad beennnngg tarte colonaoo
Big
Text Network +
ags
Image Network θ
Figure 2: Our model learns aligned embedding space for language translation by leveraging transitive relation through vision. Cross-sentence similarity Big is estimated by the path through an image collection. See Section for details.
METHOD
We present an approach that learns map words and sentences from one language to semantically similar words and sentences from different languages, for a large number of languages simultane- ously. Our approach does not require any paired data between languages, and instead only depends on lmage-language pairs. Fig, provides an overview of our framework.
3.1 SENTENCE EMBEDDING
Our approach learns an aligned embedding space for sentences across languages. Let 2f I be the learned embedding of sentence i, obtained by processing the text through language network θ, Moreover, let 3.j be the similarity between sentences z and z/, for example through the cosine similarity. Our goal is to learn the parameters of the embedding such that sentences with the same meaning are mapped to similar positions in the embedding space despite coming from different languages. After learning, we will have sentence embedding z! that we can for variety of tasks, such as retrieving or generating sentences in different languages.
We learn the parameters of the embedding space by optimizing the contrastive learning problem:
4= == - any log exp(Big/T)
with Big sim (2), 2))
(1)
En exp (Btx/)
174
In contrastive learning, we need to define which pairs of examples should be close the learned embedding space (the positives), and which pairs of examples should not (the negatives). In the above formulation, the scalar 0sj E [0.1] indicates this assignment. However, since we are in an unsupervised translation setting, we do not have ground truth pairs. Our main idea, which introduce in the next section, that we can use the visual modality to discover these pairs.
3.2 TRANSITIVE RELATIONS
Estimating the similarity for sentences of different languages challenging without labels. Unsuper- vised machine translation approaches typically rely on topological properties, such as distributional alignment or mack-translation (Lample et al.) 20188] Lample Conneau 2019). However, these constraints provide noisy gradient for learning, which makes large-scale optimization difficult.
We propose to take advantage of a transitive relation through the visual modality in order to estimate the similarity in language space Ouj. Given dataset of images and their corresponding captions, we estimate both a cross-modal similarity. Let ag be the cross-modal similarity, which indicates the alignment between image and its corresponding caption i. We also let a be the cross cross-image similarity. indicating the perceptual similarity between image: and another image This provides the transitive relation the product
centencce-magee similarity as well as a cross- cross-image timage-image
3

---

Under review as a conference paper at ICLR 2021
of similarities:
(2) and is a margin that we set to 0.4, which prevents pairs with low similarity from being used as positives. Note that aij = azs. The transitive similarity causes two sentences from different languages to be similar if they appear in similar visual contexts.
aij aij f(a oig jzgh where f(x)
(1))),
Since both a E [0, 1] and aj E [0, 1], the final similarity is in the same range, an E [0, 1]. Only when there is a strong alignment between image and its caption, and there is also another image with close perceptual similarity, will a transitive relation be formed. In realistic scenes, the correspondence for some image and caption pairs may be difficult to establish in the presence of noise, which our formulation handles by breaking the transitive relation. In other words, we only consider paths with high total similarity as positives for the contrastive objective, and discard those paths with low total similarity, since their sentences likely do not match.
3.3 LEARNING
In order optimize Equation I. we need to estimate at and aij. We parameterize both with neural network, and we train them to directly estimate the similarity also with contrastive learning.
Visual Similarity: We jointly learn a visual feature space using contrastive learning (Chen et al.) 2020 in order to estimate o. For every image, we perform two random augmentations, resulting in Two different versions of the same image. These two transformed images are run through the image network, along with the other N pairs (in a batch of N samples). This results in 2.V feature maps, For every pair (i,j) images with representations 2 and zj. we compute a contrastive loss, where all the other 2(N - 1) images are the negatives. We use the loss function:
exp (a
C. -
Σ log Exer exp (o)+/T) where aij = sim(""
= represents the learned features for image i, obtained by processing the images through an image network .. We augment images using random image cropping, random Gaussian blurring, and random color distortions, following Chen et al. (2020).
Cross-Modal Similarity: We also need estimate the similarity between images and their corre- sponding captions o The visual representation anchors inter-language alignment, and this sim- ilarity constrains the sentence embedding for each language to share the same space the image embedding. We learn this similarity metric through the contrastive objective:
Lxx -==-- Σ log E, exp (T)
exp (af/TT) + log E,exp (af)/T))
exp (exf//
with of =im sim(= 2/)
(4)
We finally also train the model with a token cloze task in order to make the language representation contextual. We follow the same loss and objective as BERT (Devlin et al. (019 over the sentence input. We label this loss L.
Token Cloze:
Full Objective: The final objective we optimize is the combination of all four losses defined above:
min L. + X, + X2Za + Agha
(5)
where Q are the neural network parameters, and À are scalar hyper parameters to the balance the terms. Over the course of optimization, the model will be estimating an aligned muuti-lingual repre- sentation 3 jointly with the transitive similarity a. As learning progresses, oij will form soft positive and negative pairs, which the model will use to learn the aligned multi-language representation. The quality of the multi-language representation will depend on the quality of transitive alignments 04g our model discovers. However, since the contrastive objective relies on statistical patterns over a large dataset, our approach is fairly robust to noise, which our experiments support.
3.4 REFINING WORD-LEVEL ALIGNMENT
Our approach learns a common embedding space between vision and sentences in multiple lan- guages, which experiments will show provides robust representation for unsupervised ma-
4

---

Under review as a conference paper at ICLR 2021
chine translation. This representation is aligned well at the sentence level. We can further refine the representation by aligning them along words as well.
To obtain word-level alignment, word embeddings. We find a linear transformation from the word embeddings of one language to the word embeddings of another language. To estimate the linear transformation, we follow standard practice and identify the anchor points by finding the k = 5 mutual nearest neighbors between the word embeddings across languages. We then proceed with the Procrustes approach from [Taitelbaum et al. (2019), which extends the original algorithm to more than two distributions. To translate words, we then directly use the transformed word embeddings.
the Procrustes algorithm (Schönemann, 1966) on the learned
3.5 ARCHITECTURE
Our method uses two-branch architecture, which extracts text and image features that share the same semantic embedding space. We briefly describe the network architecture choices below. We refer readers to the supplemental material for complete details.
Image network: To extract visual features, we apply a convolutional network over the images, which we label +,. We use R8sNet-18, initialized with ImageNet features (He al. 2016 Deng et al. 2009), and we add prediction head after the last hidden layer of the ResNet.
Text network: We use neural network to embed sentence, which label o, We single encoder with shared word embeddings across all languages, which has been shown to scale well the multilingual setting (Artetxe & Schwenk, 2019 Conneau al. 2020). All languages share the same vocabulary created using Byte Pair Encoding (Sennrich et al. (2066), which improves the alignment of embedding spaces across languages that share the same alphabet (Lample et al.) 2018a). We then use transformer from Vaswani et al. (2017), shared by all the languages. To produce outputs, we add prediction head, and normalize the outputs so that ||=||2
4 THE GLLOBETROTTER DATASET
In order to train and evaluate our approach, we have collected a federated dataset of images and captions that span 52 different languages. The full list of languages the footnote We combined three captioning datasets and translated them using Amazon Translate from Amazon Web Services. We use captions and images from the Flickr30k (Young et al., 2014), MSCOCO (Lin et al., 2014) and Conceptual Captions (Sharma al. 2018) datasets. The language in the federated dataset is diverse, covering both captions from human annotators and captions harvested from the web. The dataset contains total of 4.1M mmage-caption pairs, with English sentence mean length of 10.4 words. We will publicly release this dataset.
We split dataset into train, validation, and testing set. We make the partition ensuring that they each contain a disjoint set of images and sentences. We use 3.15M unique text-image pairs for training. 787k for validation, and 78.7k for testing. The training and validation splits contain samples corresponding to all languages, and each image only has one language associated with it. The testing set is translated all languages (the same samples), to have ground truth alignment.
EXPERIMENTAL EVALUATION
Our experiments analyze the language translation capabilities of our model, and quantify the impact of vision on the learning process. We call our model Globetrotter.
Afrikaans, Albanian, Amharic, Arabic, Azerbaijani, Bengali, Bosnian, Bulgarian, Chinese, Croatian, Czech, Danish, Dari, Dutch, English, Estonian, Finnish, French, Georgian, German, Greek, Hausa, Hebrew, Hindi, Hungarian, Indoniesian. Italian, Japanese, Korean, Latvian, Malay, Norwegian, Persian. Pashto, Pol- ish, Portuguese, Romanian, Russian, Serbian, Slovak, Slovenian, Somali, Spanish, Swahili, Swedish, Tagalog, Tamil, Thai, Turkish, Ukrainian, Urdu, Vietnamese.
5

---

Under review as a conference paper at ICLR 2021
Supervised
92.5%
Chance
0.5%
Text Only
8.9%
We evaluate our transla- tions at the sentence-level. Our ap- proach outperforms several unsuper- vised translation baselines. While unsupervised approaches are still no match for fully supervised methods, our approach uses significantly less supervision.
Figure 3:
Lample Conneau (2019)
Sigurdeson et al. (2019)
56.7%
9.0%
Globstrotter (ours)
without Lv
9.5%
without
without Lc
68.4%
Model
72.3%
0%
20%
40%
60%
80%
100%
Percentage Retrieved
Chance
0.5%
Figure 4: We evaluate our transla- tions at the sentence-level with human-generated test set. Fluent speakers for 11 of the languages manually annotated translations in the test set. Our approach outper- forms several unsupervised transla- tion basclines on this test set as well.
Text Only
11.9
+0.8%
Lample Conneau (2019)
Sigurdsson et al. (2019)
51%
(4.0%
Globetrotter (Qurs)
+2.6%
Sentence-level Supervision
91.1%
13.1 9%
0%
20%
40%
60%
80%
100%
numan-ggeneattee test set generrtte
Percentage of Retrieved Positives
5.1 BASELINES
Sigurdsson et al. (2020): The closest approach to ours is Sigurdsson et al. (2020). which is state-of-the-ar art approach for unsupervised word translation using cross-modal information. Their original model is trained to translate between just two languages, and our experiments work with fifty languages. We therefore extended their method multiple languages by creating different word embedding and adapting layer for each language, which we use as the baseline. We use the same vocabulary as our method, but train separate word embeddings for different languages.
We also compare the state-of-the-art unsupervised translation ap- proach that does not use visual information. We experimented with several baselines, and chose the one that performs the best. This baseline uses a cyyle-consistee pairs of languages. We train their method on our dataset, for all M languages simultaneously. We originally experimented with adding cycle-consistency constraints for all M2 language pairs, but this resulted in poor performance. We randomly select a total of 5M pairs, where each language appears five times the source and five times as the target. We also experimented with Lample et al. (2018b), but this performed worse than Lample & Conneau (2019).
Lample & Conneau 2019):
(or sick-trrnslation) loss between
Text-only model: To quantify the impact of vision, also train version of model where all images and image-related losses are removed, as in Devlin et al. (2019). This model is capable of learning some basic cross- gross-lingual concepts by having different languages using the same tokens.
Source: Spanish
Target: Russian
Target: Hebrew
remadelación
Buyt na ropoot e 6apa na ирıлиn View of the ciry from roofop har
neengo
An aerial view during its redevelopment
View from roef terrace
Actir asiste al estreno de los angeles celebrado
Arrep vocentar upeanepy eccautta
71221 pip DTM
los angeles premiere
Actor
season premiere
arrives premiere miu1 by mamp bB muuppp
Hustración de la niña dibujos animados
Hapиcenamnahh
color negro sobre el londo blanco
eaasa aumamons it nocymиasn na Gezscs dpoue Hand drawn sketch with soft classic couch and pit- lows on the white background
Illustration cartoon black color
Cartoon group of girls
white background
Table I:
We show some examples of sentence-level translations obtained by our approach. English only shown for visualization purposes.

---

Under review as a conference paper at ICLR 2021
All Vocabulary
Disjoint Vocabulary
Chance
0.27
0.27
Text Only
25.46
Lample & Conneau (2019)
27.67
+0,47
13,50
+2.32
Sigurdsson al. (2019)
+11.38
17.15
-10.19
Globetrotter (Ours)
44.05
2.884
26.20
+6.87
Sentence-level Superviaion
39.57
1,98
18.24
-8.71
11
22
33
44
55
8
16
24
32
40
Before Procrustes
After Procrustes
Average Recall@10
Average Recall@10
We also evaluate word-level translation. Although our approach is trained on sentence- level similarity, the word embeddings also learn provide strong word-level translation. The results can be further refined with Procrustes.
Figure 5:
Source: Spanish (English trans.)
Target: Russian (English trans.)
Target: Hebrew (English trans.)
chica (girl)
девуыиха (girl)
72'
(wife)
tenis (tennis)
zemum (prefix for tennis)
0/22
(tennis)
personas (people)
Joozen (pcople)
(pcople)
aire (air)
novy (air)
upa
(hackground)
campo (field)
House (field)
7702
(in field)
béisbol (baseball)
Getto6oz (baseball)
(baseball)
espect (prefix for show)
moy (show)
U1T
(event)
motocic (prefix for motorcycle)
MoroQqa (мCтOQqq is motorcycle)
1o
(DUDIN is motorcycle)
camion (truck)
amroffy (hus)
2012
(in street)
sombrero that)
RоCrIOS (suit)
7271
(nabr shirt)
hombre (man)
(Myл*mma man)
(man)
mientras (while)
(when)
mmx
(after the)
par (two, prefix for couple)
uapa (couple)
'07
(the second)
calle (street)
yıına (the outside)
2172
(in street)
camino (path)
11215eee (beach)
7
(path)
Table 2: We show examples of Spanish- Russian and Spanish- Hebrew word-level translations.
Fully Supervised: To understand the gap between unsupervised and supervised approaches, train our method with paired language corpora. We use our same framework, except we set the values of to 1 for paired sentences, and 0 for unpaired sentences.
Common Evaluation Setup: Throughout our experiments, we adopt a common evaluation setup to evaluate all models. We train all models for 200 epochs and select the best model on the held-out validation set. In all cases, vision is not used during testing.
5.2 SENTENCE-LEVER TRANSLATION
We evaluate sentence translation using held-out data that contains of sentences translated to all languages. We produce translations by retrieving the nearest examples given a query. From the test set, we randomly select 200 captions, for all M languages, with a total of 200M sentences. Each one of these sentences is used as query during test, and it has M - - positives (same sentence in different languages). The metric we report the percentage of positives the model ranks in the top M among all the 200M - possible options. In order to rank target sentences, we compute the similarity between them and the query sentence, and rank them according this value. We show results in Fig. 3 Our method outperforms all baselines by significant margin, underscoring the utility transitive relations across modalities.
Fig. 3 alss reports ablations of our framework when not training with each one of the four losses Eq. 5 Training without losses C, (Eq. 3 or L. (Eq. 4 implies breaking the transitive closure represented in Fig. 2 which results in a drastic decrease in performance. C, (Eq. is the loss that makes the cross-lingual alignment explicit, but importantly it is not required to close the transitive relation through the visual modality. Training without represents considerable drop in accuracy, but the results are still better than baselines. Finally, C, also contributes to the final performance, consistently with prior work (Lample & Conneau 2019:
Liu al. 2020

---

Under review as a conference paper at ICLR 2021
a)
b)
307
anomuntatrree wirh eddurimmmee fandrr
CO RE & B4 MR MEB dR
اسمر tsin rifig with actra coon track)
in
wave
man ridix& surfocard on top of vone the oowan)
train trann track going under brioge)
surtboard
d)
DCB
O77
241
C00
021
daso da babur dinde uka sace dago gidan mutum. f 'on sando.
pobjjeri medkedt dreweeo 1th
smajese
G
WART FIN SH
stufted
(a stolen motorcycle was recovered from the home person, police said!
UNI
bagne should position themsshes the stars)
camiing
(A people flying ichee serdy teach)
KPMc
3:00 MRS
Figure 6: Qualitative results. We show two examples of positive matches (top) and two examples of negative matches (bottom). We refer the reader to Section 5.4 for further analysis.
We show some examples of our sentence translations in Tab. Our approach works on all language pairs and we simply select a few for visualization purposes. These examples show how our method aligns languages following their visual semantics.
Our method does not rely on artifacts from machine-generated translations and generalizes to human-trang -ranslata data. In order to prove it, we additionally collect a test set of 200 English captions trans- We report results in Fig. 4 where we show the accuracy values both for camann-translated and machine-translated texts. We use the same metric as before, now for M = 11. Our approach outperforms the unsupervised baselines on the human-generated test well. While all methods ex- perience a small decrease in performance when tested in human- translated data, the small difference between the results in the two test sets validates the quality of the evaluation.
translated by fluent speakers to 11 different languages, for total of 2200 numan-generatee
l Fl
5.3 WORD-LEVEL. TRANSLATION
Following the evaluation in Sigurdsson al. 2020), also evaluate the word-level translation. Since we lack ground truth translation at this level, we obtain ground truth for evaluation by auto- matically matching words across languages. For every language pair. we find which words co-occur frequently in a sentence between the two languages. See Appendix B.2 Then we test each pair of languages separately. For every translation, we evaluate retrieval in both directions. Fig. S reports the average Recall @ 10 for all pairs of translations and all pairs of languages. In the right column, we exclude from the list of pairs those where the token is the same in the two languages. Even the model trained with text only which performs poorly on sentence-level translation obtains strong results, highlighting the importance of using a shared vocabulary. We show some examples of word translation in Tab. 2
5.4 ANALYSIS
Visualizing transitive matches: Fig. 6 shows examples of estimated transitive similarity values. We show predicted a" (inter-image similarity), a (cross-modal similarity), and B (inter-sentence similarity). Fig. 6 and show examples where both the similarity between images and the cross- modal similarity are high, resulting in a large a. If these pairs were to be used for training. they would be positives. The model correctly predicts high value between the two texts. Fig. 66 demonstrates the importance of using a" in addition to ax to create language pairs. In this case, the visual content between the two images corresponds, and the model detects that correctly with a high a" value. However, because web data is not always clean, the caption in the left does not correspond to the visual content. This is correctly captured in the small a² value. If we were using this pair for
2The 11 languages with ground-truth human translations are: Dutch, French, Hebrew, Hindi, Italian, Ko- rean, Polish, Portuguese, Russian, Spanish, Turkish.
8

---

Under review as a conference paper at ICLR 2021
Source: Spanish
Target: Russian
Target: Hebrew
escuchas, sikencio persons ayu- dará a entender de manerus que las pulabras simplemente pucden
ppsuuusss souucamaadi sa Gyseиu is na zepeasmmes qpone
D'mbOon 15 5tap TPoN TrrK
help you to understond in ways that words simply can wor
you listen, person
Holiday writeen on piece of paper, on background
If cuz you off frecuuee you gave sclesors
Un nuevo concepto
Jagnue wodlpaarmee MORUET arrossofiua uautzo
nembuse
mrzza
glmpse concept
Rear image of automoblle model in
Purchasing car? here tech-
nologies to look out
Un tabby gris manchado plantas veries. spoited tobby siss plants
Kрoаи iejerr оepeasje 2019
1752 CTTR harw
ofiammoR иpooeepe
A rabbit walts the foregrouni for romline
Red fox in field
Table 3: We illustrate some failure cases. Please the end of Section 5.4 for discussion.
Den
Flench
Figure 7: We show sentence-level translation accuracy by query-target language pair. In the figure, the languages are sorted by family (Ro- mance, Baltic, etc.). The block- diagonal structure shows that lan- guages from the same family easier to translate between. We also find that language isolates in our dataset perform worse overall (e.g. Tamil, the only Dravidian language).
Afriaaso
language
konengian
Slovenzhn
Crobban
Greck
Housd
Whyy
Indones
jonanese
An Pe
NONI Mach
The
Pu
Targel language
training. it would be considered negative example despite significant visual similarity. Thus, the misalignment noise is not propagated to the cross-lingual loss. Finally, Fig. 6 shows an example where both sentences accurately describe their corresponding image, but the images do not match. As expected, this would result in negative pair.
Failure cases: We show three prototypical examples of failure cases in Tab. 3 In the first example, the caption is not related to any visual concept, causing model to translate it incorrectly. The second example shows how some words are related to incorrect concepts due to spurious correlations in the training set. In this specific case, the phrase "new concept" is strongly associated to cars, since it appears in training in the context of "concept cars", i.e. vehicles from car companies to explore new designs. Therefore, the model retrieves sentences referring to cars, even though they do have any relation to the phrase "new concept".. Finally, the third failure case shows a sentence with new word ("tabby"), where the model is overreliant on context translate instead.
Translation difficulty by language: We itemize the performance of sentence-level translation by language in Fig. Languages from the same family are often easier to translate between. The most difficult language is Tamil, the only Dravidian language in our dataset.
6 CONCLUSION
Leveraging transitive relation between language and vision, experiments show framework learns a strong representation for both sentence-level and word-level machine translation without parallel corpora. We believe vision will continue be valuable for learning robust language models.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Anonymous. (VECO}:
Variable encoder-decoder pre-training for cross-lingual understanding and generation. In Submitted to International Conference on Learning Representations, 2021. URL
https: Settps:/cppeee http://pppneeeeeeeee/eeee net/ forum?id hZM3BE, under review.
Mikel Artetxe and Holger Schwenk. Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7:597-610, 2019.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. In 6th International Conference Learning Representations, ICLR 2018, 2018.
Emmanuel Azuh, David Harwath, and James Glass. Towards bilingual lexicon discovery from visually grounded speech audio. Proc. Interspeech 2019, pp. 276,280, 2019.
Timothy Baldwin. Low-cost, high-perforne ings of the 39th Annual Meeting of the Association for Computational Linguistics, pp. 18-25, Toulouse, France, July 2001. Association for Computational Linguistics. doi: 10.3115/1073012. 1073016. URL http::::ps:........... www ac web
performance translation retrieval: Dumber is better. In Proceed-
Timothy Baldwin and Hozumi Tanaka. The effects of word order and segmentation on transla- tion retrieval performance. In COLING 2000 Volume I: The 18th International Conference on Computational Linguistics, 2000.
R. Brown. Transfer-rule induction for example-based translation. In Proceedings of the MT Summit VIII Workshop on Example-Bace Based Machine Translation, 2001.
Ralf D Brown. Example-based machine translation in the pangloss system. COLING 1996 Volume 1: The 16th International Conference Computational Linguistics, 1996.
Ralf D. Brown. Automated dictionary extraction for "knowledge-free" example-based translation. In In Proceedings of the Seventh International Conference Theoretical and Methodological Issues in Machine Translation, pp. 111-118, 1997.
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A Plummer. Learning to scale multilingual representations for vision-language tasks. European Conference in Computer Vision, 2020.
lacer Calixto and Qun Liu. Sentence-level multilingual multi-modal embedding for natural lan- guage processing. In Proceedings of the International Conference Recent Advances in Natu- ral Language Processing, RANLP 2017, pp. 139-148. Varna, Bulgaria, September 2017. IN- COMA Ltd. doi: 1/978-954-455--449 020. URL pnttps://ddat www.orgg///000 266157 978 904-954-455 -0344452--055 029-522-09--6002 020
Chris Callison-Buree Miles Osbore, and Philipp Koehn. Re-evaluating the role of Bleu in machine translation research. In 1th Conference of the European Chapter of the Association for Compu- tational Linguistics, Trento, Italy, April 2006. Association for Computational Linguisties. URL https: httt:::/www.alLwe..ooo www. httpg//aaholoog//eeeeeo 1032
Konstantinos Chatzitheodorou. Improving translation memory fuzzy matching by paraphrasing. In Proceedings of the Workshop Natural Language Processing for Translation Memories, pp. 24- 30, Hissar, Bulgaria, September 2015. Association for Computational Linguistics. URL http:
7/ www. www.aacwweb... org/ hwtp://........atooooooo/------/
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Everest Hinton. A simple frame- work for contrastive learning of visual representations. Proceedings of the 37th International Conference on Machine Learning, 2020.
Yun Chen, Yang Liu, and Victor OK Li. Zero-resource neural machine translation with multi-agent communication game, In Second AAAI Conference Artificial Intelligence, 2018.
10

---

Under review as a conference paper at ICLR 2021
Zewen Chi, Li Dong, Furu Wei, Nan Yang. Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. Infoxlm: An information-
theoretic framework for cross-lingual language model pre-training. ArXiv, July 2020. URI.
infoxlm-an- hwwooo/mmaann-fffooaaio--------- ort. htg.a//wwo........oooooaaaaaaaaa - -pre-tr
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 10.18863//1/2222.aalll 747. URL https: 7/www. acwweb. org/ /ttpa//oogg//22.aaaaa...
747
Lambros Cranias, Harris Papageorgiou, and Stelios Piperdis. A matching technique in example- based machine translation. In COLING 1994 Volume 1: The 15th International Conference on Computational Linguistics, 1994. URL https: / www.ww.lweb.oo.
C94-1014
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10.18653/w1/N19. 142333.9914. URL https:
7/ 7/www. www.ww.wwwb..... wttp:/www..g../ooooo httgg//nthooooo/NlSe.. 1423
Meiping Dong, Yong Cheng, Yang Liu, Jia Xu, Maosong Sun, Tatsuya Izuha, and Jie Hao. Query lattice for translation retrieval. In Proceedings of COLING 2014, the 25th International Confer- ence on Computational Linguistics: Technical Pupers, pp. 2031-2041, 2014.
T El--Shishtawy and A El-Sammak. The best templates match technique for example based machine translation. arXiv preprint arXiv: 1241, 2014.
Yuwei Fang, Shuohang Wang, Zhe Gan, Siqi Sun, and Jingjing Liu. Filter: An enhanced fusion method for cross-lingual language understanding, 2020.
Orhan Firat, Baskaran Sankaran, Yaser Al-Onaizan, Fatos Yarman Vural, and Kyunghyun Cho.
Zero-resource translation with multi-lingual neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 268-277, 2016.
Xavier Garcia, Pierre Foret, Thibault Sellam, and Ankur Parikh. A multilingual view of unsuper- vised machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3160-3170, Online, November 2020. Association for Computational Linguistics. URL https:/ / www. acl web...b.... wowooo///aooo
findings Sindings--mnnp.288
Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. Image pivoting for learning multilingual multimodal representations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pp. 2839-2845, Copenhagen, Denmark, September 2017. Association for Computational Linguistics. doi: 10.1853//1/DD1--133. URL https: 77 www. a web. www.//atttooooooooooooooo http://annhoolooyy/11--1333
Daniela Gerz, Ivan Vulić, Edoardo Maria Ponti, Roi Reichart, and Anna Korhonen. On the relation between linguistic typology and (limitations of) multilingual language modeling. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 316-327, 2018.
David Harwath, Galen Chuang, and James Glass. Vision interlingua: Learning multilingual semantic embeddings of untranscribed speech. In 2018 IEEE International Conference on Acous- fies, Speech and Signal Processing (ICASSP), pp. 4969-4973, IEEF, 2018

---

Under review as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viégas, Martin Wattenberg, Greg Corrado, al. Google's 's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339-351, 2017.
Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, and Bryan A Plummer. Mule: Mul- timodal universal language embedding. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
T. Kimura, J. Matsuoka, Y. Nishikawa, and Y. Lepage. Analogy-based machine translation using secability. In 2014 International Conference on Computational Science and Computational In- telligence, volume 2, pp. 297-298, 2014. doi: 10. 10.1109/SCC12004..444
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In 6th International Conference on Learn- ing Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018a. URL https:/
https:://penrevview opw.reviewwwe...oooo.. net forum?
rkyTkf AZ
Guillaume Lample, Alexis Conneau, Mare' Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. Word translation without parallel data. In 6th International Conference on Learning Repre- sentations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018b. URL https: http///ppweeeww.eeeeeeeee net.iiewweeeeeeeeeeeeee.e mnetfforumiid H196sainb
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Phrase-based & neural unsupervised machine translation. In Proceedings of the 2018 Confer- ence on Empirical Methods in Natural Language Processing, pp. 5039-5049, 2018c.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Chunyang Liu, Qi Liu, Yang Liu, and Maosong Sun. Thutr: A translation retrieval system. In Proceedings of COLING 2012: Demonstration Papers, pp. 321-328, 2012.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. Multilingual denoising pre-training for neural machine translation. arXiv preprint 2r0iv:2001.082 10, 2020.
Makoto Nagao. A framework of a mechanical translation between japanese and english by analogy principle. Artificial and human intelligence, pp. 351-354, 1984.
Jason Phang, Phu Mon Htut, Yada Pruksachatkun, Haokun Liu, Clara Vania, lacer Calixto, Katha- rina Kann, and Samuel R. Bowman. English intermediate-task training improves zero-shot cross- lingual transfer too. In Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 9th International Joint Conference on Natural Language Processing, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl blog 1.8, 2019.
Peter H Schönemann. generalized solution the orthogonal procrustes problem. Psychometrika, 31(1):1-10, 1966.
12

---

Under review as a conference paper at ICLR 2021
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 1...866//11/111-110 URL https:: www. ac web. org/antho 1oyph16-11/p13133
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of ACL, 2018.
Gunnar A. Sigurdsson, Jean-Baptiste Alayrac, Aida Nematzadeh, Lucas Smaira, Mateusz Ma- linowski, João Carreira, Phil Blunsom, and Andrew Zisserman. Visual grounding video for unsupervised word translation. In CVPR, 2020. URL https:://pttttieeeeeeee www.mmicm/c///....o.o...o.......
wisuall-rounae
Lucia Specia, Stella Frank, Khalil Sima' an, and Desmond Elliott. A shared task on multimodal machine translation and crosslingual image description. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, pp. 543-553, 2016.
Yuanhang Su, Kai Fan, Nguyen Bach, C-C Jay Kuo, and Fei Huang. Unsupervised multi-modal neural machine translation. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 10482-10491, 2019.
Hagai Taitelbaum, Gal Chechik, and Jacob Goldberger. A multi-pairwise extension of procrustes analysis for multilingual word translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3551-3556, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Katharina Wäschle and Stefan Riezler. Integrating a large, monolingual corpus translation mem- ory into statistical machine translation. In Proceedings of the 18th Annual Conference of the European Association for Machine Translation, pp. 169-176. 2015.
Jonatas Wehrmann, Douglas M Souza, Mauricio Lopes, and Rodrigo C Barros. Language- agnostic visual-semantic embeddings. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5804-5813, 2019.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2: 67-78, 2014.
13

---

Under review as a conference paper at ICLR 2021
APPENDIX
We divide the appendix in two sections. In Section A we show more results, and in Section B provide more information about the implementation our method.
we
A ADDITIONAL RESULTS
A.1 FEATURE GENERALIZZTION
Training a language model, as opposed to a text representation only designed for image retrieval, has the crucial advantage that it can be finetuned perform downstream NLP tasks. In this work are interested in evaluating how well the representations generalize across languages, after training on downstream task. We evaluate our model on sentence correspondence: we split sentences in two, and half of the times we swap the second half of the sentences with other sentences of the same language. The model has determine whether or not a sentence is coherent and the beginning of the sentence corresponds to the end of the sentence. We control for uppercase, word breaks, length of sentences etc. so that the model cannot find an easy shortcut (cheat), and has rely on the semantic and syntactic structure of the sentence. We show examples of the test in Tab, for English.
We train all the models for one epoch on half of the languages in the testing split (first half in alphabetical order), and test on both held-out samples from that half, and the languages from the other half (new languages the sentence correspondence downstream task has not seen). We train single transformer layer top of our representation, with one head. For Sigurdsson et al. (2020), we do not apply the max- max-pooling over words order to have a representation for each word. We show results on Tab. [5] The results show that methods trained with language models much better at performing language tasks. It also shows that our method, trained with alignment, not only performs better on the languages the downstream task has been trained on, but also generalizes better to other languages the sentence correspondence task has never seen, indicating that the model has very aligned representation across languages. The relative decrease in accuracy computed as the percentage decrease of the difference between the accuracy and the chance accuracy.
A.2 ADAPTATION NEW LANGUAGE
We test how well our framework can adapt to incoming languages. For this purpose, test on English and Chinese (separately), which were held out during training. To do so, we precompute features for images and texts from the languages used during training, and finetune the model for the new language using the same losses as before. We train for one epoch.
After finetuning for English and Chinese, repeat the same experiments performed for the other languages, showing that our system able to adapt to new languages without losing the multilingual alignment. See Tab. [6 for translation results, and Tab. 7 for sentence correspondence results. For the sentence correspondence test, we use the head we trained before (without finetuning on the new languages).
A.3 MORE RESULTS ON TRANSLATION DIFFICULTY PER LANGUAGE
Similarly to Fig. 7 we show Fig. g the word translation accuracy matrix for every pair of lan- guages. As expected, languages that share important part of their vocabulary are the ones with highest similarity scores. Specifically, there is a very high similarity between Bosnian, Croatian and
Sentence
Corresponds
piece of cake sitting next to pastries on a white plate with red and yellow sauce
Yes
Seamless pattern with white bugs black background
Yes
A big tower with a big tv genre and common language
No
A hand holding smartphone with of picnic by lake
No
Table 4: Sentence correspondence task examples. See Appendix A.I
14

---

Under review as a conference paper at ICLR 2021
Serbian, since the three of them standardized varieties of the Cerbo-Croatian language. Also, In- donesian is very close to Malay, as the former is standardized variety of the latter. A final example the Czech and Slovak pair: the two of them are languages from the Czech-Slovak group. This shows the importance of cognates across languages. We can find similar patterns for languages that not close, but that share the same family alphabet.
Hungaran
Finnish
Estonian
Arabic
Amharic
Hebrew
Dari
Pashto
Spanish
French
Italian
Romanian
Afrikaans
German
Danish
Query language
Norwegia
Polish
Czech
Sloven
Bu gania
serbian
Groatian
Ukrainlan
Russian
Greek
Georgian
Azerbaiaai
Hausa
Somali
Swahil
Vietnamese
Thai
Hindi
Urau
Tami
Malay
Indonesian
Körean
Japanese
Hungann
Estonan
Larvan
Amtana
Habrew
Persan
Pash
Spanish
Portuguese
Italiant
Roman
Dutch
German
Nonveg sn
hollsn
Czech
Slovak
Bulgaria
Bosnian
Ubrainia an
an
RUssiar
Alaanaa a
Georn
persallle miniin
Turkish
Sarmal
Swahill
Vietnam Retnamese si
Hindi
Urdu
Bengali
Mallyy
Indonesi an Taoaloo alle
Target language
Figure 8: Word-level similarity across languages. See Appendix
A.3 for more information.
We also show in Fig. the sentence-level translation values from Fig. but now we plot A AT Instead of illustrating which language pairs close, easier to work with, shows which language pairs are asymmetric in the difficulty of the translation. Rarer languages -e.g. languages that are far from the others in the linguistic tree such as Somali, Tamil or Hindi- are easier to translate from than to translate to.
Seen accuracy (%)
Unseen accuracy (%)
Relative decrease (%)
Chance
50
50
Text only
71.54
64.94
30.64
Lample & Conneau (2019)
72.41
68.22
18.70
Sigurdsson al. (2020)
53.25
52.89
11.07
Globetrotter (Ours)
75.95
74.54
5.43
Supervised
75.64
68.73
26.95
Table 5: Sentence correspondence results, See Appendix A.I
15

---

Under review as a conference paper at ICLR 2021
English retrieved positives (%)
Chinese retrieved positives (%)
Chance Text only
0,48
0,48
19.27
12.98
Sigurdsson et al. 20000)
59.
37.96
Globetrotter (Ours)
75.67
62.81
Supervised
94.87
92.77
Table 6: Sentence translation results for finetuning. See Appendix A.2
English accuracy (%)
Chinese accuracy
Chance
50
50
Text only
65.97
55.75
Sigurdsson et al. (2020)
50.2
50.5
Globetrotter (Ours)
73.27
67.17
Supervised
69.17
62. 14
Table 7: Sentence correspondence results for finetuning. See Appendix A.2
A.4 CLUSTERING THE REPRESENTATION SPACE
In this experiment, show how differently the representation space is clustered when train with and without visual alignment. We extract features for the test set examples both for the full model and the text-only model, and cluster these features using k-means, with k 50 clusters. In Fig. T0 we show three sentences belonging to each one of the first three clusters (the selection of both the sentences and the clusters is arbitrary). When training with visual alignment the clusters have a semantic meaning, and when training without it the clusters are language-specifte proving that cross-modaa alignment necessary to obtain good semantic representations.
A.5 GENERATED TRANSLATIONS
The learned representations are not only good to do translation by retrieval, but also to generate translations. In order to do so, use a GPT-2 decoder (small version) from Radford et al. (2019). pretrained on English. Next, we finetune on English sentences from our dataset, and after that we finetune it yet again but conditioning it feature vectors from the English finetuned model from Appendix A.2] To do this we use an extra linear layer at the input, and we concatenate the results with the input word embeddings. After that, obtain GPT--2 model that generates sentences in English based on the input representation. We then test it for translation by inputting representations obtained from other languages, and generating English translations for them. The sentences used in the test were not used for any of the GPT-2 finetuning stages. We show results in Fig. [] We selected the first 10 translations that were generated, without any picery.. Interestingly, while our framework is not able to do an accurate literal translation, it does base the translation on the contextual knowledge provided by vision.
B IMPLEMENTATION DETAILS
B. TRAINING AND ARCHITECTURE DETAILS
We train transformer network with 4 attention heads and M hidden layers, with hidden size of = 512. The size of the embeddings at the output the heads (where the contrastive losses are computed) is D 128. We use batch size of 800. We set all the À values in 3 À 0.2. We train with an Adam optimizer and learning rate of 1e
normalize the feature values that ||2||2 1. Then the similarity value is computed with dot product, resulting in the cosine similarity. After that, we scale the value so that the range of the similarity is in (0, 1), instead of [-1, 1111
As mentioned in Section 3.5
16

---

Under review as a conference paper at ICLR 2021
Finnish
Estonian
Arabic
Amharic
Hebrew
Dari
Pashto
Spanish
Portuguese
French
Italian
Romamian
Afrikaans
German
Danish
Query language
Norweqian
Polish
Czech
Slovenian
Bu aanan
serbian
Crostian
Ukrainian
Russian
Albanian
Georgian
Azerbaiani
Hausa
Somali
Swahil
Thai
Hindi
Urau
Bengali
Tamil
Malay
Indonesian an
Körean
Japanese
Hungann
Estonian
Larvizn
Ammar
Hobrew Bw
Persian
Pashto
Spanish
Portiguest
Itailan an
Romana
DuEcT
"German
Dan Sn
swedd
sallsn
Czech
Slovak
Bujgarist
Shian
an
Ukrainiaa
an
Russ
Geergian
Georgiat
Turkish
llause
Swath
Vietma mesee
Hinsi
Urdu
Bengal
Malay
Indgnesion Tagaloo
Azerbal
Target language
Figure 9: Asymmetry in the direction of the sentence-level translation. See Appendix A.3
Clusters in full model
Clusters in text-only model
Cluster Savannah animals
Cluster French
(Arabic): فاك _ 4 dely 4و vid موكه كر 44 ااu شررر
un grand éléphant se tient près d'une clöture motif circulaire sur fond rouge
(Croatian): popodne provedeno igrajuči se sa slanovima (Georgian):
gafiio golshmmpa, gofhogoonn baga6ab gogemnon
homme silhouette plage
Cluster 2: Wedding
Cluster 2: Hindi
(Bengali), सनतय डरर नब
दाभ का पाह ीेद [िलाशा मे लिरर सपार पन बीनी. मंडन पहना क यीपास टeास vi it रकय सभााा IT. पाररी qua fिरs रिसाहीी zeet दिााrितत
(Slovenian): nevesta meri obleko, ki ima roza sopek (Urdu): بي فرب ك شاعي كي إلنمس
Cluster Bicycle/Motorcycle
Cluster Greek
(Swedish): en cykel kastad ner sanden på strand. (Japanese): #
потаμбς sivaı Ěva δπςσςiλλς anuslo yıa kavó. maλid πápra éva εεxaαμévo krimo прáσιva uddia ато yúpo evuōpεio.
(Tamil): a u. mus de Grraion.
Figure 10: Clustering in the representation space. When trained without visual alignment the clusters are language-specifg and when trained with visual correspondence the clusters have a semantic meaning.
B.2 GROUND TRUTH FOR WORD TRANSLATION
In order to generate the ground truth translations at the token level, we use the split of the dataset that translated to all the languages. We then create ground truth token translations for every language
17

---

Under review as a conference paper at ICLR 2021
Original sentence
Generated English translation
(Russian) коыиk отдвіжает обочине сопнечныйй петний денb (cat resting on the curb in sunny summer day)
cat lying on the grass
(German) Hardrock-
treten wahrend des Musikfesttvals auf (hard rock artists perform during music festival)
artist performs stage festival.
(Croatian) Nekoliko snowboardera koji su poletjeli niz snijeg prekriveno brdo. (A few snowboarders taking off down snow covered hill.)
some people skiing the snow
(Arabic) alay- أالي علر الفيية 3 الأررر (banknotes on black background)
silver coin on the black background
(German) Portrát auf dem blauen Himmel Hintergrund (portrait on blue sky background)
bald eagle the green background
(Georgian) @adooopnonn goom dimtgbols faddmaannaggin (additional photo property listing)
photo of the front porch
(Swedish) utsikt över sjön från rutten (view over lake from the route)
picture of the beach on sunny day
(Hungarian) légi kilátás a strand a legtöbb fehér és tiszta hamok (aerial view of the beach with the most white and clean sand)
photo of the mountain lake winter
(Croatian) šetnja kiši, (a walk in the rain
photo of the walking along the streets
(Afrikaans) akteur woon die spesiale geleentheid by (actor attends the special event)
some person attends los angeles premiere
Figure 11: Translation by generation. See Appendix A.5 for more information.
pair separately. In order do that, we follow the tf-idf algorithm. We exploit the fact that have alignments of languages at the group-of-words (sentence) level. The idea is that if the word "car" appears in an English sentence every time that the word "voiture" (car in French) appears in its French translation, they probably mean the same. In the following explanation, assume we are looking for the translation of specific token 1,4 from language A into some token 1. from language B. We just redefine the concept of "document"
the classical tf-idf algorithm to be the collection of all the words (with repetition) in language that appear in the same (translated) sentence as t:-. We call this collection (document) d.
First, we create count of tokens in language B that appear in the document d, and compute the lerm frequency (tf) using this count:
fia Eyeafy
ff
(6)
where fa is the count of the token E in a document d. Second, we compute the inverse document frequency, that takes into account how usual a token general, for all D documents:
D idf, log d D: 1h Ed
Multiplying the and idf terms we get value for each (i,j) pairs of tokens (the value is not symmetric). We store tokens t/² and t as ground truth translation if and only if t is in the top for the tf-idf value of (i.j), for all j. and t. is in the top for the tf-idf value of (j. i), for all i.
The following are some examples of translations we obtain between Spanish and English: (electr, electr), (fotograf, ograph), (ción, ction), (grande, lar), (atas, jam), (pare, couple), (decor, decor), (ventana, window), (deportivo, team), (1950, 1950), (form, form), (30, 30), (casa, hom). (lave, key), (1960, 1960), (del, the), (libro, ok), (kara, kara), (ola, surfer), (fan, fan), (viol, viol), (%, %), (dar, standard), (segundo, sec), (equipo, sports), (rojo, red), (árbol, tree), (hierba, gras), (durante, dur), (bron, ze), (mani, demonstr), (pequeño, sm), (ti, typ), (turística, attra). (corre, run), (mus, muse), (atrac, tour), (baño, bat), (mam, mom), (una, on), (element, element), (ijo, son), (ant, ol), (mural, mural), (chocola, chocola). (iste, sad), (cinta, bon), (carro, cart), (edif, bu), (planta, plant), (óc, broccoli), (prim, st), (camina, runway), (cerca, close), (pop, artist), (nacional, nation), (ustr, alian),
18

---

Under review as a conference paper at ICLR 2021
(vest, dress), (motocic, motore), (perro, dog), (largo, ong), (t, +), (ates, tom), (fram, rasp), (camina, wal), (inta, inta).
B.3 TEXT NETWORK DETAILS
The input to the text network is sequence of tokens {[SEQ, w1; w;} that represent a sentence in any language (Devlin et al. 2019). Before inputting tokens to the transformer, we encode them with fixed-length vector representation. To embed input tokens. we use a V d word embedding matrix Pu. where / is the size of the vocabulary considered by the tokenizer. We use V/ = 30, 000. We augment the input encoding with positional information (word index), translating the encoding by a learned vector: (w;) "e,,, + (w;) where Opos encodes the word position of w,,
We then input the augmented tokens to the transformer. A transformer block (Vaswani et al.) 2017) consists of multi-headed self-attention layer followed by a linear layer, that outputs a hidden repre- sentation for every token in the input sequence. These transformer blocks are concatenated in series to get deeper representations. Let H E R be the d dimensional hidden vectors at layer The transformer first computes vectors for queries Q = W H keys K = W H and values V = W,, H where each W. Raxd is matrix of learned parameters. Using these queries, keys, and values, the transformer computes the next layer representation by attending to all elements in the previous layer:
QQKT
H SV where S = softmax
(8)
Vd
In practice, the transformer uses multi-head attention, which repeats Equation S once for each head, and concatenates the results. The network produces final representation (M(SEq hid h.., hM for a stack of M transformer blocks.
As mentioned in Section 3.5 we also add prediction head. This head takes as input the final hidden
representation for the [SEQ
token, MM
[[$FQ]]
B.4 DATASET DETAILS
To collect the dataset, we used captions from the Flickr30k (Young et al. 2014), MSCOCO Lin al., 2014) and Conceptual Captions (Sharma al. 2018) datasets. Flickr30k and MSCOCO image captioning datasets that have been carefully curated and annotated in a controlled setting, so the text descriptions are accurate and thorough. However, most the images in our datasets come from Conceptual Captions, which consists of captions harvested from the web, so the visual- language alignment more noisy.
We randomly split each dataset into 52 equally sized parts, one for each language supported by the machine translation service we use. Each split is assigned a unique language, and splits with the same language across datasets are combined. The split which is assigned the English language is set aside and translated into all 51 other languages, and only used in testing. We also set aside the split translated into Chinese for fine tuning experiments. The remaining 50 splits have their original English captions discarded, and are then split 80%-20% into training and validation data. All experiments shown in Section 5 are run the reserved test data.
Note that there is no overlap at all (visual or linguistic) between the different splits, except for the test split. Please see Table for more details about the dataset.
Conceptual Captions
Flickr30k
MSCOCO
Total
Image/language pairs per language Total image/language pairs
3.1k 159k
11.91 6166
63.8k:
78.7k 78.7k
3.3M
4.1M
Table 8: Dataset statistics. There are a total of 52 languages.
19


Reviews:
[{"review_id": "rpr2sc0oEZm", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper introduces a framework that leverages transitive similarities between images and text to align multiple languages, without bi-lingual parallel training corpora. Unlike most existing unsupervised multi-modal translation works that only focus on a single (or a few) language pairs, this work proposes a one-for-many framework that can be easily applied to cover 52 languages at once.\n\nThe paper is well-written and easy to follow, and the notation is clean. The experiments are convincing with interesting case studies.\n\nThe major issue I want to raise is:\n\n(1)The methodology proposed is retrieval-based (which is popular at the time of SMT). One shortcoming of retrieval-based methods is that its applicability is limited by the size of corpora. In terms of unsupervised translation, this problem is even more severe. Because the goal of unsupervised MT is to extend the successes of MT to low-resource language pairs. I would like to hear from the authors how we can improve or alleviate this issue. \n\nSome other minor concerns are that:\n\n(2)The authors evaluate word-level translation based on ground truth obtained using statistical patterns. This somehow makes the comparison with generative models (e.g. Sigurdsson et al 2020) unfair.\n\n(3)A missing and important related work. Unsupervised Multi-modal Neural Machine Translation. Yuanhang Su et al 2020\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UojRsZVWDfw", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to leverage images to train an unsupervised machine translation (MT) model. Their main idea is that the similarity of images can be used as a proxy for the similarity of sentences describing the images. The sentences, in turn, can be in different languages, and knowledge about their similarity can be exploited as training signal for an unsupervised MT model, i.e., training without parallel sentences. Their model consists of a sentence encoder and an image encoder. For training and evaluation of the model, they use translations (multi-way for the test set) of image captioning datasets.\n\n\nThe authors evaluate on two different tasks: word-level translation and sentence-level translation. (However, the sentence-level translation is retrieval-based, i.e., no sentences are being generated.) They compare to multiple baselines, which seem to be chosen well. (The one that I would be consider missing is mBart (see below for the reference), but that's hard to train on many common GPUs, so I wouldn't necessarily expect that.)\n\nOverall, the idea the authors are presenting is convincing, the experiments are clear and well designed, and the model makes a lot of sense. I don't see any major shortcomings of this paper. Thus, I would be excited to see it being presented at ICLR.\n\nA minor shortcoming is that mBart isn't mentioned anywhere. In fact, the authors claim that they propose the first multilingual unsupervised model (lines 3 and 4 in the abstract). This should be corrected. The reference for mBart is: Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "2ljcxEkxUnj", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe authors propose a method for crosslingual sentence retrieval that uses images to ground sentences in different languages and to project these sentences into a meaningful semantic space. The data used in the model are image-caption pairs in each language, and never parallel sentences in two or more languages. Authors use existing English-language image captioning datasets (Flickr30k, MSCOCO, Google Conceptual Captions) and translate the English captions into 51 other languages using a machine translation system, therefore bootstrapping image-caption pairs in 52 languages including English. The paper is very well written and easy to read.\n\nSome positive points: (1) the method is simple and elegant, and seems to produce strong results compared to a few other baselines; (2) the collected dataset will be released with the research community; (3) parts of the experimental setup were thoughtfully done, e.g. how to create the data splits across languages/images.\n\nSome negative points: (1) the framing of the paper is not adequate, i.e. authors propose a model for \"unsupervised multilingual translation\" but they propose a model for crosslingual retrieval; (2) there are issues with the evaluation of the models, i.e. machine-translated data is used not just at training time but also as validation/test data; (3) parts of the experimental setup could be improved, i.e. a more thorough comparison to crosslingual retrieval models in the recent literature.\n\nI recommend that the paper do not be accepted for publication in its current form for all the reasons mentioned above. I will provide detailed comments on these points below.\n\nMy first comment has to do with the framing of the paper. I am not sure I would call the proposed method one for \"machine translation\". This is not a generative translation model, but rather a retrieval model that retrieves similar sentences in a foreign language. I would certainly add a very big disclaimer in the introduction, if not in the title of the paper, clearly stating this. Machine translation's main issues are due to the fact a model needs to generate text in natural language, which is (1) hard to evaluate due to the difficulty in defining what consists a good translation, (2) the lack/difficulty in having adequate automatic metrics to evaluate MT, (3) the need for reference translations to which to compare hypotheses generated by the model, etc. This paper does not address any of these problems. It is true that the authors show a few examples at the end of Appendix A where they generate translations with GPT-2, but these are utterly secondary experiments, there is no evaluation conducted on the generated translations, etc.\n\nAnother central issue in the paper: Models are trained on datasets where the non-English language sides of the data were all machine-translated from English. Validation and test sets do not seem to be human translated either. To summarize: authors train and evaluate their models on data which is machine translated to begin with. Training on MT'ed data is not an issue necessarily and can often be helpful, i.e., back-translation is an example where training on additional MT'ed source sentences paired with gold-standard target sentences can help. However, you need to control for the quality of your validation and test sets. If they are also machine translated, you are probably introducing a lot of unintended biases (see [1] for a discussion). Note that in addition to evaluating in \"translationese\", these MT'ed data were not even validated by a human, meaning we do not even know if translations used as references in the validation/test sets are correct. This makes the whole evaluation questionable.\n\nAll issues considered, since the method is a crosslingual retrieval model that uses images, the authors should compare to other baselines proposed specifically for crosslingual sentence retrieval. One such model is X-STILTs [2], which performs well on Tatoeba and BUCC, two datasets proposed specifically for crosslingual sentence retrieval. Since these datasets have no associated images, it is not straightforward to evaluate on them with the proposed model. However, authors could retrieve images for sentences in Tatoeba/BUCC using a crossmodal retrieval model, and train their proposed model on the retrieved image-sentence pairs. Since the proposed model is already robust to noisy image-sentence alignments (Section 3.2, last paragraph), perhaps authors could show how it performs compared to strong crosslingual retrieval baselines. I mentioned [2] but there are many more, check the Google XTREME Benchmark for more examples [6].\n\nA few other comments:\n\n\"Machine translation aims to learn a mapping between sentences of different languages while also maintaining the underlying intent.\" -> This is an odd way to define/introduce machine translation (MT). I'd say \"underlying semantics\", not intent.\n\n\"Experiments and visualizations show that the transitive relations through vision provide excellent self-supervision for learning neural machine translation\" -> \"In our experiments and visualizations we show (...)\"\n\nCorrect citation for \"Image pivoting for learning multilingual multimodal representations\" is the peer-reviewed paper \"https://www.aclweb.org/anthology/D17-1303/\". You are also missing the paper \"Sentence-Level Multilingual Multi-modal Embedding for Natural Language Processing\", \"https://www.aclweb.org/anthology/R17-1020/\", which additionally uses learned representations for neural machine translation. Please double-check all your citations that reference pre-prints to make sure there is no peer-reviewed version of a cited pre-print available.\n\nIn your related work, perhaps you could at least mention (the massive amount of) previous work on sentence-image ranking/retrieval, even if these works are not multilingual [3,4,5]?\n\nSection 3.2 \"However, these constraints provide a sparse gradient for learning, which makes large-scale optimization difficult.\" What do you mean by sparse gradients?\n\n\"cross-modal similarity as well as a cross-image similarity\" -> \"sentence-image similarity as well as a image-image similarity\"\n\nYou could improve the mathematical notation greatly. I would mention, for example, the different variables represented by alpha. The variable \\alpha_{ii} is especially badly named, please use more discriminative/clearer variable names. Different indices (i, j, etc.) should index different things.\n\nIf Eq.5 is being maximised, you should not call the quantity being maximised a loss. A loss is always minimised by definition, e.g. negative log-likelihood loss.\n\n[1] M. Zhang and A. Toral (2019). The Effect of Translationese in Machine Translation Test Sets. In: WMT 2019. URL: https://www.aclweb.org/anthology/W19-5208/\n[2] J. Phang et al. (2020). English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too. In: AACL 2020.\n[3] Kiros et al. (2014). Unifying visual-semantic embeddings with multimodal neural language models. In: arxiv.\n[4] Frome et al. (2015). DeViSE: A Deep Visual-Semantic Embedding Model. In: NIPS.\n[5] Faghri et al. (2018). VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In: BMVC 2018.\n[6] Google XTREME Benchmark: https://sites.research.google/xtreme", "extra_scores": {}, "extra_text": {}}, {"review_id": "-mc5nayv3gh", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a framework that leverages visual similarity to align multiple languages, using images as the bridge between them. The cross-modal alignment between language and images is used to estimate and guide the learning of cross-lingual representations. \n\nThe main contribution is to apply the contrastive loss on the cross-modality pre-training. A connection or analysis to InfoNCE should be addressed. The $L_v$ is to distinguish the image and its distortion with other images. The $L_x$ is to distinguish the image and its caption with other pairs. The $L_t$ seems to be a weighted caption level InfoNCE, where weights are calculated from the visual similarity and caption similarity. The $L_c$ is actually the same as masking token loss in BERT, which can also be formulated as mutual information maximization (https://arxiv.org/pdf/1910.08350.pdf). \n\nSince the overall loss function is a linear combination of 4 different losses, I think at least an ablation study is needed to address the importance of each loss function. Especially, I would like to see if $L_v$ or $L_x$ are removed, whether the model performance will drop significantly. \n\nAnother thing I may misunderstand is that the experiments in this paper are claimed as translation, but I think it (sentence level) is more like retrieval. I am also wondering the details of how to evaluate the retrieval task on other generative translation baselines.\n\nSome missing references:\n[1] https://arxiv.org/pdf/2002.02955.pdf\n[2] https://arxiv.org/pdf/1811.11365.pdf\n\nIn summary, this paper presents an interesting topic, but the proposed method is of less novelty and the experimental design needs more improvement. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, all reviewers generally agree that the idea of using visual similarity to unsupervised alignment of multiple languages is interesting and the proposed method and dataset are well-designed, while three of them raised some concerns related to the retrieval nature of the method. In particular,  discussions about its place as a study of machine translation and comparison with other cross-lingual retrieval baselines were the main issues. Although authors made great effort to address reviewers' concerns points and did clarify some of them, unfortunately the reviewers were not fully convinced by the response, and one reviewer decided to downgrade the initial score.  After all, three reviewers rate the paper as 'below the acceptance threshold'. Based on their opinions, I decided to recommend rejection.

I think the entire picture of the work and the logic flow could be much clearer by discussing in a top down manner why this idea should be implemented with a retrieval-based approach, rather than superficially adding "using retrieval" to some sentences.  
Decision: Reject