Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose to leverage images to train an unsupervised machine translation (MT) model. Their main idea is that the similarity of images can be used as a proxy for the similarity of sentences describing the images. The sentences, in turn, can be in different languages, and knowledge about their similarity can be exploited as training signal for an unsupervised MT model, i.e., training without parallel sentences. Their model consists of a sentence encoder and an image encoder. For training and evaluation of the model, they use translations (multi-way for the test set) of image captioning datasets.


The authors evaluate on two different tasks: word-level translation and sentence-level translation. (However, the sentence-level translation is retrieval-based, i.e., no sentences are being generated.) They compare to multiple baselines, which seem to be chosen well. (The one that I would be consider missing is mBart (see below for the reference), but that's hard to train on many common GPUs, so I wouldn't necessarily expect that.)

Overall, the idea the authors are presenting is convincing, the experiments are clear and well designed, and the model makes a lot of sense. I don't see any major shortcomings of this paper. Thus, I would be excited to see it being presented at ICLR.

A minor shortcoming is that mBart isn't mentioned anywhere. In fact, the authors claim that they propose the first multilingual unsupervised model (lines 3 and 4 in the abstract). This should be corrected. The reference for mBart is: Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation.
review_title: Great Idea for Unsupervised Machine Translation

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper introduces a framework that leverages visual similarity to align multiple languages, using images as the bridge between them. The cross-modal alignment between language and images is used to estimate and guide the learning of cross-lingual representations. 

The main contribution is to apply the contrastive loss on the cross-modality pre-training. A connection or analysis to InfoNCE should be addressed. The $L_v$ is to distinguish the image and its distortion with other images. The $L_x$ is to distinguish the image and its caption with other pairs. The $L_t$ seems to be a weighted caption level InfoNCE, where weights are calculated from the visual similarity and caption similarity. The $L_c$ is actually the same as masking token loss in BERT, which can also be formulated as mutual information maximization (https://arxiv.org/pdf/1910.08350.pdf). 

Since the overall loss function is a linear combination of 4 different losses, I think at least an ablation study is needed to address the importance of each loss function. Especially, I would like to see if $L_v$ or $L_x$ are removed, whether the model performance will drop significantly. 

Another thing I may misunderstand is that the experiments in this paper are claimed as translation, but I think it (sentence level) is more like retrieval. I am also wondering the details of how to evaluate the retrieval task on other generative translation baselines.

Some missing references:
[1] https://arxiv.org/pdf/2002.02955.pdf
[2] https://arxiv.org/pdf/1811.11365.pdf

In summary, this paper presents an interesting topic, but the proposed method is of less novelty and the experimental design needs more improvement. 
review_title: Interesting topic but need some improvements

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
This paper introduces a framework that leverages transitive similarities between images and text to align multiple languages, without bi-lingual parallel training corpora. Unlike most existing unsupervised multi-modal translation works that only focus on a single (or a few) language pairs, this work proposes a one-for-many framework that can be easily applied to cover 52 languages at once.

The paper is well-written and easy to follow, and the notation is clean. The experiments are convincing with interesting case studies.

The major issue I want to raise is:

(1)The methodology proposed is retrieval-based (which is popular at the time of SMT). One shortcoming of retrieval-based methods is that its applicability is limited by the size of corpora. In terms of unsupervised translation, this problem is even more severe. Because the goal of unsupervised MT is to extend the successes of MT to low-resource language pairs. I would like to hear from the authors how we can improve or alleviate this issue. 

Some other minor concerns are that:

(2)The authors evaluate word-level translation based on ground truth obtained using statistical patterns. This somehow makes the comparison with generative models (e.g. Sigurdsson et al 2020) unfair.

(3)A missing and important related work. Unsupervised Multi-modal Neural Machine Translation. Yuanhang Su et al 2020

review_title: Interesting idea and extensive experiments, with new dataset introduced that will benefit the community. 

final_score: rating: 5: Marginally below acceptance threshold
review_content: 
The authors propose a method for crosslingual sentence retrieval that uses images to ground sentences in different languages and to project these sentences into a meaningful semantic space. The data used in the model are image-caption pairs in each language, and never parallel sentences in two or more languages. Authors use existing English-language image captioning datasets (Flickr30k, MSCOCO, Google Conceptual Captions) and translate the English captions into 51 other languages using a machine translation system, therefore bootstrapping image-caption pairs in 52 languages including English. The paper is very well written and easy to read.

Some positive points: (1) the method is simple and elegant, and seems to produce strong results compared to a few other baselines; (2) the collected dataset will be released with the research community; (3) parts of the experimental setup were thoughtfully done, e.g. how to create the data splits across languages/images.

Some negative points: (1) the framing of the paper is not adequate, i.e. authors propose a model for "unsupervised multilingual translation" but they propose a model for crosslingual retrieval; (2) there are issues with the evaluation of the models, i.e. machine-translated data is used not just at training time but also as validation/test data; (3) parts of the experimental setup could be improved, i.e. a more thorough comparison to crosslingual retrieval models in the recent literature.

I recommend that the paper do not be accepted for publication in its current form for all the reasons mentioned above. I will provide detailed comments on these points below.

My first comment has to do with the framing of the paper. I am not sure I would call the proposed method one for "machine translation". This is not a generative translation model, but rather a retrieval model that retrieves similar sentences in a foreign language. I would certainly add a very big disclaimer in the introduction, if not in the title of the paper, clearly stating this. Machine translation's main issues are due to the fact a model needs to generate text in natural language, which is (1) hard to evaluate due to the difficulty in defining what consists a good translation, (2) the lack/difficulty in having adequate automatic metrics to evaluate MT, (3) the need for reference translations to which to compare hypotheses generated by the model, etc. This paper does not address any of these problems. It is true that the authors show a few examples at the end of Appendix A where they generate translations with GPT-2, but these are utterly secondary experiments, there is no evaluation conducted on the generated translations, etc.

Another central issue in the paper: Models are trained on datasets where the non-English language sides of the data were all machine-translated from English. Validation and test sets do not seem to be human translated either. To summarize: authors train and evaluate their models on data which is machine translated to begin with. Training on MT'ed data is not an issue necessarily and can often be helpful, i.e., back-translation is an example where training on additional MT'ed source sentences paired with gold-standard target sentences can help. However, you need to control for the quality of your validation and test sets. If they are also machine translated, you are probably introducing a lot of unintended biases (see [1] for a discussion). Note that in addition to evaluating in "translationese", these MT'ed data were not even validated by a human, meaning we do not even know if translations used as references in the validation/test sets are correct. This makes the whole evaluation questionable.

All issues considered, since the method is a crosslingual retrieval model that uses images, the authors should compare to other baselines proposed specifically for crosslingual sentence retrieval. One such model is X-STILTs [2], which performs well on Tatoeba and BUCC, two datasets proposed specifically for crosslingual sentence retrieval. Since these datasets have no associated images, it is not straightforward to evaluate on them with the proposed model. However, authors could retrieve images for sentences in Tatoeba/BUCC using a crossmodal retrieval model, and train their proposed model on the retrieved image-sentence pairs. Since the proposed model is already robust to noisy image-sentence alignments (Section 3.2, last paragraph), perhaps authors could show how it performs compared to strong crosslingual retrieval baselines. I mentioned [2] but there are many more, check the Google XTREME Benchmark for more examples [6].

A few other comments:

"Machine translation aims to learn a mapping between sentences of different languages while also maintaining the underlying intent." -> This is an odd way to define/introduce machine translation (MT). I'd say "underlying semantics", not intent.

"Experiments and visualizations show that the transitive relations through vision provide excellent self-supervision for learning neural machine translation" -> "In our experiments and visualizations we show (...)"

Correct citation for "Image pivoting for learning multilingual multimodal representations" is the peer-reviewed paper "https://www.aclweb.org/anthology/D17-1303/". You are also missing the paper "Sentence-Level Multilingual Multi-modal Embedding for Natural Language Processing", "https://www.aclweb.org/anthology/R17-1020/", which additionally uses learned representations for neural machine translation. Please double-check all your citations that reference pre-prints to make sure there is no peer-reviewed version of a cited pre-print available.

In your related work, perhaps you could at least mention (the massive amount of) previous work on sentence-image ranking/retrieval, even if these works are not multilingual [3,4,5]?

Section 3.2 "However, these constraints provide a sparse gradient for learning, which makes large-scale optimization difficult." What do you mean by sparse gradients?

"cross-modal similarity as well as a cross-image similarity" -> "sentence-image similarity as well as a image-image similarity"

You could improve the mathematical notation greatly. I would mention, for example, the different variables represented by alpha. The variable \alpha_{ii} is especially badly named, please use more discriminative/clearer variable names. Different indices (i, j, etc.) should index different things.

If Eq.5 is being maximised, you should not call the quantity being maximised a loss. A loss is always minimised by definition, e.g. negative log-likelihood loss.

[1] M. Zhang and A. Toral (2019). The Effect of Translationese in Machine Translation Test Sets. In: WMT 2019. URL: https://www.aclweb.org/anthology/W19-5208/
[2] J. Phang et al. (2020). English Intermediate-Task Training Improves Zero-Shot Cross-Lingual Transfer Too. In: AACL 2020.
[3] Kiros et al. (2014). Unifying visual-semantic embeddings with multimodal neural language models. In: arxiv.
[4] Frome et al. (2015). DeViSE: A Deep Visual-Semantic Embedding Model. In: NIPS.
[5] Faghri et al. (2018). VSE++: Improving Visual-Semantic Embeddings with Hard Negatives. In: BMVC 2018.
[6] Google XTREME Benchmark: https://sites.research.google/xtreme
review_title: Interesting idea and nice empirical results but paper framing and experimental setup could be improved.

Meta-review: Overall, all reviewers generally agree that the idea of using visual similarity to unsupervised alignment of multiple languages is interesting and the proposed method and dataset are well-designed, while three of them raised some concerns related to the retrieval nature of the method. In particular,  discussions about its place as a study of machine translation and comparison with other cross-lingual retrieval baselines were the main issues. Although authors made great effort to address reviewers' concerns points and did clarify some of them, unfortunately the reviewers were not fully convinced by the response, and one reviewer decided to downgrade the initial score.  After all, three reviewers rate the paper as 'below the acceptance threshold'. Based on their opinions, I decided to recommend rejection.

I think the entire picture of the work and the logic flow could be much clearer by discussing in a top down manner why this idea should be implemented with a retrieval-based approach, rather than superficially adding "using retrieval" to some sentences.  
Decision: Reject