Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: ___________AFTER THE RESPONSE______________

I would like to sincerely thank the authors for their additional experimental efforts, aiming to provide coherent responses to all my questions and suggestions. Including such experiments into the paper (and adding relevant discussions) would make the paper much stronger. If the focus of the work is on retrieval tasks, I strongly suggest the authors to reflect that fact in the title and also in the introduction, and add discussions focused on why they don't extent their framework to other tasks. The title and the introduction of the paper should be changed to reflect this and avoid 'overselling'. The paper should also make its connections to previous work on adapters more apparent, and clearly extract its core novelty in comparison to other related work in this field (e.g., such as the adapter-based approaches from the xGQA paper). I raise my score from 6 to 7 to acknowledge the wealth of extra experiments conducted for the response, and a very detailed response. However, this increase (as a note to the area chair as well) is conditioned on really incorporating all the promised changes and edits to the final camera-ready paper, if the paper gets accepted.
++++++++++++++++++++++++++++++++++++++++++++


This paper proposes a novel framework for multilingual multimodal learning that: 1) bypasses the need for joint and very expensive multilingual multimodal pretraining (e.g., M3P); and 2) adapts to multilingual setups to parameter-efficient modules (called language acquirers in this work, where these acquirers are basically standard bottleneck adapters). Besides more cost-friendly training of the model, the new framework, termed MultiLingual Acquisition (MLA)*, yields strong results in several cross-modal video-text and image-text retrieval tasks; the underlying model is a standard dual-strema model: CLIP-ViT-B, which is already through pretraining design bound to excel in (monolingual) retrieval tasks. The paper also presents a series of ablation studies and side experiments, covering important aspects such as training strategy, embedding initialisation, simulating low-resource languages and scenarios, zero-shot evaluations (on unseen languages) and amount of training data. There are additional qualitative and quantitative analyses available in the appendix.

*Minor: I would suggest a more precise name for the framework (that also includes multi-modality somewhere)
strengths_and_weaknesses: Strengths:
=========
- The motivation for the work is very clean and clearly stated. The core methodology is soundly described, and the paper is very easy to follow in general.
- The results in the standard cross-modal retrieval tasks are consistently positive. 
- The ablation study and additional experiments are also very well motivated and very comprehensive, and offer the readers many interesting analyses and points of discussion.
- The methodology, along with the core hypotheses, is very sound, and the experimental setup aligns well with the methodology.

Weaknesses:
==========
- The current MLA framework is tested and seems to work only on retrieval tasks. A more comprehensive analysis on other, structurall different multilingual multi-modal tasks would make the main findings (and consequently, the impact of the work) much stronger. I strongly suggest to extend the paper to more underlying models (also single-stream models), and evaluate on a carefully designed recent IGLUE benchmark https://arxiv.org/pdf/2201.11732.pdf). The paper should convince the reader that MLA is not intended only for retrieval tasks (or the other way around - if it doesn't work as well for other tasks, to narrow the scope of the work to retrieval only in the title and in the introduction).
- One of the core components of the work are 'language acquirers': they are basically standard language adapters, examined to detail in the body of work on cross-lingual text-only transfer (e.g., see again the cited MAD-X work from Pfeiffer et al.). Having said that, the idea of 'language acquirers' is basically also very related to another work of Pfeiffer et al. (cited in the paper as "[28] Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan-Martin O Steitz, Stefan Roth, Ivan Vulić, and Iryna Gurevych. xgqa: Cross-lingual visual question answering. arXiv e-prints, 2021."). More discussion is needed here so that the reader really understands the connections to language adapters and connections to the adaptation work done on multilingual multi-modal work in previous work. In my understanding, the idea is basically very similar, with two core differences: a) MLA is applied on a dual-stream model which is suitable for retrieval tasks but might not be suitable for e.g. VQA tasks; b) Pfeiffer et al., relied on MLM-ing for adaptation, while MLA relies on specific bilingual information to learn transformations from the target language to the language of the monolingual VLP model (i.e., English).
- There is no comparison with standard translation-based baselines. A very basic question: why would one want to go through the MLA procedure if translation-based approaches can work better? Some comparison numbers must be provided.
- The chosen retrieval datasets might paint a wrong picture about the difficulty of the task: I would suggest the authors again to check the IGLUE benchmark and also to evaluate on the recently proposed WIT dataset. IGLUE also covers some lower-resource languages and its tasks are generally more difficult than the standard overly optimistic evaluations on Multi30K, and MSCOCO-1k and 5k.
questions: - This is not a criticism but more a suggestion: conducting some experiments with other underlying models beyond CLIP (and/or CLIP-style models of different sizes) would offer other useful comparisons and points of reference for future work.

- It would also be interested to try out language acquirers with pretrained M-VLP models like M3P. Can MLA help M3P to adapt/specialise for a particular source/target language? This would be a very useful addition to the paper.

- Minor: I suggest to cite papers based on their publications in conference proceedings rather than arxiv versions.
limitations: The statement in limitations of learning 'coarse-grained' versus 'fine-grained' representations and limitations seems quite hand-wavy to me - what is meant by this exactly? I would also suggest to be more open with other limitations of the work (e.g., some of them are mentioned in this review - (i) there weren't any real low-resource languages in evaluation; those scenarios were mostly simulated; (ii) application and evaluation is currently tied only to retrieval tasks; (iii) there's no discussion of potential avenues for future work, etc.)
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper try to solve the problem that existing vision-language pre-trained models require huge computing resources and cannot be flexibly extended to new languages. They mainly propose a two-stage training strategy to optimize the language acquisition encoder, namely the Native Language Transfer stage and the Language Exposure stage. The authors compare their model to different baselines on multilingual image-text and video-text retrieval benchmarks, and find that their model compares well to the current state-of-the-art.

strengths_and_weaknesses: The paper main issue is the lack of clarity at certain crucial points, especially in Section 3 (the model description). The maths is at times too convoluted and should be simplified and improved. Moreover, the major concern I have about this paper is the experimental results. The work is relatively novel, although a deeper discussion of why the specific architecture used was not done and would have been important to
understand the contributions.

questions: 1.	Why use the average score over Recall@1, Recall@5, and Recall@10 as your evaluations? What's the idea behind it, and not Recall@1, Recall@5, and Recall@10, or something else?
2.	I want to know whether the Language Acquier is shared in different languages? This is very important to evaluate the size of parameters of model.
3.	I want to know the training strategy. Whether different languages are trained in parallel or serial ways? For a language, whether the model only is trained on this language? And, the model have not the multilingual ability?
4.	The comparison of Table 2 is very unfair. The baselines are on trained on many different languages at the same time. However, the model of this paper is only trained on a language for some lingual applications and the baselines are suitable for different languages. Even though the model are trained on some languages as the papers, the trained number of languages is less than M3P.
5.	The description of the model in Section 3 needs a lot of improvement. It is confusing and many times convoluted. There are too many variables being introduced.
6.	It is not clear whether you do not introduce the equations for the enumerated items, or whether you do not use the enumerated items in your model architecture. Please clarify.
7.	The paper writing is poor. I found a lot of grammar mistakes in the paper which need to be corrected.

limitations: yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a Multilingual Acquisition (MLA) framework that leverages vision and language modality at a low cost. MLA empowers monolingual VLPs with multilingual capability using much less training data through a language acquisition encoder. The authors apply the proposed method to multilingual image-text and video-text retrieval benchmarks with much less training data and parameters. In the training process, they present a two-stage training strategy to train the language acquisition encoder: 1) Native Language Transfer (NLT) stage, the model is trained to learn the connection between the native and new language, then followed by 2) Language Exposure (LE) stage, the model to learn the alignment between new languages and images.
strengths_and_weaknesses: Strengths:
- A simple two-step approach to acquire a new language by leveraging monolingual VLP by aligning the languages and modalities. The approach is straightforward and effective.
- This is a solid paper; it has comprehensive results and analysis
- The proposed approach has strong zero-shot and few-shot capability,

Weaknesses:
- Didn't find any substantial weakness to report.

Typo:
- MultiLingual => Multilingual. It is not common to write multilingual with a capital L. Another way to write it is Multi-lingual.
questions: - I was wondering why did you only choose English as the source language for the zero-shot setting other than English is the high-resource language. Have you investigated the zero-shot setting where English is not the source/pivot language? It would be interesting to find out whether other languages? Many of these evaluated languages are not closely related to English (e.g., Japanese, Chinese).
limitations: Yes, the authors address the limitations.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: ### After author response
Thanks authors for running additional experiments, including implementing MURAL. I raise the score to 6.
I think there's a merit in adapting monolingual models to multilingual ones, the approach in the paper is good, but it's overselling. I recommend the authors to reword a bit, saying what is reflected in the results. For instance, in the response the authors said MLA doesn't suffer from the curse of multilinguality based on a result of *only* 11 languages. I don't think you can extrapolate that to hundreds of languages. 

-----------------

### Summary 
The paper proposes a way to adapt English Vision-Language Pre-training (VLP) to other languages via two stages: Native Language Transfer stage and the Language Exposure stage.  In Native Language Transfer state, parallel data is utilized to adapt language encoder to the new languages using language acquisition encoder, a combination of embeddings for new languages and adapter.. In the Language Exposure stage, only Language Acquisition encoder is trained to match with images via contrastive loss. In the text-image retrieval evaluation, MLA shows strong performances compared to other multilingual VLP. The main contribution of the paper is to propose a lightweight approach turning monolingual VLP into multilingual one.

strengths_and_weaknesses: ### Strengths
- A lightweight approach for making monolingual VLP multilingual.
- Evaluation on zero-shot, finetuning English and fine-tuning all shows substantial gains over previous work on multilingual VLP (e.g., MURAL, MKD, …)
- On multilingual video-text retrieval, MLACLIP outperformed XLM-R MMP significantly.	


### Weaknesses
- Adapting monolingual language pretrained models into multilingual ones has been studied extensively in the literature. For instance, an efficient method was proposed in [[1](https://arxiv.org/abs/2002.07306)] which uses only 1 gpu. I feel that the novelty aspect is a bit short in adapting monolingual VLP to multilingual one. Nevertheless, this is not a major weakness of the paper.
- My main concern is the comparison with previous work. MLA adapts CLIP-ViT-B-32 and CLIP-ViT-B-16 to 4 languages (de, fr, cs, ja) in multilingual image-text retrieval. While MLA is compared with MURAL, it is worth pointing out that MURAL supports 109 languages. MURAL uses EOBT that consists of translations of 109 language pairs and Alt-Text (image-text pairs) covers 110 languages. Thus comparing MLA to MURAL is somewhat comparing an apple to orange. The curse of mulitlinguality tells us that adding more languages into the mix would degenerate the performance of high resource languages, a well known problem in multilingual machine translation. In my opinion, a fairer baseline would be the MURAL system but it is trained on 5 languages similar to MLA. In line 273, when the authors compare MLA to MURAL, note that MURAL has more parameters but they are mostly the embeddings of the rest 104 languages, which will never be updated during fine-tuning. So having more parameters doesn’t mean that it is easier for fine-tuning.
- On computational cost, Table 2 shows that MLA need 1 V100 GPU and 0.5 day to train. This is rather optimistic since it uses pre-trained CLIP-ViT-B-32. I think the cost of training the CLIP model should be added into the comparison. 
- While conceptually MLA can support more languages, as I mentioned earlier, adding more languages could lead to performance drop on high resource languages. So the aspect of language extensibility remains to be questioned. Additionally, MLA requires translation of image captions to train the language acquisition encoder. This is problematic because it depends on machine translation systems, which often support a certain number of languages. 

questions: ### Questions:
- For a low resource scenario (Table 6), since MURAL supports 110 languages, what is the performance of MURAL?
- What is the training size of translation data CC6L and CC69L?
- If you train your dual stream model from scratch for 5 languages in Multilingual image-text retrieval experiment, how long does it take? and what would be the final performance?

### Suggestions:
- I think the analogy to how human learning and human learning habit just makes the paper more difficult to read. From technical perspective, you want to retain as many as pre-trained parameters as possible, so you adjust the parameters of new languages. It makes sense. Maybe just say that.
limitations: Yes.
review_title: null

Meta-review: metareview: All reviewers are positive to this paper. The authors also respond actively to address the problems raised by the reviewers. I recommend acceptance. One thing is that reviewer 4qbz has pointed out the authors should be responsible to take the comments into consideration and 
do necessary changes in the paper, such as adjusting the title and the introduction and adding connections to previous work. 
confidence: Less certain
Decision: Accept