Adaptation of Large Vision-Language Models for non-English Languages
Authors: Elio Musacchio, Lucia Siciliani, Pierpaolo Basile, Giovanni Semeraro


Reviewer paper summaries: The authors present their approach of adapting LVLM for non-English languages, including data translation, instruction formatting, language adaptation and multimodal adaptation, and show experiment results against multiple multilingual VL task datasets.

Brief Summary: The paper highlights that Large Vision Language Models have been primarily trained on english data and thus not directly usable for other languages. To address this issue the authors propose a pipeline consisting of first data translation, then instruction formatting, followed by language adaptation and multimodal adaptation. 

Experiments are conducted to see if the models can answer in their main language as well is in their adapted language. To this end, a number of VQA-like datasets are used for evaluation.
Summary of strengths: New approach of adapting LVLM for non-English languages

Pros:

1. The paper is well motivated given that multimodal data encompasses multiple languages. 

2. The authors provide results on a number of datasets and models in Table 1 and further experiments for closed and open ended tasks. 

3. The authors provide the hyper-parameters to reproduce their results in appendix.
Summary of weaknesses: - Methodology is too abstract
- The requirements for adapting the proposed method to a new language are not specified. Here are the resources mentioned in the paper, which might all be required for the adaptation. Too many requirements.
  - Data translation: human translator, machine translation system, manual selection of text data for machine translation 
  - Instruction formatting: manual writing of instruction format pairs, instruction tuning data (English, non-English language)
  - Language adaptation: baseline LLM (English, non-English language), target language corpus
  - Multimodal adaptation: baseline LVLM (English, non-English language), multimodal data
- No comparison with baselines. No ablation study.

Cons:

1. I am still unclear about the main takeaway of the paper. It seems to suggest that the proposed pipeline works for using Vision-Language model in another language works?

2. The authors should compare alternative pipelines such as just translating the entire dataset to another language, and not making any other changes. 

3. The paper is more of an experimental paper with not much novelty. But I am not sure if the takeaways are interesting enough.
Comments, suggestions and typos: - Line 113: vision-language train from scratch -> vision-language training from scratch
- Line 216: Therefore, It is -> Therefore, it is

The tables are very hard to decipher. For instance, in table 2,3 instead of writing LLAMA in each row, having a column of the type of model in LLAMA, CLIP, DS would be much more readable.
Overall assessment scores: 1.5

2