ICU: Conquering Language Barriers in Vision-and-Language Modeling by Dividing the Tasks into Image Captioning and Language Understanding
Authors: Guojun Wu
Venue: EMNLP 2023 Findings (2023)
Keywords: Cross-lingual Language Understanding
Multimodality

Abstract: Most multilingual vision-and-language (V\&L) research aims to accomplish multilingual and multimodal capabilities within one model. However, the scarcity of multilingual captions for images has hindered the development. To overcome this obstacle, we propose ICU, Image Caption Understanding, which divides a V\&L task into two stages: a V\&L model performs image captioning in English, and a multilingual language model (mLM), in turn, takes the caption as the alt text and performs cross-lingual language understanding. The burden of multilingual processing is lifted off V\&L model and placed on mLM. Since the multilingual text data is relatively of higher abundance and quality, ICU can facilitate the conquering of language barriers for V\&L models. In experiments on two tasks across 9 languages in the IGLUE benchmark, we show that ICU can achieve new state-of-the-art results for five languages, and comparable results for the rest.

ICU: Conquering Language Barriers in Vision-and-Languuae Modeling by Dividing the Tasks into Image Captioning and Language Understanding
Guojun Wu
Department of Computational Linguistics, University of Zurich wuuuuh...wuuz.cch
Abstract
catail
XVNLI
1. 9y 6 putaoo in qulc
Most multilingual vision-and aan-language (V&L) research aims to accomplish multilingual and multimodal capabilities within one model. However, the scarcity multilingual captions for images has hindered the development. To overcome this obstacle, we propose ICU', Im- age Caption Understanding, which divides V&L task into two stages: V&L model per- forms image captioning in English, and mul- tilingual language model (mLM), in turn, takes the caption as the alt text and performs cross- lingual language understanding. The burden of multilingual processing is lifted off V&L model and placed on mI.M. Since the multilin- gual text data is relatively of higher abundance and quality, ICU can facilitate the conquering language barriers for V&L models. ex- periments on tasks the IGLUE benchmark, we show that ICU can achieve new state-of of-the-art results for five lan- guages, and comparable results for the rest.
contradicr
Une pomme de terre esi assez heureuse Appua calee impps!
Bpean nerser
IC
<<Iiggish Caption>>> A poor polalo is sighing on the table
unlail
XNLI
+ fe
peldocis gale
<<alt text>>>
contradict
Une pomme de terre est assez heureuse
poor potato sighing on the neutral
Вpени леrии (Tine Ital
9 languages in
Figure I: We employ XVNLI as case study to exem-
plify the partitioning of the original task into two stages. The original task comprises image premise and text hypothesis, which we display at the top. Below it, we present the two stages: image captioning (IC) and cross- lingual natural language inference (XNLI). The English tramslations of the text are provided within the brackets.
1 Introduction
In recent times, there has been growing interest in extending the success of vision-and-language (V&L) models beyond English encompass non- English languages. However, the scarcity of train- ing data has posed challenges in the development of multilingual models. To address this issue, various code-switch strategies (Ni et al., 2021; Nooralahzadeh and Sennrich, 2022) have been proposed to encourage models to learn the rela- tionships between corresponding words in differ- ent languages. Additionally, machine translation (MT) techniques have been employed to augment existing English-only datasets (Qiu et al., 2022: Zhou et al., 2021). Although some improvements have been achieved using MT-enhanced translated data, the quality of translations varies across lan- guages. Furthermore, fine-tuning strategies (Liu
et al., 2023; Nooralahzadeh and Sennrich, 2022) have been explored to enhance cross-lingual gen- eralization. However, there still exists significant performance gap between English and other lan- guages, highlighting the challenge of scarcity.
To address these challenges, this paper intro- duces ICU (Image Caption Understanding), which approaches V&L tasks by dividing them into two stages: image captioning (IC) and cross- lingull language understanding (XLU). As depicted in Fig- ure 1, guage Inference (XVNLI) task an example. Ini- tially, we employ a V&L model perform IC and generate an English caption for the image. This caption is then treated as the alt text for the im- age, enabling cross mooss-lingual natural language infer-
the Cross-lingust Visual Natural Lan-
Cooee to reproduce our results is available at https:// gi thub. com/ gjuubyron/ICUU

---

Frame
Template
0
< cleft_ccptton > < {right__cptions _ee_ptpptoo
2
Left: {left_ caption). Right: {right_ Bright_caption;
{left_caption)
3
Left: < ctallll__ppoo]]]_ >. Right: < (right_caption) >
4 There are peeeft_copion in the left image and tright_caption, in the right image.
5 The left image shows (Te (eef___aaoion while the right image shows tright_coption)
Table I: Hand-crafted templates. We use direct caption concatenation in FrameO. For clarity, enclose each caption in angle brackets in Framel and Frame3 as alt text. We also indicate their positions Frame2 and Frame3. Moreover, we seamlessly integrate these captions into detailed descriptions in Frame4 and Frame5.
ence (XNLI) using a multilingual language model (mLM). ICU leverages the strengths of both the V&L model and the mLM. Given that multilingual text data are relatively more abundant and of higher quality, ICU helps alleviate the scarcity problem. In this study, assess our approach using two tasks from IGLUE: XVNLI and MaRVL(Liu et al., 2021), a Multicultural Reasoning over Vision and Language dataset. Our findings indicate that ICU, even in zero-shot scenarios, achieves remarkable performance on both tasks. Additionally, we ob- serve that employing few-shot learning techniques for XVNLI further enhances the model's perfor- mance. Moreover, we explore frame engineering techniques, wherein we assign captions to differ- ent frames (refer Table for more details), and demonstrate that the model exhibits sensitivity to different frames when applied to MaRVI.
studied. However, for MaRVI, the model needs to determine whether textual description is true or false about pair of images. In this case, the adap- tation is achieved by assigning the two captions to different frames, illustrated in Table 1. We then treat the task zero-shot text classification (Yin al., 2019). Another challenge encountered in ICU is the mLM's s handling of code-switching. such as when the premise in English while the hypothesis is in other languages. Remarkably, we demonstrate that the mLM already achieves good performance in zero-shot scenarios, and the perfor- mance can be further improved through few-shot learning.
Experiments
In this section, will provide comprehensive description of the models employed in ICU, along with the experimental settings and evaluations con- ducted.
Our contributions are summarized as follows:
We introduce ICU, an innovative divide-and- conquer approach designed to address the challenges posed by multilingual vision-and- language tasks.
3.1 Models for ICU
We use two pre-existing models for utilization in the ICU setting. For the cross-modal part, we employ OFA (Wang et al., 2022b), sequence-to- sequence vision-and-languse cally, we select OF Aarger which has undergone fine-tuning on COCO (Lin et al., 2015). a substan- tial dataset for image captioning. For decoding. OF A Large employs beam search with a beam size of five, while incorporating a constraint of main- taining n- n-gram diversity within a context window of three. For the cross-lingual part, we use mDe- BERTaV3 Base (He et al., 2021), which achieves new state-of-the-art on XNLI (Conneau al., 2018) ()18) across 15 languages after fine-tuning, As the model is fine-tuned in a monolingual fashion (Lau- rer et al., 2023), meaning both the premises and hypotheses are in the same language, we continue
We achieve state-of-the-art results in two tasks from IGLUE benchmark, outperforming the existing multilingual methods in several lan- guages.
framework. Specifi-
We conduct experiments and analysis ex- plore efficient and computationally cheap ways further boost performance.
2 ICU: Image Caption Understanding
In this section, will discuss the challenges posed by the implementation of ICU. Firstly, crucial task to adapt the second stage (XI.U) to a suit- able NLP task. For XVNLI, this can be easily addressed since NLI has already been extensively

---

XVNLI
MaRVL
Model
ARB
SPA
FRA
RUS
avg
IND
SWA
TAM
TUR
CMN
aVg
MUNITER
46.73
56.96
59.36
51.72
53.69
54.79
51.17
52.66
54.66
55.34
53.72
XUNITER
51.98
58.94
63.32
59.71
58.49
55.14
55.51
53.06
56.19
53.06
54.59
UC
56.19
57.47
69.67
64.86
62.05
56.74
52.62
60.47
56.70
59.88
57.28
M3P
55.24
58.85
56.36
62.54
58.25
56.47
55.69
56.04
56.78
55.04
56.00
ICU
58.00
61.04
63.21
61.39
60.91
56.91
55.60
57.89
58.31
56.92
57.13
Table Zero-shot accuracy on XVNLI and MaRVI. The results of four models in the middle row are directly copied from IGLUE to enable comparison. The best performance is denoted by highlighting it in bold. (Since frame engineering is also zero-shot, we choose the best one among the frames)
categorize it as zero-shot application within our approach.
guages with machine translation, thereby enhanc- ing its multilingual capabilitiess XUNITER, M and UC² all have their initializations derived from XLM-R (Conneau et al., 2020), while MUNITER is initialized from mBERT (Devlin et al., 2019).
3.2 Few-shot Learning Setup
As the IGLUE benchmark does offer com- prehensive few-shot data for MaRVL, our few- shot learning efforts are solely focused on XVNLI. When conducting few-shot learning, we freeze the V&L model and exclusively adjust the parameters of the mLM. The process of freezing the V&L model can make it more efficient by enabling the reuse of captions and leveraging the significantly smaller mLM compared to the standard V&L mod- els. Given the scarcity of few-shot data, we re- frain from engaging hyperparameter optimiza- tion, which, while potentially arbitrary, serves the purpose of safeguarding the model from overfit- ting on such a limited dataset. We choose to use smaller batch size of 8, increase the learning rate to le-4, and limit the training to just 3 epochs. The rest hyperparameters remain the same the fine- tuning configurations of mDeBERTTV3 V3 (He al., 2021). Few-shot learning is performed separately for each language.
These models also differ in size, with MUNITER at 185M, xUNITER at 284M, UC² at 282M, and M at 377M. In contrast, the mDeBERTaV3 Base used in our approach is of smaller size at 86M.
3,4 Tasks
We assess our method through two tasks. The first task, XVNLI, involves conducting inference in a multi-lingual scenario based on the image premise and text hypothesis. It comprises 357 images and 1. l. samples across 4 languages. On the other hand, MaRVL focuses on determining the truthfulness of grounded statements regarding pairs of images. It encompasses 4.9k images and 5.7k samples across 5 languages.
Results and Analysis
In this section, we present the results of ICU in comparison to existing works within the IGLUE benchmark. Additionally, analyze the impact of few-shot learning and frame engineering tech- niques on the performance of ICU.
3.3 Baseline Models
The models in the baseline are all initialized from mLMs, and further trained with multiple objec- tives to learn multimodal representations. mU- NITER and XUNITER (Liu et al., 2021) expand the UNITER (Chen et al., 2020) architecture to en- compass multiple languages. MDP (Ni et al., 2021) additionally introduces training tasks that involve code-switching in a multimodal context, where En- glish caption words are randomly substituted with translations using a specific probability. UC² (Zhou et al., 2021) acquires data in five different lan-
4.1 Overall Results
The zero-shot results for XVNLI and MaRVL are displayed in Table 2. Among the nine languages, ICU achieves the state-of-the-art (SOTA) perfor- mance four languages, while maintaining com- parable performance in the remaining languages. However, on average, slightly lags behind the current SOTA in the IGLUE benchmark.

---

4.3 Frame Engineering
Figure 3 depicts the performance across different frames in MaRVL. Our findings indicate that em- ploying simple and concise frame generally yields better results. Conversely, incorporating lengthy around the captions does not lead improved performance.
Acchincy
A3B
5 Related Work
RUS
Image-to-text Transformation Vision-and- language Modeling TRiG (Gao et al., 2022) and PICa (Yang et al., 2022) are two prior studies that engage in image-to-text transformation as a solu- tion for addressing multimodal challenges in the context of visual question answering tasks. TRiG utilizes three types of transformations, encompass- ing image captioning, dense labeling, and optical character recognition. On the other hand, PICa employs a variety of image captioning models and tagging models to perform image transformations.
Shot
Figure 2: ICU performance across different shots XVNLL In our evaluation, define one image as one shot, although typically an image may be utilized multiple samples. On average, each shot comprises three samples.
Model
ARB
SPA
FRA
RUS
avg
MUNITER
46.91
57.73
59.36
51.80
53.95
Nevertheless, their efforts are concentrated exclu- sively the English language.
XUNITER
54.04
60.22
64.52
63.40
60.55
UC2
56.87
62.80
69.76
65.29
63.68
Dision-and-language Models Large-scale pre- training has become the cornerstone of vision-and language (V&L) research. Recent advancements have seen the development of big foundation mod- els like SimVLM, Flamingo, and GIT (Wang et al.. 2022c, Alayrac et al., 2022, Wang et al., 2022a). These models rely on training with sufficiently large datasets, typically constructed using image- text pairs obtained from web crawling, such the 400 million pairs used in CLIP (Radford al., 2021). However, due to the predominance of En- glish in the training data, these models face chal- lenges in effectively handling non-English inputs. Multilingual Language Models The success of models like mBERT (Devlin et al., 2019) and XLM (Conneau and Lample, 2019) has demon- strated that large-scale pretraining of Transform- ers across multiple languages can yield impres- sive results cross-lingual language understanding (XLU). With the addition more languages and increased training data, XLM-R (Conneau et al., 2020) has surpassed mBERT's performance on var- ious XLU benchmarks. Notably, mDeBERTaV3 (He et al., 2021) has recently achieved tfte-of-the- results on XNLI, attaining zero-shot cross- lingual accuracy of 79.8%. However, it is crucial to note that these models are primarily trained for NLP tasks and may not possess the capability to handle multimodal tasks involving both vision and
56.01
60.40
58.59
62.46
59.37
ICU
60.70
64.61
62.61
65.57
63.37
Table 3: Max-shot XVNLI results. This evaluation conducted under the max-shot setting, encompassing a total of 48 shots.
4.2 Few-shot Learning
Figure 2 illustrates the performance variations in XVNLI as the number of shots increases. Overall, consistent upward trend can be observed, indi- cating improvement in performance. Nonethe- less, we observe that when the number of shots is fewer than ten, the model's performance is in- ferior to that of zero-shot. We hypothesize that in situations with a limited number of shots, the tuning process may result in a model with reduced generality. It's only with an adequate number of shots that the model can truly achieve noteworthy performance enhancements. Furthermore, Table provides comparison of the maximum shot perfor- mance, where ICU demonstrates slight advantage over the previous SOTA approach in an additional language and successfully closes the performance gap on average.

---

Figure 3: ICU performance across different frames on MaRVL. We conduct evaluation using all the frames listed Table
language.
frame engineering techniques in enhancing perfor- mance. Our experimental results demonstrate the efficacy of recycling existing models, achieving state-of-the-art performance. Overall, ICU presents promising solution for multilingual V&L tasks and opens up avenues for future research.
Multilingual Vision-and-language Models To facilitate the learning of universal representations across different modalities and multilingual texts, the M framework (Ni et al., 2021) was intro- duced as the first pre-training framework that op- timizes multiple pre-training objectives. Another unified framework, UC (Zhou et al., 2021), pro- poses a novel architecture and introduces new pre- training tasks. Both M and UC have demon- strated improved performance various multi- lingual V&L tasks. However, there still exist ticeable performance gaps between English and non-English languages.
Limitations
While our study focuses exploring adaptations for two specific V&L tasks, it is important to knowledge that the adaptation process can be chal- lenging for other tasks. Take xGQA (Pfeiffer al., 2022) as an example, it can not easily converted to a Question Answering task, since the caption are usually too short to include the whole context of the image. The scarcity problem, particularly prevalent in low-resource languages like Tamil in the MaRVL dataset, continues to persist.
Evaluation The recently introduced IGLUE benchmark presents new challenge for multilin- gual V&L models. This benchmark encompasses five tasks spanning 20 languages, thereby expand- ing the evaluation scope beyond previous image- text retrieval tasks such Multi30k (Elliott al.,
Acknowledgments
2016) and MSCOCO (Lin et al., 2015).
We would like to thank Dr. Farhad Nooralahzadeh for the comprehensive seminar course in Multi- modal Multilingual Natural Langauge Processing, Emanuele Bugliarello for the explanation regarding the dataset, and the anonymous reviewers for their valuable comments and feedbacks.
6 Conclusion
In this paper, we introduce ICU, divide-and- conquer approach designed to address the chal- (V&L)
lenges of multilingual dasion-and-langeen tasks. ICU leverages the strengths of both V&L models and multilingual language models (mLM) tackle the inherent difficulties these tasks. By dividing the original tasks into two stages, we trans- fer the burden of multilingual processing from the V&L model to the mLM, making it more feasible objective. This approach not only helps alleviate the scarcity problem some extent but also proves be more efficient.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr. Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei. Marianne Monteiro. Jacob Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikot Birkkowsii, Ricardo Barreira, Oriol Vinyals. Andrew Zisserman, Karén monyan, 2022. Flamingo: a visual language model
We provide valuable insights into adapting V&L. tasks to be compatible with mLMs. Furthermore, we explore the benefits of few-shot learning and

---

for few-shot learning. In Advances Neural Infor- mation Processing Systems, volume 35, pages 23716- 23736. Curran Associates, Inc.
Classifying: Addressing the Data Scarcity Issue of Supervised Machine Learning with Deep Transfer Learning and BERT-NLI Political Analysis, pages 1-33.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu, 2020. Uniter: Universal image-text representation learning.
Tsung-V Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dol- lár. 2015. Microsoft coco: Common objects in con- text.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 8440- 8451, Online. Association for Computational Lin- guistics.
Chen Liu, Jonas Pfeiffer, Anna Korhonen, Ivan Vulié, and Iryna Gurevych, 2023. Delving deeper into cross- lingual visual question answering. In Findings of the Association for Computational Linguistics: EACL 2023, pages 2453-2468, Dubrovnik, Croatia, Associ- ation for Computational Linguistics.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier. and Desmond EI- liott. 2021. Visually grounded reasoning across lan- guages and cultures. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 10467-10485, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams. Samuel Bowman. Holger Schwenk, and Veselin Stoyanov. 2018. XNLI: Evaluating cross- lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Nat- ural Language Processing, pages 2475-2485, Brus- sels, Belgium. Association for Computational Lin- guistics.
Minheng Ni, Haoyang Huang, Lin Su, Edward Cui, Taroon Bharti, Lijuan Wang. Dongdong Zhang, and Nan Duan, 2021. Mšp: Learning universal repre- sentations via multitask multilingual multimodal pre- training. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 3977-3986.
Jacob Devlin, Ming-War Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Farhad Nooralahzadeh and Rico Sennrich, 2022. Im- proving the cross-lingual generalisation in visual question answering.
Jonas Pfeiffer, Gregor Geigle, Aishwarya Kamath, Jan- Martin Steitz, Stefan Roth, Ivan Vulié, and Iryna Gurevych. 2022. xGQA: Cross- Pross-lingual visual ques- tion answering. In Findings of the Association for Computational Linguistics: ACI. 2022, pages 2497- 2511, Dublin, Ireland, Association for Computational Linguistics.
Desmond Blliott, Stella Frank, Khalil Sima' an, and Lu- cia Specia. 2016. Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop on Vision and Language, pages 70- 74, Berlin, Germany, Association for Computational Linguistics.
Chen Qiu, Dan Oneată, Emanuele Bugliarello, Stella Frank, and Desmond Elliott. 2022. Multilingual multimodal learning with machine translated text. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 4 4178-4193, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Feng Gao, Qing Ping, Govind Thattai, Aishwarya Re- ganti, Ying Nian Wu, and Prem Natarajan. 2022. Natural language- visual question answer-
munnfform-rerieee--eneee
centric outside-knowleds, ing. In Proceedings of the IEET/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5067-5077.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. Debertav3: Improving deberta using electra-style pre- training with ing.
embedding shar-
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022a. Git: A generative image-to-text
Moritz Laurer, Wouter Van Atteveldt, Andreu Casas, and Kasper Welbers. 2023. Less Annotating, More

---

transformer for vision and language. arXiv preprint arXiv:2205.14100
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022b. Ofa: Unifying architectures, tasks, and modalities through a simple requencceto-sequencce learning framework. CoRR, abs/2202.03052.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- li Tsvetkov, and Yuan Cao. 2022c. Simvlm: Simple visual language model pretraining with weak super- vision.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of gpt-3 for few-shot knowledge- based vqa.
Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019. Bench- marking zero-shot text classification: Datasets, eval- uation and entailment approach. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP)), pages 3914-3923, Hong Kong, China. Association for Computational Linguistics.
Mingyang Zhou, Luowei Zhou, Shuohang Wang. Yu Cheng, Linjie Li, Zhou Yu, and Jingjing Liu. 2021. Uc2: Universal gross-lingual cross-modal vision-and- language pre-traiennee training. In IEEE Conference on Com- puler Vision and Pattern Recognition (CVPR 2021).


Reviews:
[{"review_id": "jkjd6jBiRq", "reviewer": "Reviewer_oSwq", "summary": "", "questions": "A. How are image captions generated from OFA-large? Generating captions involves several hyper-parameters such as the decoding algorithm (i.e., greedy or beam search), beam size if beam search is used, penalty factor on repetitive sequence, etc.\n\nB. How is the proposed method evaluated in the zero-shot setting? Table 2 presents the zero-shot performance while the proposed method, especially mDeBERTaV3, is trained in the downstream tasks as described in Lines 138 - 139. Thus, it is unclear what the zero-shot performance means and how it is computed.\n\nC. How is the proposed method trained in the downstream tasks? The datasets used in the paper are multilingual multimodal ones, and thus, there would be two ways of training; training a method jointly on all the languages and evaluating individually on each language, or training individually on each language and then evaluating in the same manner. Detailed experimental setup should be described for reproducibility.\n\nD. In Figure 2, why is the performance with few-shot that is less than 10 consistently worse than that of zero-shot in all the languages?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reasons To Reject: \n1. **The paper lacks the details of the implementation of the proposed method as well as the experimental setup, which makes it difficult to reproduce the experiments. See the question section below.**\n\n\n**Response:** We understand the importance of clarity and completeness in ensuring experiment reproducibility. However, our intricate setup, involving multiple data sources, two distinct models (Image Captioning and language tasks), can result in lengthy explanations. To address this, we've chosen to provide detailed instructions in our supplementary README file, available within the submitted zip file. This approach maintains manuscript readability while facilitating easy replication. We will make our code repository publically available upon paper acceptance.\n\n2. **Citation of the related work to the idea of the proposed method is missing. See the missing reference section below.**\n\n**Response:** Upon a cursory examination of those papers, we find ourselves in complete agreement on this matter. Your reminder is greatly appreciated.\n\n## Questions For The Authors:\nA. **How are image captions generated from OFA-large? Generating captions involves several hyper-parameters such as the decoding algorithm (i.e., greedy or beam search), beam size if beam search is used, penalty factor on repetitive sequence, etc.**\n\n**Response:** Regarding hyperparameters, we have adopted identical settings to those outlined in the OFA paper to which we refer. Specifically, our decoding algorithm employs beam search with a beam size of five, complemented by a penalty factor of 3.\n\nB. **How is the proposed method evaluated in the zero-shot setting? Table 2 presents the zero-shot performance while the proposed method, especially mDeBERTaV3, is trained in the downstream tasks as described in Lines 138 - 139. Thus, it is unclear what the zero-shot performance means and how it is computed.**\n\n**Response:** We apologize for any confusion caused by the lack of clarity in our initial manuscript. Throughout this study, our performance metric is accuracy. The term \"zero-shot\" indicates that mDeBERTaV3 has not been fine-tuned for this specific task, where the premise is in English while the hypothesis is presented in other languages. The zero-shot results are listed in Table 2.\n\nLines 138-139 delineate our approach in the few-shot scenario, involving the fine-tuning of mDeBERTaV3 with varying shots. The collective few-shot outcomes are visually depicted in Figure 2, while specific results for the maximum shots are summarized in Table 3.\n\nC. **How is the proposed method trained in the downstream tasks? The datasets used in the paper are multilingual multimodal ones, and thus, there would be two ways of training; training a method jointly on all the languages and evaluating individually on each language, or training individually on each language and then evaluating in the same manner. Detailed experimental setup should be described for reproducibility.**\n\n**Response:** Indeed, this aspect escaped our attention. We pursued the second approach, whereby we conducted separate training for each language and evaluated them individually.\n\nD. **In Figure 2, why is the performance with few-shot that is less than 10 consistently worse than that of zero-shot in all the languages?**\n\n**Response:** We hypothesize that in situations with a limited number of shots, the tuning process may result in a model with reduced generality. It's only with an adequate number of shots that the model can truly achieve noteworthy performance enhancements.\n\n## Typos Grammar Style And Presentation Improvements:\n\n**Response:** We are genuinely appreciative of your thoughtful and valuable feedback, as they are certain to contribute to the improvement of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed idea and method are straightforward and effective.\n2. The paper is easy to follow.", "reasons_to_reject": "1. The paper lacks the details of the implementation of the proposed method as well as the experimental setup, which makes it difficult to reproduce the experiments. See the question section below.\n2. Citation of the related work to the idea of the proposed method is missing. See the missing reference section below.", "questions_to_authors": "A. How are image captions generated from OFA-large? Generating captions involves several hyper-parameters such as the decoding algorithm (i.e., greedy or beam search), beam size if beam search is used, penalty factor on repetitive sequence, etc.\n\nB. How is the proposed method evaluated in the zero-shot setting? Table 2 presents the zero-shot performance while the proposed method, especially mDeBERTaV3, is trained in the downstream tasks as described in Lines 138 - 139. Thus, it is unclear what the zero-shot performance means and how it is computed.\n\nC. How is the proposed method trained in the downstream tasks? The datasets used in the paper are multilingual multimodal ones, and thus, there would be two ways of training; training a method jointly on all the languages and evaluating individually on each language, or training individually on each language and then evaluating in the same manner. Detailed experimental setup should be described for reproducibility.\n\nD. In Figure 2, why is the performance with few-shot that is less than 10 consistently worse than that of zero-shot in all the languages?", "paper_topic_and_main_contributions": "This paper proposes a multilingual multimodal method that divides a task into two sub-tasks: English caption generation and cross-lingual natural language understanding. The paper evaluates the proposed method on two tasks, XVNLI and MaRVL, in nine languages. Experimental results show that the proposed method outperforms existing strong baselines in four languages while the results are comparable with the baselines in the rest of the languages.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "The following works also propose generating and using image captions as an intermediate clue to solve a VQA task while it is not a multilingual task. Nevertheless, the idea is quite similar to the proposed method in this paper and these works should be acknowledged and referred to in the paper.\n\n- Gao, Feng, et al. \"Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- Yang, Zhengyuan, et al. \"An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA.\" Proceedings of the AAAI Conference on Artificial Intelligence. 2022.", "Typos_Grammar_Style_and_Presentation_Improvements": "1. In Figure 1, English translations should be appended to captions in non-English languages to help readers who do not understand these languages grasp the proposed idea.\n2. In Line 157, MARVL -> MaRVL\n3. Figure 3 should be redrawn with a bar chart or similar types of charts because there is no dependency across different frames and the scores with different frames cannot be connected to each other as a line chart.\n4. In Line 287, CONNEAU -> Conneau"}}, {"review_id": "F6sff3goO7", "reviewer": "Reviewer_ZPdR", "summary": "", "questions": "N.A", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reasons To Reject:\n\n1. **The chained impacts of image captioning and multilingual understanding model in the proposed pipeline. If the Image caption gives worse results and the final results could be worse. So The basic performance of the image caption model and multilingual language mode depends on the engineering model choice when it applies to zero-shot.**\n\n**Response:** We concur that the cascading impact is a valid consideration. This is precisely why we've opted for a state-of-the-art model that delivers high-quality captions effectively even in a zero-shot scenario. Additionally, our multilingual language model has showcased remarkable zero-shot performance, as demonstrated in Table 2.\n\nIn support of our methodology, the engineering aspect is considerably less demanding in comparison to the training processes. Hence, experimenting with various options could prove beneficial without significant effort.\n\n2. **This pipeline style method including two models does not give better average results for both XVNLI and MaRVL. Baseline models in the experiments are not well introduced.**\n\n**Response:** Our approach may not present superior average results. However, it's crucial to highlight that it does yield enhanced performance in four or five (including the max-shot scenario) out of the nine languages considered.\n\nAbsolutely, we should have provided more comprehensive insights into the baseline models. However, we'd like to emphasize a crucial aspect here. Our method boasts adjustable parameters of 86M (specifically, mDeBERTaV3 base), whereas the competing models have 185M for mUNITER, 284M for xUNITER, 282M for UC$^2$, and 377M for M$^3$P. In practical terms, our method is more parameter-efficient, and it's imperative not to underestimate the performance it can deliver.\n\nFurthermore, our approach introduces the opportunity to reuse an already trained image captioning model, a practice that undoubtedly aligns with ecological considerations. This stands in contrast to training a model from scratch, as done in the baseline, which is inherently more resource-intensive.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This simple idea gives state-of-the-art results on partial tasks of XVNLI and MaRVL of multilingual vision and language tasks.\n- With better training on the sub-module (mLM), the proposed model can give better performance.\n- The idea to divide the multilingual vision and language task into Image captioning and multilingual understanding is intuitive. ", "reasons_to_reject": "- The chained impacts of image captioning and multilingual understanding model in the proposed pipeline. If the Image caption gives worse results and the final results could be worse. So The basic performance of the image caption model and multilingual language mode depends on the engineering model choice when it applies to zero-shot. \n- This pipeline style method including two models does not give better average results for both XVNLI and MaRVL. Baseline models in the experiments are not well introduced. ", "questions_to_authors": "N.A", "paper_topic_and_main_contributions": "- What is this paper about? This paper aims to solve the multilingual vision and language task.\n- what contributions does it make? This paper proposes to divide the task into two-stage: Image captioning in English and using the English text for later multilingual understanding.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N.A", "Typos_Grammar_Style_and_Presentation_Improvements": "N.A"}, "extra_text": {}}, {"review_id": "WyIIwje71d", "reviewer": "Reviewer_UdNy", "summary": "", "questions": "A) Has something similar been proposed for tasks beyond XNLI?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reasons To Reject:\n**I'm not so sure that the vision part is interesting for this community, so maybe a more general conference such as NeurIPS or AAAI might be more suitable for this type of paper. And there is no contribution in the language aspects I know it is a short paper, but it would benefit from a clearer explanation of the tasks/datasets and the methods**\n\n**Response:** While we understand the shared concern regarding the vision component, we remain confident that our work is a suitable fit for this community, given the undeniable significance of vision as a pivotal modality. An illustrative example is the MaRVL dataset employed in our research, which not only achieved acceptance at EMNLP21 but also secured the esteemed recognition of the best paper award.\n\nFurthermore, we'd like to emphasize that our approach introduces a novel challenge within the realm of language aspects. Unlike previous works in the context of XNLI, where both the premise and hypothesis were consistently in the same languages, our approach introduces a distinctive setting where the premise is in English, and the hypothesis is in other languages. Therefore, it's important to recognize that our work does indeed make a meaningful contribution in this dimension. We deeply regret any confusion that may have arisen due to our earlier lack of clarity in explaining these aspects.\n\n## Questions For The Authors:\n\nA) **Has something similar been proposed for tasks beyond XNLI?**\n\n**Response:** We ventured into applying our method to another task, namely xGQA. However, as outlined in the limitations section, adapting our approach to this task posed considerable challenges. xGQA, at its core, is a classification task characterized by an extensive label space exceeding 1800 labels. Consequently, conventional NLP techniques such as text classification (typically effective with fewer than 10 labels), question answering (often used to extract answers from lengthy paragraphs), or text generation (which poses open domain challenges) do not seamlessly align as downstream models for processing the captions we obtain.\n\n## Typos Grammar Style And Presentation Improvements:\n\n**Response:** Indeed, that detail slipped our notice. We appreciate your pointing it out, and we indeed employed accuracy as the metric. Thank you for raising this concern.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The approach is simple and works well, and that should should be praised in my opinion\nThey report new SOTA results in multilingual vision understanding tasks ", "reasons_to_reject": "I'm not so sure that the vision part is interesting for this community, so maybe a more general conference such as NeurIPS or AAAI might be more suitable for this type of paper. And there is no contribution in the language aspects\nI know it is a short paper, but it would benefit from a clearer explanation of the tasks/datasets and the methods", "questions_to_authors": "A) Has something similar been proposed for tasks beyond XNLI?", "paper_topic_and_main_contributions": "The paper present an approach for image captioning understanding (ICU) by decomposing a single vision-to-language model into two models: one for visual understanding and another for multilingual language understanding. They make use of vanilla pre-trained models for each decomposed tasks, and report results that beat some SOTA results on the IGLUE benchmark.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It would be nice to mention which metric is used in the results reported in Table 2. I imagine it is accuracy, but I'm not sure"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method that splits multilingual multimodal understanding tasks into two stages:  English caption generation and cross-lingual natural language understanding.

Pros:  The proposed approach is intuitive and achieves SOTA results in the part of the experimental studies

Cons: The method is hard to generalize for tasks beyond XNLI due to this two stage setup. Also the method does not give better average results for both XVNLI and MaRVL

Additionally the draft needs to be updated with more experimental details like use of single-language at a time modeling, baselines and other details shared during the author discussion period.
Decision: Accept-Findings