MaXM: Towards Multilingual Visual Question Answering
Authors: Soravit Changpinyo
Linting Xue
Michal Yarom
Ashish V Thapliyal
Idan Szpektor
Julien Amelot
Xi Chen
Radu Soricut
Venue: EMNLP 2023 Findings (2023)
Keywords: VQA
multilinguality
TL;DR: New benchmark and scalable data creation for multilingual visual question answering

Abstract: Visual Question Answering (VQA) has been primarily studied through the lens of the English language. Yet, tackling VQA in other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multilingual visual question answering (mVQA), on both data and modeling fronts. We first propose a translation-based framework to mVQA data generation that requires much less human annotation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilingual captions in the Crossmodal-3600 dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark in 7 diverse languages. Finally, we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on mVQA.

MaXM: Towards Multilingual Visual Question Answering
Soravit Changpinyo, Linting Xue, Michal Yarom, Ashish V. Thapliyal Idan Szpektor, .ullien Amelot, Xi Chen, Radu Soricut
Google Research
https://gil http:://.tt.tooooooooo/eeeeeeeer /
Abstract
dog allowed free?
A: No
Visual Question Answering (VQA) has been primarily studied through the lens of the En- glish language. Yet, tackling VQA other languages in the same manner would require a considerable amount of resources. In this paper, we propose scalable solutions to multi- lingual visual question answering (mVQA), both data and modeling fronts. We first propose a translation- baaed framework to vVQ data generation that requires much less human an- notation efforts than the conventional approach of directly collection questions and answers. Then, we apply our framework to the multilin- gual captions the Crossmodal- dataset and develop an efficient annotation protocol to create MaXM, a test-only VQA benchmark diverse languages. Finally. we develop a simple, lightweight, and effective approach as well as benchmark state-of-the-art English and multilingual VQA models. We hope that our benchmark encourages further research on m VQA.
servis une assiette reclangulaire blanche avec de sauce soja?
Combien
nems, cing, cing nems (O) many spring se awred whilh rectinguter plets any sauce?
spring rolls five five spring roli))
Rिanç è ररी ? A: सहय सर, wndlh arper पर, saee it eclit स8अ it
Q: माती टिर
(Q: On whinh surface the black visitte??
road, clook road, road the blook road)
Q: atn quó no? nxur
(Q) What me name of this bind? Cassowary)
Q: Ce de cafee este ceayca albă? cu calmac, cafea lapto, cafea ou spumå, cafee ou spumå de leple, cafea neagrä spumá
(a: Whar kind of coffee is in the whic cup? with cream, cofiee with milk, coffes with foom, coffee with mili foom, Block coffee with (oom)
Q: no: rn:Tuuaauuuneeeeee 19a
A: #onn, uno
(Q: What cotor the skint dol?? whsth white)
1 Introduction
T Bltth MA tE
30 HE
Visual Question Answering (VQA), the task of an- swering visual questions grounded in images, is key to human-machine interaction in the visual world. In particular, the natural language interface in VQA makes it easy for lay people to express their needs and benefit from its applications, including accessi- bility, education, and search. Yet, VQA advances were mostly focused on English, therefore only ap- plied to a privileged subset of human populations. Arguably, the English language has dominated the field mainly because of the availability of En- glish VQA benchmarks. These benchmarks diverse, from general VQA (Zhu al., 2016; Kafle and Kanan, 2017: Krishna al., 2017: Antol et al., 2015; Goyal et al., 2017; Changpinyo al., 2022), robust VQA (Agrawal et al., 2018), compositional visual reasoning (Hudson and Manning, 2019), for
(O: Beaidio recyclatles, what alse is printed on the imahi cun?
other wustss)
Figure 1: Multilingual VQA Data in languages. The data automatically generated from multilingual cap- tions and then verified and adjusted by humans. From top to bottom: English (en). French (fr), Hindi (hi), He- brew (iw), Romanian (ro), Thai (th), and Chinese (zh).
the blind and the risually-impaire (Gurari et al., 201 8), scene-text understanding (Singh et al., 2019; Biten et al., 2019), to VQA that requires external, commonsense, world knowledge (Marino al., 2019; Zellers et al., 2019: Schwenk al., 2022). These benchmarks require considerable amount of resources to create, mostly by employing hu- man annotators to laboriously collect and verify

---

the questions and the answers for each image. et al., 2017: Agrawal et al., 2018; Gurari et al., 2018; Marino al., 2019; Singh et al., 2019; Biten et al., 2019; Sheng et al., 2021; Li et al., 2021; Changpinyo et al., 2022). The only exception is XGQA (Pfeiffer et al., 2022), an extension of the English GQA dataset (Hudson and Mamning, 2019). XGQA consists of human translations of the 12,578 English questions from 398 images in the balanced testdev split of GQA in 8 typologically diverse languages: English, German, Portuguese, Russian, Indonesian, Bengali, Korean, and Chinese. Besides the differences in the languages considered, our proposed approach to mVQA data creation com- plements xGQA (see Sect. 4.4).
To extend VQA to all languages in the world, we must make data creation more automatic. Building on recent work on automatic data creation for En- glish VQA from captions (Changpinyo et al., 2022), in this paper we propose translation- based frame- work for multilingual visual question answering (mVQA) data creation. Our framework automates much of the task of generating questions and an- swers, thus providing a scalable path to mVQA.
We apply our framework to the generation of question-answer pairs from the multilingual cap- tions of the recently-proposed Crossmodal-3600 dataset (XM3600) (Thapliyal et al., 2022). Com- bined with efficient human annotation protocol, we construct MAVERICS-XM360 (MaXM), test benchmark for mVQA in languages (see ex- amples in Fig. 1).
Beyond mVQA, training and evaluation data for multilingual multimodal models is limited. For a review of previous work, we refer the reader to the Image-Groundde Language Understanding Evaluation (IGLUE) benchmark (Bugliarello et al.. 2022), where xGQA is part of. In general, early attempts often focus on Chinese (Li et al., 2019; Wang et al., 2019), Japanese (Yoshikawa al., 2017; Aggarwal and Kale, 2020) and several Indo- European languages (e.g., German, French, and Czech) (Elliott et al., 2016, 2017; Barrault et al., 2018). However, there is a recent effort toward a wider variety of both languages and tasks. Exam- ples include image retrieval (Aggarwal and Kale, 2020) (also Russian, Korean, Turkish), visual nat- ural language inference (Bugliarello et al., 2022) (also Arabic), multilingual visual reasoning (Liu et al., 2021) (also Indonesian, Swahili, Tamil, Turk- ish), and vision-and- and-lage navigation (Ku et al., 2020) (also Hindi, Telugu). Notably, Wikipedia Image Text (WTT) (Srinivasan et al., 2021) pro- vides large-scale image-text dataset in 108 lan- guages, automatically collected form Wikipedia, and Crossmodal-366 (XM3600) (Thapliyal et al., 2022) provides human-curated test-only image cap- tions in 36 languages. Our work builds on top of XM3600, and the 7 languages that we consider are typologically, genealogically, and geographically diverse.
Finally, we use this novel benchmark to drive progress in mVQA modeling and mcasure where we stand. We leverage advances in image modeling and multilingual modeling: VIT (Dosovitskiy et al.. 2021) and mT5 (Xuc et al., 2021) and propose a unified, extensible, open-ended mVQA model, called Simple MPT, which is competitive state- of-the-art English VQA models that we adapt to apply in the m VQA setting (OFA (Wang et al., 2022b) and BLIP2 (Li et al., 2023)). Overall, there exists a large room for improvement.
In summary, main contributions are (i) scalable translation-basee framework for m mQA data generation based on captions (Sect. 3); (ii) an efficient annotation protocol, deriving a novel test benchmark called MAVERICS- XM3600 (MaXM) in 7 diverse languages: En- glish, French, Hindi, Hebrew, Romanian, Thai and Chinese (Seet. 4):Michal: Beer, why is this dataset better than XGQA? (iii) simple and lightweight mVQA modeling (Sect. 5.2, Sect. C) with strong performance; (iv) benchmarking (adap- tations of) the state-of-the-at VQA models on MaXM (Sect. 5.3).
2 Related Work
2.1 VQA and Multilingual Multimodal Benchmarks
2.2 VQA Data Creation
Previous work on VQA data creation relies heavily on humans to create questions and answers (Zhu et al., 2016; Krishna al., 2017: Goyal al., 2017: Gurari et al., 2018; Marino et al., 2019). Some works attempt to automate this process. CLEVR (Johnson et al., 2017a) uses a template-
English has been the primary language in which vision-and-language researchers study the VQA task, driven by the availability of data and bench- marks (Zhu et al., 2016; Kafle and Kanan, 2017; Krishna et al., 2017: Antol et al., 2015;
Goyal

---

based approach, but it based on synthetic images for which ground-truth annotations are available. GQA (Hudson and Manning, 2019) follows a simi- lar approach but instead starts from Visual Genome scene graphs (Krishna et al., 2017), which them- selves require large annotation efforts.
Generation generates candidate questions for each candidate answer, (iii) Answer Validation filters candidate questions that do not pass a consistency check that involves automatically answering each question from the caption and comparing this an- swer to the original extracted answer (Alberti et al., 2019; Honovich et al., 2021).
More relevant are works that rewrite image captions or video transcripts as question-answer pairs. COCOQA (Ren et al., 2015) uses a template- based approach that can only generate questions with one-word answers. WeaQA (Banerjee et al., 2021) improves upon this with semantic role label- ing, paraphrasing, and backtranslation. Recently, Changpinyo et al. (2022) and Yang et al. (2021) leverage T5 (Raffel et al., 2020) fine-tuned on question answering datasets, generating large-scale VQA datasets for images and videos, respectively. Our approach to mVQA data creation leverages VQ'A, the approach in (Changpinyo et al., 2022) (Sect. 3.1). To the best of our knowledge, besides XGQA, no other prior work on VQA data genera- tion considered languages beyond English.
Each step in VQ'A is optimized for English: Step (i) uses English spaCy and both Step (ii) and Step (iii) leverage high-capacity English-pre- trained T5 models fine-tuned on English question answering datasets (Rajpurkar et al., 2016, 2018; Kwiatkowski et al., 2019).
3.2 Translation-based VQ"A (TransVQ*A)
Inspired by VQ²A, our goal is to generate mVQA data at scale, leveraging multilingual image cap- tions. Multilingualizing each step in VQ'A can be non-trivial and essource-intensii due to the heavy reliance of English tools, models, and data (Sect. 3.1). To alleviate this, we propose a cesslation-baseed extension of VQ²A.
Given an input caption e in any language, and target language (lang), we want to generate question-answer pairs in (lang). We propose Translation-basee VQ"A (TransVQ'A as follows: Step 1 Caption Translation: Automatically trans- late a non-English caption to English ce. Step 2 Apply VQ2A: Generate a set of English question- answer pairs {ge, } from Ce- Step 3 Question- Answer Translation: Automatically translate all (ge, a2) pairs to (lang) (g, a). Step 4 Validation: Filter (g, a) pairs! in which a does not appear in the original caption c, back-translating to c's lan- guage if necessary. The upper part of Fig. 2 exem- plifies Trans VQ?A using a Chinese caption from Crossmodal-336 (Thapliyal et al., 2022).
3 Multilingual VQA Data Creation
Like in many other machine learning tasks, the main bottleneck to mVQA is obtaining high-quality labeled data. The most popular data collection framework to English VQA is to ask a set of human annotators to come up with visual ques- tions, and another set of annotator answer them (Sect. 2.2). To scale VQA to all languages, we argue that mVQA data creation must significantly reduce its use of human annotation. To this end, we study the extension of an automatic English VQA data creation method called Visual Question Generation with Question Answering validation, VQ²A (Changpinyo et al., 2022) for the purpose of m VQA data creation.
We highlight that the approach we have de- scribed so far is fully automatic and applicable to a huge set of languages that are supported by automatic translation. We note that the final val- idation is important due errors that could pile up during translation steps. This is especially acute in Step 3, since translating answers is harder due to the lack of disambiguating context in the short an- swers. We also note that TransVQ'A can generate question/answer pairs in the target (lang) from any caption. The output quality depends on the trans- lation quality, e.g. the back-translation in step 4 from (lang) to c's language. We use out-of-the-box
3.1 Background: VQ²A
The VQ'A approach leverages aligned image-text data sources that are available at scale (Ordonez
et al., 2011; Chen et al., 2015: Sharma et al., 2018;
Pont-Tuset et al., 2020; Changpinyo et al., 2021; Desai et al., 2021; Schuhmann et al., 2021) and beyond English (Srinivasan et al., 2021; Gu et al., 2022). It rewrites a declarative image caption into multiple interrogative question-answer pairs via three steps: (i) Candidate Answer Extraction ex- tracts candidate answers based on syntactic and semantic analysis of an input caption, (ii) Question
'Excluding answers to boolean questions.

---

Caption: 1
MT
Caption: Japanese ramen with tomato
VQ'A
Question: with tomato It?
addition japanesa ramen?
MT Answer Validation
MT & Answer Validation
Image (not used the ppaline)
Anewer:
Question: M BE ?
Answer:
Question: B T
Correct
Human Verification & Modification
Human Verification & Modification
Almost
Answer:
Queolion-
Figure 2: Our approach to muitilingual VQA data generation, which is easy to scale, highly automatic and only requiring humans to modify "Almost Correct" questions or correct/expand answers (left) or filter out "Incorrect" questions(right). MT is short for automatic machine translation.
translation tools in this work, and leave the explo- ration of better translation tailored for Trans VQ'A for future work.
In Sect. we employ human annotators to further clean and expand the generated data to create a high quality test benchmark.
styl
mostagred artores morts snmy cammon the snow, vira the monttinn and dead trecs.)
(Thee dim aum-etyle apwosd revldt
woodon dist.)
or hrhero on roecauo
and vr cal dish are we whil vesttr (A close up view the chirese dum sun plwced earhs)
(A mrach firee whecls and orange part rosce snends some arawy
3.3 Direct Question Generation (DirectQG)
One drawback of Trans is the low coverage of particular types of answers, such as "no". This is because the captions generally do not indicate the absence of objects or properties (e.g.. "There is no dog". "The dog is not white") To mitigate this bias, we train multilingual question generator that takes in an answer and caption in target lan- guage and generates relevant questions in the same language. We use the model to generate questions for "yes", "no"", "none" answers in each target language, as a complement to TransVQ2A.
Tun zipod0 funojinee mung. (Brow operation mounaaine
(Flve fined with
long placod
and mountere the dietence are covered with snow)
coveree while
Figure 3: The diversity of multilingual captions in XM3600. We show the captions (their English transla- tions) from 4 languages for the images of snow cannon (left) and xiao long bao (right).
annotation protocol, and analysis and discus- sion of our benchmark. Following the naming con- vention in (Changpinyo et al., 2022), we call our benchmark MAVERICS-XM3600 or MaXM short. We will release MaX. to foster research on mVQA.
Concretely, we fine-tuned mT5-XXL (Xue et al., 2021) on large-scale translated COCO Cap- tions (Chen et al., 2015) and its corresponding VQA data VQ'A-COCO (Changpinyo et al., 2022). For validation, used the subset of generated mul- tilingual VQA data in Sect. 3.2, with 330 golden examples for each language. The best checkpoint was selected based on ROUGE-L scores.
4.1 Data Sources
Language Selection. We chose 7 languages that are 1) typologically diverse, 2) genealogically di- verse, and 3) geographically diverse: English (en), French (fr), Hindi (hi). Hebrew (iw). Romanian (ro), Thai (th), and Chinese (zh).
4 MaXM: Multilingual VQA Benchmark
In this section, we leverage the approach we pre- sented in Sect. for creating multilingual VQA test-only benchmark. We next describe our data sources, how candidate data was generated, human
Image and Caption Selection. We chose a subset of the images in Crossmodal-3600

---

fr
iw
zh
Captions
7200
8562
7123
7174
English QAs
373248
499900
520080
544268
516604
415180
524252
Validated
264930
343621
346948
375629
346887
286024
362304
English QAs
(71.0%)
(68.7%)
(66.7%)
(69.05)
(67.1%)
(68.9%)
(69.1%)
Validated
264724
122644
153465
128613
121221
95531
182095
Multilingual QAs
(99.92%)
(33.85%)
(53.67%)
(37.08%)
(32.27%)
(27.53%)
(52.99%)
Table 1: Number Instances (% of Previous Stage) of automatically-geeenatii question-answer (QA) pairs based on Crossmodal-3600 captions. Validated English pairs are w.r. the QG-QA consistency filter. Validated multilingual pairs are w.r.t the caption-ansee answer consistency filter. Label
Question
Answer
Correct
Makes sense AND is relevant to the image. Satisties the question's 's intent wrt the image.
Almost Correct Incorrect
Correct but its surface form be improved (syntactic
aw.kwarduncoommnn usages.)
NOT Correct.
Table 2: Definition of Correct, Almost Correct and Incorrect labels for questions and answers in our annotation protocol.
(XM3600) (Thapliyal et al., 2022), in which high- quality multilingual image captions are available. For each language, 100 validation and test images of Open Images (Krasin et al., 2017; Kuznetsova et al., 2020) that were taken in the region(s) in which those languages were spoken were selected.
languages are filtered out based on the caption- answer consistency validation. A main reason for this the quality of question-anae -answer translation. For instance, 68% of questions with "alb" (mascu- line "white" in Romanian) are filtered out because they are not translated to the correct feminine form "alba" w.r.t the corresponding object in the question.
Our image selection criteria cover wide range of visual concepts in different cultural contexts, making the constructed VQA examples diverse and specific to the languages of the captions related to each image. For example, Fig. 3, unlike French and Romanian speakers, Hebrew and Thai speakers are less likely to know what snow cannon is. On the other hand, Thai and Chinese speakers are more likely understand what xiao long bao is, whereas dim-
DirectQG. We augment the Trans VQ questions with additional candidate questions generated by Trans VQRA (Sect. 3.3), using the XM3600 cap- tions paired with "yes", "no", or "none" in their corresponding language input.
4.3 Human Annotation
in French Hindi it could be referred sum ravioli Chinese dim sum.
We employed native speakers for each of the se- lected languages to annotate and create our bench- mark. We designed an annotation protocol to bal- ance efficiency and accuracy. In particular, we keep human-in-the-logg brief and only when an au- tomated model straggles in task, e.g., correcting translation artifacts, expanding answers, identify- ing sensitive questions. Furthermore, our protocol promotes quick discarding of examples, when the question does not make sense. We provide more details next and also Appendix B.
Another benefit of XM3600 is that the Open Images images are out-of-domain with respect to most widely-used VQA benchmarks (Ren et al., 2015; Antol et al., 2015; Zhu et al., 2016; Krishna et al., 2017; Goyal et al., 2017; Agrawal et al, 2018; Hudson and Manning, 2019; Marino et al., 2019), which are often based on MS-COCO images (Lin al., 2014).
4.2 Large-Scale m VQQ Data Creation
We apply our approach described in Sect. 3 to the XM3600 captions to generate large number of question-answer pairs for each language.
Question and Answer Validation. We define a 3-way rating system of Correct, Almost Correct and Incorrect for both the questions and the an- (see Table 2). Correct questions are kept unchanged, Almost Correct questions are manually rewritten, and Incorrect questions are discarded. Given Correct and Almost Correct questions, an annotator rates the answer and corrects it in the cases of both Almost Correct and Incorrect.
TransVQ²A A. Table reports the number of question-answer pairs at different stages in our pipeline. Overall, we are able to generate large number of question-answer pairs in all languages. We found that, across languages, approximately 30% of (translated) English question-answer pairs are filtered out due to VQ²A validation. contrast, different percentages of translated answers across
Table 3 reports label distribution for questions and answers cendomll-ssmpple from those gen-

---

fr
iw
2h
of questions evaluated
377
389
365
440
391
Correct
62.6%
65.8%
66.5%
61.6%
59.1%
47.4%
51.9%
% Almost Correct
17.5%
12.3%
9.0%
22.5%
9.3%
27.9%
25.1
% Incorrect
19.9%
21.9%
24.50%
15.9%
31.6%
24.7%
23.02%
of answers evaluated
302
304
307
301
% Correct
66.2%
72.0%
77.8%
73.9%
76.2%
82.2%
81.9%
% Almost Correct
26.5%
24.0%
17.9%
24. 1%
16.9%
7.9%
9.2%
% Incorrect
7.3%
3.9%
4.3%
2.0%
7.0%
9.9%
8.9%
Table 3: Human evaluation of the generated questions and answers.
Question
Perazntage
plastic
enbois
BUSN tommie
leaves
Prefix
en
Ir
IW
T0
15
Zh
macro
glasses
"is"
22.8
21.2
21.4
17.8
19.5
16.2
20.2
"what is"
15.8
11.3
16.0
15.2
13.5
11.6
13.7
THE
Deople
"hon many"
15.1
6.7
11.3
13.9
10.2
14.1
15.2
12.1
close-up
&
CouverL
frees
what kind""
6.0
7.2
1.4
3.8
6.5
4.0
3.6
HOU
"what
3.4
1.0
nuagou
R
"who""
3.4
3.1
9.2
2.5
1.5
2.6
2.3
"are"
3.4
0.7
4.1
3,8
3.6
3.5
2.0
3.4
7.5
3.0
3.1
1.4
3.5
2.1
3.0
2.9
vega
GITY quatr
"what type"
2.7
opans
what way"
1.0
0.3
0.0
0.5
1.2
3.3
0.7
DUnG
D'YVO
Dina
"do"
0.7
0.0
0.7
2.3
Kit
"in"
0.7
0.3
0.3
2,4
1.7
2.0
"besider"
0.3
1.0
4,4
3,2
1.2
0.5
2.3
"does"
0.3
1.7
1.5
other
11.4
16.0
12.2
15.9
14.1
13.6
15.3
Table The distribution of question types in MaXM
Figure 4: Top answer cloud is for "Whar" questions (excluding "What color"). Bottom answer clouds from left to right are for "What color", Boolean Washe//Wa//ere/D///////iii and "How many" ques-
across languages. Approximated by their corresponding English question prefixes.
erated by Trans VQ'A3. Across languages, we observe at least 75% Correct or Almost Correct questions and, given these questions, least 90% Correct or Almost Correct answers. This highlights the effectiveness of our approach.
tions, respectively.
4,4 Analysis and Discussion
Size and Question Type and Answer Distribu- tions. MaXM v1 includes 2,142 questions in 7 lan- guages: English (298), French (293), Hindi (294), Hebrew (315), Romanian (333), Thai (302), and Chinese (307).
Answer Expansion and Standardization. We split the generated questions into 4 categories: boolean, numeric, color, and others. We then asked the annotators to perform standardization of boolean, numeric, and color questions based on each language's guideline. For the rest of the ques- tions, we tasked another set of at least 2 annotators per language to expand the answers to these ques- tions with as many additionally correct (but not overly lengthy) answers as they can.
Table 4 shows breakdown of question types from MaXM. Since question prefixes some lan- guages are not indicative of question types (c.g.. Thai does not always begin the "What" questions with the Thai "What"), we estimate question's 's type using the prefix of its English version be- fore translation. We observe diverse types and high degree of linguistic variations. Fig. presents word clouds for answers for selected question types: "What", "What color", Boolean, and "How many", to further illustrate the diverse answers within MaXM for each question type.
Additional Filtering. Our raters performed an- other round of verification, filtering out examples with "ambiguous" and "Tesponsible-AT-sensie questions and/or with inappropriate image content. The raters also labeled "Collection" questions that are likely to lead to long answers that are difficult to evaluate, such as for "What is the table?" when there multiple items, without filter them out.
Comparison to xGQA. In terms of settings, one difference is the languages of the answers: xGQA operates in the "cross-lingual" setting where the input question can be non-English but the output answer is always English. While this simplifies the evaluation process, we argue that the "multilingual" setting with non non-English answers considered
Foor pairs generated by DirectQG, we did not perform exhaustive verification; we only asked the raters to annotate Cornect and Almost Correct questions related to "no" answers.

---

MaXM is more practical. Another difference is the definition of the zero- shot setting: XGQA refers to unseen languages (not images) whereas our setting is more general, refer- ring to both unseen images and languages. Finally, the type of translated data and how it is used for training are different; we only consider zero-shot setting and always use machine-translate ques- tions for training, while xGQA considers both zero- shot and few-shot settings with transatt questions involved only the few-shot case.
5.2 Models for Multilingual VQA
Inspired by approaches multilingual NLP search, we consider two main families of models for mVQA that adapt existing source English VQA datasets to target languages: Translate- Test and Translate-Train. Translate-Test leaves the train- ing data and the model as-is, but translates the test VQA data to the the source language English, apply the model, and then translate it back to the target language. On the other hand, Translate- Traina trans- lates the English VQA data a target language. trains model on this pseudo VQA data (i.e., their translations), and directly apply the trained model to the test data.
In terms of the datasets, xGQA inherits the char- acteristics of GQA, whose questions are restricted in style (e.g., generated by a probabilistic template- based question engine) and in the skills required (e.g., reasoning-based with multi-step inference of object attributes and relationships) (Hudson and Manning, 2019). contrast, MaXM's questions are more general. Additionally, xGQA considers the same set of questions for all languages, whereas MaXM considers different sets of questions guided by the captions in each language.
Translate- Test. We consider two open-source state-of-the-at VQA models: OFA-Large (Wang et al., 2022b) and BLIP2 (Li et al., 2023). Neither of them designed for mVQA.
Translate-Train We include the results from the state-of-the-at multilingual vision-and-language model PaI.I-17B (Chen et al., 2023), which pretrains on diverse VQA datasets in 35 lan- guages (Thapliyal et al., 2022) among other datasets, and then finetune on VQA2.0 in 13 lan- guages: en, bn, de, fr, hi, id, iw, ko, pt, ro, ru, th, zh. Further, we implement a lightweight version of PaLL, called Simple Multi-Language Prompted Training, Simple MPT, with much smaller model and without vision-ane and lan-language pre-training. Sim- ple MPT is trained on the data in 13 languages in a multi-task fashion. Details can be found in Ap- pendix C.
5 Evaluation
5.1 Evaluation Protocol
Evaluation Metrics. We use Exact Match Accu- racy the main evaluation measure for MaXM, following previous work on VQA (Antol et al.. 2015; Goyal et al., 2017; Gurari et al., 2018). We deem the answer as correct it matches any of the ground-truth answers. To assess the degree of strict- ness of this measure, we also consider soft text sim- ilarity metrics CIDEr (Vedantam et al., 2015) and ROUGE-L (Lin, 2004) in our experiments, where we treat each of the ground-truth answers equally one of the references (as if each of them was answered by an annotator).
5.3 Results
Main Results. Table benchmarks our proposed Simple MPT and state-of-the-art VQA models on MaXM. We observe that PaLI-17B performs best on all languages. This can be attributed to both the fact that PaLl is the strongest English VQA model and the fact that it designed be multilingual, leveraging pre-training image-text corpus in 105 languages. This result suggests it can be beneficial to design and develop multilingual VQA models from day one.
Training Data. MaXM is test-only benchmark; it cannot be used for training. We designate VQA2.0 (Goyal et al., 2017) and its translations as the default training data source for our benchmark, due to its popularity and quality, similarly to the use of COCO-Captions (Chen et al., 2015) for the nocaps benchmark (Agrawal et al., 2019) in the image captioning task. Nevertheless, we allow free use of existing VQA resources for training long the corresponding training images do not overlap with MaXM images. In our experiments, we also consider VQ²A-COCO and VQ-A- CCC3 (Chang- pinyo et al., 2022) to assess the effect of text do- main gap.
proposed Simple MPT model is strong baseline even though it is much smaller than PaLI and does not leverage multilingual pre- training data. While its English performance is on par with OFA and much worse than BLIP2, its multilingual performance excels, outperforming
Surprisingly,

---

Model
Language
fr
hi
IW
th
zh
OFA (Wang al., 2022h)
[470M]
35.6
13.3
41.5
31.7
26.4
29.5
17.6
Translate- Test
BLIP2 (Li et al., 2023)
[11B]
48.7
20.1
63.3
49.2
39.0
49.0
28.0
Simple MPT (ours)
[1.58]
36.6
36.2
55.1
40.6
42.3
50.0
30.3
Translate-Trun
Pal.1 (Chen et al., 2023)
[17B]
56.4
46.4
67.3
60.0
57.4
65.6
46.9
Table 5: Benchmarking VQA models on MaXM. Accuracy (%) of OFA-Large (OFA), BLIP2, our lightweight Simple MPT, and PaLI-17B (PaLI), with approximate parameter count in brackets. All are finetuned on VQA2.0.
English-only for Translate-Traie and 13 languages for Translate- Test. Best results are bold. Second best italized.
Model
Training Dataset
Language
en
fr
1w
r0
th
zh
Single-Langunn
VQA2.0
37.6
33.8
53.7
35.6
36.0
50.0
29.0
Simple MPT
VQA2.0
36.6
36.2
55.1
40.6
42.3
50.0
30.3
Trunslate-Tran
Simple MPT
VQ'A-COCO
48.0
43.3
56.8
42.2
45.6
52.3
34,5
Simple MPT
VQ'A-CC3M
38.3
34.1
45.6
34.9
36.9
45.0
29.0
Table 6: Effect of Training Data Sources. Accuracy of Single-Language baselines (MPT architecture) and
Accuracy (%) of MPT models trained on different training datasets.
OFA in all languages and underperforms BLIP2 only for Hindi and Hebrew.
Less Strict Metrics. In Table 7 We observe gen- erally consistent results when using CIDEr and ROUGE-L instead of the stricter Accuracy, except for Thai and Chinese, where the gaps in Accuracy are small to begin with.
Overall, our result suggests that Translate-Train may be superior approach to mVQ to Translate- Test. We note, however, that in our early experi- ments, we find that Translate-Tred -rain inferior to Translate-Test as an adaptation approach for En- glish VQA models. For instance, the answer of finetuned BLIP2 to the French question "Outre les fleurs roses, quelle autre couleur y avait-il dans le jardin?" ("Besides pink flowers, what other color was there in the garden?") is "pink" while the cor- rect answer is "blanc" ("white") wrong both in terms of language and semantics. It is not immedi- ately obvious how to adapt English VQA models with, for example, vocab and tokenizers that overfit the English language. This again suggests that the design of these multimodal models would benefit from having multilinguality in mind from the start,
No daptation via Translate-Test. Can existing English VQA models work out of the box? Table 8, we find that the answer is no. Expectedly, the models perform well on French, which is closer to English than other languages are.
Simple MPT on xGQA. Can our modeling ap- proach be extended to the cross-lingual setting in xGQA (Pfeiffer et al., 2022)? We report this result in Appendix D.
Conclusions
We take initial steps toward multilingual VQA by proposing scalable solutions on both data creation and modeling fronts. We create multilingual VQA benchmark in 7 diverse languages to drive modeling progress on multilingual VQA. We estab- lish strong unified and open-ended VQA models that work well on 13 languages as well as bench- mark state-of-the-arr models. For future work, we would like to expand native-language question gen- eration that is done in limited scope and have single one for all target answers.
Single-Language vs. Multi-Language Training, Different Training Datasets. In Table 6, our Sim- ple MPT model performs similarly or better than each of the Single- gests that modern models are capable of learning from related languages. We also find that translated COCO is overall the best training data source. We attribute this to (i) the fact that VQ²A was used to generate VQºA-COCO, and (ii) VQAA-COCO is generally more robust in the cross cross-datasct setting (Changpinyo et al., 2022). However, VQ²A-CC3M is unable to outperform VQA2.0 despite (i): apply- ing VQ²A the noisy alt-texts in CC3M (Sharma et al., 2018) is prone to errors that would only be exacerbated by automatic MT.
baselines. This sug-
Acknowledgments. We would like to thank Jialin WI, Kenton Lee, Tomer Levinboim, Nan Ding, Sarah Laszlo, Doron Kukliansky, and Tania Bedrax Weiss for their feedback and discussion. Word clouds are generated from https://wwit jasondavies. cttpiiiss.co//orccoodd//

---

Metric
Model
Language
fr
hi
IW
th
zh
Translate- Train
Simple MPT
36.6
36.2
55.1
40.6
42.3
50.0
30.3
Accuracy
OFA
35.6
13.3
41.5
31.7
26.4
29.5
17.6
Translate- Test
BLIP2
48.7
20.1
63.3
49.2
39.0
49.0
28.0
Translate-Traie Train
Simple MPT
91.5
102.0
62.0
78.6
89.6
86.7
68.4
CIDEr
OFA
88.3
49.4
66.3
78.6
60.4
70.2
51.1
Translate- Test
BLIP2
121.8
67.6
88.1
93.8
79.1
91.5
65.7
Translate-Tran
Simple MPT
45.0
47.9
57.9
42.8
49.4
57.7
37.7
ROUGE-L
OFA
47.5
27.2
52.0
44.8
38.3
44.0
31.4
Translate-
BLIP2
62.6
32.6
67.2
52.6
47.4
53.6
39.9
Table 7: Results Soft Metrics. Accuracy (%), CIDEr (x 100), and ROUGE-L (x 100) of Simple MPT, OFA, and BLIP2. All are finetuned on VQA2.0. Best results are bold.
Metric
Model
Language
hi
IW
10
2h
OFA
35.6
13.3
41.5
31.7
26.4
29.5
17.6
Translate-Test
BLJP2
48.7
20.1
63.3
49.2
39.0
49.0
28.0
Accuracy
OFA
35.6
0.3
0.3
0.3
4.8
0.0
No adapt
BLIP2
48.7
0.3
3.9
7.0
2.3
OFA
88.3
49.4
66.3
78.6
60.4
70.2
51.T
Translate-Test
BLIP2
121.8
67.6
88.1
93.8
79.1
91.5
65.7
CIDEr
OFA
83.3
4.8
3.7
4.8
12.6
7.8
3.6
No adapt
BLIP2
121.8
28.0
5.1
4.0
12.1
16.0
8.2
OFA
47.5
27.2
52.0
44.8
38.3
44.0
31.4
Translate-Test
BLIP2
62.6
32.6
67.2
52.6
47.4
53.6
39.9
ROUGE-L
OFA
47.5
2.6
4.4
4.8
5.9
7.3
6.6
No adapt
BLIP2
62.6
12.5
6.2
2.9
5.4
10.8
7.3
Table Results without adaptation for Translate- Test, Accuracy (%), CIDEr (x 100), and ROUGE-L (x 100) of OFA and BLIP2 with Translate-Test and without (No adapt).
References
Proceedings of the Third Conference on Machine Transla- tion: Shared Task Papers.
Pranav Aggarwal and Ajinkya Kalc. 2020. Towards zero-shot minssllingual image retrieval. arXiv preprint urXiv:2012.05107.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Emest Valveny, C. V. Jawahar, and Dimos- thenis Karatzas. 2019. Scene text visual question answer- ing. In ICCV.
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, Anirud- dha Kembhavi. 2018. Don't tt just assume; look and an- swer: Overcoming priors visual question answering. CVPR.
James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary. Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Me Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NNumPy programs.
Harsh Agrawal, Karan Desai, Yufci Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruy Batra. Devi Parikh, Stefan Lec, and Peter Anderson. 2019. nocaps: novel object captioning at scale. In ICCV.
Emanuele Bugliarello, Fangyu Liu, Jonas Pfciffer, Siva Reddy, Desmond Elliott, Edoardo Maria Ponti, and Ivan Vulić. 2022. IGLUE: A benchmark for transfer learning across modalities, tasks, and languages. In ICML.
Jean-Baptist Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Husson, Karel Lene, Arthur Mensch. Katie Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. arXiv preprint ar Xrxiv:2004. 14198.
Soravit Changpinyo, Doron Kukliansky, Idan Szpektor. Xi Chen, Nan Ding, and Radu Soricut. 2022. All you may need for VQA are image captions. In NAACL.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019. Synthetic QA corpora generation with roundtrip consistency. In ACL.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12M: Pushing web-scale image- text pre-training to recognize long-tail visual concepts. In CVPR.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual question answering. In ICCV.
Xi Chen, Xiao Wang. Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Alexander Good- man, Adam Gryener. Basil Mustafa, Lucas Beyer, Alexan- der Kolesnikov, Joan Puigserver, Nan Ding, Keran Rong. Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thap- liyal, James Bradbury. Weicheng Kuo, Mojtaba Seyedhos- seini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anclia Angelova, Xiaohua Zhai, Neil
Pratyay Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2021. WeaQA: Weak supervision via captions for visual question answering. In Findings of ACI-IJCNNLP.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In

---

Houlsby, and Radu Soricut, 2023. PaLI: A Hoindly-calles multilingual language-image model. In ICLR.
Kushal Kafle and Christopher Kanan. 2017. visual question answering algorithms. In ICCV.
analysis of
Xinlei Chen, Hao Fang, Tsung-Y Yi Lin, Ramakrishna Vedan- tam, Saurahh Gupta, Piotr Dollár, and C. Lawrence Zitiick 2015. Microsoft COCO Captions: Data collection and evaluation server, arxiv preprint 100vv:554.00015
Andrej Karpathy and Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In CVPR.
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El Abu-El-Haija. Alina Kuznetsova, Hassan Rom, Jasper Ui- jlings, Stefan Popov, Shahab Kamali, Matteo Malloci. Jordi Pont-TTuset, Andreas Veit, Serge Belongie, Victor Gomes, Abhinav Gupta, Chen Sun, Gal Chechik, David Cai, Zheyun Feng, Dhyanesh Narayanan, and Kevin Mur- phy. 2017. OpenImages: A public datasct for large-scale multi-label and multi-class image classification. Dataset available from https::/g.co//aataattttt coltataseVopeninuge.
Jaemin Cho. Jie Lei, Hao Tan, and Mohit Bansal. 2021. Uni- fying rision-and-langugge tasks via text generation. In ICML.
Karan Desai, Gaurav Kaul, Zubin Aysola, and Justin Johnson. 2021. RedCaps: Web-curated image-text data created by the people, for the people. In NewriPS.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Untertiner, Mostafa Dehghani, Matthiss Minderer. Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlsby. 2021. An image is worth 66 16 words: Transformers for image recognition scale. In ICI.R.
Ranjay Krishna. Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen. Yannis Kalan- tidis. Li-Jia Li, David A. Shamma, Michael Bernstein, and 1. Fei-Fei. 2017. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. IJV,, 123(1):32-33
Desmond Elliott Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. Proceedings of the Second Conference on Machine Translation.
Alexander Ku, Peter Anderson, Roma Patel, Eugene Ie, and Jason Baldridge. 2020. -aro---cros-Room Multilingual vision-and-languaae navigation with dense spatiotemporal grounding. In EMNLP.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper R. R. Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Tom Duerig, and Vittorio Ferrari. 2020. The open images dataset V4: unified im- age classification, object detection, and visual relationship detection at scale. I/CV,
Desmond Elliott, Stella Frank. Khalil Sima' an, and Lucia Specia, 2016. Multi30K: Multilingual English-German German image descriptions. Proceedings of the Sth Workshop on Vision and Language, ACI.
Yash Goyal, Tejas Khot, Douglas Summers-Stap Stay. Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Ele- vating the role of image understanding in visual question answering. In CVPR.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins. Ankur Parikh, Chris Alberti, Danielle Ep- stein, Illia Polosuktin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wee Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoe Le. and Slav Petrov. 2019. Natural Questions: a benchmark for question an- swering research. TACL, 7:453-466.
Jiaxi Gu, Xiaojun Meng, Guansong Lu, Lu Hou, Minzhe Niu, Hang Xu, Xiaodan Liang, Wei Zhang, Xin Jiang, and Chunjing Xu. 2022. Wukong: 100 million large-scale chinese modal pre-training dataset and a foundation framework. arXiv preprint urXiv:2202.0676.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. BLIP-2: Bootstrapping aanuuag-image pre- pre-training with frozen image encoders and large language models. arXiv preprint 2rXiv:2301.12599
Danna Gurari, Qing Li, Abigale J. Stangl, Anhong Guo, Chi Lin. Kristen Grauman. Jiebo Luo, and Jeffrey P. Bigham. 2018. VizWiz Grand Challenge: Answering visual ques- tions from blind people. In CVPR.
Linjie Li. Jie Lei, Zhe Gan, and Jinging Liu. 2021. Adversar- ial VQA: A new henchmark for evaluating the robustness of vqa models. In ICCV.
Or Honovich, Leshem Choshen, Roce Aharoni, Ella Neeman, Idan Szpektor. and Omri Abend. 2021. @²: Evaluating factual consistency in boowiedge-grounde dialogues via question generation and question answering. In EMNLP.
Xirong Li. Chaoxi Xa. Xiaoxu Wang. Weiyu Lan, Zhengx- iong Jia, Gang Yang, and Jieping Xu. 2019. COCO-CN for cross-lingual image tagging. captioning, and retricval. IEFE Transactions on Multimedia, 21(9):2347-2360.
Drew A. Hudson and Christopher Manning. 2019. GQA: A new dataset for real-world visual reasoning and compo- sitional question answering. In CVPR.
Chin Yew Lin. 2004. ROUGE: package for automatic
evaluation of summaries. In Text Sumnarization Branches Out.
Justin Johnson. Bharath Hariharan, Laurens Van Der Maaten, Fei-Fei, C. Lawrence Zitnckk, and Ross Girshick. 2017a. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
Tsung-Y Yi Lin. Michael Maire, Serge Belongie, Lubomir Bour- dev, Ross Girshick, James Hays, Pictro Perona, Deva Ra- manan, C. Lawrence Zitnick, and Piotr Dollár, 2014, Mi- crosoft COCO: Common objects in context. In RCCV.
Melvin Johnson, Mike Schuster, Quoe Le,,, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Femaanda Vié- gas. Martin Wattenberg, Greg Corrado. Macduff Hughes. and Jeffrey Dean, 2017b. Google's Ieells multilingual neural machine translation system: Fnabling zero-shot translation. TACL, 5:339-351.
Fangyu Liu, Emanuele Bugliarello, Edoardo Maria Ponti, Siva Reddy, Nigel Collier, and Desmond Elliott. 2021. Visu- ally grounded reasoning across languages and cultures. In EMNLP.

---

Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-VQA: visual question answering benchmark requiring external knowledge. In CVPR.
Amanpreet Singh, Vivek Natarajan. Meet Shah, Jiang. Xin- lei Chen, Dhruv Batra, Devi Parikh, and Mareus Rohrbach. 2019. Towards VQA models that can read. In CVPR.
Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Mare Najork. 2021. WIT: Wikipedia-bann based image text dataset for multimodal multilingual machine learning. In SIGIR.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. 2011 1. Im2Text: Describing images using I million captioned photographs. In NIPS
Jonas Pfciffer. Gregor Geigle, Aishwarya Kamath, Jan- Martin O Steitz, Stefan Roth, Ivan Vulié, and Iryna Gurevych. 2022. xGQA: Cross-lingual visual question answering. In Findings of ACL.
Ashish V. Thapliyal, Jordi Pont- Tustt, Xi Chen, and Radu Soricut. 2022. Crossmodal-3600 A massively multi- lingual multimodal evaluation dataset. arXiv preprint arXiv:2205. 122552.
Jordi Pont- Tusct, Jasper Uijlings, Soravit Changpinyo. Radu Soricut, and Vittorio Ferrari 2020, Connecting vision and language with localized narratives. In ECCV.
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus-based image descrip- tion evaluation. In CVPR.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yangi Zhou, Wei Li, and J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR.
Jianfeng Wang. Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. 2022a, GIT: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205. 14100.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In ACL.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022h. OFA: Unifying architectures, tasks, and modalities through a simple sequence toree-tt--seeeee oo-equuence learning framework. In ICML.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In EMNLP.
Mengye Ren, Ryan Kiros, and Richard Zemel. 2015. Ex- ploring models and data for image question answering. NIPS.
Xin Wang, Jiawei Wu, Junkun Chen, Lei I.i, Yuan-Fang Wang, and William Yang Wang. 2019. VaTeX: A large-scale, high-quality multilingual datasct for hidco-and-languaae research. In ICCV.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Ar- naud Stiegler. Arun Raja, Manan Dey. M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma. Eliza Szezechla, Tacwoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jian Jiang, Han Wang. Matten Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao. Stella Biderman, Leo Gao. Thomas Wolf. and Alexander M. Rush. 2022. Multitask prompted train- ing enables zero-shot task generalization. In ICLR.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2022c. SIMVIMM Simple visual language model pretraining with weak supervision. In ICLR.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V. Le. 2022. Finetuned language models are zero-shot learners. In ICLR.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al- Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mTS: A massively multilingual pre-trained text-to- text transformer. In NAACL.
Christoph Schuhmann, Richard Vencu, Romain Beaurnntt, Robert Kaczmarczyk, Clayton Mullis. Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. LAION-00M:: Open dataset of CLIP-fittred 400 million image-text pairs. arXiv preprint Xrxiv:21 1100221111.121 14.
Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. 2021. Just ask: Learning to answer questions from millions of narrated vidcos. In ICCV.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-
OKVQA: A benchmark for visual question answering us- ing world knowledge. arXiv preprint arXiv:2206.0177
Yuya Yoshikawa, Yutaro Shigeto, and Akikazu Takeuchi. 2017. STAIR captions: Constructing large-scale Japanese image caption datasct. In ACL.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Sorieut. 2018. Conceptual Captions: cleaned, hyper- nymed, image alt-text dataset for automatic image caption- ing. In ACL.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi 2019. From recognition to cognition: Visual commonsense reasoning. In CVPR.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICLR.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Fei-Fei Li. 2016. Visual7W: Grounded question answering in images. In CVPR.
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Al- berto Lopez Magana, Wojciech Galuba, Devi Parikh, and Douwe Kicla. 2021. Human-adversarial visual question answering. In NeurlPS.

---

Gentral
Arsrect Amnosation
and
hive cudlal
wher the Iacarnect Aloust Corract
coclenggie Use man
you umues lite annee cornet
that the eatuer reguine to speciabed tmcolcdge dge you should mamted ouertion
and priviie
cansi custorianzerr
osnissigg hil corediees.
specttic
Qupotion Annoratton
Cornect
Whing on Chmume
Caroect
yartie Correst aot to shlw nes
Jung spec
ko retueells soog
Aunliguous FUU sevavv (Jex deferdee below).
"Wh
A "eileipiinn
Viatina image baseer dinaue Chinaad's
special mooieeeee ge fel how
Vaiing saurder
from ahowla Trow noweda diued"
parks 2017
Drie andig vakor an nee whet aney sggh when
or 'cuan''
you can sustio baewd the reage the quntor when oy
Incarrecs
The sreve cos soril lra Earruct or Alouss Correct.
Alwovt Correct
The ceulie Conrect nou ropevva then syntecte crove
Usid vian castor is aebigenas, Secorract, or
rening and chabt
Answer Expanss on and/or Standardization
J.ng speckc Winngin mosiing for dlee works
Provide possible srewers Separate each snewer by "II"
sporil biliingg offees pypppeiiit mad prote' "oete" perfeot'
mining conminer pab ben' 'sann and "hunt rec'"n tumee mee'))
Then mulppe surtoss forma cornec This, prodde gaieline for major
progeeggoon tris
typos lolw.
"Yossoo: Es sa briet to possittte
Exglatc
"yee" or "ro" jost 'yes, k:
Fett toe alo uss his remoow IAU senttve terms, sach an these releed non tng.
come coses, thond" queldonoo "so"
more natural
For
say (Thero khand"
Assigucss
The pasdbur buply doo ty to seseers that we aarsouted oe disggimmntt
"il und "talli"
"Numbart Use both dight and bast with and withuut thort noun. Do not ads adjectivee
Teserrent
The sustir nertal uta any of above Unanly thi the quander diws
Esjict
"2. "woo. pooplc "wo pospe" use a chort noun not "wo tal people") Adciioral pronoura (deperding the moums) addi ceal authi
yore
to
The:
The suution shm See What sunts au Rokorr aedd
Exarplo:
used
"Coler: Use short syrtucticsll correct form.
collactian
Esgast
"green" (ie "gren towe') "slck and whith and wwite and bleck"
crngencitty othiser WWhe on the upe' when taae lo now then 3 slege the
That
Alc odd the word "odoo" is a preft.
We aay Riie (toker grear) ci
What counts as RAl-senstive?
When the tatle?
both sald
bi ewere of protected categoris included rass, calor othec natirnal onigi: ugec reliiion gioue
whhro the "oot the beo' and "on roof en buth vil cl
pregrancy,
"What kine": Use your Jegggeg soms are "Arstiguoua" and some are OK.
cilaanatip, anowtry. montal cmaue proeecedd in ttary or whtton staise pipcica morti cikabilty med ical condition geneso informaioon or cheraderi poos for hooo af fimilyy by membery staus os-3 vitiin domoosto vidence, samusl amauhtr shiliing or any other ta prani bited under festrnl, stite, local low. (AAloaet "Aalee)
tasragancher
Figure 6: Detailed Instructions on Answer Annota- tion as well as Answer Expansion and Standardiza- tion.
queequaciin
itale that coneet then ls OKC but the quedt in assuree prede group and iiss wrong undear, please mork RN or fx using most Correct.
the
proseded group ipenden
and
the quenton saice predict or describe trait a protected group, mart RAA materr what
The Church
O: Y aibill uno craik sur le mur?
"Wh'"
A: cui
Sinder lo staone it NOT OK usw seae rnder people (AMaan" 'Aaar))
(C) then cooo wal?
to ahown it OK to use ge gendr or ape to reter people since the ouecton's Intert not for prediding or doocriing a bud prokodted growp
Figure 5: Detailed Instructions on Question Annota- tion
exdit whe mm Y?
What grone the pirtur?? wrtuonj
A Considerations and Limitations
crotte nizo own?
Our dataset is intended to be used for research-only purposes.
A: youl
boilding Anabbe
Our pipeline takes in an image caption as in- put. Image captions may have mistakes and bi- ases, which could be further amplified by machine learning models used by our approach. In particu- lar, we use generative models for automatic ques- tion generation and machine translation that may create outputs with incorrect or nonfactual con- tents or outputs with Translationese artifacts. We have mitigated this manually via human in the loop and automatically via the caption-un caption-unswer answer consis- tency check (cf., Sect. 3.2). Note that the English VQ"A (Changpinyo et al., 2022) that we leverage in our pipeline also has similar filtering using the round-trip consistency check via question answer- ing. Together these significantly improve the cor-
fat fori inforsecc pa marginea drumulu??
lugin, lupin aftosstnu mow
kinguiete, linguinje,
What Rowers
the of the
rood?
kihtn, bike kypinw fowwrs puayes dhewwrs paple ssop aage purms foowwsy
Q: Moee Moru?
A: "ammsne tu lauiss, Buile fandlo and un, undt, 4 4rmmintt,, thuanis, shallo, dheedic
(0: Where are dosts?
bank, lina by the liwu ths ovei
twosth basty
C: St 1e 90 #1 65
A: A: k:
(Q. Whor ow/r is do dar the Ailt the puple on??
Figure 7: Additional MaXM examples.

---

Capy copparen dken with aliww mear souze on white
Weat oder place the carpasdoo ah aerved an??
plute carpacsio ao on?
Cagi A Amooro shot of pains loaf oundorr.
C: this macro shar?
Cap intirnn F.
garzn which ermm are sycuuhh st stry
Cop Corpoctio boru owes resooonmeenn dans ascletto
A yes
Tcukkal palm loowes
Caps cloee-up velrccee in the backppoond
the
Oap Beef ca puoddo with opasoninng ana white plato.
exsevingg
Swet carpacoo with whath
Gorpaciio boodf oves quo??
A palm loavea
G: Tuyslafour teow??
C: Whal leaves are the photo?
Tachidu
A. yeu
P.
Q: uia difinh A aeg ser ggtce
Q. la tiere creah video?
dos aramalcy
o eookcrramen des herbo, hettee anerradiuuee
aronsstes)
C: Iayilfffurr tv?
shila
plrtoe??
A. ET, st. di et
Q. urg Rive wid ert de ghow it?
accidert pickue?
Gap: Primeglaa un ou for furis albe pe care sur surt sarvie is subjin de came crudl vooi cu susan priclip cospa loculà manut..
Cas: Iu with fiagmanta road
pain
Q: The front what o otject fell the raad?
Cap: Close frrely glore
o: Wiat sind the is the photo?
Q. Frnordiss Sogor bnneerous??
the tale a place where customerrs roast thair own moat.
yrem yo newd?
How A fiie
hpr
O: #mooddraz hrnquumua?
mun urnm pra new?
Gur A andle
"pm "yr
roadl) Accar blaor sor automoolee Mast chu
Figure 8: Additional examples on our approach multilingual VQA data generation. Green, yellow, and red texts correspond to "Correct", "Almost Correct", and "Incorrect," respectively.
photo
nating?
Q: Aside from the egg: what else on the plate?
A: watfle, salad, bacon,
knife
a: Queile ast la taille du singa?
(C) How the monkey?)
Qi ser gve
A: impr atth siar, amar, smant, अнाकााi,,, सााा,, डmेr,, aiev (Q: What blve this scene? A: board, piliar, sky, the sky. sky, sky)
Q: पियर apya मंरिर forine (Or What type Buddra temple is deppcceor7)
Q: 'soonaaanduuu A: infaafuu, tomumns resindiwns ufuu, tauninule
Q: towen to monnts omiors noss?
(Q: How birds fight over the
What's parked the runway? trucks, dock bridge trucks, trucks)
A:
Q: Trenul pertocalia opreşts garä?
Figure 9: Examples of Collection questions.
(Q) Does the orange trin atop at the station?)
rectness and fluency of our pipeline. In addition, we explicitly mark examples that can be consid- ered cesponsibleeA--sensiiee
unnunnnj unnnonnn/uuuuuuuni wwwuuunnuuuuu
(Q: background is snail in?)
but not necessarily incorrect; see Sect. B for details and examples.
Another type of biases is the low coverage of par- ticular types of answers, resulting from the image captions not mentioning the absences of objects or properties. We have also taken a step toward mitigating this. See Sect. 3.3.
Q: B
(Q: Whet the shape of the mireret carving?)
Figure 10: Examples of Ambiguous questions that we flagged and filtered out.
Finally, we select diverse set of languages, alle- viating typological, genealogical, and geographical language biases presented in the VQA research community.
(XM3600) (Thapliyal et al., 2022). Open Images (Krasin al., 2017: Kuznetsova al., 2020) and the multilingual captions in XM3600 are numan-curated and cleaned, which mitigates the risks that MaXM would contain information
We mainly
use
-rossmodal-3600

---

We also highlight the richness of our candidate answers. For the "where" question in Thai, 10 answers count as correct. Similarly, the Romanian example in Fig. provides multiple diverse surface forms for "coffee with cream."
Q: What gender is the ricer of the bicycle?
Fig. 8 additional examples the Chinese one in Fig 2. These examples showcase the efficiency of our annotation process. They also provide concrete examples of "Almost Correct." For instance, in the middle example, the Thai translation of "What leaves are the photo?" is not neutral because it contains an Honorific particle 9; it ends with "khá" which signifies sign of respect to the ad- dressee and indicates that the sex of the speaker is female. Finally, these examples provide a glimpse of sources of errors. For instance, it VQ²A that hallucinates "in the video" in the Hindi example on the right.
Y a--4 une femme sur photo?
(C2 there a woman in the photo?)
Q: m''a'an minn
(Ot How would you doscrbbe the red-haired gid in the picture? prefty)
Q reek este baretul din fotografie?
(Q: What race is the bay the photo??
Collection Examples. Fig. 10 provides examples of "Collection" questions. We keep these questions as we believe they are useful in practice and as way encourage the community to work on better automatic evaluation metrics for this type of questions.
Or twdruaenn wwwp://mwluueaaaaaaaaaaas
(Ct What is the gender of porson in an olf costume?)
Figure 11: Examples Responsible-A--sensttie questions that we flagged and filtered out. Faces are hidden.
Ambiguous Examples. Fig. 9 provides examples of "Ambiguous" questions that we filter out. Rea- sons include object being too small (English) or irregular (Chinese), determining sizes being sub- jective (French), and not enough context (Hebrew, Romanian). "What kind/What type" questions are particularly difficult to answer and tend to be am- biguous.
that names or uniquely identifies individual people or offensive content.
B Human Verification and Modification B.1 Annotation Guideline
Responsible-AI-ssnniie vides examples of tions that we filter out. These cases often as- sociated with directly asking for the information about or describing a particular gender race, involving an incorrect assumption about such pro- tected attributes (c.g., girl vs. woman in the Hebrew example).
Examples. Fig. 11 pro- ques-
We provide our general instructions and detailed instructions on question annotation Fig. 5, where we explicitly ask the annotators to be wary of Responsible- -l--ssiit---sesstiivvee questions. Fig. 6 pro- vides detailed instructions on answer annotation and answer expansion and standardization.
B.2 Additional Examples
Additional Examples. Fig. 7 provides additional examples to the ones in Fig. 1. Again, we highlight the richness and diversity of our questions. For instance, it requires recognizing a cross under oc- clusion (French), type of vegetables (Hindi), the Arabic language (Hebrew), and type of flowers (Romanian). Some of these examples are specific particular languages; it would be difficult for other language speakers to answer the Hebrew ex- ample (or the Chinese example in Fig. 1, which requires OCR).
C Simple MPT
In this section, we describe Simple MPT, a lightweight model for mVQA in detail.
Design. Much the previous work on VQA is built for English. Further, VQA is often formulated as vocab-based VQA, a classification task into a pre-defined space of top (English) answer vocab- ulary; see, e.g., (Antol et al., 2015; Goyal et al.,
Pheeps: //on, sikipedia, or wwwwii/Taaiii.p.m whw.eaihihhhhrttfte Jhanerifics

---

Finetuning
Question Language
Model
Dataset
bn
de
id
ko
pt
zh
M3P (Pfeiffer et al., 2022)
GQA
58.4
17.6
24.8
18.7
19.7
26.7
24.3
19.7
mBERT< (Pfeiffer et al., 2022)
GQA
56.3
13.4
32.4
19.8
19.9
31.5
25.5
26.2
Single-I Language
VQA2.0
43.1
37.9
39.6
40.4
33.9
40.3
39.3
39.7
Simple MPT
VQA2.0
41.5
38.6
40.3
39.5
38.7
39.8
39.5
39.5
Simple MPT
VQ'A-COCO
36.6
34,3
36,1
35.5
35.1
34,6
34.5
33.4
Simple MPT
VQ'A-CC3M
34.0
30,9
33.3
33.2
32.5
32.1
32.0
32.7
PaLI (Chen al., 2023)
VQA2.0
54.2
50.0
52.2
50.6
50.4
51.3
50.3
50.6
Table 9: Zero-Shot Results on xGQA. Accuracy (%) of our Simple MPT models trained on different training datasets as well as our Single-Language baselines and baselines from (Pfeiffer al., 2022). Best results are bold. Second best italicized.
Common Crawl-based dataset covering 101 lan- guages. This allows leverage multilingual language understanding (for the questions) and gen- eration (for the answers) from the get-go. To adapt mT5 to the VQA task, we prepend patch embed- dings from the image to the question tokens. particular, we encode the image pixels using Vi- sion Transformers (ViT) (Dosovitskiy et al., 2021). We use ViT-L16 and mT5 mT5-Large in all of our exper- iments. Both mT5 and ViT are trained together in an end-to-end fashion to predict the target answer for each image-question pair, using the standard cross-entropy loss.
<Answer>
mT5 Encoder-Decodee
Image Patch
Image Patch 196
Answer (en.fr.hi.w.rro.thhhhh <Question>
ViT
Image Pixels
Multi-Language Prompted Training. We resort to multi-task prompied//nstruction training (Sanh et al., 2022; Wei et al., 2022), where task corre- sponds to VQA for a particular language. For the input question (question) in language (lang), we construct the prompt "Answer in (lang): (question)" and use it as the text input to our model, similar to modification to the input in Google's Multilin- gual Neural Machine Translation System (Johnson et al., 2017b). Such a design for multi-task learn- ing makes extending VQA to multiple languages simple; as data for additional languages become available, one can simply add them to the pool without the need for architecture changes.
Figure 12: Our Simple MPT model used in our experi- ments. We leverage ViT (Dosovitskiy et al., 2021) and mT5 (Xue al., 2021) and train them together end-to- end.
2017). The main drawback of this approach is its inability to deal with rare answers through language compositionality. Recent work considers VQA as generation (Cho et al., 2021; Wang et al., 2022c: Alayrac et al., 2022; Wang et al., 2022a), capable of open-ended VQA. We adopt this scalable and flexible modeling approach to mVQA as the language coverage increases. In particular, pose single open-ended VQA model for multiple languages. Our proposed formulation more desir- able than existing ones since it takes advantage of both compositionality in individual languages and the relationship among related languages. To this end, we first describe an encoder-decoder architec- ture for VQA in the open-ended generation setting. Then, describe how train this model for multiple languages. This is summarized in Fig. 12.
pro-
Implementation Details. We use the Flax imple- mentation (Bradbury et al., 2018). For training both our 2(lang) and 2en models, we use Adafac- tor (Shazeer and Stern, 2018) with B1 of 0 and second-monte exponential decay of 0.8. We linear warmup of 1K steps with peak Iearn- ing of learning rate of le-3 and inverse square-root decay. We set the ViT dropout rate 0 and the mT5 dropout rate 0.1. We train each model with data parallelism using 16 Cloud TPU Pods", each with a batch size of 512, for 100K steps. We use
Open-Ended VQA. Our starting architecture is mT5 (Xue al., 2021), a multilingual variant of TS (Raffel et al., 2020). mT5 is an encoder-decoder transformer-bas architecture, pre-trained on a
Chttps://clen http://lloo.oooolleeee/o con/tpu

---

standard image resolution of 224x224, We use the maximum input length of and the target output length of 8.
We consider three datasets for training Simple MPT, all are translations of existing large-scale En- glish VQA datasets to the 13 languages covered by MaXM and XGQA. We use the Karpathy training split (Karpathy and Fei-Fei, 2015) for VQA2.0 and VQºA-COCO and the standard training split for VQ3A-CC3M.
Additional Results
Our Simple MPT in the main paper predicts the answer in the same language as the question. Here, explore if our Simple MPT also be useful for the cross-lingual setting in xGQA (Pfeiffer et al., 2022), where the model always predicts the answer English.
Similar to MaXM, XGQA test-only bench- mark, the testdev split of 12,578 question-answer pairs per language from 398 images in 8 languages don.bn.de,ddkkopptruuu
To evaluate Simple MPT this dataset, we use the setting in the main paper: training VQA2.0, VQ"A-COCO, and VQ-A- COCO but do not translate the training answers.
We also use the prompt "Answer in en: (question)" instcad of "Answer (lang): (question)".
Table 9 reports the results. Our baselines are M3P and mBERTAda from (Pfeiffer et al., 2022).
Note that both M3P and mBERTAda have access to the (English) GQA training data (Hudson and Manning, 2019), where model does not. On the other hand, they do not use translated data as in our case. We outperform multilingual zero-shot baselines on all non-English languages, without access to English GQA labeled data. This further confirms that our unified approach to mQQA is ef- fective. In addition, unlike on MaXM, VQA2.0 is the best pre-training data source. We attribute this to the fact that VQA2.0 and xGQA share COCO images (Lin et al., 2014). This highlights the utility of MaXM additional out-of-domain test-only VQA evaluation data.


Reviews:
[{"review_id": "sKRIXCncYn", "reviewer": "Reviewer_NrNZ", "summary": "", "questions": "- Did you notice if this approach generates some artifacts or some wrong pattern?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R] I would appreciate an in-depth qualitative analysis of the dataset.**\n\nWe are unsure what *an in-depth qualitative analysis of the dataset* entails, but we would be happy to provide them, if the reviewer wouldn’t mind clarifying this further. Here’s what currently is available:\n\nSection 4.4 (along with Table 4 and Figure 4) discusses the statistics of our dataset and how our dataset is complementary to/different from xGQA. Appendix B2 further highlights the richness of our dataset.\n\nAppendix B2 also discusses special questions: “Collection” questions, which we included and “Ambiguous” and “RAI-sensitive” questions, which we discarded but believe can serve as guideline for future annotation processes. \n\nQualitative examples are provided in Figure 1, Figure 2, Figure 8, Figure 7, Figure 9, Figure 10, and Figure 11, with Figure 2 and Figure 8 focusing on the annotation process. Figure 3 provides a qualitative illustration of why we focus on deriving our benchmark from the XM3600 dataset.\n\n**[Q] Did you notice if this approach generates some artifacts or some wrong pattern?**\n\nMost artifacts come from translation, which we discuss and mitigate in Section 4.2. Figure 2 and Figure 8 provide examples of how we correct/remove these artifacts.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- They propose contributing to this research area with a new dataset that could help expand mVQA to a multilingual setting.\n- They perform an exhausting set of experiments and ablations.\n- The paper is well-written and easy to follow.", "reasons_to_reject": "- I would appreciate an in-depth qualitative analysis of the dataset.", "questions_to_authors": "- Did you notice if this approach generates some artifacts or some wrong pattern?", "paper_topic_and_main_contributions": "This paper proposes new solutions for multilingual visual question answering (mVQA). Specifically, they contribute to this area by presenting (i) a framework to reduce the cost in terms of manual annotation and (ii) a test benchmark for mVQA called MAVERICS-XM3600 (MaXM) translated in 7 languages.\n\nTheir proposed framework can be summarized in four steps. First, they automatically translate a non-English caption to an English one. Second, starting from the translated caption, they generate a set of QA pairs. Third, they translate the QA pairs to the target language, and lastly, they filter the translated pairs removing all the answers not correlated with the original caption.\n\nTo evaluate and test this new benchmark (MaXM), they trained a model named “simple MPT” using different data settings, and then they used existing models (e.g., OFA, BLIP2, PaLI) showing the benchmark capabilities.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}, {"review_id": "cuRAllH19F", "reviewer": "Reviewer_PM6i", "summary": "", "questions": "A. Could you elaborate on the advantages of the proposed multilingual dataset over a straightforward mix of single-language datasets? Given the relatively small size of the proposed dataset, its labor-intensive creation process, and its exclusive use as a benchmark, how does it prove superior?\n\nB. If monolingual VQA models, such as OFA or BLIP2, were fine-tuned akin to SimpleMPT of PALI using multilingual data, would this potentially enhance their performance on the mVQA task?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R1] The analysis of the dataset and the multilingual context presented in the paper is brief and lacks sufficient detail. A more comprehensive examination, particularly of the VQA data from different languages, would be highly beneficial...**\n\nWe are unsure of how our current analysis is brief and lacks sufficient detail. But we would be more than happy to make it more comprehensive, if the reviewer wouldn’t mind clarifying this further. Here’s what is currently available:\n\nSection 4.4 (along with Table 4 and Figure 4) discusses the statistics of our dataset and how our dataset is complementary to/different from xGQA. Appendix B2 further highlights the richness of our dataset.\n\nAppendix B2 also discusses special questions: “Collection” questions, which we included and “Ambiguous” and “RAI-sensitive” questions, which we discarded but believe can serve as guideline for future annotation processes. \n\nQualitative examples are provided in Figure 1, Figure 2, Figure 8, Figure 7, Figure 9, Figure 10, and Figure 11, with Figure 2 and Figure 8 focusing on the annotation process. These figures include phenomena specific to particular languages (L892-896), for example, Chinese text in Figure 1. \n\nThe discussion of language-specific context also appears in Section 4.1 and Figure 3 (why we focus on deriving our benchmark from the XM3600 dataset), L328-333, L533-537, Figure 5 (Instructions which are specific to particular languages based on our analysis), and Appendix B2.\n\n**[R2] The benefits of the proposed test dataset, as opposed to a mixed dataset comprising many single-language VQA data, are not well defined or articulated**\n\n**[QA] Could you elaborate on the advantages of the proposed multilingual dataset over a straightforward mix of single-language datasets? Given the relatively small size of the proposed dataset, its labor-intensive creation process, and its exclusive use as a benchmark, how does it prove superior?**\n\nExisting single-language VQA datasets ***do not exist*** for non-English languages, except for xGQA (L100-110), which focuses on the cross-lingual setting and reasoning (L403-431). Thus, one cannot currently perform *a straightforward mix of single-language datasets*. In theory, a mix of single-language VQA datasets may likely be more biased toward intra-cultural rather than inter-cultural phenomena (e.g., the ability to answer questions in French for images that cover Chinese culture, L296-306).\n\nFurther, the standard approach to curating VQA data for a particular language is based almost entirely on human annotators, thus resource-intensive (L46-51). Our approach is superior because it is more automatic, actually providing a much *less labor-intensive* path to the multilingual setting (L51-58).\n\nFinally, establishing an evaluation benchmark is the first step toward measuring progress in multilingual VQA. It is often regarded as the most difficult task in data creation as it requires both the data to be clean and the metric reliable. The requirement is much less strict for training data creation, where much of human annotation steps in Section 4.3 can be relaxed or removed. Its exclusive use as a test-only benchmark follows previous work such as nocaps and Crossmodal-3600, which aims to test the ability of models to generalize (to new concepts or languages, and in our cases to the new image domain as mentioned in L307-314) \n\n**[QB] If monolingual VQA models, such as OFA or BLIP2, were fine-tuned akin to SimpleMPT of PALI using multilingual data, would this potentially enhance their performance on the mVQA task?**\n\nIn our early experiments, we found that Translate-Train is inferior to Translate-Test in adapting for these English VQA models. For instance, The answer of finetuned BLIP2 to the French question *“Outre les fleurs roses, quelle autre couleur y avait-il dans le jardin?* (*“Besides pink flowers, what other color was there in the garden?”*) is *“pink”* while the correct answer is *“blanc”* (*“white”*) — wrong both in terms of language and semantics. More generally, it is not immediately obvious how to adapt English VQA models with, for example, vocab and tokenizers that overfit the English language. This suggests that the design of these multimodal models would benefit from having multilinguality in mind from the start.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposed a new benchmark evaluating the mVQA ability of current VQA models. The dataset utilized has undergone thorough human verification to ensure its quality. Furthermore, the paper offers an exhaustive comparison of monolingual and multilingual VQA models, utilizing the newly proposed benchmark as the evaluative standard.", "reasons_to_reject": "1. The analysis of the dataset and the multilingual context presented in the paper is brief and lacks sufficient detail. A more comprehensive examination, particularly of the VQA data from different languages, would be highly beneficial for the wider community.\n\n2. The benefits of the proposed test dataset, as opposed to a mixed dataset comprising many single-language VQA data, are not well defined or articulated.\n\n3. The test data size is still somewhat constrained, limiting the potential scope and applicability of the findings.", "questions_to_authors": "A. Could you elaborate on the advantages of the proposed multilingual dataset over a straightforward mix of single-language datasets? Given the relatively small size of the proposed dataset, its labor-intensive creation process, and its exclusive use as a benchmark, how does it prove superior?\n\nB. If monolingual VQA models, such as OFA or BLIP2, were fine-tuned akin to SimpleMPT of PALI using multilingual data, would this potentially enhance their performance on the mVQA task?", "paper_topic_and_main_contributions": "The paper introduces a method predicated on translation to generate multilingual Visual Question Answering (VQA) data. Following a rigorous process of human verification and filtration, it provides a benchmarking tool to gauge the multilingual comprehension capabilities of VQA models. Moreover, the paper presents a lightweight, multilingual VQA baseline model, alongside an analysis of the gathered data.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Coqb9U1JXv", "reviewer": "Reviewer_jdG6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**[R1] The multilingual data curation pipeline is…a somewhat straightforward extension of VQ2A by adding some translation steps (section 3.2)...The benchmark creation process contains little description of novelty….The MPT model could be a useful off-the-self baseline for the benchmark but bears little novelty.**\n\nAt a high level, we think it is desirable that our framework and proposed models are simple and cost effective. In other words, the significance of our contributions lies in effectively achieving our goal of scaling up VQA to non-English languages without sacrificing simplicity while controlling cost.\n\nAt the same time, our approach may *look* straightforward in hindsight but in practice it is difficult to *get the detail right*. This has a lot to do breaking down the task into small pieces and suitably assigning machines or humans to each piece (L342-350). For instance, we develop an automatic filtering approach in Section 3.2 to deal with the fact that existing translation models are not optimized for translating short answers (L237-240). We develop a simple answer bias mitigation approach for certain answer types (Section 3.3). Finally, we did have to iterate our annotation protocol and quality checks multiple times to arrive at what may appear to be straightforward in Section 4.3 and Appendix B.\n\nOverall, our experience indicates designing a rigorous annotation protocol for multilingual VQA to balance quality and efficiency is non-trivial, and it is more complex than “adding translation to VQ^2A”.\n\n\n**[R2] Experimental results are somewhat pointless**\n\nWe clarify each point below. We hope to convince the reviewer that these experiments were conducted with sound, scientific purposes and will use additional space to make their messages clearer if the paper gets accepted.\n\n**1) OFA-Large and BLIP2 are SOTA VQA models, but they are only for English not for other languages….unclear whether they are good enough baselines for the experiments.**\n\nWe put effort into adapting these English VQA models to the multilingual setting to ensure that our baselines are strong.\n\nFirst, we take a standard multilingual adaptation approach *Translate-Test* (L465-469), making them much stronger as baselines; see Table 8 for results without adaptation where these baselines perform poorly.\n\nIn addition, we found that Translate-Train is inferior to Translate-Test as an adaptation approach. For instance, The answer of finetuned BLIP2 to the French question *“Outre les fleurs roses, quelle autre couleur y avait-il dans le jardin?* (*“Besides pink flowers, what other color was there in the garden?”*) is *“pink”* while the correct answer is *“blanc”* (*“white”*) — wrong both in terms of language and semantics. More generally, it is not immediately obvious how to adapt English VQA models with, for example, vocab and tokenizers that overfit the English language. This suggests that the design of these multimodal models would benefit from having multilinguality in mind from the start.\n\nFinally, our adaptation approach in the xGQA setting outperforms approaches in xGQA, including OSCAR and mBERT. See Appendix D.\n\n**The proposed MPT model is not as good as SOTA models. As shown in Table 5, it is better than OFA, comparable to BLIP2, and worse than PaLI. This paper argues that it is much smaller, but there is no comparison in terms of model sizes and compute.**\n\nWe are happy to provide such detail, which we hope serves as a context for comparison and clarify L503-510.\n| Model | Parameter count |\n|-------------|:----------:|\n| OFA-Large | 470M |\n| BLIP2 | 11B|\n| Simple MPT | 1.5B |\n| PaLI | 17B |\n\n**The analysis experiments in Table 6-8 show somewhat obvious (Table 6, 8) or mixed results in different metrics (Table 7). For example, it is not surprising that the English trained model works well with French but no with other different languages.**\n\nWe argue that these ablation and sensitivity analysis experiments and results convey the thoroughness of our benchmarking results in Table 5. In particular,\n\nTable 6 shows what happens when we deviate from the training data source in Table 5 (language-wise and captions in the case of VQ^2A-based data). First, Simple MPT in Table 5 does not underperform Single-Language baselines, allowing us to have one unified model with strong results but low cost. Second, the result supports the use of manually-curated VQAv2 in Table 5 for training; it does not contain biases from VQ^2A (used to create our benchmark) but works well enough to beat noisy VQ^2A-CC3M.\n\nTable 7 shows what happens when we deviate from our main evaluation metric. Accuracy — what we adopted in Table 5 — is correlated well with other soft metrics, so this metric (combined with our Answer Expansion and Standardization) is useful. We do not see this as a mixed result.\n\nTable 8 shows that, without adaptation, English SOTA VQA models perform poorly, in contrast to strong results of adapted models we report in Table 5.\n\n***Minor: …The comparison with xGQA starting from line 421 lacks ground…***\n\nThank you! We will expand this discussion and provide supporting evidence, including the fact that GQA is explicitly designed for “reasoning” (e.g., abstract in [1]) and an analysis (e.g., Section 5.3) in [2], which our approach is based on.\n\n[1] https://arxiv.org/abs/1902.09506\n\n[2] https://arxiv.org/abs/2205.01883", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed MAVERICS-XM3600 dataset should be a valuable asset for multilingual VQA research. ", "reasons_to_reject": "1. Although this work summarizes the main contributions in the last paragraph of section 1, the contributions (ii)-(iv) are rather weak. \n(1) The multilingual data curation pipeline is largely based on VQ2A (Changpinyo et al., 2022). It is a somewhat straightforward extension of VQ2A by adding some translation steps (section 3.2). \n\n(2) The benchmark creation process contains little description of novelty. \n(3) The MPT model could be a useful off-the-self baseline for the benchmark but bears little novelty. \n\n2. Experimental results are somewhat pointless. \n(1) OFA-Large and BLIP2 are SOTA VQA models, but they are only for English not for other languages. It is unclear whether they are good enough baselines for the experiments. One may try some models (OSCAR or mBERT) used in the xGQA paper. \n(2) The proposed MPT model is not as good as SOTA models. As shown in Table 5, it is better than OFA, comparable to BLIP2, and worse than PaLI. This paper argues that it is much smaller, but there is no comparison in terms of model sizes and compute.  \n(3) The analysis experiments in Table 6-8 show somewhat obvious (Table 6, 8) or mixed results in different metrics (Table 7). For example, it is not surprising that the English trained model works well with French but no with other different languages. \n\n3. Some minor weaknesses \n(1) The comparison with xGQA starting from line 421 lacks ground. For example, it argues “GQA’s questions are restricted“ but “MaXM’s questions are more general” with no evidence or examples.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "1. This work contributes a novel multi-lingual VQA datasets in seven languages, named MAVERICS-XM3600. \n- There is only one other dataset in the same kind of mVQA named xGOA. The comparison between MAVERICS-XM3600 and xGOA is described in detail in section 4.4. \n\n2. It also proposes a simple and lightweight mVQA model named MPT that shows competitive mVQA performance for the benchmark. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Justification_for_Ethical_Concerns": "Since the dataset is not available for the review, it is not decidable for sure whether there is no ethical issue in this work.\n"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper creates a dataset MaXM which is a multilingual question answering benchmark. The dataset serves as a test-only mVQA dataset to validate the mVQA model performance. The reviewers have concern on the technical contribution from this paper and also request some detailed analysis on the proposed dataset. But overall, this dataset would be useful to the community as an evaluation set. The author has promised a release of this dataset in the paper.
Decision: Accept-Findings