Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering (2024)

Abstract: Building a reliable visual question answering (VQA) system across different languages is a challenging problem, primarily due to the lack of abundant samples for training. To address this challenge, recent studies have employed machine translation systems for the cross-lingual VQA task. This involves translating the evaluation samples into a source language (usually English) and using monolingual models (i.e., translate-test). However, our analysis reveals that translated texts contain unique characteristics distinct from humanwritten ones, referred to as translation artifacts. We find that these artifacts can significantly affect the models, confirmed by extensive experiments across diverse models, languages, and translation processes. In light of this, we present a simple data augmentation strategy that can alleviate the adverse impacts of translation artifacts. Our code is available at https: //github.com/ddehun/VQA_translation.

heading: 1 Introduction
text: Visual question answering (VQA) aims to answer an open-ended question by reasoning about a given image (Agrawal et al., 2015). Despite recent advances in vision-language (VL) modeling, building proficient models across various languages is still challenging. This issue primarily arises from the limited availability of annotated datasets, which are predominantly in high-resource languages such as English. Although recent efforts in developing multilingual VL models can address this issue to some extent (Zhou et al., 2021; Qiu et al., 2022; Li et al., 2023b; Geigle et al., 2023), training on datasets in the target languages is still crucial for enhanced model performance in those languages.
To mitigate the data scarcity issue, cross-lingual transfer learning focuses on utilizing extensive datasets in a source language (typically English) to
* Equal contribution
build models effective in a target language (Artetxe et al., 2020; Bugliarello et al., 2022). One of the popular approaches, namely translate-train, translates training samples into individual target languages and uses them to train models for target languages. This approach is advantageous as it does not perform translation during inference, but it requires training individual models for each target language. Furthermore, recent VL models (Singh et al., 2022; Liu et al., 2023b; Li et al., 2023a), which are mostly tailored in English, are not suitable for the translate-train approach. Another widely adopted approach, called translatetest, translates test samples written in target languages into the source language and uses VL models of the source language for the inference. These translation-based approaches have shown remarkable performance in cross-lingual tasks.
Despite the effectiveness of translation systems in cross-lingual VL tasks, using machine-translated texts as input inevitably introduces a mismatch between the training and inference phases. In the translate-test approach, models are trained on human-written texts but evaluated on machinetranslated texts. This distribution shift could hurt
5193
the generalization of models to different languages (Yu et al., 2022; Wang et al., 2022). For instance, as illustrated in Fig. 1, leveraging machinetranslated texts might lead to undesirable model outcomes, even when both questions convey the same meaning. In this paper, we refer to artifacts in translations that cause such unwanted behaviors as translation artifacts. We argue that the translation artifacts have been overlooked in previous crosslingual VQA studies despite their significance.
To explore the effect of mismatched data distribution on cross-lingual VQA, we alleviate this mismatch in the data origins1 by employing machinetranslated texts in both training and inference. Our investigation focuses on the translate-test, which can take advantage of strong monolingual models and efficiently serve multiple target languages with a single VL model. Our results reveal that models trained on machine-translated texts generally outperform those trained on human-written texts, increasing the averaged accuracy over languages and models from 51.82 to 53.14 points. This improvement, as confirmed by our qualitative analysis, is primarily attributed to the subtle nuances in translated texts (i.e., translation artifacts). Our comprehensive study covers various components in cross-lingual VQA, including 14 models, 13 languages, 5 machine translation systems, and diverse translation setups. We also observe that recent VL models (Li et al., 2023a; Dai et al., 2023) integrated with large language models also suffer from translation artifacts. Finally, we present simple data augmentation techniques, verifying their effectiveness in both human-written and translated texts.
Our contribution can be summarized as follows:
1. This is, to our knowledge, the first study to investigate translation artifacts in cross-lingual visual question answering.
2. We provide extensive analyses across a variety of languages and models, providing a foundation for future research.
3. We present simple yet effective data augmentation strategies using translated texts.

heading: 2 Related Work

heading: 2.1 Cross-lingual VQA
text: The study of VQA has predominantly focused on English and other high-resource languages (Zhu
1We refer to origin as a writer of texts (i.e., human or machine translation system).
et al., 2015; Agrawal et al., 2015; Goyal et al., 2016; Marino et al., 2019; Schwenk et al., 2022). To extend the use of VQA to various languages, researchers have introduced cross-lingual transfer techniques (Ni et al., 2021; Zhou et al., 2021; Nooralahzadeh and Sennrich, 2022; Liu et al., 2023a). One effective approach involves pretraining VL models on multilingual image-text pairs and then fine-tuning them on English VQA, which is known as zero-shot transfer (Jain et al., 2021; Lee et al., 2022; Zeng et al., 2022; Chen et al., 2022, 2023; Li et al., 2023b). Another popular approach that leverages advanced machine translation shows promise in adapting to various languages. The translate-train involves translating the text pairs from high-resource languages to the target language for finetuning (Thapliyal and Soricut, 2020; Zeng et al., 2022; Chen et al., 2023; Li et al., 2023b). On the other hand, the translate-test uses machine translation to convert test data into English, allowing the use of English-only models for inference (Jain et al., 2021; Bugliarello et al., 2022; Pfeiffer et al., 2022). This latter approach is particularly beneficial, considering the strong performance of existing English-only models (Singh et al., 2022; Li et al., 2023a; Dai et al., 2023; Gao et al., 2023).

heading: 2.2 Translation Artifacts
text: Translated texts often exhibit unique characteristics, referred to as translation artifacts or translationese (Gellerstam, 1986; Lembersky et al., 2012; Baker, 2019; Edunov et al., 2020). These characteristics can negatively influence model outcomes due to their stylistic deviations from the original texts (Volansky et al., 2015; Bizzoni et al., 2020; Yu et al., 2022). Yang et al. (2021) examined the representation discrepancies between English and other languages in the translate-train approach for various language understanding tasks. Wang et al. (2022) explored the effects of translation artifacts on model evaluation in cross-lingual summarization. To mitigate the effects of translation artifacts, researchers have proposed various methods, such as incorporating machine-translated sentences in training (Artetxe et al., 2020; Yu et al., 2022; Wang et al., 2022) or utilizing specific tags to differentiate between original and machine-translated texts (Marie et al., 2020; Riley et al., 2020; Wang et al., 2021).
However, the effect of translation artifacts on cross-lingual VQA remains largely underexplored, leading to potential risks and unexpected outcomes. While previous research has primarily focused on
the application of machine translation in VL models (Thapliyal and Soricut, 2020; Zeng et al., 2022; Bugliarello et al., 2022; Pfeiffer et al., 2022; Changpinyo et al., 2023; Chen et al., 2023), our study aims to identify the presence and impact of translation artifacts within cross-lingual VQA. We find that these translation artifacts are prevalent in VL models handling both image and text modalities.

heading: 3 Translation Artifacts in Cross-lingual Visual Question Answering
text: In this work, we analyze the impact of machine translation on cross-lingual VQA tasks, especially on the translate-test approach. To this end, we vary the origin of training datasets into human and a machine translation (MT) system and then observe how this change affects the model behavior. We use roundtrip (RT) translation to generate machine-translated training samples from the source language- English.2

heading: 3.1 Experimental Setup

heading: 3.1.1 Data
text: We use xGQA (Pfeiffer et al., 2022), a representative benchmark for the cross-lingual VQA task. Each sample in the dataset consists of an image, a structured question related to the image, and an answer. The training set is derived from the original English GQA dataset (Hudson and Manning, 2019) and consists of 72k images and 943k samples. The evaluation sets cover seven different languages - Bengali (bn), German (de), Indonesian (id), Korean (ko), Mandarian (zh), Portuguese (pt), and Russian (ru) - and is manually translated from the balanced test-dev set of the English GQA dataset by human annotators. The evaluation set consists of 398 images and 12,578 samples, and all images in xGQA datasets are sampled from the Visual Genome dataset (Krishna et al., 2017). Further details on the dataset are described in Appendix A.

heading: 3.1.2 Models
text: We conduct experiments with all multilingual and monolingual VL models addressed in Bugliarello et al. (2022). Specifically, for multilingual models, MUNITER (Qiu et al., 2022), XUNITER (Qiu et al., 2022), UC2 (Zhou et al., 2021), and M3P (Ni et al., 2021) are used. For monolingual Englishonly models, LXMERT (Tan and Bansal, 2019), UNITER (Chen et al., 2020), VILBERT (Lu et al.,
2Afterward, the source language refers to English.
2019), VisualBERT (Li et al., 2020), and VLBERT (Su et al., 2019) are used. All models are based on transformer (Vaswani et al., 2017) architecture, and both image and text are fed to the network simultaneously. In addition, we conduct experiments with recently proposed monolingual English VL models - BLIP-2 (Li et al., 2023a), InstructBLIP (Dai et al., 2023), and FLAVA (Singh et al., 2022). More details are in Appendix B.
For the cross-lingual transfer of multilingual models, the following approaches are considered: zero-shot, translate-train, and translate-test. The zero-shot approach trains a model on the original English training set in the GQA dataset and directly uses it to infer evaluation samples in the target language.3 The translate-train approach trains individual models for each target language on a translated training dataset. The translate-test approach trains a single model on an English training dataset and uses it for the evaluation of target languages along with a translation system. For monolingual models, only the translate-test approach is evaluated.

heading: 3.1.3 Training Dataset from Different Origins
text: For the translate-test, we finetune all models described above on English GQA datasets from two different origins individually: Human and MT. For Human, we use the original xGQA training set. For MT, we use the roundtrip (RT) translation to obtain training samples that are written by an MT system. We use NLLB (Costa-jussà et al., 2022) as the MT system for RT translation.4 The German (de) is used as a pivot language during RT translation (en→de→en). For the zero-shot and translate-train, we use the original English dataset and the dataset translated from English to individual target languages, respectively. More details about translation processes are in Appendix C.

heading: 3.1.4 Evaluation dataset
text: Source Language For English evaluation, we use the official evaluation set released by Pfeiffer et al. (2022) (en). Besides, we also make translated versions of English evaluation sets through RT translation (en∗). This process is to understand the impact of data origins on models more comprehensively. Target Languages For zero-shot and translatetrain evaluations, the target language questions released by Pfeiffer et al. (2022) are used. For the
3The zero-shot denotes that the language of evaluation samples differs from the language used in the finetuning phase.
4facebook/nllb-200-3.3B is used.
translate-test evaluation, each question in the target language should be translated into English. In this work, we use an official translate-test evaluation set (Bugliarello et al., 2022) generated by the Google Machine Translation (GMT) system.

heading: 3.1.5 Implementation Details
text: For finetuning VL models, we follow hyperparameters reported in Bugliarello et al. (2022) for a fair comparison. Specifically, all models are trained for 5 epochs, and the batch size and initial learning rate are set to 256 and 4e-5, respectively. AdamW (Loshchilov and Hutter, 2018) is used for optimization. All models are trained with a classification head on top of image-language representation. We evaluate models after every training epoch and choose the best checkpoint based on its
accuracy on the original English development set. More implementation details are in Appendix D.

heading: 4 Results and Analysis

heading: 4.1 Main Results
text: Multilingual Models Fig. 2 presents averaged evaluation results of multilingual models with different cross-lingual transfer approaches. The models show decreased accuracy when transferred to languages other than English. For instance, the average accuracy is 56.4 for the original English dataset, but are 51.6 and 51.3 for translate-train and translatetest approaches, respectively. Among the different cross-lingual transfer approaches, translate-train and translate-test are comparable, while the zeroshot approach usually performs worse.
Misaligned Data Origins in Translate-Test Table 1 presents translate-test evaluation results of models with different training data origins. Regarding models trained on human texts, FLAVA usually performs better than other models. Regarding the effects of different training data origins, we observe that models generally show higher accuracy when the origins of training and evaluation datasets are matched. Specifically, for the original English evaluation set, models trained on human texts consistently perform better than ones trained on MT texts. On the contrary, for the translate-test, in which all questions are generated by MT systems, models trained on MT texts outperform those trained on human texts. By only aligning the data origins of training and evaluation sets, the averaged translatetest scores across models and languages are increased from 51.82 to 53.14. Note that this trend is consistent in RT-translated English evaluation set (en∗), where the average score increases from 50.37 to 52.81. Based on our results, we suggest a reconsideration of factors contributing to lower scores of target languages in cross-lingual VQA, indicating that data origin misalignment, alongside translation errors, could significantly impact the success of language transfer.

heading: 4.2 Human Analysis
text: We next analyze translated questions in the evaluation set to examine where the increased performance of models trained on MT texts comes from. To this end, we annotate translate-test evaluation samples in which a model trained on human texts makes wrong predictions, but a model trained on MT texts makes correct ones. Note that we only consider the translate-test samples in which both models with human and MT texts correctly predicted the paired human-written English samples to avoid wrong predictions arising from sample complexity. UC2 is selected as a VQA model, and 200 questions from the Korean (ko) evaluation set are annotated. Two native speakers annotate the MT errors in each question, and the annotation schema is based on multidimensional quality metric (MQM) ontology (Mariana, 2014) following Moghe et al. (2023). More details about human annotation and annotated examples are in Appendix G.
As shown in Fig. 3, although the model trained on human texts changes its prediction from the correct to wrong ones, a majority of translated questions (>60%) do not contain crucial translation errors. In terms of translated questions without
translation errors, most of them can be regarded as paraphrased sentences of their paired English questions as shown in Fig. 4. Based on these results, we confirm that models trained on human texts often make wrong predictions about translations that convey similar meanings to human ones. In other words, subtle differences between human and translated texts caused by translation artifacts indeed influence model behavior.

heading: 4.3 Are Translation Artifacts Actually Presented in Translated Questions?
text: To enhance our comprehension of the increased translate-test accuracy of models trained on MT texts, we scrutinize model performances across samples categorized by the prevalence of translation artifacts. Specifically, we quantify the humanlikeness of each translated question and assess its influence on VQA models. To derive the humanlikeness score ph(x) for every translated question x, we train a classifier based on RoBERTa (Liu et al., 2019), designed to discern whether a given English question is written by a human or generated by round-trip (RT) translation.5 After training, the classifier assigns a score ph(x) on how likely humans write each translate-test evaluation sample. We
5The accuracy of the trained classifier is 86.97 in a classbalanced validation set.
then categorize the translated evaluation samples into two groups with the same size - human-like and NMT-like - based on their respective humanlikeness scores ph(x). The accuracy of VL-BERT models trained on different data origins (i.e., human and NMT) is compared in these groups. More experimental details are in Appendix H.
From the results in Table 2, we find that the model trained on human texts performs worse when the input questions are less likely to be written by humans. Specifically, the average accuracy across different target languages is 53.56 for human-like questions but 48.60 for NMT-like questions. Conversely, a model trained on MT texts shows less accuracy degradation on NMT-like questions compared to the one trained on human texts. These results indicate that VQA models are prone to make more errors when the given question is not likely to be written by humans, and training on (RT-) translated texts can alleviate such problems.
We next delve into the lexical diversity within each question group, inspired by previous findings that the translated texts are often simpler than human ones (Zhang and Toral, 2019; Wang et al., 2022). Specifically, we use two metrics to measure the lexical diversity of translated questions used in the translate-test approach: (1) Token Type Ratio (TTR) calculates the ratio of unique words over all words in the sentence, (2) Lexical Density (LD) calculates the ratio of content words (words that likely to convey significant meaning - nouns, verbs, adverbs, and adjectives) over all words in the sentence. Lexical diversity results in Table 3 indicate that NMT-like questions generally exhibit less
variety in word usage. We suspect that such characteristics in translated questions make a difference in training and evaluation, resulting in performance degradation of models.

heading: 4.4 Representation Analysis
text: Our previous observations reveal that translated texts exhibit distinct impacts compared to human ones when they are used for training and evaluation. We next analyze whether these different characteristics of translated texts appear in model representation. Specifically, we compare the representations of training samples from different origins (human and MT) against evaluation samples. As evaluation samples, we use the translate-test samples from different target languages and English evaluation samples written by human or RT translations. We employ the penultimate layer output of visualBERT as the sample representation, and the Fréchet Inception Distance (FID) (Heusel et al., 2017) score is used to quantify the representation distance between training and evaluation samples. Additionally, to assess the impact of finetuning on model representation, we analyze VisualBERT at checkpoints before and after finetuning.
As shown in Fig. 5, we observe clear trends indicating that translated samples cluster more closely in the model representation space. In detail, all translate-test samples show lower FID scores with MT training samples than human ones. Note that these trends are consistent for both pretrained and finetuned models. These results indicate that characteristics shared within translated texts also affect
the internal representation of VL models.

heading: 4.5 Varying NMT and Pivot Languages
text: Based on our previous results, we confirm that addressing the misalignment of data origins between training and evaluation is effective for the translatetest approach. We now aim to understand how these benefits vary with changes in the MT systems or translation setups. To this end, we conduct experiments by varying (1) the MT system used for translating the training and evaluation sets and (2) the pivot language during the RT translation. Varied MT systems We use the following four MT systems in our experiments: M2M-100418M/1.2B (Fan et al., 2021) and NLLB-200600M/-3.3B (Costa-jussà et al., 2022). Each MT system is used to make RT-translated training and translate-test evaluation sets. In detail, we use RT translation with different MT systems to make training sets, and the pivot language is fixed to German (de). All models described in Section 3.1.2 are individually trained on these four RT-translated datasets. For the evaluation set, we translate every
target language into English using different MT systems, resulting in four different evaluation sets.
Evaluation results are shown in Table 5. Notably, models trained on translated texts usually outperform those trained on human texts in translate-test sets. These results suggest that, despite a mismatch between the MT systems used for RT translation and the translate-test, leveraging RT translation for training remains advantageous for cross-lingual transfer. In terms of MT system comparison, models usually show higher accuracy when MT systems used to make training and evaluation sets are in the same model family. In the original English evaluation set, models with human texts perform best, followed by the ones with NLLB-200-3.3B texts.

heading: 4.6 Translation Quality of MT systems
text: From results in Section 4.5, we observe that the accuracy ranking in the original English set and the GMT translate-test set is NLLB-200-3.3B > NLLB200-600M > M2M-100-1.2B > M2M-100-418M. We suspect that this trend reflects the translation quality of the training data produced by each MT system. To corroborate this, we assessed these MT
systems using an IWSLT 2017 (Cettolo et al., 2017) benchmark, while maintaining the same translation direction as in RT translation (i.e., en→bn and vice versa). The IWSLT2017 dataset contains 8079 parallel sentences in these language directions, which involves multilingual text translation of TED talks. For evaluation, we utilized METEOR (Banerjee and Lavie, 2005), chrF (Popović, 2015), and SacreBLEU (Post, 2018) as evaluation metrics. As shown in Table 6, we observe the results which are clearly aligned with the previously observed trends. Further details of each metric are in Table 12. Varied Pivot Language in RT Translation We vary the pivot languages used in RT translation to make different versions of translated training sets. All target languages presented in xGQA datasets are selected as pivot languages. As an MT system, we use NLLB-200-3.3B to translate both training and evaluation samples. As shown in Table 4, models usually show higher accuracy when a pivot language matches its target language. This tendency is consistent with previous findings (Ni et al., 2022), where the texts within the same translation direction contain shared characteristics.

heading: 4.7 Experiments with MaXM dataset
text: We evaluate models trained on the xGQA dataset with MaXM (Changpinyo et al., 2023), a recently proposed evaluation-only benchmark for multilingual VQA. The MaXM dataset covers seven different languages: English (en), French (fr), Hindi (hi), Hebrew (iw), Romanian (ro), Thai (th), and Chinese (zh). Each evaluation sample consists of an
image, a question, and an answer. As the answers in the MaXM dataset are not exactly matched with the ones in xGQA that models are trained, we only use a question whose answer is either “yes” or “no”. More details about the MaXM dataset and full evaluation results are in Appendix A and Table 19, respectively. As shown in Fig. 6, we observe results consistent with the xGQA dataset. Training on RT-translated texts increases the translate-test scores except for Romanian (ro) cases.

heading: 5 Reducing the Effect of Translation Artifact on Cross-lingual VQA
text: Our findings demonstrate that training VQA models on translated texts induces higher accuracy in language transfer through the translate-test approach. Despite such gains, translated texts inevitably contain wrongly translated information due to the imperfection of MT systems. Moreover, as translations are known to be different from the naturally written human texts (Volansky et al., 2015; Zhang and Toral, 2019), training models solely on the translated texts may degrade overall performance. These problems can be observed in our previous results; in Table 1, the models trained on translated texts show a relatively low average score in the English evaluation set compared to those trained on human texts (56.91→55.13).
To resolve this, we leverage a simple data augmentation technique that uses both RT-translated texts and the original human-written texts for model training (MERGE). Furthermore, following Marie et al. (2020), we also adopt the ap-
proach that includes special tagging tokens in front of translated texts in both training and evaluation phases (TAG). As MERGE and TAG double the number of training examples, we reduce the total training steps to half for a fair comparison across methods. Results with data augmentation methods are in Table 7. The accuracy of the original English evaluation set is increased in both MERGE and TAG compared to solely using translated samples. The overall scores for the translate-test are also improved with data augmentation. These results indicate that augmenting training data with both human and MT texts is helpful for cross-lingual transfer while maintaining its performance on the original English texts.

heading: 6 Conclusion
text: In this work, we analyze the impacts of translation artifacts presented in machine-translated English texts for cross-lingual VQA. Through extensive experiments, we find that current VL models usually suffer from distributional shifts caused by translation artifacts during cross-lingual transfer, resulting in undesirable performance degradation. As a remedy, we conduct experiments with simple data augmentation strategies and observe consistent performance gains.
Our work focuses on translations that are semantically similar but written differently from human texts. In future work, we will explore mistranslation arising from context-free translation, where image information is not considered during a translation process. To this end, recently advanced multimodal translation systems can be utilized (Yao and Wan, 2020). Other important directions include considering a variance among different translations generated from a single text and devising an advanced training strategy to consider translation artifacts.
Limitations
Our study is mainly conducted on a translate-test approach for a cross-lingual VQA task. We recognize that some of our results may not generalize other tasks, like image captioning. Nevertheless, as reasoning over natural language and image is a crucial ability for vision-language models, we believe it is a fundamental step to comprehend the impacts of translation in the VQA task to transfer across different languages seamlessly. Besides, since we mainly consider the conventional finetunethen-evaluate pipelines, some experimental setups
do not directly apply to recent models that do not perform parameter updates for learning (e.g., GPT4V (OpenAI, 2023)). As discussed in Appendix J, we observe that these models also can suffer from translation artifacts to some extent when performing VL tasks. Performing extended analysis and proposals across diverse learning algorithms and models remains our future work.
Ethics Statement
Most of the models in our experiments are trained on English datasets only, so the generalizability towards other source languages is not examined. Besides, as the current MT systems are imperfect, training on translated texts may introduce unintended behaviors or favors to specific questions. Future research should investigate such undesirable bias in translations and VQA models.

heading: Acknowledgements
text: This work was supported by a grant of the KAISTKT joint research project through AI2X Laboratory, Tech Innovation Group, funded by KT (No. D23000019, Research on Multilingual Multicultural Vision-Language Representation Learning) and Institute for Information & communications Technology Promotion(IITP) grant funded by the Korea government(MSIT) (No.RS-2019II190075 Artificial Intelligence Graduate School Program(KAIST) and RS-2021-II212068, Artificial Intelligence Innovation Hub).

heading: A Dataset Details
text: xGQA (Pfeiffer et al., 2022) In our study, we used the English-balanced GQA (Hudson and Manning, 2019) training set for model training, which consists of 943k training examples and 72k training images. For model validation, the English GQA validation set, containing 132k samples and 10k images, is used. For evaluation, we used the balanced test-dev subset of the xGQA dataset, which includes 12,578 systematically structured questions with an average length of 8.5 words, associated with 398 images. The xGQA dataset extends the test-dev set of GQA by translating into seven different languages, each from a unique language family. In the translate-test approach, we used the official evaluation set released by Bugliarello et al. (2022), which translates samples written in target languages into English with the Google Machine Translation system. Further details on the xGQA dataset are provided in Pfeiffer et al. (2022).
MaXM (Changpinyo et al., 2023) The MAVERICS-XM3600 (MaXM) dataset, an evaluation-only VQA benchmark, originates from the Crossmodal-3600 dataset (XM3600) (Thapliyal et al., 2022) and consists of translation-based question-answer pairs. MaXM includes 7 languages which are chosen based on their typological, genealogical, and geographical diversity. The statistics of selected evaluation samples for each language are presented in Table 8.

heading: B Model Details
text: Table 9 summarizes the key characteristics of all models described in Section 3.1.2. For visual tokens, we utilize 36 image regions from a ResNet101 backbone (He et al., 2016), and 10 to 100 image regions from a ResNeXt-101 backbone (Xie et al., 2017). For BLIP-2, InstructBLIP,
and FLAVA, we use the official implementations released by authors. For other models, we use the implementation released by Bugliarello et al. (2022).

heading: C Translation Details
text: RT Translation We use roundtrip (RT) translation to make translated English training dataset. Unless otherwise specified, NLLB-200-3.3B is used as an MT system, and German (de) is used as a pivot language. Following Artetxe et al. (2023), we use stochastic and deterministic decoding strategies for RT translation. Specifically, for forward translation (en → de), we use nucleus sampling (Holtzman et al., 2019) with p = 0.9. For backward translation (de → en), we use beam search with beam size as 5. For both translation directions, the maximum number of repeated n-gram is set to 5.
Translate-Test Unless otherwise specified, we use the evaluation set released by Bugliarello et al. (2022) for a fair comparison. When constructing the translate-test evaluation set ourselves, as in Section 4.5, we use beam search with beam size 4.
Translate-Train We translate the original English training set into every target language in xGQA. NLLB-200-3.3B is used as an MT system for this process, and beam search is used with beam size 5.
D Implementation Details
For finetuning VL models, we follow hyperparameters described in Bugliarello et al. (2022) for a fair comparison. Specifically, we utilize the AdamW optimizer (Loshchilov and Hutter, 2018) with betas set at (0.9, 0.999) and ϵ=1e-8. The maximum number of tokens in the input sequence is set to 40, and the batch size is set to 256. The total training epochs are set to 5. The learning rate is set to 1e-4, and a linear learning late scheduler is used with a 0.5 warm-up epoch. For training, we used cross-entropy loss for all 1,842 labels available in the GQA dataset. In overall experiments, a single NVIDIA-A100 GPU with 40GB of memory is used for BLIP-2, InstructBLIP, and FLAVA, and a single model is trained in one day. Other models are trained with a 3090 RTX GPU with 24GB of memory and are trained in 5 hours. The experiments are implemented with PyTorch (Paszke et al., 2019).

heading: E Statistical Test Results
text: Based on the findings in Section 4.1, we observe an improvement in test accuracy when the data origins for training and evaluation are aligned. To demonstrate that this improvement consistently occurs in the translate-test, we train models three times with different seeds and report the average performance
of models in Table 16. Furthermore, we perform significance tests to demonstrate that this improvement consistently occurs in the translate-test. Specifically, we conduct significant tests on all the aggregated results as well as on the language-specific and model-specific results in Table 16.6 Our evaluation specifically compared the performance of models trained with MT texts against those trained with human texts in the translate-test.
First, we aggregate all the outcomes from models trained on machine-translated texts and compare them to those trained on human texts in Table 16. The results demonstrate a significant advantage in training with machine-translated texts over human texts, with a p-value of 6.49e-66.
6For significant tests, we use the paired t-test with α = 0.05.
Moving on to specific language results in Table 10, we gather all the model outcomes for each language except for English. We then compare the performance of models trained on machinetranslated texts to those trained on human texts.
When considering the performance of individual models, we collect the results from all evaluated languages for each model (i.e., collect all the results of each row in Table 16). We then compare the performance of models trained on machine-translated texts to those trained on human texts in Table 11.
As a result, we can observe that evaluating models trained on human-written English data with translated texts could negatively impact the generalization of models to other languages. By simply aligning the data origins for both training and evaluation sets, the overall performance in the translatetest can be improved.

heading: F Zero-shot Evaluation of VL models
text: Recent VL models like BLIP-2 (Li et al., 2023a) or InstructBLIP (Dai et al., 2023) can perform target downstream tasks without task-specific finetuning by relying on its language generation ability. To compare the effectiveness of finetuning on these models, we evaluate the models with and without finetuning by using xGQA evaluation samples whose answers are either “yes” or “no”. This subset contains 4,525 samples out of 12,578 total evaluation samples. For zero-shot evaluation, we prompt the model with task description as follows: ‘Answer the following question in “yes” or “no”.\n Question: <question> \n Answer: ’. Models generate the next token as an answer for the given question in the prompt with an image. We conduct post-processing steps including case-normalization or punctuation mark removal to derive the binary prediction of models. For model implementation, we use the models released by Wolf et al. (2020).
As shown in Fig. 7, although models exhibit
competitive zero-shot scores, their performance is lower than finetuned ones. These results imply that finetuning the models on task-specific datasets is also crucial for recent VL models. In this regard, it is still essential for the VL models to consider and address data origin misalignment presented in training and evaluation.

heading: G Human Evaluation Details
text: We first identified examples where questions, initially correct in English, became incorrect in the translate-test. Among these examples, we specifically focused on cases where the UC2 model, trained using the original English GQA dataset, provided incorrect results, but the UC2 model trained with RT-translated data generated correct responses. From the examples that conformed to these restrictions, we analyzed a subset of 200 examples.
Following Moghe et al. (2023), we annotated any machine translation (MT) errors in these examples, utilizing the Multidimensional Quality Metrics (MQM) ontology (Burchardt, 2013). This framework categorizes errors into a hierarchical structure, allowing for the evaluation of translations based on this hierarchy. Our analysis focused on 5 error types within the MQM ontology, including Mistranslation, Addition, Omission, Fluency, and Grammar. Two authors with a master’s degree or higher separately annotated each evaluation sample. The annotated examples from our case study are presented in Fig. 11.

heading: H Human-likeness Analysis Details
text: We use a confidence score of a text classifier to analyze the prevalence of translation artifacts in every translated question. The classifier is trained to discriminate whether the data origin of a given question is a human or MT system. Specifically, we finetune RoBERTa-base (Liu et al., 2019) to classify whether the given question is from the original human-written dataset or the translated dataset from another target language. The training epochs, batch size, and learning rate are set to 3, 24, and 2e5. The finetuned classifier assigns the confidence score ph(x) about how likely a human writes the question to each translated question in translate-test evaluation sets. This confidence score is regarded as human-likeness of translated questions.

heading: I Experiments with LLaMA-Adapter-V2
text: We examine whether LLaMA-Adapter-V2 (Gao et al., 2023), a recently proposed powerful VL model with a large language model, also suffers from translation artifacts for cross-lingual VQA tasks. To this end, we finetune LLaMA-AdapterV2 with different training options (Human, MT, MERGE, and TAG) and observe their results. Specifically, we add a classification head on top of end-of-sequence (eos) token representation in LLaMA and finetune it along with the unfrozen weights.7 The model is parameter-efficiently finetuned, where only a small portion of the total parameters are updated (15M). We use the official codes released by the authors8 for implementation, and LLaMA-7b (Touvron et al., 2023) with CLIP visual encoder (Radford et al., 2021) is used. The overall finetuning setups follow previously mentioned ones in Section 3.1.5. Note that we also finetune and evaluate models to directly generate the answer text, but the scores are usually lower compared to using the classification head.
As shown in Table 14, leveraging translated texts for training is beneficial to the translate-test approach of LLaMA-Adapter-V2, where the models trained on translated texts show higher accuracy compared to human texts. MERGE and TAG further improve accuracy in English and other target languages.
Besides, we also evaluate the model without finetuning on xGQA to probe its zero-shot ability. Since zero-shot classification with generation models requires roughly the number of forward passes with answer candidates, we choose evaluation samples whose label is either “yes” or “no”, and measure the probability of both tokens. Regard-
7LORA-BIAS-7B is used. 8https://github.com/OpenGVLab/LLaMA-Adapter/
tree/main/llama_adapter_v2_multimodal7b
ing the comparison with zero-shot and finetuning for yes/no question types in Fig. 8, the finetuned model scores better than the zero-shot approach. This result implies that although recent VL models show impressive zero-shot capability, finetuning on task-specific datasets is still required for better performance.

heading: J Experiments with GPT-4-Vision
text: In this study, we present experimental results of GPT-4-Vision (OpenAI, 2023), a cutting-edge VL model. We use 300 evaluation samples of yes/no questions described in Appendix F. We include all target languages and their corresponding original English questions. For evaluations in the target languages, inputs consist of questions either originally written in the target language or translated into English via GMT. The prompt format and the evaluation outcomes are presented in Fig. 15, and Table 13, respectively.
Our findings indicate that GPT-4 can serve as an effective multilingual VL model. Remarkably, its performance in all languages except Chinese exceeds that of English. Directly using the target language proves more efficient than relying on the translated source language, primarily due to the inherent errors in translation processes.
However, GPT-4 falls short of the finetuned monolingual models detailed in Appendix F. The direct comparison between GPT-4 and these models is nuanced, largely because of differences in evaluation settings.9 Despite these challenges, the translate-test with strong VL models yielded more favorable outcomes than using GPT-4, with scores of 63.43 compared to 76.4 and 75.89 for finetuned BLIP-2 and InstructBLIP models, respectively, Additionally, our qualitative analysis indicates that GPT-4 is also susceptible to translation artifacts, which can cause differences in predictions between human and MT texts. We present the qualitative results of GPT-4 on xGQA in Fig. 15 and 16.

heading: K Experiments on Image Captioning
text: We further probed whether the problematic issues raised by translation artifacts are also present in an image captioning task. To this end, we used a multilingual image captioning dataset, namely Crossmodal-3600 (Thapliyal et al., 2022), and a BLIP-2-large finetuned on the COCO dataset (Lin et al., 2014). In detail, we first generated English captions for all images in the test dataset and translated these generations into individual target lan-
9This complexity arises from the differences in the number of questions asked and the categorization of any response from GPT-4 other than “yes” or “no” as incorrect.
guages. This generate-and-translate process is designed to resemble the translate-test approach of cross-lingual VQA, where only an English-tailored VL model and a machine translation (MT) system were available. We then evaluated the translated captions with a reference-based metric (i.e., CiDEr (Vedantam et al., 2015)), and two types of answer captions in target languages were considered as a reference set: (1) Human-written reference captions presented in Crossmodal-3600, and (2) RT-translated reference captions created by RTtranslation (i.e., target language->en->target language) of the original reference captions. Using CiDEr, we calculated the similarity score between generated and reference captions.
As shown in Table 15, we observed clear trends that the generated captions achieve higher scores with RT-translated reference captions, which confirms that issues of cross-lingual multimodal tasks derived from translation artifacts are not just limited to VQA tasks. Intuitively, if both generated and RT-translated captions - translated texts - exhibit shared characteristics that deviate from humanwritten texts (i.e., translation artifacts), generated captions are likely to receive higher scores with RTtranslated references than human-written ones. We leave the impacts of our data augmentation strategies on these generative multimodal tasks as our future work.

heading: L Additional Results
text: Full results of Fig. 2 Fig. 9 presents full results of Fig. 2. Full results of Fig. 5 Fig. 10 presents full results
of Fig. 5 across different target languages. Full results of Table 5 Table 17 presents full results with varying MT systems for RT translation and a translate-test approach. Full results of Table 4 Table 18 presents full results with varying pivot languages used in RT translation. NLLB-200-3.3B is used as an MT system. Full results of Fig. 6 Table 19 presents the full results of different models on the MaXM dataset. NLLB-200-3.3B is used as an MT system for translate-test evaluation. Full results of Table 7 Table 20 presents the full results of models with different data sources.

heading: Future Work
text: We leave the impacts of our data augmentation strategies on these generative multimodal tasks as our future work