Venue: NAACL 2024

Lost in Translation? Translation Errors and Challenges for Fair Assessment of Text-to-Image Models on Multilingual Concepts

 Michael Saxon\({}^{\cancel{a}}\)   Yiran Luo\({}^{\cancel{b}}\)   Sharon Levy\({}^{\cancel{g}}\)

**Chitta Baral\({}^{\cancel{b}}\)   Yezhou Yang\({}^{\cancel{b}}\)   William Yang Wang\({}^{\cancel{a}}\)**

University of California, Santa Barbara \({}^{\cancel{b}}\)   Arizona State University \({}^{\cancel{g}}\)   Johns Hopkins University

_Equal contribution & corresponding: saxon@ucsb.edu, yluo97@asu.edu_

###### Abstract

Benchmarks of the multilingual capabilities of text-to-image (T2I) models compare generated images prompted in a test language to an expected image distribution over a concept set. One such benchmark, "Conceptual Coverage Across Languages" (CoCo-CroLa), assesses the tangible noun inventory of T2I models by prompting them to generate pictures from a concept list translated to seven languages and comparing the output image populations. Unfortunately, we find that this benchmark contains translation errors of varying severity in Spanish, Japanese, and Chinese. We provide corrections for these errors and analyze how impactful they are on the utility and validity of CoCo-CroLa as a benchmark. We reassess multiple baseline T2I models with the revisions, compare the outputs elicited under the new translations to those conditioned on the old, and show that a correction's impactfulness on the image-domain benchmark results can be predicted in the text domain with similarity scores. Our findings will guide the future development of T2I multilinguality metrics by providing analytical tools for practical translation decisions.

## 1 Introduction

With growth in the popularity of generative text-to-image (T2I) models has come interest in assessing their capabilities across many dimensions, including multilingual accessibility. The CoCo-CroLa [3] benchmark attempts to capture how well "concept-level knowledge" within a T2I model is accessible across different input languages. It compares the output image populations of a system under test when prompted to generate images of 193 tangible concepts in 7 _test languages_ to the images generated from a semantically equivalent prompt in a _source language_. It and similar benchmarks rely on correct translations for validity, lest "possessed" concepts be mistakenly assigned false negatives.

We find a strict _error candidate rate_ of 4.7% for Spanish (ES), 8.8% for Chinese (ZH), and 12.9% for Japanese (JA) in the CoCo-CroLa v1 (CCCL) concept translations through manual analysis by fluent speakers. These error candidates are not filtered by severity. While some candidates are severe translation errors that drive false negatives (Figure 1), others are marginal annotator disagreements that might not matter (Table 1). In this work, **we investigate when and why translation changes actually impact CCCL results** to improve future T2I multilinguality benchmarks. We:

* Write _candidate corrections_ for CCCL in ES, JA, and ZH, evaluated on four T2I models.
* Introduce a text-domain comparison metric \(\Delta\mathrm{SEM}\) to predict correction significance.
* Analyze our candidates by \(\Delta\mathrm{SEM}\) and image correctness improvement and apply impactful ones to CCCL as v1.1.
* Report insights and considerations for future semantic T2I evaluations we uncovered.

Figure 1: The CoCo-CroLa benchmark mistranslated concepts such as _bike_ in JA and _suit_ in ZH. With correct translations (right) AltDiffusion does in fact “possess” them; originally (left) they were false negatives.

 



## 2 Motivation & Approach

The **CoCo-CroLa** benchmark (CCCL) evaluates a T2I model's ability to generate images of an inventory of tangible concepts when prompted in different languages [14]. Given a tangible concept \(c\), written in language \(\ell\) as phrase \(c_{\ell}\), the \(i\)-th image produced by a multilingual T2I model \(f\) on the concept \(c_{\ell}\) can be expressed as:

\[I_{c_{\ell},i}\sim f(c_{\ell}) \tag{1}\]

The images generated in language \(\ell\) are considered _correct_ if they are faithful to their equivalent counterparts in the source language \(\ell_{s}\). This is measured by the CCCL benchmark by a **correctness metric** for a single concept \(c\) as the _crossconsistency_ score \(X_{c}(f,c_{\ell},c_{\ell_{s}})\):

\[X_{c}=\frac{1}{n^{2}}\sum_{i=0}^{n}\sum_{j=0}^{n}\mathrm{SIM}_{F}(I_{c_{\ell}, i},I_{c_{\ell_{s}},j}) \tag{2}\]

where we sample \(n\) images per-concept per-language (we use 9), and \(\mathrm{SIM}_{F}(\cdot,\cdot)\) measures the cosine similarity in feature space by image feature extractor \(F\). In practice, the default source language \(\ell_{s}\) is English and \(F\) is the CLIP visual feature extractor [35].

### Translation Errors in CoCo-CroLa

CCCL requires correct translations of each concept \(c\) from the source language \(\ell_{s}\) into a set of semantically-equivalent translations in each test language \(\ell\). saxon2023multitask built CCCL v1's concept translation list using an automated approach so as to allow new languages to be easily added without experts in each new language.

They used an ensemble of commercial machine translation systems to generate candidate translations and the BabelNet knowledge graph [67] to enforce word sense agreement. Unfortunately, this approach introduces translation errors (Table 1).

We check the Spanish, Chinese, and Japanese translations using a group of proficient speakers, following a protocol described in Appendix A.1.1, who identify a set of _translation error candidates_ that may not sufficiently capture a concept's intended semantics in English, for various reasons.

Some of the candidate errors, such as the error for _rock_ in JA (Table 1), represent severe failures to translate a concept into its common, tangible sense--it is incoherent to test a model's ability to generate pictures of rocks by prompting it with "rock music." However, other candidate errors, such as _father_ in ZH are still potentially acceptable translations, but deviate from the annotators' preferred level of formality or specificity.

To decide which corrections ought to be integrated in future T2I multilinguality benchmarks, quantifying both the significance of each translation correction is and its impact on the CCCL score for its concept is desirable.

### Quantifying Error Correction & Impact

Characterizing the _impact_ of a translation correction on model behavior is simple; we check \(\Delta X_{c}\), the change in the CCCL score going from the original concept translation \(c_{\ell}\) to the corrected \(c^{\prime}_{\ell}\),

\[\Delta X_{c}(c,\ell)=X_{c}(f,c^{\prime}_{\ell},c_{\ell_{s}})-X_{c}(f,c_{\ell}, c_{\ell_{s}}) \tag{3}\]

by comparing the generated population of images elicited from the corrected term \(I_{c^{\prime}_{\ell}}\) to the candidate translation error-conditioned images \(I_{c_{\ell}}\).

We quantify the significance of the translation correction as the _improvement in semantic similarity_\(\Delta\mathrm{SEM}(c_{\ell_{s}},c_{\ell},c^{\prime}_{\ell})\) using a text feature extractor \(F_{t}\) and cosine similarity metric \(\mathrm{SIM}(\cdot,\cdot)\)

\[\Delta\mathrm{SEM}=\mathrm{SIM}_{Ft}(c_{\ell_{s}},c^{\prime}_{\ell})-\mathrm{ SIM}_{Ft}(c_{\ell_{s}},c_{\ell}) \tag{4}\]

We use embeddings from the multilingual SentenceBERT [8] text embedder OpenAI CLIP-ViT-B32 model as \(F_{t}\).

\begin{table}
\begin{tabular}{l l l l l} \hline \hline Concept & Language & Original & Corrected & Reason for Correction \\ \hline Rock & Japanese & \(\Box>\gamma\) & \(\forall s\) & \(\exists\Rightarrow\gamma\), rokna, refers principally to “rock music” instead of stones in nature. \\ Flume & Spanish & _Ilman_ & _Jama_ & _Iamu_, though a correct translation force “flame,” coincides with the animal in English. \\ Ground & Japanese & \(\sharp\sharp\sharp\) & \(\sharp\sharp\) & \(\sharp\sharp\) refers to an electrical ground rather than the surface of the earth. \\ Table & Chinese & \(\Reamond\) & \(\Reamond\) & \(\Reamond\) & means a tabular format or a spreadsheet, not a four-legged furniture. \\ Milk & Japanese & \(\Reamond\) & \(\dashv\) & \(\sharp\) may mean breast or any kind of milk: \(\#!\) means the mild produced by cows. \\ Text & Spanish & _inferda_ & _rede_ & _compare_ & _friend alone more often means “store,“ _findade acamy_ specifies (cantping) tent. \\ Teacher & Japanese & \(\%\)\(\sharp\)\(



## 3 Results & Analyses

We generate output images using StableDiffusion I.4, 2.0, 2.1 (Rombach et al., 2022) and AltDiffusion (Chen et al., 2022), for all concepts corrected by our annotators in English, Spanish, Chinese, and Japanese, using both the original concept translations \(c_{\ell}\) from CoCo-CroLa v1 (Saxon and Wang, 2023) and the corrected translations \(c^{\prime}_{\ell}\). Model details are provided in Appendix A.4.

Figure 2 shows the relationship between \(\Delta\mathrm{SEM}\) and \(\Delta X_{c}\) for all corrected concepts for StableDiffusion 1.4, 2.0, 2.1, and AltDiffusion1. Note the pronounced, significant positive slope of the correlations between the two variables for AltDiffusion in all languages (4th row) and in Spanish for all models (third column). Here a positive slope means that _higher-improvement_ translation corrections (assessed by increased proximity to the English word in a shared embedding space) reliably correct the generated images more than the modest candidates.

Footnote 1: Error margins are 95% regression-fit confidence intervals.

These same high-slope model/language pairs (eg., JA & AltDiffusion) were found by Saxon and Wang (2023) to be "well-possessed" (high average \(X_{c}\) across correct concepts) in CoCo-CroLa v1. In other words, _valid corrections only matter for languages a model already "knows."_ Correct Klingon is just as useless as incorrect Klingon to a non-Klingon model.

Figure 2: Scatterplots showing the impact of the corrections to each concept in JA, ZH, and ES on the conceptual improvement to the CCCL correctness score, \(\Delta X_{c}\), as a function of \(\Delta\mathrm{SEM}\). Slopes \(m\) at bottom-right in **bold**.



Table 3 (subsection A.5) shows the same slopes \(m\) with PCCs, _p_-values, and intercepts for the each model and language's \(\Delta{\rm SEM}\) to \(\Delta X_{c}\) relationship. The high-slope language/model pairs also tend to have higher PCC with more statistical significance.

StableDiffusion 1.4 was trained on the primarily-Latin script LAION-en-2bSchuhmann et al. (2021), and thus lacks capabilities in non-Latin script languages JA and ZH. Consequently, there is no significant relationship between more semantically divergent corrections with high \(\Delta{\rm SEM}\) and larger improvements to concept correctness \(\Delta X_{c}\) for SD 1.4 on those languages. Meanwhile, AltDiffusion--which conditions output images on the multilingual XLM-Roberta encoder Conneau et al. (2020)-benefits from all significant corrections in all languages with statistical significance.

### Pseudocorrection Experiment

Unfortunately our ability to use the aforementioned corrections to confirm our hypothesis that _T2I model language capability can be estimated from the impact of translation corrections on image-domain performance_ is hindered by the small quantity of correction candidates we found. We bypass this problem with a _pseudocorrection experiment_--simulating a larger set of corrections by generating artificial errors in the other CCCL languages. We generate 10 synthetic erroneous _pseudo-original translations_ for each concept in German, Indonesian, and Hebrew by randomly sampling the translations for other concepts within-language. Each concept's "correction" is its original translation.

For example, we assign the concept _eye_ the Indonesian word _guru_ (EN:teacher) as its pseudo-original. We then "correct" this word to _mata_, the original correct translation, and assess \(\Delta X_{c}\) and \(\Delta{\rm SEM}\) with \(c_{\ell_{s}}\)_-eye_, \(c_{\ell}\)_-guru_ and \(c^{\prime}_{\ell}\)_-mata_.

This gives us 1,930 \(\Delta X_{c}\), \(\Delta{\rm SEM}\) pairs for each language and model, with which we evaluate the same correlation relationship as before (plot in Appendix Figure 6). We report Pearson's correlation coefficient (PCC) for each of these pairs along with the average CCCL \(X_{c}\) reported in Saxon and Wang (2023) in Figure 3. The same relationship for real corrections holds for pseudocorrections, demonstrating that text-only multilingual semantic similarity features can predict the impact of a translation correction on the output image correctness.

## 4 Discussion & Conclusions

Our findings motivate important considerations for building future T2I semantic evaluations Saharia et al. (2022); Cho et al. (2022); Huang et al. (2023).

SubjectivityA reliable T2I multilinguality assessment must report true knowledge failures--examples where a model fails to generate correct images of a concept, when it is correctly prompted to do so. Correct translations are required.

Unfortunately, choosing one "correct translation" is in inherently subjective task. This study tackled this subjectivity by casting a wide net of error candidates, and testing their impact. Consequential errors caused _false negatives_ where a concept is erroneously marked as "not possessed" (Figure 1).

CCCL's tangible concept constraint and corpus-based approach to finding concepts helps combat subjectivitySaxon and Wang (2023). In the tangible sense it's fair to say "orange" is correctly translated in Spanish to _naranja_ (the fruit) rather than _anaranjado_ (the adjective).

In prompting the T2I model we assume this tangible noun context is induced by using "a picture of an \(X\)"-style prompts. While our results show this works, it is a model-specific phenomenon and future work should examine more prompt templates.

Future work grounded in prototype theory Ando et al. (2002) may enable identification of culturally universal concepts for assessment.

Need to assess Multiple TranslationsOne challenge in multilinguality assessments is _incoming duplicates_, where multiple ways of writing a translation really are equally correct. Our homograph errors have examples, such as cigarette in Japanese. , , , and  are all translations of cigarette with identical reading, _tabako_. Why should a metric of model-language capabilities only assess one correct translation rather than all?

Figure 3: Languages with a high correlation between textual correction significance and image improvement (PCC) are more “well-understood” by the model (\(X_{c}\)), for both real- and pseudo-corrections.

 

More significant multiple translation problems arise in languages with gendered human-referent terms. For example, in Spanish _maestro_ refers to a male teacher, while _maestra_ a female one. Should a test of a model's Spanish knowledge of "teacher" as a concept test that both translations work equally well? CCCL v1 is incapable of assessing these attributes. Future benchmarks should contain this flexibility, so multiple incoming translations Savoldi et al. (2021) can be assessed for the same concept, while also tracing semantically-encoded secondary attributes such as gender between the source and test language.

Error Severity and Error TypeFigure 4 shows the distributions of error types for each language with respect to \(\Delta_{SEM}\), our proxy for correction significance or error severity. Across all three languages, the _sense errors_ (OS and IS) are the most severe, while the formality and commonality errors are the least severe (defined in subsection A.3).

Our original estimated error rate (sum of all candidates per language) is a worst-case bound, the significant-to-evaluation-validity error rate is lower. Our impact and significance results show that some of our suggestions (mainly formality and commonality errors) may be more nitpick than correction.

Some concepts in CCCL are inherently erroneous due to intangibility. For example, _history_, _film_, and _jump_ are all present in v1 of CCCL, picked up for being high-frequency noun concepts across multiple languages in the corpora. There is no sensible prototypical way to generate images "of" those concepts. We removed these for CCCL v1.1; Future benchmarks should avoid including them.

Image-Image Metric Blind SpotsWe observed interesting borderline (potential false positive) cases where CoCo-CroLa scored mistranslated concepts as possessed. For example, _bike_ in Japanese. Figure 1 shows that under the erroneous translation, AltDiffusion generates pictures of _motorcycles_ rather than _bicycles_ as it does in English. However, \(X_{c}\) doesn't actually change much under this correction as shown in Figure 2 & Table 4. The CLIP similarity score in CCCL is blind to the difference between a bicycle and motorcycle. Mistranslations where visual structural similarity is present are sometimes invisible to the image metrics.

Tangible object translation as an MT domainSingle word concepts are not central to the distribution of machine translation training data. By providing the individual English tangible nouns as input we may expect an unreasonable amount of implicit commonsense reasoning from commercial MT systems--the correct sense out of many had to be selected for success. Furthermore, the use of the BabelNet knowledge graph as a consensus mechanism reinforced some sense errors. For example, the _rock_ sense error for JA (music genre rather than physical object, Table 4) was also present in Hebrew, probably due to shared edges in the knowledge graph. Given previous interest in assessing the performance of MT translation in diverse domains Irvine et al. (2013), we think both the word-level translation of concepts under domain constraints without context (as we tried to do in CCCL previously) and treating input prompts for T2I systems (ie, captions) Hitschler et al. (2016); Singh et al. (2021) as a target domain for MT evaluation would be interesting and useful future directions.

Future benchmarks should leverage context with sentences as input to MT (eg, "watch for falling rocks") rather than the decontextualized concept words alone to improve robustness. LLMs could generate diverse English sentence examples, and could potentially also extract the final concept translations out of the multiple sentence translations.

Figure 4: Histograms for the error counts in JA, ZH, and ES vs \(\Delta_{SEM}\), colored by error type. From lightest, they are F:_formality_, C:_commonality_, A:_ambiguity_, T:_transliteration_, IS:_incoming sense error_, OS:_outgoing sense error_. The error types are defined in subsection A.3. Severe error types will exhibit more rightward distributional mass.

 



## Limitations

Trivially, human annotators for every language would remove false-negative mistranslations from future benchmarks, but there's a trade-off between easy scalability and certainty of correctness.

Our work incorporates human efforts of both native and proficient but non-native language speakers to propose and resolve translation error candidates caused by the machine translation pipeline in the original CoCo-CroLa benchmark. This could potentially bring human biases into the nuance of factors such as words' choices, introducing less culturally neural expressions as a result.

The assumption of _translatability_ that underlies CCCL in general is a challenge. As a practical use-based test of functional fairness, using heuristics and only common everyday objects that can be reasonably assumed universal is acceptable, but more linguistic and even philosophical work is needed to really motivate fairness across languages and cultures when underlying assumptions differ.

## Acknowledgements

Thanks to our December 2023 ARR reviewers and ACs, particularly Yx9V for thoughtful and detailed reviewing and conversation, and many useful suggestions. Thank you Alfonso Amayuelas for feedback on ES candidates. This work was supported in part by the National Science Foundation Graduate Research Fellowship under Grant No. 1650114, and CAREER Award under Grant No. 2048122.

## References

* Agrawal et al. (2018) Ashwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 4971-4980.
* Spain. European Language Resources Association (ELRA).
* Antol et al. (2015) Stanislaw Antol, Aishwaya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In _Proceedings of the IEEE international conference on computer vision_, pages 2425-2433.
* Chen et al. (2022) Zhongzhi Chen, Guang Liu, Bo-Wen Zhang, Fulong Ye, Qinghong Yang, and Ledell Wu. 2022. Altclip: Altering the language encoder in clip for extended language capabilities. _ArXiv preprint_, abs/2211.06679.
* Cho et al. (2022) Jaemin Cho, Abhay Zala, and Mohit Bansal. 2022. Dallesval: Probing the reasoning skills and social biases of text-to-image generative transformers.
* Conneau et al. (2020) Alexis Conneau, Kartiaky Khandelwal, Naman Goyal, Vishrav Chaudhry, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _ACL 2020_, pages 8440-8451, Online. Association for Computational Linguistics.
* Cui et al. (2021) Yuqing Cui, Apoorty Khandelwal, Yoav Artzi, Noah Snavely, and Hadar Averbuch-Elor. 2021. Who's waldo? linking people across text and images. In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_, pages 1374-1384.
* Hitschler et al. (2016) Julian Hitschler, Shigehiko Schamoni, and Stefan Riezler. 2016. Multimodal pivots for image caption translation. In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_. Association for Computational Linguistics.
* Ho et al. (2023) Matthew Ho, Aditya Sharma, Justin Chang, Michael Saxon, Sharon Levy, Yujie Lu, and William Yang Wang. 2023. Wikiwhy: Answering and explaining cause-and-effect questions. In _The Eleventh International Conference on Learning Representations_.
* Huang et al. (2023) Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2023. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. _arXiv preprint arXiv:2307.06350_.
* Huang et al. (2024) Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. 2024. T2i-compbench: A comprehensive benchmark for open-world compositional text-to-image generation. _Advances in Neural Information Processing Systems_, 36.
* Irvine et al. (2013) Ann Irvine, John Morgan, Marine Carpuat, Hal Daume III, and Dragos Munteanu. 2013. Measuring machine translation errors in new domains. _Transactions of the Association for Computational Linguistics_, 1:429-440.
* Luo et al. (2022) Yiran Luo, Pratyag Banerjee, Tejas Gokhale, Yezhou Yang, and Chitta Baral. 2022. To find waldo you need contextual cues: Debiasing who's waldo. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, page 355-361, Dublin, Ireland. Association for Computational Linguistics.
* Navigli and Ponzetto (2010) Roberto Navigli and Simone Paolo Ponzetto. 2010. BabelNet: Building a very large multilingual semantic network. In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_, pages 216-225, Uppsala, Sweden. Association for Computational Linguistics

Maitreya Patel, Tejas Gokhale, Chitta Baral, and Yezhou Yang. 2024. Conceptbed: Evaluating concept learning abilities of text-to-image diffusion models.
* Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learning transferable visual models from natural language supervision. In _ICML 2021_, volume 139 of _Proceedings of Machine Learning Research_, pages 8748-8763. PMLR.
* Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics.
* Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. 2022. High-resolution image synthesis with latent diffusion models. In _CVPR 2022_, pages 10684-10695.
* Saharia et al. (2022) Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. 2022. Photorealistic text-to-image diffusion models with deep language understanding. 35:36479-36494.
* Savoldi et al. (2021) Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. _Transactions of the Association for Computational Linguistics_, 9:845-874.
* Saxon and Wang (2023) Michael Saxon and William Yang Wang. 2023. Multilingual conceptual coverage in text-to-image models. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 4831-4848, Toronto, Canada. Association for Computational Linguistics.
* Saxon et al. (2023) Michael Saxon, Xinyi Wang, Wenda Xu, and William Yang Wang. 2023. PECO: Examining single sentence label leakage in natural language inference datasets through progressive evaluation of cluster outliers. In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_, pages 3061-3074, Dubrovnik, Croatia. Association for Computational Linguistics.
* Schuhmann et al. (2021) Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. 2021. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. _ArXiv preprint_, abs/2111.02114.
* Singh et al. (2021) Salam Michael Singh, Loitongbam Sanayai Meetei, Thoudan Doren Singh, and Sivaji Bandyopadhyay. 2021. Multiple captions embellished multilingual multi-modal neural machine translation. In _Proceedings of the First Workshop on Multimodal Machine Translation for Low Resource Languages (MMTLRL 2021)_, pages 2-11.
* Ye and Kovashka (2021) Keren Ye and Adriana Kovashka. 2021. A case study of the shortcut effects in visual commonsense reasoning. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 3181-3189.
* Zellers et al. (2019) Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. From recognition to cognition: Visual commonsense reasoning. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 6720-6731.

## Appendix A Appendix

### Contribution Statement

YL produced the Chinese and Japanese translation error candidates and the overall EC taxonomy. MS produced the Spanish candidates and checked the Japanese candidates. XL evaluated \({\rm ASEM}\), MS generated the before/after images and evaluated \(X_{c}\) and \(\Delta X_{c}\). YL produced diagrams and MS graphs.

#### a.1.1 Human Annotation Details

MS and YL produced the initial list of candidate errors and corrections. MS is a native speaker of English and literate second language speaker of Spanish and Japanese. YL is a native speaker of Chinese, professionally proficient speaker of English, and a literate proficient speaker of Japanese, with experience in literary translation and textual localization between English, Chinese, and Japanese.

Each annotator first read through the list of their languages (ES/JA and ZH/JA respectively) for about 10 minutes and marked every translation (_error candidate_) that appeared incorrect with a preliminary correction. They then verified the annotations using bilingual English-{Spanish, Japanese, Chinese} resources and consultation with native speakers where relevant as detailed below.

MS checked Spanish corrections using Spanish-language example usage notes provided in the Spanish wordreference.com dictionary, and consultation with a native speaker. MS's JA error candidates were a subset of YL's. YL also took references from language standard dictionaries used by native speakers--for Chinese _Xiandai Hanyu Cidian_ and for Japanese _Shin Meikai Kokugo Jiten_.

### Additional Resource Information

Intended Use, License and TermsWe release our corrections as a v1.1 revision to the CoCoCoCoLa benchmark [Saxon and Wang, 2023] intended to evaluate the performance of text-to-image models. It inherits v1's license and terms 

Offensive ContentSome of the erroneous translations we found can lead to offensive images, e.g. the original JA translation for _milk_ in also means "breast."

### Error candidate typology

Commonality (C)When a selected translated term doesn't appear to reflect the most common, colloquial, contemporary, or "natural" way that native speakers of the language would use in reference to the concept in a photograph or conversation. For example, in Chinese "" is a more conversational and contemporary way of writing _bottle_ than ""," which reads literary and archaic.

Outgoing Sense Error**.** (OS)The translated term picks an alternative (and often less tangible) sense from the source concept. For example, the original Chinese translation for _Table_ diverges to the sense of 'spreadsheet, tabular', instead of the presumptive home furniture item.

Incoming Sense Error**.** (IS)The translated term, while aligned to the correct source concept sense, picks a phrasing for which other senses in the target language exist that the annotators expect will confound model behavior, where another (often more common) disambiguated translation also exists. For example, the original Spanish translation for _tent_ is given as _tienda_ alone, which can also mean 'store, shop', in addition to 'a tent,' whereas the corrected translation _tienda de acampar_ refers to a camping tent alone.

Ambiguity**(A)The translated term introduces a word with multiple meanings from the unambiguous source concept. For example, the Japanese translation for _Milk_ originally uses a single character that can mean any kind of animal or human milk, or even the organ of the breast.

Formality**.** (F)The translated term uses an expression in an improper formality. For example, the original Chinese translation for _Father_ is only heard in casual conversations.

Transliteration** (T)When one of the above errors occurs with . For example, the transliteration of _Rock_ in Japanese is commonly related to 'Rock Music', rather than stones found in nature.

### Computational Experiments Details

Dataset StatisticsCCCL contains 193 multilingual concepts written in 7 languages. We have also modified 50 of these in ES, ZH, or JA with verified translations by human annotators.

Experimental SetupWe generated 9 images for each (language, model, concept) triple and evaluated \(X_{C}\) using identical methods and codeas described in CCCL Saxon and Wang (2023).

### Full Analysis Numbers

### Further Related Work

ConceptBed Patel et al. (2024) evaluates monolingual concept-level knowledge in T2I, and its concept inventory could extend and improve CCCL's. T2I-CompBench assesses compositionality in T2I Huang et al. (2024), leveraging VQA and image segmentation. Assessment model weaknesses, such as Agrawal et al. (2018)'s VQA spurious correlations Antol et al. (2015) remain a challenge.

Other benchmarks in vision-and-language also require correction and improvement. Luo et al. (2022) found and filtered unsolvable cases in _Who's Waldo_Cui et al. (2021). Ye and Kovashka (2021) exploit repeated texts in QA pairs on _VCR_Zellers et al. (2019). While manual techniques can find and clean these errors, automated approaches would be preferable, such as the PECO method Saxon et al. (2023) for finding model-used shortcuts in NLI Semi-human-in-the-loop approaches Ho et al. (2023) may improve the sourcing and cleaning of future CCCL versions.

\begin{table}
\begin{tabular}{l l c r} \hline \hline Model & \# Param & Repository (huggingface.co/...) \\ \hline StableDiffusion I.4 & 86dM & Corpus/is/stable-offvision-v1.4 \\ StableDiffusion II & NA & stabililvial/stable-diffusion-2 \\ StableDiffusion II.2 & NA & stabililvia/stable-difffusion-2 \\ AlbIllusion m9 & 1.7B & BAAI/AltOlifusion+n9 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The set of text-to-image models we evaluated with (Table adapted from Saxon and Wang (2023).

\begin{table}
\begin{tabular}{c c c c c c} \hline \hline Model & Language & PCC & \(p\) & \(m\) & \(b\) \\ \hline SD1-4 & Japanese & 0.120 & 0.577 & 0.437 & 0.049 \\ SD1-4 & Chinese & 0.018 & 0.944 & 0.051 & -0.011 \\ SD1-4 & Spanish & 0.384 & 0.307 & 1.877 & -0.064 \\ SD2 & Japanese & 0.088 & 0.684 & 0.155 & 0.020 \\ SD2 & Chinese & 0.155 & 0.554 & 0.608 & 0.000 \\ SD2 & Spanish & 0.646 & 0.060 & 3.891 & -0.067 \\ AD & Japanese & 0.734 & 0.000 & 1.519 & 0.014 \\ AD & Chinese & 0.725 & 0.001 & 4.472 & -0.010 \\ AD & Spanish & 0.895 & 0.001 & 3.588 & 0.010 \\ SD2-1 & Japanese & 0.162 & 0.448 & 0.272 & 0.013 \\ SD2-1 & Chinese & 0.078 & 0.765 & 0.340 & 0.001 \\ SD2-1 & Spanish & 0.574 & 0.106 & 3.722 & -0.075 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Stats for Pearson correlation and linear best fit between \(\Delta\mathrm{SEM}\) and \(\Delta X_{c}\) for each model and language. \(p\) represents the \(p\)-value for the PCC, \(m\) and \(b\) the slope and intercept for the best-fit line.

 

Figure 5: Qualitative examples of selected mistranslated concepts found in Coco-CroLa generated by AltDiffusion and multiple versions of Stable Diffusion - **Top left:** “Rock” in Japanese, **Top right:** “Suit” in Chinese, **Bottom left:** “Tent” in Spanish, **Bottom right:** “Table" in Chinese. Noticeably, we observe that T2I models such as Stable Diffusion 2 do not benefit from correcting the translations, as their outputs in the aforementioned languages remain irrelevant similarly to using random prompts.

 

* [14] M. C. B. Krizhevsky, A. Vedaldi, and S. Ioffe. Learning from a single image by minimizing the loss function. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 235-240, 2016.
* [15] J. L. Friedman, R. G. Olshen, and D. E. Silverstein. Training deep neural networks with applications to machine learning. _arXiv preprint arXiv:1907.08958_, 2019.

[MISSING_PAGE_POST]

 



\begin{table}
\begin{tabular}{l l c|c c c c c c} \hline \hline Concept & Original & Corrected & Type & \(\Delta\mathrm{SEM}\) & \multicolumn{3}{c}{\(\Delta X_{c}\) (CCCL Improvement) for model SD 1.4} \\  & & & & SD 1.4 & SD 2 & SD 2.1 & AD \\ \hline \multicolumn{8}{c}{_All Japanese-language error candidates:_} \\ \hline duck & \(\mathfrak{H}_{\text{cl}}\) & \(\mathcal{T}\vDash\iota\) & C & -0.092 & 0.021 & 0.008 & -0.012 & -0.055 \\ thigh & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{J}\vDash\iota\) & C & -0.091 & 0.048 & 0.007 & -0.043 & -0.124 \\ cop & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{J}_{\text{cl}}^{\text{th}}\) & \(\mathfrak{J}_{\text{cl}}^{\text{th}}\) & F & -0.053 & -0.160 & -0.029 & -0.055 & -0.140 \\ field & \(\mathfrak{H}_{\text{cl}}\) & \(\mathbb{H}_{\text{cl}}\) & A & -0.036 & 0.015 & -0.151 & -0.075 & -0.058 \\ butterfly & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{H}_{\text{cl}}\) & C & -0.022 & -0.004 & 0.025 & 0.009 & -0.020 \\ girlfriend & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{H}_{\text{cl}}\) & C & -0.013 & 0.044 & 0.166 & 0.196 & -0.030 \\ stingray & \(\gamma\eta\pi\,\lambda\) & T & C & -0.008 & -0.058 & 0.044 & -0.006 & -0.071 \\ cigarette & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{J}_{\text{cl}}\) & C & -0.007 & 0.054 & 0.043 & -0.034 & 0.078 \\ tail & \(\mathfrak{F}_{\text{cl}}\) & \(\mathfrak{W}_{\text{cl}}\) & C & -0.003 & 0.004 & 0.077 & 0.056 & 0.040 \\ woman & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & C & -0.001 & 0.108 & -0.022 & -0.014 & -0.046 \\ forest & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & C & -0.000 & 0.226 & 0.081 & 0.032 & 0.051 \\ teenager & \(\mathfrak{T}_{\text{cl}}\) & \(\mathfrak{J}_{\text{cl}}\) & C, T & 0.002 & 0.169 & 0.076 & 0.115 & 0.023 \\ flame & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & C & 0.003 & -0.062 & -0.070 & 0.009 & 0.031 \\ father & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & F & 0.010 & -0.009 & -0.010 & 0.014 & 0.003 \\ watch & \(\mathfrak{B}_{\text{cl}}\) & \(\mathfrak{B}_{\text{cl}}\) & I & IS & 0.011 & 0.487 & 0.080 & 0.062 & 0.006 \\ teacher & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & IS & 0.015 & 0.006 & -0.051 & -0.070 & 0.016 \\ kid & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & C, T & 0.017 & 0.098 & 0.070 & 0.065 & 0.068 \\ doctor & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & IS & 0.017 & -0.006 & 0.031 & 0.018 & 0.050 \\ ground & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & OS & 0.022 & -0.008 & 0.097 & 0.084 & 0.086 \\ bike & \(\mathfrak{K}_{\text{cl}}\) & \(\mathfrak{K}_{\text{cl}}\) & GS & T & 0.023 & 0.195 & 0.021 & -0.018 & 0.020 \\ detail & \(\mathfrak{T}_{\text{cl}}\) & \(\mathfrak{T}_{\text{cl}}\) & D & 0.024 & 0.002 & 0.036 & 0.043 & -0.031 \\ milk & \(\mathfrak{A}_{\text{cl}}\) & \(\mathfrak{A}_{\text{cl}}\) & OS & 0.033 & 0.141 & 0.026 & -0.002 & 0.215 \\ cafeteria & \(\mathfrak{T}_{\text{cl}}\) & \(\mathfrak{T}_{\text{cl}}\) & C, T & 0.044 & -0.192 & -0.043 & -0.034 & 0.064 \\ rock & \(\mathfrak{T}_{\text{cl}}\) & \(\mathfrak{T}_{\text{cl}}\) & IS, T & 0.067 & 0.048 & -0.029 & -0.033 & 0.104 \\ \hline \multicolumn{8}{c}{_All Chinese-language error candidates:_} \\ \hline men & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{J}_{\text{cl}}\) & A & -0.032 & 0.001 & -0.180 & -0.182 & -0.411 \\ stringay & \(\mathfrak{M}_{\text{cl}}\) & \(\mathfrak{R}_{\text{cl}}\) & C & -0.030 & 0.082 & 0.206 & 0.213 & -0.099 \\ field & \(\mathfrak{S}_{\text{cl}}\) & \(\mathfrak{H}_{\text{cl}}\) & A & -0.017 & -0.012 & -0.136 & -0.184 & 0.083 \\ boat & \(\mathfrak{M}_{\text{cl}}\) & /-\(\mathfrak{M}_{\text{cl}}\) & F & -0.001 & -0.110 & 0.009 & 0.008 & 0.017 \\ sister & \(\mathfrak{M}_{\text{cl}}\) & \(\mathfrak{M}_{\text{cl}}\) & F & -0.001 & 0.033 & 0.014 & 0.026 & -0.014 \\ wife & \(\mathfrak{M}_{\text{cl}}\) & \(\mathfrak{C}_{\text{cl}}\) & C & 0.003 & -0.021 & 0.124 & 0.177 & -0.021 \\ bottle & \(\mathfrak{M}_{\text{cl}}\) & \(\mathfrak{C}_{\text{cl}}\) & O & 0.004 & -0.062 & -0.021 & 0.032 & 0.075 \\ church & \(\mathfrak{M}_{\text{cl}}\) & \(\mathfrak{M}_{\text{cl}}\) & A & 0.005 & -0.068 & 0.076 & 0.078 & -0.018 \\ father & \(\mathfrak{G}_{\text{cl}}\) & \(\mathfrak{C}_{\text{cl}}\) & C & 0.009 & 0.027 & -0.028 & -0.059 & 0.145 \\ mouth & \(\square\) & \(\mathfrak{M}_{\text{cl}}\) & C & 0.011 & -0.054 & 0.023 & 0.010 & 0.037 \\ bell & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{M}_{\text{cl}}\) & A & 0.013 & -0.013 & 0.071 & 0.081 & -0.001 \\ cafeteria & \(\mathfrak{H}_{\text{cl}}\) & \(\mathfrak{H}_{\text{cl}}\) & A & 0.017 & -0.102 & -0.047 & -0.054 & 0.071 \\ orange & \(\mathfrak{M}_{\text{cl}}\) & \(\mathfrak{M}_{\text{cl}}\) & OS & 0.019 & 0.002 & -0.099 & -0.104 & 0.067 \\ belt & \(\mathfrak{M}_{\text{cl}}\)