SlowGuess/NAACL_2021_OCR huggingface dataset:
Figure 1 shows the results of our evaluation on the test set of 1,000 words, depicting the distributions of the different part-of-speech categories. We provide the Spearman correlation coefficient between the ground-truth and predicted concreteness scores, which shows the improved effectiveness of our BERT embeddings-based method compared to the Salle et al. (2016) embeddings employed by Hewitt et al. (2018). Using this trained model, we predict the concreteness score of each of the words in our dataset by first translating the word to English and lemmatizing it using spaCy.
Using the predicted concreteness scores, we distinguish words in our dataset to concrete (having a predicted score of  $>3$ ) and abstract ( $\leq 3$ ).
4.4 Inter-Cluster Similarity Threshold
We define a homogeneity score threshold to determine if two words  $w_{s}$  and  $w_{t}$  have sufficiently high overlap in their image embeddings to indicate a good translation. For each language pair, we compute this threshold  $h_{s,t}^{\mathrm{thld}}$  by taking the average
homogeneity score of clusters of images of 10 randomly chosen word pairs from the source  $s$  and the target language  $t$ . We take the average of these scores since we want to first be able to compare the threshold with other homogeneity scores and second to be able to capture the skew in negative thresholds as well. These pairs serve as negative examples of translation and we expect their image embeddings to be disjoint. Hence, a word pair with homogeneity score lower than this threshold means that the word pair has a good overlap in their image embeddings (i.e., a high inter-cluster similarity), which indicates a good translation.
4.5 Intra-Cluster Similarity Threshold
Similarly, we define an intra-cluster similarity threshold to determine if an image cluster associated with a word  $w$  is sufficiently tight. Since intra-cluster similarity is computed for each word (and not word pair), we compute this threshold  $\text{MEDMAX}_l^{\text{thld}}$  for each language  $l$  by constructing a negative example for the language i.e., an image cluster with a high dispersion. We create this negative example by taking five random words from the language and for each word a random sample of 20 images to build a cluster of 100 images (mimicking the typical image cluster size for a word in our dataset). We set the Median Max Cosine Similarity of this image cluster as the intra-cluster similarity threshold. A word that has an intra-cluster similarity higher than this threshold would mean that this word has a tight image cluster, a consistent meaning as represented in its images' representations.
4.6 Normalized Score
We define a normalized score  $\mathrm{NORM}_{w_s, w_t}$  to combine intra- and inter-cluster similarity scores for a word  $w_s$  in the source language and its translation in the target language  $w_t$ .
Given the intra-cluster similarity of word  $w_{s}$ $(\mathrm{MEDMAX}_{w_s})$  and that of word  $w_{t}$ $(\mathrm{MEDMAX}_{w_t})$ ; and the maximum and minimum intra-cluster similarities for the source language  $(\mathrm{MEDMAX}_s^{\max}$  and  $\mathrm{MEDMAX}_s^{\min}$ ), and those of the target language  $(\mathrm{MEDMAX}_t^{\max}$  and  $\mathrm{MEDMAX}_t^{\min}$ ); as well as the homogeneity score of the words  $h_{w_s,w_t}$  and the maximum and minimum homogeneity scores of words in the language pair  $(h_{s,t}^{\max}$  and  $h_{s,t}^{\min}$ ), we compute the normalized score  $\mathrm{NORM}_{w_s,w_t}$  as:
$$
\begin{array}{l} \mathrm {N O R M} _ {w _ {s}, w _ {t}} = \mathrm {N O R M M E D M A X} _ {w _ {s}} + \mathrm {N O R M M E D M A X} _ {w _ {t}} \\ - \mathrm {N O R M} h _ {w _ {s}, w _ {t}} \\ \end{array}
$$
$$
\mathrm {N O R M M E D M A X} _ {w _ {s}} = \frac {\mathrm {M E D M A X} _ {w _ {s}} - \mathrm {M E D M A X} _ {s} ^ {\min }}{2 \left(\mathrm {M E D M A X} _ {s} ^ {\max } - \mathrm {M E D M A X} _ {s} ^ {\min }\right)}
$$
$$
\mathrm {N O R M M E D M A X} _ {w _ {t}} = \frac {\mathrm {M E D M A X} _ {w _ {t}} - \mathrm {M E D M A X} _ {t} ^ {\min }}{2 \left(\mathrm {M E D M A X} _ {t} ^ {\max } - \mathrm {M E D M A X} _ {t} ^ {\min }\right)}
$$
$$
\mathrm {N O R M} h _ {w _ {s}, w _ {t}} = \frac {h _ {w _ {s} , w _ {t}} - h _ {s , t} ^ {\min }}{h _ {s , t} ^ {\max } - h _ {s , t} ^ {\min }}
$$
For each language pair, we also define a threshold on this normalized score (i.e.,  $\mathrm{NORM}_{s,t}^{\mathrm{thld}}$ ) by substituting  $\mathrm{MEDMAX}_{w_s}$ ,  $\mathrm{MEDMAX}_{w_t}$ , and  $h_{w_s,w_t}$  with  $\mathrm{MEDMAX}_s^{\mathrm{thld}}$ ,  $\mathrm{MEDMAX}_t^{\mathrm{thld}}$ , and  $h_{s,t}^{\mathrm{thld}}$  respectively in the equation above.
In order to compare the image translatability of two language pairs with different characteristics, we compare the ratio of the number of word pairs that are good translations divided by the total number of word pairs in each language pair:
$$
\mathrm {R A T I O} = \frac {\text {# o f w o r d p a i r s t h a t a r e g o o d t r a n s l a t i o n s}}{\text {# o f w o r d p a i r s f o r t h e l a n g u a g e p a i r}}
$$
A word pair  $w_{s}$  and  $w_{t}$  has a good translation via images (or good image translatability) if its homogeneity score  $h_{w_s,w_t}$  is lower than the homogeneity threshold  $h_{s,t}^{\mathrm{thld}}$  and its Median Max Cosine Similarities i.e.,  $\mathbf{MEDMAX}_{w_s}$  and  $\mathbf{MEDMAX}_{w_t}$  are higher than the thresholds  $\mathbf{MEDMAX}_{s}^{\mathrm{thld}}$  and  $\mathbf{MEDMAX}_{t}^{\mathrm{thld}}$ , respectively.
The higher this ratio, the more translatable the language pair is via images. When we compare two language pairs that have the same source language but different target languages (with different shared characteristics with the source), we can distinguish how different characteristics such as cultural similarity or geographical proximity affect image translatability. In Table 2, we show image translatability ratios of language pairs with the same source but different target languages side-by-side.
To understand the role of concreteness in translation via images, we also compute how many concrete words have good translations according to our image translatability measures. We consider a word pair,  $w_{s}$  and  $w_{t}$ , to be concrete if  $w_{s}$  has a concreteness score greater than 3. Source word concreteness is taken to as the pair concreteness, considering translation directionality. The ratio of how many concrete words in each language pair have good translations is also shown in Table 2.
Language Pair	Number of Words	Ratio
All	Concrete	All	Concrete
az	tr	4538	3470	0.31	0.37
az	ru	5380	2953	0.17	0.22
ko	zh	338	214	0.18	0.22
ko	ja	748	499	0.69	0.72
zh	ja	367	212	0.56	0.58
zh	ko	310	197	0.36	0.44
ja	zh	212	137	0.39	0.44
ja	ko	741	488	0.67	0.70
ar	ur	4916	3226	0.39	0.44
ar	fa	448	318	0.50	0.55
ar	he	2887	1874	0.69	0.73
ur	ar	4243	2466	0.39	0.45
ur	hi	4588	2817	0.12	0.15
es	fr	6392	3506	0.45	0.58
es	pt	7116	3920	0.40	0.53
fi	hu	5615	3190	0.29	0.40
fi	no	5336	3033	0.17	0.26
af	nl	5436	3247	0.39	0.50
af	sw	4553	2611	0.25	0.31
Table 2: Language pairs along with the numbers of word pairs and their image translatability ratios, for all and concrete word pairs. In boldface we mark the pair that has the highest ratio among pairs with the same source language whose normalized scores are significantly different and whose ratio differences are high.
To test whether the difference in image translatability between language pairs that share the same source language (e.g., Finnish to Norwegian vs. Finnish to Hungarian) is statistically significant, we conduct a simple t-test between their normalized score distributions. The resulting p-values signal the difference between their distributions. Low p-values  $(< 0.05)$  indicate statistical significance and high variation between the distributions, while higher values suggest low variation and large similarities between the language pairs (Table 3).
From the t-test, we find that the difference in distributions of pairs that share the same source language is almost in all statistically significant (p-value  $< 0.05$ , Table 3) except for Japanese to Chinese vs. Japanese to Korean.
Of other pairs whose normalized score differences are statistically significant and whose differences in translatability ratios are high (boldfaced, Table 2), we observe that the language pair with the higher image translatability ratio (i.e., Azer
Language Pair I	Language Pair II	p-value
az	tr	az	ru	2.26 × 10-25
ko	zh	ko	ja	2.2 × 10-4
zh	ja	zh	ko	10.6 × 10-5
ja	zh	ja	ko	0.43
ar	ur	ar	fa	9.44 × 10-29
ar	fa	ar	he	2.39 × 10-13
ar	he	ar	ur	16.35 × 10-5
es	fr	es	pt	4.3 × 10-47
fi	hu	fi	no	2.66 × 10-104
af	nl	af	sw	10-100
Table 3: p-values of differences between normalized score distributions of language pairs that share the same source language. In boldface, we mark pairs with a high p-value, for which we cannot assume a significant difference in their normalized score distributions.
baijani to Turkish, Korean to Japanese, Chinese to Japanese, Arabic to Hebrew, Urdu to Arabic, Finnish to Hungarian, and Afrikaans to Dutch) is always the pair that shares cultural similarity (i.e., either similar language family, similar major ethnicity, or similar major religion) even when they have little to no geographical proximity. For example, between Arabic and Hindi, Urdu's words are more translatable via images to Arabic (whose speakers share the same major religion as speakers of Urdu), despite Pakistan's geographic proximity to India. Similarly, Finnish words are more translatable via images to Hungarian (whose speakers belong to the same ethnolinguistic group as speakers of Finnish) than to Norwegian, despite Hungary not sharing any land border with Finland. In addition, there may be other language relatedness factors that result in better image translatability between languages, such as the similar writing system of Chinese and Japanese, or the similar grammatical structure of Korean and Japanese; despite their different language families.
From Table 1 we can see that Spanish shares similar characteristics with both French and Portuguese. Similarly, Japanese shares similar attributes with both Chinese and Korean (matching ethnicity and religion, different geography and language family). In such cases, where the two language pairs do not differ in characteristics, we observe that the difference in their translatability ratios is either small (in the case of Spanish to French and Spanish to Portuguese ratios in Table 2) or insignificant (in the case of Japanese to Chinese and Japanese to Korean p-value in Table 3).
On the contrary, Korean to Japanese and Korean
Figure 2: (left) PCA plot of image embeddings of the word universe in Afrikaans (heelal) and in Dutch (universum) showing their tightly overlapping image clusters, (center) images of heelal in Afrikaans, (right) images of universum in Dutch.
to Chinese pairs, and Chinese to Korean and Chinese to Japanese pairs, have at least one difference in their attributes, accounting for the pairs' results statistical significance.
In addition, we observe that concreteness of words largely affects the quality of translations due to the low diversity in its image representations, which facilitate translation between words. On average, across language pairs,  $62.4\%$  of words with normalized scores above the threshold are concrete, while only  $37.6\%$  are abstract. At the same time, in Table 2 we see that the translatability ratio is considerably higher for concrete word pairs than all pairs. This supports our idea, and other previous works, that concrete words are better represented visually and, so, more likely to have good image-aided translations, compared to abstract ones.
Our work has identified that language relatedness affect word translations via images. We observe that languages with cultural relatedness have better image translatability; suggesting that cultural relatedness should be taken into account when using images to aid translation. The image translatability measures we have defined can be used to identify a potentially good or poor translation or discover a cultural similarity or disconnect between words in two languages. For example, a word pair that has a high intra-cluster similarity and a high inter-cluster similarity in their image representations indicates that the image clusters are tight and overlapping, signaling a good translation between them. For example, the word heelal in Afrikaans and the word univers sum in Dutch have tight and overlapping image clusters (i.e., low homogeneity score) as can be seen in the PCA plot of their image embeddings and in their images (Figure 2).
On the other hand, when a word pair has tight but disjoint image clusters, it can mean that their
Figure 3: (left) PCA plot of image embeddings of the word dance in Afrikaans (dans) and in Swahili (kucheza) showing their tight but disjoint image clusters, (center) images of dans in Afrikaans, (right) images of kucheza in Swahili.
Figure 4: Images of the word park in (left) Afrikaans, (center) Dutch, (right) Swahili.
images express different meanings of the word. For example, the word dance: dans in Afrikaans and the word kucheza in Swahili have tight but disjoint image clusters (i.e., high homogeneity score) as can be seen in the PCA plot of their image embeddings and in the images (Figure 3), as kucheza means both to play and to dance in Swahili.
We observe that Afrikaans, for example, has a higher image translatability to Dutch due to their cultural (ethnolinguistic) similarity, than to Swahili, despite the relative distances of their cultural centers—South Africa is more distant geographically to the Netherlands than to Tanzania, defined in Glottolog as being the cultural center of the Swahili language. We observe higher visual similarities between words that are translations of each other in Afrikaans and Dutch than in Afrikaans and Swahili. For example, images of the word park in Afrikaans are more visually similar to images of the word park in Dutch than to images of the word park in Swahili (hifadhi) (Figures 4, 5). The images of park in Afrikaans and in Dutch refer to a Western style park, while its images in Swahili refer more to a wildlife reservation, a culturally different representation of the word park that is potentially influenced by how speakers of the different languages interact with the concept of the word. Interestingly, such connotation that is apparent in images may not be apparent in word embeddings, since hifadhi is used similarly with park in the texts of the language.
In this paper, we study when images may be useful for translating words between two languages from the perspective of their cultural and geographical relatedness. We observe that translatability of words via images vary in different language pairs, with language pairs sharing cultural similarities having better image translatability.
Figure 5: PCA plots for image embeddings of (left) park in Afrikaans to park in Dutch and (right) park in Afrikaans to hifadhi in Swahili.
In the future, it will be interesting to study image translatability of more language pairs and their characteristics, including those outside MMID, as well as extend our work to sentence-level image aided translation. It will also be of great value to study if adding considerations of cultural relatedness to image-aided MT can further improve its performance. Additionally, using a different metric for intra-cluster similarity that does not calculate similarity with respect to the origin may be more accurate depending on the application. As many similarity functions, aside from cosine similarity, have been used in the computer vision literature, improving this function could be fruitful future work.
We would like to thank the anonymous reviewers for their thoughtful comments. This work is supported in part by the U.S. NSF grant 1838193, DARPA HR001118S0044 (the LwLL program), and the Department of the Air Force FA8750-19-2-3334 (Semi-supervised Learning of Multimodal Representations). The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes. The views and conclusions contained in this publication are those of the authors and should not be interpreted as representing official policies or endorsements of DARPA, the Air Force, and the U.S. Government.
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. 2018. Unsupervised neural machine translation. In International Conference on Learning Representations.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Findings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304-323, Belgium, Brussels. Association for Computational Linguistics.
Shane Bergsma and Benjamin Van Durme. 2011. Learning bilingual lexicons using the visual similarity of labeled web images. In *IJCAI Proceedings* International Joint Conference on Artificial Intelligence, volume 22, page 1764. CiteSeer.
Shane Bergsma, David Yarowsky, and Kenneth Church. 2011. Using large monolingual and bilingual corpora to improve coordination disambiguation. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 1346-1355, Portland, Oregon, USA. Association for Computational Linguistics.
Marc Brysbaert, Amy Beth Warriner, and Victor Kuperman. 2014. Concreteness ratings for 40 thousand generally known english word lemmas. Behavior research methods, 46(3):904-911.
Andrea Burns, Donghyun Kim, Derry Wijaya, Kate Saenko, and Bryan A Plummer. 2020. Learning to scale multilingual representations for vision-language tasks. In European Conference on Computer Vision, pages 197-213. Springer.
Iacer Calixto, Qun Liu, and Nick Campbell. 2017. Multilingual multi-modal embeddings for natural language processing. CoRR, abs/1702.01101.
Shizhe Chen, Qin Jin, and Alexander Hauptmann. 2019. Unsupervised bilingual lexicon induction from mono-lingual multimodal data. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8207-8214.
Raj Dabre, Tetsuji Nakagawa, and Hideto Kazawa. 2017. An empirical study of language relatedness for transfer learning in neural machine translation. In Proceedings of the 31st Pacific Asia Conference on Language, Information and Computation, pages 282-286. The National University (Phillippines).
Johannes Dellert, Thora Daneyko, Alla Munch, Alina Ladygina, Armin Buch, Natalie Clarius, Ilja Grigorjew, Mohamed Balabel, Hizniye Isabella Boga, Zalina Baysarova, et al. 2020. Northeuralex: A wide-coverage lexical database of northern Eurasia. Language resources and evaluation, 54(1):273-301.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jon Andoni Duñabeitia, Davide Crepaldi, Antje S Meyer, Boris New, Christos Pliatsikas, Eva Smolka, and Marc Brysbaert. 2018. Multipic: A standardized set of 750 drawings with norms for six European languages. Quarterly Journal of Experimental Psychology, 71(4):808-816.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, pages 215-233, Copenhagen, Denmark. Association for Computational Linguistics.
Nicholas Evans and Stephen C. Levinson. 2009. The myth of language universals: Language diversity and its importance for cognitive science. Behavioral and Brain Sciences, 32(5):429-448.
Pascale Fung and Lo Yuen Yee. 1998. An IR approach for translating new words from nonparallel, comparable texts. In COLING 1998 Volume 1: The 17th International Conference on Computational Linguistics.
Spandana Gella, Rico Sennrich, Frank Keller, and Mirella Lapata. 2017. Image pivoting for learning multilingual multimodal representations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2839-2845, Copenhagen, Denmark. Association for Computational Linguistics.
Harald Hammarstrom, Robert Forkel, Martin Haspelmath, and Sebastian Bank. 2020. Glottolog 4.3. Jena.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition.
Nathan Keirns Heather Griffiths. 2015. Introduction to sociology 2e. In Introduction to Sociology 2e/Culture/elements-of-culture, chapter 3. OpenStax, Oxford.
John Hewitt, Daphne Ippolito, Brendan Callahan, Reno Kriz, Derry Tanti Wijaya, and Chris Callison-Burch. 2018. Learning translations via images with a massively multilingual image dataset. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2566-2576, Melbourne, Australia. Association for Computational Linguistics.
Elenaz Janfaza, A Assemi, and SS Dehghan. 2012. Language, translation, and culture. In International Conference on Language, Medias and Culture, volume 33, pages 83-87.
Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282-6293, Online. Association for Computational Linguistics.
A. Karpathy and L. Fei-Fei. 2017. Deep visual-semantic alignments for generating image descriptions. IEEE Transactions on Pattern Analysis and Machine Intelligence, 39(4):664-676.
Marc A Kastner, Ichiro Ide, Frank Nack, Yasutomo Kawanishi, Takatsugu Hirayama, Daisuke Deguchi, and Hiroshi Murase. 2020. Estimating the imageability of words by mining visual characteristics from crawled image data. Multimedia Tools and Applications, pages 1-33.
Douwe Kiela, Felix Hill, Anna Korhonen, and Stephen Clark. 2014. Improving multi-modal representations using image dispersion: Why less is sometimes more. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 835-841, Baltimore, Maryland. Association for Computational Linguistics.
Douwe Kiela, Ivan Vulić, and Stephen Clark. 2015. Visual bilingual lexicon induction with transferred ConvNet features. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 148-158, Lisbon, Portugal. Association for Computational Linguistics.
Donghyun Kim, Kuniaki Saito, Kate Saenko, Stan Sclaroff, and Bryan Plummer. 2020. Mule: Multimodal universal language embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11254-11261.
Tom Kocmi and Ondrej Bojar. 2018. Trivial transfer learning for low-resource neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 244-252, Brussels, Belgium. Association for Computational Linguistics.
Tuomo Korenius, Jorma Laurikkala, and Martti Juhola. 2007. On principal component analysis, cosine and euclidean measures in information retrieval. Information Sciences, 177(22):4893-4905.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc'Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representations.