Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations
Authors: Chengpeng Fu
Xiaocheng Feng
Yichong Huang
Wenshuai Huo
Hui Wang
Bing Qin
Ting Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: Unsupervised Machine Translation
Cross-modal Machine Translation
Word-level Image
TL;DR: Improving unsupervised machine translation with word-level visual representation to address lexical confusion

Abstract: Unsupervised neural machine translation has recently made remarkable strides, achieving impressive results with the exclusive use of monolingual corpora. Nonetheless, these methods still exhibit fundamental flaws, such as confusing similar words. A straightforward remedy to rectify this drawback is to employ bilingual dictionaries, however, high-quality bilingual dictionaries can be costly to obtain. To overcome this limitation, we propose a method that incorporates images at the word level to augment the lexical mappings. Specifically, our method inserts visual representations into the model, modifying the corresponding embedding layer information. Besides, a visible matrix is adopted to isolate the impact of images on other unrelated words. Experiments on the Multi30k dataset with over 300,000 self-collected images validate the effectiveness in generating more accurate word translation, achieving an improvement of up to $+$2.81 BLEU score, which is comparable or even superior to using bilingual dictionaries.

Enabling Unsupervised Neural Machine Translation with Word-level Visual Representations
Chengpeng 12 Xiaocheng Feng Yichong Huang', Wenshuai Huo!
Hui Wang", Bin Qin Ting Liu
Harbin Institution of Technology, Harbin, China
Pengchheng Laboratory, Shenzhen, China fuchpêstu. hit. edu. cn (huowsh, wangh06 aatcllacccom {xcfeng,
soeffengyychuang.o ychuang, qinb, tliuj@ir.hi www.rr.iii..uu....... edu.
Abstract
as confusing nouns that pertain to the same mantic category. For instance, Bapna et al. (2022) demonstrates that their models translate "lion" to "rabbit" or "snake" despite getting good CHRF scores. In the context of supervised machine trans- lation, pinpointing accurate translations of words is straightforward with aligned bilingual data. How- ever, with only monolingual data, the unsupervised model must rely on large amounts of context for word mapping. which can lead to confusion be- tween words with similar distributions, particularly when there is insufficient data. To address this is- sue, a direct solution is to integrate high-quality bilingual dictionaries into the UNMT process us- ing codeswitching or spliced methods (Jones et al., 2023). High-quality bilingual dictionaries can be obtained through manual annotation and bilingual data extraction, yet these methods either involve high costs or deviate from the original objective of the UNMT task.
Unsupervised neural machine translation has recently made remarkable strides, achieving impressive results with the exclusive of monolingual corpora. Nonetheless, these meth- ods still exhibit fundamental flaws, such as confusing similar words. A straightforward remedy to rectify this drawback is to employ bilingual dictionaries, however, high-qu qualitya bilingual dictionaries can be costly to obtain. To overcome this limitation, we propose method that incorporates images at the word level to augment the lexical mappings. Specif- ically, our method inserts visual representa- tions into the model, modifying the correspond- ing embedding layer information. Besides, a visible matrix is adopted to isolate the im- pact of images on other unrelated words. Ex- periments on the Multi30k dataset with over 300,000 self-collected images validate the ef- fectiveness in generating more accurate word translation, achieving an improvement of up +-2.1 BLEU score, which is comparable or even superior to using bilingual dictionaries.
In addition to utilizing explicit bilingual dictio- naries for word mapping, implicit methods can also be employed. As the saying goes, "A pic- ture is worth a thousand words". We can leverage the "thousand words" represented by correspond- ing images to learn the mapping between words in different languages. Visual content has the po- tential to enhance word mapping abilities in latent spaces, as the physical visual perception is congru- ent among speakers of diverse languages (Huang et al., 2020). Furthermore, the cost of annotating a bilingual dictionary is relatively high, whereas the cost of searching images with monolingual key- words is relatively low in the era of sophisticated search engines. Describing image is also less demanding for humans compared to translating word, as the former only requires mastery of one language while the latter requires at least two.
1 Introduction
Unsupervised Neural Machine Translation (UNMT) (Zhang et al., 2017: Artetxe al.. 2018; Lample et al., 2018a,b,c) seeks to avoid the necessity for high-quality bilingual data in conventional machine translation by exploring the possibility of completing translations relying solely on the monolingual corpus. Consisting of three main components, cross-lingual language model initialization, denoising auto-encoder, and iterative back translation, recent methods (Conncau and Lample, 2019; Song al., 2019) have achieved promising results.
Nonetheless, recent UNMT approaches have been observed to make particular errors in com- parison to supervised machine translation, such
Drawing on these insights, we propose to inte- grate word-level image information into the UNMT process to mitigate the issue of lexical confusion.
'The word-image dataset and associated codes are publicly available at: https://github.com com/ cow//rraaaany99UUMMMMM

---

It's worth noting that we don't use sentence-level images like some multi-modal UNMT approaches (Su et al. (2019); Huang et al. (2020)). Instead, we focused on word-level images since they are more effective and flexible in addressing the chal- lenge of lexical confusion and are easier to collect. Our approach involves augmenting the encoded images to their corresponding words, modifyyng the corresponding position and language encoding information, and utilizing visible matrix to isolate the impact of images on other words. We conduct experiments utilizing over 300,000 self-collected images, and further analysis demonstrates that we have effectively mitigated lexical confusion to certain extent. Furthermore, we achieve new state- of-the-art results for the UNMT task in certain lan- guage directions the Multi30k dataset. The main contributions of this work can be summarized as follows:
Initialization. Performing parameter initializa- tion that is non-random can introduce cross-lingual prior knowledge into the model. There are cur- rently two main initialization methods for UNMT models. The first method initializes the embed- ding layer of a UNMT model with word pretrained embeddings, while the second method uses pre- trained cross- tross-lingual language model with the same structure the UNMT encoder initialize most of the neural network parameters.
Iterative Back-translation. fack-translation is method that leverages monolingual data the target language and a reverse translation model to generate pscudo-source data, improving the qual- ity of translations in the forward direction. In UNMT, where only monolingual data is available, back-translation is used to generate aseudo-paralle, data, transforming the unsupervised problem into weakly or self-supervised one. During training, back-translation repeatedly applied an iterative manner. The loss is defined as followed:
We introduce a concise yet potent framework that integrates word-level images to alleviate the similar lexical confusion issue in UNMT, without utilizing bilingual dictionaries.
CB
(X,
Our approach has achieved promising or even unparalleled unsupervised results on the Multi30K dataset for certain language pairs.
-log (10 (1)
where PL.-LI denotes the translation prabaabiiopoobbbilltttiiii probabilities between two languages and St,-14, denotes the translation processes from language Lg to Lg.
We release the word-image datasets for five languages as open-source, which as valuable resouree the community.
Denoising Autoencoder. The introduction of the denoising autoencoder aims to guide the model producing fluent text output. This is achieved by adding noise to sentences through techniques such as replacement, deletion, and shuffling, and then allowing the model to output the correct and fluent original sentence. The loss is defined as follows:
2 Methods
this section, we first review the text-only UNMT process. Then, we present our proposed method for incorporating images, which involves the modified embedding layer and mask transformer. Finally, we introduce some training strategies.
2.1 Unsupervised Neural Machine Translation
CD
Log (X,
Successful UNMT systems typically employ an encoder-decoder architecture similar to that of su- pervised machine translation, sharing parameters between the source-to-tante target and target-to aarget-tosouure models. They share several common principles: initialization, iterative mack-translation, and denois- ing autoencoder. Let X M denote mono- lingual data in language L1 and Y (Y.) in language T-z. M and N are the sentence number monolingual data and Y respectively. 0 rep- resents the UNMT model parameters.
where (-) refers to the noise functions. PL. Lh. denotes the reconstruction probabilities in T.g. The final objective function of UNMT is:
L LB + XpLp
(3)
where Ap hyper-paramett that weights de- noise autoencoder loss term and gradually re- duced during training.

---

Lorgusge Embedcinng
EN
EN
EN
EN
EN
I
EN
EN
EN
EN
EN
Resitional
Embedding
Token Embeskding
man
in
blue
shirt
riding
motor
euboot
Mappling Layar
Kesping Layer
Kippping
Mappingg Leyer
Visucl Erocderr
Visual Encoder
Visual Enicddr
Vicual Enooder
Tokenized Inserted Sentenoe
riding
motor
eu0ooT
Tokerize
Image-
Inserted
blue
shirt
riding
motorboct
Serfeniee
Segment
Word-Tmage Lookup Table
Monalingusl
man blus shirt noiry materboo!
Figure The process of converting an image-inserted sentence into an embedding representation.
2.2
Image Fusion
In this section, present how incorporate word- level images into the UNMT process tackle the issue of lexical confusion. The main concept be- hind our approach is that words can acquire in- formation not only from contextual information but also from their corresponding images. Let X; = [x] x? ..., x denote sentence from X in language L1 and zu denotes the k-th word in sentence X,. We denote lookup table consisting of word-image pairs Dr, {word, worrd) }- where wordd an image set with one or multiple images that depict the meaning of the word.
I1S =
(5)
where ITS = (TTS)) After obtaining the IIS, we need integrate them into the UNMT model. To achieve this in- tegration, we employ a modified embedding layer and a mask transformer.
2.2.1 Modified Embedding Layer
The first modification part is the embedding layer, which generally includes token embedding, posi- tion embedding, and, in some models, language embedding. An example can be seen in Figure 1.
As shown in Figure 12, we segment an input sen- tence X; at the word level and obtain the images by retrieving the words appearing in the lookup table DL.- Then, concatenate the retrieved im- ages the corresponding words, resulting in Image-Insericee Sentence denoted IIS, Each word can be concatenated with one or multiple images. Formally, We denote IIS; as follow:
Token Embedding. In the context of ITS, both textual words and images are present. Given that most current machine translation methods operate at the subword level, we proceed to tokenize the IIS, focusing solely on the textual component of IIS. For encoding textual tokens, we continue to utilize the embedding layer from the original text-only model. Regarding images, we leverage the CLIP (Radford et al., 2021) model as the im- age encoder to extract visual features. The CLIP model is multi-moda, model trained on variety of (image, text) pairs, which possesses powerful capability represent images. To prevent nega- tive impacts on the encoding of images during the UNMT process, we freeze the visual encoder. How- ever, directly using these representations may lead
IIS; [z),
(m) I, 2 II
1
(4)
where j, is the image subset of JaK (nH can be an empty set). We define this process of injecting images as:
"Bluc" is an adjective in the figure. In our image collec- tion process, "hlue" can also appear as a noun in other training samples.

---

a gap between the image encoding space and the semantic space in UNMT. Therefore, we employ an MLP layer to bridge this gap.
Position Embedding. Positional encoding is a crucial module that supplements token structural information. For textual information in ITS, we continue to encode the position in the original or- der of the input sentence to prevent interference with sentence structure learning when integrating images into the model. For images, we assign the positional encoding of the first subword in the tok- enized result for the corresponding word.
Figure An example of the visible matrix correspond- ing to the /IS in Figure 1. The red numerical labels indicate the absolute position of the images in the IIS. In this matrix, black represents "visible" and white rep- resents "invisible".
Language Embedding, Some UNMT models in- corporate language embedding to inform the model of the input language and the desired output lan- continue to
guage. For textual information, utilize the original language encoding. However, for image information, as it does not belong to any language, we introduce new image identifier.
Q'K
+ M
= =ofotma
(7)
Vd
where Q and K are the query matrix and key matrix in the self-attention module. if two elements are invisible, the M will mask the attention score O, which means making no contribution to each other.
2.2.2 Mask Transformer
Despite modifying the original embedding layer to allow IIS to be properly fed into the model, two issues still exist. Firstly, inserting images may af- fect other irrelevant words in the original sentence and potentially disrupt the structural relationships between them. Secondly, a single word-level image may correspond multiple subwords, and during the annotation of position embedding, only the in- formation of the first subword utilized, resulting lack of relationship between the image and the remaining subwords corresponds to. To ad- dress these issues, we introduce the method of mask transformer (Liu et al., 2020).
2.3 Training Strategy
After initialization by cross-lingual pretrained lan- guage model, we train denoising autoencoder and iterative back-translation alternatively. The final loss of image fusion UNMT is:
- C + +DE
(8)
specifically, C/2 and Ct, refer to:
Ch
We create a visible matrix to shield the relation- ship between images and other textual information in the sentence and highlight the relationship be- tween the image and its corresponding subwords. The visible matrix M defined as:
log (10
-00
Wp
Wig
C
log P (X)
Mi
(6)
Wp Wq.
(10)
where Wp and Wy are elements IIS; Up o Wy denotes that up and Wig are visible to each other, while w is not. An example of a visible matrix is depicted in Figure
During the denoising autoencoder process, we first add noise to the original sentence and then integrate the images. During the iterative back- translation process, for the first round of synthetic data generation, we do not add image information to achieve a smoother transition because the pre- trained language model did not include images.
The visible matrix can be added the self- attention module mask certain attention calcula- tions. specifically, the new masked self-attention is:

---

Language
# Word
#Image
Average
Monolingual Dataset. The monolingual datasets used in our experiments are Multi30k dataset (El- liott et al.. 2016, 2017; Barrault et al., 2018), LDC dataset and WMT News Crawl dataset. The Multi30k dataset contains images and their cap- tions in four languages: English(En), French(Fr), Germany(De), and Czech(Cs). Every language has 29000 sentences for training and 1014 for valida- tion. It is worth noting that only utilized the lin- gual data from Multi30k as the images are sentence- level. To prevent the model from encountering bilingual corpora, we randomly split the training and validation sets of the Multi30k dataset into two equal parts. Each part of the training set contains 14,500 sentences, and each part of the validation set contains 507 sentences. We report results in all test sets in Multi30k including Test2016, Test2017, Test2018, and MsCOCO, For the Cs-En task, only Test2016 and Test2018 are available. Furthermore, we expanded the dataset by incorporating monolin- gual data from WMT 2008-2022 News Crawl and LDC. We also report our results on the common news domain test set, which includes WMT New- stest 2014 for En-Fr, Newstest 2021 for En-Cs, and NIST06 for En-Zh.
En
4,975
45,228
9.09
Fr
4,646
36,927
7.95
De
10,384
85,419
8.23
Cs 10,706
72,942
6.81
Zh 10,805
85,144
7.88
Total
41,516
325,660
7.84
Table I: The statistics of our constructed Word-Image dataset.
3 Experiments
In this section, we provide a detailed description of the datasets and experimental setup employed in study. Subsequently, perform in-depth analysis of the results obtained.
3.1 Datasets
Word-Image Dataset. In our approach, the most crucial component of the experiment data is the im- age dataset at the word level. Since there is no off- the-shelf word-level image dataset, we build one for performance evaluation. We collect these im- ages through the following steps: word extraction, image crawling, and image filtering. Firstly, extract the nouns from the monolingual dataset using part- of-speech analysis tools including Spacy² and Stan- ford CoreNLP'. We choose nouns because they typically correspond to real-world entities and are more likely to yield high-quality images. Next, use Google Images" to crawl images for these nouns, limiting the image size to between 200-by-200 and 10,000-by-10,000 pixels and retrieving 10 15 images per noun. The image format is restricted to JPG, JPEG, and PNG. Finally, filter the crawled images. We consider all the images collected for a single noun as cluster and encode them using the CLIP model. We then calculate pairwise similari- ties between all images and remove those with an average similarity below threshold T, which set to 0.6, they are considered anomalous. The statistics information of the Word-Image dataset can be seen in Table Our dataset covers a wide range common nouns and can be reused for vari- ous purposes.
3.2 Experimental Details
Baseline Models. We compare recent unsuper- vised text-only and multimodal MT baselines listed in the following: XLM (Conneau and Lample, 2019) uses masked language model train a cross- cross-lingual language model to initialize. (2) UMMT (Su et al.. 2019) uses visual features for de- noising autoencoder and back neck-translation. (3) PVP (Huang et al., 2020) employs multimodal back- translation and features pseudo visual pivoting. (4) UNMT-CS (Jones et al., 2023) uses the codeswitch- ing method, where words in the source sentence are swapped out for their corresponding translation from bilingual dictionaries, to solve the lexical con- fusion issue in UNMT. It is worth noting that both UMMT and PVP datasets utilize sentence-level images.
System Settings. We conducted our experiments based on the XLM codebase The model architec- ture employed is a 6-layers, 8-heads transformer, with a hidden layer representation dimension 1024. The epoch size is 100K and the tokens-per batch is 2000. we use Adam optimizer and 2000 warm-up updates. The learning rate is 1e-5. For
"hnus://ghhubb.com/rrpplsssgsggu "hanss//imagee.googgc..om

---

En-Fr
En-De
En-Cs
Approach
F16
F17
F18
C17
F16
F17
F18
C17
Fl6
F18
UMMT
39.80
23.50
PVP
52.30
33.90
XLM
53.08
46.82
35.31
43.09
31.81
26.83
25.62
22.95
25.14
20.15
UNMT-CS
54.24
46.30
36.06
43.49
33.43
27.51
26.67
24.27
26.46
22.71
Ours
54.07
46.95
36.21
44.44
33.06
29.12
27.37
24.75
26.34
22.88
+0.99
+0.13
+0.90
+1.35
+1.25
+2.29
+1.75
+1.80
+1.20
+2.73
Fr-En
De- En
Cs-En
Approach
F16
F17
F18
C17
F16
F17
F18
C17
F16
UMMT
40.50
26. 26.40
PVP
46.00
31.60*
XLM
48.04
42.14
37.03
43.58
37.51
33.21
30.73
27.98
33.13
30.29
UNMT-CS
49.28
42.86
37.19
44.39
38.83
34.48
31.44
28.07
33.91
31.33
Ours
49.10
43.01
37.41
44.57
40.32
35.20
32.70
29.81
34.29
31.56
A
+1.06
+0.87
+0.38
+0.99
+2.81
+1.99
+1.97
+1.83
+1.16
+1.27
Table 2: BLEU scores Multi30k En-Fr, En-De, and En-Cs UNMT tasks. F18, F17, F16, and C17 respectively denote the Flickr2018, Flickr2017, Flickr2016, and COCO2017 test sets in Multi30k. is the results reported from previous papers. A represents the improvement our method over the XLM method terms of BLEU.
evaluation, we use gramm BLEU (Papineni et al., 2002) scores by multi-bleu.pl in Moses?. We train all models on 4 NVIDIA 32GB V100 GPUs.
ages leads to performance improvements in all six translation directions, with a maximum increase of +2.81 BLEU points, indicating the effectiveness of approach. UNMT-CS is method that utilizes bilingual dictionaries to address the problem of lex- ical confusion via code-switching. By comparing our method with the UNMT-CS approach, we ob- served comparable results, and some directions, our method even outperformed the code-switching approach, indicating that ours is more efficient and cost-effective.
During training, first train the models 5M WMT News Crawl dataset for 70 epochs and subsequently continued training on the Multi30k dataset for another 70 epochs due to relatively small size. As for the CLIP model, we use ViT- L/14@ @ 336px. When incorporating images, we ran- domly select two images from the corresponding image set for each word and concatenate them to the text. We only incorporate images during the it- erative back-translation and denoising autoencoder stages, so our method can be used in conjunction with other pre-training enhancement methods. Re- garding the cross- lingull models used for initializa- tion, we directly employ high-quality open-source cross-lingual models in experiments involving En- Fr, En-De, and En-Zh. However, for the En-Cs, we train model using WMT monolingual dataset, with 10 million English sentences and 10 million Czech sentences, due the unavailability of any open-source models.
When compared to the best-performing method that incorporates sentence-level images, our ap- proach still achieves comparable or better results in most translation directions. This is despite the matching degree between the Multi30k monolin- gual data and word-level images being slightly lower than that of sentence-level images, as the text data in Multi30k generated based on the sentence-level images. In some directions, our ap- proach even outperforms the previous best methods by up to +4.22 BLEU points (as shown in Table 2, where our method is compared to the first two methods of the table).
3.3 Main Results
Table shows the main results on the Multi30k test set, including translating among English, French, German, and Czech. Comparing the results of XLM and our method, we observe that adding im-
Analysis
4.1
Case Study
In Table 3, we provide several examples transla- tions, where we observe that the XLM method re-

---

GT-DE
Mensehen an der Seitenlinie bei einem Fuliballspiel.
GT-EN
People on the sideline of soccer match.
XLM
People at the plate in baseball game.
Ours
People at the stands in a soccer game.
GT-DE
Fin kleines Kind in einer omangeaarhenen Rellungsweste hält ein Paddel und packlelt einem blanen Kajak auf einem Gewässer.
GTEN
A young child wearing an orange life vest holding an our paddling blue kayak in body of water.
XLM
small child in orange-red vest holds a Paddy Power puddles
cadet a body water.
Ouss
small child in orange life bolding paddle and paddling blue kayak a body of water.
homme avec veste
casquette carreaux et pantalon rayé
joue guitare électrique une
GT-RR
avec chanteur
guitariste arricre-plan.
A man black jacket and chackered hat wearing black and white striped plays electric guilar siage with singer and another guitar player in the background.
GT-EN
A man with a black jacket, striped headhand and light hrown and white shirt plays an electric guitar on stage with singer and another guitarist in the background.
XLM
A man with a black jacket, cheokered cup and black and white striped pants plays an electric guitar on a stage with singer and another guitarist the background.
Ours
Table Qualitative results the proposed model. GT denotes ground truth. Words highlighted in red are affected by the problem of lexical confusion.
incorporating images for various translation direc- tions on the Flickr2016 and Flickr2017 test sets and find that our method with image incorporation outperforms the method without images in all di- rections. This demonstrates that our method can indeed provide more accurate word translations.
74
74.
75.0
710
762
68-FR
EM-5R
FR:SN
F8-5N
F6:05 P17
FM:05
DE-EN
DE.FK
EM-CS
C5.SM
4.3 Performance on WMT and LDC Dataset
F17
128
F16
Figure 3: 1-gram precision scores on Flickr2016 and Flickr2017 data sets.
Not all sentences have a corresponding image that perfectly matches the text. On the other hand, gen- erally, each sentence has some words that can be matched with images, making the use of word- level images more flexible and less costly to collect. Hence, we also conduct experiments on a more common news dataset. We conduct experiments on En-Fr, En-Cs, and En-Zh translation directions us- ing the WMT 5M monolingual data for En-Fi and En-Cs, and the LDC dataset for En-Zh. We split the LDC bilingual data into two non-overlapping parts, each containing 0.75 million monolingual sentences. The news dataset is more abstract in content compared to the Multi30k dataset, and it is difficult to find an image that perfectly matches a sentence. This makes it challenging or almost impossible to apply previous methods that rely on sentence-leve images to this dataset. As shown in Table 4, our method can be applied to the news dataset, and it shows improvements compared to the method that only uses text. The improvement on the WMT dataset was not significant, and we speculate that this may be due to the sufficient size of the WMT dataset and the lower frequency of noun entities with images compared to Multi30k.
sults in some lexical confusion errors. For instance, in Example 1, "Fußballspiel" is translated to "base- ball" instead of "soccer", and in Example 2, "or- angefarbenen" is translated "orange-red" instead of "orange". The representation of these words is similar, which makes it difficult for the model to differentiate between them. However, when we incorporate image information, the words obtain additional information from the image representa- tion, which helps alleviate this problem to some extent. In Table 3, we can see that these words are correctly translated when we add image infor- mation. Therefore, our approach can effectively address the issue of lexical confusion UNMT.
4.2 1-gram Precision Scores
To better observe how method can alleviate the issue of lexical confusion, we report the 1-gram precision scores in Figure 3. The accuracy of the model's word translations can be reflected intu- itively in the 1 1-gram precision scores metric. We report the results of our approach with and without

---

Approach
Zh-Fm
En-Zh
Cs-Fi
En-Cs
Fr-Bi
En-Frr
Song et al., 2019; Han et al., 2021) in UNMT have achieved remarkable milestones. However, there are still certain issues associated with solely uti- lizing monolingual data for translation, such as domain mismatch problem (Marchisio et al., 2020; Kim et al., 2020), poor performance on distance language pairs (Kim et al., 2020; Chronopoulou et al., 2020; Sun et al., 2021), translationese (He et al., 2022), and lexical confusion (Bapna et al., 2022; Jones et al., 2023). Our work aims to address the issue of lexical confusion by utilizing word- level image data, thereby climinating the need for costly bilingual dictionary annotations.
XLM
8.43
7.91
25.14
17.79
33.73
35.77
Our Method
9.87
9.36
25.51
18.24
34.14
36.46
A
+1.44
+145
+0.37
+0.45
+0.41
+0.69
Table 4: BLUE scores on WMT and LDC test sets.
34.45
34.4
34.39
$4.3
34:27
34.25
34.21
34.2
34.15
Multimodal Machine Translation. Multimodal machine translation involves incorporating image information into a machine translation model enhance its performance. While most current mul- timodal machine translation systems (Caglayan et al., 2016; Yao and Wan, 2020; Yin et al., 2020; Caglayan et al., 2021; Zhang et al., 2020; Fang and Feng, 2022: Li et al., 2022) focus on supervised machine translation, it has been observed that vi- sual information is generally ignored when parallel corpora are plentiful (Caglayan et al., 2019; Wu et al., 2021). Yang al. (2022) proposed the use of image information in situations where parallel cor- pora are not abundant. On the other hand, Su et al. (2019) and Huang et al. (2020) utilized image infor- mation in an unsupervised setting. However, these approaches were based on sentence-level image information, which can be challenging to obtain since not all sentences have corresponding images. In contrast, word-level image information is eas- ier to obtain and is more flexible and lightweight. Therefore, our work aims to use word-level image information to enhance the performance of UNMT.
34.1
BEST-1
BEST-Z
RANDOM-2 RANDOM-3
Figure 4: BLEU scores En-De with different ways and numbers of adding image.
4.4 Quantity of Images
We are also curious about how the method of adding images and image quantity affect the perfor- mance. Therefore, we test several ways of adding images, including selecting the best one, the top two, randomly selecting one, randomly selecting two, and randomly selecting three. We treat all images corresponding to a single word as a class and selected the image with the highest average similarity to the other images in the same class as the best image. We report the average BLEU score of En-De on the Flickr2016 and Flickr2017 test sets as illustrated in figure 4. We find that the fixed- best method yielded lower performance compared to randomly adding images. This could be due to the limitation of the fixed-best method in capturing the full range of visual variations that may exist for given word. Among all the random selection methods, we find that adding two images randomly yielded the best performance. This could be be- cause adding three images may introduce more noise, while adding only one image may not pro- vide enough randomness.
Conclusion and Future Work
In this paper, we propose method to alleviate the problem of lexical confusion in unsupervised machine translation by utilizing word-level images and provide a concise yet potent framework for integrating them into an unsupervised neural ma- chine translation model. Through extensive exper- iments, we demonstrate that our method can ef- fectively mitigate the issue of word confusion and even outperform the use of costly bilingual dictio- naries in some directions. Furthermore, we release an open-source word-image dataset that covers 5 languages and involves over 300,000 images in to- tal. In the future, we will endeavor to explore the
5 Related Work
Unsupervised Machine Translation. Unlike common machine translation (Sutskever et al., 2014; Vaswani et al., 2017: Feng et al., 2020), UNMT don't 't use parallel corpus. Recent advance- ments (Zhang et al., 2017; Artetxe et al., 2018;
Lample et al., 201 Sa; Conneau and Lample, 2019;

---

potential of utilizing images for augmenting the translation quality of low-resource languages in a largellnguage-model-eenen translation system.
Ankur Bapna, Isaac Caswell, Julia Kreutzer, Orhan Fi- rat, Daan van Esch, Aditya Siddhant, Mengmeng Niu, Pallavi Baljekar, Xavier Garcia, Wolfgang Macherey, et al. 2022. Building machine translation systems for the next thousand languages. arXiv preprint urXiv:2205.03985
Limitations
One limitation of our work is that we have not been able to effectively handle the issuc of polysemy, The word-image dataset is obtained through direct web scraping of words, and the images collected may not all correspond to the same sense of the word. Moreover, when adding images, we have not been able to accurately select images that cor- respond to the intended meaning of the word in the sentence. However, we have attempted to alleviate this issue to some extent by randomly selecting images during the process of adding images, which has enabled the model encounter the correct im- ages to a certain degree. In the future, we plan to improve our approach by incorporating sentence- level information when inserting images.
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. 2018. Find- ings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 304-323.
Ozan Caglayan, Loie Barrault, and Fethi Bougares. 2016. Multimodal attention for neural machine trans- lation. arXiv preprint arXiv: 160.03960
Ozan Caglayan, Menekse Kuyu Mustafa Serean Amac. Pranava Madhyastha, Erkut Erdem, Aykut Erdem, and Lucia Specia. 2021. Cross-lingus visual pre- training for multimodal machine translation. In Pro- ceedings of the 16th Conference of the European Chapter of the Association for Computational Lin- guisties: Main Volume, pages 1317-1324, Online. Association for Computational Linguistics.
Ethics Statement
Ozan Caglayan, Pranava Madhyastha, Lucia Specia, and Loic Barrault. 2019. Probing the need for visual context in multimodal machine translation. In Pro- ceedings of the 2019 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Pupers), pages 4159-4170, Min- neapolis, Minnesota. Association for Computational Linguistics.
In our experimental endeavors, we have amassed a substantial collection of images, which naturally prompts discussions pertaining to these issues. is imperative to clarify that we do not possess the copyright for these images. Instead, they will be exclusively available to researchers and educators seeking to utilize the dataset for non com-cmamecaaal research and/or educational purposes.
Alexandra Chronopoulou, Dario Stojanovski. and Alexander Fraser. 2020. Reusing a Pretrained Lan- guage Model Languages with Limited Corpora for Unsupervised NMT. In Proceedings the 2020 Con- ference on Empirical Methods in Natural Language Processing (FMNLP)), pages 2703-2711, Online. As- sociation for Computational Linguistics.
Acknowledgements
Xiaocheng Feng is the corresponding author of this work. We thank the anonymous reviewers for their insightful comments. This work was sup- ported by the National Key RD Program of China via grant No. 2021ZD01 12905, National Natural Science Foundation of China (NSFC) via grant 62276078, the Key RD Program of Heilongjiang via grant 2022ZX01 ation Project of PCLPCL2022001 and the Funda- mental Research Funds for the Central Universities (Grant No.HIT.0CEE..0230188)
Alexis Conneau and Guillaume Lample. 2019. Cross- lingual language model pretraining. Advances in neural information processing systems, 32.
the International Cooper-
Desmond Elliott, Stella Frank, Loic Barrault Fethi Bougares, and Lucia Specia. 2017. Findings of the second shared task on multimodal machine transla- tion and multilingual image description. In Proceed- ings of the Second Conference on Machine Transla- tion, pages 215-233, Copenhagen, Denmark. Associ- ation for Computational Linguistics.
References
Mikel Artetxe, Gorka Labaka, Eneko Agiree, and Kyunghyun Cho. 2018 Unsupervised neural ma- chine translation. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC. Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net.
Desmond Elliott, Stella Frank, Khalil Sima' an, and Lu- cia Specia. 2016. Multi30K: Multilingual English- German image descriptions. In Proceedings of the 5th Workshop Vision and Language, pages 70- 74, Berlin, Germany. Association for Computational Linguistics.

---

Qingkai Fang and Yang Feng. 2022. Neural machine translation with phrase-lever universal visual repre- sentations. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Pupers), pages 5687-5698, Dublin, Ireland. Association for Computational Linguisties.
In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 5039-5049, Brussels, Belgium. Association for Com- putational Linguistics.
Bei Li, Chuanhao Lv, Zefan Zhou, Tao Zhou, Tong Xiao, Anxiang Ma, and JingBo Zhu. 2022. On vi- sion features in multimodal machine translation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 6327-6337, Dublin, Ireland. As- sociation for Computational Linguistics.
Xiaocheng Feng, Zhangyin Feng. Wanlong Zhao, Bing Qin, and Ting Liu. 2020. Enhanced neural machine translation by joint decoding with word and pos- tagging sequences. Mobile Networks and Applica- tions, 15222-217228.
Jesse Michael Han, Igor Babuschkin, Harrison Edwards, Arvind Neelakantan, Tao Xu, Stanislas Polu, Alex Ray, Pranav Shyam, Aditya Ramesh, Alec Radford, al. 2021. Unsupervised neural machine translation with generative language models only, arXiv preprint 10..5:21
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang. Qi Ju, Haotang Deng, and Ping Wang, 2020. K-bert: En- abling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2901-2908.
Zhiwei Hc, Xing Wang, Rui Wang, Shuming Shi, and Zhaopeng Tu. 2022. Bridging the data gap between training and inference for unsupervised neural chine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 661 1-6623, Dublin, Ireland. Association for Computational Lin- guistics.
Kelly Marchisio, Kevin Duh, and Philipp Koehn. 2020. When does unsupervised machine translation work? In Proceedings of the Fifth Conference on Machine Translation, pages 571-583, Online. Association for Computational Linguistics.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing 7hu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- national Linguistics, pages 31 1-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Po-Yao Yao Huang, Junjie Hu, Xiaojun Chang, and Alexan- der Hauptmann. 2020. Unsupervised multimodal neural machine translation with pseudo visual pivot- ing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8226-828 8237, Online. Association for Computational Linguistics.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International confer- ence on machine learning, pages 8748-8763 PMLR.
Alex Jones, Isaac Caswell, Ishank Saxena, and Orhan Firat. 2023. Bilex rx: Lexical data augmentation for massively multilingual machine translation. CoRR, abs/2303.15265.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019. Mass: Masked sequence to sequence pre- training for language generation. In International Conference on Machine Learning, pages 5926-5936. PMLR
Yunsu Kim, Miguel Graça, and Hermann Ney. 2020. When and why is unsupervised neural machine trans- lation useless? In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 3544, Lisboa, Portugal. Euro- pean Association for Machine Translation.
Yuanhang Su, Kai Fan, Nguyen Bach, C-C Jay Kuo, and Fei Huang. 2019. Unsupervised multi-modal neural machine translation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pat- tern Recognition, pages 10482-10491.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Mare' Aurelio Ranzato. 2018a. Unsupervised machine translation using monolingual corpora only. In International Conference on Learning Representa- tions.
Haipeng Sun, Rui Wang, Masao Utiyama, Benjamin Marie, Kehai Chen, Eiichiro Sumita, and Tiejun Zhao, 2021. Unsupervised neural machine trans- lation for similar and distant language pairs: An em- pirical study. ACM Transactions on Asian and Low- Resource Language Information Processing (TAL- LIP), 20(1):1-17.
Guillaume Lample, Alexis Conneau, Marc' Aurelio Ran- zato, Ludovic Denoyer, and Hervé Jégou. 2018b. Word translation without parallel data. In 6th In- lernational Conference on Learning Representations. ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenRe- view.net.
Ilya Sutskever, Oriol Vinyals, and Quoe V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems. 27.
Guillaume Lample, Myle Ott, Alexis Conneau, Ludovic Denoyer, and Mare Aurelio Ranzato. 2018c. Phrase- based & neural unsupervised machine translation.

---

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
the Ground-truth bilingual dictionaries provided in MUSE (Lample et al., 2018b). When training. we substituted words 50% of the sentences, while the remaining sentences were left unchanged. All other parameters used in the XLM and UNMT-CS methods are identical to those used in our proposed method, described in our paper.
Zhiyong Wu, Lingpeng Kong, Wei Bi, Xiang Li, and Ben Kao. 2021. Good for misconceived reasons: An empirical revisiting on the need for visual context in multimodal machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers). pages 1153-61666 Online. Association for Computational Linguistics.
B
The Implicit Mapping Relationship Between Image Representations
In fact, our approach doesn't explicitly provide the model with a mapping relationship. Rather, we introduce the (word-image) pairs relationship into the model, enabling it to extract information from similar images and thus enhancing the mapping relationship between words.
Zhe Yang, Qingkai Fang, and Yang Feng. 2022. Low- resource neural machine translation with cross-modal alignment. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing, pages 10144, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
We are also curious about the impact on results the implicit mapping relationship between image representations was absent. Therefore, during the second stage of multi30k training, we replace all images from one language with randomly gener- ated representations while keeping the other lan- guage's s images unchanged. The En En->De direction results for the Flickr2016 and Flickr2017 datasets experience a significant decline, decreasing from 40.32/35.20 to 39.17/34.29. This confirms the cru- cial importance of the mapping relationship be- tween images for our approach.
Shaowei Yao and Xiaojun Wan. 2020. Multimodal transformer for multimodal machine translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 4346- 4350, Online. Association for Computational Lin- guistics.
Yongjing Yin, Fandong Meng, Jinsong Su, Chulun Zhou, Zhengyuan Yang. Jie Zhou, and Jiebo Luo. 2020. A novel graph-based based multi-modal fusion en- coder for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025-3035, On- line. Association for Computational Linguistics.
Meng Zhang, Yang Liu, Huanbo Luan, and Maosong Sun. 2017. Adversarial training for unsupervised bilingual lexicon induction. In Proceedings of the 55th Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pages 1959-19700 Vancouver, Canada. Association for Computational Linguistics.
Zhuosheng Zhang, Kehai Chen, Rui Wang, Masao Utyaama, Eiichiro Sumita, Zuchao Li, and Hai Zhao. 2020. Neural machine translation with universal vi- sual representation. In International Conference on Learning Representations.
A
The Implementation Details of the XLM and UNMT-CS Method
The XLM and UNMT-CS methods are re- implemented on the Multi30k dataset in our paper. During training, both these two methods are trained on the 5 million sentences from the WMT dataset for 70 epochs and subsequently continued training for 70 epochs on the Multi30k dataset. For the bilin- gual dictionary in codeswitching method, we use


Reviews:
[{"review_id": "IJYoVqABgH", "reviewer": "Reviewer_eGcr", "summary": "", "questions": "a) How do you select the nouns for the word-image lookup table? and How do you ensure the coverage and quality of the word-image pairs?\n\nb) How do you handle noisy or irrelevant images that may be crawled from Google Images?\n\nc) How do you handle words that have multiple meanings or senses, and may correspond to different images depending on the context? Do you use any word sense disambiguation technique or select images dynamically based on the sentence?\n\nd) How do you deal with named entities? Do you use any placeholder or fallback mechanism? How does this affect the performance of UNMT?\n\ne) How do you evaluate the impact of word-level images on the latent space of UNMT? Do you use any visualization or analysis tool to show how the images affect the word representations or alignments? Do you have any examples or cases that illustrate this impact?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you so much for your insightful comments and affirmation of this work. Please allow us to address the issues you raise:\n\n**Reply to Questions**\n\n**a) How to select the nouns and how do you ensure the coverage.**\n\nWe elaborated on these details in Section 3.1 of our paper. Specifically, we utilize tools such as Spacy to perform part-of-speech analysis on the dataset and extract nouns. By choosing all the nouns within the monolingual dataset, we can ensure a guaranteed coverage rate.\n\n**b) How do you handle noisy or irrelevant images.**\n\nWe presented the filter details in Section 3.1 of our paper. We encode all the images using the CLIP model and then calculate pairwise similarities. We remove those outlier images whose average similarity with other images falls below a certain threshold. We also randomly selected a subset of the data for manual evaluation, where the relevance was found to be greater than 95%.\n\n**c) Polysemy.**\n\nWe discussed polysemy in Section Limitations.\nTo mitigate this issue in some degree, we randomly select images during the image fusion process, which has provided the model with exposure to the correct images to a certain extent. \nA more precise approach to address this issue is to calculate the similarities between the context-aware word embeddings and image representations via the multilingual multi-modal language models (MMLMs). However, existing MMLMs rely on bilingual parallel text, which betrays the intention of UNMT. Considering the costly endeavor of training a new MMLM from scratch without using bilingual corpora, we deem the efficient resolution of polysemy as a valuable area for future exploration. \n\n**d) Deal with named entities.**\n\nIn our paper, we specifically address the issue of word confusion with simple nouns and we have not yet considered complex named entities, as these entities are less prone to confusion.\nCertainly, we can employ the same methodology for handling named entities. In this case, we would first need to identify the named entities in the text after which we can gather images at the granularity of named entities.\nWe value your suggestion and will take it into consideration for future research, aiming to optimize the treatment of named entities in our methodology.\n\n**e) Visualization.**\n\nThank you so much for your advice. We will include some visualization to evaluate the impart of word-level images on the latent space. Specifically, we will utilize the t-SNE tool to demonstrate the representation of words before and after the insertion of images. \n\n**Rebuttal to Reject Reasons**\n\n**f) Analysis of the impact of different types or numbers of images on the translation quality.**\n\nWe conducted analysis experiments on different methods of selecting images and the number of images in Section 4.4. We found that randomly selecting two images on the Multi30k dataset yields favorable results. However, it is crucial to emphasize that all of these selection methods yield comparable results and make a significant contribution in addressing the issue of word confusion to some extent.\n\n\n**g) Word-level images in other NLP tasks.**\n\nIn the context of UNMT, images play a significant role by acting as intermediaries that help bridge the language gap and facilitate alignment. However, in general NLP tasks, images are primarily utilized for enhancing representations. Certainly, we also desire to compare our work with relevant research in other NLP tasks. However, we couldn't locate the specific papers you mentioned. Could you please provide the accurate titles of the papers? Thank you so much.\n\n**h) Experiments on more challenging language pairs or domains.**\n\nThe current unsupervised machine translation research often focuses on datasets such as WMT and Multi30k. \nIn our study, we also included an additional evaluation on the EN-ZH (language pair with different scripts) using part of LDC dataset. \nAs for different domains, Our approach is domain-agnostic and can be applied universally. We also test our method on WMT19 Medline EN-DE testset in following table and find that our method showed promising results.\n\n| | DE-EN | DE_EN | EN_DE | EN-DE |\n|---|---|---|---|---|\n|Test Set|Medline 18|Medline 19|Medline 18|Medline 19|\n|XLM|9.75|11.67|7.19|6.16|\n|Ours|10.55|13.87|7.79|6.95|\n\n**i) It could raise ethical or social issues related to the use of images for NLP.**\n\nIn our experimental endeavors, we have amassed a substantial collection of images, which naturally prompts discussions pertaining to these issues. \nIt is imperative to clarify that we do not possess the copyright for these images. Instead, they will be exclusively available to researchers and educators seeking to utilize the dataset for non-commercial research and/or educational purposes. \nWhen publishing the dataset, we will also strive to mitigate privacy, fairness and other concerns related to the images. For instance, we take measures to remove images containing recognizable faces to ensure privacy.\nWe sincerely appreciate you highlighting this matter, and we will ensure to incorporate this information into the paper or documentation accompanying the datasets.\n\n**j) It could encounter technical or practical challenges related to the scalability or efficiency of the proposed method.**\n\nThe challenges faced in these practical applications are indeed common issues in multi-modal machine translation tasks. However, our word-level images provide distinct advantages when compared to the approaches of searching or storing a suitable sentence-level image for each individual sentence. In practice, sentence-level images should be typically obtained at the dataset level, ranging from tens of thousands to millions. However, word-level images usually require a vocabulary-level size, typically in the range of thousands. Our smaller scale results in reduced costs in terms of collection, storage, and processing requirements.\n\n**k) It could face theoretical or conceptual limitations related to the use of word-level images for NLP.**\n\nWhile word-level images may encounter limitations in conceptual coverage, we possess distinct advantages over sentence-level images. We provide greater flexibility and are effective in addressing a significant portion of word confusion challenges. Looking ahead, our future plans will involve some integrating concepts at the conceptual level by combining multiple words.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The strengths of this paper are:\n+ It addresses a challenging and important problem in UNMT, which is lexical confusion, and proposes a novel and simple solution that uses word-level images as an implicit bilingual resource.\n+ It provides a comprehensive and rigorous experimental setup, including multiple datasets, language pairs, baselines, and evaluation metrics, and shows consistent and significant improvements over text-only and multimodal baselines.\n+ It introduces a large-scale word-image dataset that covers five languages and involves over 300,000 images, which can be useful for various multimodal NLP tasks and applications.\n\nThe main benefits to the NLP community if this paper were to be presented at the conference or accepted into Findings are:\n+ It would inspire more research on using word-level images for NMT and other NLP tasks, such as cross-lingual image captioning, image retrieval, or visual question answering.\n+ It would provide a new perspective on how to leverage visual information for NLP, especially for low-resource or distant languages, where bilingual dictionaries or parallel corpora are scarce or unavailable.\n+ It would offer a valuable dataset that can facilitate multimodal NLP research and development, and potentially lead to new tasks or challenges that combine vision and language.", "reasons_to_reject": "The weaknesses of this paper are:\n+ It does not provide a thorough analysis of the impact of different types or numbers of images on the translation quality, such as how to select the most relevant or diverse images for a given word, or how to balance the trade-off between image quantity and quality.\n+ It does not compare its method with other methods that use word-level images for NMT or other NLP tasks, such as Wang et al. (2019) or Huang et al. (2021), which could provide more insights into the advantages and limitations of different approaches.\n+ It does not conduct experiments on more challenging language pairs or domains, such as languages with different scripts or writing systems, or domains with more abstract or specialized vocabulary, which could test the robustness and generalizability of the proposed method.\n\nThe main risks of having this paper presented at the conference (other than lack of space to present better papers) or accepted into Findings are:\n+ It could raise ethical or social issues related to the use of images for NLP, such as privacy, consent, bias, or fairness, especially when dealing with sensitive or personal data, such as names, faces, locations, or identities.\n+ It could encounter technical or practical challenges related to the scalability or efficiency of the proposed method, such as how to collect, store, process, or access large amounts of word-level images for different languages or domains, or how to handle noisy or irrelevant images that may degrade the performance.\n+ It could face theoretical or conceptual limitations related to the use of word-level images for NLP, such as how to account for the ambiguity, variability, or context-dependency of visual information, or how to integrate other modalities or sources of information that may complement or contradict the images.", "questions_to_authors": "a) How do you select the nouns for the word-image lookup table? and How do you ensure the coverage and quality of the word-image pairs?\n\nb) How do you handle noisy or irrelevant images that may be crawled from Google Images?\n\nc) How do you handle words that have multiple meanings or senses, and may correspond to different images depending on the context? Do you use any word sense disambiguation technique or select images dynamically based on the sentence?\n\nd) How do you deal with named entities? Do you use any placeholder or fallback mechanism? How does this affect the performance of UNMT?\n\ne) How do you evaluate the impact of word-level images on the latent space of UNMT? Do you use any visualization or analysis tool to show how the images affect the word representations or alignments? Do you have any examples or cases that illustrate this impact?", "paper_topic_and_main_contributions": "This paper is about using word-level images to enhance unsupervised neural machine translation and reduce lexical confusion.\nThe problem this paper addresses is how to improve the performance and accuracy of UNMT, especially for words that are easily confused due to similar distributions or meanings in different languages. The main contributions that it makes towards a solution are:\n+ A method that leverages the visual information of word-level images to augment the lexical mappings between languages, and a modified embedding layer and a mask transformer to fuse image and text information in the model.\n+ An extensive evaluation that demonstrates the effectiveness of the proposed method in mitigating lexical confusion and achieving comparable or better results than previous methods that use sentence-level images or bilingual dictionaries.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ic5nKlTwFT", "reviewer": "Reviewer_jZHw", "summary": "", "questions": "While the authors have justified that mapping layers would bring the embeddings space together, I feel alignment might still not be fully present between XLM text embeddings and CLIP embeddings. I am curious to know what makes authors think that adding a single mapping layer and using it during training would be able to align these embeddings properly.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and the endorsement of our research. Here is our response to your inquiries:\n\n**a) Hugely reliant on search engine results.**\n1) The images we retrieve are based on nouns, which results in relatively fewer biases compared to other complex search. \n2) To mitigate this issue, we utilize a clustering algorithm during the data collection process, enabling the implementation of an automatic filtering method to remove low-quality images. While this approach may seem straightforward, our results shows that appears to be sufficient. \n3) In cases where a higher quality dataset is desired, manual selection can be employed, albeit at a higher cost. However, it is important to note that manual image selection is still less expensive compared to annotating bilingual dictionaries, as it requires proficiency in only one language, whereas the latter necessitates proficiency in two languages.\n\n**b) \"Blue\" in the image.**\n\n\"Blue\" is indeed an adjective in this image. However, in our image collection process, \"blue\" can also appear as a noun in other training samples, which is why it is included in our word-image lookup table. When inserting images in our fusion method, we do not differentiate between whether a word is a noun or not. We appreciate your observation and will ensure to include this clarification in the paper.\n\n**c) The selection of what images to take and how many images to take.**\n\nIn section 4.4, we conduct validation experiments and find that randomly selecting two images on the Multi30k dataset yields the best results. While these methods may introduce slight performance differences, it is important to emphasize that all of these approaches contribute to addressing the issue of word confusion to some extent.\n\n**d) The results on the WMT dataset did not show significant improvements.**\n\nThe motivation behind our method stems from the need to address word confusion in UNMT, which tends to be more prevalent in nouns. Consequently, when the occurrence of proper nouns is limited or when the dataset is abundant, the issue of word confusion becomes less prominent, and the effectiveness of our method may not be significant.\n\n**e) $\\Delta$ represents the improvement of our method over the XLM not the best.**\n\nOur method builds upon XLM as our fundamental baseline, incorporating modifications that directly address word confusion without relying on expensive bilingual dictionaries. However, we acknowledge that the initial presentation may have been slightly misleading. We appreciate you raising this question, and we will ensure to include an additional comparison with the best results in future versions to provide a more comprehensive and accurate assessment.\n\n**f) Single mapping layer works.**\n\nIn fact, previous studies such as FROMAGe *[1]*,  LLaVAR *[2]*, and VPGTrans *[3]* have successfully employed a straightforward fully connected layer to align textual and visual spaces. These works have provided compelling evidence that a simple fully connected layer is highly effective in addressing the task at hand.\n\n**Reference**\n\n*[1] Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal inputs and ouputs,ICML,2023*\n\n\n*[2] Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. arXiv e-prints, pages arXiv–2306, 2023*\n\n*[3] Ao Zhang, Hao Fei, Yuan Yao, Wei Ji, Li Li, Zhiyuan Liu, and Tat-Seng Chua. Transfer visual prompt generator across llms,2023*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is extremely well structured and conveys its ideas effectively. The experimental design has been given a thoughtful consideration for instance the use of masked transformer and the use of mapping layer before using it as token embedding to bring text and image embeddings to the same embeddings space. The proposed approach does show experimental gains that translate into increased BLEU score in comparison to the baseline. Overall, if selected some of the ideas might be borrowed from this paper which could help advance the field.", "reasons_to_reject": "While the paper tries an interesting approach, it does have certain weaknesses based on my understanding and would love to hear back from the authors on these pointers:\n(1) The paper is hugely reliant on search engine results. If the search engine results are biased, this would eventually bring that bias into their model which might raise some ethical concerns if they don't curate the images properly and if the curation is done properly the paper loses its edge on scalability over dictionary-based approaches. Also, in the image I feel \"blue\" is an \"adjective\" and not a noun which also raises the point the approach also depends on efficacy of the tools like \"spacy\" and \"core nlp\" and might need some addition of context and issues like polysemy.\n(2) The selection of what images to take and how many images to take is also not fully justified and reasoning why 2 images work best did not seem satisfactory. Moreover, the improvements were not observed in WMT dataset (which authors justify saying lower frequency of nouns, my counter is in real life you would have many cases where nouns are not present) which raises some questions over the approach and need for more evidence. Also, in table 3, XLM was used as baseline to show delta imrovement in BLEU score in table 2 which to me felt like slightly misleading in first go. Shouldn't the deltas be compared from the best model in column.", "questions_to_authors": "While the authors have justified that mapping layers would bring the embeddings space together, I feel alignment might still not be fully present between XLM text embeddings and CLIP embeddings. I am curious to know what makes authors think that adding a single mapping layer and using it during training would be able to align these embeddings properly.", "paper_topic_and_main_contributions": "The paper is centered around using multimodality (image signal) for improving Unsupervised machine translation. It is based on the fact that the images would have same meaning irrespective of the language and could be useful in improving UNMT. Instead of a dictionary they leverage search engines to extract images that can be used for training with denoising autoencoder loss and backtranslation based loss. The paper's main contribution lies in the fact that they design the framework to integrate these image signals very effectively into the training pipeline.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "NA", "Typos_Grammar_Style_and_Presentation_Improvements": "Maybe \"Blue\" example in figure is debatable if this can be treated as noun or an adjective"}, "extra_text": {}}, {"review_id": "Mnx5PUbTXO", "reviewer": "Reviewer_Latj", "summary": "", "questions": "1. What will happen if we replace the corresponding images with random images?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and support of this work. We appreciate the opportunity to address the concerns you have raised:\n\n**a) What will happen if we replace the corresponding images with random images?**\n\nThank you for your suggestion. We will add another ablation study to this paper. As shown in Table, we have observed a significant decline when using random images during the multi30k training stage. We will include the complete results in our paper at a later stage.\n\n| | FR-EN | FR_EN | DE-EN | DE_EN |\n|---|---|---|---|---|\n|Test Sets|F16|F17|F16|F17|\n|Corresponding Image| 49.10 | 43.01 | 40.32 | 35.20 |\n|Random Image | 48.11 | 41.69 | 39.02 | 34.38 |\n\n\n**b) Unfair comparison to baselines.**\n\n1) All the baselines uses pre-trained models including PVP which they *[1]* introduce in section 4.3 in their paper. \n2) We employ self-collected images due to the lack of readily available word-level image datasets. Existing datasets such as Multi30k and COCO primarily consist of sentence-level images, making them unsuitable for our specific needs. This utilization of self-collected word-level images is one of the primary contributions of our paper. By comparing our method with approaches that utilize sentence-level images, we aim to highlight the unique flexibility and advantages offered by word-level images.\n\n\n**c) Random are better than best method in Section 4.4.**\n\n\nThe \"Random method\" refers to the process of selecting one corresponding image from multiple images associated with a word, in which each image is chosen randomly for each training sample, allowing the model to have exposure to all the images during training including the best image. However, the \"best method\", which involves fixing the best image, may limit the model's perspective. Therefore, it is expected that the best method would yield slightly lower results compared to the random method.\n\n**d) Use XLM not MASS as the text-only baseline.**\n\nUNMT encompasses two distinct training stages: pre-training and NMT training. In our approach, we specifically focus on the NMT training stage, leveraging word-level images. It is important to note that our contribution is independent and complementary to the MASS approach, which primarily emphasizes the pre-training stage. Although our method can be applied to MASS, for this particular study, our primary focus is on enhancing the more classical and widely utilized XLM model, which serves as our text-only baseline.\n\n**Reference**\n\n*[1] Po-Yao Huang, Junjie Hu, Xiaojun Chang, and Alexander Hauptmann. Unsupervised multimodal neural machine translation with pseudo visual pivoting. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8226–8237, Online, July 2020. Association for Computational Linguistics.*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The authors propose a novel multi-modal UNMT method based on word level instead of sentence level.", "reasons_to_reject": "- Unfair comparison to baselines: The authors employ existing high-quality pre-trained cross-lingual models, while the baselines, like PVP, train their model without using pre-trained models. Moreover, the images used in this paper are self-collected, whereas other baselines use images from Multi30K and COCO datasets.\n\n- The absence of an ablation study to demonstrate the role of word-level images is noted. Additionally, in Section 4.4, the results of the random pick are higher than the best pick. This raises a question: what will happen if we replace the corresponding images with random images?\n\n- The authors use XLM as the text-only baseline, however, the MASS has better performance than XLM in UNMT.", "questions_to_authors": "1. What will happen if we replace the corresponding images with random images?", "paper_topic_and_main_contributions": "This paper introduces a novel multi-modal unsupervised neural machine translation method based on word-level images. The authors propose an image fusion technique in the embedding layer of the Transformer and collect a word-image dataset containing 300K images for conducting experiments. To enhance the integration of images with words in the original sentences, the authors further propose a visible matrix to shield the relationship between images and words.\n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes unsupervised NMT improvement by grounding words in multiple languages to images, so that visual words representations are mixed with text embeddings. The authors show +2.81 BLEU improvement, which is comparable to using a bilingual dictionary.

Pros: The paper presents a simple and original method. The scale of the experiment and the data released to the community is valuable (300k images with captions, 5 languages).  Results show consistent improvements in translation quality. 
Cons:  The experimentation and analysis of the approach are somehow limited and preliminary. There are many open questions that would need to be addressed, as pointed out by the reviewers.  Also, experimental results depend on the images retrieved by a specific search engine. While this is fine for the case of nouns covered by the paper, it is not clear how this solution scales to other words or terms. Finally, there are ethical concerns about the data set that needs to be addressed before its release.
Decision: Accept-Findings