Neural Machine Translation with Universal Visual Representation

1 INTRODUCTION . Visual information has been introduced for neural machine translation in some previous studies ( NMT ) ( Specia et al. , 2016 ; Elliott et al. , 2017 ; Barrault et al. , 2018 ; Ive et al. , 2019 ) though the contribution of images is still an open question ( Elliott , 2018 ; Caglayan et al. , 2019 ) . Typically , each bilingual ( or multilingual ) parallel sentence pair is annotated manually by one image describing the content of this sentence pair . The bilingual parallel corpora with manual image annotations are used to train a multimodal NMT model by an end-to-end framework , and results are reported on a specific data set , Multi30K ( Calixto & Liu , 2017 ; Calixto et al. , 2017 ) . One strong point of the multimodal NMT model is the ability to use visual information to improve the quality of the target translation . However , the effectiveness heavily relies on the availability of bilingual parallel sentence pairs with manual image annotations , which hinders the image applicability to the NMT . As a result , the visual information is only applied to the translation task over a small and specific multimodal data set Multi30K ( Elliott et al. , 2016 ) , but not to large-scale text-only NMT ( Bahdanau et al. , 2014 ; Gehring et al. , 2017 ; Vaswani et al. , 2017 ) and low-resource ∗Corresponding author . Zhuosheng Zhang and Zuchao Li were internship research fellows at NICT when this work was done . Hai Zhao was partially supported by Key Projects of National Natural Science Foundation of China ( No . U1836222 and No . 61733011 ) . Rui Wang was partially supported by JSPS grant-in-aid for early-career scientists ( 19K20354 ) : “ Unsupervised Neural Machine Translation in Universal Scenarios ” and NICT tenure-track researcher startup fund “ Toward Intelligent Machine Translation. ” text-only NMT ( Fadaee et al. , 2017 ; Lample et al. , 2018 ; Ma et al. , 2019 ; Zhou et al. , 2019 ) . In addition , because of the high cost of annotation , the content of one bilingual parallel sentence pair is only represented by a single image , which is weak in capturing the diversity of visual information . The current situation of introducing visual information results in a bottleneck in the multimodal NMT and is not feasible for text-only NMT and low-resource NMT . In this paper , we present a universal visual representation ( VR ) method1 relying only on image-monolingual annotations instead of the existing approach that depends on image-bilingual annotations , thus breaking the bottleneck of using visual information in NMT . In detail , we transform the existing sentence-image pairs into a topic-image lookup table from a small-scale multimodal data set Multi30K . During the training and decoding process , a group of images with a similar topic to the source sentence will be retrieved from the topic-image lookup table learned by the term frequency-inverse document frequency , and thus is encoded as image representations by a pretrained ResNet ( He et al. , 2016 ) . A simple and effective attention layer is then designed to fuse the image representations and the original source sentence representations as input to the decoder for predicting target translations . In particular , the proposed approach can be easily integrated into the text-only NMT model without annotating large-scale bilingual parallel corpora . The proposed method was evaluated on four widely-used translation datasets , including the WMT ’ 16 Englishto-Romanian , WMT ’ 14 English-to-German , WMT ’ 14 English-to-French , and Multi30K which are standard corpora for NMT and multimodal machine translation ( MMT ) evaluation . Experiments and analyses show effectiveness . In summary , our contributions are primarily three-fold : 1 . We present a universal visual representation method that overcomes the shortcomings of the bilingual ( or multilingual ) parallel data with manual image annotations for MMT . 2 . The proposed method enables the text-only NMT to use the multimodality of visual information without annotating the existing large scale bilingual parallel data . 3 . Experiments on different scales of translation tasks verified the effectiveness and generality of the proposed approach . 2 RELATED WORK . Building fine-grained representation with extra knowledge is an essential topic in language modeling ( Li et al. , 2020a ; b ; Zhang et al. , 2020b ; a ) , among which adopting visual modality could potentially benefit the machine with a more comprehensive perception of the real world . Inspired by the studies on the image description generation ( IDG ) task ( Mao et al. , 2014 ; Elliott et al. , 2015 ; Venugopalan et al. , 2015 ; Xu et al. , 2015 ) , a new shared translation task for multimodal machine translation was addressed by the machine translation community ( Specia et al. , 2016 ) . In particular , the released dataset Multi30K ( Elliott et al. , 2016 ) includes 29,000 multilingual ( English , German , and French ) parallel sentence pairs with image annotations ( Elliott et al. , 2017 ; Barrault et al. , 2018 ) . Subsequently , there has been a rise in the number of studies ( Caglayan et al. , 2016 ; 2017 ; Calixto et al. , 2016 ; Huang et al. , 2016 ; Libovickỳ & Helcl , 2017 ; Helcl et al. , 2018 ) . For example , Calixto et al . ( 2017 ) proposed a doubly-attentive multimodal NMT model to incorporate spatial-visual features , improving translation performance . Compared with spatial-visual features , Calixto & Liu ( 2017 ) further incorporated global image features as words in the source sentence and to enhance the encoder or decoder hidden state . In contrast , some recent studies indicated that the visual modality is either unnecessary ( Zhang et al. , 2017 ) or only marginally beneficial ( Grönroos et al. , 2018 ) . More recently , Ive et al . ( 2019 ) showed that visual information is only needed in particular cases , such as for ambiguous words where the textual context is not sufficient . However , these approaches only center around a small and specific Multi30K data set to build a multimodal NMT model , which hinders image applicability to NMT . The reason would be the high cost of image annotations , resulting potentially in the image information not being adequately discovered . We believe that the capacity of MMT has not yet been excavated sufficiently , and there is still a long way to go before the potential of MMT is fully discovered . In this work , we seek to break this constraint and enable visual information to benefit NMT , especially text-only NMT . 1The code is publicly available at https : //github.com/cooelf/UVR-NMT . 3 UNIVERSAL VISUAL RETRIEVAL . Algorithm 1 Topic-image Lookup Table Conversion Algorithm Require : Input sentences , S = { X1 , X2 , . . . XI } and paired images E = { e1 , e2 , . . . , eI } Ensure : Topic-image lookup table Q where each word is associated with a group of images 1 : Obtain the TF-IDF dictionary F = TF-IDF ( S ) 2 : Transform sentence-image pair to topic-image lookup table Q = LookUp ( S , E , F ) 3 : procedure TF-IDF ( S ) 4 : for each sentence in S do 5 : Filter stop-words in the sentence 6 : Calculate the TF-IDF weight for each word 7 : end for 8 : return TF-IDF dictionary F 9 : end procedure 10 : procedure LOOKUP ( S , E , F ) 11 : for For each pair { Ti , ei } ∈ zip { S , E } do 12 : Rank and pick out the top-w “ topic ” words in the sentence according to the TF-IDF score in the dictionary F , and each sentence is reformed as T = { t1 , t2 , . . . , tw } 13 : Pair the w words with the corresponding image ei 14 : for For each word tj in T do 15 : if ei not in Q [ tj ] then 16 : Add ej to the corresponding image set Q [ tj ] for word tj 17 : end if 18 : end for 19 : end for 20 : return Topic-image lookup table Q 21 : end procedure In this section , we will introduce the proposed universal visual representation method . Generally , the default input setting of the MMT is a sentence-image pair . Our basic intuition is to transform the existing sentence-image pairs into topic-image lookup table2 , which assumes the topic words in a sentence should be relevant to the paired image . Consequently , a sentence can possess a group of images by retrieving the topic-image lookup table . Topic-image Lookup Table Conversion To focus on the major part of the sentence and suppress the noise such as stopwords and low-frequency words , we design a filtering method to extract the “ topic ” words of the sentence through the term frequency-inverse document frequency ( TF-IDF ) 3 inspired by Chen et al . ( 2019 ) . Specifically , given an original input sentence X = { x1 , x2 , . . . , xI } of length I and its paired image e , X is first filtered by a stopword list4 and then the sentence is treated as a document g. We then compute TF-IDF TIi , j for each word xi in g , TIi , j = oi , j∑ k ok , j × log |G| 1 + |j : xi ∈ g| , ( 1 ) where oi , j represents the number of occurrences of the word xi in the input sentence g , |G| the total number of source language sentences in the training data , and |j : xi ∈ g| the number of source sentences including word xi in the training data . We then select the top-w high TF-IDF words as the new image description T = { t1 , t2 , . . . , tw } for the input sentence X . After preprocessing , each filtered sentence T is paired with an image e , and each word ti ∈ T is regarded as the topic word for image e. After processing the whole corpus ( i.e. , Multi30K ) , we form a topic-image lookup tableQ as described in Algorithm 1 , in which each topic word ti would be paired with dozens of images . Image Retrieval For the input sentence , we first obtain its topic words according to the text preprocessing method described above . Then we retrieve the associated images for each topic word 2We use the training set of the Multi30K dataset to build the topic-image lookup table . 3We describe our methods by regarding the processing unit as word though this method can also be applied to a subword-based sentence for which the subword is considered to be the processing unit . 4https : //github.com/stopwords-iso/stopwords-en from the lookup table Q and group all the retrieved images together to form an image list G. We observe that an image might be associated with multiple topic words so that it would occur multiple times in the list G. Therefore , we sort the images according to the frequency of occurrences in G to maintain the total number of images for each sentence at m. Figure 1 illustrates the retrieval process5 . In the left block , we show six examples of sentence-image pairs in which the topic words are in boldface . Then we process the corpus using the topic-image transformation method demonstrated above and obtain the topic-image lookup table . For example , the word dog is associated with 1,512 images . For an input source sentence , we obtain the topic words ( in boldface ) using the same preprocessing . Then we retrieve the corresponding images from the lookup table for each topic word . Now we have a list of images , and some images appear multiple times as they have various topics ( like the boxed image in Figure 1 ) . So we sort the retrieved image list by the count of occurrence to pick out the top-m images that cover the most topics of the sentence . At test time , the process of getting images is done using the image lookup table built by the training set , so we do not need to use the images from the dev and test sets in Multi30K dataset6 . Intuitively , we do not strictly require the manual alignment of the word ( or concept ) and image , but rely on the co-occurrence of topic word and image , which is simpler and more general . In this way , we call our method as universal visual retrieval .

Summary: The authors propose to augment NMT with a grounded inventory of images.  The intuition is clear and the premise is very tempting.  The key architectural choice is to allow the transformer to use language embeddings to attend into a topic-image lookup table.  The proportion is learned to balance how much signal comes from each source.    Figure 4, attempts to investigate the importance of this sharing and its effects on performance.