Similarly, INLINEFORM0 is the probability of generating INLINEFORM1 in the target language, which shares INLINEFORM2 with the source language. By sharing the same parameters across different languages in the encoder and decoder, both the visual features and the learned word embeddings for different languages are enforced to project in a joint semantic space. To be noted, the proposed multi-lingual parameter sharing strategy is not constrained to the presented image captioning model, but can be applied in various image captioning models such as show-tell model  and so on. We use maximum likelihood as objective function to train the multi-lingual caption model, which maximizes the log-probability of the ground-truth captions: DISPLAYFORM0

Evaluation of Multi-lingual Image Caption	We first evaluate the captioning performance of the proposed multi-lingual caption model, which serves as the foundation stone for our bilingual lexicon induction method. We compare the proposed multi-lingual caption model with the mono-lingual model, which consists of the same model structure, but is trained separately for each language. Table TABREF22 presents the captioning results on the multi30k dataset, where all the languages are from the Latin family. The multi-lingual caption model achieves comparable performance with mono-lingual model for data sufficient languages such as English and German, and significantly outperforms the mono-lingual model for the data-scarce language French with absolute 3.22 gains on the CIDEr metric. For languages with distinctive grammar structures such as English and Japanese, the multi-lingual model is also on par with the mono-lingual model as shown in Table TABREF29 . To be noted, the multi-lingual model contains about twice less of parameters than the independent mono-lingual models, which is more computation efficient. We visualize the learned visual groundings from the multi-lingual caption model in Figure FIGREF32 .

Therefore, INLINEFORM6 and INLINEFORM7 not only encode the linguistic information of different languages but also share the embedding space which enables direct cross-lingual similarity comparison. We refer the linguistic features of source and target words INLINEFORM8 and INLINEFORM9 as INLINEFORM10 and INLINEFORM11 respectively. For the visual similarity, the multi-lingual caption model locates the image region to generate each word base on the spatial attention in Eq ( EQREF13 ), which can be used to calculate the localized visual representation of the word. However, since the attention is computed before word generation, the localization performance can be less accurate. It also cannot be generalized to image captioning models without spatial attention. Therefore, inspired by  , where they occlude over regions of the image to observe the change of classification probabilities, we feed different parts of the image to the caption model and investigate the probability changes for each word in the sentence. Algorithm SECREF16 presents the procedure of word localization and the grounded visual feature generation. Please note that such visual-grounding is learned unsupervisedly from the image caption data.

Multi-lingual Image Caption Model	Suppose we have mono-lingual image caption datasets INLINEFORM0 in the source language and INLINEFORM1 in the target language. The images INLINEFORM2 in INLINEFORM3 and INLINEFORM4 do not necessarily overlap, but cover overlapped object or scene classes which is the basic assumption of vision-based methods. For notation simplicity, we omit the superscript INLINEFORM5 for the data sample. Each image caption INLINEFORM6 and INLINEFORM7 is composed of word sequences INLINEFORM8 and INLINEFORM9 respectively, where INLINEFORM10 is the sentence length. The proposed multi-lingual image caption model aims to generate sentences in different languages to describe the image content, which connects the vision and multi-lingual sentences. Figure FIGREF15 illustrates the framework of the caption model, which consists of three parts: the image encoder, word embedding module and language decoder. The image encoder encodes the image into the shared visual space. We apply the Resnet152  as our encoder INLINEFORM0 , which produces INLINEFORM1 vectors corresponding to different spatial locations in the image: DISPLAYFORM0  where INLINEFORM0 .

Experimental Setup	For the multi-lingual caption model, we set the word embedding size and the hidden size of LSTM as 512. Adam algorithm is applied to optimize the model with learning rate of 0.0001 and batch size of 128. The caption model is trained up to 100 epochs and the best model is selected according to caption performance on the validation set. We compare our approach with two baseline vision-based methods proposed in  ,  , which measure the similarity of two sets of global visual features for bilingual lexicon induction: CNN-mean: taking the similarity score of the averaged feature of the two image sets. CNN-avgmax: taking the average of the maximum similarity scores of two image sets. We evaluate the word translation performance using MRR (mean-reciprocal rank) as follows: DISPLAYFORM0  where INLINEFORM0 is the groundtruth translated words for source word INLINEFORM1 , and INLINEFORM2 denotes the rank of groundtruth word INLINEFORM3 in the rank list of translation candidates. We also measure the precision at K (P@K) score, which is the proportion of source words whose groundtruth translations rank in the top K words. We set K as 1, 5, 10 and 20.

Visual-guided Word Representation	The proposed multi-lingual caption model can induce similarities of words in different languages from two aspects: the linguistic similarity and the visual similarity. In the following, we discuss the two types of similarity and then construct the source and target word representations. The linguistic similarity is reflected from the learned word embeddings INLINEFORM0 and INLINEFORM1 in the multi-lingual caption model. As shown in previous works  , word embeddings learned from the language contexts can capture syntactic and semantic regularities in the language. However, if the word embeddings of different languages are trained independently, they are not in the same linguistic space and we cannot compute similarities directly. In our multi-lingual caption model, since images in INLINEFORM2 and INLINEFORM3 share the same visual space, the features of sentence INLINEFORM4 and INLINEFORM5 belonging to similar images are bound to be close in the same space with the visual constraints. Meanwhile, the language decoder is also shared, which enforces the word embeddings across languages into the same semantic space in order to generate similar sentence features.

The recent WMT workshop  has proposed a subtask of multi-lingual caption generation, where different strategies such as multi-task captioning and source-to-target translation followed by captioning have been proposed to generate captions in target languages. Our work proposes a multi-lingual image caption model that shares part of the parameters across different languages in order to benefit each other.

We propose a multi-lingual image caption model trained on multiple mono-lingual image caption data, which is able to induce two types of word representations for different languages in the joint space. The first is the linguistic feature learned from the sentence context with visual semantic constraints, so that it is able to generate more accurate translations for words that are less visual-relevant. The second is the localized visual feature which attends to the local region of the object or scene in the image for the corresponding word, so that the visual representation of words will be more salient than previous global visual features. The two representations are complementary and can be combined to induce better bilingual word translation. We carry out experiments on multiple language pairs including German-English, French-English, and Japanese-English. The experimental results show that the proposed multi-lingual caption model not only achieves better caption performance than independent mono-lingual models for data-scarce languages, but also can induce the two types of features, linguistic and visual features, for different languages in joint spaces. Our proposed method consistently outperforms previous state-of-the-art vision-based bilingual word induction approaches on different languages. The contributions of this paper are as follows:

The vision-based methods exploit images to connect different languages, which assume that words corresponding to similar images are semantically alike.  collects images with labeled words in different languages to learn word translation with image as pivot.  improves the visual-based word translation performance via using more powerful visual representations: the CNN-based  features. The above works mainly focus on the translation of nouns and are limited in the number of collected languages. The recent work  constructs the current largest (with respect to the number of language pairs and types of part-of-speech) multimodal word translation dataset, MMID. They show that concrete words are easiest for vision-based translation methods while others are much less accurate. In our work, we alleviate the limitations of previous vision-based methods via exploring images and their captions rather than images with unstructured tags to connect different languages. Image captioning has received more and more research attentions. Most image caption works focus on the English caption generation  ,  , while there are limited works considering generating multi-lingual captions.

Datasets	For image captioning, we utilize the multi30k  , COCO  and STAIR  datasets. The multi30k dataset contains 30k images and annotations under two tasks. In task 1, each image is annotated with one English description which is then translated into German and French. In task 2, the image is independently annotated with 5 descriptions in English and German respectively. For German and English languages, we utilize annotations in task 2. For the French language, we can only employ French descriptions in task 1, so the training size for French is less than the other two languages. The COCO and STAIR datasets contain the same image set but are independently annotated in English and Japanese. Since the images in the wild for different languages might not overlap, we randomly split the image set into two disjoint parts of equal size. The images in each part only contain the mono-lingual captions. We use Moses SMT Toolkit to tokenize sentences and select words occurring more than five times in our vocabulary for each language. Table TABREF21 summarizes the statistics of caption datasets. For bilingual lexicon induction, we use two visual datasets: BERGSMA and MMID.

Unsupervised Bilingual Lexicon Induction from Mono-lingual Multimodal Data	Bilingual lexicon induction, translating words from the source language to the target language, is a long-standing natural language processing task. Recent endeavors prove that it is promising to employ images as pivot to learn the lexicon induction without reliance on parallel corpora. However, these vision-based approaches simply associate words with entire images, which are constrained to translate concrete words and require object-centered images. We humans can understand words better when they are within a sentence with context. Therefore, in this paper, we propose to utilize images and their associated captions to address the limitations of previous approaches. We propose a multi-lingual caption model trained with different mono-lingual multimodal data to map words in different languages into joint spaces. Two types of word representation are induced from the multi-lingual caption model: linguistic features and localized visual features. The linguistic feature is learned from the sentence contexts with visual semantic constraints, which is beneficial to learn translation for words that are less visual-relevant. The localized visual feature is attended to the region in the image that correlates to the word, so that it alleviates the image restriction for salient visual representation.

Unsupervised Bilingual Lexicon Induction	Our goal is to induce bilingual lexicon without supervision of parallel sentences or seed word pairs, purely based on the mono-lingual image caption data. In the following, we introduce the multi-lingual image caption model whose objectives for bilingual lexicon induction are two folds: 1) explicitly build multi-lingual word embeddings in the joint linguistic space; 2) implicitly extract the localized visual features for each word in the shared visual space. The former encodes linguistic information of words while the latter encodes the visual-grounded information, which are complementary for bilingual lexicon induction.

Conclusion	In this paper, we address the problem of bilingual lexicon induction without reliance on parallel corpora. Based on the experience that we humans can understand words better when they are within the context and can learn word translations with external world (e.g. images) as pivot, we propose a new vision-based approach to induce bilingual lexicon with images and their associated sentences. We build a multi-lingual caption model from multiple mono-lingual multimodal data to map words in different languages into joint spaces. Two types of word representation, linguistic features and localized visual features, are induced from the caption model. The two types of features are complementary for word translation. Experimental results on multiple language pairs demonstrate the effectiveness of our proposed method, which leads to significant performance improvement over the state-of-the-art vision-based approaches for all types of part-of-speech. In the future, we will further expand the vision-pivot approaches for zero-resource machine translation without parallel sentences.

Generalization to Diverse Language Pairs	Beside German-to-English word translation, we expand our approach to other languages including French and Japanese which is more distant from English. The French-to-English word translation performance is presented in Table TABREF39 . To be noted, the training data of the French captions is five times less than German captions, which makes French-to-English word translation performance less competitive with German-to-English. But similarly, the fusion of linguistic and visual features achieves the best performance, which has boosted the baseline methods with 4.2% relative gains on the MRR metric and 17.4% relative improvements on the P@20 metric. Table TABREF40 shows the Japanese-to-English word translation performance. Since the language structures of Japanese and English are quite different, the linguistic features learned from the multi-lingual caption model are less effective but still can benefit the visual features to improve the translation quality. The results on multiple diverse language pairs further demonstrate the generalization of our approach for different languages.

The two types of features are complementary for word translation. Experimental results on multiple language pairs demonstrate the effectiveness of our proposed method, which substantially outperforms previous vision-based approaches without using any parallel sentences or supervision of seed word pairs.

Evaluation of Bilingual Lexicon Induction	We induce the linguistic features and localized visual features from the multi-lingual caption model for word translation from the source to target languages. Table TABREF30 presents the German-to-English word translation performance of the proposed features. In the BERGSMA dataset, the visual features achieve better translation results than the linguistic features while they are inferior to the linguistic features in the MMID dataset. This is because the vocabulary in BERGSMA dataset mainly consists of nouns, but the parts-of-speech is more diverse in the MMID dataset. The visual features contribute most to translate concrete noun words, while the linguistic features are beneficial to other abstract words. The fusion of the two features performs best for word translation, which demonstrates that the two features are complementary with each other. We also compare our approach with previous state-of-the-art vision-based methods in Table TABREF30 . Since our visual feature is the averaged representation, it is fair to compare with the CNN-mean baseline method where the only difference lies in the feature rather than similarity measurement. The localized features perform substantially better than the global image features which demonstrate the effectiveness of the attention learned from the caption model.

The visual-relevant concrete words are easier to translate such as `phone' and `red'. But our approach still generates reasonable results for abstract words such as `area' and functional words such as `for' due to the fusion of visual and sentence contexts. We also evaluate the influence of different image captioning structures on the bilingual lexicon induction. We compare our attention model (`attn') with the vanilla show-tell model (`mp')  , which applies mean pooling over spatial image features to generate captions and achieves inferior caption performance to the attention model. Table TABREF35 shows the word translation performance of the two caption models. The attention model with better caption performance also induces better linguistic and localized visual features for bilingual lexicon induction. Nevertheless, the show-tell model still outperforms the previous vision-based methods in Table TABREF30 .

Introduction	The bilingual lexicon induction task aims to automatically build word translation dictionaries across different languages, which is beneficial for various natural language processing tasks such as cross-lingual information retrieval  , multi-lingual sentiment analysis  , machine translation  and so on. Although building bilingual lexicon has achieved success with parallel sentences in resource-rich languages  , the parallel data is insufficient or even unavailable especially for resource-scarce languages and it is expensive to collect. On the contrary, there are abundant multimodal mono-lingual data on the Internet, such as images and their associated tags and descriptions, which motivates researchers to induce bilingual lexicon from these non-parallel data without supervision. There are mainly two types of mono-lingual approaches to build bilingual dictionaries in recent works. The first is purely text-based, which explores the structure similarity between different linguistic space. The most popular approach among them is to linearly map source word embedding into the target word embedding space  ,  . The second type utilizes vision as bridge to connect different languages  ,  ,  . It assumes that words correlating to similar images should share similar semantic meanings.

So previous vision-based methods search images with multi-lingual words and translate words according to similarities of visual features extracted from the corresponding images. It has been proved that the visual-grounded word representation improves the semantic quality of the words  . However, previous vision-based methods suffer from two limitations for bilingual lexicon induction. Firstly, the accurate translation performance is confined to concrete visual-relevant words such as nouns and adjectives as shown in Figure SECREF2 . For words without high-quality visual groundings, previous methods would generate poor translations  . Secondly, previous works extract visual features from the whole image to represent words and thus require object-centered images in order to obtain reliable visual groundings. However, common images usually contain multiple objects or scenes, and the word might only be grounded to part of the image, therefore the global visual features will be quite noisy to represent the word. In this paper, we address the two limitations via learning from mono-lingual multimodal data with both sentence and visual context (e.g., image and caption data) to induce bilingual lexicon. Such multimodal data is also easily obtained for different languages on the Internet  .

Related Work	The early works for bilingual lexicon induction require parallel data in different languages.  systematically investigates various word alignment methods with parallel texts to induce bilingual lexicon. However, the parallel data is scarce or even unavailable for low-resource languages. Therefore, methods with less dependency on the availability of parallel corpora are highly desired. There are mainly two types of mono-lingual approaches for bilingual lexicon induction: text-based and vision-based methods. The text-based methods purely exploit the linguistic information to translate words. The initiative works  ,  utilize word co-occurrences in different languages as clue for word alignment. With the improvement in word representation based on deep learning,  finds the structure similarity of the deep-learned word embeddings in different languages, and employs a parallel vocabulary to learn a linear mapping from the source to target word embeddings.  improves the translation performance via adding an orthogonality constraint to the mapping.  further introduces a matching mechanism to induce bilingual lexicon with fewer seeds. However, these models require seed lexicon as the start-point to train the bilingual mapping. Recently,  proposes an adversarial learning approach to learn the joint bilingual embedding space without any seed lexicon.

The BERGSMA dataset  consists of 500 German-English word translation pairs. Each word is associated with no more than 20 images. The words in BERGSMA dataset are all nouns. The MMID dataset  covers a larger variety of words and languages, including 9,808 German-English pairs and 9,887 French-English pairs. The source word can be mapped to multiple target words in their dictionary. Each word is associated with no more than 100 retrieved images. Since both these image datasets do not contain Japanese language, we download the Japanese-to-English dictionary online. We select words in each dataset that overlap with our caption vocabulary, which results in 230 German-English pairs in BERGSMA dataset, 1,311 German-English pairs and 1,217 French-English pairs in MMID dataset, and 2,408 Japanese-English pairs.

The parameter INLINEFORM1 of the encoder is shared for different languages in order to encode all the images in the same visual space. The word embedding module maps the one-hot word representation in each language into low-dimensional distributional embeddings: DISPLAYFORM0  where INLINEFORM0 and INLINEFORM1 is the word embedding matrix for the source and target languages respectively. INLINEFORM2 and INLINEFORM3 are the vocabulary size of the two languages. The decoder then generates word step by step conditioning on the encoded image feature and previous generated words. The probability of generating INLINEFORM0 in the source language is as follows: DISPLAYFORM0  where INLINEFORM0 is the hidden state of the decoder at step INLINEFORM1 , which is functioned by LSTM  : DISPLAYFORM0  The INLINEFORM0 is the dynamically located contextual image feature to generate word INLINEFORM1 via attention mechanism, which is the weighted sum of INLINEFORM2 computed by DISPLAYFORM0 DISPLAYFORM1  where INLINEFORM0 is a fully connected neural network. The parameter INLINEFORM1 in the decoder includes all the weights in the LSTM and the attention network INLINEFORM2 .

Word Translation Prediction	Since the word representations of the source and target language are in the same space, we could directly compute the similarities across languages. We apply l2-normalization on the word representations and measure with the cosine similarity. For linguistic features, the similarity is measured as: DISPLAYFORM0  However, there are a set of visual features associated with one word, so the visual similarity measurement between two words is required to take two sets of visual features as input. We aggregate the visual features in a single representation and then compute cosine similarity instead of point-wise similarities among two sets: DISPLAYFORM0  The reasons for performing aggregation are two folds. Firstly, the number of visual features is proportional to the word occurrence in our approach instead of fixed numbers as in  ,  . So the computation cost for frequent words are much higher. Secondly, the aggregation helps to reduce noise, which is especially important for abstract words. The abstract words such as `event' are more visually diverse, but the overall styles of multiple images can reflect its visual semantics. Due to the complementary characteristics of the two features, we combine them to predict the word translation. The translated word for INLINEFORM0 is DISPLAYFORM0

The combination of visual and linguistic features also significantly improves the state-of-the-art visual-based CNN-avgmax method with 11.6% and 6.7% absolute gains on P@1 on the BERGSMA and MMID dataset respectively. In Figure FIGREF36 , we present the word translation performance for different POS (part-of-speech) labels. We assign the POS label for words in different languages according to their translations in English. We can see that the previous state-of-the-art vision-based approach contributes mostly to noun words which are most visual-relevant, while generates poor translations for other part-of-speech words. Our approach, however, substantially improves the translation performance for all part-of-speech classes. For concrete words such as nouns and adjectives, the localized visual features produce better representation than previous global visual features; and for other part-of-speech words, the linguistic features, which are learned with sentence context, are effective to complement the visual features. The fusion of the linguistic and localized visual features in our approach leads to significant performance improvement over the state-of-the-art baseline method for all types of POS classes. Some correct and incorrect translation examples for different POS classes are shown in Table TABREF34 .

Acknowledgments	This work was supported by National Natural Science Foundation of China under Grant No. 61772535, National Key Research and Development Plan under Grant No. 2016YFB1001202 and Research Foundation of Beijing Municipal Science & Technology Commission under Grant No. Z181100008918002.

Though there is certain mistakes such as `musicians' in the bottom image, most of the words are grounded well with correct objects or scenes, and thus can obtain more salient visual features.

Therefore, every word can be represented as a set of grounded visual features (the set size equals to the word occurrence number in the dataset). We refer the localized visual feature set for source word INLINEFORM0 as INLINEFORM1 , for target word INLINEFORM2 as INLINEFORM3 . Generating localized visual features. Encoded image features INLINEFORM0 , sentence INLINEFORM1 . Localized visual features for each word INLINEFORM2 each INLINEFORM3 compute INLINEFORM4 according to Eq ( EQREF10 ) INLINEFORM5 INLINEFORM6 INLINEFORM7

Question: Which languages are used in the multi-lingual caption model?
Answer: German-English, French-English, and Japanese-English