SlowGuess/NAACL_2021_OCR huggingface dataset:
Quanyu Long $^{1}$ , Mingxuan Wang $^{2}$ , and Lei Li $^{2}$
$^{1}$ Nanyang Technological University, Singapore
2ByteDance AI Lab, China
quanyu001@e.ntu.edu.sg;
{wangmingxuan.89, lileilab}@bytedance.com
There are common semantics shared across text and images. Given a sentence in a source language, whether depicting the visual scene helps translation into a target language? Existing multimodal neural machine translation methods (MNMT) require triplets of bilingual sentence - image for training and tuples of source sentence - image for inference. In this paper, we propose ImagiT, a novel machine translation method via visual imagination. ImagiT first learns to generate visual representation from the source sentence, and then utilizes both source sentence and the "imagined representation" to produce a target translation. Unlike previous methods, it only needs the source sentence at the inference time. Experiments demonstrate that ImagiT benefits from visual imagination and significantly outperforms the text-only neural machine translation baselines. Further analysis reveals that the imagination process in ImagiT helps fill in missing information when performing the degradation strategy.
Visual foundation has been introduced in a novel multimodal Neural Machine Translation (MNMT) task (Specia et al., 2016; Elliott et al., 2017; Barrault et al., 2018), which uses bilingual (or multilingual) parallel corpora annotated by images describing sentences' contents (see Figure 1(a)). The superiority of MNMT lies in its ability to use visual information to improve the quality of translation, but its effectiveness largely depends on the availability of data sets, especially the quantity and quality of annotated images. In addition, because the cost of manual image annotation is relatively high, at this stage, MNMT is mostly applied on a small and specific dataset, Multi30K (Elliott et al., 2016), and is not suitable for large-scale text-only Neural Machine Translation (NMT) (Bahdanau et al.,
(a) Multimodal NMT
(b) ImagiT
Figure 1: The problem setup of our proposed ImagiT is different from existing multimodal NMT. A multimodal NMT model takes both text and paired image as the input, while ImagiT takes only sentence in the source language as the usual NMT task. ImagiT synthesizes an image and utilize the internal visual representation to assist translation.
2015; Vaswani et al., 2017). Such limitations hinder the applicability of visual information in NMT.
To address the bottlenecks mentioned above, Zhang et al. (2020) propose to build a lookup table from an image dataset and then using the search-based method to retrieve pictures that match the source language keywords. However, the lookup table is built from Multi30K, which leads to a relatively limited coverage of the pictures, and potentially introduces much irrelevant noise. It does not always find the exact image corresponding to the text, or the image may not even exist in the database. Elliott and Kádár (2017) present a multitask learning framework to ground visual representation to a shared space. Their architecture called "imagination" shares an encoder between a primary NMT task and an auxiliary task of ranking the vi
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5738-5748 June 6-11, 2021. ©2021 Association for Computational Linguistics
sual features for image retrieval. However, neither the image is explicitly generated, nor the visual feature is directly leveraged by the translation decoder, the model simply learns the visual grounded shared encoder. Based on other researchers' earlier exploration, we hypothesize that the potential of vision in conventional text-only NMT has not been fully discovered. Different with Elliott and Kádár (2017) implicit approach, we understand "imagination" to be more like "picturing", since it is similar to humans who can visually depict figures in the mind from an utterance. Our approach aims to explicitly imagine a "vague figure" (see Figure 1(b)) to guide the translation, since A picture is worth a thousand words, and imagining the picture of a sentence is the instinctive reaction of a human being who is learning bilingualism.
In this paper, we propose a novel end-to-end machine translation model that is embedded in visual semantics with generative imagination (ImagiT) (see Figure 1(b)). Given a source language sentence, ImagiT first encodes it and transforms the word representations into visual features through an attentive generator, which can effectively capture the semantics of both global and local levels, and the generated visual representations can be considered as semantic-equivalent reconstructions of sentences. A simple yet effective integration module is designed to aggregate the textual and visual modalities. In the final stage, the model learns to generate the target language sentence based on the joint features. To train the model in an end-to-end fashion, we apply a visual realism adversarial loss and a text-image pair-aware adversarial loss, as well as text-semantic reconstruction loss and target language translation loss based on cross-entropy.
In contrast with most prior MNMT work, our proposed ImagiT model does not require images as input during the inference time but can leverage visual information through imagination, making it an appealing method in low-resource scenario. Moreover, ImagiT is also flexible, accepting external parallel text data or non-parallel image captioning data. We evaluate our Imagination modal on the Multi30K dataset. The experiment results show that our proposed method significantly outperforms the text-only NMT baseline. The analysis demonstrates that imagination help the model complete the missing information in the sentence when we perform degradation masking, and we also see improvements in translation quality by pre-training
the model with an external non-parallel image captioning dataset.
To summarize, the paper has the following contributions:
1. We propose generative imagination, a new setup for machine translation assisted by synthesized visual representation, without annotated images as input;
2. We propose the ImagiT method, which shows advantages over the conventional MNMT model and gains significant improvements over the text-only NMT baseline;
3. We conduct experiments to verify and analyze how imagination helps the translation.
MNMT As a language shared by people worldwide, visual modality may help machines have a more comprehensive perception of the real world. Multimodal neural machine translation (MNMT) is a novel machine translation task proposed by the machine translation community, which aims to design multimodal translation frameworks using context from the additional visual modality (Specia et al., 2016). The shared task releases the dataset Multi30K (Elliott et al., 2016), which is an extended German version of Flickr30K (Young et al., 2014), then expanded to French and Czech (Elliott et al., 2017; Barrault et al., 2018). In the three versions of tasks, scholars have proposed many multimodal machine translation models and methods. Huang et al. (2016) encodes word sequences with regional visual objects, while Calixto and Liu (2017) study the effects of incorporating global visual features to initialize the encoder/decoder hidden states of RNN. Caglayan et al. (2017) models the image-text interaction by leveraging elementwise multiplication. Elliott and Kádár (2017) propose a multitask learning framework to ground visual representation to a shared space and learn with the auxiliary triplet alignment task. The common practice is to use convolutional neural networks to extract visual information and then using attention mechanisms to extract visual contexts (Caglayan et al., 2016; Calixto et al., 2016; Libovický and Helcl, 2017). Ive et al. (2019) propose a translate-and-refine approach using two-stage decoder. Calixto et al. (2019) put forward a latent variable model to capture the multimodal interactions between visual and textual features. Caglayan et al.
(2019) show that visual content is more critical when the textual content is limited or uncertain in MMT. Recently, Yao and Wan (2020) propose multimodal self-attention in Transformer to avoid encoding irrelevant information in images, and Yin et al. (2020) propose a graph-based multimodal fusion encoder to capture various relationships.
Text-to-image synthesis Traditional Text-to-image (T2I) synthesis mainly uses keywords to search for small image regions, and finally optimizes the entire layout (Zhu et al., 2007). After generative adversarial networks (GANs) (Goodfellow et al., 2014) were proposed, scholars have presented a variety of GAN-based T2I models. Reed et al. (2016) propose DC-GAN and design a direct and straightforward network and a training strategy for T2I generation. Zhang et al. (2017) propose stackGAN, which contains multiple cascaded generators and discriminators, and the higher stage generates better quality pictures. In previous work, scholars only considered global semantics. Xu et al. (2018) proposed AttnGAN to apply the attention mechanism to capture fine-grained word-level information. MirrorGAN (Qiao et al., 2019) employs a mirror structure, which reversely learns from the inverse task of T2I to further validate whether generated images are consistent with the input texts. The inverse task is also known as image captioning.
3 ImagiT model
As shown in Figure2, ImagiT embodies the encoder-decoder structure for end-to-end machine translation. Between the encoder and the decoder, there is an imagination step to generate semantic-equivalent visual representation. Technically, our model is composed of following modules: source text encoder, generative imagination network, image captioning, multimodal aggregation and decoder for translation. We will elaborate on each of them in the rest of this section.
3.1 Source text encoder
Vaswani et al. (2017) propose the state-of-art Transformer-based machine translation framework, which can be written as follows:
$$
\overline {{\mathbf {H}}} ^ {l} = L N \left(\operatorname {A t t} ^ {l} \left(\mathbf {Q} ^ {l - 1}, \mathbf {K} ^ {l - 1}, \mathbf {V} ^ {l - 1}\right) + \mathbf {H} ^ {l - 1}\right), \tag {1}
$$
$$
\mathbf {H} ^ {l} = L N \left(F F N ^ {l} \left(\overline {{\mathbf {H}}} ^ {l}\right) + \overline {{\mathbf {H}}} ^ {l}\right), \tag {2}
$$
Where  $Att^l$ ,  $LN$ , and  $FFN^l$  are the self-attention module, layer normalization, and the feed-forward network for the  $l$ -th identical layer respectively. The core of the Transformer is the multi-head self-attention, in each attention head, we have:
$$
z _ {i} = \sum_ {j = 1} ^ {n} \alpha_ {i j} \left(x _ {j} W ^ {V}\right), \tag {3}
$$
$$
\alpha_ {i j} = \operatorname {s o f t m a x} \left(\frac {\left(x _ {i} W ^ {Q}\right) \left(x _ {j} W ^ {K}\right) ^ {\top}}{\sqrt {d}}\right). \tag {4}
$$
$W^{V}, W^{Q}, W^{K}$  are layer-specific trainable parameter matrices. For the output of final stacked layer, we use  $w = \{w_{0}, w_{1}, \dots, w_{L-1}\}$ ,  $w \in \mathbb{R}^{d \times L}$  to represent the source word embedding,  $L$  is the length of the source sentence. Besides, we add a special token to each source language sentence to obtain the sentence representation  $s \in \mathbb{R}^{d}$ .
3.2 Generative imagination network
Generative Adversarial Network (Goodfellow et al., 2014) has been applied to synthesis images similar to ground truth (Zhang et al., 2017; Xu et al., 2018; Qiao et al., 2019). We follow the common practice of using the conditioning augmentation (Zhang et al., 2017) to enhance robustness to small perturbations along the conditioning text manifold and improve the diversity of generated samples.  $F^{ca}$  represents the conditioning augmentation function, and  $s^{ca}$  represents the enhanced sentence representation.
$$
s ^ {c a} = F ^ {c a} (s), \tag {5}
$$
$\{F_0,F_1\}$  are two visual feature converters, sharing similar architecture.  $F_{0}$  contains a fully connected layer and four deconvolution layers (Noh et al., 2015) to obtain image-sized feature vectors. Furthermore, we define  $\{f_0,f_1\}$  are the visual features after two transformations with different resolution. For detailed layer structure and block design, please refer to (Xu et al., 2018).
$$
f _ {0} = F _ {0} (z, s ^ {c a}), \tag {6}
$$
$$
f _ {1} = F _ {1} \left(f _ {0}, F ^ {\text {a t t n}} \left(f _ {0}, s ^ {\text {c a}}\right)\right), \tag {7}
$$
<sup>1</sup>Zhang et al. (2017) also mentions that the randomness in the Conditioning Augmentation is beneficial for modeling text to image semantic translation as the same sentence usually corresponds to objects with various poses and appearances.
Figure 2: Overview of the framework of the proposed ImagiT.  $F_{0}$  and  $F_{1}$  are text-to-image converters, sharing similar structures, comprising of perceptron, residual, and unsampling blocks.  $\mathrm{L} \times$  represents L identical layers. Noting that we only need to obtain the generated visual feature to guide the translation, for the whole pipeline, up-sampling this feature to image is redundant.
Where  $f_0 \in \mathbb{R}^{M_0 \times N_0}$ ,  $z$  is the noise vector, sampled from the standard normal distribution, and it will be concatenated with  $s^{ca}$ . Each column of  $f_i$  is a feature vector of a sub-region of the image, which can also be treat as a pseudo-token. To generate fine-grained details at different subregions of the image by paying attention to the relevant words in the source language, we use image vector in each sub-region to query word vectors by leveraging attention strategy.  $F^{attn}$  is an attentive function to obtain word-context feature, then we have:
$$
F ^ {a t t n} \left(f _ {0}, s ^ {c a}\right) = \sum_ {l = 0} ^ {L - 1} \left(U _ {0} w _ {l}\right) \left(\text {s o f t m a x} \left(f _ {0} ^ {T} \left(U _ {0} w _ {l}\right)\right)\right) ^ {\top}, \tag {8}
$$
Word feature  $w_{l}$  is firstly converted into the common semantic space of the visual feature,  $U_{0}$  is a perceptron layer. Then it will be multiplied with  $f_{0}$  to acquire the attention score.  $f_{1}$  is the output of the imagination network, capturing multiple levels (word level and sentence level) of semantic meaning.  $f_{1}$  is denoted as the blue block "generated visual feature" in Figure2. It will be utilized directly for target language generation, and it will also be passed to the discriminator for adversarial training. Note that for the whole pipeline, upsampling  $f_{1}$  to
an image is redundant.
Comparing to T2I synthesis works which use cascaded generators and disjoint discriminators(Zhang et al., 2017; Xu et al., 2018; Qiao et al., 2019), we only use one stage to reduce the model size and make our generated visual feature  $f_{1}$  focus more on text-image consistency, but not the realism and authenticity.
3.3 Image captioning
Image captioning (I2T) can be regarded as the inverse problem of text-to-image generation, generating the given image's description. If an imagined image is semantic equivalent to the source sentence, then its description should be almost identical to the given text. Thus we leverage the image captioning to translate the imagined visual representation back to the source language(Qiao et al., 2019), and this symmetric structure can make the imagined visual feature act like a mirror, effectively enhancing the semantic consistency of the imagined visual feature and precisely reflect the underlying semantics. Following Qiao et al. (2019), we utilize the widely used encoder-decoder image captioning framework(Vinyals et al., 2015), and fix the parameters of the pre-trained image captioning framework when end-to-end training other modules in
ImagiT.
$$
p _ {t} = \operatorname {D e c o d e r} \left(h _ {t - 1}\right), t = 0, 1, \dots , L - 1, \tag {9}
$$
$$
\mathcal {L} _ {I 2 T} = - \sum_ {t = 0} ^ {L - 1} \log p _ {t} \left(T _ {t}\right). \tag {10}
$$
$p_t$  is the predicted probability distribution over the words at  $t$ -th decoding step, and  $T_t$  is the  $T_t$ -th entry of the probability vector.
3.4 Multimodal aggregation
After obtaining the imagined visual representation, we aggregate two modalities for the translation decoder. Although the vision carries richer information, it also contains irrelevant noise. Comparing to encoding and integrating visual feature directly, a more elegant method is to induce the hidden representation under the guide of image-aware attention and graph perspective of Transformer (Yao and Wan, 2020), since each local spatial regions of the image can also be considered as pseudo-tokens, which can be added to the source fully-connected graph. In the multimodal self-attention layer, we add the spatial feature of the generated feature map in the source sentence, that is, the attention query vector is the combination of text and visual embeddings, getting  $\tilde{x} \in \mathbb{R}^{(L + M) \times d}$ . Then perform image-aware attention, the key and value vectors are just text embeddings, we have:
$$
c _ {i} = \sum_ {j = 0} ^ {L - 1} \tilde {\alpha} _ {i j} \left(w _ {j} W ^ {V}\right), \tag {11}
$$
$$
\tilde {\alpha} _ {i j} = \operatorname {s o f t m a x} \left(\frac {\left(\tilde {x} _ {i} W ^ {Q}\right) \left(w _ {j} W ^ {K}\right) ^ {\top}}{\sqrt {d}}\right). \tag {12}
$$
3.5 Objective function
During the translation phase, similar to equation 10, we have:
$$
\mathcal {L} _ {\text {t r a n s}} = - \sum_ {t} \log p _ {t} \left(T _ {t}\right), \tag {13}
$$
To train the whole network end-to-end, we leverage adversarial training to alternatively train the generator and the discriminator. Especially, as shown in Figure 3, the discriminator take the imagined visual representation, source language sentence, and the real image as input, and we employ two adversarial losses: a visual realism adversarial
Figure 3: Training objective. The discriminator takes source language sentences, generated images, and real images as input, then computes two adversarial loss: realism loss and text-image paired loss.  $\mathcal{L}_{I2T}$  is designed to guarantee the semantic consistency, and  $\mathcal{L}_{trans}$  is the core loss function to translate integrated embedding to the target language.
loss, and a text-image pair-aware adversarial loss computed by the discriminator (Zhang et al., 2017; Xu et al., 2018; Qiao et al., 2019).
$$
\begin{array}{l} \mathcal {L} _ {G _ {0}} = - \frac {1}{2} \mathbb {E} _ {f _ {1} \sim p _ {G}} [ \log (D (f _ {1}) ] \tag {14} \\ - \frac {1}{2} \mathbb {E} _ {f _ {1} \sim p _ {G}} [ \log (D (f _ {1}, s) ], \\ \end{array}
$$
$f_{1}$  is the generated visual feature computed by equation 7 from the model distribution  $p_G$ ,  $s$  is the global sentence vector. The first term is to distinguish real and fake, ensuring that the generator generates visually realistic images. The second term is to guarantee the semantic consistency between the input text and the generated image.  $\mathcal{L}_{G_0}$  jointly approximates the unconditional and conditional distributions. The final objective function of the generator is defined as:
$$
\mathcal {L} _ {G} = \mathcal {L} _ {G _ {0}} + \lambda_ {1} \mathcal {L} _ {I 2 T} + \lambda_ {2} \mathcal {L} _ {\text {t r a n s}}. \tag {15}
$$
Accordingly, the discriminator  $D$  is trained by minimizing the following loss:
$$
\begin{array}{l} \mathcal {L} _ {D} = - \frac {1}{2} \mathbb {E} _ {I \sim p _ {d a t a}} [ \log (D (I) ] \\ - \frac {1}{2} \mathbb {E} _ {f _ {1} \sim p _ {G}} [ \log (1 - D (f _ {1}) ] (16) \\ - \frac {1}{2} \mathbb {E} _ {I \sim p _ {d a t a}} [ \log (D (I, s) ] (16) \\ - \frac {1}{2} \mathbb {E} _ {f _ {1} \sim p _ {G}} [ \log (1 - D (f _ {1}, s) ]. \\ \end{array}
$$
Where  $I$  is from the true image distribution  $p_{data}$ . The first two items are unconditional loss, the latter two are conditional loss.
Model	En⇒De	En⇒Fr
Test2016	Test2017	Test2016	Test2017
BLEU	METEOR	BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
Multimodal Neural Machine Translation Systems
IMGD(Calixto and Liu, 2017)	37.3	55.1	N/A	N/A	N/A	N/A	N/A	N/A
NMTSRC+IMG(Calixto et al., 2017)	36.5	55.0	N/A	N/A	N/A	N/A	N/A	N/A
fusion-conv (Caglayan et al., 2017)	37.0	57.0	29.8	51.2	53.5	70.4	51.6	68.6
trg-mul (Caglayan et al., 2017)	37.8	57.7	30.7	52.2	54.7	71.3	52.7	69.5
VAG-NMT(Zhou et al., 2018)	N/A	N/A	31.6	52.2	N/A	N/A	53.8	70.3
Transformer+Att (Ive et al., 2019)	38.0	55.6	N/A	N/A	59.8	74.4	N/A	N/A
Multimodal (Yao and Wan, 2020)	38.7	55.7	N/A	N/A	N/A	N/A	N/A	N/A
ImagiT + ground truth	38.6	55.7	32.4	52.5	59.9	74.3	52.8	68.6
Text-only Neural Machine Translation Systems
Transformer (Vaswani et al., 2017)	37.6	55.3	31.7	52.1	59.0	73.6	51.9	68.3
Multitask (Elliott and Kádár, 2017)	36.8	55.8	N/A	N/A	N/A	N/A	N/A	N/A
VMMTF(Calixto et al., 2019)	37.6	56.0	N/A	N/A	N/A	N/A	N/A	N/A
Lookup table (Zhang et al., 2020)	36.9	N/A	28.6	N/A	57.5	N/A	48.5	N/A
ImagiT	38.5	55.7	32.1	52.4	59.7	74.0	52.4	68.3
Table 1: Main result from the Test2016, Test2017 for the  $\mathrm{En}\Rightarrow \mathrm{De}$  and  $\mathrm{En}\Rightarrow \mathrm{Fr}$  MNMT task. The first category (Multimodal Neural Machine Translation Systems) collects the existing MNMT systems, which take both source sentences and paired images as input. The second category illustrates the systems that do not require images as input. Since our method falls into the second group, the baselines are the text-only Transformer (Vaswani et al., 2017) and the aforementioned works (Zhang et al., 2020; Elliott and Kádár, 2017).
We evaluate our proposed ImagiT model on two datasets, Multi30K (Elliott et al., 2016) and Ambiguous COCO (Elliott et al., 2017). To show its ability to train with external out-of-domain datasets, we adopt MS COCO (Lin et al., 2014) in the next analyzing section.
Multi30K is the largest existing human-labeled collection for MNMT, containing  $31K$  images and consisting of two multilingual expansions of the original Flickr30K(Young et al., 2014) dataset. The first expansion has five English descriptions and five German descriptions, and they are independent of each other. The second expansion has one of its English description manually translated to German by a professional translator, then expanded to French and Czech in the following shared task (Elliott et al., 2017; Barrault et al., 2018). We only apply the second expansion in our experiments, which has 29,000 instances for training, 1,014 for development, and 1,000 for evaluation. We present our results on English-German (En-De) English-French (En-Fr) Test2016 and Test2017.
Ambiguous COCO is a small evaluation dataset collected in the WMT2017 multimodal machine translation challenge (Elliott et al., 2017), which collected and translated a set of image descriptions that potentially contain ambiguous verbs. It contains 461 images from the MS COCO(Lin et al.,
2014) for 56 ambiguous vers in total.
MS COCO is the widely used non-parallel text-image paired dataset in T2I and I2T generation. It contains 82,783 training images and 40,504 validation images with 91 different object types, and each image has 5 English descriptions.
Our baseline is the conventional text-only Transformer (Vaswani et al., 2017). Specifically, each encoder-decoder has a 6-layer stacked Transformer network, eight heads, 512 hidden units, and the inner feed-forward layer filter size is set to 2048. The dropout is set to  $p = 0.1$ , and we use Adam optimizer (Kingma and Ba, 2015) to tune the parameter. The learning rate increases linearly for the warmup strategy with 8,000 steps and decreases with the step number's inverse square root. We train the model up to 10,000 steps, the early-stop strategy is adopted. We use the same setting as Vaswani et al. (2017). We use the metrics BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014) to evaluate the translation quality.
For the imagination network, the noise vector's dimension is 100, and the generated visual feature is  $128 \times 128$ . The upsampling and residual block in visual feature transformers consist of  $3 \times 3$  stride 1 convolution, batch normalization, and ReLU activation. The training is early-stopped if the dev set BLEU score do not improve for 10 epochs, since