SlowGuess/ACL_2020_OCR huggingface dataset:
Po-Yao Huang $^{1}$ , Junjie Hu $^{1}$ , Xiaojun Chang $^{2}$ , Alexander Hauptmann $^{1}$
$^{2}$ Faculty of Information Technology, Monash University
{poyaoh, junjieh, alex}@cs.cmu.edu, cxj273@gmail.com
Unsupervised machine translation (MT) has recently achieved impressive results with monolingual corpora only. However, it is still challenging to associate source-target sentences in the latent space. As people speak different languages biologically share similar visual systems, the potential of achieving better alignment through visual content is promising yet under-explored in unsupervised multimodal MT (MMT). In this paper, we investigate how to utilize visual content for disambiguation and promoting latent space alignment in unsupervised MMT. Our model employs multimodal back-translation and features pseudo visual pivoting in which we learn a shared multilingual visual-semiotic embedding space and incorporate visually-pivoted captioning as additional weak supervision. The experimental results on the widely used Multi30K dataset show that the proposed model significantly improves over the state-of-the-art methods and generalizes well when images are not available at the testing time.
Neural machine translation (Kalchbrenner and Blunsom, 2013; Sutskever et al., 2014) has achieved near human-level performance (Wu et al., 2016). However, its effectiveness strongly relies on the availability of large-scale parallel corpora. Unfortunately, preparing the parallel data remains a challenge as there are more than 6,500 languages in the world, and recruiting translators with bilingual or multilingual knowledge to cover all those languages is impractical.
As a result, developing methods alleviating the need of well-annotated large parallel corpora has recently attracted increasing attention in the community. These methods fall into two broad categories. The first type of methods use a third language as the pivot (Firat et al., 2016; Chen et al., 2017; Cheng et al., 2017; Johnson et al., 2017)
to enable zero-resource translation. Although the progress is encouraging, pivoting with a third language still demands bilingual knowledge for collecting large-scale parallel source-pivot and pivot-target corpora. The second type of methods explore unsupervised approaches (Conneau et al., 2018a; Artetxe et al., 2018; Lample et al., 2018a) have recently achieved impressive translation quality. These methods rely only on monolingual data and back-translation (Sennrich et al., 2016a). However, as discussed in (Lample et al., 2018b), the alignment of source-target sentences is uncertain and highly subject to proper initialization.
Using visual content for unsupervised MT (Chen et al., 2018; Su et al., 2019) is a promising solution for pivoting and alignment based on its availability and feasibility. Abundant multimodal content in various languages are available online (e.g. Instagram and YouTube). It is also easier to recruit monolingual annotators to describe an image than to find multilingual translators to translate sentences. Importantly, visual content is eligible to improve the alignments in the language latent spaces since the physical visual perception is similar among people speaking different languages (e.g. similar "blue car" for a German and a French).
Based on these insights, we propose a novel unsupervised multimodal MT framework incorporating images as pseudo pivots promoting latent space alignment. In addition to use features of visual objects for multimodal back-translation, we align a shared multilingual visual-semantic embedding (VSE) space via leveraging disjoint image-sentence pairs in different languages. As illustrated in Figure 2, for sentences approximately pivoted by similar images (src-image-tgt), drawing embeddings of corresponding image-sentence pairs closer results in better alignments of semantically equivalent sentences in the language latent spaces. Inspired by back-translation, we further explore another pseudo pivoting strategy, which approximates multilingual
8226
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8226-8237 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
sentence pairs (src-image-tgt) conditioned on a real image via captioning. Instead of using annotation of images for pivoting as in (Chen et al., 2018), we generate sentences in two languages pivoted on the real image, and then approximately pairing them as weak supervision for training unsupervised MT system. This approach is analogous to a cross-modal version of back-translation.
We make the following contributions: (1) Building a unified view of employing visual content for pseudo pivoting. (2) We learn and improve the alignments in the shared multilingual multimodal embedding space for unsupervised MMT with disjoint image-text pairs in different languages. (3) Our model achieves state of the art on Multi30K and generalizes well to the text-only scenario.
Neural Machine Translation Typical NMT models are based on the encoder-decoder framework with attention (Bahdanau et al., 2015). Let  $\mathbf{x} = (x_{1},\dots ,x_{N})$  denotes a source sentence and  $\mathbf{y} = (y_1,\dots ,y_M)$  denotes a target sentence, where  $(\mathbf{x},\mathbf{y})\in (\mathcal{X},\mathcal{Y})$ . The encoder-decoder model learns to estimate the following likelihood from the source sentence to the target sentence:
$$
p _ {x \rightarrow y} (\mathbf {y} | \mathbf {x}) = \prod_ {i = 1} ^ {M} p \left(y _ {i} \mid \mathbf {y} _ {<   i}, \mathbf {x}\right) \tag {1}
$$
When a parallel corpus is available, the maximum likelihood estimation (MLE) is usually adopted to optimize the (source to target language) NMT model by minimizing the following loss:
$$
\mathcal {L} _ {x \rightarrow y} ^ {M T} = \mathbb {E} _ {(\mathbf {x}, \mathbf {y}) \sim (\mathcal {X}, \mathcal {Y})} [ - \log p _ {x \rightarrow y} (\mathbf {y} | \mathbf {x}) ] \quad (2)
$$
Among all encoder-decoder models, the Transformer (Vaswani et al., 2017) architecture recently achieves state-of-the-art translation quality. Instead of using recurrent or convolutional operations, it facilitates multi-head self-attention (Lin et al., 2017). In this paper, we choose the Transformer as the underlying architecture for both the translation and the captioning modules.
Unsupervised Machine Translation While conventional MT systems rely on the availability of a large parallel corpus, translation with zero-resource (unsupervised MT) (Lample et al., 2018a; Artetxe et al., 2018; Lample et al., 2018b) has drawn increasing research attention. Only monolingual sentences are presented at the training and validation phase, i.e., only  $\mathbf{x} \in \mathcal{X}$  and  $\mathbf{y} \in \mathcal{Y}$  are available.
Successful unsupervised MT systems share several common principles. First, they require the pre-training step to initialize the model and establish strong monolingual language models properly. For example, XLM (Conneau and Lample, 2019) utilizes the masked language model objective in BERT (Devlin et al., 2019). MASS (Song et al., 2019) utilizes a span-based sequence-to-sequence masking objective for language model pre-training.
Second, these systems transform the unsupervised problem into a weakly or self-supervised one by automatically generating pseudo sentence pairs via back-translation (Sennrich et al., 2016a). The idea behind can be analogous to the cycle-consistency objective in CycleGAN (Zhu et al., 2017) for image-image translation with unpaired data. Specifically, let us denote by  $h^{*}(\mathbf{y}) = (\hat{x}_{1},\dots ,\hat{x}_{N})$  the sentence in the source language inferred from  $\mathbf{y}\in \mathcal{V}$  such that  $h^{*}(\mathbf{y}) = \mathrm{argmax}p_{y\to x}(\mathbf{x}|\mathbf{y})$ . Similarly, let us denote by  $g^{*}(\mathbf{x}) = (\hat{y}_{1},\dots ,\hat{y}_{M})$  the sentence in the target language inferred from  $\mathbf{x}\in \mathcal{X}$  such that  $g^{*}(\mathbf{x}) = \mathrm{argmax}p_{x\to y}(\mathbf{y}|\mathbf{x})$ . Then the "pseudo" parallel sentences  $(h^{*}(\mathbf{y}),\mathbf{y})$  and  $(\mathbf{x},g^{*}(\mathbf{x}))$  can be further used to train two two MT models  $(\mathcal{X}\rightarrow \mathcal{Y})$  and  $\mathcal{Y}\rightarrow \mathcal{X})$  by minimizing the following back-translation loss:
$$
\begin{array}{l} \mathcal {L} _ {x \leftrightarrow y} ^ {B T} = \mathbb {E} _ {\mathbf {x} \sim \mathcal {X}} [ - \log p _ {y \rightarrow x} (\mathbf {x} | g ^ {*} (\mathbf {x})) ] \tag {3} \\ + \mathbb {E} _ {\mathbf {y} \sim \mathcal {Y}} [ - \log p _ {x \rightarrow y} (\mathbf {y} | h ^ {*} (\mathbf {y})) ] \\ \end{array}
$$
Although reinforcement learning-based approaches (He et al., 2016a) and Gumbel-softmax reparametrization (Maddison et al., 2017) have been used to handle back-propagation thorough non-differentiable "argmax" predictions. In this paper, we do not back-propagate through  $h^{*}(\mathbf{y})$  and  $g^{*}(\mathbf{x})$  to simplify the training process.
3 Unsupervised Multimodal Machine Translation
As illustrated in Figure 1, our model is composed of seven modules: Two encoder-decoder pairs for translation, two decoders for captioning, and one shared visual encoder. In this section, we first detail our basic MMT model architecture and the unsupervised setup. Then we introduce pseudo visual pivoting: learning multilingual VSE and pivoted captioning.
3.1 Multimodal MT
Multimodal machine translation (Specia et al., 2016) (MMT) considers additional images as a
8227
Figure 1: The proposed model structure (English  $\leftrightarrow$  German). We incorporate visual objects for unsupervised multimodal MT and improve the language latent space alignment with pseudo visual pivoting ( $\S 3.3-\S 3.4$ ).
complementary information source for MT. An image  $\mathbf{z}$  and the description in two languages form a triplet  $(\mathbf{x},\mathbf{y},\mathbf{z})\in (\mathcal{X},\mathcal{Y},\mathcal{Z})$ . The Transformer encoder reads the source sentence and encodes it with hierarchical self-attention into  $\mathbf{h}^x = \{\mathbf{h}_1^x,\dots ,\mathbf{h}_N^x\}$ ,  $\mathbf{h}_i^x\in \mathbb{R}^d$ , where  $d$  is the dimension of the embedding space. The visual encoder encodes the image into  $\mathbf{h}^z = \{\mathbf{h}_1^z,\dots ,\mathbf{h}_K^z\}$ ,  $\mathbf{h}_i^z\in \mathbb{R}^d$ ,  $K_{\mathrm{max}} = 36$ . Most previous work (Chen et al., 2018; Su et al., 2019) use 2D  $(K = 14\times 14)$  feature maps of ImageNet pre-trained ResNet (He et al., 2016b). In contrast, we utilize the regional features of  $K$  salient visual objects in an image extracted by Faster-RCNN (Ren et al., 2015) and a 1-layer MLP as the encoder to encode visual objects.
Various attention strategies for sequence-to-sequence learning have been addressed in (Libovický and Helcl, 2017). Our model employs the hierarchical multi-head multimodal attention for decoding. For decoding at time stamp  $i$ , the textual attention  $\mathrm{Attn}(\mathbf{h}_i^y,\mathbf{h}^x)$  computes the context vector  $\mathbf{c}_i = \sum_j\alpha_j\mathbf{h}_j^x$  via a attention-based alignment  $\alpha_{j} = \mathrm{Align}(\mathbf{h}_{i}^{y},\mathbf{h}_{j}^{x})$ , where  $\sum_{j}\alpha_{j} = 1$  and  $\mathbf{h}_i^y$  is the decoder state. Essentially, the one-head attention in Transformer is implemented as  $\mathbf{c}_i =$ $\mathrm{softmax}(\mathbf{Q}_i(\mathbf{K}^x)^\top /\sqrt{d})\mathbf{V}^x$  where  $\{\mathbf{Q},\mathbf{K}^x,\mathbf{V}^x\}$  are the packed  $d$ -dimensional Query, Key, Value vectors, which are the mapped and packed version of  $\{\mathbf{h}_i^y,\mathbf{h}^x,\mathbf{h}^x\}$ . For decoding with encoded visual and textual inputs, we utilize multimodal attention to compute the context vector  $\mathbf{c}_i$ :
$$
\mathbf {c} _ {i} ^ {x} = \operatorname {A t t n} \left(\mathbf {h} _ {i - 1} ^ {y}, \mathbf {h} ^ {x}\right) + \lambda_ {v} \operatorname {A t t n} \left(\mathbf {h} _ {i - 1} ^ {y}, \mathbf {h} ^ {z}\right) \tag {4}
$$
In practice we set  $\lambda_v = 1.0$ . Our multimodal decoder models the likelihood to predict the next token as:
$$
p \left(y _ {i} \mid \mathbf {y} _ {<   i}, \mathbf {x}, \mathbf {z}\right) = \operatorname {s o f t m a x} \left(f \left(\mathbf {c} _ {i}, y _ {i - 1}, \mathbf {h} _ {i - 1} ^ {y}\right), \right. \tag {5}
$$
where  $f(.)$  denotes the aggregated non-linear feature mapping in Transformer.
3.2 Unsupervised Learning
Unsupervised multimodal MT (Nakayama and Nishida, 2017; Chen et al., 2018; Su et al., 2019) poses a new yet challenging problem. On both the source and target sides, only non-overlapping monolingual multimodal data are presented for training and validation. Specifically, the data available are:  $(\mathbf{x},\mathbf{z}_x)\in (\mathcal{X},\mathcal{Z}),(\mathbf{y},\mathbf{z}_y)\in (\mathcal{Y},\mathcal{Z})$ , such that  $\{\mathbf{x}\} \cap \{\mathbf{y}\} = \phi$ ,  $\{\mathbf{z}_x\} \cap \{\mathbf{z}_y\} = \phi$ . Note that there are no parallel translation pairs available (unsupervised), and the images are mutually exclusive for different languages.
For multimodal back-translation, the generated pseudo target sentence conditioned on the source sentence and image can be re-written as  $g^{*}(\mathbf{x},\mathbf{z}_{x}) = \mathrm{argmax}p_{xz\to y}(\mathbf{y}|\mathbf{x},\mathbf{z}_{x})$ , where  $p_{xz\to y}(\mathbf{y}|\mathbf{x},\mathbf{z}) = \prod_{i = 1}^{M}p(y_{i}|\mathbf{y}_{< i},\mathbf{x},\mathbf{z})$ . Similar for  $p_{yz\to x}(\mathbf{x}|\mathbf{y},\mathbf{z})$  and  $h^{*}(\mathbf{y},\mathbf{z}_{y})$ . For unsupervised multimodal MT, the multimodal back-translation objective can be extended as:
$$
\begin{array}{l} \mathcal {L} _ {x \leftrightarrow y} ^ {M B T} = \mathbb {E} _ {(\mathbf {x}, \mathbf {z} _ {x})} \left[ - \log p _ {y z \rightarrow x} \left(\mathbf {x} \mid g ^ {*} (\mathbf {x}, \mathbf {z} _ {x}), \mathbf {z} _ {x}\right)\right] \\ + \mathbb {E} _ {\left(\mathbf {y}, \mathbf {z} _ {y}\right)} \left[ \right. - \log p _ {x z \rightarrow y} \left(\mathbf {y} \mid h ^ {*} \left(\mathbf {y}, \mathbf {z} _ {y}\right), \mathbf {z} _ {y}\right)\left. \right) \tag {6} \\ \end{array}
$$
We simplify the notation of expectation for clarity.
Aligning the latent spaces of the source and target languages without supervision is challenging, as discussed in (Lample et al., 2018b). However, as people speak different languages biologically share similar visual systems, we envision that the shared visual space can serve as the pivot for alignment. Unlike most previous work (Chen et al., 2018; Su et al., 2019) treating images merely as a feature, we propose two visual pivoting approaches: (1) Aligning the multilingual VSE space; (2) Image pseudo pivoting via captioning. As illustrated in Figure 2, for (1), we use images as the approximate pivots connecting real non-parallel sentences. (src-image-tgt.) In (2), for each pivoting real image, we gener
8228
Figure 2: Pseudo visual pivoting: (1) multilingual VSE (src-Img-tgt, in fact src-img $_1$ , tgt-img $_2$ ), and (2) pivoted captioning (src-img-tgt). The italic items do not exist and are approximated (pseudo). (src, img, tgt) is colored in (green, yellow, blue). Solid red and black lines indicate captioning and translation without updates. Encoder-decoder are updated with dashed lines to improve the alignments in the multilingual multimodal embedding space.
ate captions in both languages to construct “pseudo” source-target sentence pairs. (src-Img-tgt), where the italic item is “pseudo”. We collectively term the proposed approach pseudo visual pivoting.
3.3 Multilingual Visual-Semantic Embedding
We posit that for  $\mathcal{X},\mathcal{Y},\mathcal{Z}$ , the two language spaces  $\mathcal{X},\mathcal{Y}$  could be properly associated by respectively aligning two monolingual VSE spaces  $\mathcal{X}\leftrightarrow \mathcal{Z}$  and  $\mathcal{Y}\leftrightarrow \mathcal{Z}$ . We leverage the contrastive objective in cross-modal retrieval (Kiros et al., 2014; Huang et al., 2019b) for aligning multimodal inputs in the shared VSE space where the embeddings are close if they are semantically associated or paired.
Specifically, we generalize the fine-grained (object-level and token-level), monolingual textual-to-visual, and visual-to-textual attention (Lee et al., 2018; Huang et al., 2019c) into the multilingual setup. For fine-grained image-sentence alignment, let  $s_{ij} = \cos(\mathbf{h}_i^x, \mathbf{h}_j^z)$  denotes the cosine similarity between the  $i$ -th encoded token and the  $j$ -th encoded visual object. The image-sentence similarity can be measured by averaging the cosine similarities between the visually-attend sentence embeddings and the visual embeddings of the objects. The visually-attended sentence embeddings  $\mathbf{h}^{zx}$  are the weighted combination of the encoded tokens  $\mathbf{h}^x$ . Precisely, we compute  $\mathbf{h}_j^{zx} = \sum_{i=1}^{N} \alpha_{ij} \mathbf{h}_i^x$ , where  $j = 1 \cdots K$  and  $\alpha_{ij} = \text{softmax}_i(s_{ij})$ . Let us denote by  $S(\mathbf{x}, \mathbf{z}) = \frac{1}{2K} \sum_{j=1}^{K} \cos(\mathbf{h}_j^{zx}, \mathbf{h}_j^z) + \frac{1}{2N} \sum_{i=1}^{N} \cos(\mathbf{h}_i^{xz}, \mathbf{h}_i^x)$  as the image-sentence similarity, the contrastive triplet loss encouraging image-sentence alignment in the VSE space can be written as:
$$
\begin{array}{l} \mathcal {L} _ {c} (\mathbf {x}, \mathbf {z}) = \max  _ {\tilde {\mathbf {x}}} \left[ \gamma - S (\mathbf {x}, \mathbf {z}) + S (\tilde {\mathbf {x}}, \mathbf {z}) \right] _ {+} \tag {7} \\ + \max _ {\tilde {\mathbf {z}}} \big [ \gamma - S (\mathbf {x}, \mathbf {z}) + S (\mathbf {x}, \tilde {\mathbf {z}}) \big ] _ {+}, \\ \end{array}
$$
where  $[.]_{+}$  is the hinge function, and  $\tilde{\mathbf{x}}$  and  $\tilde{\mathbf{z}}$  are the non-paired (negative) instances for  $\mathbf{x}$  and  $\mathbf{z}$ . Intuitively, when the loss decreases, the matched images and sentences will be drawn closer down to a margin  $\gamma$  than the hardest non-paired ones. Formally, we minimizing the following objective for cross-modal alignments in the two VSE spaces:
$$
\mathcal {L} _ {x, y, z} ^ {V S E} = \mathbb {E} _ {(\mathbf {x}, \mathbf {z} _ {x})} \left[ \mathcal {L} _ {c} (\mathbf {x}, \mathbf {z} _ {x}) \right] + \mathbb {E} _ {(\mathbf {y}, \mathbf {z} _ {y})} \left[ \mathcal {L} _ {c} (\mathbf {y}, \mathbf {z} _ {y}) \right] \tag {8}
$$
3.4 Image Captioning for Pseudo Pivoting
Inspired by back-translation with monolingual corpora, we propose a novel cross-modal approach to generate weakly-supervised pairs to guide language space alignment for unsupervised MMT. Precisely, we leverage image captioning to synthesize pseudo sentence pairs (pivoted and conditioned on the image) for back-translation and paired-translation.
Image Captioning Image captioning models are akin to MT models besides the non-sequential visual encoder. For example, an image-to-source captioning model estimates the likelihood as  $p_{z \to x}(\mathbf{x}|\mathbf{z}) = \prod_{i=1}^{N} p(x_i|\mathbf{x}_{<i},\mathbf{z})$ , where  $\mathbf{z}$  is the encoded image. Essentially, the captioning model learns to minimize the following loss:
$$
\mathcal {L} _ {z \rightarrow x} ^ {C A P} = \mathbb {E} _ {\left(\mathbf {z} _ {x}, \mathbf {x}\right)} \left[ - \log p _ {z \rightarrow x} (\mathbf {x} | \mathbf {z} _ {x}) \right] \tag {9}
$$
As illustrated in Figure 2, we incorporate two captioning models  $\mathcal{Z} \to \mathcal{X}$  and  $\mathcal{Z} \to \mathcal{Y}$  to generate additional "pseudo" parallel sentences pivoted on the image as additional weak supervision to better align language latent spaces in unsupervised MMT. For example, with Image  $\rightarrow$  English and Image  $\rightarrow$  German, the generated pseudo (English, German)
8229
pair is then pivoted on the Image. Learning captioning models is practical as it is easier to collect large-scale image-text pairs than translation pairs. We pre-train these captioning models and use them to generate sentences in two languages depicting the same image, i.e.,  $c_{x}^{*}(\mathbf{z}_{x}) = \mathrm{argmax}p_{z\to x}(\mathbf{x}|\mathbf{z}_{x})$  and  $c_{y}^{*}(\mathbf{z}_{x}) = \mathrm{argmax}p_{z\to y}(\mathbf{y}|\mathbf{z}_{x})$ . The pivoted captions then enable the following two objectives:
Pivoted Captioning for Back-Translation We utilize the synthetic multilingual captions (i.e.,  $c_{x}^{*}(\mathbf{z}_{x}), c_{y}^{*}(\mathbf{z}_{x})$  from the source images and  $c_{x}^{*}(\mathbf{z}_{y}), c_{y}^{*}(\mathbf{z}_{y})$  from the target images) to reversely reconstruct the synthetic captions from their translations in both directions. Formally, we compute the following caption-based back-translation loss:
$$
\begin{array}{l} \mathcal {L} _ {x \leftrightarrow y} ^ {C B T} = \mathbb {E} _ {\mathbf {z} _ {x}} \left[ - \log p _ {y z \rightarrow x} \left(c _ {x} ^ {*} (\mathbf {z} _ {x}) | g ^ {*} \left(c _ {x} ^ {*} (\mathbf {z} _ {x}), \mathbf {z} _ {x}\right), \mathbf {z} _ {x}\right)\right. \\ \left.\left. - \log p _ {x z \rightarrow y} \left(c _ {y} ^ {*} (\mathbf {z} _ {x}) \mid g ^ {*} \left(c _ {y} ^ {*} (\mathbf {z} _ {x}), \mathbf {z} _ {x}\right), \mathbf {z} _ {x}\right)\right]\right] \\ + \mathbb {E} _ {\mathbf {Z} _ {y}} \left[ - \log p _ {y z \rightarrow x} \left(c _ {x} ^ {*} (\mathbf {z} _ {y}) | h ^ {*} (c _ {x} ^ {*} (\mathbf {z} _ {y}), \mathbf {z} _ {y}), \mathbf {z} _ {y}\right)\right. \\ \left. - \log p _ {x z \rightarrow y} \left(c _ {y} ^ {*} (\mathbf {z} _ {y}) | h ^ {*} \left(c _ {y} ^ {*} (\mathbf {z} _ {y}), \mathbf {z} _ {y}\right), \mathbf {z} _ {y}\right)\right] \tag {10} \\ \end{array}
$$
Pivoted Captioning for Paired-Translation With the synthetic "pseudo" paired (source, target) captions pivoted on a image  $(e.g. (c_{y}^{*}(\mathbf{z}_{x}), c_{x}^{*}(\mathbf{z}_{x}))$ , the caption-based paired-translation loss is defined as:
$$
\begin{array}{l} \mathcal {L} _ {x \leftrightarrow y} ^ {C P T} = \mathbb {E} _ {\mathbf {z} _ {x}} \left[ - \log p _ {x z \rightarrow y} \left(c _ {y} ^ {*} (\mathbf {z} _ {x}) \mid c _ {x} ^ {*} (\mathbf {z} _ {x}), \mathbf {z} _ {x}\right)\right] \\ + \mathbb {E} _ {\mathbf {z} _ {y}} \left[ - \log p _ {y z \rightarrow x} \left(c _ {x} ^ {*} \left(\mathbf {z} _ {y}\right) \mid c _ {y} ^ {*} \left(\mathbf {z} _ {y}\right), \mathbf {z} _ {y}\right)\right] \tag {11} \\ \end{array}
$$
Note that similar to the text back-translation, for  $\mathcal{L}_{x\leftrightarrow y}^{CPT}$  and  $\mathcal{L}_{x\leftrightarrow y}^{CBT}$ , we do not back-prop through the captioning step. For optimization, we sample mini-batches and minimizing the following loss:
$$
\mathcal {L} = \mathcal {L} _ {x \leftrightarrow y} ^ {M B T} + \mathcal {L} _ {x, y, z} ^ {V S E} + \mathcal {L} _ {x \leftrightarrow y} ^ {C B T} + \mathcal {L} _ {x \leftrightarrow y} ^ {C P T} \tag {12}
$$
Here we drop the weights  $w$  of each loss for clarity. In practice, all the weights are set to 1.0 except for  $w_{CPT}$  where we employ a decreasing learning scheduler specified in the next section.
We first describe the implementation details and the experimental setup. Then we compare our approach with baselines with detailed analysis.
4.1 Dataset and Preprocessing
We conduct experiments on the Multi30K (Elliott et al., 2016) dataset, the benchmark dataset for mul
timodal MT. It contains 29K training, 1K validation, and 1K testing images. Each image has three descriptions in English/German/French, which are translations of each other.
To ensure the model never learn from parallel sentences, we randomly split Multi30K training and validation sets in half for one language and use the complementary half for the other. The resulting M30k-half are two corpora with non-overlapping 14,500 training and 507 validation image-sentence pairs, respectively.
For text pre-processing, we use Moses (Koehn et al., 2007) scripts for tokenization and apply the Byte Pair Encoding (BPE) (Sennrich et al., 2016b) from XLM. To identify and extract features of visual objects in images, we use the Faster-RCNN (Ren et al., 2015) model in (Anderson et al., 2018) to detect up to 36 salient visual objects per image and extract their corresponding 2048-dim regional features.