SlowGuess/ACL_2020_OCR huggingface dataset:
Yongjing Yin $^{1*}$ , Fandong Meng $^{2}$ , Jinsong Su $^{1\dagger}$ , Chulun Zhou $^{1}$ , Zhengyuan Yang $^{3}$ , Jie Zhou $^{2}$ , Jiebo Luo $^{3}$
$^{1}$ Xiamen University, Xiamen, China
$^{2}$ Pattern Recognition Center, WeChat AI, Tencent Inc, Beijing, China
$^{3}$ Department of Computer Science, University of Rochester, Rochester NY 14627, USA
yinyongjing@stu.xmu.edu.cn fandongmeng@tencent.com
jssu@xmu.edu.cn
Multi-modal neural machine translation (NMT) aims to translate source sentences into a target language paired with images. However, dominant multi-modal NMT models do not fully exploit fine-grained semantic correspondences between semantic units of different modalities, which have potential to refine multi-modal representation learning. To deal with this issue, in this paper, we propose a novel graph-based multi-modal fusion encoder for NMT. Specifically, we first represent the input sentence and image using a unified multi-modal graph, which captures various semantic relationships between multi-modal semantic units (words and visual objects). We then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions to learn node representations. Finally, these representations provide an attention-based context vector for the decoder. We evaluate our proposed encoder on the Multi30K datasets. Experimental results and in-depth analysis show the superiority of our multi-modal NMT model.
Multi-modal neural machine translation (NMT) (Huang et al., 2016; Calixto et al., 2017) has become an important research direction in machine translation, due to its research significance in multimodal deep learning and wide applications, such as translating multimedia news and web product information (Zhou et al., 2018). It significantly extends the conventional text-based machine translation by taking images as additional inputs. The assumption behind this is that the translation is expected to be more accurate compared to purely text-based
translation, since the visual context helps to resolve ambiguous multi-sense words (Ive et al., 2019).
Apparently, how to fully exploit visual information is one of the core issues in multi-modal NMT, which directly impacts the model performance. To this end, a lot of efforts have been made, roughly consisting of: (1) encoding each input image into a global feature vector, which can be used to initialize different components of multi-modal NMT models, or as additional source tokens (Huang et al., 2016; Calixto et al., 2017), or to learn the joint multi-modal representation (Zhou et al., 2018; Calixto et al., 2019); (2) extracting object-based image features to initialize the model, or supplement source sequences, or generate attention-based visual context (Huang et al., 2016; Ive et al., 2019); and (3) representing each image as spatial features, which can be exploited as extra context (Calixto et al., 2017; Delbrouck and Dupont, 2017a; Ive et al., 2019), or a supplement to source semantics (Delbrouck and Dupont, 2017b) via an attention mechanism.
Despite their success, the above studies do not fully exploit the fine-grained semantic correspondences between semantic units within an input sentence-image pair. For example, as shown in Figure 1, the noun phrase "a toy car" semantically corresponds to the blue dashed region. The neglect of this important clue may be due to two big challenges: 1) how to construct a unified representation to bridge the semantic gap between two different modalities, and 2) how to achieve semantic interactions based on the unified representation. However, we believe that such semantic correspondences can be exploited to refine multimodal representation learning, since they enable the representations within one modality to incorporate cross-modal information as supplement during multi-modal semantic interactions (Lee et al., 2018; Tan and Bansal, 2019).
*This work is done when Yongjing Yin was interning at Pattern Recognition Center, WeChat AI, Tencent Inc, China.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3025-3035 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Figure 1: The multi-modal graph for an input sentence-image pair. The blue and green solid circles denote textual nodes and visual nodes respectively. An intra-modal edge (dotted line) connects two nodes in the same modality, and an inter-modal edge (solid line) links two nodes in different modalities. Note that we only display edges connecting the textual node "playing" and other textual ones for simplicity.
In this paper, we propose a novel graph-based multi-modal fusion encoder for NMT. We first represent the input sentence and image with a unified multi-modal graph. In this graph, each node indicates a semantic unit: textual word or visual object, and two types of edges are introduced to model semantic relationships between semantic units within the same modality (intra-modal edges) and semantic correspondences between semantic units of different modalities (inter-modal edges) respectively. Based on the graph, we then stack multiple graph-based multi-modal fusion layers that iteratively perform semantic interactions among the nodes to conduct graph encoding. Particularly, during this process, we distinguish the parameters of two modalities, and sequentially conduct intra- and inter-modal fusions to learn multi-modal node representations. Finally, these representations can be exploited by the decoder via an attention mechanism.
Compared with previous models, ours is able to fully exploit semantic interactions among multimodal semantic units for NMT. Overall, the major contributions of our work are listed as follows:
- We propose a unified graph to represent the input sentence and image, where various semantic relationships between multi-modal semantic units can be captured for NMT.
- We propose a graph-based multi-modal fusion encoder to conduct graph encoding based on the above graph. To the best of our knowledge, our work is the first attempt to explore multimodal graph neural network (GNN) for NMT.
- We conduct extensive experiments on Multi30k datasets of two language pairs.
Experimental results and in-depth analysis indicate that our encoder is effective to fuse multi-modal information for NMT. Particularly, our multi-modal NMT model significantly outperforms several competitive baselines.
- We release the code at https://github.com/DeepLearnXMU/GMNMT.
2 NMT with Graph-based Multi-modal Fusion Encoder
Our multi-modal NMT model is based on attentional encoder-decoder framework with maximizing the log likelihood of training data as the objective function.
2.1 Encoder
Essentially, our encoder can be regarded as a multimodal extension of GNN. To construct our encoder, we first represent the input sentence-image pair as a unified multi-modal graph. Then, based on this graph, we stack multiple multi-modal fusion layers to learn node representations, which provides the attention-based context vector to the decoder.
2.1.1 Multi-modal Graph
In this section, we take the sentence and the image shown in Figure 1 as an example, and describe how to use a multi-modal graph to represent them. Formally, our graph is undirected and can be formalized as  $G = (V, E)$ , which is constructed as follows:
In the node set  $V$ , each node represents either a textual word or a visual object. Specifically, we adopt the following strategies to construct these two kinds of nodes: (1) We include all words as separate textual nodes in order to fully exploit textual
Figure 2: The architecture of our NMT model with the graph-based multi-modal fusion encoder. Note that we actually do not apply a Visual FFN to the last layer in the encoder.
information. For example, in Figure 1, the multimodal graph contains totally eight textual nodes, each of which corresponds to a word in the input sentence; (2) We employ the Stanford parser to identify all noun phrases in the input sentence, and then apply a visual grounding toolkit (Yang et al., 2019) to detect bounding boxes (visual objects) for each noun phrase. Subsequently, all detected visual objects are included as independent visual nodes. In this way, we can effectively reduce the negative impact of abundant unrelated visual objects. Let us revisit the example in Figure 1, where we can identify two noun phrases "Two boys" and "a toy car" from the input sentence, and then include three visual objects into the multi-modal graph.
To capture various semantic relationships between multi-modal semantic units for NMT, we consider two kinds of edges in the edge set  $E$ : (1) Any two nodes in the same modality are connected by an intra-modal edge; and (2) Each textual node representing any noun phrase and the corresponding visual node are connected by an inter-modal edge. Back to Figure 1, we can observe that all visual nodes are connected to each other, and all textual nodes are fully-connected. However, only nodes  $v_{o_1}$  and  $v_{x_1}$ ,  $v_{o_1}$  and  $v_{x_2}$ ,  $v_{o_2}$  and  $v_{x_1}$ ,  $v_{o_2}$  and  $v_{x_2}$ ,  $v_{o_3}$  and  $v_{x_6}$ ,  $v_{o_3}$  and  $v_{x_7}$ ,  $v_{o_3}$  and  $v_{x_8}$  are connected by inter-modal edges.
2.1.2 Embedding Layer
Before inputting the multi-modal graph into the stacked fusion layers, we introduce an embedding
layer to initialize the node states. Specifically, for each textual node  $v_{x_i}$ , we define its initial state  $H_{x_i}^{(0)}$  as the sum of its word embedding and position encoding (Vaswani et al., 2017). To obtain the initial state  $H_{o_j}^{(0)}$  of the visual node  $v_{o_j}$ , we first extract visual features from the fully-connected layer that follows the ROI pooling layer in Faster-RCNN (Ren et al., 2015), and then employ a multi-layer perceptron with ReLU activation function to project these features onto the same space as textual representations.
2.1.3 Graph-based Multi-modal Fusion Layers
As shown in the left part of Figure 2, on the top of embedding layer, we stack  $L_{e}$  graph-based multimodal fusion layers to encode the above-mentioned multi-modal graph. At each fusion layer, we sequentially conduct intra- and inter-modal fusions to update all node states. In this way, the final node states encode both the context within the same modality and the cross-modal semantic information simultaneously. Particularly, since visual nodes and textual nodes are two types of semantic units containing the information of different modalities, we apply similar operations but with different parameters to model their state update process, respectively.
Specifically, in the  $l$ -th fusion layer, both updates of textual node states  $\mathbf{H}_x^{(l)} = \{H_{x_i}^{(l)}\}$  and visual node states  $\mathbf{H}_o^{(l)} = \{H_{o_j}^{(l)}\}$  mainly involve the following steps:
Step1: Intra-modal fusion. At this step, we employ self-attention to generate the contextual representation of each node by collecting the message from its neighbors of the same modality.
Formally, the contextual representations  $\mathbf{C}_x^{(l)}$  of all textual nodes are calculated as follows:
$$
\mathbf {C} _ {x} ^ {(l)} = \operatorname {M u l t i H e a d} \left(\mathbf {H} _ {x} ^ {(l - 1)}, \mathbf {H} _ {x} ^ {(l - 1)}, \mathbf {H} _ {x} ^ {(l - 1)}\right), \tag {1}
$$
where MultiHead(Q, K, V) is a multi-head self-attention function taking a query matrix Q, a key matrix K, and a value matrix V as inputs. Similarly, we generate the contextual representations  $\mathbf{C}_o^{(l)}$  of all visual nodes as
$$
\mathbf {C} _ {o} ^ {(l)} = \operatorname {M u l t i H e a d} \left(\mathbf {H} _ {o} ^ {(l - 1)}, \mathbf {H} _ {o} ^ {(l - 1)}, \mathbf {H} _ {o} ^ {(l - 1)}\right). \tag {2}
$$
In particular, since the initial representations of visual objects are extracted from deep CNNs, we apply a simplified multi-head self-attention to preserve the initial representations of visual objects, where the learned linear projects of values and final outputs are removed.
Step2: Inter-modal fusion. Inspired by studies in multi-modal feature fusion (Teney et al., 2018; Kim et al., 2018), we apply a cross-modal gating mechanism with an element-wise operation to gather the semantic information of the cross-modal neighbours of each node.
Concretely, we generate the representation  $M_{x_i}^{(l)}$  of a text node  $v_{x_i}$  in the following way:
$$
M _ {x _ {i}} ^ {(l)} = \sum_ {j \in A \left(v _ {x _ {i}}\right)} \alpha_ {i, j} \odot C _ {o j} ^ {(l)}, \tag {3}
$$
$$
\alpha_ {i, j} = \operatorname {S i g m o i d} \left(\mathbf {W} _ {1} ^ {(l)} C _ {x _ {i}} ^ {(l)} + \mathbf {W} _ {2} ^ {(l)} C _ {o _ {j}} ^ {(l)}\right), \tag {4}
$$
where  $A(v_{x_i})$  is the set of neighboring visual nodes of  $v_{x_i}$ , and  $\mathbf{W}_1^{(l)}$  and  $\mathbf{W}_2^{(l)}$  are parameter matrices. Likewise, we produce the representation  $M_{o_j}^{(l)}$  of a visual node  $v_{o_j}$  as follows:
$$
M _ {o _ {j}} ^ {(l)} = \sum_ {i \in A \left(v _ {o _ {j}}\right)} \beta_ {j, i} \odot C _ {x _ {i}} ^ {(l)}, \tag {5}
$$
$$
\beta_ {j, i} = \operatorname {S i g m o i d} \left(\mathbf {W} _ {3} ^ {(l)} C _ {o _ {j}} ^ {(l)} + \mathbf {W} _ {4} ^ {(l)} C _ {x _ {i}} ^ {(l)}\right), \tag {6}
$$
where  $A(v_{o_j})$  is the set of adjacent textual nodes of  $v_{o_j}$ , and  $\mathbf{W}_3^{(l)}$  and  $\mathbf{W}_4^{(l)}$  are also parameter matrices.
The advantage is that the above fusion approach can better determine the degree of inter-modal fusion according to the contextual representations of
each modality. Finally, we adopt position-wise feed forward networks  $\mathrm{FFN}(*)$  to generate the textual node states  $\mathbf{H}_x^{(l)}$  and visual node states  $\mathbf{H}_o^{(l)}$ :
$$
\mathbf {H} _ {x} ^ {(l)} = \operatorname {F F N} \left(\mathbf {M} _ {x} ^ {(l)}\right), \tag {7}
$$
$$
\mathbf {H} _ {o} ^ {(l)} = \operatorname {F F N} \left(\mathbf {M} _ {o} ^ {(l)}\right), \tag {8}
$$
where  $\mathbf{M}_x^{(l)} = \{M_{x_i}^{(l)}\}$ ,  $\mathbf{M}_o^{(l)} = \{M_{o_j}^{(l)}\}$  denote the above updated representations of all textual nodes and visual nodes respectively.
2.2 Decoder
Our decoder is similar to the conventional Transformer decoder. Since visual information has been incorporated into all textual nodes via multiple graph-based multi-modal fusion layers, we allow the decoder to dynamically exploit the multi-modal context by only attending to textual node states.
As shown in the right part of Figure 2, we follow Vaswani et al. (2017) to stack  $L_{d}$  identical layers to generate target-side hidden states, where each layer  $l$  is composed of three sub-layers. Concretely, the first two sub-layers are a masked self-attention and an encoder-decoder attention to integrate target-and source-side contexts respectively:
$$
\mathbf {E} ^ {(l)} = \operatorname {M u l t i H e a d} \left(\mathbf {S} ^ {(l - 1)}, \mathbf {S} ^ {(l - 1)}, \mathbf {S} ^ {(l - 1)}\right), \tag {9}
$$
$$
\mathbf {T} ^ {(l)} = \operatorname {M u l t i H e a d} \left(\mathbf {E} ^ {(l)}, \mathbf {H} _ {x} ^ {\left(L _ {e}\right)}, \mathbf {H} _ {x} ^ {\left(L _ {e}\right)}\right), \tag {10}
$$
where  $\mathbf{S}^{(l - 1)}$  denotes the target-side hidden states in the  $l$ -th layer. In particular,  $\mathbf{S}^{(0)}$  are the embeddings of input target words. Then, a position-wise fully-connected forward neural network is uesd to produce  $\mathbf{S}^{(l)}$  as follows:
$$
\mathbf {S} ^ {(l)} = \operatorname {F F N} \left(\mathbf {T} ^ {(l)}\right). \tag {11}
$$
Finally, the probability distribution of generating the target sentence is defined by using a softmax layer, which takes the hidden states in the top layer as input:
$$
P (Y | X, I) = \prod_ {t} \operatorname {S o f t m a x} \left(\mathbf {W S} _ {t} ^ {(L _ {d})} + b\right), \tag {12}
$$
where  $X$  is the input sentence,  $I$  is the input image,  $Y$  is the target sentence, and  $\mathbf{W}$  and  $b$  are the parameters of the softmax layer.
We carry out experiments on multi-modal English  $\Rightarrow$  German (En  $\Rightarrow$  De) and English  $\Rightarrow$  French (En  $\Rightarrow$  Fr) translation tasks.
<sup>1</sup>For simplicity, we omit the descriptions of layer normalization and residual connection.
Datasets We use the Multi30K dataset (Elliott et al., 2016), where each image is paired with one English description and human translations into German and French. Training, validation and test sets contain 29,000, 1,014 and 1,000 instances respectively. In addition, we evaluate various models on the WMT17 test set and the ambiguous MSCOCO test set, which contain 1,000 and 461 instances respectively. Here, we directly use the preprocessed sentences  ${}^{2}$  and segment words into subwords via byte pair encoding (Sennrich et al., 2016) with 10,000 merge operations.
Visual Features We first apply the Stanford parser to identify noun phrases from each source sentence, and then employ the visual ground toolkit released by Yang et al. (2019) to detect associated visual objects of the identified noun phrases. For each phrase, we keep the visual object with the highest prediction probability, so as to reduce negative effects of abundant visual objects. In each sentence, the average numbers of objects and words are around 3.5 and 15.0 respectively. Finally, we compute 2,048-dimensional features for these objects with the pre-trained ResNet-100 Faster-RCNN (Ren et al., 2015).
Settings We use Transformer (Vaswani et al., 2017) as our baseline. Since the size of training corpus is small and the trained model tends to be over-fitting, we first perform a small grid search to obtain a set of hyper-parameters on the  $\mathrm{En} \Rightarrow \mathrm{De}$  validation set. Specifically, the word embedding dimension and hidden size are 128 and 256 respectively. The decoder has  $L_{d} = 4$  layers $^{4}$  and the number of attention heads is 4. The dropout is set to 0.5. Each batch consists of approximately 2,000 source and target tokens. We apply the Adam optimizer with a scheduled learning rate to optimize various models, and we use other same settings as (Vaswani et al., 2017). Finally, we use the metrics BLEU (Papineni et al., 2002) and METEOR (Denkowski and Lavie, 2014) to evaluate the quality of translations. Particularly, we run all models three times for each experiment and report the average results.
Figure 3: Results on the  $\mathrm{En} \Rightarrow \mathrm{De}$  validation set regarding the number  $L_{e}$  of graph-based multi-modal fusion layers.
Baseline Models In addition to the text-based Transformer (Vaswani et al., 2017), we adapt several effective approaches to Transformer using our visual features, and compare our model with them<sup>5</sup>:
- ObjectAsToken(TF) (Huang et al., 2016). It is a variant of the Transformer, where all visual objects are regarded as extra source tokens and placed at the front of the input sentence.
- Enc-att(TF) (Delbrouck and Dupont, 2017b). An encoder-based image attention mechanism is incorporated into Transformer, which augments each source annotation with an attention-based visual feature vector.
- Doubly-att(TF) (Helcl et al., 2018). It is a doubly attentive Transformer. In each decoder layer, a cross-modal multi-head attention sublayer is inserted before the fully connected feed-forward layer to generate the visual context vector from visual features.
We also display the performance of several dominant multi-modal NMT models such as Doubly-att(RNN) (Calixto et al., 2017), Soft-att(RNN) (Delbrouck and Dupont, 2017a), Stochastic-att(RNN) (Delbrouck and Dupont, 2017a), Fusion-conv(RNN) (Caglayan et al., 2017), Trg-mul(RNN) (Caglayan et al., 2017), VMMT(RNN) (Calixto et al., 2019) and Deliberation Network(TF) (Ive et al., 2019) on the same datasets.
3.2 Effect of Graph-based Multi-modal Fusion Layer Number  $L_{e}$
The number  $L_{e}$  of multi-modal fusion layer is an important hyper-parameter that directly determines
5We use suffixes "(RNN)" and "(TF)" to represent RNN- and Transformer-style NMT models, respectively.
<sup>2</sup>http://www.statmt.org/wmt18/multimodal-task.html
3There is no parsing failure for this dataset. If no noun is detected for a sentence, the object representations will be set to zero vectors and the model will degenerate to Transformer.
4The encoder of the text-based Transformer also has 4 layers.
Model	En→De
Test2016	Test2017	MSCOCO
BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
Existing Multi-modal NMT Systems
Doubly-att(RNN) (Calixto et al., 2017)	36.5	55.0	-	-	-	-
Soft-att(RNN) (Delbrouck and Dupont, 2017a)	37.6	55.3	-	-	-	-
Stochastic-att(RNN) (Delbrouck and Dupont, 2017a)	38.2	55.4	-	-	-	-
Fusion-conv(RNN) (Caglayan et al., 2017)	37.0	57.0	29.8	51.2	25.1	46.0
Trg-mul(RNN)(Caglayan et al., 2017)	37.8	57.7	30.7	52.2	26.4	47.4
VMMT(RNN) (Calixto et al., 2019)	37.7	56.0	30.1	49.9	25.5	44.8
Deliberation Network(TF) (Ive et al., 2019)	38.0	55.6	-	-	-	-
Our Multi-modal NMT Systems
Transformer (Vaswani et al., 2017)	38.4	56.5	30.6	50.4	27.3	46.2
ObjectAsToken(TF) (Huang et al., 2016)	39.0	57.2	31.7	51.3	28.4	47.0
Enc-att(TF) (Delbrouck and Dupont, 2017b)	38.7	56.6	31.3	50.6	28.0	46.6
Doubly-att(TF) (Helcl et al., 2018)	38.8	56.8	31.4	50.5	27.4	46.5
Our model	39.8	57.6	32.2	51.9	28.7	47.6
Table 1: Experimental results on the  $\mathrm{{En}} \Rightarrow  \mathrm{{De}}$  translation task.
the degree of fine-grained semantic fusion in our encoder. Thus, we first inspect its impact on the EN  $\Rightarrow$  DE validation set.
Figure 3 provides the experimental results using different  $L_{e}$  and our model achieves the best performance when  $L_{e}$  is 3. Hence, we use  $L_{e} = 3$  in all subsequent experiments.
3.3 Results on the  $\mathbf{E}\mathbf{n}\Rightarrow \mathbf{D}\mathbf{e}$  Translation Task
Table 1 shows the main results on the  $\mathrm{En}\Rightarrow \mathrm{De}$  translation task. Ours outperforms most of the existing models and all baselines, and is comparable to Fusion-conv(RNN) and Trg-mul(RNN) on METEOR. The two results are from the state-of-the-ar system on the WMT2017 test set, which is selected based on METEOR. Comparing the baseline models, we draw the following interesting conclusions
First, our model outperforms ObjectAsToken(TF), which concatenates regional visual features with text to form attendable sequences and employs self-attention mechanism to conduct intermodal fusion. The underlying reasons consist of two aspects: explicitly modeling semantic correspondences between semantic units of different modalities, and distinguishing model parameters for different modalities.
Second, our model also significantly outperforms Enc-att(TF). Note that Enc-att(TF) can be considered as a single-layer semantic fusion encoder. In addition to the advantage of explicitly modeling semantic correspondences, we conjecture that multi-layer multi-modal semantic interactions are also beneficial to NMT.
Third, compared with Doubly-att(TF) simply using an attention mechanism to exploit visual in
Figure 4: BLEU scores on different translation groups divided according to source sentence lengths.
Figure 5: BLEU scores on different translation groups divided according to source phrase numbers.
formation, our model achieves a significant improvement, because of sufficient multi-modal fusion in our encoder.
Besides, we divide our test sets into different groups based on the lengths of source sentences and the numbers of noun phrases, and then compare the performance of different models in each group. Figures 4 and 5 report the BLEU scores on these groups. Overall, our model still consistently achieves the best performance in all groups. Thus, we confirm again the effectiveness and gen
Model	En→De
Test2016	Test2017	MSCOCO
BLEU	METEOR	BLEU	METEOR	BLEU	METEOR
Our model	39.8	57.6	32.2	51.9	28.7	47.6
w/o inter-modal fusion	38.7	56.7	30.7	50.6	27.0	46.7
visual grounding ⇒ fully-connected	36.4	53.4	28.3	47.0	24.4	42.9
different parameters ⇒ unified parameters	39.2	57.3	31.9	51.4	27.7	47.4
w/ attending to visual nodes	39.6	57.3	32.0	51.3	27.9	46.8
attending to textual nodes ⇒ attending to visual nodes	30.9	48.6	22.3	41.5	20.4	38.7