SlowGuess/ACL_2020_OCR huggingface dataset:
Zhan Shi*, Xu Zhou†, Xipeng Qiu†, Xiaodan Zhu*
*Ingenuity Labs Research Institute, Queen's University
*Department of Electrical and Computer Engineering, Queen's University
†School of Computer Science, Fudan University
{z.shi,xiaodan.zhu}@queensu.ca,{16210240095,xpqiu}@fudan.edu.cn
Image captioning is a multimodal problem that has drawn extensive attention in both the natural language processing and computer vision community. In this paper, we present a novel image captioning architecture to better explore semantics available in captions and leverage that to enhance both image representation and caption generation. Our models first construct caption-guided visual relationship graphs that introduce beneficial inductive bias using weakly supervised multi-instance learning. The representation is then enhanced with neighbouring and contextual nodes with their textual and visual features. During generation, the model further incorporates visual relationships using multi-task learning for jointly predicting word and object/predicate tag sequences. We perform extensive experiments on the MSCOCO dataset, showing that the proposed framework significantly outperforms the baselines, resulting in the state-of-the-art performance under a wide range of evaluation metrics. The code of our paper has been made publicly available.
Automatically generating a short description for a given image, a problem known as image captioning (Chen et al., 2015), has drawn extensive attention in both the natural language processing and computer vision community. Inspired by the success of encoder-decoder frameworks with the attention mechanism, previous efforts on image captioning adopt variants of pre-trained convolution neural networks (CNN) as the image encoder and recurrent neural networks (RNN) with visual attention as the decoder (Lu et al., 2017; Anderson et al., 2018; Xu et al., 2015; Lu et al., 2018).
Many previous methods translate image representation into natural language sentences without
Figure 1: Visual relationship graphs from a pre-trained detection model (Yao et al., 2018) (upper) and from the ground-truth caption (bottom).
explicitly investigating semantic cues from texts and images. To remedy that, some research has also explored to detect high-level semantic concepts presented in images to improve caption generation (Wu et al., 2016; Gan et al., 2017; You et al., 2016; Fang et al., 2015; Yao et al., 2017). It is believed by many that the inductive bias that leverages structured combination of concepts and visual relationships is of importance, which has led to better captioning models (Yao et al., 2018; Guo et al., 2019; Yang et al., 2019). These approaches obtain visual relationship graphs using models pretrained from visual relationship detection (VRD) datasets, e.g., Visual Genome (Krishna et al., 2017), where the visual relationships capture semantics between pairs of localized objects connected by predicates, including spatial (e.g., cake-on-desk) and non-spatial semantic relationships (e.g., man-eat-food) (Lu et al., 2016).
As in many other joint text-image modeling problems, it is crucial to obtain a good semantic representation in image captioning that bridges semantics in language and images. The existing approaches, however, have not yet adequately leveraged the semantics available in captions to con
$^{1}$  https://github.com/Gitsamshi/WeakVRD-Captioning
7454
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7454-7464 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
struct image representation and generate captions. As shown in Figure 1, although VRD detection models present a strong capacity in predicting salient objects and the most common predicates, they often ignore predicates vital for captioning (e.g., "grab" in this example). Exploring better models would still be highly desirable.
A major challenge for establishing a structural connection between captions and images is that the links between predicates and the corresponding object regions are often ambiguous: within the "image-level" label  $(obj_{1}, pred, obj_{2})$  extracted from captions, there may exist multiple object regions corresponding to  $obj_{1}$  and  $obj_{2}$ . In this paper, we propose to use weakly supervised multi-instance learning to detect if a bag of object (region) pairs in an image contain certain predicates, e.g., predicates appearing in ground-truth captions here (or in other applications, they can be any given predicates under concerns). Based on that we can construct caption-guided visual relationship graphs.
Once the visual relationship graphs (VRG) are built, we propose to adapt graph convolution operations (Marcheggiani and Titov, 2017) to obtain representation for object nodes and predicate nodes. These nodes can be viewed as image representation units used for generation.
During generation, we further incorporate visual relationships—we propose multi-task learning for jointly predicting word and tag sequences, where each word in a caption could be assigned with a tag, i.e., object, predicate, or none, which takes as input the graph node features from the above visual relationship graphs. The motivation for predicting a tag in each step is to regularize which types of information should be taken into more consideration for generating words: predicate nodes features, object nodes features, or the current state of language decoder. We study different types of multi-task blocks in our models.
As a result, our models consist of three major components: constructing caption-guided visual relationship graphs (CGVRG) with weakly-supervised multi-instance learning, building context-aware CGVRG, and performing multi-task generation to regularize the network to take into account explicit predicate object/predicate constraints. We perform extensive experiments on the MSCOCO (Lin et al., 2014) image captioning dataset with both supervised and Reinforcement
learning strategy (Rennie et al., 2017). The experiment results show that the proposed models significantly outperform the baselines and achieve the state-of-the-art performance under a wide range of evaluation metrics. The main contributions of our work are summarized as follows:
- We propose to construct caption-guided visual relationship graphs that introduce beneficial inductive bias by better bridging captions and images. The representation is further enhanced with neighbouring and contextual nodes with their textual and visual features.
- Unlike existing models, we propose multi-task learning to regularize the network to take into account explicit object/predicate constraints in the process of generation.
- The proposed framework achieves the state-of-the-art performance on the MSCOCO image captioning dataset. We provide detailed analyses on how this is attained.
Image Captioning A prevalent paradigm of existing image captioning methods is based on the encoder-decoder framework which often utilizes a CNN-plus-RNN architecture for image encoding and text generation (Donahue et al., 2015; Vinyals et al., 2015; Karpathy and Fei-Fei, 2015). Soft or hard visual attention mechanism (Xu et al., 2015; Chen et al., 2017) has been incorporated to focus on the most relevant regions in each generation step. Furthermore, adaptive attention (Lu et al., 2017) has been developed to decide whether to rely on visual features or language model states in each decoding step. Recently, bottom-up attention techniques (Anderson et al., 2018; Lu et al., 2018) have also been proposed to find the most relevant regions based on bounding boxes.
There has been increasing work focusing on filling the gap between image representation and caption generation. Semantic concepts and attributes detected from images have been demonstrated to be effective in boosting image captioning when used in the encoder-decoder frameworks (Wu et al., 2016; You et al., 2016; Gan et al., 2017; Yao et al., 2017). Visual relationship (Lu et al., 2016) and scene graphs (Johnson et al., 2015) have been further employed for image encoder in a unimodal (Yao et al., 2018) or multi-modal (Yang et al., 2019; Guo et al., 2019) manner to improve the over
7455
Figure 2: An overview of the proposed image captioning framework.
all performance via the graph convolutional mechanism (Marcheggiani and Titov, 2017). Besides, Kim et al. (2019) proposes a relationship-based captioning task to lead better understanding of images based on relationship. As discussed in introduction, we will further explore the relational semantics available in captions for both constructing image representation and generating caption.
Visual Relationship Detection Visual relations between objects in an image have attracted more studies recently. Conventional visual relation detection have dealt with  $\langle$  subject-predicate-object  $\rangle$  triples, including spatial relation and other semantic relation. Lu et al. (2016) detect the triples by performing subject, object, and predicate classification separately. Li et al. (2017) attempt to encode more distinguishable visual features for visual relationships detection. Probabilistic output of object detection (Dai et al., 2017; Zhang et al., 2017) is also considered to reason about the visual relationships.
3 The Models
Given an image  $\pmb{I}$ , the goal of image captioning is to generate a visually grounded natural language sentence. We learn our model by minimizing the cross-entropy loss with regard to the ground truth caption  $S^{*} = \{w_{1}^{*}, w_{2}^{*}, \dots, w_{T}^{*}\}$ :
$$
\begin{array}{l} L _ {X E} = - \log p \left(\boldsymbol {S} ^ {*} \mid \boldsymbol {I}\right) (1) \\ = - \sum_ {t = 1} ^ {T} \log p \left(w _ {t} ^ {*} \mid w _ {<   t} ^ {*}, I\right) (2) \\ \end{array}
$$
The model is further tuned with a Reinforcement Learning (RL) objective (Rennie et al., 2017) to maximize the reward of the generated sentence  $S$ :
$$
J _ {R L} = E _ {\boldsymbol {S} \sim p (\boldsymbol {S} | \boldsymbol {I})} \left(d \left(\boldsymbol {S}, \boldsymbol {S} ^ {*}\right)\right) \tag {3}
$$
where  $d$  is a sentence-level scoring metric.
An overview of our image captioning framework is depicted in Figure 2, with the detail of the components described in the following sections.
3.1 Caption-Guided Visual Relationship Graph (CGVRG) with Weakly Supervised Learning
A general challenge of modeling  $p(S|I)$  is obtaining a better semantic representation in the multimodal setting to bridge captions and images. Our framework first focuses on constructing caption-guided visual relationship graphs (CGVRG).
3.1.1 Extracting Visual Relationship Triples and Detecting Objects
The process of constructing CGVRG first extracts relationship triples from captions using textual scene graph parser as described in (Schuster et al., 2015). Our framework employs Faster R-CNN (Ren et al., 2015) to recognize instances of objects and returns a set of image regions for objects:  $V = \{v_{1}, v_{2}, \dots, v_{n}\}$ .
3.1.2 Constructing CGVRG
The main focus of CGVRG is constructing visual relationship graphs. As discussed in introduction, the existing approaches use pre-trained VRD (visual relationship detection) models, which often ignore key relationships needed for captioning. This gap can be even more prominent if the domain/data used to train image-captioning is farther from where VRD is pretrained. A major challenge to use predicate triples from captions to construct CGVRG is that, the links between predicates and the corresponding object regions are often ambiguous as discussed in introduction. To solve this problem, we use weakly supervised, multi-instance learning.
7456
Obtaining Representation for Object Region Pairs For an image  $I$  with a list of salient object regions obtained in object detection  $\{v_{1}, v_{2}, \dots, v_{n}\}$ , we have a set of region pairs  $U = \{\mathbf{u}_{1}, \mathbf{u}_{2}, \dots, \mathbf{u}_{N}\}$ , where  $N = n(n - 1)$ . As shown in Figure 3(b), the visual features of any two object regions and their union box will be collected to compute  $p_{\mathbf{u}_n}^{r_j}$ , the probability that a region pair  $\mathbf{u}_n$  is associated with the predicate  $r_j$ , where  $r_j \in R$  and  $R = \{r_1, r_2, \dots, r_M\}$  include frequent predicates obtained from the captions in training data. The feed-forward network of Figure 3(b) will be trained in weakly supervised training.
Weakly Supervised Multi-Instance Training As shown in Figure 3(c), during training, one object pair  $t = (o_1, o_2)$ , e.g., (women, hat), can correspond to multiple pairs of object regions: the four women-hat combinations between the two women and two hats. To make our description clearer, we refer to  $t = (o_1, o_2)$  as an object pair, and the four women-hat pairs in the image as object region pairs. Accordingly, for a triple we extracted  $t = (o_1, r, o_2)$ ,  $r \in R$ , e.g., (woman, in, hat), the predicate  $r$  (i.e., in) can be associated with multiple object region pairs (here,  $(w0, h0)$ ,  $(w0, h1)$ ,  $(w1, h0)$ , and  $(w1, h1)$ ).
To predict predicates over object region pairs, we propose to use Multi-Instance Learning (Fang et al., 2015) as our weakly supervised learning approach. Multi-Instance Learning receives a set of labeled bags, each bag containing a set of instances. A bag would be labeled negative if all the instances in it are negative. On the other hand, a bag is labeled positive if there is at least one positive instance in the bag.
In our problem, an instance is a region pair. Therefore for a candidate predicate  $r \in R$  (e.g., in), we use  $\mathcal{N}_r$  to denote the object region pairs corresponding to predicate  $r$ . If  $r$  appears in the caption  $S$ ,  $\mathcal{N}_r$  would be a positive bag. We use  $\mathcal{N} \setminus \mathcal{N}_r$  to denote the negative bag for  $r$ . When  $r$  is not contained in the caption, the entire  $\mathcal{N}$  would be the negative bag (the last row of Figure 3(c)). The probability of a bag  $b$  having the predicate  $r_j$  is measured with "noisy-OR":
$$
p _ {b} ^ {r _ {j}} = 1 - \prod_ {n \in b} \left(1 - p _ {\mathbf {u} _ {n}} ^ {r _ {j}}\right) \tag {4}
$$
where  $p_{\mathbf{u}_n}^{r_j}$  has been introduced above. We adopt the cross-entropy loss on the basis of all predicate
(b)
(c)
Figure 3: Subcomponents in constructing CGVRG: (a) detecting objects and extracting triples; (b) obtaining representation for object region pairs; (c) examples of positive and negative bags in multi-instance learning for predicate "in" and "feed", respectively. Here,  $w$ ,  $h$ , and  $g$  denote woman, hat, and giraffe, respectively.
(w0,h0)	(w0,h1)	(w1,h0)	(w1,h1)	...	(w0,g0)	(w0,g1)	(w0,g2)	(w1,g0)	(w1,g1)	(w1,g2)
in	Positive Bag	Negative Bag
feed	Negative Bag	Positive Bag
--	Negative Bag
probabilities over bags, given an image  $\pmb{I}$  and caption  $S$ :
$$
\begin{array}{l} L (\boldsymbol {I}) = - \sum_ {j = 1} ^ {M} \left[ \mathbb {1} _ {\left(r _ {j} \in \boldsymbol {S}\right)} \left(\log p _ {\mathcal {N} _ {r _ {j}}} ^ {r _ {j}} + \log \left(1 - p _ {\mathcal {N} \backslash \mathcal {N} _ {r _ {j}} ^ {r _ {j}}}\right)\right) \right. \\ \left. + \mathbb {1} _ {(r _ {j} \notin S)} \left(\log \left(1 - p _ {\mathcal {N}} ^ {r _ {j}}\right)\right) \right] \tag {5} \\ \end{array}
$$
where the indicator function  $\mathbb{1}_{condition} = 1$  if the condition is true, otherwise  $\mathbb{1}_{condition} = 0$ .
Constructing the Graphs Once obtaining the trained module, we can build a CGVRG graph  $\mathcal{G} = (\mathcal{V},\mathcal{E})$  for a given image  $I$ , where the node set  $\mathcal{V}$  includes two types of nodes: object nodes and predicate nodes. We denote  $o_i$  as the  $i^{th}$  object node and  $r_{ij}$  as a predicate node that connects  $o_i$  and  $o_j$  (refer to Figure 1 or the middle part of Figure 2). The edges in  $\mathcal{E}$  are added based on triples; i.e.,  $(o_i,r_{ij},o_j)$  will assign two directed edges from node  $o_i$  to  $r_{ij}$  and from  $r_{ij}$  to  $o_j$ , respectively.
Note that due to the use of the proposed weakly supervised models, the acquired graphs can now contain predicates that exist in captions but not in the VRD models used in the previous work that does not explicitly consider predicates in captions. We will show in our experiments that this improves captioning quality.
3.2 Context-Aware CGVRG
We further enhance CGVRG in the context of both modalities, images and text, using graph convolution networks. We first integrate visual and textual features: the textual features for each node are
from a word embedding and the visual features are regional visual representations extracted via RoI pooling from Faster R-CNN. The specific features  $\mathbf{g}_{o_i},\mathbf{g}_{r_{ij}}$  for object  $o_i$  and predicate  $r_{ij}$  are shown as follows:
$$
\mathbf {g} _ {o _ {i}} = \phi_ {o} ([ \mathbf {g} _ {o _ {i}} ^ {t}; \mathbf {g} _ {o _ {i}} ^ {v} ]) \tag {6}
$$
$$
\mathbf {g} _ {r _ {i j}} = \phi_ {r} \left(\mathbf {g} _ {r _ {i j}} ^ {t}\right) \tag {7}
$$
where  $\phi_r$  and  $\phi_o$  are feed-forward networks using ReLU activation;  $\mathbf{g}_{o_i}^t$ ,  $\mathbf{g}_{r_{ij}}^t$ , and  $\mathbf{g}_{o_i}^v$  denote textual features of  $o_i$ ,  $r_{ij}$  and visual features of  $o_i$ , respectively.
We present the process of encoding  $\mathcal{G}$  to produce a new set of context-aware representation  $\mathcal{X}$ . The representation of predicate  $r_{ij}$  and  $o_i$  are computed as follows:
$$
\mathbf {x} _ {r _ {i j}} = f _ {r} ([ \mathbf {g} _ {o _ {i}}; \mathbf {g} _ {o _ {j}}; \mathbf {g} _ {r _ {i j}} ]) \tag {8}
$$
$$
\begin{array}{l} \mathbf {x} _ {o _ {i}} = \frac {1}{N _ {i}} \left[ \sum_ {r \in \mathcal {N} _ {\text {o u t}} (o _ {i})} f _ {\text {o u t}} ([ \mathbf {g} _ {o _ {i}}; \mathbf {g} _ {r} ]) \right. \tag {9} \\ + \sum_ {r \in \mathcal {N} _ {i n} (o _ {i})} f _ {i n} ([ \mathbf {g} _ {o _ {i}}; \mathbf {g} _ {r} ]) \Bigg ] \\ \end{array}
$$
where  $f_{r}, f_{in}, f_{out}$  are feed-forward networks using ReLU activation.  $\mathcal{N}_{in}$  and  $\mathcal{N}_{out}$  denote the adjacent nodes with  $o_i$  as head and tail, respectively.  $N_{i}$  is the total number of adjacent nodes.
3.3 Multi-task Caption Generation
Unlike the existing image-captioning models, we further incorporate visual relationships into generation — we propose multi-task learning for jointly predicting word and tag sequences as each word in a caption will be assigned a tag, i.e., object, predicate, or none. The module takes as input the graph node features from the context-aware CGVRG. The output of the generation module is hence the sequence of words  $\mathbf{y} = \{y_1,\dots ,y_T\}$  as well as the tags  $\mathbf{z} = \{z_{1},\dots ,z_{T}\}$ . Two different approaches are leveraged to train the two tasks jointly.
The bottom LSTM is used to align a textual state to graph node representations:
$$
\mathbf {h} _ {t} ^ {1} = \operatorname {L S T M} \left(\mathbf {h} _ {t - 1} ^ {1}, \left[ \mathbf {h} _ {t - 1} ^ {2}; \overline {{\mathbf {x}}}; \mathbf {e} _ {w _ {t}} \right]\right) \tag {10}
$$
where LSTM means one step of recurrent unit computation via LSTM;  $\overline{\mathbf{x}}$  is the mean-pooled representation of all nodes in the graph;  $\mathbf{h}_{t - 1}^{1}$  and  $\mathbf{h}_{t - 1}^{2}$
denote hidden states of bottom and top LSTM in time step  $t - 1$ , respectively;  $\mathbf{e}$  is the word embedding table.
The state  $h_t^1$  is then used as a query to attend over graph node features  $\{\mathbf{x}_o\}$  and  $\{\mathbf{x}_r\}$  separately to get attended features  $\hat{\mathbf{x}}_t^T$  and  $\hat{\mathbf{x}}_t^o$ :
$$
\hat {\mathbf {x}} _ {t} ^ {r} = \operatorname {A T T} \left(\mathbf {h} _ {t} ^ {1}, \left\{\mathbf {x} _ {r} \right\}\right) \tag {11}
$$
$$
\hat {\mathbf {x}} _ {t} ^ {o} = \operatorname {A T T} \left(\mathbf {h} _ {t} ^ {1}, \left\{\mathbf {x} _ {o} \right\}\right) \tag {12}
$$
where ATT is a soft-attention operation between a query and graph node features.
The top LSTM works as a language model decoder, in which the hidden state  $\mathbf{h}_0^2$  is initialized with the mean-pooled semantic representation of all detected predicates  $\{r\}$ . In time step  $t$ , the input consists of the output from the bottom LSTM layer  $h_t^1$  and attended graph features  $\hat{\mathbf{x}}_t^r$ ,  $\hat{\mathbf{x}}_t^o$ :
$$
\mathbf {h} _ {t} ^ {2} = \operatorname {L S T M} \left(\mathbf {h} _ {t - 1} ^ {2}, \left[ \mathbf {h} _ {t} ^ {1}; \hat {\mathbf {x}} _ {t} ^ {o}; \hat {\mathbf {x}} _ {t} ^ {r} \right]\right) \tag {13}
$$
3.3.1 Multi-task Learning
We propose two different blocks to perform the two tasks jointly, as shown in Figure 4. In each step, a multi-task learning block deals with task  $s_1$  as predicting a tag  $z_t$  and task  $s_2$  as predicting a word  $y_t$ . Specifically MT-I treats the two tasks independent of each other:
$$
p \left(z _ {t} \mid y _ {<   t}, I\right) = \operatorname {s o f t m a x} \left(f _ {z} \left(\mathbf {h} _ {t} ^ {2}\right)\right) \tag {14}
$$
$$
p \left(y _ {t} \mid y _ {<   t}, I\right) = \operatorname {s o f t m a x} \left(f _ {y} \left(\mathbf {h} _ {t} ^ {2}\right)\right) \tag {15}
$$
where  $f_{z}$  and  $f_{y}$  are feed-forward networks with ReLU activation. Inspired by the adaptive attention mechanism (Lu et al., 2017), MT-II further exploits the probability from  $p(z_{t}|y_{< t},\boldsymbol{I})$  to integrate the representation of current hidden state  $h_t^2$  and attended features from graph  $\hat{\mathbf{x}}_t^r$ ,  $\hat{\mathbf{x}}_t^o$ :
$$
p \left(y _ {t} \mid y _ {<   t}, \boldsymbol {I}\right) = \operatorname {s o f t m a x} \left(f _ {y} \left(\hat {\mathbf {h}} _ {t} ^ {2}\right)\right), \tag {16}
$$
$$
\hat {\mathbf {h}} _ {t} ^ {2} = \mathbf {h} _ {t} ^ {2} p _ {n a} + \hat {\mathbf {x}} _ {t} ^ {r} p _ {r} + \hat {\mathbf {x}} _ {t} ^ {o} p _ {o} \tag {17}
$$
$$
p \left(z _ {t} \mid y _ {<   t}, I\right) = \operatorname {s o f t m a x} \left(f _ {z} \left(\mathbf {h} _ {t} ^ {2}\right)\right) \tag {18}
$$
where  $p_{na}, p_r, p_o$  denote the probabilities of tag  $z_t$  being "none", "predicate", and "object", respectively. The multi-task loss function is as follows:
$$
L _ {M T} (\boldsymbol {I}) = - \sum_ {t = 1} ^ {T} \log p \left(y _ {t} \mid y _ {<   t}, \boldsymbol {I}\right) + \gamma \log p \left(z _ {t} \mid y _ {<   t}, \boldsymbol {I}\right) \tag {19}
$$
where  $\gamma$  is the hyper-parameter to balance the two tasks.
7458
(b)
Figure 4: An overview of multi-task caption generation module. Subfigure (a) is a two-layer LSTM; Subfigure (b) depicts two different types of multi-task block.
The overall training process can be broken down into two parts: the CGVRG detection module training period and the caption generator training period; the latter includes cross-entropy optimization and the CIDEr-D optimization. For CGVRG detection module training, the detection module is optimized with the multi-instance learning loss in Equation 5. For caption generator training, the model is first optimized with the cross-entropy loss in Equation 19, and then we directly optimize the model with the expected sentence-level reward (CIDEr-D in this work) shown in Equation 3 by self critical sequence learning (Rennie et al., 2017).
In the inference stage, given an image, the CGVRG detection module obtains a graph upon them. The graph convolution network encodes graphs to obtain the context aware multi-modal representations. Then graph object/predicate node features are further provided to the multi-task caption generation module to generate sequences with beam search.
MSCOCO We perform extensive experiments on the MSCOCO benchmark (Lin et al., 2014). The Karpathy split (Karpathy and Fei-Fei, 2015) is adopted for our model selection and offline testing, which contains 113K training images, 5K validation images and 5K testing images. As for the online test server, the result is trained on the entire training and validation set (123K images). To evaluate the generated captions, we employ
standard evaluation metrics: SPICE (Anderson et al., 2016), CIDEr-D (Vedantam et al., 2015), METEOR (Denkowski and Lavie, 2014), ROUGE-L (Lin, 2004), and BLEU (Papineni et al., 2002).