SlowGuess/ACL_2020_OCR huggingface dataset:
Qingbao Huang $^{1,2}$ , Jielong Wei $^{2}$ , Yi Cai $^{1*}$
Changmeng Zheng<sup>1</sup>, Junying Chen<sup>1</sup>, Ho-fung Leung<sup>3</sup>, Qing Li<sup>4</sup>
$^{1}$ School of Software Engineering, South China University of Technology, Guangzhou, China
$^{2}$ School of Electrical Engineering, Guangxi University, Nanning, Guangxi, China
<sup>3</sup>The Chinese University of Hong Kong, Hong Kong SAR, China
<sup>4</sup>The Hong Kong Polytechnic University, Hung Hom, Kowloon, Hong Kong, China qbhuang@gxu.edu.cn, 1712306010@st.gxu.edu.cn, ycai@scut.edu.cn
Visual question answering aims to answer the natural language question about a given image. Existing graph-based methods only focus on the relations between objects in an image and neglect the importance of the syntactic dependency relations between words in a question. To simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question, we propose a novel dual channel graph convolutional network (DC-GCN) for better combining visual and textual advantages. The DC-GCN model consists of three parts: an I-GCN module to capture the relations between objects in an image, a Q-GCN module to capture the syntactic dependency relations between words in a question, and an attention alignment module to align image representations and question representations. Experimental results show that our model achieves comparable performance with the state-of-the-art approaches.
As a form of visual Turing test, visual question answering (VQA) has drawn much attention. The goal of VQA (Antol et al., 2015; Goyal et al., 2017) is to answer a natural language question related to the contents of a given image. Attention mechanisms are served as the backbone of the previous mainstream approaches (Lu et al., 2016; Yang et al., 2016; Yu et al., 2017), however, they tend to catch only the most discriminative information, ignoring other rich complementary clues (Liu et al., 2019).
Recent VQA studies have been exploring higher level semantic representation of images, notably using graph-based structures for better image understanding, such as scene graph generation (Xu et al., 2017; Yang et al., 2018), visual relationship detection (Yao et al., 2018), object counting (Zhang et al.,
(a) O: What color is the vampire's shirt
(b) O: What color is the vampire's shirt
Ground True Answer: blue
Predicted Answer: black
(c) Dependency parsing of the question
Figure 1: (a) The question and the ground true answer. (b) The wrong answer is predicted by a state-of-the-art model, which focuses on the highlighted region in the image. The depth of the color indicates the weights of the words in the question, where deeper color represents higher weight. The question is performed by syntactic dependency parsing. (c) The dependency parsing of the question is obtained by the universal Standford Dependencies tool (De Marneffe et al., 2014).
2018a), and relation reasoning (Cao et al., 2018; Li et al., 2019; Cadene et al., 2019a). Representing images as graphs allows one to explicitly model interactions between two objects in an image, so as to seamlessly transfer information between graph nodes (e.g., objects in an image).
Very recent research methods (Li et al., 2019; Cadene et al., 2019a; Yu et al., 2019) have achieved remarkable performances, but there is still a big gap between them and human. As shown in Figure 1(a), given an image of a group of persons and the corresponding question, a VQA system needs to not only recognize the objects in an image (e.g., batter,UMPire and catcher), but also grasp the textual information in the question "what color is theUMPire's shirt". However, even many competitive VQA models struggle to process them accurately, and as a result predict the incorrect answer (black) rather than the correct answer (blue), including the
* Corresponding author: Yi Cai (ycai@scut.edu.cn).
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7166-7176 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
state-of-the-art methods.
Although the relations between two objects in an image have been considered, the attention-based VQA models lack building blocks to explicitly capture the syntactic dependency relations between words in a question. As shown in Figure 1(c), these dependency relations can reflect which object is being asked (e.g., the word *umpire*'s modifies the word *shirt*) and which aspect of the object is being asked (e.g., the word *color* is the direct object of the word *is*). If a VQA model only knows the word *shirt* rather than the relation between words *umpire*'s and *shirt* in a question, it is difficult to distinguish which object is being asked. In fact, we do need the modified relations to discriminate the correct object from multiple similar objects. Therefore, we consider that it is necessary to explore the relations between words at linguistic level in addition to constructing the relations between objects at visual level.
Motivated by this, we propose a dual channel graph convolutional network (DC-GCN) to simultaneously capture the relations between objects in an image and the syntactic dependency relations between words in a question. Our proposed DC-GCN model consists of an Image-GCN (I-GCN) module, a Question GCN (Q-GCN) module, and an attention alignment module. The I-GCN module captures the relations between objects in an image, the Q-GCN module captures the syntactic dependency relations between words in a question, and the attention alignment module is used to align two representations of image and question. The contributions of this work are summarized as follows:
1) We propose a dual channel graph convolutional network (DC-GCN) to simultaneously capture the visual and textual relations, and design the attention alignment module to align the multimodal representations, thus reducing the semantic gaps between vision and language.
2) We explore how to construct the syntactic dependency relations between words at linguistic level via graph convolutional networks as well as the relations between objects at visual level.
3) We conduct extensive experiments and ablation studies on VQA-v2 and VQA-CP-v2 datasets to examine the effectiveness of our DC-GCN model. Experimental results show that the DC-GCN model achieves competitive performance with the state-of-the-art approaches.
Visual Question Answering Attention mechanism has been proven effective on many tasks, such as machine translation (Bahdanau et al., 2014) and image captioning (Pedersoli et al., 2017). A number of methods have been developed so far, in which question-guided attention on image regions is commonly used. These can be categorized into two classes according to the types of employed image features. One class uses visual features from some region proposals, which are generated by Region Proposal Network (Ren et al., 2015). The other class uses convolutional features (i.e., activations of convolutional layers).
To learn a better representation of the question, the Stacked Attention Network (Yang et al., 2016) which can search question-related image regions is designed by performing multi-step visual attention operations. A co-attention mechanism that jointly performs question-guided visual attention and image-guided question attention is proposed to solve the problems of which regions to look at and what words to listen to (Shih et al., 2016). To obtain more fine-grained interaction between image and question, some researchers introduce rather sophisticated fusion strategies. Bilinear pooling method (Kim et al., 2018; Yu et al., 2017, 2018) is one of the pioneering works to efficiently and expressively combine multimodal features by using an outer product of two vectors.
Recently, some researchers devoted to overcome the priors on VQA dataset and proposed the methods like GVQA (Agrawal et al., 2018),  $UpDn + Q-Adv + DoE$  (Ramakrishnan et al., 2018), and RUBi (Cadene et al., 2019b) to solve the language biases on the VQA-CP-v2 dataset.
Graph Networks Graph networks are powerful models that can perform relational inferences through message passing. The core idea is to enable communication between image regions to build contextualized representations of these regions. Below we review some of the recent works that rely on graph networks and other contextualized representations for VQA.
Recent research works (Cadene et al., 2019a; Li et al., 2019) focus on how to deal with complex scene and relation reasoning to obtain better image representations. Based on multimodal attentional networks, (Cadene et al., 2019a) introduces an atomic reasoning primitive to represent interactions between question and image region by a rich vecto
7167
Figure 2: Illustration of our proposed Dual Channel Graph Convolutional Network (DC-GCN) for VQA task. The Dependency Parsing constructs the semantic relations between words in a question, and Q-GCN Module updates every word's features by aggregating the adjacent word features. In addition, the I-GCN Module builds the relations between image objects, and the Attention Alignment Module use question-guided image attention mechanism to learn a new object representation thus align the images and questions. All punctuations and upper cases have been preprocessed. The numbers in red are the weight scores of image objects and words.
rial representation and model region relations with pairwise combinations. GCNs, which can better explore the visual relations between objects and aggregate its own features and neighbors' features, have been applied to various tasks, such as text classification (Yao et al., 2019), relation extraction (Guo et al., 2019; Zhang et al., 2018b), scene graph generation (Yang et al., 2018; Yao et al., 2018).
To answer complicated questions about an image, a relation-aware graph attention network (ReGAT) (Li et al., 2019) is proposed to encode each image into a graph and model multi-type interobject relations via a graph attention mechanism, such as spatial relations, semantic relations and implicit relations. One limitation of ReGAT (Li et al., 2019) lies in the fact that it solely considers the relations between objects in an image while neglect the importance of text information. In contrast, our DC-GCN simultaneously capture visual relations in an image and textual relations in a question.
Similar to (Anderson et al., 2018), we extract the image features by using a pretrained Faster RCNN (Ren et al., 2015). We select  $\mu$  object proposals for each image, where each object proposal is represented by a 2048 dimensional feature vector. The obtained visual region features are denoted as  $h_v = \{h_{vi}\}_{i=0}^{\mu} \in \mathbb{R}^{\mu \times 2048}$ .
To extract the question features, each word is embedded into a 300-dimensional Glove vector
(Pennington et al., 2014). The word embeddings are input into a LSTM (Hochreiter and Schmidhuber, 1997) to encode, which produces the initial question representation  $h_{q} = \{h_{qj}\}_{j=0}^{\lambda} \in \mathbb{R}^{\lambda \times d_{q}}$ .
3.2 Relation Extraction and Encoding
3.2.1 I-GCN Module
Image Fully-connected Relations Graph By treating each object region in an image as a vertex, we can construct a fully-connected undirected graph, as shown in Figure 3(b). Each edge represents a relation between two object regions.
Pruned Image Graph with Spatial Relations S- spatial relations represent an object position in an image, which correspond to a 4-dimensional spatial coordinate  $[x_{1},y_{1},x_{2},y_{2}]$  . Note that  $(x_{1},y_{1})$  is the coordinate of the top-left point of the bounding box and  $(x_{2},y_{2})$  is the coordinate of the bottom-right point of the bounding box.
Identifying the correlation between objects is a key step. We calculate the correlation between objects by using spatial relations. The steps are as follows: (1) The features of two nodes are input into multi-layer perceptron respectively, and then the corresponding elements are multiplied to get a relatedness score. (2) The intersection over union of two object regions is calculated. According to the overlapping part of two object regions, different spatial relations are classified into 11 different categories, such as inside, cover, and overlap (Yao et al., 2018). Following the work (Yao et al., 2018), we utilize the overlapping region between
7168
two object regions to judge whether there is an edge between two regions. If two object regions have large overlapping part, it means that there is a strong correlation between these two objects. If two object regions haven't any overlapping part, we consider two objects have a weak correlation, which means there are no edges to connect these two nodes. According to the spatial relations, we prune some irrelevant relations between objects and obtain a sparse graph, as shown in Figure 3(c).
(b)
Figure 3: (a) Generate region proposals by pretrained model (Anderson et al., 2018). For display purposes, we only highlight some object regions. (b) Construct the relations between objects. (c) Prune the irrelevant object edges and calculate the weight between objects. The numbers in red are the weights of edges.
Image Graph Convolutions Following the previous studies (Li et al., 2019; Zhang et al., 2018b; Yang et al., 2018), we use GCN to update the representations of objects. Given a graph with  $\mu$  nodes, each object region in an image is a node. We represent the graph structure with a  $\mu \times \mu$  adjacency matrix  $A$ , where  $A_{ij} = 1$  if there is overlapping region between node  $i$  and node  $j$ ; else  $A_{ij} = 0$ .
Given a target node  $i$  and a neighboring node  $j \in \mathcal{N}(i)$  in an image, where  $\mathcal{N}(i)$  is the set of nodes neighboring with node  $i$ , and the representations of node  $i$  and node  $j$  are  $h_{vi}$  and  $h_{vj}$ , respectively. To obtain the correlation score  $s_{ij}$  between node  $i$  and  $j$ , we learn a fully connected layer over concatenated node features  $h_{vi}$  and  $h_{vj}$ :
$$
s _ {i j} = w _ {a} ^ {T} \sigma (W _ {a} [ h _ {v i} ^ {(l)}, h _ {v j} ^ {(l)} ]), \qquad (1)
$$
where  $w_{a}$  and  $W_{a}$  are learned parameters,  $\sigma$  is the non-linear activation function, and  $[h_{vi}^{(l)}, h_{vj}^{(l)}]$  denotes the concatenation operation. We apply a softmax function over the correlation score  $s_{ij}$  to obtain weight  $\alpha_{ij}$ , as shown in Figure 3(c) where the numbers in red represent the weight scores:
$$
\alpha_ {i j} = \frac {\exp (s _ {i j})}{\sum_ {j \in \mathcal {N} (i)} \exp (s _ {i j})}. \tag {2}
$$
The  $l$ -th layer representations of neighboring nodes  $h_{vj}^{(l)}$  are first transformed via a learned linear transformation  $W_{b}$ . Those transformed representations
are then gathered with weight  $\alpha_{ij}$ , followed by a non-linear function  $\sigma$ . This layer-wise propagation can be denoted as:
$$
h _ {v i} ^ {(l + 1)} = \sigma \left(h _ {v i} ^ {(l)} + \sum_ {j \in \mathcal {N} (i)} A _ {i j} \alpha_ {i j} W _ {b} h _ {v j} ^ {(l)}\right). \tag {3}
$$
Following the stacked  $L$  layer GCN, the output of I-GCN module  $H_{v}$  can be denoted as:
$$
H _ {v} = h _ {v i} ^ {(l + 1)} (l <   L). \tag {4}
$$
3.2.2 Q-GCN Module
In practice, we observe that two words in a sentence usually hold certain relations. Such relations can be identified by the universal Standford Dependencies (De Marneffe et al., 2014). As shown in Table 1, we list a part of commonly-used dependency relations. For example, the sentence what color is
Figure 4: The question is performed by syntactic dependency parsing. The word  $is$  is the root node of dependency relations while the words in blue (e.g., det, dobj) are dependency relations. The direction of arrow indicates that two words exist a relation.
theUMPIRE'sshirtis parsed to obtain the relations between words (e.g.,cop,det and nmod),as shown in Figure 4.The words in blue are the dependency relations.The ending of arrow indicates that this word is a modifier.The word root in purple is used to indicate which word is the root node of dependency relations.
Question Fully-connected Relations Graph By treating each word in a question as a node, we construct a fully-connected undirected graph, as shown in Figure 5(a). Each edge represents a relation between two words.
Pruned Question Graph with Dependency Relations Irrelevant relations between two words may bring noises. Therefore, we need to prune some unrelated relations to reduce the noises. By parsing the dependency relations of a question, we obtain the relations between words (cf. Figure 4). According to dependency relations, we prune some edges between two nodes which do not have dependency relations. A sparse graph is obtained, as shown in Figure 5(b).
Relations	Relation Description
det	determiner
nsubj	nominal subject
case	prepositions, postpositions
nmod	nominal modifier
cop	copula
dobj	direct object
amod	adjective modifier
aux	auxiliary
advmod	adverbial modifier
compound	compound
dep	dependent
acl	claussal modifier of noun
nsubjpass	passive nominal subject
auxpass	passive auxiliary
root	root node
Table 1: The main categories of relations classified by the dependency parsing tool (De Marneffe et al., 2014).
Figure 5: (a) A fully-connected graph network is built where each word is a node and each word may have relations with other words. (b) the Stanford Syntactic Parsing tool (De Marneffe et al., 2014) is used to obtain the dependency relations between words. According to these relations, we can prune the unrelated edges and obtain a sparse graph. (c) The numbers in red are the weight scores. For the node  $\text{umpire}$ 's, the weight of word  $\text{the}$  is 0.1 while the weight of word  $\text{shirt}$  is 0.9. The weight scores reflect the importance of words. The phrase  $\text{umpire}$ 's  $\text{shirt}$  describes an object, thus the word  $\text{shirt}$  is more important than word  $\text{the}$ .
Question Graph Convolutions Following the previous works (Li et al., 2019; Zhang et al., 2018b; Yang et al., 2018), we use GCN to update the node representations of words. Given a graph with  $\lambda$  nodes, each word in a question is a node. We represent the graph structure with a  $\lambda \times \lambda$  adjacency matrix  $B$  where  $B_{ij} = 1$  if there is a dependency relation between node  $i$  and node  $j$ ; else  $B_{ij} = 0$ .
Given a target node  $i$  and a neighboring node  $j \in \Omega(i)$  in a question,  $\Omega(i)$  is the set of nodes neighboring with node  $i$ . The representations of node  $i$  and  $j$  are  $h_{qi}$  and  $h_{qj}$ , respectively. To obtain the correlation score  $t_{ij}$  between node  $i$  and  $j$ , we learn a fully connected layer over concatenated node features  $h_{qi}$  and  $h_{qj}$ :
$$
t _ {i j} = w _ {c} ^ {T} \sigma \left(W _ {c} \left[ h _ {q i} ^ {(l)}, h _ {q j} ^ {(l)} \right]\right), \tag {5}
$$
where  $w_{c}$  and  $W_{c}$  are learned parameters,  $\sigma$  is the non-linear activation function, and  $[h_{qi}^{(l)}, h_{qj}^{(l)}]$  de
notes the concatenation operation. We apply a softmax function over the correlation score  $t_{ij}$  to obtain weight  $\beta_{ij}$ :
$$
\beta_ {i j} = \frac {\exp (t _ {i j})}{\sum_ {j \in \Omega (i)} \exp (t _ {i j})}. \tag {6}
$$
As shown in Figure 5(c), the numbers in red are the weight scores. The  $l$ -th layer representations of neighboring nodes  $h_{qj}^{(l)}$  are first transformed via a learned linear transformation  $W_{d}$ . Those transformed representations are gathered with weight  $\beta_{ij}$ , followed by a non-linear function  $\sigma$ . This layer-wise propagation can be denoted as:
$$
h _ {q i} ^ {(l + 1)} = \sigma \left(h _ {q i} ^ {(l)} + \sum_ {j \in \Omega (i)} B _ {i j} \beta_ {i j} W _ {d} h _ {q j} ^ {(l)}\right). \tag {7}
$$
Following the stacked  $L$  layer GCN, the output of Q-GCN module  $H_{q}$  is denoted as:
$$
H _ {q} = h _ {q i} ^ {(l + 1)} (l <   L). \tag {8}
$$
3.3 Attention Alignment Module
Based on the previous works (Gao et al., 2019; Yu et al., 2019), we use self-attention mechanism (Vaswani et al., 2017) to enhance the correlation between words in a question and the correlation between objects in an image, respectively.
To enhance the correlation between words and highlight the important words, we utilize the self-attention mechanism to update question representation  $H_{q}$ . The updated question representation  $\tilde{H}_{q}$  is obtained as follows:
$$
\tilde {H} _ {q} = \operatorname {s o f t m a x} \left(\frac {H _ {q} H _ {q} ^ {T}}{\sqrt {d _ {q}}}\right) H _ {q}, \tag {9}
$$
where  $H_{q}^{T}$  is the transpose of  $H_{q}$  and  $d_{q}$  is the dimension of  $H_{q}$ . The level of this self-attention is set to 4.
To obtain the image representation related to question representation, we align the image representation  $H_{v}$  by utilizing the question representation  $\tilde{H}_q$  as the guided vector. The similarity score  $r$  between  $H_{v}$  and  $\tilde{H}_q$  is calculated as follows:
$$
r = \frac {\tilde {H} _ {q} H _ {v} ^ {T}}{\sqrt {d _ {v}}}, \tag {10}
$$
where  $H_{v}^{T}$  is the transpose of  $H_{v}$  and  $d_v$  is the dimension of  $H_{v}$ . A softmax function is used to normalize the score  $r$  to obtain the weight score  $\tilde{r}$ :
$$
\tilde {r} = \left[ \tilde {r} _ {1}, \dots , \tilde {r} _ {i} \right] = \frac {\exp (r _ {i})}{\sum_ {j \in \mu} \exp (r _ {j})} \tag {11}
$$
7170
where  $\mu$  is the number of image regions.
By multiplying the weight  $\tilde{r}$  and the image representation  $H_{v}$ , the updated image representation  $\tilde{H}_{v}$  is obtained:
$$
\tilde {H} _ {v} = \tilde {r} \cdot H _ {v}. \tag {12}
$$
The level of this question guided image attention is set to 4. The final outputs of the attention alignment module are  $\tilde{H}_q$  and  $\tilde{H}_v$ .
3.4 Answer Prediction
We apply the linear multimodal fusion method to fuse two representations  $\tilde{H}_q$  and  $\tilde{H}_v$  as follows:
$$
H _ {r} = W _ {v} ^ {T} \tilde {H} _ {v} + W _ {q} ^ {T} \tilde {H} _ {q}, \tag {13}
$$
$$
\operatorname {p r e d} = \operatorname {s o f t m a x} \left(W _ {e} H _ {r} + b _ {e}\right), \tag {14}
$$
where  $W_{v}, W_{q}, W_{e}$ , and  $b_{e}$  are learned parameters, and pred means the probability of the classified answers from the set of answer vocabulary which contains  $M$  candidate answers. Following (Yu et al., 2019), we use binary cross-entropy loss function to train an answer classifier.
VQA-v2 (Goyal et al., 2017) is the most commonly used VQA benchmark dataset which is split into train, val, and test-standard sets. Among test-standard set,  $25\%$  are served as test-dev set. Each question has 10 answers from different annotators. Answers with the highest frequency are treated as the ground truth. All answer types can be divided into Yes/No, Number, and Other. VQA-CP-v2 (A-grawal et al., 2018) is a derivation of the VQA-v2 dataset, which is introduced to evaluate and reduce the question-oriented bias in VQA models. Due to significant difference of distribution between train set and test set, the VQA-CP-v2 dataset is harder than VQA-v2 dataset.
We use the Adam optimizer (Kingma and Ba, 2014) with parameters  $\alpha = 0.0001$ ,  $\beta_{1} = 0.9$ , and  $\beta_{2} = 0.99$ . The size of the answer vocabulary is set to  $M = 3,129$  as used in (Anderson et al., 2018). The base learning rate is set to 0.0001. After 15 epochs, the learning rate is decayed by 1/5 every 2 epochs. All the models are trained up to 20 epochs with the same batch size 64 and hidden size 512. Each image has  $\mu \in [10,100]$  object regions, all
questions are padded and truncated to the same length 14, i.e.,  $\lambda = 14$ . The levels of stacked layer  $L$  and attention alignment module are both 4.
Table 2 shows the performance of our DC-GCN model and baseline models trained with the widely-used VQA-v2 dataset. All results in our paper are based on single-model performance. For a fair comparison, we also train our model with extra visual genome dataset (Krishna et al., 2017). Bottom-Up