SlowGuess/ACL_2020_OCR huggingface dataset:
Mahmoud Khademi
Burnaby, BC, Canada
mkhademi@sfu.ca
We introduce a new neural network architecture, Multimodal Neural Graph Memory Networks (MN-GMN), for visual question answering. The MN-GMN uses graph structure with different region features as node attributes and applies a recently proposed powerful graph neural network model, Graph Network (GN), to reason about objects and their interactions in an image. The input module of the MN-GMN generates a set of visual features plus a set of encoded region-grounded captions (RGCs) for the image. The RGCs capture object attributes and their relationships. Two GNs are constructed from the input module using the visual features and encoded RGCs. Each node of the GNs iteratively computes a question-guided contextualized representation of the visual/textual information assigned to it. Then, to combine the information from both GNs, the nodes write the updated representations to an external spatial memory. The final states of the memory cells are fed into an answer module to predict an answer. Experiments show MN-GMN rivals the state-of-the-art models on Visual7W, VQA-v2.0, and CLEVR datasets.
Visual question answering (VQA) has been recently introduced as a grand challenge for AI. Given an image and a free-form question about it, the VQA task is to produce an accurate natural language answer. VQA has many applications, such as image retrieval and search. This paper proposes a new neural network architecture for VQA based on the recent Graph Network (GN) (Battaglia et al., 2018).
The pairwise interactions between various regions of an image and spatial context in both horizontal and vertical directions are important to answer questions about objects and their interactions in the scene context. For example, to answer How many cats are in the picture? (see Figure 1), a
Figure 1: An example from Visual Genome (https://visualgenome.org/). The region-grounded captions provide useful clues to answer questions. For example, to answer Where are the cats?, orange and white cat laying on a wooden bench is informative.
model needs to aggregate information from multiple, possibly distant, regions; hence applying a convolutional neural network may not be sufficient to perform reasoning over the regions. Our new architecture (see Figure 2), Multimodal Neural Graph Memory Network (MN-GMN), uses a
7177
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7177-7188
graph structure to represent pairwise interactions between visual/textual features (nodes) from different regions of an image. GNs provide a context-aware neural mechanism for computing a feature for each node that represents complex interactions with other nodes. This enables our MN-GMN to answer questions that need reasoning about complex arrangements of objects in a scene.
Previous approaches such as Memory Networks (MN) (Sukhbaatar et al., 2015) and Dynamic Memory Networks (DMN) (Kumar et al., 2015) combined a memory component and an attention mechanism to reason about a set of inputs. The DMN was first proposed for text QA. The text QA task is composed of a question, and a set of statements, called facts, in the order that describes a short story. Only a subset of the facts is required to answer a question. DMN includes four modules: input, question, episodic memory, and answer. The input and question modules encode the question and the facts. Then, the episodic memory takes as input the question and aggregates the facts to produce a vector representation of the relevant information. This vector is passed to the answer module to predict an answer. Previous applications of the MN and DMN for VQA either represent each image region independently as a single visual fact (Xu and Saenko, 2015) or represent the regions of an image like facts of a story with a linear sequential structure (Xiong et al., 2016). But, whereas a linear order may be sufficient for text QA, it is insufficient to represent the 2D context of an image.
The major novel aspect of our approach is that we exploit the flexibility of GNs to combine information from two different sources: visual features from different image regions and textual features based on region-grounded captions (RGCs). An RGC detector is learned by transfer learning from a dataset with region-grounded captions. Like visual features, an RGC is specified with a bounding-box. The RGCs capture object attributes and relationships that are often useful to answer visual questions. For example, in Figure 2, to answer Is the water calm?, a wave in the ocean is informative; the water is blue specifies an attribute of water; surfer riding a wave describe interactions between objects. Captions also incorporate commonsense knowledge. Our multimodal graph memory network comprises a visual GN and a textual GN, one for each information source. Each node of the two GNs iteratively computes a question-guided
Q: Is the water calm? How many surfers are in the picture? What color is the water?
Figure 2: Multimodal Neural Graph Memory Networks for VQA. The visual features/captions are extracted from white/black bounding-boxes and are used as node features to construct the visual/textual Graph Network.
contextualized representation of the visual/textual information at the bounding-box assigned to it. The third component in our multimodal graph memory module is an external spatial memory, which is designed to combine information across the modalities. Each node writes the updated representations to the external spatial memory, which is composed of memory cells arranged in a 2D grid. The final state of the memory cells is then fed into the answer module to predict an answer. The external spatial memory resolves the redundancy introduced by overlapping bounding-boxes, which causes difficulties, for example, with counting questions.
7178
- We introduce a new memory network architecture, based on graph neural networks, which can reason about complex arrangements of objects in a scene to answer visual questions.
- To the best of our knowledge, this is the first work that explicitly incorporates local textual information (RGCs) of the image via a transfer learning technique into a multimodal memory network to answer visual questions.
- Our architecture, which can be seen as a multimodal relational extension to DMN, rivals the state-of-the-art on three VQA datasets.
An important part of the VQA task is to understand the given question. Most approaches utilize a neural network architecture that can handle sequences of flexible length and learn complex temporal dynamics using a sequence of hidden states. Such architectures include Recurrent Neural Networks (RNNs), Long Short-Term Memory (LSTM), and the Gated Recurrent Unit (GRU). To encode a given image, most VQA approaches employ a Convolutional Neural Network (CNN) pre-trained on ImageNet, such as VGGNet and ResNet, to extract visual information from an image. These two recent trends of applying CNNs and RNNs have been successfully applied to image captioning and visual grounding (Johnson et al., 2015) tasks. Grounding connects words to their visual meaning. Our approach sees VQA as first grounding the question in the image and then predicting an answer.
Most early deep neural-based VQA models produce an answer conditioned on a global visual feature vector and the embedded question. However, since many questions and answers relate to a specific region in an image, these models often cannot predict a precise answer. To overcome this issue, many attention-based models are proposed. The attention-based models compute an attention weight of spatially localized CNN features based on the question to predict an answer (Xu and Saenko, 2015; Xiong et al., 2016). Teney et al. (2018) used the Bottom-Up Attention model (Anderson et al., 2018) to obtain a set of features at different regions of the image and computed an attention weight for each region based on the encoded question to predict an answer. In Lu et al. (2016),
the authors proposed a hierarchical co-attention model that jointly implements both image-guided question attention and question-guided visual attention. Fukui et al. (2016) proposed a VQA model based on multimodal compact bilinear (MCB) pooling to get a joint representation for image and question. Similarly, Yu et al. (2018); Kim et al. (2018) utilized higher-order fusion techniques to combine the question with visual features more efficiently. Cadene et al. (2019) proposed a bilinear fusion algorithm to represent interactions between question and image regions.
In Jabri et al. (2016), the authors introduced a model called Relation Networks, which uses multilayer perceptron models to reason over all pairs of local image features extracted from a grid of image regions. Dynamic tree structures have been used in VQA to capture the visual context of image objects (Tang et al., 2019). Yi et al. (2018) proposed a model called neural-symbolic visual question answering (NS-VQA). The NS-VQA uses symbolic structure as prior knowledge to answer questions that need complex reasoning. This model first extracts a structural scene representation from the scene and a program trace from the given question. Then, it applies the program to the scene representation to predict an answer.
Recently, a few models are proposed which can learn the interactions between image regions. The graph learner model (Norcliffe-Brown et al., 2018) merges a graph representation of the image based on the question with a graph convolutional network, to learn visual features that can represent question specific interactions. Yang et al. (2018) proposed to reason over a visual representation of the image called scene graph which represents objects and their relationships explicitly. Li et al. (2019) introduced a VQA model called Relation-aware Graph Attention Network (ReGAT). Guided by the question, ReGAT encodes an image into a graph that represents relations among visual objects. The ReGAT is trained on Visual Genome dataset (Krishna et al., 2016).
Most of the above models need datasets with annotated object relationship triplets for training. Because annotating triplets is difficult, such datasets are relatively small. Instead, our VQA architecture exploits the rich textual information of an image via incorporating the RGCs to learn the attributes of an image region and the interactions between a set of image regions enclosed by an RGC bounding-box.
7179
This information is much easier to obtain because large caption datasets are available.
More recently, Hudson and Manning (2019a) proposed a model called Neural State Machine (NSM) for the visual questions that need compositionality and multi-step inference. Given an image, the NSM first predicts a probabilistic graph as a structured semantic representation of the image. Then, NSM executes sequential reasoning guided by the input question over the predicted graph, by iteratively traversing the nodes of the graph. The authors show that the proposed model can achieve state-of-the-art results on VQA-CP (Agrawal et al., 2018) and GQA (Hudson and Manning, 2019b) datasets. Shrestha et al. (2019) introduced a VQA model called Recurrent Aggregation of Multimodal Embeddings Network (RAMEN), which is suitable for both natural image understanding and the synthetic datasets that need compositional reasoning. The RAMEN processes visual and question features in three steps: early fusion of spatially localized image features with question features, learning bimodal embeddings, and aggregating them across the image by applying a bidirectional GRU to capture the interactions between bimodal embeddings.
3 Graph Networks
In this section, we briefly explain the graph networks (GN) framework (Battaglia et al., 2018). The GN extends several other graph neural networks such as message-passing neural networks (Gilmer et al., 2017), and non-local neural networks (Wang et al., 2018). In a GN framework, a graph is represented by a 3-tuple  $\mathcal{G} = (\mathbf{u},\mathcal{V},\mathcal{E})$ , where  $\mathbf{u}$  is a graph-level attribute. The  $\mathcal{V} = \{\mathbf{v}_i\}_{i = 1:N}$  is a set of node attributes, where  $\mathbf{v}_i$  is a node attribute of node  $i$ , and  $N$  is the number of nodes. The  $\mathcal{E} = \{(\mathbf{e}_k,r_k,s_k)\}_{k = 1:M}$  is a set of edges, where  $\mathbf{e}_k$  is an edge attribute for the edge going from node  $s_k$  to node  $r_k$ , and  $M$  is the number of edges.
A GN block has three update functions  $\phi$  and three aggregation functions  $\rho$ . Given an input graph, a GN block updates the graph using the update and aggregation functions. The computational steps in a GN are represented in Algorithm 1. The function  $\phi^e$  is mapped over entire edges to calculate per-edge updates,  $\phi^v$  is mapped over entire nodes to calculate per-node updates, and  $\phi^u$  is used to update the global attribute. The  $\rho$ 's should be unvarying to permutations of their inputs and must be flexible to a varying number of arguments, such
as maximum, summation, etc.
Algorithm 1: Computational steps in a Graph Network block.
Input :A graph  $G = (\mathbf{u},\mathcal{V},\mathcal{E})$
Output:Updated graph  $G^{\prime} = (\mathbf{u}^{\prime},\mathcal{V}^{\prime},\mathcal{E}^{\prime})$
(1) Function GraphNetwork  $(\mathcal{E},\mathcal{V},\mathbf{u})$
(2) for  $k \gets 1$  to  $M$  do
(3)  $\mathbf{e}_k^{\prime}\gets \phi^{e}(\mathbf{e}_{k},\mathbf{v}_{r_{k}},\mathbf{v}_{s_{k}},\mathbf{u})\triangleright$  Compute new edge attributes
(4) end
(5) for  $i\gets 1$  to  $N$  do
(6)  $\mathcal{E}_i^{\prime}\gets \left\{(\mathbf{e}_k^{\prime},r_k,s_k)\right\}_{r_k = i,k = 1:M}$
(7)  $\bar{\mathbf{e}}_i^{\prime}\gets \rho^{e\to v}(\mathcal{E}_i^{\prime})\quad \triangleright A g g r e g a t e e d g e$  attributes for each node
(8)  $\begin{array}{rlr}\mathbf{v}_i^{\prime}\gets \phi^v (\bar{\mathbf{e}}_i^{\prime},\mathbf{v}_i,\mathbf{u}) & \triangleright \text{Compute new}\\ \text{node attributes} \end{array}$
(9) end
(10)  $V^{\prime}\gets \{\mathbf{v}_{i}^{\prime}\}_{i = 1:N}$
(11)  $\mathcal{E}' \gets \left\{(\mathbf{e}_k', r_k, s_k)\right\}_{k=1:M}$
(12)  $\bar{\mathbf{e}}^{\prime}\gets \rho^{e\rightarrow u}(\mathcal{E}^{\prime})$  Aggregat edge attributes for the whole graph
(13)  $\bar{\mathbf{v}}^{\prime}\gets \rho^{v\to u}(\mathcal{V}^{\prime})$  Aggregatenode attributes for the whole graph
(14)  $\mathbf{u}' \gets \phi^u(\bar{\mathbf{e}}', \bar{\mathbf{v}}', \mathbf{u}) \triangleright$  Compute new global attribute
(15) return  $(\mathbf{u}',\mathcal{V}',\mathcal{E}')$
4 Our Proposed Architecture
Figure 2 shows our MN-GMN architecture, which is composed of four modules: input, question, multimodal graph memory network, and answer. We now describe these modules.
4.1 Input Module
The input module has two components: A deep CNN, e.g., Bottom-Up Attention (Anderson et al., 2018), ResNet (He et al., 2015), etc. and a region-grounded caption (RGC) encoder which encodes the RGCs. The RGCs are generated by a dense captioning model. Then, they are encoded with a GRU and a parser (Schuster et al., 2015). The RGCs are useful to answer questions about object attributes and their relationships. We now describe the details and motivation for these components.
Visual Feature Extraction. To extract visual features, we use the Bottom-Up Attention model. The features are obtained via Faster R-CNN and 101-layer ResNet, which attend to specific image
7180
regions. Using a fixed threshold on object detection, we extract  $N$  2048-dimensional image features from  $N$  different regions of the image. The value of  $N$  depends on the image and ranges from 10 to 100. Each feature vector has a bounding-box specified by its coordinates  $\mathbf{r} = (r_x,r_y,r_{x'},r_{y'})$ , where  $(r_x,r_y)$  and  $(r_{x'},r_{y'})$  are the top-left and bottom-right corners of the bounding-box which are normalized to have a values between 0 and 1 based on the height and width of the image. We concatenate each feature vector with its bounding-box to obtain a vector denoted by  $\mathbf{x}_i$ ,  $(i = 1,\dots ,N)$ . Note that  $\mathbf{x}_i$  only describes the image at its bounding-box without exploiting the global spatial context.
Captions. To extract a set of RGCs for the image, we use a dense captioning model proposed by Johnson et al. (2015). This model contains a CNN, a dense localization layer, and an RNN language model that generates the captions (https://github.com/jcjohnson/densecap). The model is trained on RGCs from the Visual Genome dataset. The training set that we use does not include VQAv2.0/Visual7W test images. Through transfer learning, our model is leveraging the caption annotations. Each RGC has a caption, a bounding-box, and a confidence score. To encode a caption, we first create a dictionary using all words in the captions and questions. We preprocess the captions and questions with basic tokenization by converting all sentences to lower case and throwing away non-alphanumeric characters.
We map the words to a dense vector representation using a trainable word embedding matrix  $\mathbf{L} \in L \times D$ , where  $D$  is the dimensionality of the semantic space, and  $L$  is the size of the dictionary. To initialize the word embeddings, we use the pretrained GloVe vectors. The words that don't occur in the pretrained word embedding model are initialized with zeros. We encode a caption using a GRU and a parser. The parser takes a caption and parses it into a set of objects with their attributes and a set of relationship triplets. The encoded RGC is a vector representation denoted by  $\tilde{\mathbf{x}} \in \mathbb{R}^D$ . See appendix A for more detail about the RGC encoding.
4.2 Question Module
We encode a question using the same dictionary as we use for captions. This enables our model to match the words in a caption with the words in a question and attend to the relevant caption. The final hidden state of a GRU, denoted by  $\mathbf{q}$ , is used
as the representation of the question.
4.3 Multimodal Graph Memory Network
Given a set of visual feature vectors, a set of encoded RGCs, and the encoded question, the multimodal graph memory network module produces a representation of the relevant information based on the encoded question. The memory chooses which parts of the inputs to focus on using an attention mechanism. Unlike previous work (Xu and Saenko, 2015; Xiong et al., 2016), our memory network module is multimodal and relational. That is, it employs both textual and visual information of the input image regions, and it exploits pair-wise interactions between each pair of visual/textual features using a visual/textual GN. Similar to visual features, most of the RGCs may be irrelevant to the given question. Thus, the memory module needs to learn an attention mechanism for focusing on the relevant RGCs.
Formally, the multimodal graph memory network is composed of a visual GN  $\mathcal{G} = (\mathbf{u},\mathcal{V},\mathcal{E})$  with  $N$  nodes, a textual GN  $\tilde{\mathcal{G}} = (\tilde{\mathbf{u}},\tilde{\mathcal{V}},\tilde{\mathcal{E}})$  with  $\tilde{N}$  nodes, and an external spatial memory. Each node of the visual GN represents a visual feature with an associated bounding-box. Similarly, each node of the textual GN has a bounding-box corresponds to a detected RGC of the image. In both GNs, we connect two nodes via two forward and backward edges if they are nearby. That is, we connect two nodes if the Euclidean distance between the normalized center of their bounding-boxes is less than  $\gamma = 0.5$ . Note that even if two nodes of a GN are not neighbors, they may still communicate via the message passing mechanism of the GN.
The external memory is a network of memory cells arranged in a  $P \times Q$  grid. Each cell has a fixed location that corresponds to a specific  $(H / P) \times (W / Q)$  region in the image, where  $H$  and  $W$  are height and width of the image. Each node of the visual/textual GN sends its information to a memory cell if its bounding-box covers the location of the cell. Since the bounding-boxes may overlap, a cell may get information from multiple nodes. The external memory network is responsible for aggregating the information from both GNs and eliminating redundancy introduced by overlapping bounding-boxes. This makes our architecture less sensitive to the number of detected bounding-boxes. Since the input to the spatial memory is the output of the GNs, the state of the GN nodes can be seen
7181
as an internal memory, and the state of the spatial memory can be seen as an "external" memory like Neural Turing Machines (Graves et al., 2014).
Initialization. To initialize each node attribute of the visual GN, we combine a visual feature vector extracted from a region of the image with the encoded question using MCB pooling as  $\mathbf{v}_i = \mathbf{q} \star \mathbf{x}_i$ , where  $\star$  represents the MCB pooling. Similarly, we initialize each node attribute of the textual GN as  $\tilde{\mathbf{v}}_i = \mathbf{q} \odot \tilde{\mathbf{x}}_i$ , where  $\odot$  is the element-wise multiplication. We use the MCB to combine the visual features with the encoded question since the question and visual features are from different modalities. The global attribute  $\mathbf{u}$  is initialized by a global feature vector of the image extracted from the last layer of the 101-layer ResNet. This helps to answer questions that need the global features of the scene. The global attribute  $\tilde{\mathbf{u}}$  is initialized with the encoded question. The edge features of the GNs and memory cells are initialized with zero vectors.
Updates. At each iteration, we first update the GNs. Then, we update the content of the memory cells. We update the edge attributes, node attributes, and global attribute of both GNs as described in Algorithm 1. For each GN, we use three different GRUs to implement the functions  $\phi^e$ ,  $\phi^v$ , and  $\phi^u$ . The  $\rho^{e\rightarrow v}$  is an element-wise summation. The  $\rho^{v\rightarrow u}$  and  $\rho^{e\rightarrow u}$  for visual GN are implemented as
$$
\bar {\mathbf {v}} ^ {\prime} = \psi \left(\sum_ {i} \sigma \left(\mathbf {W} _ {1} \mathbf {v} _ {i} + \mathbf {b} _ {1}\right) \odot \psi \left(\mathbf {W} _ {2} \mathbf {v} _ {i} + \mathbf {b} _ {2}\right)\right)
$$
$$
\bar {\mathbf {e}} ^ {\prime} = \psi \left(\sum_ {k} \sigma \left(\mathbf {W} _ {3} \mathbf {e} _ {k} + \mathbf {b} _ {3}\right) \odot \psi \left(\mathbf {W} _ {4} \mathbf {e} _ {k} + \mathbf {b} _ {4}\right)\right)
$$
where,  $\sigma$  and  $\psi$  are the sigmoid and tangent hyperbolic activation functions, and  $\mathbf{W}_i$ ,  $\mathbf{b}_i$ ,  $i = 1, \ldots, 4$ , are trainable parameters. This allows to incorporate information from the question for computing the attention weights using the sigmoid function for each node/edge. The  $\rho^{v \to u}$  and  $\rho^{e \to u}$  for the textual GN are implemented in a similar way. Let,  $\bar{\mathbf{v}}_{p,q} = \frac{1}{|\mathcal{N}_{p,q}|} \sum_{i \in \mathcal{N}_{p,q}} \mathbf{v}_i$  and  $\bar{\tilde{\mathbf{v}}}_{p,q} = \frac{1}{|\tilde{\mathcal{N}}_{p,q}|} \sum_{i \in \tilde{\mathcal{N}}_{p,q}} \tilde{\mathbf{v}}_i$ , where  $\mathcal{N}_{p,q}$  and  $\tilde{\mathcal{N}}_{p,q}$  are the set of nodes which are connected to the memory cell  $(p,q)$  in the visual and textual GNs, respectively. Each memory cell is updated as
$$
\bar {\mathbf {m}} _ {p, q} = f \left(\mathbf {m} _ {p - 1, q}, \mathbf {m} _ {p, q - 1}, \mathbf {m} _ {p, q + 1}, \mathbf {m} _ {p + 1, q}\right)
$$
$$
\mathbf {m} _ {p, q} ^ {\prime} = \mathrm {G R U} ([ \bar {\mathbf {v}} _ {p, q}, \bar {\tilde {\mathbf {v}}} _ {p, q}, \bar {\mathbf {m}} _ {p, q} ], \mathbf {m} _ {p, q})
$$
where  $f$  is a neural network layer which aggregates the memories from the neighboring cells. We repeat these steps for two iterations. Applying one
iteration decreases the accuracy by about 2.0 points. As observed by Kumar et al. (2015), iterating over the inputs allows the memory network to take several reasoning steps which some questions require.
4.4 Answer Module