SlowGuess/ACL_2020_OCR huggingface dataset:
Given a caption, the hidden state of the caption GRU for the  $i$ -th word is computed by  $\mathbf{c}_i = \mathrm{GRU}(\mathbf{s}_i,\mathbf{c}_{i - 1})$ , where  $\mathbf{s}_i$  is the semantic representation of the  $i$ -th word in the caption. The final hidden state of the GRU, denoted by  $\mathbf{c}\in \mathbb{R}^{D}$ , is used as a vector representation of the caption. We reset the GRU after feeding each caption.
The encoded captions may not properly represent the objects in an image, the relationships between them, and the object attributes, especially for a caption with a complex parse tree. Thus, we enrich this representation by utilizing a parser developed by Schuster et al. (2015) that takes a single caption and parses it into a set of relationship triplets (possibly empty), a set of objects, and their attributes. For example, given orange and white cat laying on a wooden bench, the parser outputs a triplet cat-lay on-bench, objects cat and bench, and attributes cat-white, cat-orange and bench-wooden.
For the Visual Genome dataset, the parser produces less than three relationships for about  $98$
7187
of the captions. We obtain a fixed-length representation for the output of the parser, denoted by  $\tilde{\mathbf{c}}\in \mathbb{R}^{14D}$  by allocating the embedding of 14 words: 6 words for up to two relationship triplets and 8 words for up to 4 objects and their attributes. For the aforementioned example, the fixed-length representation is the concatenation of the embedding of each word in sequence  $\prec$  cat-lay on-bench,x-x-x,bench-wooden,cat-orange,cat-white,x-x>, where  $x$  is a special token to represent an empty slot. To create the sequences, we use a fixed arbitrary order. Each RGC has also a bounding-box specified by its coordinates  $\tilde{\mathbf{r}} = (\tilde{r}_x,\tilde{r}_y,\tilde{r}_{x'},\tilde{r}_{y'})$ , where  $(\tilde{r}_x,\tilde{r}_y)$  and  $(\tilde{r}_{x'},\tilde{r}_{y'})$  are the top-left and bottom-right corners of the bounding-box which are normalized to have a value between 0 and 1 based on the height and width of the image. For each RGC, we project the concatenation of  $\tilde{\mathbf{r}}$ ,  $\mathbf{c}$  and  $\tilde{\mathbf{c}}$  to a space of dimensionality  $D$  using a densely-connected layer with ReLU activation function to obtain a vector representation denoted by  $\tilde{\mathbf{x}}\in \mathbb{R}^D$ .
B More Examples for VQA Task
Figure 6 shows a VQA example on CLEVR dataset. Figure 7 shows how MN-GMN can answer a ques
Figure 6: Example VQA with N-GMN+ on CLEVR.
tion correctly by incorporating the region-grounded captions, whereas N-GMN gives the wrong answer. Figure 8 illustrates the visualization of the attention weights with MN-GMN to answer a Number question. For this example, we compute the attention weights that are used to obtain  $\bar{\mathbf{v}}^{\prime}$  for each spatial memory cell. More precisely, the magnitude of the sigmoid output that implements  $\rho^{v\rightarrow u}$  for the external spatial memory is visualized. Each attention weight shows the importance of a fixed region in a  $14\times 14$  grid of memory cells to the question.
Q: What color are the man's shoes?  
A: White (MN-GMN), Blue (N-GMN)
Figure 7: The MN-GMN provides the correct answer using white and black tennis shoes.
Q: How many zebras are pictured? A: 3
Figure 8: Visualization of the attention weights for a  $14 \times 14$  grid of cells. Red regions get higher attention.
7188