SlowGuess/ACL_2020_OCR huggingface dataset:
The answer module predicts an answer using a GN called answer GN. The nodes of the answer GN are the external spatial memory cells. However, there is an edge between every ordered pair of the nodes (cells), hence the answer GN is a complete graph. This supports reasoning across distant regions of the image. Let  $\mathbf{m}_{p,q}^{\circ}$  be the final state of the memory cell at location  $(p,q)$ . We initialize the node attributes of the answer GN denoted by  $\mathbf{v}_{p,q}^{\circ}$  as  $\mathbf{v}_{p,q}^{\circ} = \mathbf{m}_{p,q}^{\circ}$ . The edge attributes are initialized using the one-hot representation of the location of the sender and receiver memory cells. That is, the edge attribute of the edge going from the memory cell at location  $(p,q)$  to  $(p',q')$ , is initialized with a vector of size  $2P + 2Q$  which is computed by concatenating the one-hot representation of  $p,q,p'$  and  $q'$ . The global attribute of the answer GN is initialized with a vector of zeros.
Then, we update the edge attributes, the node attributes and the global attribute of the answer GN as described in Algorithm 1. As before, we use three different GRUs to implement functions  $\phi^e$ ,  $\phi^v$ , and  $\phi^u$ . The  $\rho^{e\rightarrow v}$  is a simple elementwise summation. The  $\rho^{v\to u}$  and  $\rho^{e\to u}$  are implemented as before, but with different set of parameters. The answer module predicts an answer as  $\hat{\mathbf{p}} = \sigma \bigl (\mathbf{W}g(\mathbf{u}^{\circ}) + \tilde{\mathbf{W}}\tilde{g} (\mathbf{u}^{\circ}) + \mathbf{b}\bigr)$  where,  $\mathbf{u}^{\circ}$  is the updated global attribute of the answer GN,  $\mathbf{W}\in \mathbb{R}^{Y\times 2048}$ ,  $\tilde{\mathbf{W}}\in \mathbb{R}^{Y\times 300}$ ,  $\mathbf{b}\in \mathbb{R}^{Y}$  are trainable parameters,  $g$ ,  $\tilde{g}$  are non-linear layers, and  $Y$  is the number of possible answers.
Following Teney et al. (2018), to exploit prior linguistic information about the candidate answers, the GloVe embeddings of the answer words are used to initialize the rows of the  $\tilde{\mathbf{W}}$ . Initialization with the Glove embeddings improves the performance by about 1.0 point. Similarly, to utilize prior visual information about the candidate answers, a visual embedding is used to initialize the rows of  $\mathbf{W}$ . The visual embedding is obtained by retrieving 10 images from Google Images for each word. Then, the images are encoded using the ResNet-101 pretrained on ImageNet to obtain a feature vector of size 2048. For each word, the average of the feature vectors is used to initialize a
7182
row of  $\mathbf{W}$ . The loss for a single sample is defined as  $\mathcal{L} = -\sum_{i=1}^{Y} p_i \log (\hat{p}_i) + (1 - p_i) \log (1 - \hat{p}_i)$  where,  $\hat{p}_i$  is the  $i$ th element of  $\hat{\mathbf{p}}$ , and  $p_i$  is the  $i$ th element of the ground-truth vector  $\mathbf{p}$  ( $p_i = 1.0$  if  $A \geq 3$  annotators give the  $i$ th answer word, otherwise  $p_i = A / 3$ ). For multiple choice task, the candidate answers are encoded by the last state of a GRU and concatenated with  $\mathbf{u}^\circ$  using a neural network layer as  $\acute{p} = \sigma \left(\hat{\mathbf{w}} \acute{f}([\mathbf{u}^\circ, \mathbf{a}] + \acute{b})\right)$  where,  $\mathbf{a}$  is an encoded answer choice,  $\acute{f}$  is a non-linear layer, and  $\hat{\mathbf{w}}, \acute{b}$  are trainable parameters. For multiple choice task, the binary logistic loss  $-p \log (\acute{p}) - (1 - p) \log (1 - \acute{p})$  is used, where  $p$  is 1.0 for an (image, question, answer) triplet, if the answer choice is correct, otherwise  $p$  is 0.
Training Details and Optimization. The MN-GMN is implemented in TensorFlow. We use a library from https://github.com/deepmind/graph_nets to implement the GNs. We follow VQA tips in Teney et al. (2018) to train our models. More specifically, to apply an ensemble technique, 20 instances of the model is trained with various initial random seeds. For test images, the scores for the answers by all models are summed, and the answer is predicted using the highest summed score. To minimize the loss, we apply the RMSprop optimization algorithm with a learning rate of 0.0001 and minibatches of size 100.
Dropout with probability 0.5 and early stopping are applied to prevent overfitting. Dropout is used after the layer that computes the updated global attribute of the answer GN. During training, all parameters are tuned except for the weights of the CNN and RGC detector to avoid overfitting. For VQA-v2.0 and Visual7W datasets, we augment the training dataset with Visual Genome/GQA images and QA pairs. The training set that we use does not include the VQA-v2.0/Visual7W test or Visual7W validation images. The output dimension of the MCB and the dimension of the hidden layer in both RGC and question GRUs are set to 512. Also, we set  $P, Q = 14$  and  $D = 512$ . The full model takes around 6 hours to train on two Titan X GPUs.
We explain the datasets, baseline models, and evaluation metric that we use in our experiments. Then, the experimental results are discussed.
Datasets. VQA-v2.0 (Antol et al., 2015) includes 82,783 training images, 40,504 validation images,
and 81,434 testing images. There are 443,757 training questions, 214,354 validation questions, and 447,793 test questions in this dataset. A subset of the standard test set, called test-dev, contains 107,394 questions. Each question has 10 candidate answers generated by humans. We choose correct answers that appear more than 8 times. This makes  $Y = 3,110$  candidate answers. We use the standard metric (Antol et al., 2015), which is an answer is correct if at least 3 people agree.
Visual7W dataset (Zhu et al., 2015) includes 47,300 images. We train and evaluate our model on telling questions of the Visual7W which includes 28,653 images. This set uses six types of questions: what  $(6\%)$ , where  $(48\%)$ , when  $(16\%)$ , who  $(5\%)$ , why  $(10\%)$ , how  $(15\%)$ . The training, validation and test splits, contain  $50\%$ ,  $20\%$ ,  $30\%$  of the QA pairs, respectively. For evaluation, Visual7W provides four candidate answers. The Visual7W has fewer language biases compared to VQA.
We also experiment on CLEVR dataset (Johnson et al., 2017a). CLEVR evaluates different aspects of visual reasoning, such as attribute recognition, counting, comparison, logic, and spatial relationships. Each object in an image has the following attributes: shape (cube, sphere, or cylinder), size (large or small), color (8 colors), and material (rubber or metal). An object detector with 96 classes is trained using all combinations of the attributes by the Tensorflow Object Detection API. We use Faster R-CNN NasNet trained on the MS-COCO dataset as the pretrained model. Given an image, the output of the object detector is a set of object bounding-boxes with their feature vectors. For CLEVR, we omit the textual GN, since CLEVR images do not have rich textual information.
Baselines. We compare our model with several architectures developed recently, including the state-of-the-art models ReGAT, BAN, VCTREE, and MuRel. For comparison, we also include three related models in Table 1 that have been proposed more recently in Arxiv preprints during the preparation of this work: LXRT, MSM@MSRA, and MIL@HDU. The ReGAT exploits supervision from Visual Genome relationships. MAN is a memory-augmented neural network which attends to each training exemplar to answer visual questions, even when the answers infrequently happen in the training set. The Count (Zhang et al., 2018) is a neural network model designed to count objects from object proposals. For Visual7W, we
7183
compare our models with Zhu et al. (2015), MCB, MAN, and MLP. The MCB leverages the Visual Genome QA pairs as additional training data and the 152-layer ResNet as a pretrained model. The MLP method uses (image, question, answer) triplets to score answer choices. For CLEVR, we compare our models with several baselines proposed by Johnson et al. (2017a) as well as the state-of-the-art models RAMEN, PROGRAM-GEN, and NS-VQA. N2NMN learns to predict a layout based on the question and compose a network using a set of neural modules. The CNN+LSTM+RN learns to infer a relation using a neural network model called Relation Networks. The PROGRAM-GEN exploits supervision from functional programming, which is used to generate CLEVR questions.
Ablation Study. We implement several lesion architectures. The MN+ResNet model does not use any GNs and is designed to evaluate the effect of using GN. This model is similar to MN (Sukhbaatar et al., 2015). It applies a soft attention for  $14 \times 14$  ResNet feature maps (the last  $14 \times 14$  pooling layer) and generates a representation  $\mathbf{u}^{\circ} = h(\mathbf{q}) \star h'(\sum_{i=1}^{196} \alpha_i \mathbf{x}_i)$ . Here  $h, h'$  are non-linear layers, and  $\alpha_i$  is an attention weight computed as  $\alpha_i = \text{softmax}(\mathbf{wh}''([\mathbf{x}_i, \mathbf{q}]))$ , where  $\mathbf{w}$  is a learned parameter vector and  $h''$  is a non-linear layer. Then, an answer is predicted as described before.
The N-GMN model only uses the visual GN (no textual GN nor spatial memory). This model evaluates the effect of incorporating RGCs. After two iterations, the global feature vector of the visual GN is used as  $\mathbf{u}^{\circ}$  to generate an answer. The  $\mathrm{N - GMN^{+}}$  model only uses the visual GN and the external spatial memory components (no textual GN). This model is used for the CLEVR dataset since CLEVR images do not have rich textual information. The MN-GMN $^{-}$  model does not use the external spatial memory. After two iterations, the global feature vector of the visual and textual GNs are concatenated and fed into a non-linear layer to generate  $\mathbf{u}^{\circ}$ . Finally, MN-GMN is our full model.
Results and Discussion. Our experimental results on VQA-v2.0 dataset are reported in Table 1. For LXRT, MSM@MSRA, and MIL@HDU, the numbers are reported from the VQA Challenge 2019 Leaderboard (using an ensemble of models). Across all question types, N-GMN outperforms MN+ResNet. This shows that applying the visual GN with explicit object bounding-boxes provides
a usefully richer representation than a grid of fixed visual features. MN-GMN $^{-}$ outperforms N-GMN. This shows that RGCs help to improve accuracy. RGCs are especially useful for answering the Other and Yes/No question types. Our full model MN-GMN outperforms MN-GMN $^{-}$ . This shows that applying external spatial memory is effective, especially for Number questions. The full model's accuracy is higher than the baselines.
Model	Test-dev	Test-standard
Y/N	Num	Other	All	Y/N	Num	Other	All
\( MAN^1 \)	-	-	-	-	79.2	39.5	52.6	62.1
\( Count^2 \)	83.1	51.6	59.0	68.1	83.6	51.4	59.1	68.4
\( MFH^3 \)	84.3	50.7	60.5	68.8	-	-	-	-
\( Bottom-Up^4 \)	81.8	44.2	56.1	65.3	-	-	-	65.7
\( G-learner^5 \)	-	-	-	-	82.9	47.1	56.2	66.2
\( v-AGCN^6 \)	82.4	45.9	56.5	65.9	82.6	45.1	56.7	66.2
\( RAMEN^7 \)	-	-	-	66.0	-	-	-	-
\( MuRel^8 \)	84.8	49.8	57.9	68.0	-	-	-	68.4
\( VCTREE^9 \)	84.3	47.8	59.1	68.2	84.6	47.4	59.3	68.5
\( BAN^{10} \)	85.4	54.0	60.5	70.0	-	-	-	70.4
\( ReGAT^{11} \)	86.1	54.4	60.3	70.3	-	-	-	70.6
\( LXRT^{12} \)	89.3	56.9	65.1	74.2	89.5	56.7	65.2	74.3
\( MSM@MSRA^{13} \)	89.8	58.9	65.4	74.7	89.8	58.4	65.7	74.9
\( MIL@HDU^{14} \)	90.1	59.2	65.7	75.0	90.4	59.2	65.8	75.2
MN+ResNet	84.2	43.4	58.1	67.3	84.5	44.0	58.1	67.5
N-GMN	86.1	53.5	61.2	70.6	86.7	53.6	61.8	71.2
\( MN-GMN^- \)	88.0	53.5	63.8	72.6	88.5	53.7	64.2	73.1
MN-GMN	88.2	56.0	64.2	73.2	88.3	56.1	64.5	73.5
Table 1: Accuracy percentage on the VQA-v2.0 dataset. The references are Ma et al. (2018)<sup>1</sup>, Zhang et al. (2018)<sup>2</sup>, Yu et al. (2018)<sup>3</sup>, Teney et al. (2018); Anderson et al. (2018)<sup>4</sup>, Norcliffe-Brown et al. (2018)<sup>5</sup>, Yang et al. (2018)<sup>6</sup>, Shrestha et al. (2019)<sup>7</sup>, Cadene et al. (2019)<sup>8</sup>, Tang et al. (2019)<sup>9</sup>, Kim et al. (2018)<sup>10</sup>, Li et al. (2019)<sup>11</sup>, Tan and Bansal (2019)<sup>12</sup>, Liu et al. (2019)<sup>13</sup>, and Yu et al. (2019)<sup>14</sup>.
Model	What	Where	When	Who	Why	How	Avg
Human1	96.5	95.7	94.4	96.5	92.7	94.2	95.7
LSTM-ATT1	51.5	57.0	75.0	59.5	55.5	49.8	54.3
Concat+ATT2	47.8	56.9	74.1	62.3	52.7	51.2	52.8
MCB+ATT2	60.3	70.4	79.5	69.2	58.2	51.1	62.2
MAN3	62.2	68.9	76.8	66.4	57.8	52.9	62.8
MLP4	64.5	75.9	82.1	72.9	68.0	56.4	67.1
N-GMN	66.2	77.2	83.3	74.0	69.2	58.5	68.6
MN-GMN-	67.1	77.4	84.0	75.1	70.1	59.2	69.3
MN-GMN	67.3	77.4	84.0	75.0	70.3	59.4	69.5
Table 2: Accuracy percentage on Visual7W dataset. The references are Zhu et al. (2015)<sup>1</sup>, Fukui et al. (2016)<sup>2</sup>, Ma et al. (2018)<sup>3</sup>, and Jabri et al. (2016)<sup>4</sup>.
Our results on Visual7W are reported in Table 2. Our N-GMN, MN-GMN $^{-}$ , and MN-GMN outperform the baselines MLP, MAN, and MCB+ATT. The results for our N-GMN $^{+}$  on CLEVR in Table 3 are competitive with the state-of-the-art RAMEN, PROGRAM-GEN, and NS-VQA. We emphasize that, unlike PROGRAM-GEN, our algorithm does
7184
not exploit supervision from functional programming. Also, unlike NS-VQA, our model is not tailored to synthetic datasets only, since it performs well on both natural and artificial datasets that need multi-step compositional reasoning.
Model	All	Exist	Count	Cmp-Int	Q-At	Cmp-At
HUMAN1	92.6	96.6	86.7	86.5	95.0	96.0
Q-TYPE MODE1	41.8	50.2	34.6	51.0	36.0	51.3
LSTM1	46.8	61.1	41.7	69.8	36.8	51.3
CNN+BOW1	48.4	59.5	38.9	51.1	48.3	51.8
CNN+LSTM1	52.3	65.2	43.7	67.1	49.3	53.0
CNN+LSTM+MCB1	51.4	63.4	42.1	66.4	49.0	51.0
CNN+LSTM+SA1	68.5	71.1	52.2	73.5	85.3	52.3
N2NMN2	83.3	85.7	68.5	85.0	90.0	88.8
CNN+LSTM+RN3	95.5	97.8	90.1	93.6	97.9	97.1
PROGRAM-GEN4	96.9	97.1	92.7	98.7	98.2	98.9
RAMEN5	96.9	98.9	94.1	88.5	98.9	99.3
NS-VQA6	99.8	99.9	99.7	99.9	99.8	99.8
N-GMN-	95.6	97.7	90.3	93.5	98.0	97.3
N-GMN+	96.3	98.0	91.8	94.8	98.1	98.1
Table 3: Accuracy on CLEVR dataset. The references are Johnson et al. (2017a)<sup>1</sup>, Hu et al. (2017)<sup>2</sup>, Santoro et al. (2017)<sup>3</sup>, Johnson et al. (2017b)<sup>4</sup>, Shrestha et al. (2019)<sup>5</sup>, and Yi et al. (2018)<sup>6</sup>.
Q: Is it a cloudy day?
A: Yes (MN-GMN), No (N-GMN)
Figure 3: N-GMN versus MN-GMN. The MN-GMN provides the correct answer using a cloudy blue sky.
Figure 3 shows how MN-GMN can answer a question correctly by incorporating RGCs, whereas N-GMN gives the wrong answer. Figure 4 illustrates the visualization of the attention weights with MN-GMN to answer a Number question. We compute the attention weights that are used to obtain  $\bar{\mathbf{v}}^{\prime}$  for each spatial memory cell. More precisely, the magnitude of the sigmoid output that implements  $\rho^{v\rightarrow u}$  for the spatial memory is visualized. Each attention weight shows the importance of a fixed region in a  $14\times 14$  grid of cells to the question. Figure 5 shows a VQA example on the CLEVR dataset. Appendix B provides more examples.
Q: How many motorcycles are there? A: 2
Figure 4: Visualization of the attention weights for a  $14 \times 14$  grid of cells. Red regions get higher attention.
Figure 5: Example VQA with  $\mathbf{N}$  -GMN  $^+$  on CLEVR.
Multi-modal Neural Graph Memory Networks are a new architecture for the VQA task. The MN-GMN represents bimodal local features as node attributes in a graph. It leverages a graph neural network model, Graph Network, to reason about objects and their interactions in a scene. In experiments on three datasets, the MN-GMN showed superior quantitative and qualitative performance compared to the lesion approaches and rivals the state-of-the-art models. A future research direction is to combine RGCs with distant supervision by an external knowledge base to answer the visual questions that need external knowledge; for example Which animal in this photo can climb a tree?
7185
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4971-4980.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6077-6086.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 2425-2433.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. 2018. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261.
Remi Cadene, Hedi Ben-Younes, Matthieu Cord, and Nicolas Thome. 2019. Murel: Multimodal relational reasoning for visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1989-1998.
Akira Fukui, Dong Huk Park, Daylen Yang, Anna Rohrbach, Trevor Darrell, and Marcus Rohrbach. 2016. Multimodal compact bilinear pooling for visual question answering and visual grounding. arXiv preprint arXiv:1606.01847.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. 2017. Neural message passing for quantum chemistry. In International Conference on Machine Learning, pages 1263-1272.
Alex Graves, Greg Wayne, and Ivo Danihelka. 2014. Neural tuning machines. arXiv preprint arXiv:1410.5401.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. arXiv preprint arXiv:1512.03385.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. 2017. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 804-813.
Drew Hudson and Christopher D Manning. 2019a. Learning by abstraction: The neural state machine. In Advances in Neural Information Processing Systems, pages 5901-5914.
Drew A Hudson and Christopher D Manning. 2019b. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6700-6709.
Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. 2016. Revisiting visual question answering baselines. In European conference on computer vision, pages 727-739. Springer.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017a. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2901-2910.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017b. Inferring and executing programs for visual reasoning. In Proceedings of the IEEE International Conference on Computer Vision, pages 2989-2998.
Justin Johnson, Andrej Karpathy, and Li Fei-Fei. 2015. Densecap: Fully convolutional localization networks for dense captioning. arXiv preprint arXiv:1511.07571.
Jin-Hwa Kim, Jaehyun Jun, and Byoung-Tak Zhang. 2018. Bilinear attention networks. In Advances in Neural Information Processing Systems, pages 1564-1574.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. 2016. Visual genome: Connecting language and vision using crowdsourced dense image annotations. arXiv preprint arXiv:1602.07332.
Ankit Kumar, Ozan Irsoy, Jonathan Su, James Bradbury, Robert English, Brian Pierce, Peter Ondruska, Ishaan Gulrajani, and Richard Socher. 2015. Ask me anything: Dynamic memory networks for natural language processing. arXiv preprint arXiv:1506.07285.
Linjie Li, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Relation-aware graph attention network for visual question answering. arXiv preprint arXiv:1903.12314.
Bei Liu, Zhicheng Huang, Zhaoyang Zeng, Zheyu Chen, and Jianlong Fu. 2019. Learning rich image region representation for visual question answering. arXiv preprint arXiv:1910.13077.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In Advances In Neural Information Processing Systems, pages 289–297.
Chao Ma, Chunhua Shen, Anthony Dick, Qi Wu, Peng Wang, Anton van den Hengel, and Ian Reid. 2018. Visual question answering with memory-augmented networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6975-6984.
Will Norcliffe-Brown, Stathis Vafeias, and Sarah Parisot. 2018. Learning conditioned graph structures for interpretable visual question answering. In Advances in Neural Information Processing Systems, pages 8334-8343.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Tim Lillicrap. 2017. A simple neural network module for relational reasoning. In Advances in neural information processing systems, pages 4967-4976.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D Manning. 2015. Generating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the fourth workshop on vision and language, pages 70-80.
Robik Shrestha, Kushal Kafle, and Christopher Kanan. 2019. Answer them all! toward universal visual question answering models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10472-10481.
Sainbayar Sukhbaatar, Jason Weston, Rob Fergus, et al. 2015. End-to-end memory networks. In Advances in neural information processing systems, pages 2440-2448.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490.
Kaihua Tang, Hanwang Zhang, Baoyuan Wu, Wenhan Luo, and Wei Liu. 2019. Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6619-6628.
Damien Teney, Peter Anderson, Xiaodong He, and Anton van den Hengel. 2018. Tips and tricks for visual question answering: Learnings from the 2017 challenge. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4223-4232.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. 2018. Non-local neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Caiming Xiong, Stephen Merity, and Richard Socher. 2016. Dynamic memory networks for visual and textual question answering. arXiv preprint arXiv:1603.01417.
Huijuan Xu and Kate Saenko. 2015. Ask, attend and answer: Exploring question-guided spatial attention for visual question answering. arXiv preprint arXiv:1511.05234.
Zhuoqian Yang, Jing Yu, Chenghao Yang, Zengchang Qin, and Yue Hu. 2018. Multi-modal learning with prior visual relation reasoning. arXiv preprint arXiv:1812.09681.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. 2018. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems, pages 1031-1042.
Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. 2019. Deep modular co-attention networks for visual question answering. arXiv preprint arXiv:1906.10770.
Zhou Yu, Jun Yu, Chenchao Xiang, Jianping Fan, and Dacheng Tao. 2018. Beyond bilinear: Generalized multimodal factorized high-order pooling for visual question answering. IEEE transactions on neural networks and learning systems, 29(12):5947-5959.
Yan Zhang, Jonathon Hare, and Adam Prügel-Bennett. 2018. Learning to count objects in natural images for visual question answering. arXiv preprint arXiv:1802.05766.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. 2015. Visual7w: Grounded question answering in images. arXiv preprint arXiv:1511.03416.
A Encoding Region-Grounded Captions