Title: REASONING-AWARE GRAPH CONVOLUTIONAL NET-WORK FOR VISUAL QUESTION ANSWERING

Abstract: Relational reasoning methods based on graph networks are currently state-of-theart models for Visual Question Answering (VQA) tasks involving real images. Although graph networks are used in these models to enrich visual representations by encoding question-adaptive inter-object relations, these simple graph networks is arguably insufficient to perform visual reasoning for VQA tasks. In this paper, we propose a Reasoning-Aware Graph Convolutional Networks (RA-GCN) that goes one step further towards visual reasoning for GCNs. Our first contribution is the introduction of visual reasoning ability into conventional GCNs. Secondly, we strengthen the expressive power of GCNs via introducing node-sensitive kernel parameters based on edge features to address the limitation of shared transformation matrix for each node in GCNs. Finally, we provide a novel iterative reasoning network architecture for solving VQA task via embedding the RA-GCN module into an iterative process. We evaluate our model on the VQA-CP v2, GQA and Clevr dataset. Our final RA-GCN network successfully achieves state-of-the-art accuracy which is 42.3% on the VQA-CP v2, and highly competitive 62.4% accuracy on the GQA, as well as 90.0% on val split of Clevr dataset.

1. INTRODUCTION:

Since the Convolutional Neural Networks (CNNs) have successuflly tackled many classic computer vision problems such as image classification (Krizhevsky et al., 2012) (Simonyan & Zisserman, 2014) , object detection (He et al., 2017) (Ren et al., 2015a) and generation (Radford et al., 2015) , generalizing CNNs to inputs with graph-like structures is an important topic in the field of deep learning. Graph Neural Networks (GNNs) were introduced in Kipf & Welling (2016) Scarselli et al. (2008) as a common solution to handle arbitrary graph data. Beyond GNNs' outstanding performances when applied to 3D mesh deforamtion (Ranjan et al., 2018) , image captioning (Yao et al., 2018) , scene understanding (Yang et al., 2018) , GNNs have also been successfully used for Visual Question Answering (VQA) task (Li et al., 2019) (Norcliffe-Brown et al., 2018b) (Hu et al., 2019) . VQA requires a high level understanding of images and questions and is often considered to be a good proxy for visual reasoning. However, like the CNNs, it is not straightforward to use GNNs in a situation where a high level of reasoning is required. In this paper, we investigate the usage of GNNs for visual reasoning requried by VQA task, which is a core problem of comuter vision tasks in a lot of real-world applications.

How should we build a model based on GNNs to perform reasoning in VQA task? Recently, most state-of-the-art approaches based on graph networks to VQA (Cadene et al., 2019) (Li et al., 2019 ) (Norcliffe-Brown et al., 2018b) (Hu et al., 2019) , are focusing on encoding question-adaptive interobject relations to enrich visual representations via different graph networks. Figure 1 shows an overview of the framwork of methods mentioned above. Hu et al. (2019) enhances each object features in the scene with a relational contextualized feature collected by multiple iterations of meassage passing between objects. Cadene et al. (2019) provides an iterative reasoning process of a MuRel cell which is designed to encode interactions between question and image objects. Li et al. (2019) adopts a graph attention network to learn multi-type question-adaptive relation representations, in order to enrich visual representations. Norcliffe-Brown et al. (2018b) constructs question-adaptive graph structure and uses a spatial GCN to learn the visual representaions. We stress that the enriched visual features from above methods will then fuse with question embeddings via a multimodal fusion module to produce a joint representation, which is used in the answer prediction. Without the last multimodal fusion module, those methods will lost huge performance. To some extent, the graph networks adopted by above approaches are mainly responsible for learning question-adaptive inter-object relationships instead of visual reasoning.

In this work, we propose a novel Reasoning-Aware Graph Convolutional Network (RA-GCN) that goes one step further towards visual reasoning for GCNs. Our first contribution is to introduce the RA-GCN module which achieves visual reasoning capability to modulate visual representations based on question information. Our second contribution is to introduce node-sensitive kernel parameters based on edge features to address the limitation of shared transformation matrix for each node in GCNs. By building edge features to capture relationships between nodes and providing node-specific kernel parameters, the expressive power of model visual feature patterns is enhanced. Our last contribution is to provide a novel iterative reasoning network architecture for solving VQA task via embedding the RA-GCN module into an iterative process.

In the experments, we provide various ablative studies to validate our RA-GCN module and the iterative reasoning architecture. By evaluating our model on the VQA-CP v2 (Agrawal et al., 2018) , GQA (Hudson & Manning, 2019) and Clevr (Johnson et al., 2017) 

3. REASONING-AWARE GRAPH CONVOLUTIONAL NETWORKS:

Our VQA approach is depicted in Figure 2 . Given an image v ∈ I and a question q ∈ Q, we want to predict an answer â ∈ A that matches the ground truth answer a * . As common practice in the VQA literature, this can be defined as a classification problem:

â = arg max a∈A p θ (a|v, q) (1)

where p θ is our trained model. In our system, the image is represented by a set of objects V = {v i } K i=1 , where each object v i is associated with visual feature vector v i ∈ R dv and a boundingbox spatial coordinates b i . Each b i = [x, y, w, h] corresponds to 4-dimensional spatial coordinate, where (x, y) denotes the coordinate of the top-left point and (h, w) is the height, width of the bounding box. Note that x and w (respectively y and h) are normalized by the width (resp. height) of the image. For the question, we adopt dynamic RNN with a GRU cell to provide a sentence embedding q ∈ R dq (d q = 1024 in our experiments) as suggested in Norcliffe-Brown et al. (2018b) .

In Section 3.1, we provide the background of GCNs and a feature-wise linear modulation (FiLM) transformation approach (Perez et al., 2017) . Next, in Section 3.2, we present the RA-GCN module, a novel graph convolution network that learns to perform visual reasoning operations by blending conditional question information into the set of spatially grounded visual representations. Finally, in Section 3.3, by leveraging the reasoning power of RA-GCN module, a novel architecture is established by iterating through the RA-GCN module to reason about the visual representations with respect to a question.

3.1. PRELIMINARIES:

GCN. We will start by a brief recap of the 'vanilla' as proposed in Kipf & Welling (2016) . A graph is represented as G = {V, E} where V is the set of K nodes and E are edges, while x l i ∈ R D l and x l+1 i ∈ R D l+1 are the feature of node x i before and after the l-th convolution respectively. Generally, there has two steps to perform a graph convolutional filter on node x i . First, Node representations are transformed by a learnable parameter matrix W ∈ R D l+1 ×D l . Second, node x i gathers these transformed node representations from its neighbour nodes j ∈ N (i), followed by a non-linear function (ReLU). If node representations are collected into a matrix X l ∈ R D l ×K , the convolutional operation can be written as:

EQUATION

)

where Ã is symmetrically normalized from A in conventional GCNs. A ∈ [0, 1] K×K is the adjacency maxtrix of G, and we have a ij = 1 for node j ∈ N (i) and a ii = 1.

FiLM. Feature-wise linear modulation (FiLM) transformation approach (Perez et al., 2017 ) is a general-purpose conditioning method for granting neural networks the reasoning ability based on conditioning information. Specifically, it modulate one feature via a simple, feature-wise transformation based on another domain feature. More formally, FiLM learns functions f and h which output γ i,c and β i,c as a function of condition encoding x i :

EQUATION

Image Encoder:

Faster R-CNN

Quenstion Encoder:

"What animal in the box?"

Object Features:

Step 1

RA-GCN:

Step 2

Step 3

Step where γ i,c and β i,c modulate features F i,c , whose subscripts refer to the i th input's c th feature, via a feature-wise affine transformation:.

F iLM (F i,c |γ i,c , β i,c ) = γ i,c F i,c + β k,c .

(4)

3.2. RA-GCN MODULE:

The RA-GCN module takes as input a set of K visual features v i ∈ R dv , along with their bounding box coordinates b i . Before using a GCN filter to learn representation for each object, we need construct a graph (adjacency matrix) of image objects based on those relationships among objects.

It is natural that graph convolutions should focus not only on the objects, but also on the object ralationships that are the most relevant to the question. Hence, we first provide a graph learner module to introduce a sparser graph, in which relationships are learned based on question-guided edge features. With the learned question-specific sparse graph, we provide a novel GCN filter which not only learns to capture semantic information between objects but is also able to perform conditional visual reasoning.

Graph learner:

The goal of this sub-module is to produce a sparse graphical representations of objects based on a question. Specifically, we learn to define the most k-relevant nodes for each node as its neighbours based on question. This sparsity constraint make sense as most VQA questions requires attending only to a small subset of the graph nodes and it can also reduce the amount of computation. As the interactive dynamics between different objects is very crucial for answering semantically-complicated questions. Here, we try to exploit both geometric and semantic relationships between objects via explictly constructing edge features concated by question-adaptive semantic edge features and geometric edge features. Such edge features allows the storing of more specific information about what precise characteristic of a particular relationship is important in a given question context.

We seek to construct a adjacency matrix A ∈ R K×K and each a ij resprents the importance of edge e ij . For building semantic feature of each edge e ij ∈ E, we first fuse the neighbour node features v i and v j as F use(v i , v j ). Then we choose the FiLM method to modulate these fused features based on question feature q as, e s ij = F iLM (F use(v i , v j )|q). The reasoning ability of FiLM layer can ensure that the semantic edge features is question-specific. Zhuang et al. (2017) has shown that the relative spatial feature between two objects is a strong representation to encode their relationship. Similarly, we model the geometric edge features between node v i and v j based on their relative spatial information. Suppose the top-left coordinate, bottom-right coordinate, width and height of node v i are represented as [x tli , y tli , x bri , y bri , w i , h i ], then the geomtric edge features are represented as, e g ij = [

x tl i -x tl j wi , y tl i -y tl j hi , After constructing edge features, we provide a simple discriminative network F e consisting of two fully-connected layers and a softmax layer to measure its importance a ij = F e (e ij ). Such definition does not impose any constraints on the graph sparsity, and therefore yield a fully connected adjacency matrix. Like the works (Norcliffe-Brown et al., 2018b) (Gao & Ji, 2019) , we adapt ranking strategy for each node to learn a sparse heighbourhood system:

x

EQUATION

where topm returns the indices of the m largest values of an input vector, and a i denotes the i th row of the adjacency matrix. For each row a i , we choose a non-linear function to normalizes the subset {a ij , j ∈ N (i)}, have a ii = 1 and a ij = 0 for j ∈ N (i). In other words, the neighbourhood system of a given node will correspond to the nodes which it has the strongest connections to.

Conditioned Graph Convolution:

Given a question specific graph structure, we then exploit a novel graph convolution filter to learn new object representations that are informed by performing visual reasoning among a neighbouhood system tailored to answer the given question. Except for adding the reasoning ability, we enhance the conventional GCN's representation power by addressing the limitaion of shared transformation matrix for each node.

First, motivated by Perez et al. (2017) , we use the FiLM layer to update each node features in the context of question as F iLM (X l |q) ∈ R D l+1 . Although such FiLM transformation is still shared for each node, the question specific way enables the learning of each node's representation to explore the most relevant information with question and fuse question information into node features. Then the Eq. 2 is transformed to:

EQUATION

Second, in order to make the graph convolution work on nodes with arbitrary topologies, the learned kernel matrix W is shared for all nodes. In contrast, CNN learns a different transformation matrix for each position inside the kernel and owns expressive power to model feature patterns. By building on the concept of CNN, we try to adapt node-specific kernels in our graph convolution filter. Su et al. (2019) solves the limitation of spatially shared weights in CNN via multipling filter weights with a spatially varying kernel. The spatially varying kernel is generated by learnable, locally pixel features. Motivated by this, we use the edge features to genearate node-specific kernel parameters, which are spatial sensitive.

Specifically, we use a simple neural network F w to convert edge features into kernel parameters as m ij ∈ R D l+1 and then perform channel-wise multiplication with updated node feature F iLM (x l j |q). The generated kernel parameters not only make the transformation of node features node-adaptive, but also can fully explore semantic interaction as well as graph structures information into node representation. The network F w consists of two fully-connected layers and a tanh layer which limits the element of weight vector to range [-1, 1] to avoid increasing the scale of output features. All the kernel parameters can be collected into a tensor P ∈ R K×K×D l+1 . Then Eq. 6 is transformed to:

EQUATION

where represents channel-wise concatenation, F iLM d outputs the channel d of output features and M d ∈ R K×K is the d-th slice across channel in M.

The learnable weighting matrix based on edge features introduced in Zhao et al. ( 2019) is the most related work to ours but with following sharp difference. Our design is based on the concept of CNN that has espressive power to model feature patterns because the kernal parameters can be both positive and negative. So we use a tanh layer to generate the kernel paramters. However, Zhao et al. (2019) use edge features to genearate a wighting mask which is always positive as they use a sofamax layer. As a result, our model is more similar to CNN and owns better capability to fit the data mapping, showed in Section.

3.3. NETWORK ARCHITECTURE:

Our network architecture mimics a simple form of iterative reasoning by leverage the reasoning power of our RA-GCN module to iteratively adjust visual features based on question information to answer questions. As we can see in Figure 2 , the object features {v i } are updated by RA-GCN module through multiple steps. More specifically, for each step t = 1..T where T is the total number of steps, a RA-GCN module processes and updates the object features as follows:

EQUATION

At step t = T , the object representations {v t i } are aggregated with a global pooling operation to provide a single vector v ∈ R dv . This vector contains information about all the objects, their spatial and semantic relations conditioned on questions for current step T .

Like the CNN architecures, the information about the inputs or gradients can vanish when our architecture becomes increasingly deep. We opt to use dense-wise connections proposed in Huang et al. (2017) to concatenate all the global pooled features from each RA-GCN module as the last vector feature. Without fusing this vector feature with questin information, we directly use a classifier layer consisting of 2-layers MLP with ReLU actiations to predict the answer.

We stress that our model relies solely on RA-GCN module to use question information to modulate the object features. This method distinguishes itself from previous methods (Cadene et al., 2019 ) (Hu et al., 2019) (Li et al., 2019 ) Norcliffe-Brown et al. (2018b) which fuse object features and question information into a single embedding via element-wise produce, concatenation, attention, and/or more advanced methods.

4. EXPERIMENTS:



4.1. EXPERIMENTAL SETUP:



Datasets::

We evaluate our proposed model on VQA-CP v2 (Agrawal et al., 2018) , GQA (Hudson & Manning, 2019) and Clevr (Johnson et al., 2017) datasets for question-conditioned visual reasoning. Although VQA 2.0 (Goyal et al., 2017) dataset is the most used dataset, it still contains some kinds of language bias. Here we adapt VQA-CP v2 dataset, a derivation of the VQA 2.0 (Goyal et al., 2017) dataset, which was introduced to evaluate and reduce the question-oriented biases in the VQA models. In particular, the distribution of answers differs between training and test splits. This dataset can denonstrate the generalization ability of our model. We provide a fine grained analysis on the test set. Then, we use the GQA dataset to further demonstrate our model's visual reasoning capacity as the VQA dataset pays less attetion to reasoning, since 19.5% of its questions have relations, 8% have spatial reasoning questions and only 3% have compoistional questions. while, the GQA dataset makes much effort on generating questions that need multi-step reasoning and balancing the answer distributions to overcome the question-condition biases. Finally, we also use the Clevr dataset to construct a more detailed analysis of our model's performance on very complicated relational questions, such as what number of other objects are there of the same size as the brown shiny object. The (Shrestha et al., 2019) appoints that a good VQA model should be capable of datasets across from natural images to synthetic datasets that test reasoning and most methods do not generalize across the two domains.

Hyper-parameters::

No matter which dataset we use, we both adapt the same GRU based question encoder provided in Norcliffe-Brown et al. (2018b) . For the VQA-CP v2 dataset, we use the Bottomup features provided by Norcliffe-Brown et al. (2018b) to represent our image as a set of 36 localized regions. We also use the same encoder from Norcliffe-Brown et al. (2018b) to embed the question tokens. For the GQA dataset, we adopt the same preprocess operation to process dataset as the Hu et al. (2019) . We use the object detection features of size N det × 2048 (where N det is the number of detected objects in each image with a maximum of 100 per image) and object bounding box coordinates provided by GQA dataset itself. The embedding process of question tokens is same as Hu et al. (2019) . For the Clevr dataset, we use the 14 × 14 × 1024 convolutional grid features extracted from the C4 block of an ImageNet-pretrained ResNet-101 network (He et al., 2016) as the local features x l oc (i.e. each x l oc is a 1024-dimensional vector and K = 196). We adopt the same embedding process as Perez et al. (2017) to embed question tokens. For all datasets, we both use Adam as optimizer, set batchsize as 64 and learning rate as 3e -4. 

Ablation study:

For all the ablation studies, we set iterative reasoning steps as three since such setting make us achieve the state-of-the-art performance in VQA-CP v2 dataset.

In Table 1 , we compare four ablated instances of RA-GCN module. First, we validate the benifits of FiLM transformation. Removing it from RA-GCN module will lead huge loss of its accuray acrossing all the datasets. As our network does not include multimodal fusion module, removing the FiLM transformation will let our model lack question information. It make sense that the performance will decreases a lot while lacking FiLM layer. Second, we validate the benifits of FiLM transformation. When the FiLM transformation layer is added, using the node-specific kernel parameters will leads to higher accuracy on every datasets.

Number of reasoning steps:

In Table 2 , we perform an analysis ot the iterative process. We train four different RA-GCN networks on the VQA-CP v2 train split, each with a different number of iterations over the RA-GCN module.

Performance is reported on test split. Networks with two and three steps respectively provide a gain of +1.2 and +2.5 in overall accuracy over the network with a single step. An intersting aspect of the iteractive process of RA-GCN is that network with 4 steps reports a decrease in overall accuracy over the network with 3 steps while the amount of parameters increase. The reason behinds this maybe is a GCNs with multiple convolutional layers will suffer from an over-smoothing problem (Luan et al., 2019) .

4.3. STATE OF THE ART COMPARISON:



VQA-CP v2:

In Table 3 , we compare our model to the most recent contributions on the VQA-CP v2 dataset and our RA-GCN model achieves a new remarkably overall state-of-the-art performance. We observe that RA-GCN provides a substantial gain over other methods. Given the different distribution between train and val splits, models that only focus on linguistic biases to answer the question are systematically penalized on their test splits. This property of VQA-CP v2 implies that our RA-GCN method is less prone to question-based overfitting.

Among these methods in the Table 3 , LCGN and LCGS models are trained by ourself as they have not been originly evaluated on the VQA-CP v2 dataset. The LCGN method surports realtional reasoning and improves performance across GQA and Clevr datasets. The LCGS approach combines a graph learner and spatial graph convolutions, which is the mostly similar to our method. So we train them using Bottom-up region representations to fully demonstrate our method's effectivness. Interestingly, our model surpasses MUREL, ReGAT and LCGN methods, which correspond to some of the latest development in relational reasoning models for VQA. This tends to indicate our RA- (Cadene et al., 2019) 39.54 ReGAT (Li et al., 2019) 40.42 LCGN (Hu et al., 2019) 40.21 RA-GCN (ours) 42.3 GQA In Table 4 , we compare our model to the contributions on the GQA dataset. CNN+LSTM and Bottom-Up are simple fusion approaches between the text and the image, using the released GQA object detection features respectively. The MAC model is a multi-step attention and memory model with specially designed control, reading and writing cells, and is trained on the same object detection features as our model. Our approach outperforms the MAC model that performs multistep inference. This shows our RA-GCN provides much stronger visual reasoning ability on GQA dataset. Finally, even though we did not extensively tune the hyperparameters of our model, our overal score on the val split is highly competitive with state-of-the-art methods.

Clevr One of the core aspect of VQA models lies in their ability to handle different datasets, especially with cross datasets of natural images and synthetic images that many methods do not generalize well. In Table 5 , we compare our model to the contributions on the Clevr dataset. Although our model only achieve 90.0 score which is much less than LCGN model's 97.9, there may be two factors in our model affecting our accuray. The first factor is our RA-GCN module is originly designed to handle graph inputs, not the convolutional grid features provided in Clevr dataset. The size of convolutional grid features is 14 * 14 * 1024, which leads to 196 objects if we treat the grid features as object features. Due to the limitaion of our gpu resources, we can not train our model to handle 196 objects per image at the same time. So we use a convolution filter with stride 2 to downsample the grid features to size 7 * 7. For above reasons, it is not fair to compare our result with other methods. However, our model still achieves 90.0 scores, meaning our RA-GCN model is able to perform visual reasoning well.

5. CONCLUSION:

In this paper, we introduced RA-GCN, a resoning-aware graph convolutional network for Visual Question Answering task. Our system is based on the concept that FiLM layer can infer neural networks reasoning ability and node-sensitive transformation matrix can augments the expressive power of GCNs. To the best of our knowledge, RA-GCN is the first GCN designed for visual reasoning tasks.

We validated our approach on three challenging datasets: VQA-CP v2, GQA, and Clevr. We exhibted various ablation studies to validate our RA-GCN model. Our final RA-GCN network achieves 

Model convolutional grid features Accuracy:

LCGN (Hu et al., 2019) 97.9 MAC (Hudson & Manning, 2018) 98.9 RA-GCN (ours) 90.0 state-of-the-art performance in VQA-CP v2, and is very competitive with state-of-the-art performance on GQA. Additonally, our model is capable of handling both datasets of natural images and synthetic images.

:



Figures and Tables:

Figure 1: Overview of the framwork of the current Relational reasoning networks based on graph networks.

Figure 2: RA-GCN network. The RA-GCN network performs visual reasoning by iterating through several RA-GCN modules and concatenates all the global pooling information from each module as the last vector feature for classification.

br i -x br j wi , y br i -y br j hi , wi * hi , wj +hj wi+hi ]. Finally, the edge features are represented as e ij = [e s ij , e g ij ].

Ablation study of RA-GCN on the impacts of FiLM transformation and node-specific kernel paramerters

Table content: <table><tr><td colspan="5">FiLM transformation node-specific parameters VQA-CP v2 GQA Clevr</td></tr><tr><td/><td/><td>42.3</td><td>62.4</td><td>90.0</td></tr><tr><td/><td>×</td><td>40.3</td><td>59.3</td><td>87.6</td></tr><tr><td>×</td><td/><td>32.5</td><td>43.2</td><td>65.6</td></tr><tr><td>×</td><td>×</td><td>31.3</td><td>44.7</td><td>63.2</td></tr></table>

Number of iterations. Impact of the number of steps in the iterative process on VQA-CP v2 test split.

Table content: <table><tr><td>Model</td><td colspan="4">1 step 2 steps 3 steps 4 steps</td></tr><tr><td colspan="2">RA-GCN 39.8</td><td>41.0</td><td>42.3</td><td>41.6</td></tr><tr><td>4.2 MODEL VALIDATION</td><td/><td/><td/></tr></table>

State-of-the-art comparison on the VQA-CP v2 dataset. Results on test split. All these models were trained on the same training set.

Table content: <table><tr><td>Model</td><td>Bottem up Accuracy</td></tr><tr><td>LCGS (Norcliffe-Brown et al., 2018b)</td><td>39.4</td></tr><tr><td>MuRel</td><td/></tr></table>

State-of-the-art comparison on the GQA dataset. Results on val split. All these models were trained on the same training set.

Table content: <table><tr><td>Model</td><td colspan="2">object features from detection Accuracy</td></tr><tr><td>CNN+LSTM (Hudson &amp; Manning, 2019)</td><td/><td>49.2</td></tr><tr><td>MAC (Hudson &amp; Manning, 2018)</td><td>×</td><td>57.5</td></tr><tr><td>LCGN (Hu et al., 2019)</td><td/><td>63.8</td></tr><tr><td>RA-GCN (ours)</td><td/><td>62.4</td></tr></table>

State-of-the-art comparison on the Clevr dataset. Results on val split. All these models were trained on the same training set.

Table content: <table/>

Peer review (iclr2020):
Overview In this paper, the authors proposed a new version of graph convolutional network for visual question answering. To enable the “reasoning” ability for visual question answering model, the authors proposed a new module called reasoning-awere GCN (RA-GCN). In RA-GCN module, it has a graph learner to learn the connections for visual graph extracted from the image. Then, a conditional graph convolution is performed to modulate the node features in the visual graph based on the question and then update the feature for each node specifically based on the edge context. Based on this RA-GCN module, the authors further perform an iterative process to perform visual question answering. In the experiments, the authors worked on three datasets, VQA-CP, GQA and CLEVR. On these three datasets, the proposed method achieves better or comparable results to previous SOTA. A detailed ablation study further helps to understand the RA-GCN model more closely. Pros: 1. The authors proposed a new module called RA-GCN for visual question answering. In this new module, the authors used a graph learner to learn the structure of the visual graph, and then used a conditioned convolution to update the node features in the graph. 2. In the experiments, the authors evaluated the proposed model and previous model on three datasets and achieved better or comparable performance compared. The ablation studies further helped to understand the model more clearly Cons: 1. It is not very clear why the proposed model is called reasoning-aware GCN. Though the proposed RA-GCN module perform message passing across graph conditioned on language, i do not think it has learned to perform reasoning for visual question answering. In this sense, the claimed contribution is a bit exaggerated. 2. The difference between the proposed model and other graph-based visual question answering model is not very clear to me. The graph learner in RA-GCN is similar to the dynamic connections learned in LCGCN, while the conditioned convolution is also used on LCGCN. 3. The motivation for RA-GCN is not clear to me. First, why the authors used a graph learner to prune the edges in the graph. Pruning the graph seems not reliable without any supervision on the connections. Second, why the authors used a node-specific updating by introducing a edge tensor. As it is in GCN, a straightforward way is directly regard the edges as nodes in the graph and then perform graph convolution to propagate the context from edges to object nodes. Also, the description of RA-GCN module is vague at some points. First, it is not clear what the Fuse function is in graph learner. Second, it is also not clear how to get the tensor P for node-specific graph convolution. 4. In experiment, it is shown that the proposed model either outperforms or underperforms previous methods. On VQA-CP v2, all models achieved close performance. On GQA and CLEVR, the proposed model is worse than state-of-the-art. Based on the above comments and the experimental results, it is hard to justify the novelty and significant contributions behind the proposed model. What are the main merits in the proposed model is not clear to me. 5. Finally, it would be great if the authors can present some visualization results for the proposed model, showing how the graph is pruned and how the node feature updating is modulated. These would help the reader to understand the “reasoning” process behind the model. Conclusion: In this paper, the authors proposed a new module called reasoning-aware GCN for visual question answering. It is reasonable to use question to modulate the graph convolution. However, from the method description and the experimental results, it is hard to verify the novelty and significant contributions behind the proposed model. I would suggest the authors explain the motivations behind the model and explain it more clearly. Also, the authors should present some visualization results for the VQA task.