Robust Visual Reasoning via Language Guided Neural Module Networks
Authors: Arjun Reddy Akula
Varun Jampani
Soravit Changpinyo
Song-Chun Zhu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Module Networks
Adaptive Convolutions

Abstract: Neural module networks (NMN) are a popular approach for solving multi-modal tasks such as visual question answering (VQA) and visual referring expression recognition (REF). A key limitation in prior implementations of NMN is that the neural modules do not effectively capture the association between the visual input and the relevant neighbourhood context of the textual input. This limits their generalizability. For instance, NMN fail to understand new concepts such as “yellow sphere to the left" even when it is a combination of known concepts from train data: “blue sphere", “yellow cube", and “metallic cube to the left". In this paper, we address this limitation by introducing a language-guided adaptive convolution layer (LG-Conv) into NMN, in which the filter weights of convolutions are explicitly multiplied with a spatially varying language-guided kernel. Our model allows the neural module to adaptively co-attend over potential objects of interest from the visual and textual inputs. Extensive experiments on VQA and REF tasks demonstrate the effectiveness of our approach. Additionally, we propose a new challenging out-of-distribution test split for REF task, which we call C3-Ref+, for explicitly evaluating the NMN’s ability to generalize well to adversarial perturbations and unseen combinations of known concepts. Experiments on C3-Ref+ further demonstrate the generalization capabilities of our approach.

Robust Visual Reasoning via Language Guided Neural Module Networks
Arjun R. Akula', Varun Jampani', Soravit Changpinyo', Song-Chun Zhu UCLA Center for Vision, Cognition, Learning, and Autonomy, 'Google Research
'Beijing Institute for General Artificial Intelligence (BIGAI) "Tsinghua University, *Peking University
aakula@ucla. { (varun jampani, schangpi) (eaaaaanniiigooooooooooeooo @google com
8 c. zhu@pku. edu.cn en
Abstract
Neural module networks (NMN) are a popular approach for solving multi-modal tasks such as visual question answering (VQA) and visual referring expression recognition (REF). A key limitation in prior implementations of NMN is that the neural modules do not effectively capture the association between the visual input and the relevant neighbourhood context of the textual input. This limits their generalizability. For instance, NMN fail understand new concepts such "yellow sphere to the left" even when it is a combination of known concepts from train data: "blue sphere", "yellow cube". and "metallic cube the left". this paper, address this limitation by introducing language tanguage-guided adaptive convolution layer (LG-Conv) into NMN, in which the filter weights of convolutions are explicitly multiplied with a spatially varying Language-guider kernel. Our model allows the neural module to adaptively co-attend over potential objects interest from the visual and textual inputs. Extensive experiments VQA and REF tasks demonstrate the effectiveness of our approach. Additionally, we propose new challenging out-of-distribution test split for REF task, which call C3- Ref+, for explicitly evaluating the NMN's ability to generalize well to adversarial perturbations and unseen combinations of known concepts. Experiments C3- Ref+ further demonstrate the generalization capabilities of our approach.
Introduction
Visual question answering (VQA) [1] 8 7 and visual referring expression recognition (REF) [37 27 are fundamental language-to- vision matching tasks that have several downstream applications such as robot navigation, image retrieval, and natural language interfaces
39 The high- level goal of these tasks is to perform joint reasoning over visual and textual queries. In the recent years, neural module networks (NMN) [10 20 33 attracted increasing attention due to their superior performance on these tasks [33 26). Briefly, NMN models learn to parse textual queries as executable programs composed of learnable neural modules. Each of these modules implements a single step of reasoning (e.g. count, filter, compare) and are dynamically assembled to perform multi-step reasoning over text. In addition to the good performance, NMN also provide high model interpretability thanks to their transparent, hierarchical and semantically motivated architecture [45 6 32).
42
50
Despite great success, the current NMN implementations require large amount of training data and are less effective in generalizing to unseen but known language constructs (2 I3 51. For example, NMN fail to understand new concepts such as "yellow sphere the left" that are constructed using a combinations of known concepts from train data such as "blue sphere", "'elloo cube". and "metallic
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

filter_ shape eylinder)
ReLU
The first metallia
Guidance Encoder
Conv
cylinder from right
ReLU
LG-Conv
ReLU
W
LG Conv
Figure I: An example from the CLEVR-Ref+ dataset. Existing NMN implementations only provide the visual features (Um) as inputs to the neural modules. In this work, we additionally condition each module on textual expression (ạ) by replacing the standard convolution layers with content adaptive convolution layers LG-Conv which modify the convolution by explicitly multiplying the filter weights (W) with spatially varying language-guider kernel G. x denotes element-wise multiplication and + denotes summation.
cube to the left". One of the main reasons for this that the neural modules existing works either use a shallow, indirect language guidance 40) 19 2 or pre-define the textual inputs in the module instantiation 226 33,, ignoring the rich correlations among the visual inputs and the relevant context from the textual inputs. For example, the neural module that filters based on the object size, "filter_sis _iee ( (smalli smallest)) needs to localize a tiny sphere or medium sized sphere in the image depending on the object relationships in the expression (e.g. "the smallest thing among the spheres" vs. "the metallic sphere smaller than all the large cylinders") and the different sizes of spheres and cylinders available in its visual input. We believe that explicitly conditioning the neural modules on the joint textual and visual context helps in inferring robust visiolinguistic relationships which further enhances the compositional reasoning skills.
In this work, we address the aforementioned issues by explicitly providing the relevant objects and relationships in the textual expression to neural modules. To do this, as shown in Figure [ we replace the standard convolution operations in the neural modules with novel language- adaptive convolution operation, which we call LG-Conv. More specifically, the filter weights W of LG-Conv are explicitly multiplied with spatially varying language-guie guided kernel G, which allows the module to adaptively co-attend over potential objects of interest from the visual input and textual input by altering the convolution. Although content-adaptive convolutions 22
15 44 are used in several vision tasks, we are not aware of any prior works that does this filter adaptation using language as guidance. We propose two novel and effective methods namely, bi-salient attentional guidance (BiSAtt) network and co-salient attentional guidance guidance (CoSAtt) network to learn the guidance kernel from textual and visual inputs.
We conduct extensive experiments on VQA and REF tasks using CLEVR 25 CLOSURE B3]
and CLEVR-Reft (33] datasets. On the recently released VQA benchmark CLOSURE B3, our approach significantly outperforms all the previous works with 11.6% improvements in accuracy. On the REF benchmark CLEVR-Ref+ (33, we outperform competing approaches by as much as 49.8% accuracy single-referent split (S-Ref) and +4.7% on full-referent split (F-Ref), suggesting the importance of language-guiie guidance. Most significant gains with S-Ref, which consists of only 30% of the training data in CLEVR-Ref demonstrate the superior generalization of our model in learning

---

ReLU
ReLU
I q Marg
Guidance Encoder
Conv
LG-Conv
ReLU
ReLU
Guidance Kernl (G)
Conv
LG-Con Conv
ReLU
ReLU
Conv
LG-ConV
(a)
Vm
(b)
Vm
Figure 2: (a) Architecture of neural module (m) in existing NMN 126 consuming visual input Um (b) Our proposed architecture replacing Conv layers with content adaptive convolution layers guided by the input image I, input query q and parameterized textual input marg.
from fewer training samples We further evaluate the generalization capabilities of our approach by collecting dataset consisting of unseen compositions and contrasting samples for CLEVR-Ref+ benchmark [17 and call our new dataset C3-Reft.
Our key contributions summarized as follows:
1. We propose novel language-guide adaptive convolution layer for NMN that guide modules in adaptively selecting informative visiolinguistic relationships and in attending to relevant objects of interest from the visual and textual inputs;
2. We demonstrate the superiority of our approach by achieving new state-of-the-ans results on multiple tasks and benchmarks;
3. We introduce new benchmark explicitly test the model's ability generalize to adver- sarial perturbations and novel compositions of concepts unseen during training. We show that our model is more robust and generalizable compared to previous approaches.
Related Work
Multi-modal Grounding. Early approaches for tackling grounding tasks such as VQA and REF used recurrent networks with CNNs and attention- models to jointly understand visual and
language inputs [6 48 34).
{ proposed bottom-up and top-down approach to learn attention over image regions obtained from pretrained object detector. However, these models are shown to be heavily driven by annotation artifacts the training data []. Balanced and synthetic datasets such as CLEVR (25), CLOSURE [13 CLEVR Ref+
33 are proposed by explicitly controlling the bias seeeiii-then--raaaee
approach, have shown superior performance these datasets. However, these models fail to learn robust visio-linguistic representations and are shown to exploit the imbalanced distribution the train and test splits [5] [4]
and language priors. Transformers [36]
47 using
Neural Module Networks. Neural module networks (NMN) leverage specialized modules to com- pute basic reasoning tasks. These modules can be assembled to perform complex and compositional reasoning I0 25 26 19] 9 proposed dynamic NMN that learns and adapts the structure of the
execution layouts to question. Recently, (26) proposed homogeneous (IEP) and generic neural modules, unlike fixed and hand-crafted neural module, in which the semantics of each neural module Iearnt during training. IEP model achieves promising performance on CLEVR dataset. [33 proposed IEP-Ref by extending IEP model to ELEVR-Ref+ dataset and outperformed all the prior
S Ref is subset of full CLEVR- REf+ dataset containing expressions that refer to only single target object in the image.

---

Visual Attention
8X20X20
BiSAtt Guidance Kernel (G)
Linear
128
128
LSTM
LSTM
Visual Attention
Filter Bank
Language Attention
Blue
Module Argument
Input Image 128X20X20
Are
there
blue
Input Query
Bi-Salient Attentional Guidance Encoder (BiSAtt): In BiSAtt Encoder architecture, we first encode text inputs and then use learn set of adaptive weights to linearly combine the basis filters which produces the convolution filters applied on input image.
Figure 3:
works. Although, compositional by design, these models lack robust generalization abilities and fail to ground novel combinations of known linguistic constructs [13]. Few implementations of NMN such FiLM 40] and N2NMN [9] condition the neural modules textual guidance. However, the visiolinguistic context in these modules is rather shallow as they cannot jointly co-attend over potential objects of interest directly from the visual input and textual inputs. The major difference between our work and the prior works of NMN is that we explicitly condition the neural modules on aanuuag--uiddacee by directly altering and adapting the convolution operation.
Adaptive Convolutions. There is an extensive literature on content-adaptive convolutions, in which standard 2D convolutions are generalized high-dimensional convolutions 112
23 41 [8]. [23)
proposes to use bilateral filtering layers inside CNN architectures. [24] proposed dynamic filter
networks in which input-specific custom filter weights are predicted by different network branch. 44 proposed pixel-adaptivy convolutions which modifies the filters in position-specie fashion. While these prior works study adaptive convolution techniques in CNN representation tasks, has not been explored before for guiding NMN for multi-modal language- vision tasks. Closest to our work is the FiLM (40] model that learn parameters for scaling up or down the CNN activations by conditioning on the textual input. Our model differs from FiLM primarily in using language guidance that depends on learnable, local pixel and textual features.
Approach
Problem Setup and Notation. Given image and natural language query q as input, our goal to develop NMN model that selects an answer E the query from a fixed set of possible answers. We generalize this notation for both VQA and REF tasks; g. denote question and a natural language answer respectively in VQA, whereas they represent a referring expression and a bounding box of the target object respectively in REF, We represent input image set of image regions
ordered sequence of (ro. F1. Ti...., TN) and the query as the set of words (wI, W2, ..., w.) where w; is the i-th word, N is the number of image regions extracted from input image 1, and L is the total number of the tokens in the input query.
Similar to (26], we use two-stage model for generating answer: (1) Program Generation Model p(2lg; 0,)): where the query parsed to z representing the reasoning steps required to answer query, and (2) Program Execution Model p(a z, I; 0.): where the predicted program z is used to
assemble a input-specific neural network that is composed from a of neural modules m and exccuted to produce distribution over answers.

---

8X20X20
CoSAtt
LSTM
Guidance Kernel
Co-Attention
Filter
Bank
Module Argument
*
Blue
on left
Co-Attention
Filter
Bank
Input Image 128X20X20
there
blue
Input
Query
Figure 4: Co-Salient Attentional Guidance Encoder (CoSAtt): CoSAtt Encoder jointly attends over the input image and text inputs (early fusion) identify co-salient regions and relationships in visual and language features that are contextually associated with each other.
As shown in Figure 2)a), , the neural modules in current implementations of NMN [25] 33 typically
standard Residual Convolution Block (RCB), consisting of convolution layers and ReLU activations. Formally, module (fx) of arity n receives feature maps (F₁) of shape 128 x 20 x 20 and outputs same-sized tensor fo Jn (F F2. F2...., F......F)) Fn).
hm = ReLU (conv1 (Fi))
fo ReLU (conv2 (hm) @ hm)
As we can see, these modules 25 26 are not explicitly conditioned on the input expression q, and therefore fails to extract robust visiolinguistic relationships. In contrast, as shown in Figure (b), we explicitly condition the neural modules on g. in addition to visual inputs, by replacing the standard convolution operations RCB block with a novel adaptive and language- guided convolution operation, which we call LG-Conv. Also, parameterize the module arguments, i.e., for example, we treat "filler Luatte__materialaa module as parameterized by argument (Marg) "rubber" instead of as standalone module "filter_mater mamariiiaaa rial [rubber]] ". As result of this parametrization. the number distinct set of modules used in the parameterized model reduce by 75%. We condition our LG-Conv layer on both query q and the module argument Mary (See Figure 2 In the following, we describe the LG-Conv operation and detail its formal specification.
3.1 Guidedd Convolution
Our high-level goal is empower the neural modules to learn adapting visiolingustic features from both visual and language inputs. We achieve this by introducing novel LG-Conv layers which allows potential objects of interest from the visual and textual inputs by altering the convolution [24] 15 [4] Formally, a standard conv layer in the RCB block performing
the module adaptively co-attend
spatial convolution operation the image pixels P (p1, .P.) is given
p Σ W(G W + jEN(
where W denotes the filter weights, C; denote the coordinates of the pixels in the image, b denotes biases, and (i)) defines convolution window. This convolution operation, with spatially shared weights, is agnostic to pixel features and independent of language features. As shown in Figure I
5

---

modify this to depend on both pixel features and language features using spatially varying guidance
kernel G as follows:
p = Σ G(gi Bi W(c; -cllps
(3)
JEN(I)
The spatial convolution W is adapted each pixel the visual input using the guidance kernel G. Similar to [44], we represent G' using a fixed parametric Gaussian: G(g., 9;) exp(-2 (a:
(g, (g--9))), where g represents guidance features that we learn using the following two methods (a) Bi-Salient Attentional Guidance (BiSAtt Encoder): We generate spatial guidance features using the architecture shown in Figure 3 The input image of dimensions 1 28 x 20 x 20, the input query 7. and the module's ' parametrized text argument are used in producing the guidance features. Specifically, in BiSAtt architecture, we add visual attention layers over I generate spatial guidance from non-spatial
(b) Co-Salient Attentional Guidance (CoSAtt Encoder): Here, we apply joint attention over q. and the module argument identify co-salient regions and relationships in visual and language features that are contextually associated with each other. The architecture is shown in Figure In comparison BiSAtt, as we show in our experiments, CoSAtt improves the relevance and interaction between objects in the image and the query. For efficient implementation, we use the same learned guidance across all the LG-Conv layers in a RCB block.
As our parametrized model require only a few number of modules, the total number of parameters in our NMN is significantly less compared to the state-of-the-ar models, even though the network parameters in our parameterized module slightly increase due to the additional conv and LSTM units in the guidance encoder.
Program Generator. We implement program generator using an attention-based sequence to se- quence (seq2seq) model with an encoder-decodes structure [46] 25 to map the input query q into an executable program 2. Both the encoder and decoder have two hidden layers with 256-dim hidden vector. Similar to [25] we convert the decoded sequence of program functions to syntax trees (in an in-order traversal) which each node contains RCB module.
Execution Engine. The execution engine assembles a neural network using the predicted program by mapping function each node in syntax tree to its corresponding neural module. The parent modules in the syntax tree takes the outputs from the child modules. Since we use a homogeneous architecture for designing our modules, the output generated from all modules is of same shape 128 x 20 x 20. We flatten the final feature map before passing it to multi-layer perceptron classifier, producing a distribution over all possible answers.
Training. During training, we find the optimal module parameters by maximizing the likelihood of the data. We optimize p(2|g;60 0p) using policy gradient method.
(Op))) = ElVlog (z) 0p) .r]
(4)
where is the reward and the expectation is taken with respect to rollouts of the policy. In order to enforce the network for generating the most accurate predictions, we then train the execution engine directly by maximizing log p(a) z, I, q: 0&)) with respect 0e.
Elooog p(2)q; 6p) log p(a|z,11 I, I.q:
(5)
Experiments
In this section, we start by discussing the datasets and the baselines considered in evaluating our approach. We show the superiority of our approach by performing quantitative and qualitative comparisons between our method and the baselines. We then demonstrate the importance of the proposed language guided adaptive convolution through ablation studies. Finally, we present our new benchmark, C3-Ref+, and show that our approach robust adversarial inputs and generalizes well to novel compositions.
We experimented more forms of guidance kernel discussed in [44, but we did not find significant improve- ments in NMN performance with these other kernels.

---

Model
CLEVR-Dec CLEVR Test CLOSURE
IEP-Ref [33
08.720.3
97.1 =0.2
59.8
FiLM 40
96.2
96.9
58.9
MAC [22]
99.1
98.2
71.6
Vector NMN 13
98.8
97.6
71.0
NS-VQA 49
99.2
99.4
76.4
LCGN [21
NA
NA
NA
ViLBERT 35
95.3
93.0
51.2
Visual BERT 35
96.0
92.8
50.6
Ours (with BiSAtt) Ours (with CoSAtt)
98.9 08.920.2 g2022
99.2=0.1
86.1 =0.1
98.9
99.2
88.0+002
Table 1: Performance of our approach and baselines CLEVR, CLOSURE benchmarks.
4.1 Datasets and Baselines
We evaluate our approach on both VQA and REF benchmarks. We use CLEVR 25 as the VQA benchmark, consisting of synthetically generated image and question pairs. Specifically, it consists of 100K images and 860K questions. We train our model on CLEVR train split and evaluate the performance on its val and test splits. In addition, using the model trained on CLEVR, we evaluate the performance on CLOSURE benchmark [33. consisting of novel compositions of objects and relations not seen in CLEVR train split. We then report results on CLEVR-Ref+ [33, a synthetic benchmark for referring expressions. contains nearly 0.8M referring expressions which 32% of expressions refer to only single object (Single-referent) and 68% refer to more than one object (Multi-referent). We refer the full dataset F-Ref and the single-referent subset as S-Ref.
We compare the performance of our approach against several NMN baselines such as FiLM (Feature- wise Linear Modulation), MAC 222 IEP-Ref [ VectorNMN 13, and non-NMN baselines such
as NS- AS-VQQA 49 LCGN 211 ViLBERT 35 and VisualBERT
30
Model
S-D
S-T
F-D
F-T
IEP-Ref
49.8 10.1
51 51.5 51..6
80.5 10.2
78.2
Model
CLS
S-T
F-T
FiLM
44.9
46.7
76.5
75.7
MAC
46.3
49.2
81.3
77.4
Ours
88.0
63.3
84.3
Vector NMN
53.5
83.2
77.1
C1 Ours-L+G
62.1
51.6
77.8
NS-VQA
51.5
52.9
82.5
79.6
C2 Ours-L-G
61.5
52.1
75.2
LCGN
46.8
48.0
77.0
74.8
C3 Ours+++(G w/o /)
80.2
57.7
79.9
ViLBERT
42.4
44.3
69.3
68.7
C4 Ous+L++G w/o q)
78.8
54.1
76.2
Visual BERT
41.7
43.2
69.8
63.2
C5 Ours+L+ w/o marg)
82.1
61.7
80.9
Ours (with BiSA
61.1
#0:3
59.7
20:2
87.2
83.5
Table Ablations. Performance our model with and without LG-Conv layer (L) and CoSAut encoder (G) on CLOSURE (CI.S), S-Ref Test, and F-Ref Test.
Ours (with CoSAtt)
62.3
63.3
89.1+0.2
89.1
$4.3
Table 2: Performance of language-guided NMN models and state-of-the-an models on S-Ref Dev (S-D), S-Ref Test (S-T), F-Ref Dev (F-D) and F-Ref Test (F-T).
4.2 Implementation Details
Similar to 25 we use 18K tround-truth programs to train the program generator (PG). We train PG and the execution engine using Adam [28) with learning rates 0.0005 and 0.0001, respectively. Our trained for a maximum iterations. We employ early stopping based on validation set accuracy. While reporting accuracies on S-Ref test split, we use the model trained on S-Ref train split. We repeat the experiment times on each benchmark and report the
32K iterations, while EE trained for a maximum of 450K
on each of them.
7

---

4.3 Evaluation
Table[aand SableeSsoow results in comparison with the baselines. We find that our model outperforms all prior work on CLOSURE, and CLEVR-Rets benchmarks, while showing on-par performance on CLEVR test split. This demonstrates the effectiveness of the proposed language guided convolutions in capturing visiolinguistic relations and contextual dependencies from the longer CLEEV-like expressions. In particular, we achieve +1 +11.6% in accuracy on CLOSURE test split compared to the best prior model Vector-NMN, indicating that model generalizes well to unseen compositions. The multi-modal transformer based approaches ViLBERT and VisualBERT performed poorly on both CLOSURE and CLEVR- Ref+, probably due to the mismatched image distribution in pre-training (with conceptual captions (Sl) and fine-tuning. Our model improves the accuracy on CLEVR- RReff test splits by 9.8% on S-Ref and 4.7% on F-Ref, compared with the current state-of-the-ar method IEP-Ref. Significant gains on S-Test also suggest the superior generalization skills of our model in learning from fewer training samples. Relatively more improvements with CoSAtt encoder compared to BiSAtt encoder shows that early fusion of image and text features facilitate in generating more robust guidance kernel.
To gain better insight into the relative contribution of the design choices we made, we perform experiment with the following five ablated models:
C1: Conv vs. LG-Conv (L). We investigate the contribution of the proposed content adaptive convolution layer in the RCB block by replacing LG-Conv layer with standard convolution. In this setting, we use guidance (G) from CoSAtt encoder for directly scaling up or down the CNN activations in the RCB block.
C2: Conditioning on CoSAtt Guidance (G). In this ablation, we use LG-Conv layers but skip the CoSAut encoder verify the importance of module level conditioning on the interaction between image and text features. We instead only pass the module argument (Marg) as guidance to the LG-Conv layer.
C3: CoSAtt w/o. Image (I) We encode guidance using only input query q and the module argument to test the importance of conditioning on image I in the CoSAtt encoder.
C4: CoSAtt w/o. Query (q) We encode guidance using only input image to test the importance of conditioning on input query q in the CoSAtt encoder.
C5: CoSAtt w/o. Module Arg (mxnp) In this variant, we keep q and I, but skip Mary in the CoSAtt encoder.
The results are shown in Table 3 As we can see, all the above five variants underperform, confirming the importance of our proposed adaptite convolutions and guidance kernel. Results show that module argument in the CoSAtt guidance has less significant effect compared to other components, suggesting that our model is able infer the semantic context of the module. Figure illustrate the qualitative differences of fil filter_com color (red) module trained using IEP-Ref and our approach. With IEP-Ref, the model selects all red objects from the image, ignoring the context in the expression. On the other hand, our approach correctly locates objects based on their contextual relevance.
4.4 The C3-Ref+ Dataset
we construct a new benchmark, C3-Ref+, to critically examine the generaliza- tion capabilities of NMN in grounding oft-oin (o.o.d) referring expressions a fundamental expressive power of human intelligence. Specifically, C3-Ref+ consists of two kinds of samples con- structed using S-Ref split of CLEVR-Ref+ dataset: (a) Novel Compositions, consisting of samples that evaluate the model on combinations of objects and their spatial relationships not seen in S-Ref train split; and (b) Contrast Sets, consisting of samples that help in exposing model brittleness by probing a model's decision boundary local to examples in the S-Ref test set. Table shows few examples.
Following
17 38
In constructing novel compositions of samples, we first extract all the combinations of simple and complex noun phrases (e.g. "The metallic object", "The metallic object to the left") from the R-Ref train split and then manually construct new expressions using the unseen pairs of these phrases. To construct a contrast sample, we manually perturb the semantics of various parts of the referring expressions in the S-Ref test split such that the ground-truth referent object changes. For example, we modify the expression first one of the tiny rubber thing from left to first one of the tiny metallic thing from right. We verify and validate the correctness of the collected samples and their ground-truth annotations using three human annotators. The annotations that are not consistent among the three
8

---

Ground Truth
TEP-Ref: filter color(red)
Our Model: filter_cee _ccoor(red))
Look the eyan object Look ryan objject that tovrl: Look the eyan object that the left of [red syliodrical thingles
the left of cylindrical thing
the left red sylindrical thing
?2: Find the large sphere to the right of r:: Find the large sphere the right of
r2: Find the large sphere to the right of thing
red
Ching
thing
f2
r3: The eyan thing behind the ismall red abject)
(::
The cyan thing behind the small Tobject
r3:: The cyan thing behind the Jobjent
Figure 5: Qualitative examples showing attention heatmaps of palter_colos (red) module outputs trained using IEP-Ref and our model on CLEVR-Ref+ dataset. el, e2, e3 highlight the red objects that are referred in the input expressions r1, r2, and r3, respectively.
Haarer
Figure 6: Accuracy of models on S-Ref and C3-Ref+
human annotators are removed from the final split We collected a total of 1412 expressions spanning 980 images with vocabulary of 56. The average length of expressions is 18.3. More than 60% of our collected expressions have or 3 relations such as "in the front" and "from the right". As shown in Figure performance of state-of-the-at models drop by up 18% on C3-Ref+. Our proposed method shows least drop (<5%) in performance indicating its superiority in grounding new unseen compositions and adversarial perturbations.
We also perform an additional round of filtering to remove inconsistent samples that could point more than object in the image.

---

Original: Any other yellow metal thing(s) of the same size the first one of the cyan metal thing(s) from right
C3-Ref+: Any other tiny thing(s) of the same size the first one of the brown metal thing(s) from left
Original: The shiny thing(s) that are left of the first one of the small sphere object(s) from right and behind the fourth one of the rubber object(s) from front.
C3-Ref+: The shiny thing(s) that are right of the first one of the small cubical object(s) from right and behind the fourth one of the rubber object(s) from front.
Original: Find matte thing that is on the left side of the cyan object that is behind the second one of the metallic object(s) from front; The last one of the object(s) from front that are behind it.
C3-Reft Find matte thing that is the left side of the cyan object that is behind the second one of the metallic object(s) from front; The second one of the object(s) from right that are in front of it.
Table 4: Random examples of contrast sets in C3-Ref+ and their original annotations in S-Ref.
Conclusion
Neural module networks (NMN) widely used in language and vision tasks. We show that explicitly conditioning neural modules on the language guidance through adaptive convolutions improve their grounding and generalization abilities, achieving new state-of-the-art results the visual question answering and visual referring expression recognition tasks. Our analysis on CLOSURE, CLEVR-Ref+ and compositional and contrastive split C3-Ref+ demonstrate that our proposed method enhances NMN' ability adaptively selecting and exploiting informative visiolinguistic relationships.
Broader Impact
This work contributes to improve the joint understanding of image and textual content, which in turn very important component in several vision and language grounding tasks. Our research can promote the development of multi-modal interaction system and facilitate people's daily lives. This work does not present any foresecable socictal consequence.
7 Funding Disclosure
Funding (financial activities supporting the submitted work):
None
Competing Interests (financial activities outside the submitted work):
None
References
[1] A. Agrawal, D. Batra, and D. Parikh. Analyzing the behavior of visual question answering models. arXiv preprint 1606 07356, 2016.
10

---

[2] A. Akula, R. Sangal, and R. Mamidi. A novel approach towards incorporating context processing capabilities nlidb system. In Proceedings of the sixth international joint conference on natural language processing, pages 1216-1222 2013.
[3] A. R. Akula, A novel approach towards building generic, portable and contextual nlidh system. Interna- tional Institute of Information Technology Hyderabad, 2015.
[4] A. R. Akula and S.-C. Zhu. Visual discourse parsing. ArXiv preprint, abs/ abss/100.00255,, 2019. URL pttpa://arxis orge /arg///b//a 1 903. 02252
[5] A. R. Akula, S. Gella, Y. Al-Onaizan, S.-C Zhu, and S. Reddy, Words aren't 't enough. their order matters: On the robustness of grounding visual referring expressions. In Association for Computational Linguistics (ACL), 2020.
[6] A. R. Akula, S. Wang, and S. Zhu. Cocox: Generating conceptual and counterfactual explanations via fault-lines. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Secoodd Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, AAAT/
2020, pages 2594-2601, AAAI Press, 2020. URL.tttp:: //a::://aaaaaee ottp:/aii.ro//////////////////// a Seticle/view/S55
[7] A. Akula, B. Changpinyo, Gong, P. Sharma, S.-C. Zhu, and R. Soricut. Crossvqa: Scalably generating benchmarks for systematically testing vqa generalization. 2021.
[8] P. Anderson, X. He. C. Buchler, D. Teney. M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086, 2018.
[9] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Learning to compose neural networks for question answering, arXiv preprint arXiv:1601.010 01705, 2016.
[10] J. Andreas, M. Rohrbach, T. Darrell, and D. Klein. Neural module networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 39-48, 2016.
[11] S. Antol, A. Agrawal, J. Lu, M. Mitchell, D. Batra, C. Lawrence Zitnick, and D. Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pages 2425-2433, 2015.
[12] V. Aurich and J. Weule. Non-linear gaussian filters performing edge preserving diffusion. In Mustererken- mung 1995, pages 138-545. Springer, 1995.
[13] D. Babdanau, H. de Vries, T. J. O' O'Donnell, S. Murty, P. Beaudoin, Y. Bengio, and A. Courville. Closure: Assessing systematic generalization of clevr models. arXiv preprint arXiv:/912.05775 2019.
[14] V. Cirik, Kirkpatik,,,,,, and L.-P. Morency. Using syntax to ground referring expressions natural images. In AAAI Conference on Artificial Intelligence, 2018.
[15] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 764-773, 2017.
[16] A. Fukui, D. H. Park, D. Yang. A. Rohrbach, T. Darrell, and M. Rohrbach. Multimodal compact bilinear pooling for visual question answering and visual grounding. 2016.
[17] M. Gardner, Y. Artzi, V. Basmova, J. Berant, B. Bogin, S. Chen, P. Dasigi. D. Dua, Y. Elazar, A. Got- tumukkala, al. Evaluating nlp models via contrast sets. arXiv preprint arXiv:2004.02777 2020.
[18] K. He, J. Sun, and X. Tang. Guided image filtering. In European conference on computer vision, pages 1-14. Springer, 2010.
[19] R. Hu, Andreas, M. Rohrbach, T. Darrell, and K. Sacnko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 104-813,,7 2017.
[20] R. Hu, M. Rohrbach, J. Andreas, T. Darrell and K. Saenko. Modeling relationships in referential expressions with compositional modular networks. In IEFE Conference on Computer Vision and Pattern Recognition, pages 11 111-1124, 2017.
[21] R. Hu, A. Rohrbach, T. Darrell, and K. Saenko. reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10294 10303, 2019.
graph networks for relational

---

[22] D. A. Hudson and C. D. Manning. Learning by abstraction: The neural state machine. In Proceedings of NeurIPS, 2019.
[23] V. Jampani, M. Kiefel, and P. V. Gehler. Learning sparse high dimensional filters: Image filtering, dense crfs and bilateral neural networks. In Proceedings the IEEE Conference Computer Vision and Puttern Recognition, pages 4452 15-224446.222 2016.
[24] X. Jia, B. De Brabandere, T. Tuytelaars, and L. V. Gool. Dynamic filter networks. In Advances in neural information processing systems, pages 667-675, 2016.
[25] J. Johnson, B. Hariharan, van der Maaten. L. Fei-Fei, C. L. Zitnick, and R. Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. IEEE Conf. Comput. Vis. Puttern Recog., 2017.
[26] Johnson, B. Hariharan, L. van der Maaten, Hoffman, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017.
[27] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[28] D. Kingma and J. Ba. Adam: method for stochastic optimization. Proceedings of ICIR, 2015.
[29] B. M. Lake and M. Baroni. Generalization without systematicity; On the compositional skills of sequence- to-sequence recurrent networks. arXiv preprint /71 2017.
[30] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsich, and K.-W. Chang. VisualBERT: simple and performant baseline for vision and language. arXiv preprint arXiv: 1908. 2019.
[31] L. Li, M. Yatskar, D. Yin, C.-J. Hsich, and K. K. Chang. Visualbert: A simple and performant baseline for vision and language, 2019.
[32] Q Li, S. Huang, Y. Hong. Y. Chen, N. Wu, and S.-C. Zhu, Closed loop neural-symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning. In International Conference on Machine Learning, pages 5884-589 5894, PMLR, 2020.
[33] R. Liu, C. Liu, Y. Bai, and A. L. Yuille, Clevr-ref+: Diagnosing visual reasoning with referring expressions. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 41 4185-4194, 2019.
[34] J. Lu, J. Yang. D. Batra, and D. Parikh. Hierarchical question-imet image co-attention for visual question answering, In Adv. Neural Inform. Process. Syst., 2016.
[35] J. Lu, D. Batra, D. Parikh, and S. Lee. ViLBERT: Pretraining task-agnostic visiolinguistic representations for assion-and-languaae tasks. In NeurlPS, 2019.
[36] J. Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task agnostii visiolinguistic representations for vision-and aar--and--aan and-language tasks. In Advances in Neural Information Processing Systems, pages 13 13-23, 2019.
[37] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension of unambiguous object descriptions. In IEEE conference on computer vision and pattern recognition, pages 11-20, 2016.
[38] Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.1 14599, 2019.
[39] A. Palakurthi, R. S M. A. Akula, and R. Mamidi. Classification of attributes in natural language query into different SQL clauses. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 497-506. Hissar, Bulgaria, 2015. INCOMA Ltd. Shoumen, BULGARIA. URL
[40] E. Perez, F. Strub, H. Vries, V. Dumoulin, and A. C. Courville, Film: Visual reasoning with general conditioning layer. In AAAI, 2018.
[41] S. Pouyanfar, Tao, A. Mohan, H. Tian, A. Kaseb, K. Gauen, R. Dailey, Aghajanzadeh, Y.-H. Lu, S.-C. Chen, et al. Dynamic sampling in convolutional neural networks for imbalanced data classification. In 2018 IEEE conference on multimedia information processing and retrieval (MIPR), pages 112-117. IEEE, 2018.
[42] Y. Qi. Q Wu, P. Anderson, X. Wang. W. Y. Wang, C. Shen, and A. van den Hengel. Reverie: Remote embodied visual referring expression in real indoor euvironments. arXiv preprint 10Xiv:190..01151 2019.
12

---

[43] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned. hypernymed, image all-text dataset for automatic image captioning. In Association for Computational Linguistics, pages 2556-2565, 2018.
[44] H. Su, V. Jampani, D. Sun, O. Gallo, E. Leamed-Millee and Kautz. Pixel-adaptive convolutional neural networks. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 11166-1177,, 2019.
[45] S. Subramanian, B. Bogin, N. Gupta, Wolfson, S. Singh, Berant, and M. Gardner. Obtaining faithful interpretations from compositional neural networks. arXiv preprint vrXiv:2005.00722 2020.
[46] Sutskever, O. Vinyals, Q. Le. Sequence to sequence learning with neural networks. Advances in neural information processing systems, pages 3104-3112, 2014.
[47] H. and Bansal. Lxmert: Learning cross pross-modality encoder representations from transformers. arXiv preprint Xiv: (7490, 2019.
[48] C. Xiong, S. Merity, and R. Socher. Dynamic memory networks for visual and textual question answering. In International conference on machine learning, pages 2397-2406, 2016.
[49] J. Wu, C. Gan, A. Torralba, Kohli, and Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. In Advances in neural information processing systems, pages 1031-1042, 2018.
[50] P. Young, A. Lai, M. Hodosh, and J. Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78, 2014.
[S1] C. Zhang, F. Gao, B. Jia, Y. Zhu, and S.-C. Zhu, Raven: A dataset for relational and analogical visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5317-5327, 2019.
13


Reviews:
[{"review_id": "zMGqgLafXyt", "reviewer": "Reviewer_e2WV", "summary": "This paper proposes a novel extension to the neural module network (NMN) to tackle vision-language tasks. The proposed method applies a language-guided convolution operator to the neural modules, which improves information exchange and fusion between the visual and language modes. The proposed framework is applied to various benchmarks in the visual question answering (VQA) and referring expression recognition (REF) domains. Experiment results demonstrate a superior performance of the proposed method.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The capability to process complex information from multimodal perception is a hallmark of the power of modern deep learning systems. This paper focuses on the vision-language tasks, a direction currently subject to huge interest in both research and application. The proposed method introduces a way to further enhance the guidance to the reasoning modules by language inputs, and is claimed to be the first model to adapt convolutional filters to languages. Large amounts of carefully conducted experiments show improvements in performance over a wide range of baseline models on both the VQA and RFE tasks. The paper is well written and easy to follow, despite missing a few technical details which will help readers from outside the community to understand better.\n\nMy challenges and question to this paper are listed below. No further experiments are requested in the rebuttal.\n\n1. The idea behind the design of both NMN models and the CLEVR dataset is to guide machines to learn \"universal\" neural modules that perform atomized logic operations through compositional reasoning. This means that the module parameters should not only be independent to the visual input, but also the language input. For example, in both \"What size is the red sphere?\" and \"How many large red objects are there?\" the \"filter_color[red]\" operator should map to the same logic operation regardless of the context input. This paper to some extend violates this design philosophy, therefore it becomes important to address how much the model is overfitted. \n\n2. The language dependency of NMN mainly falls on the inference of programs. Is the language-guided execution modules able to fine-tune an imperfect program generator (for example, in VQA, first use a few thousand programs to train a weak program generator then jointly fine-tune with RL)? This might also suggest how overfitted the model is.\n\n3. Any specific reason for choosing the Gaussian kernel? How do the results depend on the type of kernel and their sizes?\n\n4. Since all guidance kernels are shared by all modules, would \"modularizing\" the guidance kernels further contribute to the performance? \n\nOverall I think this paper has very strong potential and the experiments are solidly conducted. However, due to concerns about the model design, possible overfitting, limited novelty, I hereby give a \"borderline reject\" score.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations are briefly mentioned but not systematically discussed.\nSocial impact of this work is briefly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "jRePhyGfJvg", "reviewer": "Reviewer_Sgvd", "summary": "This paper introduces a language-guided adaptive convolution layer into neural module networks (NMN) to address the limited generalizability of NMN to unseen concepts. The proposed method is able to achieve on-par performance to state-of-the-art models on CLEVR VQA task while significantly outperforms state-of-the-art models on CLEVR-Ref+ and CopsRef.  The model is also able to generalize better to novel compositions of objects and relations in CLOSURE. \n\nIn addition, a new challenging out-of-distribution test split for REF task is introduced, to evaluate on model generalizability to adversarial perturbations and unseen combinations of known concepts. The proposed method has shown a clear advantage on this new test split. ", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOverall, the paper is well written, with extensive experiments conducted on a comprehensive list of datasets and tasks. The performance improvements from the proposed model are significant, especially on several referring expression recognition tasks. \n\nIn addition, the proposed dataset extends existing datasets of testing model generalizability to another dimension: adversarial perturbation. It can be potentially widely adopted by future works along this direction.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "PwPLQvgHOs", "reviewer": "Reviewer_vcX9", "summary": "The paper presents an incremental NMNs with a language-guided adaptive convolution layer into vanilla NMN. The goal o LG-Conv is to have a module that can adaptively co-attend over potential objects of interest from the visual input and textual input by altering the convolution.  \n\nTo achieve this, they present biSAtt and CoSAtt networks to learn the so-called guidance kernel G from textual and visual inputs. \n\nExperiments are done on multiple widely accepted visual reasoning benchmarking datasets, as well as a new benchmark that is designed to explicitly test the model's ability to generalize to adversarial perturbations. \n", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The power of NMNs were argued as leveraging specialized modules and could be assembled to perform complex and compositional reasoning. The decomposed module decouples the visual pathway and language pathway, forming an elegant structure. The argument of this work to me violates this design elegancy by agglomerate language signal again back into the convolution operations. The fact of achieving high performance on CLEVR-ref and others, as shown by the authors, could also be interpreted  CLEVR like dataset is way too artificial that could not reflect the power of NMNs following its original designing principle.  \n\nThis is being shown in Figure 4. The filter_color(red) is designed to be a modular operator only to locate red thing but not attending to other so-called objects with contextual relevance. By arguing the filter_color(red) should also consider contextual relevance goes against the NMNs principle. \n\nIf an incremental implementation does not follow a vanilla architecture's original designing principle but achieves better performance, how shall we as a community interpret it? \n\nMy other concern is with the CLEVR-Ref+ construction. For example, in figure 5, both E1 and E4 refers to an object that is to the left of yellow thing (and there is only one yellow thing). However, the GT for E1 is r1, and E4 is r4, one is to the left, and the other is to the right. My interpretation could be wrong, but from this single example, there seems to be inconsistent GT already. The authors mentioned three human annotators validate the creation of the Ref+ dataset. However, if the dataset fidelity is not well maintained, all the experimental results won't be convincing for making solid arguments. \n\nFurthermore, one argument of this work is \"NMNs fail to understand new concepts such\nas “yellow sphere to the left\" even when it is a combination of known concepts\nfrom train data: “blue sphere\", “yellow cube\", and “metallic cube to the left\".\"  I am wondering if the authors have verified in the new Ref+ dataset what is the percentage of such a combination of known concepts? How would you make sure when the human labeler is editing the phrases, they will create new ones that were not in other examples? \n\nOther places need clarification:\nIn some places, you use NMNs. Some use NMN. Would you please make sure to unify them? \nFigure 5, E3 is constructed from E1 and E2, but having a new phrase, \"big blue rubber thing\" instead of \"big blue metallic thing\". Are you also perturbing these phrases manually? Also, could you explain why the GT of E2 is <box:r2>. It doesn't seem to be correct to me. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "A brief broader impact narrative is provided. I do not see any explicit potential negative societal impact of this work. ", "review_text": "", "extra_scores": {"ethical_concerns": "None. ", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "Aao9-XQ3jl", "reviewer": "Reviewer_tcpV", "summary": "The authors introduce three new tools to improve compositionality in visual reasoning tasks:\n1. They apply pixel-adaptive conv nets [Su et. al., 2019] to the modules of Neural Module Networks. The guidance features, which weight the convolution computation, are computed via attention over the full image and text query. They present two different attention architectures to compute this feature: one separates the attention computation over vision and language and the other co-attends.\n2. A new CLEVR test set that is constructed from the S-ref split of CLEVR. The new test set consists of (a) object, spatial relationships not seen during training and (b) contrast sets [Gardner et. al., 2020].\n3. They parameterize modules by argument (e.g., `blue`) to reduce the number of modules to be learned.\nTheir model achieves competitive performance on CLEVR and improves over all baselines on the S-Ref and F-Ref splits. On their new split (C3-Ref+) they achieve the lowest drop in accuracy and maintain performance as the number of comparison relationships stays high.\n\nPixel-Adaptive Convolutional Neural Networks. Su et. al., 2019.\nEvaluating Models’ Local Decision Boundaries via Contrast Sets. Gardner et. al., 2020.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The paper brings together a variety of modeling techniques for the first time: an adaptive convolution together with attention computations over image and text queries all inside a neural module network.\n\nQuality: The evaluation was thorough, but some of the design decisions left me puzzled. The authors show results across a variety of splits, including a new split designed to probe robustness and appropriately ablate every aspect of their system. Their new system leaves me puzzled because it seems to break the benefits of neural modules by incorporating contextual input into the modules. If the benefit of the LG-Conv layer is to add textual relevance and whole image context, why use modules at all? For example, in their qualitative results, they show that their `filter_color(red)` module  better localizes the red cylindrical object in the scene compared to IEP-Ref, which just localizes every red object in the scene. Conceptually, the latter seems _more_ correct. The contribution seems to come more from the attention architectures rather than their interaction with the modules. To understand the contribution of attention further, it would be helpful for the authors to describe how ViLBERT and VisualBERT were applied to CLEVR and contrast it with their approach.\n\nClarity: The paper was clear when discussing background and experiment design. I found the method and motivation a little confusing. I also had to rely on the original pixel-adaptive conv paper to fully understand their changes (details under comments), but that did not affect the overall quality.  \n\nSignificance: The paper puts forth a fine-grained analysis of a large number of methods interacting together with the goal of modularity. These insights could be useful in developing more compositional modules across a variety of visual tasks.\n\nQuestions:\n- Equation (4): why is the optimization described as policy gradient? Is standard cross-entropy not appropriate?\n- Are Visual BERT and ViLBERT given whole image information or are they restricted to object features extracted from the image?\n\nComments to authors (not considered in review score):\n- The indexing in equation (2) is confusing. It would help to specify that these indices are 2-dimensional and to state explicitly that [] is an index operation.\n- L201, Table 1: ViLBERT and Visual BERT resolve to the same reference", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "In my view, the authors have adequately addressed the impact of their work.", "review_text": "", "extra_scores": {"ethical_concerns": "After reviewing the ethics guidelines, I believe there are no issues.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a new model for compositional reasoning with extensive evaluation on synthetic and  real image datasets. Additionally, the paper contributes a new challenging ref test set based on CLEVR which is manually verified. 

Given the interesting results and technical contributions the paper provides, I recommend to accept this paper under the expectation that the authors will revise the paper as promised in the author response, including:
1) Fix notational issues / references, consistent notation
2) Additional analysis using different types of kernel and sizes
3) Adding results on Modularizing the guidance kernels

Several reviewers raised the concern how "modular" this design is and how good the grounding is. I encourage the authors to include more of the discussion they provide in the author response to the final version of the paper. I think it would be very convincing to see a quantitative analysis on this, on the precision (and maybe recall for completeness as well) of the attention, i.e. a quantify the analysis of Figure 4, ideally comparing to baselines and prior work. I believe this should be reasonable easy given the annotations provided with CLEVR, and significantly increase the insight of this work and thus increase its impact.
Decision: Accept (Poster)