Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a language-guided adaptive convolution layer into neural module networks (NMN) to address the limited generalizability of NMN to unseen concepts. The proposed method is able to achieve on-par performance to state-of-the-art models on CLEVR VQA task while significantly outperforms state-of-the-art models on CLEVR-Ref+ and CopsRef.  The model is also able to generalize better to novel compositions of objects and relations in CLOSURE. 

In addition, a new challenging out-of-distribution test split for REF task is introduced, to evaluate on model generalizability to adversarial perturbations and unseen combinations of known concepts. The proposed method has shown a clear advantage on this new test split. 
main_review: 
Overall, the paper is well written, with extensive experiments conducted on a comprehensive list of datasets and tasks. The performance improvements from the proposed model are significant, especially on several referring expression recognition tasks. 

In addition, the proposed dataset extends existing datasets of testing model generalizability to another dimension: adversarial perturbation. It can be potentially widely adopted by future works along this direction.
limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper presents an incremental NMNs with a language-guided adaptive convolution layer into vanilla NMN. The goal o LG-Conv is to have a module that can adaptively co-attend over potential objects of interest from the visual input and textual input by altering the convolution.  

To achieve this, they present biSAtt and CoSAtt networks to learn the so-called guidance kernel G from textual and visual inputs. 

Experiments are done on multiple widely accepted visual reasoning benchmarking datasets, as well as a new benchmark that is designed to explicitly test the model's ability to generalize to adversarial perturbations. 

main_review: The power of NMNs were argued as leveraging specialized modules and could be assembled to perform complex and compositional reasoning. The decomposed module decouples the visual pathway and language pathway, forming an elegant structure. The argument of this work to me violates this design elegancy by agglomerate language signal again back into the convolution operations. The fact of achieving high performance on CLEVR-ref and others, as shown by the authors, could also be interpreted  CLEVR like dataset is way too artificial that could not reflect the power of NMNs following its original designing principle.  

This is being shown in Figure 4. The filter_color(red) is designed to be a modular operator only to locate red thing but not attending to other so-called objects with contextual relevance. By arguing the filter_color(red) should also consider contextual relevance goes against the NMNs principle. 

If an incremental implementation does not follow a vanilla architecture's original designing principle but achieves better performance, how shall we as a community interpret it? 

My other concern is with the CLEVR-Ref+ construction. For example, in figure 5, both E1 and E4 refers to an object that is to the left of yellow thing (and there is only one yellow thing). However, the GT for E1 is r1, and E4 is r4, one is to the left, and the other is to the right. My interpretation could be wrong, but from this single example, there seems to be inconsistent GT already. The authors mentioned three human annotators validate the creation of the Ref+ dataset. However, if the dataset fidelity is not well maintained, all the experimental results won't be convincing for making solid arguments. 

Furthermore, one argument of this work is "NMNs fail to understand new concepts such
as “yellow sphere to the left" even when it is a combination of known concepts
from train data: “blue sphere", “yellow cube", and “metallic cube to the left"."  I am wondering if the authors have verified in the new Ref+ dataset what is the percentage of such a combination of known concepts? How would you make sure when the human labeler is editing the phrases, they will create new ones that were not in other examples? 

Other places need clarification:
In some places, you use NMNs. Some use NMN. Would you please make sure to unify them? 
Figure 5, E3 is constructed from E1 and E2, but having a new phrase, "big blue rubber thing" instead of "big blue metallic thing". Are you also perturbing these phrases manually? Also, could you explain why the GT of E2 is <box:r2>. It doesn't seem to be correct to me. 

limitations_and_societal_impact: A brief broader impact narrative is provided. I do not see any explicit potential negative societal impact of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors introduce three new tools to improve compositionality in visual reasoning tasks:
1. They apply pixel-adaptive conv nets [Su et. al., 2019] to the modules of Neural Module Networks. The guidance features, which weight the convolution computation, are computed via attention over the full image and text query. They present two different attention architectures to compute this feature: one separates the attention computation over vision and language and the other co-attends.
2. A new CLEVR test set that is constructed from the S-ref split of CLEVR. The new test set consists of (a) object, spatial relationships not seen during training and (b) contrast sets [Gardner et. al., 2020].
3. They parameterize modules by argument (e.g., `blue`) to reduce the number of modules to be learned.
Their model achieves competitive performance on CLEVR and improves over all baselines on the S-Ref and F-Ref splits. On their new split (C3-Ref+) they achieve the lowest drop in accuracy and maintain performance as the number of comparison relationships stays high.

Pixel-Adaptive Convolutional Neural Networks. Su et. al., 2019.
Evaluating Models’ Local Decision Boundaries via Contrast Sets. Gardner et. al., 2020.
main_review: Originality: The paper brings together a variety of modeling techniques for the first time: an adaptive convolution together with attention computations over image and text queries all inside a neural module network.

Quality: The evaluation was thorough, but some of the design decisions left me puzzled. The authors show results across a variety of splits, including a new split designed to probe robustness and appropriately ablate every aspect of their system. Their new system leaves me puzzled because it seems to break the benefits of neural modules by incorporating contextual input into the modules. If the benefit of the LG-Conv layer is to add textual relevance and whole image context, why use modules at all? For example, in their qualitative results, they show that their `filter_color(red)` module  better localizes the red cylindrical object in the scene compared to IEP-Ref, which just localizes every red object in the scene. Conceptually, the latter seems _more_ correct. The contribution seems to come more from the attention architectures rather than their interaction with the modules. To understand the contribution of attention further, it would be helpful for the authors to describe how ViLBERT and VisualBERT were applied to CLEVR and contrast it with their approach.

Clarity: The paper was clear when discussing background and experiment design. I found the method and motivation a little confusing. I also had to rely on the original pixel-adaptive conv paper to fully understand their changes (details under comments), but that did not affect the overall quality.  

Significance: The paper puts forth a fine-grained analysis of a large number of methods interacting together with the goal of modularity. These insights could be useful in developing more compositional modules across a variety of visual tasks.

Questions:
- Equation (4): why is the optimization described as policy gradient? Is standard cross-entropy not appropriate?
- Are Visual BERT and ViLBERT given whole image information or are they restricted to object features extracted from the image?

Comments to authors (not considered in review score):
- The indexing in equation (2) is confusing. It would help to specify that these indices are 2-dimensional and to state explicitly that [] is an index operation.
- L201, Table 1: ViLBERT and Visual BERT resolve to the same reference
limitations_and_societal_impact: In my view, the authors have adequately addressed the impact of their work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes a novel extension to the neural module network (NMN) to tackle vision-language tasks. The proposed method applies a language-guided convolution operator to the neural modules, which improves information exchange and fusion between the visual and language modes. The proposed framework is applied to various benchmarks in the visual question answering (VQA) and referring expression recognition (REF) domains. Experiment results demonstrate a superior performance of the proposed method.
main_review: The capability to process complex information from multimodal perception is a hallmark of the power of modern deep learning systems. This paper focuses on the vision-language tasks, a direction currently subject to huge interest in both research and application. The proposed method introduces a way to further enhance the guidance to the reasoning modules by language inputs, and is claimed to be the first model to adapt convolutional filters to languages. Large amounts of carefully conducted experiments show improvements in performance over a wide range of baseline models on both the VQA and RFE tasks. The paper is well written and easy to follow, despite missing a few technical details which will help readers from outside the community to understand better.

My challenges and question to this paper are listed below. No further experiments are requested in the rebuttal.

1. The idea behind the design of both NMN models and the CLEVR dataset is to guide machines to learn "universal" neural modules that perform atomized logic operations through compositional reasoning. This means that the module parameters should not only be independent to the visual input, but also the language input. For example, in both "What size is the red sphere?" and "How many large red objects are there?" the "filter_color[red]" operator should map to the same logic operation regardless of the context input. This paper to some extend violates this design philosophy, therefore it becomes important to address how much the model is overfitted. 

2. The language dependency of NMN mainly falls on the inference of programs. Is the language-guided execution modules able to fine-tune an imperfect program generator (for example, in VQA, first use a few thousand programs to train a weak program generator then jointly fine-tune with RL)? This might also suggest how overfitted the model is.

3. Any specific reason for choosing the Gaussian kernel? How do the results depend on the type of kernel and their sizes?

4. Since all guidance kernels are shared by all modules, would "modularizing" the guidance kernels further contribute to the performance? 

Overall I think this paper has very strong potential and the experiments are solidly conducted. However, due to concerns about the model design, possible overfitting, limited novelty, I hereby give a "borderline reject" score.
limitations_and_societal_impact: Limitations are briefly mentioned but not systematically discussed.
Social impact of this work is briefly discussed.
review_title: null

Meta-review: The paper proposes a new model for compositional reasoning with extensive evaluation on synthetic and  real image datasets. Additionally, the paper contributes a new challenging ref test set based on CLEVR which is manually verified. 

Given the interesting results and technical contributions the paper provides, I recommend to accept this paper under the expectation that the authors will revise the paper as promised in the author response, including:
1) Fix notational issues / references, consistent notation
2) Additional analysis using different types of kernel and sizes
3) Adding results on Modularizing the guidance kernels

Several reviewers raised the concern how "modular" this design is and how good the grounding is. I encourage the authors to include more of the discussion they provide in the author response to the final version of the paper. I think it would be very convincing to see a quantitative analysis on this, on the precision (and maybe recall for completeness as well) of the attention, i.e. a quantify the analysis of Figure 4, ideally comparing to baselines and prior work. I believe this should be reasonable easy given the annotations provided with CLEVR, and significantly increase the insight of this work and thus increase its impact.
Decision: Accept (Poster)