How to Design Sample and Computationally Efficient VQA Models

Abstract: In multi-modal reasoning tasks, such as visual question answering (VQA), there have been many modeling and training paradigms tested. Previous models propose different methods for the vision and language tasks, but which ones perform the best while being sample and computationally efficient? Based on our experiments, we find that representing the text as probabilistic programs and images as object-level scene graphs best satisfy these desiderata. We extend existing models to leverage these soft programs and scene graphs to train on question answer pairs in an end-to-end manner. Empirical results demonstrate that this differentiable end-to-end program executor is able to maintain state-of-the-art accuracy while being sample and computationally efficient.

Reviews and author responses:

**PAPER SUMMARY**

The paper presents a method for visual question answering (VQA) that makes use of a differentiable program executor that softly approximates the execution of neural modules that read and write to a stack. Experiments on CLEVR demonstrate that the model can achieve high performance using small amounts of supervision (varying the amount of question/answer pairs, image/object annotations, and question/program annotation used during training).

**STRENGTHS**
- The method achieves differentiable program execution without resorting to training with policy gradients
- Experiments demonstrate high accuracy on CLEVR, even using very little training data

**WEAKNESSES**
- Critical missing citation / comparison: Prob-NMN (Vedantam et al, ICML 2018)
- The model hardcodes a lot of task-specific knowledge; is this useful?
- The writing is very unclear in some places

**UNCLEAR WRITING**

I find the paper to be quite unclear about exactly what type of supervision is used where. Section 3.4 states that “We pre-train our models with a very small percentage of visual and textual data” -- but what kind of data is this exactly? If I understood the following discussion, I think that “image data” refers to (image, object) annotations used to train the Mask R-CNN model, while “textual data” refers to (question, program) pairs used to supervise the program executor. Then in Table 3, “X% pre-train” refers to the amount of (image, object) and (question, program) annotations used to supervise the model, and “Y% of Training Data” refers to the number of (question, answer) pairs used to supervise the model. I think that the clarity of the paper would be much improved by making it more clear as to what types of supervision are used at different stages.

I similarly found Section 4.2 and Tables 4 and 5 very unclear, and I’m not entirely sure what experiments are being performed; what does it mean to “fix vision and train text from 0.1% pretraining” in Table 4, or “fix text and train vision from scratch” in Table 5?

**MISSING COMPARISON: PROB-NMN**

One of the main draws of the proposed method is that it is extremely data-efficient on CLEVR, e.g. achieving 99.1% accuracy when supervised with 10% of (question, answer) pairs and 1% of (image, object) and (question, program) pairs.

This sort of data-efficient learning is also a key feature of Prob-NMN [1]; this is a critical piece of prior work which is not cited; the proposed method be discussed in light of this prior work, and the method should be compared against it. From Table 2 of [1], Prob-NMN achieves 97.73% test-set accuracy on CLEVR when supervised with 100% of (question, answer) pairs and 0.143% of (question, program) pairs; this outperforms the 93.1% reported for DePe in Table 3 of the submission which uses 100% of (question, answer) pairs and 0.1% of (question, program) and (image, object) pairs. However this comparison is not fully valid, since Prob-NMN uses slightly more program annotations but uses no explicit object detector. The authors should set up a more controlled experiment to compare their method against Prob-NMN.

[1] Vedantam et al, “Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering”, ICML 2018

**HARDCODED REASONING**

The proposed model achieves data-efficiency by encoding a huge amount of task-specific prior knowledge into the model architecture:
- The use of any ground-truth programs assumes that the set of primitive operators needed to answer questions are known
- The module designs in Table 2 go a step further, and assume that the implementation of all of these primitive operators is known -- the operators in Table 2 are effectively a differentiable approximation to the ground-truth CLEVR program executor
- The use of an object detector assumes that all questions can be answered in terms of object positions and properties; that the vocabulary of semantic concepts needed to answer questions is known; and that furthermore the mapping of images -> object locations and properties can be supervised during training

Taken together, the model essentially hardcodes the data-generation process of the CLEVR dataset into the model. This is evident from Table 3: if I’ve read and understood the experimental setup, training with 1% of (question, program) annotations and (image, object) annotations and zero (question, answer) annotations achieves >90% test-set accuracy on CLEVR! To me this is not a positive result -- instead it signifies that the model architecture itself is overfit to the dataset.

The higher-level research goal of building neuro-symbolic models that elegantly blend prior knowledge with learning is extremely exciting. But testing this kind of model on CLEVR is not. CLEVR is a synthetic dataset, and its questions are not natural: they are generated programmatically, and there is a lossless and deterministic mapping between question text and semantics (encoded as functional programs). CLEVR images are likewise not natural, and the image semantics necessary for answering any CLEVR question can likewise be losslessly encoded into a scene graph. As such, the fact that this model achieves data-efficient learning on CLEVR is nearly as uninteresting as the fact that the ground-truth CLEVR program execution engine achieves perfect accuracy when fed with ground-truth scene graphs: it is nearly guaranteed by construction, and only works by exploiting the synthetic nature of the benchmark.

My biggest concern with this line of work is that I don’t see how it could possibly be used to achieve visual reasoning on real-world data, where none of the above can be exploited: for true natural language and images, we do not know a vocabulary of symbolic primitives that can losslessly represent the world and we certainly can’t hardcode the implementation of all symbolic primitives necessary for performing real-world reasoning.

So although this method may indeed achieve extremely data-efficient learning on CLEVR, this is not an interesting achievement on its own unless there is some hope that the method used to do so can also be used for non-synthetic, real-world data. And the submission presents no evidence, or even an argument, for how this might be done.

**OVERALL**

On the whole, I think that this paper is not ready for publication due to unclear writing, and the fact that it is missing a critical comparison to Prob-NMN.

I also take issue with the underlying approach, since I think the methods used to achieve data-efficiency on CLEVR cannot scale to real-world data. However I can accept that different people in the community feel differently from me on this issue (as evidenced by excitement over papers like the Neuro-Symbolic Concept Learner), so I would not necessarily consider this as a disqualifier if it were my only concern with the paper.


**AFTER REBUTTAL**

I appreciate the author's efforts to improve the clarity of the submission and to add a comparison with Prob-NMN. However I remain unconvinced that this style of model can possibly scale to more realistic data. I've upgraded my score from 4 to 5, but I still generally lead toward rejection.


Dear Reviewers and AC,

We have made major revisions to the paper draft to improve the readability and to address your specific comments. We encourage you to take a look! In general we spend more time exploring and analyzing different VQA components, reflected in these high level changes:

1) There is a new section that describes the vision, language, reasoning components in more detail.

2) The order of which we present the experiments is updated. At each experiment we arrive at clear a conclusion for which method is most efficient for each component.

3) After experimenting on individual components we show that we need a new framework to handle object-level representations and soft program execution. We highlight two key challenges in the main paper to address this integration: an updated memory and optimization procedure. The rest of previous model details have been moved to the appendix. 

4) Our final experiments now also compare the model efficiencies across vision, language, and reasoning jointly to show that this model integration works. We also have comparison results to other models as before.

We are still working on GQA experiments, and we hope that the current revision provides interesting insights.


Thank you for the detailed feedback and suggestions! We have updated the overall paper so the writing flows better in general. Here we address specific comments as well.

### Training Data:

When we indicate pre-training on a small amount of data, we indicate that we train our text and vision classification models on a fraction of the fine grained function program and scene graph labels provided in the dataset. So when we say 1% pre-trained we pre-train these models on 1% of the 700k QA function programs and their corresponding scene graphs (7k each). We thus test ranges from 0.1% - 1% pre-training, and our updated results show that the method still scales well with smaller pre-training percentages (0.1%, 700 program and scene graphs).

The vision pre-training is done using the scene graphs to train the MLP models corresponding to object centric attributes (shape, color, etc.) and relations between objects. Here it is assumed that the general object bounding boxes can be estimated, as used in previous work such as NS-CL. While this is a hard assumption to make, there has been more progress towards unsupervised detection and segmentation, which remains out of the scope of the current work. The vision pre-training is done as you indicated, by the text and program pairs.

The training data is the number of question and answer pairs used for end-to-end supervision, where no detailed program annotations or scene graphs are provided. We have made this more clear in our updated revision.

### Experiments:

Please view our global comment above for more details regarding the experiments.

### Prob-NMN:

Prob-NMN is a good comparison method to our methods with its pre-training and fine tuning setup, and we will add it to our paper. Compared to our original scores for our 0.1% pre-training 93.1% accuracy on 100% QA pairs while Prob-NMN achieves 97.7% on 0.14% pre-training. However we noticed that our $\alpha$ and $\beta$ hyperparameters (described in the optimization section 3.4) were not set, which led to a significant dip in our accuracies. After setting these correctly for 0.1% pre-training we achieve **99%** and 98% validation accuracy using 100% and 10% QA pairs respectively. 

### Hardcoded Reasoning: 

The use of primitive operators that we use follow the CLEVR convention, but we believe that they are general enough to use. Primitives such as filter, relate, count can be generally used for different datasets and even different domains such as text. The high level reasoning to parse the natural question to a traversal over the subject domain, images in our case. The CLEVR functions defined are just an implementation choice, or a good bias, which can be used on other image datasets. We are currently working on experiments that show that we can leverage these same definitions on a more real-world dataset such as GQA.

CLEVR is indeed a baseline dataset for neuro-symbolic reasoning, and we use it to emphasize various vision and text model configuration efficiencies, in addition to the end-to-end accuracy. Based on the experiments we have a concrete path to testing such a method on real image data efficiently. As for out of distribution vocabulary in GQA we use the attributes corresponding to the top-k primitives per attribute for practical purposes. Similarly a method can be extended to use a dense embedding representation for the attributes with cosine metric to softly match primitives, instead of the current sparse one-hot selection. In such a setting, if new primitives are detected, their dense embeddings can be added to an attribute set without retraining the entire pipeline.


Thank you for the constructive comments. We are updating the overall paper so the writing flows better in general. Here we address specific comments as well.

### Novelty:

Indeed pieces of the model are motivated from NS-CL, Stack-NMN, MAC and others. The key contribution here is first looking at a combination of these methods which empirically show to be sample and statistically efficient, which are the probabilistic program executions and the object level detections. 

Please view our global comment above for more details regarding the motivation.

### Real World Data:

Like previous approaches we are curious to see if our method can scale on more real world datasets. We are currently experimenting on the GQA dataset to test our performance against other methods. 

### Mask-RCNN:

Regarding the mask R-CNN we followed the convention laid out by NS-VQA and NS-CL. We emphasize that the R-CNN is just used to identify the object bounding boxes, but the class labels are not used. Instead the object feature based MLP models are used to classify object attributes and relationships. In practice for a task like this, faster R-CNN works about the same as Mask RCNN for identifying bounding boxes of the objects. The high level idea is we can use such pipelines on more natural image tasks. These images have R-CNN methods that already are pre-trained and the idea is that we should be able to drop them into our object-centric vision pipelines. While this is not directly comparable to pixel wise methods such as MAC, we show that such object based architectures are important for efficiency and are working to show that they work for real image datasets.

### Takeaway:

Throughout our experiments we observe that methods that can continuously optimize end-to-end perform the most efficiently. More specifically for VQA this requires leveraging object-centric representations, an intermediate soft logic representation, and a probabilistic way of representing soft logic programs from text. While at a high level this makes sense, we empirically show this in our vision and text training experiments. This motivates future work that requires reasoning paths to retain probabilistic program distributions over object centric representations to compute the expected answer, such as in GQA.


We thank the reviewer for the constructive feedback. Below we address specific comments.

### Writing:

We are updating the overall paper to provide more analysis and so the writing flows better in general. More specifically we will modify the writing to explore these efficiency tradeoffs of existing models and then propose our model architecture based on these results. 

### Motivation:

For the motivation we show that to handle the vision, language, and reasoning there are many core components that VQA models look at and compare and contrast these components with respect to sample and statistical efficiency in the experiments. 

Please view our global comment above for more details regarding the motivation.

### Experiments:

Please view our global comment above for more details regarding the experiments.

### Real Datasets:

We see great benefits from the MAC and Stack-NMN operating on the real world domains. Such models show the benefit of pixel wise attention and general text neural modules when operating on these open domains. We are currently working on experiments on GQA to show that our method can scale to more real world datasets as well.


We thank the reviewers for their feedback so far. We currently focusing on two main directions with the current work: 

1) We are restructuring the paper to primarily be an exploration of current models and methods for efficient VQA, this will motivate the design of final model we propose and test.
2) We currently running experiments on a more real-world dataset: GQA

Some clarifications useful for all reviewers are regarding the following topics:

### Experiment Explanation

In section 4.2 we explore the different model components from Table 1 given different sample complexities. Here we want to explore which model architecture would be more sample efficient for the vision and language components. The table 4 shows that if the models were provided perfect vision predictions, we can isolate the effects of the text training. Here the train ratio is what percentage of the CLEVR question answer training pairs are provided for end-to-end training and the corresponding validation QA accuracies. This tests the sample efficiency of symbolic methods which choose a discrete program trace versus ours that leverage soft probabilistic programs. For the vision we fine tune the program parsers perfectly and train the vision models from scratch to test abduction, pixel-wise, and object centric representations. The takeaways from these experiments is that the soft program layouts and the object-level predictors provide the best sample complexities for the text and vision models respectively.

Similarly in 4.3 we follow the same setup as in 4.2, but instead of varying the sample complexity we measure the computational complexity of how long these approaches take to converge. In our experience these REINFORCE and abductive methods for NS-VQA, NS-CL text and NGS vision take many samples and iterations to converge. Moreover due to the high variance of the REINFORCE methods, it is very difficult to reproduce the numbers reported in the original papers, motivating our continuous approach. 

In the updated paper we will clearly break down these components and test them on existing models, which will motivate the design of the proposed model.

### New Model Motivation

The motivation for the new model builds on the works of NS-CL and Stack-NMN. NS-CL leverages **soft logic** and **object-level** vision models, which has proven key for sample efficient and accurate results. For the text it executes a discrete program layout, making it statistically inefficient due to the REINFORCE to optimization procedure and discards uncertainty regarding the program parse. 

We also leverage Stack-NMN, which uses a **soft program layout** to capture the uncertainty of the text to program conversion, but uses a less accurate pixel-wise image model and corresponding neural modules. They leverage this pixel-wise approach to store the image embedding of the same dimension in its intermediate stack. This makes it stack architecture simpler and the corresponding neural modules only have to deal with one type of input.

Based on these observations, our proposed method leverages object-level vision, and soft program layouts, and soft logic. This led to key architectural differences in how the stack was defined and manipulated, as well as how the final model inference is carried out. This is because the stack needs to store the intermediate results of all probabilistic program executions of the soft logic, whose function signatures and output types. 

Furthermore we explore challenges of jointly optimizing program and vision prediction in a true **end-to-end** fashion. In comparison NS-CL iteratively tunes its program parser and vision models, while Stack-NMN leverages coarser neural modules instead of more interpretable programs. This leads to the name **Differentiable end-to-end Program execution (DePe)**.


This paper studies visual question-answering (VQA). The authors proposed an end-to-end differentiable framework that performs "soft" logical reasoning based on object-centric scene representations and attention-based language parsing. The authors claimed that the new model is more data-efficient.

While I think this direction is interesting and useful, I think the submission really needs more work before it may be published at a top-tier conference. The current manuscript is hard to understand and the results are not convincing.

First, the method is not well-motivated. The intro mostly discussed related papers and their differences, so I was expecting an analysis paper, but it turns out the rest of the paper is about a new model. There is no justification for why we need a new model, what is new about it, and why we expect it to do better. 

Apart from this, the writing is in general unclear. For example, the model has an acronym DePe, but the full name was never fully told. The experiment section is also poorly written. In 4.2, it's unclear what "Train Ratio" means in Table 4 or 5, and these tables were never referred to in the main text.  I also don't understand the setup in Section 4.3, either, as essential descriptions are completely missing.

This makes it hard to assess the results.  In addition, all results are on the CLEVR dataset. One selling point of methods such as MAC, NS-CL, and Stack-NMN is that they can naturally be applied to real images such as the GQA dataset. Without experiments on real images, the analyses presented in this paper are unlikely to convince researchers in this area.



The paper proposes a neuro-symbolic model for sample-efficient VQA, which turns each question into a probabilistic program which is then softly executed. The problem explored in the paper and its background and context presented clearly and it does a good job in motivating its importance and trade-offs between possible solutions. While the use of a probabilistic program to represent the questions might be too stiff / inflexible in my opinion and may not generalize well to less constrained natural language, this direction is still of course important and interesting. It also does a great job in presenting the existing approaches and comparing their properties. The writing is good and the model is presented clearly with a very useful diagram. 

However, the novelty of the paper seems limited to me, as it mainly combines together ideas that have been extensively explored in many prior works which are mentioned by the paper. Turning the question into a series of attentions over semantic factors appears in the NSM and partially in MAC models. The iterative memory updates appeared in MAC. Combining together small operations and functions defined by hand as dictated by programs as in page 6 of the model is the main idea of Module network. End-to-end differentiability for VQA models has also been extensively explored and multiple solutions have been proposed: relations networks, soft variants of NMN, and also MAC and NSM, etc. The use of stacks has been explored too in the stack-NMN model. I therefore feel the paper mostly recombines and tunes together these ideas rather than offering one particular new idea or insight. 

The paper presents results on CLEVR only, which goes back into my concern about the inflexibility of the probabilistic programs. Especially for this type of models, it will be useful to explore it on tasks beyond CLEVR such as VQA/GQA to show whether it can work for natural or richer language. The use of Mask R-CNN on CLEVR is also quite unreasonable in my opinion: the task has meant to be visually simple, so using a very strong visual model on it nullifies the visual aspect of it completely, making the model working on perfect semantic scene graph inputs rather than on "real/natural" uncertain and more noisy inputs. It also gives unfair advantage to the model when comparing to baselines which didn’t use object detectors on CLEVR but rather work directly with the image, e.g. MAC and others (presented in the table in the experiments section).

At the same time, it is important to mention the model does get quantitative improvements in scores and especially sample efficiency, but the paper doesn’t make it clear what is the particular property or part of the model that allows for the improved numbers, and so the paper doesn’t leave the reader with a clear new takeaway message.


Meta-review: The reviewers are in agreement that this paper could benefit further improvement. There are several areas: novelty of the proposed approach and evaluation on real-world datasets (beyond just CLEVR).

Decision: withdrawn-rejected-submissions