How to Design Sample and Computationally Efficient VQA Models
Authors: Karan Samel
Zelin Zhao
Kuan Wang
Robin Luo
Binghong Chen
Le Song
Venue:  (2021)
Keywords: vqa
visual question answering
neural modules
probabilistic logic

Abstract: In multi-modal reasoning tasks, such as visual question answering (VQA), there have been many modeling and training paradigms tested. Previous models propose different methods for the vision and language tasks, but which ones perform the best while being sample and computationally efficient? Based on our experiments, we find that representing the text as probabilistic programs and images as object-level scene graphs best satisfy these desiderata. We extend existing models to leverage these soft programs and scene graphs to train on question answer pairs in an end-to-end manner. Empirical results demonstrate that this differentiable end-to-end program executor is able to maintain state-of-the-art accuracy while being sample and computationally efficient.

Under review as a conference paper at ICLR 2021
How TO DESIGN SAMPLE AND COMPUTATIONALLY EFFICIENT VQA MODELS
Anonymous authors Paper under double-blind review
ABSTRACT
In mudti-modal reasoning tasks, such visual question answering (VQA), there have been many modeling and training paradigms tested. Previous models pro- pose different methods for the vision and language tasks, but which ones perform the best while being sample and computationally efficient? Based on our experi- ments, we find that representing the text as probabilistic programs and images as object-level scene graphs best satisfy these desiderata. We extend existing models to leverage these soft programs and scene graphs to train on question answer pairs in end-to-end manner. Empirical results demonstrate that this differentiable end-to-end program executor is able to maintain state-of-the-an accuracy while being sample and computationally efficient.
INTRODUCTION
Many real-world complex tasks require both perception and reasoning (or System and System intelligence (Sutton & Barto 2018)), such as VQA. What is the best way to integrate perception and reasoning components in a single model? Furthermore, how would such an integration lead to accurate models, while being sample and computationally efficient? Such questions are important to address when scaling reasoning systems to real world use cases, where empirical computation bounds must be understood in addition to the final model performance.
There is a spectrum methods in the literature exploring different ways of integrating perception and reasoning. Nowadays, the perception is typically carried out via neural models: such as CNNs for vision, and LSTMs (Gers et al., 1999) or Transformers (Vaswani et al. 2017) for language. Depending on the representation of perception input and their reasoning interface, a method can be either more towards the neural end of the spectrum or more toward the symbolic end.
For the vision part, models either use pixel-level object-level symbolic representation. For the language part, models can generate either textual attention or programs, where the text is decom- posed into sequence of functions. Within the program representations, models typically operate on a selected discrete program or on probabilistic programs. The reasoning part used to produce the final answer can either neural models, symbolic reasoning, something in between, such as neural module networks (NMN) or soft logic blocks.
Existing works for NMN methods leverage pixel-level representations and program representations such as NMN (Hu et al. 2017). Prob-NMN (Vedantam et al., 2019), and Stack-NMN (Hu et al., 2018). Representative models that use object-level vision also leverage both neural and symbolic language and reasoning. Models that are more neural are LXMERT (Tan & Bansal 2019) and NSM (Hudson Manning 2019), while those that are more symbolic are NS- VQA (Yi et al. 2018), NS- CL (Mao et al. 2019) and NGS (Li et al. 2020 A systematic comparison across these models is
illustrated in Table I with more details in Appendix
Overall, neural models have more expressive power but with more parameters, while more-symbolic models have more prior structures built into them but with fewer parameters. There is an interesting bias-variance trade-off in the model design. By encoding much bias into the model as possible, one could reduce sample requirements.
The different choices of perception and reasoning components also limit how the QA models will be trained. both components are chosen as neural modules, then the training can be done a very efficient end-to-end fashion. If the reasoning carried out using discrete operations,

---

Under review as a conference paper at ICLR 2021
Vision
Language
Reasoning
Training
Symbolic Progs.
Object-level level
End-to-End
Pixel Att.
Text Att.
Soft Progs.
Symbolic
Logic
Sampling
Neural
LXMERT (Prob-)NMM
Stack-NMN
NSM
NS-VQA
NS-CL NGS
Table I: A breakdown of VQA models by indicating which method is used with respect to their vision, language, inference, and training components. Refer [appenii[Affor a detailed description of these methods.
then the perception model needs to sample discrete outputs or take discrete inputs to interface with downstream reasoning. For instance, if symbolic reasoning used, REINFORCE (Williams 1992) is typically used to train the perception models, which may require many samples during the opti- mization process. Alternatively, one can also use expensive abduction (Li et al. 2020) to manipulate the perception models outputs to provide the correct reasoning and then optimize these perception models using these pseudo-labels. Overall, more neural models will be easier to optimize, while more symbolic models will need additional expensive discrete sampling during optimization. To highlight this interesting fact, we call it the neuro-symbolic trade-off.
This neuro-symbolic trade-off also affects sample efficiency and computational efficiency. To be more sample efficient, the model needs to be less neural, yet, a more neural model can be more com- putationally efficient during training. Thus method that achieve overall good performance in terms both sample and computation efficiency will require systematically determining which perception and reasoning components should be used and how integrate them. To design such model, we first test which method within each perception and reasoning component works the most efficiently. From this neuro-symbolic trade-off exploration we can design model that uses these most efficient components and compare its overall performance against existing models.
PROBLEM SETTING
Before the exploration, we formally define the different choices for the vision, language, and reason- ing components. the general VQA setting we are provided with an image I, natural language question Q, and an answer A. We now define how these basic inputs are used in each component.
2.1 REPRESENTATION FOR VISION
Given the image / there are two predominant visual representations: pixel and object-level attention.
Pixel Attention. Given an image one can leverage traditional deep learning architectures used for image representations and classification such as ResNets (He et al.) 2016). Here the image passed through many residual convolution layers before entering a MLP sub-network perform classification task. From one of these MLP linear layers, an intermediate dense image representation feature fI E RD be extracted, denoted by ResNet(T). These features used further down the VQA pipeline, where the downstream model computes attention over the relevant part of the feature based the question asked.
Object-level. Another paradigm is to leverage object detection models such as Faster R-CNNs (Ren et al. 2015) to identify individual objects within images. Given objects in the image,
2

---

Under review as a conference paper at ICLR 2021
conduct more object-level or symbolic reasoning over the image, instead of reasoning through pixel by pixel basis.
In this object-level representation, set of object location bounding boxes (bbox) can be detected and labeled directly by using R-CNN as O = {bbox, (. ... labely]] = R-CNN(I) O can be labeled "small red shiny ball" or "large
for a preset number of T objects. Here green tray" based on what is in the image.
Another approach is to factor the joint bounding box and label prediction into individual components to be handled by separate models. First the bounding boxes are extracted from the R-CNN as [bbox:)?== R-CNN(T). Then these be passed into a separate MLP network retrieve the labels {label, ] MLP(ResNett I [[boxx])), where 7[bbox] is cropped image at that bounding box location. These can be used to define the final set of objects: O {bboxx la
In such setup, the benefit is that the R-CNN can be trained just as an object detector for a generic object class versus the background, whose annotations are easier to obtain. Furthermore, the number of supervised data the label MLP uses for training can be controlled separately. This is a useful mechanic during model efficiency analysis where work under the assumption that object bounding box is almost perfect, while object labeling is imperfect and expensive to annotate.
2.2 REPRESENTATION FOR LANGUAGE
The language representations operates on the natural text question Q. Some data sets also provide intermediate representations of each through function program layout F P.. FP represents the Fi
question as a sequence of abstract functions F as FP = [F1. F2..., Fil for F; E F. These function programs are used jointly with the visual representations to conduct reasoning to arrive at answer A. Details about potential realizations of F are described in the following reasoning representation section 2.3 Given the question Q and its representation PP we can define different approaches for representing the text.
Just using the embedded text tokens E model can embed the question Q through recurrent network to generate final question representation li,, where T is the maximum length sequence. Then ht can be put through an recurrent decoder to obtain a latent function at each step c through an attentive combination of the hidden states C. "Er a. hy
Text Attention.
Symbolic Program. we want to explicitly produce a FP for a corresponding question Q, we similarly encode the text done for text attention. During decoding c is passed through a MLP to predict a valid function token. Then the most likely program is selected as arg max P(FP Q)-
Soft Program. When choosing discrete symbolic program, the uncertainty of other function pro- gram parses is thrown out. Instead the probabilities for each program can be saved and an expected program be computed TE| [( (FP P Q)]. Intuitively all the possible programs have be con- sidered in this scenario which can be intractable. Instead soft program methods such as Stack-NMN factor this E[P(F Q)] II P(F, Q)] II E[P Q)]. This enables preserving distribution of functions at each step instead of selecting single one.
2.3 REPRESENTATION FOR REASONING
Given the visual and language representations, the reasoning component use these representations to derive the final answer A. Here we discuss methods that are neural, symbolic, and soft logic based.
Neural Reasoning. Reasoning can be made directly with the image feature f1 and encoded question lIT such as MIP(( ; [r]) in a purely neural fashion. Other approaches can leverage the object representations O. This is done by modulating the attention over which O correspond to final answer A, conditioned on fr,, as done in NSM or LCGN. LXMERT uses modal attention between text embeddings E and O to prediet the final answer. All these methods are more neural, but the FP can be leveraged as well to incorporate better biases through symbolic and soft programs.
Symbolic Representations. From the question can define abstract functions FF generate FP as described the previous section. Representing F in a symbolic form enables encoding general knowledge certain dataset's 's domain specifie language (DSL.) into model. This improves model
3

---

Under review as a conference paper at ICLR 2021
interpretability and provides better inductive biases well. Here further describe two classes of these functions: fine grained and coarse.
A fine grained representation of FP is sequence of n-ary predicates, fune- tions with n arguments, composing F. For example, given the question Q "What is the shape of the thing left of the sphere?", sample fine grained program fined as FP = [filter_shape (sphere, о) relate (left, о) spperyshaape (O)] Here the visual representation (O or fr) and higher level argument concepts, such as sphere, used as inputs to each function. We observe clear biases encoded into the function architecture, as given a scene graph of objects O and their relations, one could walk along this graph using FP to get the final answer A. The trade-off is that the model has to deal with more complex functions, whose input arguments and output types can vary. For example relate shape and re aete return a subset of objects, while query shaaee returns string. Such formulations are used by more neuro-symbolic methods such as NS- VQA and NS-CL.
be de-
Coarse function types consist of simpler predicates whose arity is fixed, typically 1, over F. Given the previous question Q, a coarse function can be defined as FP [filters (Jr), relatee (fr), queryo(fi)]. Here less structure is required with respect to the language and visual representation where each function can be parameterized as a NMNs. These re- quire more parameters than DSL functions but are syntactically casier to handle they typically just operate on fixed dimensional image feature fI, thus implicitly encoding the function arguments.
Symbolic Reasoning. Using any coarse fine representation type for F, the symbolic rea- soning can take place over the selected symbolic program FP. We define the high level execution of the symbolic reasoner arrive at the answer by executing over FP as A (F P, image representation) In the fine grained and coarse samples this would look like:
A (FP,O)s query_shape (relate (left, filter_shan (sphere, ))))
A (FP. fi)s querya (relate) (filter, (fi)))
Since the structure of the reasoning discrete, to update the visual and language model weights requires sampling based learning such as REINFORCE or abductive methods.
Soft Logic Reasoning. When conducting the symbolic reasoning we discard the uncertainty of the visual representations when generating the labels for O. Instead the probabilities for O can be saved. Here the uncertainty from the detections can propagated during the execution in a soft manner to arrive at A. We can similarly define the soft logic reasoning A (FP, T) EO--R.cNNDDKEEP..0 s) Due to the probabilistic execution, this can be optimized end-to-end.
Now that the components and their corresponding methods have been defined, we explore which methods are the most efficient for their respective task.
3 NEURO-SYMBOLE TRADE-OFF EXPLORATION
Many deep VQA models have been developed in the literature with a range of design elements, as can be seen from Table D Which one of these methods is the key factor in making deep VQA model sample efficient while at the same time achieving state-of-the-art? In this comparison using the CLEVR dataset (Johnson al. 2017]. we aim understand which design elements individually perform the best. Based on these findings, better end-to-ent models can be designed from the indi- vidual methods selected. More specifically, we will explore the sample and computational efficiency for the following components:
Visual representations through pixel attention and object-level methods.
Reasoning execution through neural modules, symbolic execution, and soft logic.
Language encoding for questions through text attention, symbolic and soft programs.
For the representations of language and reasoning, we observe that these two components are tightly coupled. In language define _ and the corresponding FP given Q. In reasoning these functions
4

---

Under review as a conference paper at ICLR 2021
get executed for fine grained functions, network gets constructed from neural modules in the coarse case. For this reason we found it difficult to isolate the language and reasoning exploration. to initially observe the interactions between the vision and reasoning given fixed FP. Then by iteratively selecting the best vision and reasoning components, we explore the most efficient language and reasoning combination. Each method is also explained in more detail in Appendix D
This motivated
3.1 VISUAL REASONING
To test the visual perception and the reasoning components we break down the tests into two parts. First we first determine which visual representation is more efficient: pixel attention or object- centric. Second we find the reasoning method that best complements the selected visual representa- tion.
Computation Efficiency
Sample Efficiency
Validation Accuracy
100
80
QA Data %
Pixel Att. Stack-NMN
Object-level NS-CI.
60
40
1%
47.1
96.1
NS
5%
66.8
97.2
20
Stack NMN
10%
93.5
98.8
M
0
0.2
0.4
0.6
0.8
Train Samples
-10
Figure 1: Sample and computational efficiency for pixel attention and object-level vision represen- tations. The sample efficiencies experiments just use a total percentage of the available QA pairs and report the validation score after convergence. The computational efficiency is evaluated on the 10 % QA data and are run 5 times. The train samples indicate the number of examples seen by the model over time (iteration batch size).
Pixel Attention versus Object-level. We compare pixel attention Stack-NMN and object-level NS- CL representation methods. We chose these two models as their visual representations differ but had continuous end-to-end reasoning methods given a fixed FP.
We set up this experiment by training and freezing each model's language components CLEVR question program pairs (Q, FP) to isolate the effects visual methods. Then the visual representa- tion is trained from scratch using different percentages of CLEVR text question and answer (Q, A) pair data (QA Data %). where no object level supervision is provided.
The sample and computational efficiencies illustrated in Table representations work better. Object-level detectors are able to leverage cheap object locations labels for localization and can leverage MIPs classify objects given dense feature E IR.. In trast, pixel-wise methods needs to learn the object attention maps through more parameter intensive convolutional layers over Lx where < L W channels.
I which indicate object-level
Symbolic versus Soft Logic Reasoning, Since we will use fine grained object-level O information, don't need to conduct parameter intensive pure neural reasoning over f1. This lets us leverage of function programs FP. Similarly, given O, we don't 't use coarse NMN functions, which are only compatible operating over fi. We focus on testing fine grained F determine the best way to reason over the answer A = (FP, .)., either using symbolic or soft logic.
We use NS-CL which already performs soft logic over (Q, pairs. To test the symbolic reasoning we replace the soft functions F defined in NS-CL by discrete counterparts, similar to the symbolic execution performed by NS-VQA. To train the symbolic reasoning with QA data, we test REIN- FORCE and abduction based optimization, denoted as NS-RL and NS-AB respectively. The results in Table 2 to gains computational efficiency well as final performance. Abduction has benefits over RE- INFORCE as it is able to selectively sample the object probabilities, which shows to a more efficient procedure the accuracy of the vision model increases.
indicate that propagating the uncertainty of the object predictions through soft logic leads
5

---

Under review as a conference paper at ICLR 2021
Computation Efficiency
Sample Efficiency
Validation Accuracy
100
80
QA Data %
60
NS-RL.
NS-AB
NS-CL
40
1%
47.1
59.8
96.1
20
5%
72.8
76.6
97.2
10%
81.3
84.2
98.8
0
0.2
0.4
0.6
0.8
Train Samples
-10
Figure 2: Sample and computational efficiency for soft versus symbolic reasoning. We take soft logic NS-CL and modify the reasoning to become symbolic. Then we optimize these symbolic methods through REINFORCE and abduction for NS-RL and NS-AB respectively.
At this point we have been testing the visual components and present the benefits of object-level representation and soft logic reasoning. Given these two methods, we now explore which language representation would be the most efficient use.
3.2 LINGUISTIC REASONING
To test our language representation we want to determine the most efficient representation for Q. Taking into account the vision experiments, we find NS-CL's operation over object-level representa- tions and executing soft logic reasoning over fixed FP to be the most suitable approach. To build off of this, we want to understand the best representation of FP for reasoning. Therefore we look at symbolic and soft approaches. Recall the tight integration between the language representation and reasoning means that we investigate these two components in a joint fashion.
Computation Efficiency
Sample Efficiency
Validation Accuracy
100
80
QA Data %
Symbolic Progs. NS-CL
Soft Progs. Stack-NMN
60
40
1 %
86.5
81.3
NS-CL Suck-NMN
20
5%
89.2
90.2
10%
92.1
94.3
0
0.2
0.4
0.6
0.8
Train Samples
-10
Table 2: Sample and computational efficiency for pixel attention and object-centric vision represen- tations.
Symbolic versus Soft Execution. To test the language representation, we similarly train and freeze the visual representations isolate the effects of the language models. These language models are then trained are also trained end-to-end on (Q, A) pairs without ground truth FP.
We compare NS-CL, which uses fine grained symbolic programs and soft logic reasoning over O to Stack-NMN which uses coarse soft programs and neural reasoning over fr- For Stack-NMN the language and vision are trained jointly, but for NS-CL. the language parser trained disjointly from the vision. During testing the computation efficiency results, equally divided the iterations by the curriculum learning setup described in their work. This led to more stable accuracy improvements over training random samples done in NS-VQA.
The results are presented Table and we generally see slower convergence than the vision tri- als. This is due to the fact that the program produced has to be perfect to reason correctly, while some mis-labeled objects may not lead to incorrect reasoning. Furthermore we encounter spurious predictions of incorrect F leading the correct answer, prolonging training.

---

Under review as a conference paper at ICLR 2021
Looking closer at the results, at 1% QA data, the symbolic representation is top. We posit that this is due to model exhaustively sampling the program space over an end-to-end approach given limited data. However, the amount of data increases, the end-to-end soft programs show better accuracies and computational efficiency. With the results from these language and vision experiments, now have understanding of which methods are the most efficient for VQA.
4 VISION AND LANGUAGE END-TO-END REASONING.
We iteratively experimented on different visual, language, and reasoning VQA components to deter- mine which methods had the best accuracy and efficiency trade-offs. Based on this, we determined the following desiderata for efficient VQA methods:
Soft programs to propagate language parsing uncertainty.
Object-level detection models with pre-trained localization for object based reasoning.
Soft logic functions to propagate the uncertainties from the vision and language branches.
End-to-end differentiability to efficiently optimize the perception and reasoning jointly.
We are motivated to combine the existing Stack-NMN and NS-CL frameworks we tested to syn- thesize such a model. However, during our trade-off exploration found it non-trivial simply combine different methods across visual, language and reasoning components. We address two challenges regarding storing the intermediate computation in memory and the overall optimization when using these selected methods.
4.1 MEMORY
The first challenge is incompatibility at the reasoning level. NS-CL leverages fine grained functions that leverage objects O. The outputs all of Stack-NMN's soft programs that operate only on fr passed through coarse NMNs. To make such soft programs from Stack-NMN compatible with fine grained functions and object-level data, the memory storage handling the intermediate functional computation has store object-level data as well.
We design such a memory structure by pre-assigning different object-level data modalities to differ- ent parts of a memory matrix M E (x+++++) Here the first dimension is the stack dimen- sion used to store intermediate function outputs for future use, similar to Stack-NMN. The second (A + C + 1) dimension are the rows that store the heterogeneous values while Stack-NMN only stores D dimensional image feature fr. The first A dimensions in the row indicate the object set attentions miee, or which objects the model is paying attention to at that step. The C' stores concate- nated categorical value outputs possible by our vision models such as meolor and msize. The final dimension Mzum is reserved for some accumulated score such count or boolean bit for true or false questions. For the CLEVR specific case the object attentions meet E RAA The categorical values are [moooor; mahape: Mizzurrr Maise] E RCC and the numeric slot M.num E IR.
This enables computing the reasoning softly over NS-CL like object-level predictions and Stack- NMN function program layouts as BO- KITr P(F, O)s) After reasoning, the answers can be directly read from the memory instead of being inferred as a multi-class label. This is done by directly predicting the the question type from Q and accessing the correspond- ing memory location. For example if the question was asking about the color then we would return arg meotar- Now that have this fully differentiable architecture, focus the optimiza- tion procedure.
4.2 OPTIMIZATION
The second challenge is that Stack-NMN trains end-to-end while NS-CL iterates between REIN- FORCE for program parsing and end-to-end training for visual concept learning. NS-CL fixes the language model while end-to-end training vision, while want to jointly optimizing both lan- guage and vision models. This results in a much larger optimization landscape prone to spurious predictions from end-to-end training on QA data.

---

Under review as a conference paper at ICLR 2021
To mitigate this initially start by training the the vision MLP and language LSTM models with a small amount the direct supervision, using between 0.1% 1% of such data. This is done over the object (f/bbx.], label) and language (Q, FP) pairs available in CLEVR. Then train end-to-end on (Q. A) pair data where we have losses for categorical and numerical answers using, found it useful add the initial direct
cross entropy and MSE losses respectively. Additionally, supervision losses and data when training end-to-end as well, weighted by hyperparameters a and 3. We formulate this a regularization method that prevents the models from diverging spurious predictions when only provided with QA labels. We further provide details and demonstrate the efficacy of this regularization in Appendix C
All these terms give our overall loss as:
LEZE = LQAXEEt + LQAMSE + aLoxX + BLFP_EE
From these extensions over Stack-NMN and NS-CL we construct a fully Differentiable end-to-end Program executor (DePe). A more detailed description DePe's architecture and examples can be found in Appendix
EXPERIMENTS
We built DePe using the desiderata over efficient VQA methods and now its overall end-to- end performance jointly on vision and language. First we compare it to our base NS-CL and Stack- NMN methods in terms of our desired sample and computational complexity. Then we compare DePe's accuracy against other VQA methods.
5.1 EFFICIENCY PERFORMANCE
Computation Efficiency
Sample Efficiency
Validation Accuracy
100
80
QA Data
60
Stack-NMN
NS-CL
DePe
40
1%
43.6
68.1
65.4
20
5%
66.6
86.7
87.7
10%
80.6
98.8
98.0
0
0.2
0.4
0.6
0.8
Train Samples
-107
Figure 3: A comparison of efficiencies during joint training of vision and language. We test DePe with 0.1 % directly supervised labels.
We compare the efficiencies of DePe, Stack-NMN, and NS-CL in Table 3 To test the computational efficiency of NS-CL we trained the program parser till each iteration step, fixed the parser, and then trained the vision components end-to-end given the fixed parser.
Overall the results reflect the results reported during our component-wise testing. In terms of the computational efficiency, NS-CL. (Mao et al., 2019) uses REINFORCE to optimize its FP parser, which requires many more iterations to converge. Stack-NMN (Hu et al. 2018) also able to optimize end-to-end, but trains the fi representation requiring more training samples. DePe ing Stack-NMN soft programs and object-level execution from NS-CL able to more efficiently optimize over either method alone.
In terms of sample efficiency NS-CL and DePe are comparable given enough data since the models execute similarly once NS-CL language models fine tuned. Since DePe is trained some directly supervised data we also attempted directly supervising Stack-NMN and NS-CL, but saw similar performance discussed in the model detail in Appendix D
8

---

Under review as a conference paper at ICLR 2021
Method
Vision
Language
Reasoning
Training
100% Data
10% Data
MAC
Pixel
Text Att.
Neural
E2E
98.9
67.3
TbD
Pixel
Symbolic Prog.
Neural
E2E
99.1
54.2
Prob-NMN
Pixel
Symbolic Prog.
Neural
E2E
97.1
88.2
Stack-NMN
Pixel
Soft Prog.
Neural
E2E
96.3
86.8
LXMERT
Object
Text Att.
Neural
E2E
97.9
66.5
LCGN
Object
Text Att.
Neural
E2E
97.9
64.8
NGS
Object
Symbolic Prog.
Symbolic
Abduction
100.0
87.3
NS-VQA
Object
Symbolie Prog.
Symbolic
RL
99.8
92.1
NS-CI.
Object
Symbolic Prog.
Soft Logic
B2E + RL.
99.2
98.8
DePe (0.1%)
Object
Soft Prog.
Soft Logic
E2E
99.0
98.0
DePe (1.0%)
Object
Soft Prog.
Soft Logic
E2E
99.5
99.3
Table 3: Performance of DePe and other VQA models using 100% and 10% QA pair supervision. Here show DePe using 0. and 1% of the direct supervision on and FP.
5.2 COMPARATIVE PERFORMANCE
We present the accuracy comparison across differrnnt VQA models in Table 3 All models are able to achieve 96%+ given the full data set, but we are more interested in the results with lower sample complexity. At 10% data we observe that methods with continuous inference and DSL based neural modules, such DePe, NS-CL. Stack-NMN and Prob-NMN (Vedantam et al., 2019) scale better. The other methods are more on the neural side, such as TbD (Mascharka et al. 2018), MAC (Hudson & Manning 2018). LXMERT (Tan & Bansal 2019), and LCGN (Hu et al., 2019), require more data to converge. Methods that discretely sample, such as NS-VQA Yi et al., 2018)) or NGS (Li et al., 2020),, can also achieve high accuracies given more training data, but practice, require many iterations to converge and result high variance results compared to continuous methods.
CONCLUSION AND FUTURE WORK
In VQA there are different paradigms of modeling the visual, language, and reasoning representa- tions. In addition to the final model performance, ciency in these complex perception and reasoning tasks. First we introduced the existing models with their corresponding vision, language, reasoning, and training components used. Then we for- mally defined these components and the common method representations used within each compo- nent. In order determine which methods were the most sample and computationally efficient, iteratively tested the vision, reasoning, and language components. These results showed that object- level, soft program, soft logic, and end-to-end training were important for efficient VQA. Following this we modified existing models to leverage all of these efficient methods in joint manner. We showed that this model, DePe, was the most efficient while retaining state-of-the-art performance.
interested in understanding model effi
Based on these methods we look forward to testing DePe on larger real-world data sets. Since our model uses generic function programs to operate over language and vision can be extended to different data sets with minimal modifications. Furthermore we plan to investigate how to use concepts embeddings similar to NS-CL within our memory instead of one-hot representations. We will also be interested in testing how the object-level representations work questions involving both object and image level reasoning.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Felix A Gers, Jürgen Schmidhuber, and Fred Cummins. Learning to forget: Continual prediction with Istm. 1999.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference computer vision, pp. 2961-2969, 2017.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pp. 804-813, 2017.
Ronghang Hu, Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In Proceedings of the European conference on computer vision (ECCV), pp. 53-69, 2018.
Ronghang Hu, Anna Rohrbach, Trevor Darrell, and Kate Saenko. Language-conditioned graph net- works for relational reasoning. In Proceedings the IEEE International Conference on Computer Vision, pp.
2019.
Drew Hudson and Christopher D Manning. Learning by abstraction: The neural state machine. In Advances in Neural Information Processing Systems, pp. 5903-5916, 2019.
Drew A Hudson and Christopher Manning. Compositional attention networks for machine rea- soning. In International Conference Learning Representations, 2018.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
the
Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun. Zhu. Closed loop neural-symbolie learning via integrating neural perception, grammar parsing, and symbolic reasoning. International Conference on Machine Learning (ICML), 2020.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Super- vision. In International Conference Learning Representations, 2019. URL https:// openreview www.neee/oooooddeoeeeeu
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Clos- ing the gap between performance and interpretability in visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4942 4950, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing (EMNLP- IJCNLP), pp. 5103-5114, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 1998-6008. 2017.
10

---

Under review as a conference paper at ICLR 2021
Ramakrishna Vedantam, Karan Desai, Stefan Lee, Marcus Rohrbach, Dhruv Batra, and Devi Parikh. Probabilistic neural symbol- models for interpretable visual question answering. arXiv preprint arXiv:1902.07864, 2019.
Ronald J Williams.
Simple statistical gradient-folloingg algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):22--256, 1992.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural- symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, pp. 1031-1042, 2018.

---

Under review as a conference paper at ICLR 2021
A MODEL COMPARISONS
We explore few comparative works that contain variety of training strategies used for VQA. Each method type handles the vision, language, reasoning and training in different fashions.
LXMERT. Reasoning over images based on questions is carried out via Transformer like deep architecture. Such neural reasoning module can be easily interfaced with neural perception modules, and optimized end-to-end with perception module jointly. Since such model incorporate few prior structures into the model, it contains lots of parameters and requires lots of question answer pairs in order to achieve good results.
NMN methods. With Neural Module Networks (NMN), the language is turned into discrete function program over neural modules which act on pixel level attention to answer questions. The discrete function program allows reasoning steps be executed exactly. However, such design also makes the entire model not end-to-end differentiable. One needs to use REINFORCE to optimize the model for producing the function program.
An extension of NMN is Prob-NMN where the predicted program is supervised over a small set of samples. These ground truth programs provide a prior distribution over valid programs. This prior is used to determine future valid programs and enforce systematic program layouts.
In Stack Neural Module Network, reasoning instructions in the question are executed neural program over neural modules which act pixel level attention to answer questions. This neural program execution approach produces a soft function program, where discrete reasoning structures, such as differentiable pointer and stack, are incorporated into the neural program executor. This enables Stack-NMN to maintain uncertainty over which reasoning steps are selected.
GNN Methods. In Neural State Machine (NSM) and Language-Conditione Graph Neural Net- works (LCGN), images are represented as object and relations, and graph neural networks condi- tioned on the language feature are used reasoning modules. Graph neural networks are structured networks, which can represent logical classifier in graphs. Such graph neural networks and deep perception architectures end-to-end differentiable. However, the architecture is quite generic, and requires large number of guestion-answer pairs to train.
NS-CL. In Neural Symbolic Concept Learner, questions turned into symbolic program with soft logic function, which are then executed end-to-end on object attention produced by the vision model. The soft logic makes the reasoning step end-to-end differentiable. However, the functional programs are still discrete in structure, making the entire model not end-to-end differentiable. One needs to use REINFORCE to optimize the model for producing the function program.
NGS. Sairra-Grramm-SSymmmbr performs abduction, where both the image and language are turn into discrete object by sampling from the perception model, and symbolic program is exected to generate the answers.
In abductive learning, the symbolic reasoning step is directly interpretable, and many existing logic reasoning engine can be used. However, the model is not end-to-end differentiable. Discrete opti- mization needed generate supervision for the vision and language model to be updated.
B DIFFERENTIABLE END-TO-END PROGRAM EXECUTOR
The overall DePe architecture, as shown in Figure consists of multiple sub-parts. The percep- tion models that encode the vision and the question. The soft logic functions F closely follow the domain-specifi language (DSL) provided by CLEVR. We implement them in probabilistic man- ner based on the signature, detailed in Appendix E The function execution results are stored in a differentiable memory that maintains stack of these operations. As model executes step by step, this memory used as a buffer for the reasoning chain's probabilistic steps leading to a final answer.
B. PERCEPTION
Object detection. For our vision, we use object-level detection through a Mask R-CNN (He et al. 2017) to extract the corresponding bounding boxes of the objects. We then take detected objects and
12

---

Under review as a conference paper at ICLR 2021
DePe Overall
Object Parser
Repeat for K reasoning steps
VQR Cell
VQR
VQR Cell
What is the shape of the object behind the tiny red thing?
Question Parser
*sphere
Memory
Question
Question
Parser
Question Parser
Encoder
VQR Cell
Decoder
Object Parser
Arguments
Function Attention
Q Type Head
Function Head
Function Argument Heads
Function Executor
Fused Memory
Function
Memory
query_shape (filter_size) 0.8, ...) (small: 0.9, ...) (fiiterccolor 0.8, ...) (red: 0.9, ...]
Memory
Update
Figure 4: Our model ingests the image and extracts object-centric information. The question (textual input) is used by the question parser, which first embeds the question through an LSTM encoder. This embedding is used to predict the question type, which will be used to retrieve the answer from memory at the final step. The encoded text is passed to a decoder, which at every step predicts attention over which functions execute and each function's arguments at the current step. The objects, arguments, and any preceding memory inputs are used by each function to execute, stored in memory belonging to each function. These function memories are weighted by functional attention and fused, which updates the original memory. The decoding and the VQR Cell execution run fixed number of times, and then the answer is extracted from the final memory.
pass them through pre-trained ResNet-50 model (He et al.) 2016) extract image features. These features are fed into models to predict object attributes from the image features and their bounding boxes' object relations.
Question parser. The parser takes in the text input and encodes it through a Bi-LSTM to generate the final hidden state. This hidden state is used in the program parser processes where a function model predicts the attention over which soft function is to be used at that step. Additionally, these functions may contain arguments, such filter weteersshape shape [cube], so for each class of functions, we have model to predict a distribution over the possible attributes. The hidden state is also used predict question type, which is used select the final memory answer at the end of the execution.
For the vision and the language, all prediction models are small MLPs. We denote the set of trainable vision and text models parameters as Opision and Oxext respectively.
B.2 MEMORY
In the CLEVR DSL,, the functions have different types of inputs and outputs. Stack-NMN used neural modules approximate these DSL functions, thus were able make the image attention output of a consistent length. Since we are using the soft version of the DSL, we had to design memory structure that could handle varying input and output modalities and sizes.
Memory block. To create a heterogeneous memory structure, we pre-assigned different modalities to different parts of a matrix. This memory matrix block M E (T++A+++1) used to store the intermediate results of the function computation.
13

---

Under review as a conference paper at ICLR 2021
Here the first dimension T is the stack dimension on which the function outputs stored in dif- ferent steps of the execution. The second (A+ + C' + 1) dimension are the rows that store the het- erogencous values. For the rows, A elements in the row indicate the object set attentions madets or which objects the model is paying attention to at that step. The C' stores the categorical value outputs possible by vision models such Meotor msize- For example if there 6 possible colors, then Mootor E RR The final dimension Mmum is reserved for some accumulated score such as count or a boolean bit for true or false questions. For the CLEVR specific case the object at- tentions deet E RA.. The categorical values are [Meolor: Msappe; Miteturre Mgizel E RC, and the numeric slot deet E R.
Stack structure. Choosing which row & E T of the memory is accessed is handled by the stack. This enables the model accurately retrieve the previous one or two stack values as needed by the functions, instead of to predict which locations in memory to use.
Some functions may require just looking at the previous output from the VQR Cell, such as chain-like questions over a scene graph through functions such as filter, relate, sum. Such functions will pop from and then push to the same memory row. There are situations where functions need multiple inputs as well, such comparison functions count_equal. These functions will thus pop two values from the stack and will only push back one. These function signatures are summarized in Table iin Appendix E
Stack manipulation. Each function will have access to the stack, which abstracts the memory management appropriately through push and pop operations. The stack memory M is initialized at random and a one-hot memory pointer p E IR"", starting at po = 1. Each function returns a row m or specific slice such Meator be updated in memory.
To push a row m onto the stack we follow the Stack-NMN convention updating the pointer as p Id_conv(p, [0, 0, 1)) and the memory row M; M; (1 p.) + m mm-pi. Here the convolution is just moving the one hot pointer left by one in a differentiable manner. Then only the memory row that contains the one hot pointer updated with the row m. Similarly to pop value we perform the following retrieve the row m Σi= Ps M, and push back the pointer right by p cd_con(p,, [1, 0))
B.3 VQR CELL
The visual question reasoning (VQR) Cell is responsible for the step by step executions over the vision and the text to get to the answer. Compared to previous methods that executed these DSL functional programs in a discrete sequence, we generate probabilistic output over all the functions F at each step. With this approach, we can propagate the uncertainties of the object detections and the question parser for end-to-end learning. An example of this execution is visible in Figure [
Soft function execution. Each function has different return values and signatures. Since functions can have different input and output requirements, they require to be at different positions of the stack. Therefore each function is given its copy of the memory stack to operate over.
Once the memory outputs for each function are com- puted, they have to be weighted by which function our model is most likely needed for this step. This is done by using the function attention weights computed by the question parser. These weights scale each function's 's memory and the pointer and then computes weighted sum, which is then used to update the global memory M and pointer p, shown in Algorithm [ This global memory is then copied for all the function memories in the next Cell iteration. The cell executes for fixed num- ber of iterations and is set to number such that it can cover most ground truth program lengths.
Algorithm 1: VQR Cell itera-
tion j [[] do My = (M, end Mave Σ Wy. Mj p softmax (E)=) Wz.Pp) PT)
for
= M (1 p) + Mave p
Final answer. After the last iteration of VQR Cell execution, the final answer is chosen by the memory corresponding to most probable question type. For example if we predict the question is asking about the color, then we return )rrr If we predict the question type is about count, then return Miuam Since our memory structure is heterogeneous, taking argmax over the
14

---

Under review as a conference paper at ICLR 2021
what is the shape of the object behind the tiny red thing?
Function Head
Function Argument Heads
Object Parser
Memory
Iter 1. P(Filter Size) 0.9
P(small) = .9
P(small) = [.96, .88, .23)
= =[.77, 711, .18] -> Mast
Iter 2. P(Filter Color) 0.72
P(small) .88
P(small) = [.01, 91,, .88)
=[.00, .41, .10] >> Moet
Iter P(Unique) 0.99
sharpen(Mae) [.00. (00) matt
Iter P(Relate)
P(behind)
P(behind) [001, .97))
[[00, 00000000..00 .72] Moet
Iter P(Unique) 0.99
sharpen(Mee) [.00, .98, 000] >> moet
Iter 6. P(Query Shape) 0.72
P(shape=sphere obj = .89
[.00, .41, .10] -> masape
Answer: "Sphere"
(cylinder: 00, sphere: 62, cube: .00)
Figure 5: Here is a sample execution for the vision and text DePe. For the simplicity of the diagram, don't include all the probability traces and just show the ones corresponding to the max probability. For the vision, the predictions correspond to the objects detected in the scene (marked with 2, 3 in the picture). For each execution, store the multiplication in the corresponding memory row, and retrieve the final memory to answer the question.
entire row m may lead to an incorrect label due to parts of the memory storing values at different ranges, such probabilities [0, 1] or counts Z
B.4 OPTIMIZATION
The objective is minimize the overall loss based the predictions retrieved from memory. This involves a multi-task loss as we are predicting categorical and numeric answers to questions to op- timize over Oseat: Ovision. This is done by minimizing the cross entropy loss for categorical answers and mean squared error loss for numeric ones: L LQAXXER + LQA.MSE-
This optimization, particularly for the has posed be difficult problem from scratch there are many candidate functions at each step in the function program. Furthermore spurious programs can be proposed in early stages of training. corresponding to the shortest programs that answer the question, but don't it align with the ground truth semantics.
To address this, models such as NS-CL, employ structural constraints when training their text parser. They employ a fixed number of concepts tokens used to parse the question and discrete templates for the parser to follow. In addition they leverage curriculum learning and switch between optimizing the parser and vision models to support both the discrete and continuous training methods.
Other models, such as NS-VQA, train on small portion of the supervised labeled data. In this manner such restrictions the concepts, templates, curriculum learning, but require labeling of ground truth programs. In this work we explore the pre-trained route. We pre-train our models with small percentages (0.1-1%) of visual and textual data. When use X% pre-training data, we sample X% of the training questions. Then we directly supervise our question parser and vision models those QA programs and their corresponding vision scenes.
Additionally, we found it useful to add these direct supervision losses when training end-to-end with corresponding weights and B. We formulate this as regularization method that prevents the
15

---

Under review as a conference paper at ICLR 2021
models from diverging to spurious predictions when the training signal is only coarse QA labels.
= Ent + LQAMSE + aLou).XEnt + 3CTent.EEnt
The full definitions and learning strategies for the losses available in Appendix
C OPTIMIZATION DETAILS
If given and Image and a question Q our model makes a prediction as follows. We first make predictions of the objects and text components as:
DePe(T, (, Q; Otext, Ovision)
We look to minimize the following loss L.
N
C
LQA.XEnt =
D. Vig log ÚJ
N
N
LQAMSE N
Σ
I Enam 1188 - Inum 12
L LOA + LQAMSE
We note that for stable convergence we include the pre-training data to our loss function. These are the cross entropy loss for the object attribute and relation predictions Lobj Ext. These also the cross entropy loss for the question parser predictions of the functions at each step X Ent- This gives our final loss function:
L = LQAXEE + LQAMSE XLot + 3LText.XEm
We include two scalar terms for the pre-training losses to balance the models behavior to find local minima within the QA losses and focusing too much optimizing the smaller number of pre- training samples. Setting these hyperparameters more prelevant when the ratio of pre-training to QA samples low, since the pre-training samples provide a weak signal, but should be respected. We find that setting = 3 = /% of QA data, works well in practice to balance this ratio. The gains of setting these hyperparmeters can be seen in Table
% pre-train weight
0.5%
pre-train weight
1% pre-train weight
% QA Data
3
base
A
base
A
base
A
5%
3
87.7
97.9
10.2
97.1
99.1
2.0
98.8
99.2
0.4
10%
6
92.3
98.0
5.7
96.6
99.2
2.6
99.1
99.3
0.2
100%
10
93.1
99.0
5.9
97.7
99.4
1.7
99.4
99.5
0.1
Table 4: Absolute validation accuracy gains by adding pre-training loss weights. The gains were computed against the baseline method of a = 3 =
We use Adam to optimize with Ir le 1e-4_ 4, 1 = 0.9, 32 = 0.999, c = and set our batch size 1024.
D EXPERIMENTS
Here outline the details of the training procedure for the end-to-end training and efficiency periments. We follow the default training procedure for the relevant work it not modified this section.
16

---

Under review as a conference paper at ICLR 2021
D.1 LXMERT
For LXMERT we were interested in exploring if the cross modal architecture could implicitly encode our attention based reasoning process, as conveyed by the recent Transformer literature. There were a couple of approaches that we tried during our tests.
The initial tests and the results we reported were the based on the original architecture proposed in the paper well as using the pre-trained weights the authors provided. This used 9, 5, and 5 layers of the language encoder N,. the cross modality encoder, and the region of interest (Rol) object encoder NR respectively.
We also test the initializing the Rol encoder from scratch since the CLEVR features are more simpler than the ones used in pre-training on real images. Additionally we tested LXMERT with fewer
layers, N 4, Nx = 2, NR = 2, which we report in Table
% QA Data
LXMERT Pre-trained
LXMERT Scratch
LXMERT 422
100
97.9
98.0
95.6
95.3
93.5
90.4
25
89.2
69.27
70.71
10
66.5
50.55
52.72
Table LXMERT CLEVR performance on different subsets of data and configurations.
D.2 STACK-NMN
When experimenting with Stack-NMN we first trained the entire model on 100% of the CLEVR data using the ground truth layout and save it. For the visual representation experiments we take the trained model and freeze the LSTM weights while resetting the vision weights. For the language representation we freeze the CNN and NMN weights without the expert layout.
When training Stack-NMN for direct supervision, we train the model with the ground truth layout but only on 0.1% of the data. Then take load this model and ran it on the corresponding 1, 5, and 10% QA data without the expert layout. With only 0. 0.1%, we didn't see any sample of computational performance differences on end-to-end training.
We used the same settings as the original paper and trained all variations for 200k iterations.
D.3 NS-CL
When testing NS-CL end-to-end we require iterating between optimizing the language program parser and training the vision concept models, done in REINFORCE and end-to-end fashion spectively. When conducting the computational complexity experiments we had to train and test the program parser and vision models fixed number of training samples. To do this divide up the total number of training samples uniformly into 10 curriculum lessons used in the original NS-CL. We then tune the program parser with the data belonging to the cumulative curriculum up those number of training samples. Given this program parser we tune the vision models end-to-end over the same number of training samples. This way we have a better picture of NS-CL's behavior as a function of the training iterations.
To make NS- CL more comparative to DePe, we also use the 0.1% direct supervision data improve the NS-CL initial performance. We similarly train the program parser on the (Q, FP) pairs, thus start off at similar accuracies. Unlike DePe, NS-CL uses similarities between text and vision con- cepts, soit was unclear how to directly supervise the vision model given the object data. Similar to Stack-NMN we test a burn-in strategy where we train the entire model end-to-end with the QA Data from the corresponding 0.1% 1% direct supervision then train using the 10% QA data for 150 epochs. Here we observed no significant gains from just start off with the supervised vision parser.
In the reasoning representation experiments we test NS-CL with discrete reasoning. Here we took the original soft logic F defined in the NS-CL paper and replaced them with symbolic logic as used in NS- VQA. Then we used REINFORCE optimize when training on QA data in NS-RL. In NS-
17

---

Under review as a conference paper at ICLR 2021
AB use abduction to find the most likely change in the object concept prediction to make the resulting reasoning provide the correct answer. We discuss this the next section.
D.4 NS-AB ABDUCTION
Abduction involves determining which detection mide changes required to correctly answer the question given a fixed discrete function program.
At high level these changes done by making the most likely change that makes the output answer right. NGS they use a tree based formulation and in our implementation we retain our stack implementation. Our stack implementation lets us test abduction proposals in greedy efficient manner, but leads to smaller search of the detection space leading to the ground truth answer instead of spurious one. We describe the details of implementation below.
D.4.1 POLICY DEFINITION
The function program Q (Ai) E Ftells to execute sequence of operations functions f;. If there are multiple reasoning branches that are aggregated, the execution trace follows a DAG dependency structure. We use Markov decision process (MDP) formulation for the execution process:
A policy maet To(mm)s Im:. I) with vision models parameters Opision will take the operation execution from the previous stage m., the operator indicator Sr and image 7, and output action a₁ Memory from the previous stage m could be any memory slice such as Myett Mottr; Myum- This action m corresponds to the object selections made for the current operation from the input image.
The action will cause the state m to be updated to m and the transition is described by P(mę+1 m,, medet ft). This is the operation execution for the current stage. This is carried by logic function ft as described in Table
After this sequence of operations, state with the answer A, and tell us whether the answer correct or not. If the answer is correct R and otheriwse O. Then the expected correctness of answer given the uncertainty in Ty and P is
obtain the reward R( R(A, 1) by comparing the last selected
E[R(A, m] where
(1)
m ~ p(m++1 madets fe), at ~ To(m Im,, I), m = VI = 1,...,7 T
(2)
D.4.2 OPTIMIZATION
Given m data points expressed as image I. questions Q and answer triplets:
(T1, Q1, A1)...,, (Im, Qmy Am), the learning problem can be expressed
TØ
arg max Ty
Σ E( mf1+1]]
(3)
Label abduction. If for particular data point (I, (A the answer according to the model is incorrect, one can try to find corrections by making minimal corrections e; the model detections as follows:
ci. eτ -argmin
D(m
(4)
s.t. R ml+1) 11 makes the answer right.
(5)
m P( Im,, Ct, ft), Vt I,...,
(6)
where D(I) is some distance measure between the corrections e; and the model predictions m For instance, D(I)) can be the negative likelihood of the correction under model no. the edit distance.
Intuitively, want attempt the most likely corrections iteratively to find solution by minimizing D(I)). Sampling all possible corrections at once, can cause right answers through spurious label changes, which we want to mitigate.
18

---

Under review as a conference paper at ICLR 2021
Sampling methods. Due to the compositional nature of the reasoning tasks, attempt to optimize oF in a greedy fashion at each step of the program from to instead of jointly from i S (l)...,T).
This better enforces the consistency constraint when single of update leads to valid program exccu- tions. Valid executions mean that the predicted abduced labels lead to final answer, whether right or wrong. This opposed to making conflicting changes in all c Cj where i * j. leading failed program execution due the manual abduced changes. If this greedy approach fails to find an answer, we fall back on exhaustively sampling mett ~ Ty at all program levels for a fixed number of iterations.
E MODULAR SOFT-LOGIC FUNCTIONS
The descriptions of the variables and constants used to describe memory components are listed in Table 6 The functions used are in Table 7 For notation simplicity, the function arguments are assumed be popped from that function's memory copy or predicted from the or vision models.
Name
Module
Description
Values
D
Hyperparameter
Number of objects
Z+
Y
Hyperparameter
Shift value
Hyperparameter
Scalar value
R
Mddet
Memory
Gates determining the active object detections
{ 1}}
Mham
Memory
Storage for numerical and boolean operations
R
Mattr
Memory
Probability distribution over that attribute
Blattr]
Purg
Text
Function argument probability distribution
R |uttrl, Rlrell
Prel
Vision
Object pairwise relation predictions
R DxDx]rell
Pattr
Vision
Image attribute prediction per object
DX|attr]
Table 6: Description of functional arguments and constants.
19

---

Under review as a conference paper at ICLR 2021
Signature
Implementation
scene()
Mdet == 1
unique(mae:)
==
Mater
sum
mater
count(mact)
Myum = sum(Madt)
exist(matt)
Maum = max(mdet)
intersect(m m m2zr)
Mydt == min maet)
union(m ders m2
Midet == max m²ct)
equal integer(m) m mmmm
Maxm := sigmoid( |M m 1)/(())))
greater Muum)
Mnam sigmoid ((m)um Mmum - 7)/
less_than(m1hum M mmmm
Mnam sigmoid (mmuum M 7)/T)
relate(madt- Prete Parg)
Maeet (mac)) (Prat)
Atter.attr(Mdde.. Pattr. Parg)
Mieet min mide (Patte aetrr (Parg))
query attr( Pattrs Porg)
Mattr min(Pargy (Pore)) (Maez)
same attr(mmaet: Pattr)
Made
((Patte) (Pattrr)aaana (Pattr) @ (1 (Maee))
equal attr(m m matte)
mmum = (m (m
Table 7: Implementation details for each modular function. mdet, Myums Mattr correspond to differ- ent parts in the memory representing attentional masks, numerical results, and attributes. Pory the distribution of functional arguments produced by the question parser, while Pret and Pattr rela- tion and attribute predictions given by the perception module. Hyper-parameters D = 50, r = 0.25 and y 0.5 and attribute functions are split further by each attr (shope, color, material size).
20


Reviews:
[{"review_id": "wbhmMu7HwXc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**PAPER SUMMARY**\n\nThe paper presents a method for visual question answering (VQA) that makes use of a differentiable program executor that softly approximates the execution of neural modules that read and write to a stack. Experiments on CLEVR demonstrate that the model can achieve high performance using small amounts of supervision (varying the amount of question/answer pairs, image/object annotations, and question/program annotation used during training).\n\n**STRENGTHS**\n- The method achieves differentiable program execution without resorting to training with policy gradients\n- Experiments demonstrate high accuracy on CLEVR, even using very little training data\n\n**WEAKNESSES**\n- Critical missing citation / comparison: Prob-NMN (Vedantam et al, ICML 2018)\n- The model hardcodes a lot of task-specific knowledge; is this useful?\n- The writing is very unclear in some places\n\n**UNCLEAR WRITING**\n\nI find the paper to be quite unclear about exactly what type of supervision is used where. Section 3.4 states that “We pre-train our models with a very small percentage of visual and textual data” -- but what kind of data is this exactly? If I understood the following discussion, I think that “image data” refers to (image, object) annotations used to train the Mask R-CNN model, while “textual data” refers to (question, program) pairs used to supervise the program executor. Then in Table 3, “X% pre-train” refers to the amount of (image, object) and (question, program) annotations used to supervise the model, and “Y% of Training Data” refers to the number of (question, answer) pairs used to supervise the model. I think that the clarity of the paper would be much improved by making it more clear as to what types of supervision are used at different stages.\n\nI similarly found Section 4.2 and Tables 4 and 5 very unclear, and I’m not entirely sure what experiments are being performed; what does it mean to “fix vision and train text from 0.1% pretraining” in Table 4, or “fix text and train vision from scratch” in Table 5?\n\n**MISSING COMPARISON: PROB-NMN**\n\nOne of the main draws of the proposed method is that it is extremely data-efficient on CLEVR, e.g. achieving 99.1% accuracy when supervised with 10% of (question, answer) pairs and 1% of (image, object) and (question, program) pairs.\n\nThis sort of data-efficient learning is also a key feature of Prob-NMN [1]; this is a critical piece of prior work which is not cited; the proposed method be discussed in light of this prior work, and the method should be compared against it. From Table 2 of [1], Prob-NMN achieves 97.73% test-set accuracy on CLEVR when supervised with 100% of (question, answer) pairs and 0.143% of (question, program) pairs; this outperforms the 93.1% reported for DePe in Table 3 of the submission which uses 100% of (question, answer) pairs and 0.1% of (question, program) and (image, object) pairs. However this comparison is not fully valid, since Prob-NMN uses slightly more program annotations but uses no explicit object detector. The authors should set up a more controlled experiment to compare their method against Prob-NMN.\n\n[1] Vedantam et al, “Probabilistic Neural-symbolic Models for Interpretable Visual Question Answering”, ICML 2018\n\n**HARDCODED REASONING**\n\nThe proposed model achieves data-efficiency by encoding a huge amount of task-specific prior knowledge into the model architecture:\n- The use of any ground-truth programs assumes that the set of primitive operators needed to answer questions are known\n- The module designs in Table 2 go a step further, and assume that the implementation of all of these primitive operators is known -- the operators in Table 2 are effectively a differentiable approximation to the ground-truth CLEVR program executor\n- The use of an object detector assumes that all questions can be answered in terms of object positions and properties; that the vocabulary of semantic concepts needed to answer questions is known; and that furthermore the mapping of images -> object locations and properties can be supervised during training\n\nTaken together, the model essentially hardcodes the data-generation process of the CLEVR dataset into the model. This is evident from Table 3: if I’ve read and understood the experimental setup, training with 1% of (question, program) annotations and (image, object) annotations and zero (question, answer) annotations achieves >90% test-set accuracy on CLEVR! To me this is not a positive result -- instead it signifies that the model architecture itself is overfit to the dataset.\n\nThe higher-level research goal of building neuro-symbolic models that elegantly blend prior knowledge with learning is extremely exciting. But testing this kind of model on CLEVR is not. CLEVR is a synthetic dataset, and its questions are not natural: they are generated programmatically, and there is a lossless and deterministic mapping between question text and semantics (encoded as functional programs). CLEVR images are likewise not natural, and the image semantics necessary for answering any CLEVR question can likewise be losslessly encoded into a scene graph. As such, the fact that this model achieves data-efficient learning on CLEVR is nearly as uninteresting as the fact that the ground-truth CLEVR program execution engine achieves perfect accuracy when fed with ground-truth scene graphs: it is nearly guaranteed by construction, and only works by exploiting the synthetic nature of the benchmark.\n\nMy biggest concern with this line of work is that I don’t see how it could possibly be used to achieve visual reasoning on real-world data, where none of the above can be exploited: for true natural language and images, we do not know a vocabulary of symbolic primitives that can losslessly represent the world and we certainly can’t hardcode the implementation of all symbolic primitives necessary for performing real-world reasoning.\n\nSo although this method may indeed achieve extremely data-efficient learning on CLEVR, this is not an interesting achievement on its own unless there is some hope that the method used to do so can also be used for non-synthetic, real-world data. And the submission presents no evidence, or even an argument, for how this might be done.\n\n**OVERALL**\n\nOn the whole, I think that this paper is not ready for publication due to unclear writing, and the fact that it is missing a critical comparison to Prob-NMN.\n\nI also take issue with the underlying approach, since I think the methods used to achieve data-efficiency on CLEVR cannot scale to real-world data. However I can accept that different people in the community feel differently from me on this issue (as evidenced by excitement over papers like the Neuro-Symbolic Concept Learner), so I would not necessarily consider this as a disqualifier if it were my only concern with the paper.\n\n\n**AFTER REBUTTAL**\n\nI appreciate the author's efforts to improve the clarity of the submission and to add a comparison with Prob-NMN. However I remain unconvinced that this style of model can possibly scale to more realistic data. I've upgraded my score from 4 to 5, but I still generally lead toward rejection.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "tPVKmIR-xV-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a neuro-symbolic model for sample-efficient VQA, which turns each question into a probabilistic program which is then softly executed. The problem explored in the paper and its background and context presented clearly and it does a good job in motivating its importance and trade-offs between possible solutions. While the use of a probabilistic program to represent the questions might be too stiff / inflexible in my opinion and may not generalize well to less constrained natural language, this direction is still of course important and interesting. It also does a great job in presenting the existing approaches and comparing their properties. The writing is good and the model is presented clearly with a very useful diagram. \n\nHowever, the novelty of the paper seems limited to me, as it mainly combines together ideas that have been extensively explored in many prior works which are mentioned by the paper. Turning the question into a series of attentions over semantic factors appears in the NSM and partially in MAC models. The iterative memory updates appeared in MAC. Combining together small operations and functions defined by hand as dictated by programs as in page 6 of the model is the main idea of Module network. End-to-end differentiability for VQA models has also been extensively explored and multiple solutions have been proposed: relations networks, soft variants of NMN, and also MAC and NSM, etc. The use of stacks has been explored too in the stack-NMN model. I therefore feel the paper mostly recombines and tunes together these ideas rather than offering one particular new idea or insight. \n\nThe paper presents results on CLEVR only, which goes back into my concern about the inflexibility of the probabilistic programs. Especially for this type of models, it will be useful to explore it on tasks beyond CLEVR such as VQA/GQA to show whether it can work for natural or richer language. The use of Mask R-CNN on CLEVR is also quite unreasonable in my opinion: the task has meant to be visually simple, so using a very strong visual model on it nullifies the visual aspect of it completely, making the model working on perfect semantic scene graph inputs rather than on \"real/natural\" uncertain and more noisy inputs. It also gives unfair advantage to the model when comparing to baselines which didn’t use object detectors on CLEVR but rather work directly with the image, e.g. MAC and others (presented in the table in the experiments section).\n\nAt the same time, it is important to mention the model does get quantitative improvements in scores and especially sample efficiency, but the paper doesn’t make it clear what is the particular property or part of the model that allows for the improved numbers, and so the paper doesn’t leave the reader with a clear new takeaway message.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "mW0MLgq153h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies visual question-answering (VQA). The authors proposed an end-to-end differentiable framework that performs \"soft\" logical reasoning based on object-centric scene representations and attention-based language parsing. The authors claimed that the new model is more data-efficient.\n\nWhile I think this direction is interesting and useful, I think the submission really needs more work before it may be published at a top-tier conference. The current manuscript is hard to understand and the results are not convincing.\n\nFirst, the method is not well-motivated. The intro mostly discussed related papers and their differences, so I was expecting an analysis paper, but it turns out the rest of the paper is about a new model. There is no justification for why we need a new model, what is new about it, and why we expect it to do better. \n\nApart from this, the writing is in general unclear. For example, the model has an acronym DePe, but the full name was never fully told. The experiment section is also poorly written. In 4.2, it's unclear what \"Train Ratio\" means in Table 4 or 5, and these tables were never referred to in the main text.  I also don't understand the setup in Section 4.3, either, as essential descriptions are completely missing.\n\nThis makes it hard to assess the results.  In addition, all results are on the CLEVR dataset. One selling point of methods such as MAC, NS-CL, and Stack-NMN is that they can naturally be applied to real images such as the GQA dataset. Without experiments on real images, the analyses presented in this paper are unlikely to convince researchers in this area.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in agreement that this paper could benefit further improvement. There are several areas: novelty of the proposed approach and evaluation on real-world datasets (beyond just CLEVR).
Decision: Reject