Supervising the Transfer of Reasoning Patterns in VQA
Authors: Corentin Kervadec
Christian Wolf
Grigory Antipov
Moez Baccouche
Madiha Nadri
Venue: NeurIPS 2021 Poster (2021)
Keywords: Visual Question Answering
Visual Reasoning
Deep Learning

Abstract: Methods for Visual Question Anwering (VQA) are notorious for leveraging dataset biases rather than performing reasoning, hindering generalization. It has been recently shown that better reasoning patterns emerge in attention layers of a state-of-the-art VQA model when they are trained on perfect (oracle) visual inputs. This provides evidence that deep neural networks can learn to reason when training conditions are favorable enough. However, transferring this learned knowledge to deployable models is a challenge, as much of it is lost during the transfer.
We propose a method for knowledge transfer based on a regularization term in our loss function, supervising the sequence of required reasoning operations.
We provide a theoretical analysis based on PAC-learning, showing that such program prediction can lead to decreased sample complexity under mild hypotheses. 
We also demonstrate the effectiveness of this approach experimentally on the GQA dataset and show its complementarity to BERT-like self-supervised pre-training. 

Supervising the Transfer of Reasoning Patterns in VQA
Corentin Kervadec*1,2 Christian Wolf*2 Grigory Antipov
Moez Baccouche!
Madiha Nadri
Orange Innovation, France
LIRIS, INSA-Lyon, France LAGEPP, Université de Lyon, France firatname. lastnamedorange com, christian.wol// www.w@isa-lllnnn.... fr, madiha. madiha.nadri@ll nadri@lagep.pp... univ-lyon1.fr *equal contribution
Abstract
Methods for Visual Question Anwering (VQA) are notorious for leveraging dataset biases rather than performing reasoning. hindering generalization. It has been recently shown that better reasoning patterns emerge in attention layers of state- of-the-art VQA model when they are trained on perfect (oracle) visual inputs. This provides evidence that deep neural networks conditions are favorable enough. However, transferring this learned knowledge deployable models is challenge, much of is lost during the transfer. We propose a method for knowledge transfer based on a regularization term in our loss function, supervising the sequence of required reasoning operations. We provide a theoretical analysis based on PAC-learning, showing that such program prediction can lead to decreased sample complexity under mild hypotheses. We also demonstrate the effectiveness of this approach experimentally on the GQA dataset and show its complementarity BERT-like self self-supereiee pre-traie pre-training.
learn reason when training
Introduction
Reasoning images is the main goal of Visual Question Anwering (VQA), task where model asked to answer questions over images. This problem a test bed for the creation of agents capable of high-level reasoning, as it involves multi-modal and high-dimensional data well as complex decision functions requiring latent representations and multiple hops. State-of-the-art models are notorious for leveraging dataset biases and shortcuts in learning rather than performing reasoning, leading to lack of generalization, as evidenced by extensive recent work on bias oriented benchmarks for rision-and-langual reasoning 20 21 31). Even large-scale semi-supervised pre-training methods, which successfully managed to increase overall VQA performance, still struggle to address questions whose answers are rare given a context (20].
has been recently shown that reasoning patterns emerge in attention layers of a SOTA VQA model when trained perfect (oracle) visual inputs, which provides evidence that deep neural networks can learn to reason, when training conditions are favorable enough (21). In particular, uncertainty and noise in visual inputs seems be major cause for shortcut learning in VQA. While this kind of methods provide strong empirical results and insights on the bottlenecks in problems involving learning to reason, they still suffer from significant loss in reasoning capabilities during the transfer phase, when the model is required to adapt from perfectly clean visual input to the noisy input will encounter after deployment. We conjecture, that reasoning noisy data involves additional functional components not necessary in the clean case due to different types of domain shifts: (1) presence shift, caused by imperfect object detectors, leading to missing visual objects necessary for
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Leamed knowweddge of the reasoningg
Latent variables necessary for reusoning over multiple hops [used reasening processes)
Decamposi of underlying (unlmown)) reasoning function
( Clastical training, CE loss on answers y
(b) CE loss on program prediction
Figure I: VQA takes visual input e and a question and predicts distribution over answers (a) Classical discriminative training encodes the full reasoning function the network parameters while the network activations contain latent variables necessary for reasoning over multiple hops. (b) Additional program supervision requires intermediate network activations to contain information on the reasoning process, simplifying learning the reasoning function Under the hypothesis of it's decomposition into multiple reasoning modes, intermediate supervision favors separately learning the mode selector and each individual mode function. This intuition is analyzed theoretically in section 4
reasoning, or to multiple (duplicate) detections: (2) an appearance shift causing variations in object embeddings (descriptors) for the same class of objects due to different appearance.
In this paper, we propose a new method for transferring reasoning patterns from models learned on perfect visual imput models trained noisy visual representations. Key to the success is regularization term minimizing loss of the reasoning capabilities during transfer. In particular, we address this problem through program prediction the sequence of reasoning operations along with their textual and/or visual arguments. To maintain a strong link between the learned function and its objective during the knowledge transfer phase, when inputs are switched from clean oracle inputs to noisy input, the neural model is required to continue to predict complex reasoning programs from different types of inputs.
additional auxiliary loss, i.e. supervision of
As a second justification, we claim that program supervision in itself leads to a simpler learning problem, the underlying reasoning function is decomposed into a set of tasks, each of which is casier to learn individually than the full joint decision function. We backup this claim through a theoretical analysis showing decreased sample complexity under mild hypotheses.
As summary, we present the following contributions: we propose new program supervision module added on top of vision-language transformer models; (ii) we provide theoretical analysis of the benefit of supervising program prediction in VQA deriving bounds on sample complexity: (iii) we experimentally demonstrate the efficiency of program supervision and show that it increases VQA performance on both in- and out-o--distribution sets, even when combined with BERT-like pre-training [30] [0], and that it improves the quality of oracle transfer initially proposed by (21).
Related Work
Transformers in VQA VQA as a task was introduced in various datasets 4 12 16 including GQA [15) which is ceennaaiccall-ggeeratt
from real-world images. This growing amount of diverse datasets has been accompanied by the development of more and more sophisticated VQA models. While their exhaustive survey is out of the scope of this paper, one can mention some foundational categories of approaches, e.g. those based on object-level attention 2 and tensor decomposition [5]. In this work, we focus VQA models which are based Transformers (33, due to their wide adoption and their impressive results in several tasks (including VQA). In particular, we rely on the [00 llke pretraining which was shown be beneficial for VQA in recent works [19] 30). More recently, (21) focused on the study of so-called reasoning patterns such Transformer- bassd VQA models. The authors analyzed how various VQA tasks are encoded in different attention heads, by applying an -ased analysis inspired by [24). The analysis was performed using perfect-sighted oracle Transformer model (which is trained with near-to- to-perfect visual information, and thus, is much less prone to exploit dataset biases) in order to identify which patterns lead to better reasoning. Then, the authors showed that these reasoning patterns can be transferred from the oracle to a Transformer-based VQA model, thus improving both overall accuracy and accuracy on infrequent answers. In this work, we argue that using program prediction an additional supervision signal is catalyst for the transfer of these reasoning patterns.
combination of Transformers with a large-scale BERT 10

---

Biases and shortcut learning in VQA - In addition to the popular VQA datasets mentioned above, other benchmarks were proposed evaluate specific reasoning capabilities of VQA systems. In particular, they address the issue of shortcut learning [11] in deep learning. where models learn decision rules which are ineffective when tested on domain or distribution different from the training one. For instance, VQA-CP [ explicitly inverts the answer distribution between train and test splits. To cope with recent criticisms regarding these evaluations [31 28), the GQA-0O1 dataset [[8] introduced new split of GQA focusing on rare -f / OOD) question-answer pairs, and showed that many VQA models strongly rely on dataset biases. Following this work, the VQA-CE dataset [8] introduced new evaluation approach related to the VQA v2 dataset. By studying the multi-modas shortcuts in the training set and mining some trivial predictive rules (e.g. co-occurrences of words and visual objects), the evaluation set is generated using questions where these mined rules lead to incorrect answers. As for GQA-OOD, this dataset demonstrated that SOTA models do not perform well when they can't rely on shortcuts, even models which use bias-reduction techniques. Based on these benchmarks, several methods have been conceived to reduce shortcut learning in VQA (see [3 [9] for comprehensive study of the different techniques). However none of them achieve significant performance improvement on GQA-OOD [8] or VQA-CE 8
Connections with symbolic VQA- Our work is also related to neuro-symbolic reasoning
35 2
and neural module networks (NMN) 13, which generally encode a set of pre-defined functions into unique neural modules, then dynamically compose them execute question- related programs. Several improvements of standard NMNs have been proposed to make them end-to- and-to-end trainable through reinforcement learning 17 or, more recently, to enhance their scalability and generalizability [6]. However, the common point of all these works is that they are generally based on the prediction of reasoning programs whose elementary functions through program supervision. In contrast to this work, our approach uses program supervision to enrich its internal representations instead of inferring program's 's execution.
learned jointly with program prediction itself,
Mcasuring complexity of learning problems - and thus generalization, has been a goal of theo- retical machine learning since the early days, with large body of work based on PAC-Learning 32 277. Traditionally, bounds have been provided ignoring data distributions and focusing uniquely on hypothesis classes (network structures in neural network language), e.g. as measured by VC- dimension. Surprising experimental results on training networks on random samples have seemingly contradicted learning theory (37. in particular Rademacher Complexity. Recently, building on statistics of gradient descent, bounds have been proposed which take into account data distributions, notably 26) Algorithmic alignment between neural network structures and the decomposition of underlying reasoning functions has been studied in (34] with a focus on algorithms based on dynamic programming. Our theoretical contribution in section builds on the latter two methodologies and extends this type of analysis to intermediate supervision of reasoning programs.
3 Knowledge transfer and program supervision
We propose a method for transferring reasoning patterns from models learned on perfect visual inputs to models trained on noisy visual representations. In the lines of oracle transfer [2], we first pre-train a model on ground-truth visual data and then fine-tune on standard visual embeddings extracted with an object detector. The underlying hypothesis that the noise and uncertainty visual input prevents the model from learning to reason and leads to learning shortcuts.
To minimize the loss of reasoning capabilities during the transfer over the presence shift between oracle and noisy visual objects, we propose a regularization technique, which supervises the prediction of reasoning steps required to answer the question. We therefore assume the existence of the following ground truth annotation of reasoning programs. A given data sample consists of sequence {q,} of input question word embeddings, set (v.) of input visual objects, the ground truth answer class y" as well as the groundtruth reasoning program, which is structured as tree involving operations and arguments. Operations {o]] } are elements of a predefined set cchoose color, filter size, ...). The arguments of these operations may be taken from (i) all question words, (ii) all visual objects, (iii) all operations - when an operation takes as argument the result of another operation. Hence, arguments are annotated as many-to-many relationships. In the question "Is there motorbike plane?", for instance, the operation "or" depends on the result of the two operations checking the existence of specific object in the image. This is denoted as a E 1} where atj =1 means that
3

---

On
O2
On-1
Coarse: Operation (GRU)
Does the boot to the left of the look small or large?
Uni-modal
Cross-modal
ng
h1
An-1
Tokenizer
Lang- Only
Vis. to lang-
Select:
4u
flag
Program decoder
Object Detector
Vis. Only
Lang. vis.
Yela
Relate:
boot
left
1/
Choose size:
Input (params. ore frozen)
VL-Transformr
Supervision
small
Fine: Argument
large
A vision+language transformer with an attached program decoder. The decoder is fed with the VL- "L-Tranffommer's s penultimate embedding (just before the VQA classification head) and generates programs using a coarse-to to-see approach: C coarse program is generated using a GRU, consisting of a sequence of program operations embeddings {o: [ejo--(x--1) @ It is then re-fined by predicting the visual a. and textual af arguments using an affinity score between operation and input embecdings. Not shown: prediction of the operation's 's dependencies.
Figure 2:
operation i is associated with question word j as argument and, similarly, a indicating a visual argument and a. =1 an operation result argument.
We propose apply the regularization on top of the "L-Transformer architecture proposed in [0], based on sequences of self- and mooss-todlliyy attention. For this purpose, we define trainable module for program generation (prugram decoder), added output of the VL-Trrasfformer model as shown in Fig. 2 - an adaptation to other architectures would be straightforward.
Program decoder In the lines of [6], the program decoder has been designed in coarse-to-fin fashion. It first generates ① coarse sketch of the program consisting only of the operations, which then @ refined by predicting textual and visual arguments and dependencies between operations.
Coarse: operation - this module only predicts the sequence of operations [o:)aeio.n-11 using a recurrent neural network (RNN/ GRU) (7) variant, whose initial hidden state is initialized with the yCLS token embedding of the VQA transformer - the same embedding from which elassically the final answer y is predicted, cf. Figure 2 Inference is stopped when the special "STOP" operation is predicted. At each GRU time step i, a new hidden state h, is computed, from which the operation o, is classified with linear projection. It is supervised with cross-entropy loss Lap Le (o,, oū).
@ Fine: input arguments - the coarse program is then refined by predicting the operations' arguments. We first deal with textual and visual arguments only. Affinity scores at, between each operation's 's hidden h, and token q, embeddings are computed with 2-layer feed-forward network from concatenated embeddings. They represent argument set of operation o,, Similar scores at are computed for operations and visual objects. They supervised with BCE losses Lgarg E.fLBCE( (a.,; at) and Learg = EIcCBCE(OK) (a"/.- a.j))
probability the word q, to belong to
Fine: op arguments - next the dependencies are predicted, i.e. arguments which correspond to results of other operations, and which structure the program into tree. We deal with these arguments differently, and compute the set of dependency arguments for each operation o, with another GRU, whose hidden state initialized with the hidden state h; the operation. The argument index at is
a linear projection of the hidden state and supervised with BCE: Ldep = Eng LBe ( a
Program supervision - The coarse-to-fine program decoder trained with the four additional losses weighted by hyperparameters a, B, 6.
Luga + C.Lop B. + +ygagr + 8. Lwaags VQA
Program supervision
Ground truth programs- - We use ground truth information from the GQA [5] dataset, whose questions have been automatically generated from real images. Each sample contains program describing the operations and arguments required derive the answer for each question. However, the GT programs have been created for GT visual arguments (GT objects), which do not exactly

---

match the visual input of an object detector used during training and inference 12]. We therefore construct soft target, by computing intersection-on petrecttoooovvr--uuee over-union (IoU) between GT and detected objects.
Oracle transfer - Our method uses program supervision to regularize knowledge transfer from a visual oracle noisy input, introduced in (21). Oracle transfer consists in pretraining the VL.- Transformer model on ground-truth one-hot visual inputs before performing BERT-like pre-training. offers training conditions which are more favorable for learning reasoning capabilities. We perform the following steps: (1) Oracle pre-training on GT visual input on the GQA dataset; (2) (optionally) BERT-like pre-training on data from GQA unbalanced; (3) Finetuning on the final VQA-objective on the GQA dataset. Each one of these steps are regularized using program supervision.
4 Sample complexity of program supervision
We provide a theoretical analysis indicating that the prediction and supervision of reasoning pro- grams improve learnability in vision and language reasoning under some assumptions. In what follows, we denote with ! "true" (but unknown) underlying reasoning functions, and by f functions approximating them, implemented as neural networks. The goal is learn a function g able to predict a distribution over answer classes given an input question and an input image, see Fig[Dp. While in the experimental part we use state-of-the-an tranformer based models, in this theoretical analysis, we consider a simplified model, which takes as input the two vectorial embeddings and corresponding to, respectively, the question and the visual information (image), for instance generated by a language model and a convolutional neural network, and produces answers y g(q. v).
We restrict this analysis two-layer MIPs, as they easier to handle theoretically than modern attention based models. The reasoning function is approximated by neural network / parametrized by vector & and which predicts output answers y [(q, v, 0).
Our analysis uses PAC-Icarning 32 and builds on recent results providing bounds on sample complexity taking into account the data distribution itself. We here briefly reproduce Theorem 3.5. from paper (34). which, as an extension of a result [26), provides a bound for sample complexity of overparametrized MLPs with vectorial outputs, i.e. MLPs with sufficient capacity for learning given task:
Theorem 4.1 (Sample complexity for overparametrized MLPs). Let A be an overparametrized and randomly initialized two-layer MLP trained with gradient descent for a sufficient number of iterations. (i)
Suppose _dd + R. with components g(z)(+) = E, oj (8) xyp;" where 363) E IRd, ( E R,
(1) and pj = 1 or Pj
(8) = 21,1 l.l. E N4. The sample complexity Calg,e, c,0 is (9) c,) 0 max; Σ₁p; aj H11 l12 + log(?)
(0)
(c/m)²
We use the following Ansatz: since each possible input question requires potentially different form of reasoning over the visual content, our analysis is based on the following assumption.
Assumption 1. The unknown reasoning function g() is mixture model which decomposes as follows
y" = π TT, = Σ TT
(3)
where the different mixture components correspond different forms of reasoning related to different questions. The mixture components can reason on the visual input only, and the mixture weights are determined by the question q, i.e. the weights π depend on the question q, e.g. π 9n (q).
We call (.) the reasoning mode estimator. One hypothesis underlying this analysis is that learning to predict reasoning programs allows the model to more easily decompose into this form B L.e. that the network structure closely mimicks this decomposition, information on the different reasoning modes r is likely to be available in the activations of intermediate layers, of. Figure [ This will be formalized in assumption 3 and justified further below.
Considering the supposed "true" reasoning function y ggg, v) and its decomposition given in 3 suppose that each individual reasoning module gr can be approximated with multi-variat
5

---

polynomial, in particular each component h of the vector h,,
h - g-( = Σ
a 10 (9loT r. o) D)pn with parameters w - (ogi P3
3 (
(4)
trivial lower bound on the complexity of the reasoning mode estimator 9s (.) is the complexity of the identity function, which is obtained in the highly unlikely case where the question embeddings contain the 1-in-K encoding of the choice reasoning mode r. We adopt a more realistic case as the following assumption.
The input question embeddings q are separated into clusters according to reasoning modes r, such that the underlying reasoning mode estimator ga can be realized NN classifier with dot-product similarity in this embedding space.
Assumption 2.
Under this assumption, the reasoning mode estimator can be expressed generalized linear model, Le. a linear function followed by a soft-max a,
9- (q) (11 ạ, 71 ....,..),
where the different "Ir are the cluster centers of the different reasoning modes r in the question embedding space. As the softmax is monotonic non-linear function, its removal will not decrease sample complexity" [1] and the complexity can be bounded by the logits ा,, = 7 q. Plugging this into
(3) we obtain that each component y (+) of the answer is expressed as the following function:
Σ (n (1 20 (3 0)p%
We can reparametrize this function by concatenating the question q and the visual input v into single input vector æ. which are then masked by two different binary masks, which can be subsumed into the parameters and ( respectively, giving
y0 = ΣΣ (o! ap?
Extending Theorem 3.5. from (34, we can give our main theoretical result as the sample complexity of this function expressed as the following theorem.
Theorem 4.2
(Sample complexity for multi-mode reasoning functions). Let A be an over-
parametrized and randomly initialized two-layer MLP trained with gradient descent for
sufficient number of iterafions. Suppose g Ted + R with components g(x)
Σ, En
()( 2), (8!3) x)P's where "r E Rd, 3 E R..
(6) a E R. and Prs = 1 or
(8)
Prj -1,1 N. The sample complexity e,
c)( 0 inax; E, mpnj ax 11 12-rll2-1122
+ log(m/0)
(c/m)²
The proof of this theorem is given in the supplementary material (Appendix A).
Theorem 4.2 provides the sample complexity of the reasoning function g() under classical training. In the case of program supervision, our analysis is based on the following assumption (see also Fig. Пp::
Assumption 3. Supervising reasoning programs encodes the choice of reasoning modes into the hidden activations of the network f. Therefore, learning is separated into several different processes,
(a) learning of the reasoning mode estimator g-() approximated as network branch f () connected to the program output;
(b) learning of the the different reasoning modules gr() approximated as network branches fr() connected to the different answer classes yr each one of these modules is learned independently.
In principle. there should exist special degenerate cases, where an additional softmax could reduce sample complexity: however, in our case it is applied to linear function and thus generates non-linear function.

---

Oracle transf.
Prog. sup.
GQA-OOD 18
GQA 15
AUC1
Model
acc-tail
acc-head
test-dev
binary"
open
test-std
prog.
+xmeert scratch
(a) Bascline
42.9
49,5
52,4
(b) Oracle transfer
V
48 48.2.0.3
54.611.1
57.010.3
74.5
42.1
57.3
(c) Ours
V
48.8
56.1
57.8+0.2
75.4
43.0
58.2
97.1
(d) Baseline
47.5
55.2
58.5
(e) Oracle transfer
47.1
54.8
58.4
77.1
42.6
58.8
Ours
48.0
56.6
59.3
77.3
44.1
59.7
96.4
Table 1: Impact of program supervision Oracle transfer [21] for vision- -laguuge transformers. LXMERT 30] pre-training is done on the GQA unbalanced training set. We report scores on GQA [15 (test-dev and test-std) and GQA-OOD (test). binary and open scores computed the test-std; i we evaluate visual argument prediction by computing AUC@0.66 on GQA-val.
We justify Assumption a through supervision directly, which separates 9x from the rest of the reasoning process. We Justify Assumption 3 by the fact, that different reasoning modes will lead to different hidden activations of the nelwork. Later layers will therefore see different inputs for different modes r, and selector neurons can identify responsible inputs for each branch f=(). effectively switching off irrelevant input.
We that these complexities are lower than the sample complexity of the full reasoning function (5)
given in theorem 4.2, since for a given combination of i, r. the term 1| Pri dominates
the corresponding term || Recalling that the vectors y correspond to the cluster centers of reasoning modes in language embedding space. Under the assumption that the embeddings q have been created with batch normalization, a standard technique in deep learning, each value ( follows a normal distribution [00, 1). Dropping indices i, r, to ease notation, we can then compare the over the distribution of o and derive the following relationship:
expectation of the term ||
1 r r(m)
B wh 111121111111
C|IB1I v2
131
(8)
where I' is the Gamma special function and is the dimension of the language embedding y. We provide a proof for this equality in the supplementary material (Appendix A).
Discussion and validity of our claims the difference in sample complexity is determined by the factor C in equation (). which monotonically grows with the size of the embedding space m. which is typically in the hundreds. For the order of m=512 to m=7 used for state-of-the-art LXMERT models [30], complexity grows by a factor of around ~00 We would like to point out that this analysis very probably under-estimates the difference in complexity, as the difference highly depends on the complexity of the reasoning estimator π, which we have simplified as a linear function in equation ( Taking into account just the necessary soft-max alone would probably better appreciate the difference in complexity between the two methods, which we leave for future work. Our analysis also based on several assumptions, among which the simplified model (an over over-parametrizeed MLP instead of an attention based network). as well as assumptions of Theorem 4.2 from (34) and 26). on which our analysis based. Lastly, we would like to comment on the fact that we compare two different bounds: (i) the one on sample complexity for learning the full multi-modal reasoning given Theorem 4.2 and (ii) the one for learning a single reasoning mode given by Theorem 4. While comparing bounds does not provide definitive answers on the order of models, both bounds have been derived by the same algebraic manipulations and we claim that they are comparable.
Experimental results
Setup architecture: - we perform our experiments with a compact version of the Vision- Language (VL)-Tansfore
used in [30 (cf. Fig. 2). with hidden embedding size of d=128 and 4=4 heads per layer (only 26M trainable parameters). Dataset: our models are trained on the balanced GQA [15] training set (~ question-answer pairs). However, LXMERT pretraining is done on the unbalanced training set (~155M question-answa pairs). The latter contains more questions and
programs, but the same number of images (~100K images). Evaluation: is performed on GQA [15]
7

---

GQA-0OD [8] acc-tail (val.)
GQA 15
GQA-OOD [
GQA [5]
Ablations
Ablations
val.
acc-tail (val.)
val.
VQA only
46.9
62.2
(6) No prog
50.0
66.4
(2) Coarse only
46.5
62.5
(7) Uni-modal
49.9
66.5
(3) Coarse dep.
46.8
62.8
(8) Cross-modal
50.4
67.4
(4) Full w/o v.arg
47.3
63.7
Table 3: We study the impact of the program
(5) Full (ours)
49.9
66.2
supervision posistion: after uni-modal layers or after cross- cross-modal layers (standard configuration). The supervision is more efficient when used after modal interactions. Setting=oracle transfer, no Ixmert
Table 2: Ablation of different types of program supervision (compact model, LXMERT/BERT pre-training, no Oracle), on GQA val. varg superv. of visual arguments.
and GQA-GOD test sets. GQA a dataset with question-anse answer pairs automatically generated from real images, and is particularly well suited for evaluating a large variety of reasoning skills. GQA-OOD a benchmark dedicated to the out-of-domain VQA evaluation, and gives information on the generalization capabilities of the model. (for GQA) or validation (for GQA-OOD) sets. When specified (with =) we provide the average accuracy and standard deviation computed three runs with different random seeds. Visual input: following (2). we use bottom-up visual features extracted using a pre-trained object detector (we keep its parameters frozen during the training). If not specified, we use faster-RCh RCNN (25 with 36 objects per-images. In addition to that, we experiment with designed objects, and also with the VinVL (38) features which (unlike Baster-RCNN ones) conceived specifically for rision-languse tasks.
are selected either on the testdev
Program supervision improves visual reasoning - Tab. | reports the effectiveness of program prediction when combined with oracle and BERT-like pretraining on the GQA dataset and corrobo- rates the results found in the theoretical analysis. addition, when using both program supervision and LXMERT (30 but without oracle transfer, achieve an accuracy of 58.8 the tesidev set of GQA. This is lower than oracle transfer's 's accuracy, demonstrating the complementary methods. We note that the majority of the gain is achieved on the more challenging open questions. In addition, results on GQA-OOD (ace-tail and acc-head) suggest that the gains are obtained in, both, out- and in in-distribution settings. However, as already observed in (21). LXMERT pre-training tends to decrease the acc-tail gains brought by oracle transfer plus program supervision. We evaluate the program prediction performance by measuring the area under the ROC curve (AUC) the visual argument prediction with an loU treshold of =00666. Models (c) and (e) achieve, respectively, 97.1 and 96.4 AUC scores, demonstrating the effectiveness of the program decoder.
the two
Program supervision decreases sample complexity as shown in Fig. 3 We vary the amount of training data from 5% to 100% and observe that adding program super- vision allows reach an accuracy similar the baseline while using less data (no oracle transfer, no LXMERT pretraining).
GQA accuracy (46)
30
Visual arguments are the key We study the impact of different types of program supervision in Tab. [ We can see the importance of supervising arguments, (4) and
40
prog sup.
(5). The supervision of visual arguments (5) contributes most to the gain in performance, again corroborating that visual uncertainty is the main bottleneck for reasoning.
baseline
10
60
80
100
Portion of training data (%)
Program supervision enhances cross-modal interac- tions - In Tab. 3 we study how the inputs of the pro- gram prediction module influence the VQA accuracy. In particular, we test two settings: (7) uni-modat where the programs are predicted from the vision and language em- beddings right after the uni-moda layers (language and vision only in Fig. 2 and (8) cross-modal,
Figure 3: Empirical evaluation of sample
complexity of program supervision.
where the programs are predicted after the cross-modal layers (as shown in Fig. 2 We observe that,
contrary to the latter, the former does not improve the baseline ((7) vs (6) in Tab 3). This highlights the fact that the program supervision mainly impacts the operations the cross modal layers, where the most complex reasoning operations performed.
8

---

Visual features
Oracle transf.
Prog. sup.
GQA 1S
Model
test-dev
binary"
open"
test-std
(g) Oracle transfer
57.020.4
100 RCNN 2
(h) Ours
v
58.210.1
(i) Oracle transfer
59.6#0.1
(i) Ours
v
60.9
VinVL. 38
(k) Oracle transfer +lxmert
61.4
79.6
47.5
62.5
(I) Ours +ixmert
61.8
80.1
48.0
63.0
Table 4: Impact of improved visual inputs while using program supervision on Vision-Language *binary/open are computed on test-std.
Transformers. Scores on GQA [5]
Visual
Additional supervision
Training data (M)
GQA-OOD
GQA
L5]
Method
feats.
Img
Sent
acc-tail
ace-head
bin.
open
all
BAN4 22
RCNN
~ 0.1
R1
47.2
51.9
76.0
40.4
57.1
MCAN 36
RCNN
2
> 0.1
=1
46.5
53.4
75.9
42.2
58.0
Oracle transfer 21
RCNN
2
~0.18
=1
48.3
55.5
75.2
44.1
58.7
MMN 6
RCNN
Program
=00.1
11
48.0
55.5
78.9
44,9
60.8
LXMERT
RCNN
A00.18
49.8
57.7
77.8
45.0
60.3
Ours
VinVL
38
Program
200.1
115
49.1
59.7
80.1
48.0
63.0
NSM
14
SG 14
Scene graph
200.1
R/
78.9
49.3
63.2
OSCAR+VinLL 38
VinVL
38
~5.7
R99
82.3
48.8
64.7
Table Comparison with the state of the art on the GQA 15] (test-std) and GQA-OOD 18] (test)
sets. For fair comparison, we provide information about the required training data and supervision.
Program supervision allows to take advantage of the improved visual inputs - We analyse the impact of using our method with better input image representation. Increasing the number objects from 36 to 100 per image ((g) and (h) in Tab. 4) allows to further increase the gains brought by our method. On the contrary, the score of the baseline model remains unchanged, showing that the a bigger number of object proposals. Similarly, replacing the faster-RCNN features by the more recent and more accurate VinVL ones (()-(1) in Tab. 4 results in better performances.
program supervision allows to take advantage
Comparison with SOTA We report in Tab. the results obtained by approach compared to the current SOTA on the GQA and GQA-OOD datasets. In order to ensure a fair comparison, we also provide, for each method, information regarding the amount of data (images and sentences) used during training. As shown in Tab. 5 our approach compares favourably with SOTA since it obtains the second best accuracy (with 0.2 points gap) the GQA test-std among the approaches which use extra training data. The results also remain competitive when comparing to the OSCAR-viM. 38 while being trained with 50 times less images. On GQA-OOD,
approach obtains the second best acc-tail score (and the best acc-head one) with a much less complex architecture than current SOTA (26M vs 212M trainable parameters compared LXMERT (30
6 Conclusion
We have demonstrated that it is possible to improve the reasoning abilities of VQA models when providing additional supervision program annotations. In particular, our method designed to improve the transfer of reasoning patterns from models learned perfect visual input to models trained on noisy visual representations. Our experiments are supported by a theoretical analysis, demonstrating that program supervision decrease sample complexity under reasonable hypothesis. The proposed method relies on the availability of reasoning program annotations, which are costly to annotate, especially when dealing with human generated questions. Recent work has already managed to gather such kind of annotations [9. The next step will be to extend the method to configurations where the program annotation is rare or incomplete. Another promising research path could be explicitly conditioning the VQA system's 's answer by the program prediction module, following the recent work of Chen al. 6 instead of using it as catalyst for knowledge transfer.
Acknowledgment- - Wolf acknowledges support from ANR through grant "Remember" (ANR- 20-1H.

---

References
[1] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. CVPR, 2018.
[2] Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pages 1077-6086 2018.
[3] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016.
[4] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015.
[5] Hedi Ben- Younes. Rémi Cadene, Matthieu Cord, and Nicolas Thome. Mutan: Multimodal tucker fusion for visual question answering. In ICCV, 2017.
[6] Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng. William Wang, and Jingjing Liu. Meta module network for compositional visual reasoning. In WACV, 2021.
[7] Kyunghyun Cho, Bart van Merrienboer, Çaglar Güllşehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using mn encoder-decoder for statistical machine translation. In EMNLP, 2014.
[8] Corentin Dancette, Remi Cadene, Damien Teney, and Matthieu Cord. Beyond question-based biases: Assessing multimodal shortcut learning in visual question answering. arXiv preprint arXiv:2104003149 2021.
[9] Abhishek Das, Harsh Agrawal, C. Lawrence Zitnick, Devi Parikh, and Dhruv Batra. Human Attention in Visual Question Answering: Do Humans and Deep Networks Look at the Same Regions? In EMNLP, 2016.
[10] Jacob Devlin, Ming- rectional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2019.
Bing-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi-
[11] Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665 673, 2020.
[12] Yash Goyal, Tejas Khot, Douglas Summers- Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In CVPR, pages 6904-6913, 6913, 2017.
[13] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proceedings of the IEEE International Conference on Computer Vision, pages 804-813, 2017.
[14] Drew Hudson and Christopher D Manning. Learning by abstraction: The neural state machine, In Advances in Neural Information Processing Systems, pages 5901-5914, 2019.
[15] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages (10006709,15 2019.
[16] Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, pages 2901-2910, 2017.
[17] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zit- nick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the IEFE International Conference on Computer Vision, pages 2989-2998, 2017.
[18] C. Kervadee, Antipov, M. Baccouche, and C. Wolf. Roses Are Red, Violets Are Blue... but Should VQA Expect Them To? Pre-print: arxiv:2006.0512/ 2020.
[19] Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Weak supervision helps emergence of word-object alignment and improves vision-language tasks. In European Conference on Artificial Intelligence, 2019.
10

---

[20] Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should vqa expect them to? In CVPR, 2021.
[21]
Corentin Kervadec, Theo Jaunet, Grigory Antipov, Moez Baccouche, Romain Vuillemot, and Christian Wolf. How transferable reasoning patterns in vqa? In CVPR, 2021.
[22]
Jin-Hwa Kim, Jachyun Jun, and Byoung- Tak Zhang. Bilinear attention networks. In Advances in Neural Information Processing Systems, pages 1564-1574, 2018.
[23] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro-Syue -auro-yymbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision. In International Conference on Learning Representations. 2019.
[24] Hubert Ramsauer, Bernhard Schäfl, Johannes Lehner, Philipp Seidl, Michael Widrich, Lukas Gruber, Markus Holzleitner, Milena Pavlović. Geir Kjetil Sandve, Victor Greiff, Hopfield networks is all you need. arXiv preprint arXiv: 2222::20000022.7, 2020.
[25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: with region proposal networks. In Advances in neural information processing systems, pages 91-99, 2015.
Towards real-time object detection
[26] S.S. Du Arora, W. Hu, Z. Li, and Wang. Fine-grained Analysis optimization and generalization for overparametrized two-layer neural networks. In ICML, 2019.
[27] Shai S. Shalev-Shwart and S. Ben-David. Understanding Machine Learning From Theory to Algorithms. Cambridge University Press, 2014.
[28] Robik Shrestha, Kushal Kafle, Christopher Kanan. negative case analysis of visual grounding methods for vqa. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020.
[29] Robik Shrestha, Kushal Kafle, and Christopher Kanan, An investigation of critical issues in bias mitigation techniques. arXiv preprint arXiv:2104.0011 2021.
[30] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP, pages 5103-5114, 2019.
[31] Damien Teney, Ehsan A Abbasnejad, Kushal Kafle, Robik Shrestha, Christopher Kanan, and Anton van den Hengel. On the value of out-of-distribttion testing: An example of goodhart's 's law. In NewrIPS, 2020.
[32] L.G. Valiant. A theory of the learnable. In Communications of the ACM, volume 27(11), 1984.
[33] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkorcit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008. 2017.
[34] K. J. Li, M. Zhang, S.S. Du, K.-I. K., and S. Jegelka. What can Neural Networks Reason About. In ICLR, 2020.
[35] Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B Tenenbaum. Neural- Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In Advances in Neural Information Processing Systems (NIPS), 2018.
[36] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep modular co-attention networks for visual question answering. In CVPR, pages 6281-6290, 2019.
[37] C. 7hang, S. Bengio, M. Hardt, B. Recht, and O. Vinyals. Understanding deep learning requires rethinking generalization. In ICIR, 2017.
[38] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. arXiv preprint arXiv:2101.005:
11

---

Broader Impact - Beyond the exciting scientific reasons for exploring work on visual reasoning. we welcome the potentially high interest for society in VQA systems targeting increased accessibility, As an example, helping the visually impaired to query their environment based on a camera input. We are currently not aware of existing applications abusing VQA systems for unethical goals. Potential future unethical abuse could involve their use to automatic solving of Captchas (automatic Turing tests), and the creation of false dialogs and discussions in online forums ("troll farms") requiring the examination of posted text as well as accompanying images. The latter is risk inherent to any powerful system capable predicting text, but for the moment does not realistically apply to VQA, where very short answers predicted through classification over the full answer dictionary (even though our contribution is independent of the chosen VQA model).
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Sections 4 and Conclusion
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See "broader impact" Conclusion
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section
(b) Did you include complete proofs of all theoretical results? [Yes] See Section 4 and in supplementary materials.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [Yes] We do not include the code, but we provide instructions needed to reproduce our experimental results in Section 3
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section and supplementary material
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes] In Section 5 we report std and average accuracy after running experiments three times with random seeds. When the std is not given, the experiment is has been run only
time
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supp. mat.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
12


Reviews:
[{"review_id": "nStjzkTUlD", "reviewer": "Reviewer_8H1F", "summary": "This paper targets on the application problem of visual question answering, whose recent state of the art was set by attention (typically transformer) - based deep neural networks. This work propose to use program supervision to introduce extra loss terms in the training of these transformer based models, arguing that the guidance from these signals improve the model's robustness against noise and fluctuations in the visual input during transfer learning. Theoretical analysis and experimental evaluations are conducted on program supervision, leading to plausible findings and results.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper discusses a fundamental problem of how to assess and improve the \"reasoning\" capabilities of neural networks. In general, this problem is challenging to tackle as reasoning and pattern recognition are strongly entangled in most end-to-end networks with multi-modal inputs. On this note, using program supervision to guide representation learning and then perform transfer learning on more noisy inputs seems a plausible direction.\n\nMy questions challenges to this paper are listed as follows:\n\n1. It seems to me that the CLEVR dataset is specifically designed for addressing a very similar if not the same issue of neural network being dominated by perceptual bias in VQA. Therefore it is the most natural dataset to study under the context of this paper. Is there a good reason why this dataset is not used?\n\n2. This paper claims that program supervision improves the robustness against VISUAL inputs and surround all discussions and experiments around this claim. While the results seems plausible, it remains skeptical to me why this is the case. In most cases, programs are mainly depend on the question, not the visual context (in CLEVR and GQA, the program is one-to-one mapped to the question). Hence the program supervision will mostly lead to a better question parser rather than a more robust visual branch.\n\n3. Following 1 and 2, I believe it would be more natural to study transfer learning against noise and biases in the language space rather than the visual space.\n\nMy overall score to this paper is \"Marginally below threshold\". No further experiments are requested in rebuttal.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Briefly discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "_IkGliUMtAX", "reviewer": "Reviewer_Ta31", "summary": "The paper studies the problem of training VQA models on noisy/imperfect visual input (e.g., representations from existing object detectors). It proposes using program supervision as an additional task to assist in transferring reasoning patterns learned through clean/oracle visual inputs to the imperfect inputs. The experiments are performed on the GQA/GQA-OOD benchmarks, and the empirical results show gains on both frequent (head) and infrequent (tail) classes, showing improvement in reasoning skills. Furthermore, the work theoretically shows that predicting programs can reduce sample complexity.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the paper is well written for most parts. The program supervision loss is simple and sensible, but of course requires ground truth programs to be available, which may not always be realistic. The theoretical result relating program supervision to reduction in sample complexity is an important contribution, but the empirical setup could be much better.\nI will now present my concerns:\n\n[C1] The phrase: ‘transfer of reasoning patterns’ seems a bit odd for the setup considered in this paper. With such a title/description, I was expecting more focus on the reasoning patterns themselves, instead of visual quality. For instance, the title would be more appropriate for the following scenarios: applying previously learned reasoning patterns to images from new domains or questions phrased differently or to novel concepts/compositions of reasoning steps, but these are not what is being studied here.\n\n[C2] The theoretical result about program supervision reducing sample complexity in reasoning is an important contribution. However, the empirical setup leaves some open questions. Firstly, it would be nice to validate it empirically by fine-tuning on fewer samples. Secondly, does this apply to pre-training stages too? Would it apply to both visual oracle pre-training and BERT-style pre-training? It would be a plus to have those answers or at least have those discussions somewhere.\n\n[C3] Obviously, there are some concerns about the practicality of the proposed method. Firstly, having access to oracle visual annotations is not realistic, yet most of the gains seem to come from them. Second, having access to the ground truth programs is also not realistic. Both of these concerns could be addressed if the sample complexity could be validated empirically i.e., would having program supervision help even if not all instances had oracle visual inputs? Furthermore, how much benefit would it show if only a subset of the instances had program supervision? The results on head/tail do not quite cut it since they were based answer type/answer frequencies if I recall correctly, and may not provide clear analyses based on training phase/type of annotation. I understand that these may require extra experiments, but even some discussions regarding these would help.\n\n[C4] It is important to clarify early on that the program supervision refers to predicting the reasoning operations/arguments, but it is not necessary that the model actually follows that reasoning pattern (or does it?). It would be a big plus if the paper ran MMN [C2], which actually performs those reasoning operations with oracle pre-training too. For instance, if program supervision alone is sufficient and we do not actually need a separate modular architecture to run those reasoning operations, then that would be a very important finding for the community. But this is missing from the paper.\n\n[C5] Some of the details were not completely clear:\nI was confused about what ‘reasoning patterns’ meant in the context of this paper, and had to refer to [1] for clarification.\nHow much does the model trained after oracle pre-training alone achieve?\n\n[C6] Missing citation: HAN [3] could be cited since it is also graph-based and outperforms the other graph-based method mentioned in the paper: NSM [4].\n\nReferences:\n\n[1] Kervadec, Corentin, et al. \"How Transferable are Reasoning Patterns in VQA?.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n[2] Chen, Wenhu, et al. \"Meta module network for compositional visual reasoning.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2021.\n\n[3] Kim, Eun-Sol, et al. \"Hypergraph attention networks for multimodal learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.\n\n[4] Hudson, Drew A., and Christopher D. Manning. \"Learning by abstraction: The neural state machine.\" arXiv preprint arXiv:1907.03950 (2019).\n\n[5] Selvaraju, Ramprasaath R., et al. \"Taking a hint: Leveraging explanations to make vision and language models more grounded.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper acknowledges the limitations of the theoretical work i.e., analysis on MLPs instead of the models actually used in the paper. However, I think there are at least a few other limitations: a) assuming access to oracle visual inputs/programs and, b) lack of empirical studies on sample complexity for pre-training/fine-tuning tasks.", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MAwNcjCgBW_", "reviewer": "Reviewer_uBf9", "summary": "This paper describes a novel approach for doing more robust VQA that doesn’t just learn to exploit the biases in the data. They build on previous work which demonstrated that training a model on oracle visual data makes it less susceptible to learning biases in the data. While the previous work showed that the better reasoning skills learnt by the models trained on oracle visuals decline when transferred to the noisier visuals extracted by pre-trained vision models, here the authors demonstrate that adding the additional supervision step of making the model predict the reasoning program alongside the VQA answer makes it more robust to such distributional shifts.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is exceptionally well written. It provides both theoretical and empirical results, and discusses ethical implications and risks well. \n\nThe only problem is that it doesn’t achieve SOTA VQA results. Yes, the SOTA model (OSCAR [39]) requires 50 times more images, but the proposed model requires 2x more sentences and the additional program supervision, which is hard to obtain as the authors acknowledge. Saying this, it does better than the comparable baseline with program supervision trained on the same amount of data.\n\nThe authors show theoretical results suggesting that their additional program supervision results in better sample complexity by learning re-usable reasoning modules implicitly. This is probably the most interesting result of the paper, and I would have liked to see it explored more experimentally. For example, I am not sure what experimental results support the sample complexity claim. Also, could the authors maybe use the program module to analyse the failure cases? Does it bring extra interpretability to the model? \n\nThe program prediction module is entirely independent from the VQA module. Why do the authors believe that it helps regularise VQA learning? Have they considered making VQA answer module dependent on the reasoning performed in the program module?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "-MNPqmCkn6e", "reviewer": "Reviewer_NU5A", "summary": "This paper proposes a new loss terms to encourage Transformer-based models perform \"explicit reasoning\", which is beneficial for transferring knowledge from oracle-trained VQA models to deployable ones. Specifically, the novel loss is similar with the one commonly used in modular network (e.g. converting questions to executable program/functions), but instead of relying on the predicted programs to assemble executable programs during inference, in this paper this loss is only used during training as an auxiliary regularization on top of regular Transformer-based VQA models. The authors provide theoretical proof about the benefits of using the additional loss, and empirically evaluated the advantage on GQA dataset (including out-of-distribution evaluation on GQA-OOD). The proposed system demonstrates not the best but still competitive performance especially considering the model size and training efficiency (without using extra data).\n\n========================================================================================\n\nThanks for the response from the authors. I've updated my rating after reading the rebuttal as well as the comments from other reviewers.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality\n- This work is an extension following a very recent work (paper [21] in the reference) and the goal is to improve the knowledge transfer effectiveness from oracle-trained VQA systems to deployable (e.g. visual signals coming from upstream visual recognition systems), rather than directly training VQA models. This is a promising direction for building future VQA models.\n- The proposed loss term is similar to existing work from modular networks (e.g. paper [6]), but the usage is different: in previous work the predicted executable program is used for producing execution graph especially during inference, but in this work the entire module is only used as regularization during training. \n- The authors provide theoretically analysis about their novel regularization term on a simplified framework (MLP).\n\n2. Quality\n- I don't see any obvious technical flaws in this paper, and the experiments are quite extensive with proper ablations. \n- One typo in L154: I assume the loss should be $L_{dep}$ rather than $L_{varg}$ (which is already defined in L149).\n\n3. Clarity\n- This paper is well-written and easy to follow. The main idea is straightforward, and the experimental sections are clear for readers to reproduce.\n- I only quickly checked the math but not carefully verifying all the details in the proof (I'm not an expert in theory).\n\n4. Significance\n- This paper explores a novel approach (following a recent work) of building VQA systems, which aims at leveraging oracle-trained VQA systems (with perfect visual signals) and transferring the knowledge to accommodate noisy visual inputs. The high-level idea is different from the tradition (training from scratch with noisy visual signals) and the promising results may attract more future research efforts.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- I didn't find obvious mistakes from the theoretical analysis (I'm not an expert though), but it seems from the experimental validations there is no observation to directly or indirectly (with a measurable proxy) support the claim? More specifically, is there any metrics can be used to provide more straightforward evidence about the statement \"... such program prediction can lead to decreased sample complexity ...\" (from the abstract).\n\n- One probably missing ablation about the proposed loss term is to directly train a VQA system with the loss from scratch but without oracle transfer. I feel this can be a good evidence to verify whether the proposed regularization is only helpful for knowledge transfer or generally can push the Transformer models to better handle the reasoning.\n\n- Some minor issues: (1) why in Table 1 the accuracy numbers of binary, open and test-std for baseline models are missing? (2) L294 mentions an accuracy of 58.8 on testdev, when program supervision+lxmert is used but not oracle transfer. The oracle transfer+lxmert seems to be 58.4 according to the table, which is against the claim of L295 (\"this is lower than oracle transfer's accuracy...\")?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors addressed most of the reviewers concerns and all reviewers who engaged in post-rebuttal discussion recommend to accept the paper, one reviewer increasing their score.
The paper contributes an interesting approach to transfer reasoning patterns with program supervision in the context of visual question answering and a solid evaluation.

I recommend accepting the paper under the expectation that the authors will address the concerns as promised in the author response, including:
1) The empirical evaluation of the theoretical claim on sample complexity
2) improve clarity, typos
3) discuss related work Chen et al.

If possible, please also include a better failure analysis as suggested by reviewer uBf9.
Decision: Accept (Poster)