Supervising the Transfer of Reasoning Patterns in VQA

1 Introduction . Reasoning over images is the main goal of Visual Question Anwering ( VQA ) , a task where a model is asked to answer questions over images . This problem is a test bed for the creation of agents capable of high-level reasoning , as it involves multi-modal and high-dimensional data as well as complex decision functions requiring latent representations and multiple hops . State-of-the-art models are notorious for leveraging dataset biases and shortcuts in learning rather than performing reasoning , leading to lack of generalization , as evidenced by extensive recent work on bias oriented benchmarks for vision-and-language reasoning [ 1 , 20 , 21 , 31 ] . Even large-scale semi-supervised pre-training methods , which successfully managed to increase overall VQA performance , still struggle to address questions whose answers are rare given a context [ 20 ] . It has been recently shown that reasoning patterns emerge in attention layers of a SOTA VQA model when trained on perfect ( oracle ) visual inputs , which provides evidence that deep neural networks can learn to reason , when training conditions are favorable enough [ 21 ] . In particular , uncertainty and noise in visual inputs seems to be a major cause for shortcut learning in VQA . While this kind of methods provide strong empirical results and insights on the bottlenecks in problems involving learning to reason , they still suffer from significant loss in reasoning capabilities during the transfer phase , when the model is required to adapt from perfectly clean visual input to the noisy input it will encounter after deployment . We conjecture , that reasoning on noisy data involves additional functional components not necessary in the clean case due to different types of domain shifts : ( 1 ) a presence shift , caused by imperfect object detectors , leading to missing visual objects necessary for 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . reasoning , or to multiple ( duplicate ) detections ; ( 2 ) an appearance shift causing variations in object embeddings ( descriptors ) for the same class of objects due to different appearance . In this paper , we propose a new method for transferring reasoning patterns from models learned on perfect visual input to models trained on noisy visual representations . Key to the success is a regularization term minimizing loss of the reasoning capabilities during transfer . In particular , we address this problem through program prediction as an additional auxiliary loss , i.e . supervision of the sequence of reasoning operations along with their textual and/or visual arguments . To maintain a strong link between the learned function and its objective during the knowledge transfer phase , when inputs are switched from clean oracle inputs to noisy input , the neural model is required to continue to predict complex reasoning programs from different types of inputs . As a second justification , we claim that program supervision in itself leads to a simpler learning problem , as the underlying reasoning function is decomposed into a set of tasks , each of which is easier to learn individually than the full joint decision function . We backup this claim through a theoretical analysis showing decreased sample complexity under mild hypotheses . As a summary , we present the following contributions : ( i ) we propose a new program supervision module added on top of vision-language transformer models ; ( ii ) we provide a theoretical analysis of the benefit of supervising program prediction in VQA deriving bounds on sample complexity ; ( iii ) we experimentally demonstrate the efficiency of program supervision and show that it increases VQA performance on both in- and out-of-distribution sets , even when combined with BERT-like pre-training [ 30 , 10 ] , and that it improves the quality of oracle transfer initially proposed by [ 21 ] . 2 Related Work . Transformers in VQA — VQA as a task was introduced in various datasets [ 4 , 12 , 16 ] , including GQA [ 15 ] which is automatically-generated from real-world images . This growing amount of diverse datasets has been accompanied by the development of more and more sophisticated VQA models . While their exhaustive survey is out of the scope of this paper , one can mention some foundational categories of approaches , e.g . those based on object-level attention [ 2 ] and tensor decomposition [ 5 ] . In this work , we focus on VQA models which are based on Transformers [ 33 ] , due to their wide adoption and their impressive results in several tasks ( including VQA ) . In particular , we rely on the combination of Transformers with a large-scale BERT [ 10 ] -like pretraining which was shown to be beneficial for VQA in recent works [ 19 , 30 ] . More recently , [ 21 ] focused on the study of so-called reasoning patterns in such Transformer-based VQA models . The authors analyzed how various VQA tasks are encoded in different attention heads , by applying an energy-based analysis inspired by [ 24 ] . The analysis was performed using a perfect-sighted oracle Transformer model ( which is trained with near-to-perfect visual information , and thus , is much less prone to exploit dataset biases ) in order to identify which patterns lead to better reasoning . Then , the authors showed that these reasoning patterns can be transferred from the oracle to a Transformer-based VQA model , thus improving both overall accuracy and accuracy on infrequent answers . In this work , we argue that using program prediction as an additional supervision signal is a catalyst for the transfer of these reasoning patterns . Biases and shortcut learning in VQA — In addition to the popular VQA datasets mentioned above , other benchmarks were proposed to evaluate specific reasoning capabilities of VQA systems . In particular , they address the issue of shortcut learning [ 11 ] in deep learning , where models learn decision rules which are ineffective when tested on a domain or distribution different from the training one . For instance , VQA-CP [ 1 ] explicitly inverts the answer distribution between train and test splits . To cope with recent criticisms regarding these evaluations [ 31 , 28 ] , the GQA-OOD dataset [ 18 ] introduced a new split of GQA focusing on rare ( Out-Of-Distribution / OOD ) question-answer pairs , and showed that many VQA models strongly rely on dataset biases . Following this work , the VQA-CE dataset [ 8 ] introduced a new evaluation approach related to the VQA v2 dataset . By studying the multi-modal shortcuts in the training set and mining some trivial predictive rules ( e.g . co-occurrences of words and visual objects ) , the evaluation set is generated using questions where these mined rules lead to incorrect answers . As for GQA-OOD , this dataset demonstrated that SOTA models do not perform well when they can ’ t rely on shortcuts , even models which use bias-reduction techniques . Based on these benchmarks , several methods have been conceived to reduce shortcut learning in VQA ( see [ 31 , 29 ] for a comprehensive study of the different techniques ) . However none of them achieve significant performance improvement on GQA-OOD [ 18 ] or VQA-CE [ 8 ] . Connections with symbolic VQA — Our work is also related to neuro-symbolic reasoning [ 35 , 23 ] and neural module networks ( NMN ) [ 3 , 13 ] , which generally encode a set of pre-defined functions into unique neural modules , then dynamically compose them to execute question-related programs . Several improvements of standard NMNs have been proposed to make them end-to-end trainable through reinforcement learning [ 17 ] or , more recently , to enhance their scalability and generalizability [ 6 ] . However , the common point of all these works is that they are generally based on the prediction of reasoning programs whose elementary functions are learned jointly with program prediction itself , through program supervision . In contrast to this work , our approach uses program supervision to enrich its internal representations instead of inferring program ’ s execution . Measuring complexity of learning problems — and thus generalization , has been a goal of theoretical machine learning since the early days , with a large body of work based on PAC-Learning [ 32 , 27 ] . Traditionally , bounds have been provided ignoring data distributions and focusing uniquely on hypothesis classes ( network structures in neural network language ) , e.g . as measured by VCdimension . Surprising experimental results on training networks on random samples have seemingly contradicted learning theory [ 37 ] , in particular Rademacher Complexity . Recently , building on statistics of gradient descent , bounds have been proposed which take into account data distributions , notably [ 26 ] . Algorithmic alignment between neural network structures and the decomposition of underlying reasoning functions has been studied in [ 34 ] , with a focus on algorithms based on dynamic programming . Our theoretical contribution in section 4 builds on the latter two methodologies and extends this type of analysis to intermediate supervision of reasoning programs . 3 Knowledge transfer and program supervision . We propose a method for transferring reasoning patterns from models learned on perfect visual inputs to models trained on noisy visual representations . In the lines of oracle transfer [ 21 ] , we first pre-train a model on ground-truth visual data and then fine-tune on standard visual embeddings extracted with an object detector . The underlying hypothesis is that the noise and uncertainty in visual input prevents the model from learning to reason and leads to learning shortcuts . To minimize the loss of reasoning capabilities during the transfer over the presence shift between oracle and noisy visual objects , we propose a regularization technique , which supervises the prediction of reasoning steps required to answer the question . We therefore assume the existence of the following ground truth annotation of reasoning programs . A given data sample consists of a sequence { qi } of input question word embeddings , a set { vi } of input visual objects , the ground truth answer class y∗ as well as the groundtruth reasoning program , which is structured as a tree involving operations and arguments . Operations { o∗i } are elements of a predefined set { choose color , filter size , ... } . The arguments of these operations may be taken from ( i ) all question words , ( ii ) all visual objects , ( iii ) all operations — when an operation takes as argument the result of another operation . Hence , arguments are annotated as many-to-many relationships . In the question “ Is there a motorbike or a plane ? ” , for instance , the operation “ or ” depends on the result of the two operations checking the existence of a specific object in the image . This is denoted as aq∗ij ∈ { 0 , 1 } where a q∗ ij =1 means that operation i is associated with question word j as argument and , similarly , av∗ij =1 indicating a visual argument and ad∗ij =1 an operation result argument . We propose to apply the regularization on top of the VL-Transformer architecture proposed in [ 30 ] , based on sequences of self- and cross-modality attention . For this purpose , we define a trainable module for program generation ( program decoder ) , added to the output of the VL-Transformer model as shown in Fig . 2 — an adaptation to other architectures would be straightforward . Program decoder — In the lines of [ 6 ] , the program decoder has been designed in a coarse-to-fine fashion . It first generates À a coarse sketch of the program consisting only of the operations , which are then Á refined by predicting textual and visual arguments and dependencies between operations . À Coarse : operation — this module only predicts the sequence of operations { oi } i∈ [ 0 , n−1 ] using a recurrent neural network ( RNN / GRU ) [ 7 ] variant , whose initial hidden state is initialized with the yCLS token embedding of the VQA transformer — the same embedding from which classically the final answer y is predicted , cf . Figure 2 . Inference is stopped when the special “ STOP ” operation is predicted . At each GRU time step i , a new hidden state hi is computed , from which the operation oi is classified with a linear projection . It is supervised with a cross-entropy loss Lop = ∑ iLCE ( oi , o∗i ) . Á Fine : input arguments — the coarse program is then refined by predicting the operations ’ arguments . We first deal with textual and visual arguments only . Affinity scores aqij between each operation ’ s hidden hi and token qj embeddings are computed with a 2-layer feed-forward network from concatenated embeddings . They represent the probability of the word qj to belong to the argument set of operation oi . Similar scores avij are computed for operations and visual objects . They are supervised with BCE losses Lqarg = ∑ ijLBCE ( a q ij , a q∗ ij ) and Lvarg = ∑ ijLBCE ( avij , av∗ij ) . Fine : op arguments — next the dependencies are predicted , i.e . arguments which correspond to results of other operations , and which structure the program into a tree . We deal with these arguments differently , and compute the set of dependency arguments for each operation oi with another GRU , whose hidden state is initialized with the hidden state hi of the operation . The argument index adij is a linear projection of the hidden state and supervised with BCE : Ldep = ∑ ijLBCE ( adij , ad∗ij ) . Program supervision — The coarse-to-fine program decoder is trained with the four additional losses weighted by hyperparameters α , β , γ , δ. L = Lvqa︸︷︷︸ VQA +α.Lop + β.Ldep + γ.Lqarg + δ.Lvarg︸ ︷︷ ︸ Program supervision , ( 1 ) Ground truth programs — We use ground truth information from the GQA [ 15 ] dataset , whose questions have been automatically generated from real images . Each sample contains a program describing the operations and arguments required to derive the answer for each question . However , the GT programs have been created for GT visual arguments ( GT objects ) , which do not exactly match the visual input of an object detector used during training and inference [ 2 ] . We therefore construct a soft target , by computing intersection-over-union ( IoU ) between GT and detected objects . Oracle transfer — Our method uses program supervision to regularize knowledge transfer from a visual oracle to noisy input , as introduced in [ 21 ] . Oracle transfer consists in pretraining the VLTransformer model on ground-truth one-hot visual inputs before performing BERT-like pre-training . It offers training conditions which are more favorable for learning reasoning capabilities . We perform the following steps : ( 1 ) Oracle pre-training on GT visual input on the GQA dataset ; ( 2 ) ( optionally ) BERT-like pre-training on data from GQA unbalanced ; ( 3 ) Finetuning on the final VQA-objective on the GQA dataset . Each one of these steps are regularized using program supervision .

Summary: This paper proposes a new loss terms to encourage Transformer-based models perform "explicit reasoning", which is beneficial for transferring knowledge from oracle-trained VQA models to deployable ones. Specifically, the novel loss is similar with the one commonly used in modular network (e.g. converting questions to executable program/functions), but instead of relying on the predicted programs to assemble executable programs during inference, in this paper this loss is only used during training as an auxiliary regularization on top of regular Transformer-based VQA models. The authors provide theoretical proof about the benefits of using the additional loss, and empirically evaluated the advantage on GQA dataset (including out-of-distribution evaluation on GQA-OOD). The proposed system demonstrates not the best but still competitive performance especially considering the model size and training efficiency (without using extra data).  ========================================================================================  Thanks for the response from the authors. I've updated my rating after reading the rebuttal as well as the comments from other reviewers.