How to Design Sample and Computationally Efficient VQA Models

1 INTRODUCTION . Many real-world complex tasks require both perception and reasoning ( or System I and System II intelligence ( Sutton & Barto , 2018 ) ) , such as VQA . What is the best way to integrate perception and reasoning components in a single model ? Furthermore , how would such an integration lead to accurate models , while being sample and computationally efficient ? Such questions are important to address when scaling reasoning systems to real world use cases , where empirical computation bounds must be understood in addition to the final model performance . There is a spectrum of methods in the literature exploring different ways of integrating perception and reasoning . Nowadays , the perception is typically carried out via neural models : such as CNNs for vision , and LSTMs ( Gers et al. , 1999 ) or Transformers ( Vaswani et al. , 2017 ) for language . Depending on the representation of perception input and their reasoning interface , a method can be either more towards the neural end of the spectrum or more toward the symbolic end . For the vision part , models can either use pixel-level or object-level symbolic representation . For the language part , models can generate either textual attention or programs , where the text is decomposed into a sequence of functions . Within the program representations , models typically operate on a selected discrete program or on probabilistic programs . The reasoning part used to produce the final answer can either use neural models , symbolic reasoning , or something in between , such as neural module networks ( NMN ) or soft logic blocks . Existing works for NMN methods leverage pixel-level representations and program representations such as NMN ( Hu et al. , 2017 ) , Prob-NMN ( Vedantam et al. , 2019 ) , and Stack-NMN ( Hu et al. , 2018 ) . Representative models that use object-level vision also leverage both neural and symbolic language and reasoning . Models that are more neural are LXMERT ( Tan & Bansal , 2019 ) and NSM ( Hudson & Manning , 2019 ) , while those that are more symbolic are NS-VQA ( Yi et al. , 2018 ) , NSCL ( Mao et al. , 2019 ) and NGS ( Li et al. , 2020 ) . A systematic comparison across these models is illustrated in Table 1 with more details in Appendix A . Overall , neural models have more expressive power but with more parameters , while more-symbolic models have more prior structures built into them but with fewer parameters . There is an interesting bias-variance trade-off in the model design . By encoding as much bias into the model as possible , one could reduce sample requirements . The different choices of perception and reasoning components also limit how the QA models will be trained . If both components are chosen as neural modules , then the training can be done in a very efficient end-to-end fashion . If the reasoning is carried out using more discrete operations , then the perception model needs to sample discrete outputs or take discrete inputs to interface with downstream reasoning . For instance , if symbolic reasoning is used , REINFORCE ( Williams , 1992 ) is typically used to train the perception models , which may require many samples during the optimization process . Alternatively , one can also use expensive abduction ( Li et al. , 2020 ) to manipulate the perception models outputs to provide the correct reasoning and then optimize these perception models using these pseudo-labels . Overall , more neural models will be easier to optimize , while more symbolic models will need additional expensive discrete sampling during optimization . To highlight this interesting fact , we call it the neuro-symbolic trade-off . This neuro-symbolic trade-off also affects sample efficiency and computational efficiency . To be more sample efficient , the model needs to be less neural , yet , a more neural model can be more computationally efficient during training . Thus a method that can achieve an overall good performance in terms of both sample and computation efficiency will require systematically determining which perception and reasoning components should be used and how to integrate them . To design such a model , we first test which method within each perception and reasoning component works the most efficiently . From this neuro-symbolic trade-off exploration we can design a model that uses these most efficient components and compare its overall performance against existing models . 2 PROBLEM SETTING . Before the exploration , we formally define the different choices for the vision , language , and reasoning components . In the general VQA setting we are provided with an image I , a natural language question Q , and an answer A . We now define how these basic inputs are used in each component . 2.1 REPRESENTATION FOR VISION . Given the image I there are two predominant visual representations : pixel and object-level attention . Pixel Attention . Given an image one can leverage traditional deep learning architectures used for image representations and classification such as ResNets ( He et al. , 2016 ) . Here the image is passed through many residual convolution layers before entering a MLP sub-network to perform a classification task . From one of these MLP linear layers , an intermediate dense image representation feature fI ∈ RD can be extracted , denoted by fI = ResNet ( I ) . These features are used further down the VQA pipeline , where the downstream model computes attention over the relevant part of the feature based on the question asked . Object-level . Another paradigm is to leverage object detection models such as Faster R-CNNs ( Ren et al. , 2015 ) to identify individual objects within images . Given objects in the image , one can conduct more object-level or symbolic reasoning over the image , instead of reasoning through a pixel by pixel basis . In this object-level representation , a set of object location bounding boxes ( bbox ) can be detected and labeled directly by using R-CNN asO = { ( bbox1 , label1 ) , · · · , ( bboxT , labelT ) } = R-CNN ( I ) for a preset number of T objects . Here o ∈ O can be labeled as “ small red shiny ball ” or “ large green tray ” based on what is in the image . Another approach is to factor the joint bounding box and label prediction into individual components to be handled by separate models . First the bounding boxes are extracted from the R-CNN as { bboxi } Ti=1 = R-CNN ( I ) . Then these can be passed into a separate MLP network to retrieve the labels { labeli } Ti=1 = MLP ( ResNet ( I [ bboxi ] ) ) , where I [ bbox ] is cropped image at that bounding box location . These can be used to define the final set of objects : O = { ( bboxi , labeli ) Ti=1 } . In such a setup , the benefit is that the R-CNN can be trained just as an object detector for a generic object class versus the background , whose annotations are easier to obtain . Furthermore , the number of supervised data the label MLP uses for training can be controlled separately . This is a useful mechanic during our model efficiency analysis where we work under the assumption that object bounding box is almost perfect , while object labeling is imperfect and expensive to annotate . 2.2 REPRESENTATION FOR LANGUAGE . The language representations operates on the natural text question Q . Some data sets also provide intermediate representations of each Q through a function program layout FP . FP represents the question as a sequence of abstract functionsF as FP = [ F1 , F2 , ... , Ft ] forFi ∈ F . These function programs are used jointly with the visual representations to conduct reasoning to arrive at answer A . Details about potential realizations of F are described in the following reasoning representation section 2.3 . Given the question Q and its representation FP we can define different approaches for representing the text . Text Attention . Just using the embedded text tokens E a model can embed the question Q through a recurrent network to generate a final question representation hT , where T is the maximum length sequence . Then hT can be put through an recurrent decoder to obtain a latent function at each step ct through an attentive combination of the hidden states ct = ∑ T at · ht . Symbolic Program . If we want to explicitly produce a FP for a corresponding question Q , we similarly encode the text as done for text attention . During decoding ct is passed through a MLP to predict a valid function token . Then the most likely program is selected as arg maxFP P ( FP | Q ) . Soft Program . When choosing a discrete symbolic program , the uncertainty of other function program parses is thrown out . Instead the probabilities for each program can be saved and an expected program can be computed as E [ P ( FP | Q ) ] . Intuitively all the possible programs have to be considered in this scenario which can be intractable . Instead soft program methods such as Stack-NMN factor this as E [ P ( FP | Q ) ] = E [ ∏ T P ( Ft | Q ) ] = ∏ T E [ P ( Ft | Q ) ] . This enables preserving a distribution of functions at each step instead of selecting a single one . 2.3 REPRESENTATION FOR REASONING . Given the visual and language representations , the reasoning component use these representations to derive the final answer A . Here we discuss methods that are neural , symbolic , and soft logic based . Neural Reasoning . Reasoning can be made directly with the image feature fI and encoded question hT such asA = MLP ( [ hT ; fI ] ) in a purely neural fashion . Other approaches can leverage the object representationsO . This is done by modulating the attention over whichO correspond to final answer A , conditioned on hT , as done in NSM or LCGN . LXMERT uses cross-modal attention between text embeddings E and O to predict the final answer . All these methods are more neural , but the FP can be leveraged as well to incorporate better biases through symbolic and soft programs . Symbolic Representations . From the question we can define abstract functions F to generate FP as described in the previous section . Representing F in a symbolic form enables encoding general knowledge or certain dataset ’ s domain specific language ( DSL ) into a model . This improves model interpretability and provides better inductive biases as well . Here we further describe two classes of these functions : fine grained and coarse . A fine grained representation of FP is sequence of n-ary predicates , functions with n arguments , composing F . For example , given the question Q = “ What is the shape of the thing left of the sphere ? ” , a sample fine grained program can be defined as FP = [ filter shape ( sphere , O ) , relate ( left , O ) , query shape ( O ) ] Here the visual representation ( O or fI ) and higher level argument concepts , such as sphere , are used as inputs to each function . We observe clear biases encoded into the function architecture , as given a scene graph of objects O and their relations , one could walk along this graph using FP to get the final answer A . The trade-off is that the model has to deal with more complex functions , whose input arguments and output types can vary . For example relate shape and relate return a subset of objects , while query shape returns a string . Such formulations are used by more neuro-symbolic methods such as NS-VQA and NS-CL . Coarse function types consist of simpler predicates whose arity is fixed , typically 1 , over F . Given the previous question Q , a coarse function can be defined as FP = [ filterθ ( fI ) , relateθ ( fI ) , queryθ ( fI ) ] . Here less structure is required with respect to the language and visual representation where each function can be parameterized as a NMNs . These require more parameters than DSL functions but are syntactically easier to handle as they typically just operate on a fixed dimensional image feature fI , thus implicitly encoding the function arguments . Symbolic Reasoning . Using any coarse or fine representation type for F , the symbolic reasoning can take place over the selected symbolic program FP . We define the high level execution of the symbolic reasoner to arrive at the answer by executing over FP as A = 〈FP , image representation〉S . In the fine grained and coarse samples this would look like :

Summary: The paper proposes a neuro-symbolic model for sample-efficient VQA, which turns each question into a probabilistic program which is then softly executed. The problem explored in the paper and its background and context presented clearly and it does a good job in motivating its importance and trade-offs between possible solutions. While the use of a probabilistic program to represent the questions might be too stiff / inflexible in my opinion and may not generalize well to less constrained natural language, this direction is still of course important and interesting. It also does a great job in presenting the existing approaches and comparing their properties. The writing is good and the model is presented clearly with a very useful diagram.