Robust Visual Reasoning via Language Guided Neural Module Networks

1 Introduction . Visual question answering ( VQA ) [ 11 , 8 , 7 ] and visual referring expression recognition ( REF ) [ 37 , 27 ] are fundamental language-to-vision matching tasks that have several downstream applications such as robot navigation , image retrieval , and natural language interfaces [ 42 , 50 , 4 , 3 , 39 ] . The highlevel goal of these tasks is to perform joint reasoning over visual and textual queries . In the recent years , neural module networks ( NMN ) [ 10 , 20 , 33 ] attracted increasing attention due to their superior performance on these tasks [ 33 , 26 ] . Briefly , NMN models learn to parse textual queries as executable programs composed of learnable neural modules . Each of these modules implements a single step of reasoning ( e.g . count , filter , compare ) and are dynamically assembled to perform multi-step reasoning over text . In addition to the good performance , NMN also provide high model interpretability thanks to their transparent , hierarchical and semantically motivated architecture [ 45 , 5 , 6 , 32 ] . Despite great success , the current NMN implementations require a large amount of training data and are less effective in generalizing to unseen but known language constructs [ 29 , 13 , 51 ] . For example , NMN fail to understand new concepts such as “ yellow sphere to the left '' that are constructed using a combinations of known concepts from train data such as “ blue sphere '' , “ yellow cube '' , and “ metallic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . cube to the left '' . One of the main reasons for this is that the neural modules in existing works either use a shallow , indirect language guidance [ 40 , 19 , 2 ] or pre-define the textual inputs in the module instantiation [ 26 , 33 ] , ignoring the rich correlations among the visual inputs and the relevant context from the textual inputs . For example , the neural module that filters based on the object size , “ filter_size ( smallest ) ” , needs to localize a tiny sphere or a medium-sized sphere in the image depending on the object relationships in the expression ( e.g . “ the smallest thing among the spheres '' vs. “ the metallic sphere smaller than all the large cylinders '' ) and the different sizes of spheres and cylinders available in its visual input . We believe that explicitly conditioning the neural modules on the joint textual and visual context helps in inferring robust visiolinguistic relationships which further enhances the compositional reasoning skills . In this work , we address the aforementioned issues by explicitly providing the relevant objects and relationships in the textual expression to neural modules . To do this , as shown in Figure 1 , we replace the standard convolution operations in the neural modules with a novel language-guided adaptive convolution operation , which we call LG-Conv . More specifically , the filter weights W of LG-Conv are explicitly multiplied with a spatially varying language-guided kernel G , which allows the module to adaptively co-attend over potential objects of interest from the visual input and textual input by altering the convolution . Although content-adaptive convolutions [ 24 , 15 , 44 ] are used in several vision tasks , we are not aware of any prior works that does this filter adaptation using language as guidance . We propose two novel and effective methods namely , bi-salient attentional guidance ( BiSAtt ) network and co-salient attentional guidance guidance ( CoSAtt ) network to learn the guidance kernel G from textual and visual inputs . We conduct extensive experiments on VQA and REF tasks using CLEVR [ 25 ] , CLOSURE [ 13 ] , and CLEVR-Ref+ [ 33 ] datasets . On the recently released VQA benchmark CLOSURE [ 13 ] , our approach significantly outperforms all the previous works with 11.6 % improvements in accuracy . On the REF benchmark CLEVR-Ref+ [ 33 ] , we outperform competing approaches by as much as +9.8 % accuracy on single-referent split ( S-Ref ) and +4.7 % on full-referent split ( F-Ref ) , suggesting the importance of language-guidance . Most significant gains with S-Ref , which consists of only 30 % of the training data in CLEVR-Ref+ , demonstrate the superior generalization of our model in learning from fewer training samples1 . We further evaluate the generalization capabilities of our approach by collecting a new dataset consisting of unseen compositions and contrasting samples for CLEVR-Ref+ benchmark [ 17 ] , and call our new dataset C3-Ref+ . Our key contributions are summarized as follows : 1 . We propose a novel language-guided adaptive convolution layer for NMN that guide modules in adaptively selecting informative visiolinguistic relationships and in attending to relevant objects of interest from the visual and textual inputs ; 2 . We demonstrate the superiority of our approach by achieving new state-of-the-art results on multiple tasks and benchmarks ; 3 . We introduce a new benchmark to explicitly test the model ’ s ability to generalize to adversarial perturbations and novel compositions of concepts unseen during training . We show that our model is more robust and generalizable compared to previous approaches . 2 Related Work . Multi-modal Grounding . Early approaches for tackling grounding tasks such as VQA and REF used recurrent networks with CNNs and attention-based models to jointly understand visual and language inputs [ 16 , 48 , 34 ] . [ 8 ] proposed bottom-up and top-down approach to learn attention over image regions obtained from a pretrained object detector . However , these models are shown to be heavily driven by annotation artifacts in the training data [ 1 ] . Balanced and synthetic datasets such as CLEVR [ 25 ] , CLOSURE [ 13 ] , CLEVR-Ref+ [ 33 ] are proposed by explicitly controlling the bias and language priors . Transformers [ 36 , 31 , 47 ] , using pretrain-then-transfer approach , have shown superior performance on these datasets . However , these models fail to learn robust visio-linguistic representations and are shown to exploit the imbalanced distribution in the train and test splits [ 5 , 14 ] . Neural Module Networks . Neural module networks ( NMN ) leverage specialized modules to compute basic reasoning tasks . These modules can be assembled to perform complex and compositional reasoning [ 10 , 25 , 26 , 19 ] . [ 9 ] proposed dynamic NMN that learns and adapts the structure of the execution layouts to the question . Recently , [ 26 ] proposed homogeneous ( IEP ) and generic neural modules , unlike fixed and hand-crafted neural module , in which the semantics of each neural module is learnt during training . IEP model achieves promising performance on CLEVR dataset . [ 33 ] proposed IEP-Ref by extending IEP model to CLEVR-Ref+ dataset and outperformed all the prior 1S-Ref is a subset of full CLEVR-Ref+ dataset containing expressions that refer to only a single target object in the image . works . Although , compositional by design , these models lack robust generalization abilities and fail to ground novel combinations of known linguistic constructs [ 13 ] . Few implementations of NMN such as FiLM [ 40 ] and N2NMN [ 19 ] condition the neural modules on textual guidance . However , the visiolinguistic context in these modules is rather shallow as they can not jointly co-attend over potential objects of interest directly from the visual input and textual inputs . The major difference between our work and the prior works of NMN is that we explicitly condition the neural modules on the language-guidance by directly altering and adapting the convolution operation . Adaptive Convolutions . There is an extensive literature on content-adaptive convolutions , in which standard 2D convolutions are generalized to high-dimensional convolutions [ 12 , 23 , 41 , 18 ] . [ 23 ] proposes to use bilateral filtering layers inside CNN architectures . [ 24 ] proposed dynamic filter networks in which input-specific custom filter weights are predicted by a different network branch . [ 44 ] proposed pixel-adaptive convolutions which modifies the filters in a position-specific fashion . While these prior works study adaptive convolution techniques in CNN representation tasks , it has not been explored before for guiding NMN for multi-modal language-vision tasks . Closest to our work is the FiLM [ 40 ] model that learn parameters for scaling up or down the CNN activations by conditioning on the textual input . Our model differs from FiLM primarily in using language guidance that depends on learnable , local pixel and textual features . 3 Approach . Problem Setup and Notation . Given an image I and a natural language query q as input , our goal is to develop a NMN model that selects an answer a ∈ A to the query from a fixed set A of possible answers . We generalize this notation for both VQA and REF tasks ; q , a denote question and a natural language answer respectively in VQA , whereas they represent a referring expression and a bounding box of the target object respectively in REF . We represent input image I as an ordered sequence of a set of image regions R = ( r0 , r1 , ... , rN ) and the query q as the set of words ( w1 , w2 , ... , wL ) where wi is the i-th word , N is the number of image regions extracted from input image I , and L is the total number of the tokens in the input query . Similar to [ 26 ] , we use a two-stage model for generating answer : ( 1 ) Program Generation Model p ( z|q ; θp ) : where the query is parsed to z representing the reasoning steps required to answer the query , and ( 2 ) Program Execution Model p ( a|z , I ; θe ) : where the predicted program z is used to assemble a input-specific neural network that is composed from a set of neural modules m and is executed to produce a distribution over answers . As shown in Figure 2 ( a ) , the neural modules in current implementations of NMN [ 25 , 33 ] typically use a standard Residual Convolution Block ( RCB ) , consisting of convolution layers and ReLU activations . Formally , a module ( fn ) of arity n receives n feature maps ( Fi ) of shape 128× 20× 20 and outputs a same-sized tensor fo = fn ( F1 , F2 , ... , Fn ) . hm = ReLU ( conv1 ( Fi ) ) fo = ReLU ( conv2 ( hm ) ⊕ hm ) ( 1 ) As we can see , these modules [ 25 , 26 ] are not explicitly conditioned on the input expression q , and therefore fails to extract robust visiolinguistic relationships . In contrast , as shown in Figure 2 ( b ) , we explicitly condition the neural modules on q , in addition to visual inputs , by replacing the standard convolution operations in RCB block with a novel adaptive and language-guided convolution operation , which we call LG-Conv . Also , we parameterize the module arguments , i.e. , for example , we treat “ filter_material '' module as parameterized by argument ( marg ) “ rubber '' instead of as a standalone module “ filter_material [ rubber ] '' . As a result of this parametrization , the number of a distinct set of modules used in the parameterized model reduce by 75 % . We condition our LG-Conv layer on both query q and the module argument marg ( See Figure 2 ) . In the following , we describe the LG-Conv operation and detail its formal specification .

Summary: This paper proposes a novel extension to the neural module network (NMN) to tackle vision-language tasks. The proposed method applies a language-guided convolution operator to the neural modules, which improves information exchange and fusion between the visual and language modes. The proposed framework is applied to various benchmarks in the visual question answering (VQA) and referring expression recognition (REF) domains. Experiment results demonstrate a superior performance of the proposed method.