Robust Visual Reasoning via Language Guided Neural Module Networks

1 Introduction . Visual question answering ( VQA ) [ 11 , 8 , 7 ] and visual referring expression recognition ( REF ) [ 37 , 27 ] are fundamental language-to-vision matching tasks that have several downstream applications such as robot navigation , image retrieval , and natural language interfaces [ 42 , 50 , 4 , 3 , 39 ] . The highlevel goal of these tasks is to perform joint reasoning over visual and textual queries . In the recent years , neural module networks ( NMN ) [ 10 , 20 , 33 ] attracted increasing attention due to their superior performance on these tasks [ 33 , 26 ] . Briefly , NMN models learn to parse textual queries as executable programs composed of learnable neural modules . Each of these modules implements a single step of reasoning ( e.g . count , filter , compare ) and are dynamically assembled to perform multi-step reasoning over text . In addition to the good performance , NMN also provide high model interpretability thanks to their transparent , hierarchical and semantically motivated architecture [ 45 , 5 , 6 , 32 ] . Despite great success , the current NMN implementations require a large amount of training data and are less effective in generalizing to unseen but known language constructs [ 29 , 13 , 51 ] . For example , NMN fail to understand new concepts such as “ yellow sphere to the left '' that are constructed using a combinations of known concepts from train data such as “ blue sphere '' , “ yellow cube '' , and “ metallic 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . cube to the left '' . One of the main reasons for this is that the neural modules in existing works either use a shallow , indirect language guidance [ 40 , 19 , 2 ] or pre-define the textual inputs in the module instantiation [ 26 , 33 ] , ignoring the rich correlations among the visual inputs and the relevant context from the textual inputs . For example , the neural module that filters based on the object size , “ filter_size ( smallest ) ” , needs to localize a tiny sphere or a medium-sized sphere in the image depending on the object relationships in the expression ( e.g . “ the smallest thing among the spheres '' vs. “ the metallic sphere smaller than all the large cylinders '' ) and the different sizes of spheres and cylinders available in its visual input . We believe that explicitly conditioning the neural modules on the joint textual and visual context helps in inferring robust visiolinguistic relationships which further enhances the compositional reasoning skills . In this work , we address the aforementioned issues by explicitly providing the relevant objects and relationships in the textual expression to neural modules . To do this , as shown in Figure 1 , we replace the standard convolution operations in the neural modules with a novel language-guided adaptive convolution operation , which we call LG-Conv . More specifically , the filter weights W of LG-Conv are explicitly multiplied with a spatially varying language-guided kernel G , which allows the module to adaptively co-attend over potential objects of interest from the visual input and textual input by altering the convolution . Although content-adaptive convolutions [ 24 , 15 , 44 ] are used in several vision tasks , we are not aware of any prior works that does this filter adaptation using language as guidance . We propose two novel and effective methods namely , bi-salient attentional guidance ( BiSAtt ) network and co-salient attentional guidance guidance ( CoSAtt ) network to learn the guidance kernel G from textual and visual inputs . We conduct extensive experiments on VQA and REF tasks using CLEVR [ 25 ] , CLOSURE [ 13 ] , and CLEVR-Ref+ [ 33 ] datasets . On the recently released VQA benchmark CLOSURE [ 13 ] , our approach significantly outperforms all the previous works with 11.6 % improvements in accuracy . On the REF benchmark CLEVR-Ref+ [ 33 ] , we outperform competing approaches by as much as +9.8 % accuracy on single-referent split ( S-Ref ) and +4.7 % on full-referent split ( F-Ref ) , suggesting the importance of language-guidance . Most significant gains with S-Ref , which consists of only 30 % of the training data in CLEVR-Ref+ , demonstrate the superior generalization of our model in learning from fewer training samples1 . We further evaluate the generalization capabilities of our approach by collecting a new dataset consisting of unseen compositions and contrasting samples for CLEVR-Ref+ benchmark [ 17 ] , and call our new dataset C3-Ref+ . Our key contributions are summarized as follows : 1 . We propose a novel language-guided adaptive convolution layer for NMN that guide modules in adaptively selecting informative visiolinguistic relationships and in attending to relevant objects of interest from the visual and textual inputs ; 2 . We demonstrate the superiority of our approach by achieving new state-of-the-art results on multiple tasks and benchmarks ; 3 . We introduce a new benchmark to explicitly test the model ’ s ability to generalize to adversarial perturbations and novel compositions of concepts unseen during training . We show that our model is more robust and generalizable compared to previous approaches . 2 Related Work . Multi-modal Grounding . Early approaches for tackling grounding tasks such as VQA and REF used recurrent networks with CNNs and attention-based models to jointly understand visual and language inputs [ 16 , 48 , 34 ] . [ 8 ] proposed bottom-up and top-down approach to learn attention over image regions obtained from a pretrained object detector . However , these models are shown to be heavily driven by annotation artifacts in the training data [ 1 ] . Balanced and synthetic datasets such as CLEVR [ 25 ] , CLOSURE [ 13 ] , CLEVR-Ref+ [ 33 ] are proposed by explicitly controlling the bias and language priors . Transformers [ 36 , 31 , 47 ] , using pretrain-then-transfer approach , have shown superior performance on these datasets . However , these models fail to learn robust visio-linguistic representations and are shown to exploit the imbalanced distribution in the train and test splits [ 5 , 14 ] . Neural Module Networks . Neural module networks ( NMN ) leverage specialized modules to compute basic reasoning tasks . These modules can be assembled to perform complex and compositional reasoning [ 10 , 25 , 26 , 19 ] . [ 9 ] proposed dynamic NMN that learns and adapts the structure of the execution layouts to the question . Recently , [ 26 ] proposed homogeneous ( IEP ) and generic neural modules , unlike fixed and hand-crafted neural module , in which the semantics of each neural module is learnt during training . IEP model achieves promising performance on CLEVR dataset . [ 33 ] proposed IEP-Ref by extending IEP model to CLEVR-Ref+ dataset and outperformed all the prior 1S-Ref is a subset of full CLEVR-Ref+ dataset containing expressions that refer to only a single target object in the image . works . Although , compositional by design , these models lack robust generalization abilities and fail to ground novel combinations of known linguistic constructs [ 13 ] . Few implementations of NMN such as FiLM [ 40 ] and N2NMN [ 19 ] condition the neural modules on textual guidance . However , the visiolinguistic context in these modules is rather shallow as they can not jointly co-attend over potential objects of interest directly from the visual input and textual inputs . The major difference between our work and the prior works of NMN is that we explicitly condition the neural modules on the language-guidance by directly altering and adapting the convolution operation . Adaptive Convolutions . There is an extensive literature on content-adaptive convolutions , in which standard 2D convolutions are generalized to high-dimensional convolutions [ 12 , 23 , 41 , 18 ] . [ 23 ] proposes to use bilateral filtering layers inside CNN architectures . [ 24 ] proposed dynamic filter networks in which input-specific custom filter weights are predicted by a different network branch . [ 44 ] proposed pixel-adaptive convolutions which modifies the filters in a position-specific fashion . While these prior works study adaptive convolution techniques in CNN representation tasks , it has not been explored before for guiding NMN for multi-modal language-vision tasks . Closest to our work is the FiLM [ 40 ] model that learn parameters for scaling up or down the CNN activations by conditioning on the textual input . Our model differs from FiLM primarily in using language guidance that depends on learnable , local pixel and textual features . 3 Approach . Problem Setup and Notation . Given an image I and a natural language query q as input , our goal is to develop a NMN model that selects an answer a ∈ A to the query from a fixed set A of possible answers . We generalize this notation for both VQA and REF tasks ; q , a denote question and a natural language answer respectively in VQA , whereas they represent a referring expression and a bounding box of the target object respectively in REF . We represent input image I as an ordered sequence of a set of image regions R = ( r0 , r1 , ... , rN ) and the query q as the set of words ( w1 , w2 , ... , wL ) where wi is the i-th word , N is the number of image regions extracted from input image I , and L is the total number of the tokens in the input query . Similar to [ 26 ] , we use a two-stage model for generating answer : ( 1 ) Program Generation Model p ( z|q ; θp ) : where the query is parsed to z representing the reasoning steps required to answer the query , and ( 2 ) Program Execution Model p ( a|z , I ; θe ) : where the predicted program z is used to assemble a input-specific neural network that is composed from a set of neural modules m and is executed to produce a distribution over answers . As shown in Figure 2 ( a ) , the neural modules in current implementations of NMN [ 25 , 33 ] typically use a standard Residual Convolution Block ( RCB ) , consisting of convolution layers and ReLU activations . Formally , a module ( fn ) of arity n receives n feature maps ( Fi ) of shape 128× 20× 20 and outputs a same-sized tensor fo = fn ( F1 , F2 , ... , Fn ) . hm = ReLU ( conv1 ( Fi ) ) fo = ReLU ( conv2 ( hm ) ⊕ hm ) ( 1 ) As we can see , these modules [ 25 , 26 ] are not explicitly conditioned on the input expression q , and therefore fails to extract robust visiolinguistic relationships . In contrast , as shown in Figure 2 ( b ) , we explicitly condition the neural modules on q , in addition to visual inputs , by replacing the standard convolution operations in RCB block with a novel adaptive and language-guided convolution operation , which we call LG-Conv . Also , we parameterize the module arguments , i.e. , for example , we treat “ filter_material '' module as parameterized by argument ( marg ) “ rubber '' instead of as a standalone module “ filter_material [ rubber ] '' . As a result of this parametrization , the number of a distinct set of modules used in the parameterized model reduce by 75 % . We condition our LG-Conv layer on both query q and the module argument marg ( See Figure 2 ) . In the following , we describe the LG-Conv operation and detail its formal specification .

Summary: This paper introduces a language-guided adaptive convolution layer into neural module networks (NMN) to address the limited generalizability of NMN to unseen concepts. The proposed method is able to achieve on-par performance to state-of-the-art models on CLEVR VQA task while significantly outperforms state-of-the-art models on CLEVR-Ref+ and CopsRef.  The model is also able to generalize better to novel compositions of objects and relations in CLOSURE.   In addition, a new challenging out-of-distribution test split for REF task is introduced, to evaluate on model generalizability to adversarial perturbations and unseen combinations of known concepts. The proposed method has shown a clear advantage on this new test split.