Systematic Generalization: What Is Required and Can It Be Learned?

1 INTRODUCTION . In recent years , neural network based models have become the workhorse of natural language understanding and generation . They empower industrial machine translation ( Wu et al. , 2016 ) and text generation ( Kannan et al. , 2016 ) systems and show state-of-the-art performance on numerous benchmarks including Recognizing Textual Entailment ( Gong et al. , 2017 ) , Visual Question Answering ( Jiang et al. , 2018 ) , and Reading Comprehension ( Wang et al. , 2018 ) . Despite these successes , a growing body of literature suggests that these approaches do not generalize outside of the specific distributions on which they are trained , something that is necessary for a language understanding system to be widely deployed in the real world . Investigations on the three aforementioned tasks have shown that neural models easily latch onto statistical regularities which are omnipresent in existing datasets ( Agrawal et al. , 2016 ; Gururangan et al. , 2018 ; Jia & Liang , 2017 ) and extremely hard to avoid in large scale data collection . Having learned such dataset-specific solutions , neural networks fail to make correct predictions for examples that are even slightly out of domain , yet are trivial for humans . These findings have been corroborated by a recent investigation on a synthetic instruction-following task ( Lake & Baroni , 2018 ) , in which seq2seq models ( Sutskever et al. , 2014 ; Bahdanau et al. , 2015 ) have shown little systematicity ( Fodor & Pylyshyn , 1988 ) in how they generalize , that is they do not learn general rules on how to compose words and fail spectacularly when for example asked to interpret “ jump twice ” after training on “ jump ” , “ run twice ” and “ walk twice ” . An appealing direction to improve the generalization capabilities of neural models is to add modularity and structure to their design to make them structurally resemble the kind of rules they are ∗Equal contribution supposed to learn ( Andreas et al. , 2016 ; Gaunt et al. , 2016 ) . For example , in the Neural Module Network paradigm ( NMN , Andreas et al . ( 2016 ) ) , a neural network is assembled from several neural modules , where each module is meant to perform a particular subtask of the input processing , much like a computer program composed of functions . The NMN approach is intuitively appealing but its widespread adoption has been hindered by the large amount of domain knowledge that is required to decide ( Andreas et al. , 2016 ) or predict ( Johnson et al. , 2017 ; Hu et al. , 2017 ) how the modules should be created ( parametrization ) and how they should be connected ( layout ) based on a natural language utterance . Besides , their performance has often been matched by more traditional neural models , such as FiLM ( Perez et al. , 2017 ) , Relations Networks ( Santoro et al. , 2017 ) , and MAC networks ( Hudson & Manning , 2018 ) . Lastly , generalization properties of NMNs , to the best of our knowledge , have not been rigorously studied prior to this work . Here , we investigate the impact of explicit modularity and structure on systematic generalization of NMNs and contrast their generalization abilities to those of generic models . For this case study , we focus on the task of visual question answering ( VQA ) , in particular its simplest binary form , when the answer is either “ yes ” or “ no ” . Such a binary VQA task can be seen as a fundamental task of language understanding , as it requires one to evaluate the truth value of the utterance with respect to the state of the world . Among many systematic generalization requirements that are desirable for a VQA model , we choose the following basic one : a good model should be able to reason about all possible object combinations despite being trained on a very small subset of them . We believe that this is a key prerequisite to using VQA models in the real world , because they should be robust at handling unlikely combinations of objects . We implement our generalization demands in the form of a new synthetic dataset , called Spatial Queries On Object Pairs ( SQOOP ) , in which a model has to perform spatial relational reasoning about pairs of randomly scattered letters and digits in the image ( e.g . answering the question “ Is there a letter A left of a letter B ? ” ) . The main challenge in SQOOP is that models are evaluated on all possible object pairs , but trained on only a subset of them . Our first finding is that NMNs do generalize better than other neural models when layout and parametrization are chosen appropriately . We then investigate which factors contribute to improved generalization performance and find that using a layout that matches the task ( i.e . a tree layout , as opposed to a chain layout ) , is crucial for solving the hardest version of our dataset . Lastly , and perhaps most importantly , we experiment with existing methods for making NMNs more end-to-end by inducing the module layout ( Johnson et al. , 2017 ) or learning module parametrization through soft-attention over the question ( Hu et al. , 2017 ) . Our experiments show that such end-to-end approaches often fail by not converging to tree layouts or by learning a blurred parameterization for modules , which results in poor generalization on the hardest version of our dataset . We believe that our findings challenge the intuition of researchers in the field and provide a foundation for improving systematic generalization of neural approaches to language understanding . 2 THE SQOOP DATASET FOR TESTING SYSTEMATIC GENERALIZATION . We perform all experiments of this study on the SQOOP dataset . SQOOP is a minimalistic VQA task that is designed to test the model ’ s ability to interpret unseen combinations of known relation and object words . Clearly , given known objects X , Y and a known relation R , a human can easily verify whether or not the objects X and Y are in relation R. Some instances of such queries are common in daily life ( is there a cup on the table ) , some are extremely rare ( is there a violin under the car ) , and some are unlikely but have similar , more likely counter-parts ( is there grass on the frisbee vs is there a frisbee on the grass ) . Still , a person can easily answer these questions by understanding them as just the composition of the three separate concepts . Such compositional reasoning skills are clearly required for language understanding models , and SQOOP is explicitly designed to test for them . Concretely speaking , SQOOP requires observing a 64 × 64 RGB image x and answering a yes-no question q = XRY about whether objects X and Y are in a spatial relation R. The questions are represented in a redundancy-free XRY form ; we did not aim to make the questions look like natural language . Each image contains 5 randomly chosen and randomly positioned objects . There are 36 objects : the latin letters A-Z and digits 0-9 , and there are 4 relations : LEFT OF , RIGHT OF , ABOVE , and BELOW . This results in 36 · 35 · 4 = 5040 possible unique questions ( we do not allow questions about identical objects ) . To make negative examples challenging , we ensure that both X and Y of a question are always present in the associated image and that there are distractor objects Y ′ 6= Y a : S above T ? Yes b : W left of A ? No Figure 2 : A positive ( top ) and negative ( bottom ) example from the SQOOP dataset . and X ′ 6= X such that XRY ′ and X ′RY are both true for the image . These extra precautions guarantee that answering a question requires the model to locate all possible X and Y then check if any pair of them are in the relation R. Two SQOOP examples are shown in Figure 2 . Our goal is to discover which models can correctly answer questions about all 36 ·35 possible object pairs in SQOOP after having been trained on only a subset . For this purpose we build training sets containing 36 · 4 · k unique questions by sampling k different right-hand-side ( RHS ) objects Y1 , Y2 , ... , Yk for each left-hand-side ( LHS ) object X . We use this procedure instead of just uniformly sampling object pairs in order to ensure that each object appears in at least one training question , thereby keeping the all versions of the dataset solvable . We will refer to k as the # rhs/lhs parameter of the dataset . Our test set is composed from the remaining 36 · 4 · ( 35− k ) questions . We generate training and test sets for rhs/lhs values of 1,2,4,8 and 18 , as well as a control version of the dataset , # rhs/lhs=35 , in which both the training and the test set contain all the questions ( with different images ) . Note that lower # rhs/lhs versions are harder for generalization due to the presence of spurious dependencies between the words X and Y to which the models may adapt . In order to exclude a possible compounding factor of overfitting on the training images , all our training sets contain 1 million examples , so for a dataset with # rhs/lhs = k we generate approximately 106/ ( 36 · 4·k ) different images per unique question . Appendix D contains pseudocode for SQOOP generation . 3 MODELS . A great variety of VQA models have been recently proposed in the literature , among which we can distinguish two trends . Some of the recently proposed models , such as FiLM ( Perez et al. , 2017 ) and Relation Networks ( RelNet , Santoro et al . ( 2017 ) ) are highly generic and do not require any taskspecific knowledge to be applied on a new dataset . On the opposite end of the spectrum are modular and structured models , typically flavours of Neural Module Networks ( Andreas et al. , 2016 ) , that do require some knowledge about the task at hand to be instantiated . Here , we evaluate systematic generalization of several state-of-the-art models in both families . In all models , the image x is first fed through a CNN based network , that we refer to as the stem , to produce a feature-level 3D tensor hx . This is passed through a model-specific computation conditioned on the question q , to produce a joint representation hq x. Lastly , this representation is fed into a fully-connected classifier network to produce logits for prediction . Therefore , the main difference between the models we consider is how the computation hq x = model ( hx , q ) is performed . 3.1 GENERIC MODELS . We consider four generic models in this paper : CNN+LSTM , FiLM , Relation Network ( RelNet ) , and Memory-Attention-Control ( MAC ) network . For CNN+LSTM , FiLM , and RelNet models , the question q is first encoded into a fixed-size representation hq using a unidirectional LSTM network . CNN+LSTM flattens the 3D tensor hx to a vector and concatenates it with hq to produce hq x : hq x = [ flatten ( hx ) ; hq ] . ( 1 ) RelNet ( Santoro et al. , 2017 ) uses a network g which is applied to all pairs of feature columns of hx concatenated with the question representation hq , all of which is then pooled to obtain hq x : hq x = ∑ i , j g ( hx ( i ) , hx ( j ) , hq ) ( 2 ) where hx ( i ) is the i-th feature column of hx . FiLM networks ( Perez et al. , 2017 ) use N convolutional FiLM blocks applied to hx . A FiLM block is a residual block ( He et al. , 2016 ) in which a feature-wise affine transformation ( FiLM layer ) is inserted after the 2nd convolutional layer . The FiLM layer is conditioned on the question at hand via prediction of the scaling and shifting parameters γn and βn : [ γn ; βn ] =W n q hq + b n q ( 3 ) h̃nq x = BN ( W n 2 ∗ReLU ( Wn1 ∗ hn−1q x + bn ) ) ( 4 ) hnq x = h n−1 q x +ReLU ( γn h̃nq x ⊕ βn ) ( 5 ) where BN stands for batch normalization ( Ioffe & Szegedy , 2015 ) , ∗ stands for convolution and stands for element-wise multiplications . hnq x is the output of the n-th FiLM block and h 0 q x = hx . The output of the last FiLM block hNq x undergoes an extra 1 × 1 convolution and max-pooling to produce hq x. MAC network of Hudson & Manning ( 2018 ) produces hq x by repeatedly applying a Memory-Attention-Composition ( MAC ) cell that is conditioned on the question through an attention mechanism . The MAC model is too complex to be fully described here and we refer the reader to the original paper for details .

Summary: The paper focuses on comparing the impact of explicit modularity and structure on systematic generalization by studying neural modular networks and “generic” models. The paper studies one instantiation of this systematic generalization for the setting of binary “yes” or “no” visual question answering task.  They introduce a new dataset called in which model has to answer questions that require spatial reasoning about pairs of randomly scattered letters and digits in the image. While the models are evaluated on all possible object pairs, they are trained on a smaller subset. They observe that NMNs generalize better than other neural models when an appropriate choice of layout and parametrization is made. They also show that current end-to-end approaches for inducing model layout or learning model parametrization fail to generalize better than generic models.