How Modular should Neural Module Networks Be for Systematic Generalization?

1 Introduction . The combinatorial nature of vision [ 1 ] is an open challenge for learning machines tackling Visual Question Answering ( VQA ) [ 2–7 ] . The amount of possible combinations of object categories , attributes , relations , context and visual tasks is exponentially large , and any training set contains a limited amount of such combinations . The ability to generalize to combinations not included in the training set , ie . the so-called systematic generalization [ 8–10 ] , is a hallmark of intelligence . Recent research has shown that Neural Module Networks ( NMNs ) are a promising approach for systematic generalization in VQA [ 5 , 11 ] . NMNs infer a program layout containing a sequence of sub-tasks , where each sub-task is associated to a module consisting of a neural network [ 12 , 13 ] . The aspects that facilitate systematic generalization in NMNs are not fully understood . Recent works demonstrate that modular approaches are helpful for object recognition tasks with novel combinations of category and viewpoints [ 14 ] and also novel combinations of visual attributes [ 15 ] . Bahdanau et al . [ 11 ] have demonstrated the importance of using an appropriate program layout for systematic generalization , and Bahdanau et al . [ 5 ] in another paper have shown that the module ’ s network architecture also impacts systematic generalization . These results highlight that using modules to tackle sub-tasks and using compositions of these modules facilitates systematic generalization . However , there is a central question at the heart of modular approaches that remains largely unaddressed : what sub-tasks should be tackled by a module ? Namely , is it best for systematic generalization to use a large degree of modularity such that modules tackle very specific sub-tasks , or is it best to use a smaller degree of modularity ? In this paper , we investigate what sub-tasks a module should perform in order to facilitate systematic generalization in VQA . We analyse different degrees of modularity as in the example of Figure 1 , where NMNs could have three different degrees of modularity : ( i ) a unique module for tackling all 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sub-tasks , ( ii ) modules tackling groups of sub-tasks , or ( iii ) many modules each one for tackling sub-tasks at the highest degree of modularity . Also , we investigate the degree of modularity at all the stages of the network , as NMNs have a structure divided in three stages : an image encoder for the extraction of visual features , a set of intermediate modules arranged based on the program layout , and a classifier that provides the output answer . We examined the degree of modularity in three families of VQA datasets : VQA-MNIST , the SQOOP dataset [ 11 ] , and the Compositional Generalization Test ( CoGenT ) of CLEVR [ 2 ] . Our results reveal two main findings that are consistent across the tested datasets : ( i ) an intermediate degree of modularity by grouping sub-tasks leads to much higher systematic generalization than for the NMNs ’ modules introduced in previous works , and ( ii ) modularity is mostly effective when is defined at the image encoder stage , which is rarely done in the literature . Furthermore , we show that these findings are directly applicable to improve the systematic generalization of state-of-the-art NMNs . 2 Libraries of Modules . NMNs consist of three components : a program generator , a library of modules and an execution engine . The program generator translates VQA questions into program layouts , consisting of compositions of sub-tasks . The library of modules contains neural networks that perform sub-tasks , and these modules are used by the execution engine to put the program layout into action . Given that the use of a correct program layout plays a fundamental role to achieve systematic generalization with NMNs [ 11 , 13 ] , we follow the approach in previous works that fix the program layout to the ground-truth program , such that it does not interfere in the analysis of the library [ 5 , 11 ] . Thus , we follow a reductionist approach , which facilitates gaining an understanding of the crucial aspects of NMNs in systematic generalization . As we gain such understanding , we will be in a position to study the interactions between these aspects in future works . Here , we aim at understanding whether the choice of a library with a specific degree of modularity facilitates systematic generalization in NMNs . Figure 1a shows examples of several libraries of modules ( one library per column ) , characterized by different degrees of modularity at different stages of the network . Each row identifies a stage in the network , while the use of single or multiple modules identifies the degree of modularity at each stage . In particular , a library with a single module shared across all sub-tasks is called all . A library with modules tackling groups of sub-tasks is called group . Finally , a library with modules that tackle each single sub-task is called sub-task . These different ways of defining the degree of modularity in the library are detailed in Figure 1b . Note that in the intermediate stage , it is sometimes necessary to use modules that operate differently depending on an input argument related to the sub-task to be performed by the module . For example , a module whose sub-task is to determine whether an object is of a specific color or not , could be implemented by a module specialized to detect colors with an input argument that indicates the color to be detected . It could also be implemented by a module specialized to detect a specific color , which would not require an additional input argument . In Figure 1c , we provide an example of usage of the libraries given the question “ Is the green object left of ‘ 2 ’ ? ” . The ground-truth program layout is a tree structure with two leaves and a root that identify respectively the pair of objects and the spatial relation . In the figure we depict how the same program layout can be implemented with libraries with different degrees of modularity . Note that the degree of modularity does not change the final number of modules in the program layout , but the number of modules in the library . In the following , we first introduce the definitions of the modules in the library for the image encoder and the classifier stages , and then we introduce existing implementations of intermediate modules and propose a new one . 2.1 Libraries of Modules at the Image Encoder and Classifier Stages . The image encoder is usually a single convolutional neural network module common to all sub-tasks ( first two columns of Figure 1a ) . We analyze image encoders defined per group of sub-tasks , as in the third and forth columns in Figure 1a . The library with maximum degree of modularity at the image encoder stage corresponds to having an image encoder per sub-task of the VQA problem , as depicted in the last column of Figure 1a . For a VQA task with binary answers , libraries with different degrees of modularity can be equally defined on the classifier . 2.2 Libraries of Modules at the Intermediate Modules Stage . The architecture of the intermediate modules varies across the literature . We consider two architectures which previous works [ 5 , 11 ] have focused on : the Find [ 16 ] and the Residual [ 13 ] modules . Let f ( k , sx , sy ) be the function that represents an intermediate module , where each sub-task is denoted by an index k that represents the input argument of the module , and ( sx , sy ) are the data inputs to the module , which can be the representation of the input image or the output of the precedent intermediate modules according to the program layout . If a module requires a single input for the given sub-task , sy is the output of the image encoder given a null input image . In the following , we show that the Find module can be interpreted as a library with a single intermediate module shared across all sub-tasks , with the neural representation modulated by an embedding related to the sub-task . Then , we show that the Residual module corresponds to a library with maximum degree of modularity , in which each sub-task corresponds a single intermediate module . Finally , we introduce a new module based on these two , which has an intermediate degree of modularity . A single intermediate module ( aka . Find module [ 16 ] ) . The definition is the following : k = Embedding ( k ) , ( 1 ) f ( k , sx , sy ) = ReLU ( W1 ⇤ ( k ReLU ( W2 ⇤ [ sx ; sy ] + b2 ) ) + b1 ) , ( 2 ) where [ sx ; sy ] is the concatenation of the two inputs and [ W1 ; b1 ; W2 ; b2 ] are the weights and bias terms of the convolutional layer . These weights and bias terms are the same across all sub-tasks , ie . all the sub-tasks are tackled with the same module . The module performs a sub-task by “ modulating ” the neural activity via the element-wise product involving k. Figure 1b depicts the Find module at the first column . One module per sub-task ( aka . Residual module [ 13 ] ) . This represents a library with a much finer degree of modularity than the previous one . For each sub-task k , we have an independent set of convolutional weights indexed by k , ie . [ W k1 ; bk1 ; W k2 ; bk2 ; W k3 ; bk3 ] , as in the following : s̃k = ReLU ( W k3 ⇤ [ sx ; sy ] + bk3 ) , ( 3 ) f ( k , sx , sy ) = ReLU ( s̃k + W k1 ⇤ ReLU ( W k2 ⇤ s̃k + bk2 ) + bk1 ) . ( 4 ) Note that here the mechanism to tackle a sub-task is to use an independent set of weights per sub-task , as depicted in Figure 1b , at the third column . One module per group of sub-tasks . We introduce a new library of modules that has a degree of modularity halfway between the two presented above ( depicted in Figure 1b , second column ) . Each modules tackles a group of sub-tasks via a Find module . We use [ W g1 ; b g 1 ; W g 2 ; b g 2 ] to denote the weights and bias terms of a module for the group of sub-tasks indexed by g. Let g = Group ( k ) be a mapping from the sub-task k to the index of the group , g. Thus , this library of modules is defined as follows : g = Group ( k ) , k = Embedding ( k ) , ( 5 ) f ( k , sx , sy ) = ReLU ( W g1 ⇤ ( k ReLU ( W g 2 ⇤ [ sx , sy ] + b g 2 ) ) + b g 1 ) . ( 6 ) Note that this library of modules uses both Find and Residual architectures to tackle a sub-task—the division of sub-tasks in groups of separated weights reflects the design of the Residual architecture , while the modulation mechanism that allows the modules to adjust their representation among sub-tasks in the same group reflects the design of the Find architecture .

Summary: This paper studies neural module networks (NMNs) for systematic generalization on VQA tasks. Of particular interest is understanding how the degree of modularity across various stages of information processing affects the ability to generalize systematically.   To that extent three stages are distinguished: (1) the encoding stage from input image to features, (2) the intermediate stage where modules compare and relate image content according to the correct program layout, (3) and the classifier stage, which processes the final representation to answer the question. In each stage, the responsible networks can be shared across sub-tasks or they can be fully modularized by associating separate networks (having unique weights) for each sub-task. In the former case, this requires conditioning the shared network during the intermediate stage via a sub-task embedding to achieve specific behavior, i.e. as done in prior work (eg. the "find module" in [16]). Alternatively, an intermediate degree of modularity can be achieved by organizing sub-tasks according to groups (such as focusing on colors or categories) with sharing weights only among the modules within a particular group and using conditioning within a group to further specialize their functionality.   To evaluate how these different choices affect systematic generalization two families of datasets are proposed: Multi-Attribute MNIST contains attribute extraction tasks (for one or more colored digits) and comparison tasks (between objects and spatial locations). Here, the learner is provided only a few combinations of visual attributes during training and is expected to generalize systematically to all possible combinations. The second family of datasets is based on SQOOP [11] that  involves comparing visual objects spatially and provides only few combinations of objects during training. Here also a modified version is considered where the amount of combinations of objects in the scene is also limited. Towards the end two variants of CLEVR are considered.  On Multi-attribute MNIST it can be observed how additional modularity improves systematic generalization (i.e. using a fully modular setting, or an intermediate degree), and how modularity in the encoding stage (1) is particularly helpful. It is also shown how tree-NMNs [11] can be improved on SQOOP by increasing modularity at the intermediate stage, and how vector-NMNs [5] can be improved on CoGenT by inducing modularity at the encoding stage, although the opposite effect is observed on COGENT.