How Modular should Neural Module Networks Be for Systematic Generalization?

1 Introduction . The combinatorial nature of vision [ 1 ] is an open challenge for learning machines tackling Visual Question Answering ( VQA ) [ 2–7 ] . The amount of possible combinations of object categories , attributes , relations , context and visual tasks is exponentially large , and any training set contains a limited amount of such combinations . The ability to generalize to combinations not included in the training set , ie . the so-called systematic generalization [ 8–10 ] , is a hallmark of intelligence . Recent research has shown that Neural Module Networks ( NMNs ) are a promising approach for systematic generalization in VQA [ 5 , 11 ] . NMNs infer a program layout containing a sequence of sub-tasks , where each sub-task is associated to a module consisting of a neural network [ 12 , 13 ] . The aspects that facilitate systematic generalization in NMNs are not fully understood . Recent works demonstrate that modular approaches are helpful for object recognition tasks with novel combinations of category and viewpoints [ 14 ] and also novel combinations of visual attributes [ 15 ] . Bahdanau et al . [ 11 ] have demonstrated the importance of using an appropriate program layout for systematic generalization , and Bahdanau et al . [ 5 ] in another paper have shown that the module ’ s network architecture also impacts systematic generalization . These results highlight that using modules to tackle sub-tasks and using compositions of these modules facilitates systematic generalization . However , there is a central question at the heart of modular approaches that remains largely unaddressed : what sub-tasks should be tackled by a module ? Namely , is it best for systematic generalization to use a large degree of modularity such that modules tackle very specific sub-tasks , or is it best to use a smaller degree of modularity ? In this paper , we investigate what sub-tasks a module should perform in order to facilitate systematic generalization in VQA . We analyse different degrees of modularity as in the example of Figure 1 , where NMNs could have three different degrees of modularity : ( i ) a unique module for tackling all 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . sub-tasks , ( ii ) modules tackling groups of sub-tasks , or ( iii ) many modules each one for tackling sub-tasks at the highest degree of modularity . Also , we investigate the degree of modularity at all the stages of the network , as NMNs have a structure divided in three stages : an image encoder for the extraction of visual features , a set of intermediate modules arranged based on the program layout , and a classifier that provides the output answer . We examined the degree of modularity in three families of VQA datasets : VQA-MNIST , the SQOOP dataset [ 11 ] , and the Compositional Generalization Test ( CoGenT ) of CLEVR [ 2 ] . Our results reveal two main findings that are consistent across the tested datasets : ( i ) an intermediate degree of modularity by grouping sub-tasks leads to much higher systematic generalization than for the NMNs ’ modules introduced in previous works , and ( ii ) modularity is mostly effective when is defined at the image encoder stage , which is rarely done in the literature . Furthermore , we show that these findings are directly applicable to improve the systematic generalization of state-of-the-art NMNs . 2 Libraries of Modules . NMNs consist of three components : a program generator , a library of modules and an execution engine . The program generator translates VQA questions into program layouts , consisting of compositions of sub-tasks . The library of modules contains neural networks that perform sub-tasks , and these modules are used by the execution engine to put the program layout into action . Given that the use of a correct program layout plays a fundamental role to achieve systematic generalization with NMNs [ 11 , 13 ] , we follow the approach in previous works that fix the program layout to the ground-truth program , such that it does not interfere in the analysis of the library [ 5 , 11 ] . Thus , we follow a reductionist approach , which facilitates gaining an understanding of the crucial aspects of NMNs in systematic generalization . As we gain such understanding , we will be in a position to study the interactions between these aspects in future works . Here , we aim at understanding whether the choice of a library with a specific degree of modularity facilitates systematic generalization in NMNs . Figure 1a shows examples of several libraries of modules ( one library per column ) , characterized by different degrees of modularity at different stages of the network . Each row identifies a stage in the network , while the use of single or multiple modules identifies the degree of modularity at each stage . In particular , a library with a single module shared across all sub-tasks is called all . A library with modules tackling groups of sub-tasks is called group . Finally , a library with modules that tackle each single sub-task is called sub-task . These different ways of defining the degree of modularity in the library are detailed in Figure 1b . Note that in the intermediate stage , it is sometimes necessary to use modules that operate differently depending on an input argument related to the sub-task to be performed by the module . For example , a module whose sub-task is to determine whether an object is of a specific color or not , could be implemented by a module specialized to detect colors with an input argument that indicates the color to be detected . It could also be implemented by a module specialized to detect a specific color , which would not require an additional input argument . In Figure 1c , we provide an example of usage of the libraries given the question “ Is the green object left of ‘ 2 ’ ? ” . The ground-truth program layout is a tree structure with two leaves and a root that identify respectively the pair of objects and the spatial relation . In the figure we depict how the same program layout can be implemented with libraries with different degrees of modularity . Note that the degree of modularity does not change the final number of modules in the program layout , but the number of modules in the library . In the following , we first introduce the definitions of the modules in the library for the image encoder and the classifier stages , and then we introduce existing implementations of intermediate modules and propose a new one . 2.1 Libraries of Modules at the Image Encoder and Classifier Stages . The image encoder is usually a single convolutional neural network module common to all sub-tasks ( first two columns of Figure 1a ) . We analyze image encoders defined per group of sub-tasks , as in the third and forth columns in Figure 1a . The library with maximum degree of modularity at the image encoder stage corresponds to having an image encoder per sub-task of the VQA problem , as depicted in the last column of Figure 1a . For a VQA task with binary answers , libraries with different degrees of modularity can be equally defined on the classifier . 2.2 Libraries of Modules at the Intermediate Modules Stage . The architecture of the intermediate modules varies across the literature . We consider two architectures which previous works [ 5 , 11 ] have focused on : the Find [ 16 ] and the Residual [ 13 ] modules . Let f ( k , sx , sy ) be the function that represents an intermediate module , where each sub-task is denoted by an index k that represents the input argument of the module , and ( sx , sy ) are the data inputs to the module , which can be the representation of the input image or the output of the precedent intermediate modules according to the program layout . If a module requires a single input for the given sub-task , sy is the output of the image encoder given a null input image . In the following , we show that the Find module can be interpreted as a library with a single intermediate module shared across all sub-tasks , with the neural representation modulated by an embedding related to the sub-task . Then , we show that the Residual module corresponds to a library with maximum degree of modularity , in which each sub-task corresponds a single intermediate module . Finally , we introduce a new module based on these two , which has an intermediate degree of modularity . A single intermediate module ( aka . Find module [ 16 ] ) . The definition is the following : k = Embedding ( k ) , ( 1 ) f ( k , sx , sy ) = ReLU ( W1 ⇤ ( k ReLU ( W2 ⇤ [ sx ; sy ] + b2 ) ) + b1 ) , ( 2 ) where [ sx ; sy ] is the concatenation of the two inputs and [ W1 ; b1 ; W2 ; b2 ] are the weights and bias terms of the convolutional layer . These weights and bias terms are the same across all sub-tasks , ie . all the sub-tasks are tackled with the same module . The module performs a sub-task by “ modulating ” the neural activity via the element-wise product involving k. Figure 1b depicts the Find module at the first column . One module per sub-task ( aka . Residual module [ 13 ] ) . This represents a library with a much finer degree of modularity than the previous one . For each sub-task k , we have an independent set of convolutional weights indexed by k , ie . [ W k1 ; bk1 ; W k2 ; bk2 ; W k3 ; bk3 ] , as in the following : s̃k = ReLU ( W k3 ⇤ [ sx ; sy ] + bk3 ) , ( 3 ) f ( k , sx , sy ) = ReLU ( s̃k + W k1 ⇤ ReLU ( W k2 ⇤ s̃k + bk2 ) + bk1 ) . ( 4 ) Note that here the mechanism to tackle a sub-task is to use an independent set of weights per sub-task , as depicted in Figure 1b , at the third column . One module per group of sub-tasks . We introduce a new library of modules that has a degree of modularity halfway between the two presented above ( depicted in Figure 1b , second column ) . Each modules tackles a group of sub-tasks via a Find module . We use [ W g1 ; b g 1 ; W g 2 ; b g 2 ] to denote the weights and bias terms of a module for the group of sub-tasks indexed by g. Let g = Group ( k ) be a mapping from the sub-task k to the index of the group , g. Thus , this library of modules is defined as follows : g = Group ( k ) , k = Embedding ( k ) , ( 5 ) f ( k , sx , sy ) = ReLU ( W g1 ⇤ ( k ReLU ( W g 2 ⇤ [ sx , sy ] + b g 2 ) ) + b g 1 ) . ( 6 ) Note that this library of modules uses both Find and Residual architectures to tackle a sub-task—the division of sub-tasks in groups of separated weights reflects the design of the Residual architecture , while the modulation mechanism that allows the modules to adjust their representation among sub-tasks in the same group reflects the design of the Find architecture .

Summary: This work investigates how the degree and location of modularity in neural module networks (NMN) impacts out-of-domain generalization on visual question answering (VQA) tasks. The authors tested NMNs on various VQA datasets (multi-attribute MNIST, SQOOP, CLEVR). On the first two, they separated the stages of the NMN so they were either shared among all tasks, shared only among groups of sub-tasks (like colours or categories), or shared only for particular subtasks (like individual colours or numbers). Group-level modularity, particularly at the encoder layer, improved out-of-domain test-set performance. Given these results, they showed that using a modular encoder also improved systematic generalization on the CLEVR dataset over a shared-encoder baseline.