How Modular should Neural Module Networks Be for Systematic Generalization?
Authors: Vanessa D'Amario
Tomotake Sasaki
Xavier Boix
Venue: NeurIPS 2021 Poster (2021)
Keywords: Neural Module Networks
Systematic Generalization
Out of Distribution
VQA

Abstract: Neural Module Networks (NMNs) aim at Visual Question Answering (VQA) via composition of modules that tackle a sub-task. NMNs are a promising strategy to achieve systematic generalization, i.e., overcoming biasing factors in the training distribution. However, the aspects of NMNs that facilitate systematic generalization are not fully understood. In this paper, we demonstrate that the degree of modularity of the NMN have large influence on systematic generalization. In a series of experiments on three VQA datasets (VQA-MNIST, SQOOP, and CLEVR-CoGenT), our results reveal that tuning the degree of modularity, especially at the image encoder stage, reaches substantially higher systematic generalization. These findings lead to new NMN architectures that outperform previous ones in terms of systematic generalization.

How Modular Should Neural Module Networks Be for Systematic Generalization?
Vanessa D'Amario 1.3
Tomotake Sasaki 2.3
Xavier Boix 1.3
Massachusetts Institute Technology, USA 2 Fujitsu Limited, Japan Center for Brains, Minds and Machines, USA
manessad@mit.edy Mamotake.ssakee wttpaakeessaaaiffffuiua.aa @fjitssu com, xboixxmit.
Abstract
Neural Module Networks (NMNs) aim Visual Question Answering (VQA) via composition of modules that tackle sub-task. NMNs are a promising strategy achieve systematic generalization, ie. overcoming biasing factors the training distribution. However, the aspects of NMNs that facilitate systematic generaliza- tion are not fully understood. In this paper, we demonstrate that the degree of modularity of the NMN have large influence systematic generalization. In a series of experiments on three VQA datasets (VQA-MNIST. SQOOP, and CLEVR- CoGenT), our results reveal that tuning the degree of modularity, especially at the image encoder stage, reaches substantially higher systematic generalization. These findings lead new NMN architectures that outperform previous ones in terms of systematic generalization.
1
Introduction
The combinatorial nature of vision an open challenge for learning machines tackling Visual Question Answering (VQA) 7 The amount of possible combinations of object categories, attributes, relations, context and visual tasks is exponentially large, and any training set contains limited amount of such combinations. The ability generalize to combinations not included in the training set, ie. the so-called systematic generalization S SH a hallmark of intelligence.
Recent research has shown that Neural Module Networks (NMNs) are promising approach for systematic generalization VQA M. NMNs infer program layout containing sequence of
sub-tasks, where each sub-task is associated to a module consisting of a neural network [12 1k]
The aspects that facilitate systematic generalization in NMNs not fully understood. Recent works demonstrate that modular approaches are helpful for object recognition tasks with novel combinations of category and viewpoints [4] and also novel combinations of visual attributes [ Bahdanau al. have demonstrated the importance of using an appropriate program layout for systematic
generalization, and Bahdanau el al. [5] in another paper have shown that the module's 's network archi- tecture also impacts systematic generalization. These results highlight that using modules to tackle sub-tasks and using compositions these modules facilitates systematic generalization. However, there is a central question at the heart of modular approaches that remains largely unaddressed: what sub-tasks should be tackled by a module? Namely, is it best for systematic generalization use large degree of modularity such that modules tackle very specific sub-tasks, or is it best to use smaller degree modularity?
In this paper, we investigate what sub-tasks a module should perform in order to facilitate systematic generalization in VQA. We analyse different degrees of modularity as in the example of Figure [ where NMNs could have three different degrees of modularity: (i) unique module for tackling all
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

sub-tasks, (ii) modules tackling groups of sub-tasks, or (iii) many modules each one for tackling sub-tasks at the highest degree of modularity, Also, we investigate the degree of modularity at all the stages of the network, as NMNs have a structure divided in three stages: an image encoder for the extraction of visual features, set of intermediate modules arranged based on the program layout, and classifier that provides the output answer.
We examined the degree of modularity in three families of VQA datasets: VQA-MNIST, the SQOOP dataset M... and the Compositional Generalization Test (CoGenT) of CLEVR [2]. Our results reveal two main findings that are consistent across the tested datasets: (i) an intermediate degree of modularity by grouping sub-tasks leads to much higher systematic generalization than for the NMNs' modules introduced in previous works, and (ii) modularity is mostly effective when is defined the image encoder stage, which is rarely done in the literature. Furthermore, we show that these findings are directly applicable to improve the systematic generalization of state-of-the-ar NMNs.
2 Libraries of Modules
NMNs consist of three components: program generator, a library of modules and an execution engine. The program generator translates VQA questions into program layouts, consisting of compositions of sub-tasks. The library of modules contains neural networks that perform sub-tasks, and these modules are used by the execution engine to put the program layout into action.
Given that the use of a correct program layout plays a fundamental role to achieve systematic generalization with NMNs D1 13 we follow the approach in previous works that fix the program layout to the ground-trute program, such that it does not interfere in the analysis of the library M Thus, we follow a reductionist approach, which facilitates gaining an understanding of the crucial aspects of NMNs in systematic generalization. As we gain such understanding, we will be in position to study the interactions between these aspects in future works.
Here, we aim at understanding whether the choice of a library with a specific degree of modularity facilitates systematic generalization in NMNs. Figure Ta shows examples of several librarics of modules (one library per column), characterized by different degrees of modularity at different stages of the network. Each row identifies a stage in the network, while the use of single or multiple modules identifies the degree of modularity each stage. In particular, library with single module shared across all sub-tasks is called all. A library with modules tackling groups of sub-tasks is called group. Finally, a library with modules that tackle each single sub-task is called sub-task. These different ways of defining the degree of modularity in the library are detailed in Figure Db.
Note that in the intermediate stage, it is sometimes necessary to use modules that operate differently depending on an input argument related to the sub-task to be performed by the module. For example, module whose sub-task is to determine whether object is of specific color or not, could be implemented by a module specialized to detect colors with an input argument that indicates the color to be detected. It could also be implemented by a module specialized to detect specific color, which would not require an additional input argument.
In Figure Dc. we provide an example of usage of the libraries given the question "Is the green object left of '2 7" The ground-trut truth program layout tree structure with two leaves and root that identify respectively the pair of objects and the spatial relation. In the figure we depict how the same program layout can be implemented with libraries with different degrees of modularity. Note that the degree of modularity does not change the final number of modules in the program layout, but the number of modules in the library.
In the following, we first introduce the definitions of the modules in the library for the image encoder and the classifier stages, and then we introduce existing implementations of intermediate modules and propose new one.
2.1 Libraries of Modules at the Image Encoder and Classifier Stages
The image encoder is usually a single convolutional neural network module common to all sub-tasks (first two columns Figure Tb.. We analyze image encoders defined per group of sub-tasks, as the third and forth columns in Figure II. The library with maximum degree of modularity at the image
2

---

all
all
group all all
group
sub-task
|
all
group
group
sub-task
all
group
group
sub-task
IMAGE ENCODER(S) to obtain visual features
all
group 1
group 1
sub-tass 1
A INTERMEDIATE MODULE(S)
group [arg]
all [arg]
group 1 [arg]
carry sub-taakg
[arg]
sub-task
V
CLASSIFIER(S) to provide an answer
a.ll
group
all
group
sub-task
(a)
all
group
sub-task
shage [=10", >]
color
[<red, ... >]
'0'
all [<0] ... offt_of__.. ..
size (<small, >]
light condition
[<bright,
>]
small
relations [<leftoff
left_of
(b)
green '2'
2
Question: "Is green object left of '2'?"
Progrsm layout:
left_of
all
all
group
group
sub-task
all
group
all
group
sub-task
all
group
all
group
sub-task
uolor
cstegory
aolor
category
[ereen]
all
aalar [ereen]]
catertory ('8")
[green]
all
['2]]
all
oolor [green]
cateyoory ['2']
green
'g'
All
relations [left_of]
sll
relations [left_of]
left. of
[eft_or]
[een_oof]
All
relations
all
relations
left. of
(c)
Figure Libraries of modules. (a) Example five libraries with different degrees of modularity (all, group, and sub-task) analyzed in this paper. Some modules have additional input arguments to indicate specifics of the sub-task. (b) Intermediate modules from libraries with all, group, and sub-task degrees of modularity, for the VQA-MNIST dataset. (c) Composition of modules leveraging on the libraries (a), given the question "Is the green object left of 22'" and an image containing a pair of objects (the former identified by its color, the latter by its category).
3

---

encoder stage corresponds to having an image encoder per sub-task of the VQA problem, as depicted in the last column of Figure Ma.
For a VQA task with binary answers, libraries with different degrees of modularity can be equally defined on the classifier.
2.2 Libraries of Modules at the Intermediate Modules Stage
The architecture of the intermediate modules varies across the literature. We consider two architectures which previous works [5] [ have focused on: the Find [I6 and the Residual []] modules. Let [(k 8y; 8y) be the function that represents an intermediate module, where each sub-task is denoted by an index & that represents the input argument of the module, and (#n, 8y) are the data inputs to the module, which can be the representation of the input image or the output of the precedent intermediate modules according to the program layout. If module requires a single input for the given sub-task, Sy is the output of the image encoder given a null input image.
In the following, we show that the Find module can be interpreted as a library with a single intermedi- ate module shared across all sub-tasks, with the neural representation modulated by an embedding related to the sub-task. Then, we show that the Residual module corresponds to a library with maxi- mum degree of modularity, which each sub-task corresponds a single intermediate module, Finally, we introduce new module based on these two, which has an intermediate degree of modularity.
A single intermediate module (aka. Find module [6
The definition is the following:
The Embedding(k)), fk:, S,, s,y) = ReLU(W, * (7h @ ReLU(W, * [sx; syl + b2)) + b,).
(2)
where [sg; sy] the concatenation of the two inputs and [W1; by; W2; by) are the weights and bias terms of the convolutional layer. These weights and bias terms the same across all sub-tasks, ie. all the sub-tasks are tackled with the same module. The module performs a sub-task by "modulating" the neural activity via the element product involving Yk. Figure Tb depicts the Find module at the first column.
One module per sub-task (aka. Residual module [13.. This represents a library with much finer degree of modularity than the previous one. For each sub-task k. we have an independent set of convolutional weights indexed by k, ie. [Wf; bf; W by; W bg]. as in the following:
5 = ReL * [s sy] + 6g).
(3)
f(;, Sy, syy) = (5 + w * ReLU 5 bb))+ b+).
(4)
Note that here the mechanism to tackle a sub-task to use an independent set of weights per sub-task, as depicted in Figure Ip, at the third column.
One module per group of sub-tasks. We introduce a new library of modules that has a degree of modularity halfway between the two presented above (depicted in Figure Tb. second column). Each modules tackles group of sub-tasks via Find module. We use [W]:bb] by: W.y ; bū] to denote the weights and bias terms of module for the group of sub-tasks indexed by Let = Group(k) (k) be mapping from the sub-task to the index the group, g. Thus, this library of modules defined as follows:
g Gk)up(k), 7 Embeddinggk)),
(5)
f(k 82, = ReL.U(W * (7x @ ReI.U(W3 * [82, + b2) +
(6)
Note that this library of modules uses both Find and Residual architectures to tackle a sub-task the division of sub-tasks in groups of separated weights reflects the design of the Residual architecture, while the modulation mechanism that allows the modules to adjust their representation among sub-tasks in the same group reflects the design of the Find architecture
3 Datasets for Analysing Systematic Generalization in VQA
In this section, we introduce the VQA-MNIST and the SQOOP datasets to study the effect of libraries with different degrees of modularity systematic generalization.

---

VQA-MNIST
SQ0OP
Attribute extraction
Comparison
4
H
g
there & green object in the image??
Are the two objects the same color?
Is the green object left '2'?
Ds left of G?
Is Dleft G?
object '?"?
[oolor]
[oolor]
[green]
[
[D]
[G]
[J
[G]
["]
[green]
[same__clor]
[eft_oof]
[ief_of]]
[left_or]
VQA datasets to evaluate systematic generalization. From top to bottom: Visual input, question and corresponding program layout. The VQA-MNIST contains objects that are combinations of four attributes (category, color, size and illumination condition). The tasks consist on attribute extraction (from a single object and from multiple objects), and attribute comparisons (between separated object pairs and spatial locations). The training distribution has limited amount of attribute combinations. The SQOOP dataset contains objects at different positions and the task consists on comparing the position of two given objects. The training distribution has limited number of co-occurring objects the image. The original SQOOP dataset contains five objects per image [1]], and we further limit the training distribution by only allowing two objects per image.
Figure 2:
3.1 VQA-MNIST: limited combinations of visual attributes
We introduce VQA-MNIST, a family four VQA datasets of which two of them are related to attribute extraction, while the other two deal with comparisons of attributes and spatial positions. All tasks require a binary answer ('yes' or 'no'). Each object is an MNIST digit [[7] with multiple attributes:
category, color, size and illumination condition among respectively ten, five, three, and three different options. This leads to total amount of attribute instances equal to 21, and a total amount of combinations of attribute instances equal to 450. See Appendix [A. for more details.
To generate a limited set combinations of attributes, we a parameter r and we randomly extract combinations among all the 450 possibilities, with the requirement that each of the 21 attribute instances appears least r times (see Appendix A.2 for details). In order to study the effect of training data diversity, of attribute combinations and for a fair comparison, we evaluate them under the same testing conditions, as described next. We define data-run six different training datasets originated from training combinations obtained with six different r's (1, 2, 5, 8, 10, 20)), such that the set of training combinations extracted for smaller values of is included in the training combinations for larger values r. We evaluate systematic generalization for all datasets the data-run on the same images that come from novel combinations, obtained by fixing examples as described in Appendix A.3] The amount of training examples for each dataset in the data-run is fixed 210K for training, and 42K for testing systematic generalization. In this way, all NMNs in the data-run are trained with the same amount of data, ie. we evaluate different amounts of training data diversity for the same number of training images.
compare NMNs trained with datasets with different number
5. We generate the training and test
In Figure g we depict the four VQA tasks divided into attribute extraction and attribute comparison, which are summarized in the following:
Attribute extraction. Starting from the left Figure we show datasets, attribute extraction from single object and attribute extraction from multiple objects. In the former case, the image contains a single object (image size is 28 x 28 pixels), the latter, always contains pair of objects (image size is 64 64 pixels). The second object plays the role of a confounding factor. both datasets, the questions inquire about one the possible attribute instances. The ground-truth program layout corresponds to single sub-task, which is identified by the attribute instance in the question. Thus, the program layout in the intermediate module corresponds to single module identifying the sub-task (the sub-tasks in Figure 2 are identified by '7' and green).
5

---

Artribute comparison. At the center of Figure 2 we show two datasets for attribute comparison. These have ground-truth program layout that is a tree structure. The first dataset consists on attribute comparison between pairs of separated objects, in which comparisons are done between pair of objects contained separated images (of size 28 x pixels). The comparison are with respect to an attribute of the dataset. The two leaves in the program layout extract the attribute and the root tackles the comparison between them. The second dataset consists on a comparison between spatial positions, as shown in Figure 2 (image size 64 x 64 pixels). The leaves of the program layout identify the objects in the image, and the root tackles the spatial comparison (ie. above, below, left of, or right of).
3.2 SQOOP: limited amount of co-occurrence of objects in the images
The SQOOP dataset tackles the problem of spatial relations between two objects |01 Namely, the questions consist on comparing the position of two objects, e.g., Is left of G? The dataset contains four spatial relations (above, below, left of, or right of) on thirty-six possible objects. All objects can appear in the image, but specific pairs of objects are not compared during training. By limiting the possible pairs of objects compared during training, systematic generalization is measured as the ability to compare novel pairs of objects.
In the original SQOOP dataset [1]], each image contains five objects, as in the left example of the SQOOP block in Figure 2 We increase the bias of the SQOOP dataset, make more challenging for NMNs. To do so, we reduce the objects in the image to the pair in the question. We generate a dataset with the smallest amount of objects co-occurring with another object in an image, such that each object only co-occurs with another object. Given a generic training question e.g., "Is D left of G?", all the training images related to this question contain the D and objects only. This procedure substantially reduces the training data diversity of the original SQOOP datasets with five objects per image.
4 Results
In this section, we report the systematic generalization accuracy of NMNs with different libraries of modules on the VQA-MNIST and the SQOOP datasets.
4.1 VQA-MNIST
To analyze the impact of the degree of modularity of the library in systematic generalization, we report results using the following libraries, which are depicted in Figure D
all all all: This is a library with a single image encoder module, single classifier module, and intermediate module defined as the Find module. This library is commonly used in NMN [16]. while the following ones are explored for the first time in this paper.
all - group- group: This library contains a single image encoder, and intermediate modules and classifiers separated per group. We divide sub-tasks related to object categories, colors, spatial relations and so forth in different groups
group- all all: This is a library with multiple image encoders separated per group, where sub-tasks related to object categories, colors, spatial relations and forth belong to different groups. It has single intermediate module and a single classifier shared across all sub-tasks, respectively.
group- group- group: This library has one image encoder module per each group of sub-tasks, as in the image encoder of group - all all. The intermediate modules and the classifier also tackle groups of sub-task, as in the intermediate modules and classifier of all group group.
sub-task sub-task sub-task: This is the library with the maximum degree of modularity. The modules address a single sub-task at each stage of the NMN.
Other libraries of modules are analyzed in Appendix B such as other state-of-the-art NMN imple- mentations and different libraries that help to isolate the effect of the degree of modularity at the different stages. The analysis of the libraries in the Appendix serves to strengthen the evidence that support the conclusions obtained with the libraries analysed in this section.

---

group group
group
group group group
sub task
1.0
OOD test accuracy
0.8
0.8
0.7
0.7
0.6
0.5
2.4
5.1
13.45
21.55
26.85
53.65
2.55
5.25
13.45
21.2
26.55
52.9
Average amount of training combinations (%)
Average amount of training combinations (%)
(a)
(b)
1.0
1.0
OOD test accuracy
0.8
test
0.7
0.7
0.6
000.0
0.6
2.65
5.1
13.25
21.0
26.2
52.9
2.55
5.25
13.45
21.2
26.55 52.9
Average amount of training combinations (%)
Average amount training combinations (%)
(c)
(d)
Figure 3: Results in VQA-MNIST. Systematic generalization accuracy (referred OOD test accuracy in the plot) that evaluates VQA on novel combinations of visual attributes, for different amount of training combinations (%). ie. training data diversity, Results compare different libraries modules in the four types of tasks of VQA-MNIST: (a) attribute extraction from single object, (b) attribute extraction from multiple objects, (c) attribute comparison between pairs of separated objects, (d) comparison between spatial positions.
To measure systematic generalization accuracy, it needs to guaranteed that the novel combinations of attributes used for testing are not used in any stage of learning [6]. We perform a grid-search over the NMNs' paper-parameters, and evaluate the accuracy using an in-distribution validation split at the last iteration of the training. The model with highest in-distribution validation accuracy is selected, such the novel combinations of attributes are not used for hyper-parameter tuning (see Appendix B.2 for more details).
Figure shows the systematic generalization accuracy novel combinations of attributes on the four tasks in VQA-MNIST. Across all plots, each value on the horizontal-axis displays the average percentage of training combinations for fixed r (recall that is the minimum amount of times an attribute instance appears in the training combinations) across two data-runs, where the bars are grouped based on the value of r. the legend, we specify the library of modules used in the NMN. On the vertical-axis, we report mean and standard deviation of the systematic generalization accuracy across the two data-runs.
Systematic generalization compared to in-distribution generalization is much more challeng- ing. Even though NMNs achieve the highest systematic generalization in the literature 5 [T]. Figure shows that the systematic generalization accuracy across VQA tasks can be very low when the amount of training combinations is small. This highlights the difficulty to generalize to novel combinations of attributes when the training datasets biased. Recall that VQA-MNIST is synthetic dataset which could be considered one of the simplest. Yet, state-of-the approaches in systematic generalization have remarkable low accuracy in this dataset. Appendix (B.3sshww the in-distribution accuracy, which is in stark contrast with the systematic generalization accuracy, as it is close to 100% accuracy in all evaluated cases. We underline that the low systematic generalization accuracy depends on the amount of training combinations, ie. the training data diversity, which has been reported other contexts beyond VQA, such in object recognition tasks [8]
[4 15
and in navigation

---

Table 1: Results in the SQOOP dataset. Systematic generalization accuracy (%) for NMNs with tree program layout trained on SQOOP. Top row: SQOOP dataset with five objects per image, bottom row: SQOOP dataset with two objects per image. The all sub-task all library reaches higher systematic generalization in the more difficult case of two objects per image.
all all all 99.8 +: 0.2
all sub-task- all
Five objects per image
99.96 + 0.06
(as in previous work (III) Two objects per image
84 + 2
88.5 + 0.5
Libraries with an intermediate degree of modularity, especially in the image encoder stage, substantially improve systematic generalization. The group -all all and the group group group libraries achieve the highest systematic generalization accuracy across all VQA tasks. These libraries also clearly outperform the sub-task sub-task sub-task library, which has the highest degree of modularity. The library with worst systematic generalization is all all all, which has the lowest amount of modularity. Observe that the improvements of systematic generalization accuracy are large in some cases (there are gaps of about 15% accuracy between libraries of modules). Also, note that the group all all library clearly superior to all group group, which suggests that higher degree of modularity is more helpful at the image encoder stage. Taking these results together, we can conclude that a library with an intermediate degree of modularity (group), specially at the image encoder stage, is crucially important to improve systematic generalization for novel combinations of visual attributes. In Appendix B we report several control experiments that support this conclusion. Namely, in Appendix B.4 we validate the conclusions with other libraries of modules commonly used in the literature (such as all - sub-task all, and variants with batch normalization applied to the all libraries, which is when it is only possible). In Appendix B.5 B.6 and B.7 further validate the conclusion analysing variants for the image encoder, intermediate modules and image classifier, respectively.
Higher systematic generalization is not reachable with larger amounts of training examples. Since using libraries with larger number of modules may lead to modules being trained with lower number of training examples, we controlled that our conclusions are not affected by increasing the amount of training examples. We consider the task of attribute extraction from single object and attribute comparison between objects separated images, and increased the training set size of a factor ten (for total of 2.1M training examples), while keeping the same percentage of training combinations of the previous experiments in Figure 3 (a.c). Results are reported in Appendix [.8]] which show that the systematic generalization accuracy is not improved when increasing the number of training examples, and conclusions are consistent to those in Figure 3 (a,c). Thus, the amount of training examples in these experiments already achieves the highest possible systematic generalization accuracy by increasing the dataset size, ie. our conclusions are not consequence of having libraries with data-starved modules.
4.2 SQOOP
We use the two libraries of modules used in previous work [II] all all all and all sub-task all. To measure systematic generalization, we consider the model its last training iteration using the hyper-parameters in previous works (batch normalization is used in the image encoder and classifier stages).
In Table I we show the systematic generalization accuracy (mean and standard deviation across five trials). Results are consistent with the conclusions obtained in VQA-MNIST. On the original SQOOP dataset with five objects per image, both libraries achieve almost perfect systematic generalization over novel test questions. For the dataset with two objects in the image, the dataset more biased because of the lower amount of co-occurring objects, and hence, it increases the difficulty of generalizing to novel question. In this case, the all sub-task all library achieves significantly higher systematic generalization performance than the all all all library. Finally, in Appendix C we report the
performance of other VQA models that not use modules (ie. FiLM [T9] and MAC
11 20 The
results provide re-assurance that NMNs achieve higher systematic generalization.
8

---

Table 2: Results in the CLEVR-CoGens dataset. Mean and standard deviation of test accuracy (%) across five repetitions. NMNs are tested on in-distribution and out-of-distribution (systematic generalization..
Tensor-NMN with all image encoder 97.9 = 0.1 72.7 + 0.5
Vector-NMN with all image encoder 98.0 = 0.2 73.2 + 0.2
Vector- NMtN
with group image encoder (ours) 94.4 = 0.3 77.3 + 1.3
in-distribution
syst. generalization
5 Application on CLEVR-CoGenT Split for Systematic Generalization
In this section, we apply the findings in the previous section on existing NMN architectures used in practice. To do so, we use CLEVR, diagnostic dataset in VQA [2] This dataset consists of complex 3D scenes with multiple objects and ground-truth program layouts formed by compositions of sub-tasks. This dataset comes with additional splits test systematic generalization, namely the Compositional Generalization Test (CLEVR-CoGen where cubes and cylinders appear in limited amount of colors, that inverted between training and testing (see Appendix D.I for details). In this way. we can measure systematic generalization to novel combinations of visual attributes (shape and color this case).
CLEVR-CoGent is divided in two conditions
We now show that the analysis in previous sections is helpful to improve state-of of-te--o--h-hrrr art NMNs. We focus on Vector-NMN, which recently has been shown to outperform all previous NMNs [S]. Vector-NMN uses a single image encoder shared across all sub-tasks. The Vector-NMN's image encoder takes as input the features extracted from a pre-trained ResNet-101 [21]. and consists of two trainable convolutional layers shared among all sub-tasks. The details of Vector-NMN definition can be found in Appendix D.2 Our variant of Vector-NMN is based tuning the degree of modularity in the encoder stage, as we have previously shown this can substantially improve the systematic generalization capabilities. Thus, we modify the library of Vector-NMIN by using a higher degree of modularity at the image encoder stage, ie. one module for each group of sub-tasks. Concretely, we define the groups as: (i) counting tasks, (ii) tasks related to colors, (iii) materials, (iv) shapes, (v) sizes, (vi) spatial relations, (vii) logical operations, (viii) input scene (for details, see Appendix D.3).
The hhper-paramette parameters are fixed to the ones in the previous work [5]. and the ground-truth program layout is given. Tables D 3and3ishan show the mean and standard deviation accuracy (across five runs with different network initialization each) of the original Vector-NMN, a more modular (group) image encoder, and also Tensor-NMN, which is another NMN commonly used in the literature that here serves baseline. Table shows the in-distribution accuracy and the systematic generalization accuracy in objects with novel combinations of shape and color. Results shows the wide gap between in-distribution and out-of-distribution for all NMNs. Our Vector-NMN with more modular (group) image encoder achieves the best systematic generalization performance. This comes with a reduction of in-distribution generalization, which can be explained by the trade-off between in an-distriuttinn and out-of-distribution generalization reported in previous works [4] 22).
version of Vector-NMN with
Table shows the break-down of systematic generalization for each type of question. Note that the systematic generalization accuracy varies depending on questions types. The questions that mostly affected by the limited amount of object shapes and color combinations in CLEVR- CoGenT, ie. query_shaps and encoder brings improvement of the accuracy of 12% and 7%, respectively. A broader comparison of our approach with other non-modular approaches such as FiLM [19 and MAC [5] 20). and other NMN variants is reported in Appendix D.4] These results further demonstrate the higher systematic generalization accuracy of our approach.
color, is where a higher degree of modularity in the image
6 Conclusions and Discussion
Our results demonstrate that NMNs with a library of modules with an intermediate degree of modularity, especially at the image encoder stage, substantially improves systematic generalization. This finding is consistent across datasets of different complexity and NMN architectures, and casily applicable to improve state-of-the-a NMNs by using modular image encoders.

---

Table 3: Breakdown of the results in the CLEVR-CoGent dataset. Breakdown by question type.
Systematic generalization accuracy (%) reported, and it the average across five trials.
Tensor-NMN
Vector-NMIN with all image encoder
Vector-NMIN
with all image encoder
with group image encoder (ours)
count
69.7 = 0.8
70.4 = 0.4
71 =
equal_colla color
75.6 + 0.8
74 +
80 + 1
equal integer
82.7 0.3
78 2
85 = 2
equal material
74 2
74.2 = 0.7
84 2
equal_shapa
91 2
89 = 3
79 2
equal_size
75 + 1
75 + 1
88 = 2
exist
84.2 +: 0.4
84.4 + 0.4
84.4 = 0.5
greater_thar
83.8 = 0.6
83.6 = 0.4
89 = 1
than_thnn
80.7 0.9
82.0 = 0.5
87 = 2
query_
58 +
60 +
67 +
query_material
84.1 = 0.9
84.7 = 0.4
88.2 = 0.8
query_shape
37 1
40 3
52 + 3
query_size
83.5 + 0.6
84.7 + 0.7
89.5 0.5
This work also has led to new research questions. While we have shown that modularity has large impact in systematic generalization, we have tuned the degree of modularity at hand and in practical applications is desirable to do this automatically. Also, it unclear how other types of bias, such as bias in the program layout, could affect systematic generalization. To motivate follow-up work in other types of bias beyond the ones analyzed in this work, in Appendix E we report a pilot experiment that shows that bias in the program layout could degrade systematic generalization of the NMNs introduced. This suggests that there may be trade-off between the degree of modularity and bias in the program layout, which will be investigated in future works.
Finally, we are intrigued about the neural mechanisms that facilitate systematic generalization and an answer have been
how the degree of modularity affects those mechanisms. Some hints towards pointed out by [4], which shows that the emergence invariance to nuisance factors at the individual neuron level improves systematic generalization for object recognition.
Acknowledgments and Disclosure of Funding
We would like to thank Moyuru Yamada, Hisanao Akima, Pawan Sinha and Tomaso Poggio for useful discussions and insightful advice provided during this project, and Ramdas Pillai and Atsushi Kajita from Fixstars Solutions for their assistance executing the large scale computational experiments. This work has been supported by the Center for Brains, Minds and Machines (funded by NSF STC award CCF-1231216), the R01EY020517 grant from the National Eye Institute (NIH) and Fujitsu Limited (Contract No. 40008819 and 40009105),
Code and Data Availability
Code and data can be found in the following github repository: https://gt thb..bb... thub. wttp/aaneesaaamarro. htwpaa//wrwddddddtadnnnnnngn hewpiiinngrasoiinnnneo git
References
[1] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia David A Shamma, al. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-73, 2017.
[2] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In
10

---

Proceedings of the 30th IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR 2017). pages 2901-2910. 2017.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual question answering. In Proceedings of the 15th IEEE International Conference on Computer Vision (ICCV 2015), pages 2425-2433 2015.
[4] Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't t just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), pages 4971-4980, 2018
[5] Dzmitry Bahdanau, Harm de Vries, Timothy J O' D'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville, CLOSURE: Assessing systematic generalization of CLEVR models, arXiv preprint xxV1:19112.5578332, 2020.
[6] Damien Teney, Kushal Kafle, Robik Shrestha, Ehsan Abbasnejad, Christopher Kanan, and Anton van den Hengel. On the value of out-of-distribution testing: An example of Goodhart's 's law. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pages 407-417, 2020.
[7] Kushal Kafle and Christopher Kanan. Visual question answering: Datasets, algorithms, and future challenges. Computer Vision and Image Understanding, 163:3-20, 2017.
[8] Brenden Lake and Marco Baroni, Generalization without systematicity: On the compositional skills of eqquence-to-sqquencc recurrent networks. In Proceedings of the the 35th International Conference on Machine Learning (ICML 2018), pages 2873-2882, 2018.
[9] Joāo Loula, Marco Baroni, and Brenden Lake. Rearranging the familiar: Testing compositional generaliza- tion in recurrent networks. In Proceedings the 2018 EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neural Networks for NLP, pages 108-114, 2018.
[10] Jerry A Fodor and Zenon Pylyshyn. Connectionism and cognitive architecture: critical analysis. Cognition, 28(1-2):3-71, 1988.
[11] Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In Proceedings of the 7th International Conference on Learning Representations (ICLR 2019), 2019.
[12] Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. Proceedings of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2016), pages 39-48, 2016.
[13] Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zit- nick, and Ross Girshick. Inferring and executing programs for visual reasoning. In Proceedings of the 16th IEFE International Conference on Computer Vision (ICCV 2017), pages 2989-2998, 2017.
[14] Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how do CNNs generalize to -ittt---iittiittionn category-viewpoint combinations? arXiv preprint arXiv: 08032222,, 2021.
[15] Senthil Purushwalkam, Maximilian Nickel, Abhinav Gupta, and Marc' Aurelio Ranzato. Task-driven modular networks for zero-shot compositional learning. In Proceedings of the 17th IEEF/CVF International Conference on Computer Vision (ICCV 2019), pages 3593-3602, 2019.
[16] Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Sacnko. Learning reason: End-to-end module networks for visual question answering. In Proceedings of the 16th IEEE International Conference on Computer Vision (ICCV 2017), pages 804-813, 2017.
[17] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEFF, 86(1 1):2278-2324, 1998.
[18] Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. In Advances Neural Information Processing Systems 33 (NewrIPS 2020), pages 19861-19872, 2020.
[19] Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual reasoning with a general conditioning layer. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence (8),I-18), 2018
11

---

[20] Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018). 2018.
[21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the 29th IEEE Conference Computer Vision and Pattern Recognition (CVPR 2016).
pages 770-778, 2016.
[22] Syed Suleman Abbas Zaidi, Xavier Boix, Neeraj Prasad, Sharon Gilad-Gutnick, Shlomit Ben-Ami, and Pawan Sinha. robustness transformations driven by invariant neural representations? arXiv preprint arXiv:2007.001 12, 2020.
12


Reviews:
[{"review_id": "nHBicutJwm8", "reviewer": "Reviewer_w7Fj", "summary": "This work investigates how the degree and location of modularity in neural module networks (NMN) impacts out-of-domain generalization on visual question answering (VQA) tasks. The authors tested NMNs on various VQA datasets (multi-attribute MNIST, SQOOP, CLEVR). On the first two, they separated the stages of the NMN so they were either shared among all tasks, shared only among groups of sub-tasks (like colours or categories), or shared only for particular subtasks (like individual colours or numbers). Group-level modularity, particularly at the encoder layer, improved out-of-domain test-set performance. Given these results, they showed that using a modular encoder also improved systematic generalization on the CLEVR dataset over a shared-encoder baseline.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Major suggestions:\n\n- Re: Fig 2, there might be other interesting combinations of modular structure, for example, no model looked at combining all levels of modularity together in one model (something like sub-task - group - all). There would be 27 different arrangements in total (3 types of modularity for encoder, intermediate, and classifier). If these were explored, that should be mentioned. Otherwise, were there other reasons they weren’t investigated? Is there a technical difficulty I’m missing that makes some structures more difficult than others? Would be good to mention this. \n\n- Regarding the results that modularity at the encoder level is important, I would have liked some suggestions on why this is the case? Is it something specific to the structure of these tasks? One could imagine that other tasks might require a shared encoder. Could be nice to interface with multi-tasking literature here (ie https://arxiv.org/abs/1706.05098)\n\n- Finally, I wonder about the number of parameters in each of the models. For example, would the vector-NMN with 8 groups have 8 times the number of parameters in the ‘encoding’ stage? If so, it might be good to control for this, though I personally doubt that is what is responsible for the effect. \n\nMinor suggestions:\n\n- Acronyms should be defined right away (ie VQA in abstract)\n\n- Figure 2: it is not obvious at a glance that columns in a) and b) refer to corresponding models since columns are not well aligned, should make that obvious\n\n- The description of what neural module networks are was a bit unclear (needed to seek out other literature to get a better understanding)\n\n- Should name the module mentioned on line 94 (maybe the group module)\n\n- Some general structural issues, some further editing of sentence structure/grammar would help, and it was confusing that there was a Datasets section, a Results section, and then a 5th section with a new dataset/results, and table 3 title could be more clear. \n\n- there is a tendency to treat systematic generalization in a binary fashion (networks are tested in-domain or out-of-domain). Really, this should be a graded measure, and would be good to be explicit for each dataset about exactly what features are out of domain and by how much. \n\nOverall:\n\nThis work deals with an important topic that is relevant in many areas, namely, how does modularity interact with systematicity? The work provides some compelling evidence that modularity is important, especially at the encoding layer. Despite some of the issues outlined above I think the community would be interested in this result. \n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "ih4TBeLl9x", "reviewer": "Reviewer_No9k", "summary": "This paper studies neural module networks (NMNs) for systematic generalization on VQA tasks. Of particular interest is understanding how the degree of modularity across various stages of information processing affects the ability to generalize systematically. \n\nTo that extent three stages are distinguished: (1) the encoding stage from input image to features, (2) the intermediate stage where modules compare and relate image content according to the correct program layout, (3) and the classifier stage, which processes the final representation to answer the question. In each stage, the responsible networks can be shared across sub-tasks or they can be fully modularized by associating separate networks (having unique weights) for each sub-task. In the former case, this requires conditioning the shared network during the intermediate stage via a sub-task embedding to achieve specific behavior, i.e. as done in prior work (eg. the \"find module\" in [16]). Alternatively, an intermediate degree of modularity can be achieved by organizing sub-tasks according to groups (such as focusing on colors or categories) with sharing weights only among the modules within a particular group and using conditioning within a group to further specialize their functionality. \n\nTo evaluate how these different choices affect systematic generalization two families of datasets are proposed: Multi-Attribute MNIST contains attribute extraction tasks (for one or more colored digits) and comparison tasks (between objects and spatial locations). Here, the learner is provided only a few combinations of visual attributes during training and is expected to generalize systematically to all possible combinations. The second family of datasets is based on SQOOP [11] that  involves comparing visual objects spatially and provides only few combinations of objects during training. Here also a modified version is considered where the amount of combinations of objects in the scene is also limited. Towards the end two variants of CLEVR are considered.\n\nOn Multi-attribute MNIST it can be observed how additional modularity improves systematic generalization (i.e. using a fully modular setting, or an intermediate degree), and how modularity in the encoding stage (1) is particularly helpful. It is also shown how tree-NMNs [11] can be improved on SQOOP by increasing modularity at the intermediate stage, and how vector-NMNs [5] can be improved on CoGenT by inducing modularity at the encoding stage, although the opposite effect is observed on COGENT.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The aim of this paper is to answer the question of how modular neural module networks should be for systematic generalization on VQA datasets. Three different kinds of modularity are distinguished (\"all\", \"group\" and \"sub-task\") applied to three different stages of information processing (\"encoding\", \"intermediate\", \"classification\"). \n\nIt should be noted that the \"all\" configuration, using a shared network and conditioning, during the intermediate stage already makes use of the ground-truth program layout to structure information processing and thus is already quite modular (compared to say a system that directly maps images to answers). From this perspective, assessing the right level of modularity at the intermediate stage is really just a matter of balancing the amount of data that is made available to the module via weight sharing across sub-tasks with the complexity of learning a module that is adept at performing each of those sub-tasks. In the \"all\" case, all data is made available, but many different sub-tasks have to be learned. In the \"sub-task\" case only a subset of the data is made available (i.e. the '2' detector only sees data when the question contains a '2'), yet only a specific function has to be learned. Thus, it is not surprising that using \"group\" split across the ground-truth attribute types can provide a sweet spot. However, it is also clear that such a grouping (as well as the \"sub-task\" configuration) is not easily obtained in the general case. Further, the performance of NMNs when the groups are not \"optimal\" and do not reflect the ground-truth separation of attributes is not explored.\n\nMore interesting, thus, is understanding the benefits of modularity at the encoding level and classification level. From Figure 4 (comparing all-group-group and group-group-group) it can be seen that a grouped visual encoder tends to help, although other variations that could better confirm this are not explored. For example, it seems natural to me to compare all-all-all to group-all-all and to sub-task-all-all to really understand how modularity at the encoding level affects performance. The same holds true for the classification stage where, since the classification task is binary, it is not clear to me whether additional modularity will help. For example, can't the classifier just naively read out the final embedding produced by the intermediate stage to trivially answer the question? In any case, a comparison between all-all-all, all-all-group, and all-all-sub-task is not provided, which makes it difficult to say something about this. More generally, from Figure 4 there appears to be a trend towards more modular networks performing better, which is intuitive but also somewhat expected and therefore of limited significance in my view. The reason that this can be expected is because additional modularity at the properly level such as in \"sub-task\" will ensure that networks do not specialize to a particular input combination, but learn to act as stand-alone feature/relation detectors that can be composed (as is done in providing the correct program layout) to generalize systematically to novel combinations. Perhaps a surprising insight that was only briefly highlighted is that the performance of fully modular networks is still quite poor when only few combinations are provided (even when the amount of training data is increased), which I did not expect due to the amount of supervision is that provided for a configuration like sub-task-sub-task-sub-task. I was also surprised to see the overall better performance of NMNs on the relational questions compared to the attribute questions on Multi-Attribute MNIST, since I would have expected the relational questions to be more difficult. \n\nThere are some additional results provided on other datasets (SQOOP, CoGenT, and CLOSURE), but a systematic evaluation is lacking and there are additional differences that may act as confounders, which makes it difficult to draw broad conclusions. For example on SQOOP it is shown how using \"sub-task\" modularization at the intermediate stage is beneficial when combined with the tree-NMN method from [11], but \"sub-task\" modularity at the encoding stage or at the classification stage is not explored. Similarly, on CoGenT and CLOSURE the vector-NMN method from [5] is considered as well as a variation with a \"group\" modular image encoder by organizing sub-tasks according to groups. However, other kinds of modularity and at different stages, as was explored for Multi-attribute MNIST, are not considered. Here another difference is that the encoder does not act on the raw input image, but rather on features extracted from a pre-trained ResNet-101, which yields a different kind of modularity for the encoder compared to what was done previously. Nonetheless it can be observed that a modular image encoder is beneficial on this dataset. In contrast, in Table 9 in the appendix it can be seen how this same modification yields consistently worse performance on CLOSURE. There it is argued that this is due to the differences in program layouts during training and testing and future research is needed to understand this behavior in this case. \n\nIn general, I don't find that enough significant conclusions can be drawn from the current experiments presented in this paper and therefore argue for rejection. There is certainly potential, and a more systematic exploration of different degrees of modularity at different stages will help address this. It is also important that such an analysis covers multiple different data sets and uses the same set of methods, but without introducing other potential confounders, to allow for a good comparison. Finally, I encourage the authors to broaden the analysis to settings where \"perfect\" modularity is misspecified (such as by having groups that do not nicely partition along attributes), which will help understand whether these conclusions remain the same in more practical settings where that amount of supervision is not available.\n\n**Minor comments**\n\n* Regarding the data generation process for Multi-attribute MNIST it was not quite clear to me how the data is obtained. Algorithm 1 in the appendix provides pseudo-code for sampling train and test-combinations where it is ensured that test-combinations do not appear in the training combination set. However, it is not clear to me how the fraction of train/test-combinations is controlled (as reported on the x-axis in Figure 4) and how this relates to the parameter \"r\" that ensures that each attribute value is observed at least \"r\" times. It would be great if you could clarify this.\n\n* There are a number of references to tables / figures broken, which made it difficult to read (eg. line 205 should be figure 7 in the appendix) and there are many more.\n\n* The importance of incorporating modularity in the encoder is in my view intimately related to the binding problem in neural networks and the \"superposition catastrophe\" when superimposing information about multiple objects using distributed representations. A recent survey on this topic by Greff et al. (2020), see problem of segregation, may be used to draw a connection and strengthen this argument further.\n\n* The variation with batch norm explored in the appendix seems an arbitrary choice. It would be helpful to motivate why you are exploring this particular variation. \n\n* The terms \"systematic generalization\" and \"out of distribution\" are interchanged in the text without pointing out their connection.\n\n* It would be great to be consistent about the number of seeds across the paper and increase the number of seeds for Figure 4 from 2 to 5.\n\n* There are a number of naming conventions that currently hamper readability. For example, the \"Find module\" and \"Residual module\" that are not very descriptive of the broader concept. Inputs such as s_x and s_y. variations such as \"Condition A\" and \"Condition B\", etc.\n\n* Please take a moment to go through the references to correctly cite prior work that has been published (as opposed to citing the arxiv version)\n\n**References**\n\nGreff, K., van Steenkiste, S., & Schmidhuber, J. (2020). On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208.\n\n**Post-rebuttal Update**\n\nI remain with my current score of a reject (4). The reasons for this I have outlined in detail in a public reply above.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No societal impact statement is provided. Limitations are adequately discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "UCSLC96rHt", "reviewer": "Reviewer_BWzo", "summary": "This paper investigates the effect of the degree of modularity on the systematic generalization of neural module networks (NMNs). Specifically, the authors study how the degree of separation or sharing of the weights across sub-tasks at the image encoder, intermediate modules, and classifier levels affect out-of-distribution generalization on multi-attribute MNIST, SQOOP, and CLEVR-CoGenT with systematic test splits. The sub-tasks are either grouped under a single group called \"all\", or in groups of high-level functional classes, or in groups of 1. Weights within a group are shared, but the module operation depends on a sub-task embedding (this is done through straightforward architectures in the extremes of modularity). The authors explore four combinations of modularity settings across image encoder, intermediate modules, and classifiers. In experiments with multi-attribute MNITST and CLEVR-CoGenT, the authors show the benefit of separating groups in the image encoder. For SQOOP, they show an advantage of using sub-task-level intermediate modules.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an interesting paper that provides insights into desirable architectures for modules in NMNs that can systematically generalize. The paper presents new studies of the effect of the degree of modularity in neural module networks. The authors also propose adding modularity in the image encoder as a novel contribution, which significantly impacts the generalization for the chosen datasets. They also show a benefit of an intermediate level of modularity for the intermediate modules.\n\nMain concern: For systematic generalization in module networks, a desirable property is that each module learns to perform a distinct sub-task, independent of the program layout. In this way, each module is a distinct computation block, and modules can be arranged arbitrarily in any legal order to represent a valid computation graph that performs the desired computation. For this goal, I appreciate the authors using ground-truth program layouts to focus on the ability of the modules to learn such behavior without the added noise from the program generator. However, a good test for this type of systematicity is evaluating with systematically OOD test program layouts, to see if the learned modules can simply be organized in different layouts to still achieve correct predictions. By focusing more on OOD image distributions, the cause for better generalization is a bit less clear. The authors change the image distribution for both multi-attribute MNIST and CLEVR by using CoGenT. For instance, between CLEVR-CoGenT and CLOSURE, I believe CLOSURE is a better test of systematic generalization than CLEVR-CoGenT for this study since it considers novel module arrangements (because the question distribution changes) for the same image distribution. I do see that the authors included CLOSURE results in the appendix, but the proposed method does not provide benefits in that case. In SQOOP, the bias in the question setting does not provide any useful signal since both models achieve near-perfect systematic generalization. I think it is fine not to beat existing baselines as long as there is something to learn, but the experiments for these settings are minimal.\n\nOther concerns/questions:\n- With the three types of groupings presented in the paper, along with three levels of the pipeline, there could be a total of $3^3=27$ possible combinations of modularity at different levels. Did the authors experiment with all of them? If not, how did they choose the specific subset to experiment with?\n- Lines 78-79: Is modularity only imposed on the classifier in the presence of a binary classification task? What happens in the case of CLEVR, where it is not a binary classification task? For the one network per sub-task setting, is each sub-task classifier a binary classifier for all datasets, including CLEVR?\n- Why is Section 5 a separate section instead of just another result in Section 4?\n\nOther notes:\n- The paper lacks a related work section to set up the context in the literature thoroughly. But it does provide many of the important references in the paper body.\n- Lines 25-27: \"et al.\" does not need to be italicized. Also, it is synonymous with \"and others\", so \"X et al.\" are multiple people. Thus, it would be best if you said \"X et al. have shown...\" instead of \"X et al. has shown...\" (same with \"report\" vs \"reports\" in line 27).\nLines 65-68: It is a bit difficult at first to follow the authors' introduction of the terms \"sub-task\" and \"group\", how they relate to \"blocks\". It would be great to illustrate these terms with a short example program.\n\nThe results of the paper are significant and important to the community. However, it is difficult to take away the message clearly from the limited presented experiments that test the compositionality of the modules in the in-distribution image, out-of-distribution layout setting.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors adequately discuss the limitations of having manually tuned the degree of modularity, and that of using ground-truth layouts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "9"}, "extra_text": {}}, {"review_id": "H85-JFVAhC", "reviewer": "Reviewer_1KRM", "summary": "The authors analyze the preferred granularity of the modules at different stages of modular neural networks, which leads to the best systematic generalization. Based on Multi-attribute MNIST, they conclude that the modules are most useful when responsible to a specific semantically meaningful feature (like color, shape, etc), as opposed to having a single module or having modules for specific instances of those features (like green, 2, etc). They also conclude, somewhat surprisingly, that modularity is important already in the early stages of the network, so the feature extractor should also be modular. They show the advantage of the modular feature extractor on the more complex CLEVR dataset as well.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality\nTo the best of my knowledge, the provided analysis on the desired granularity of modules is original.\n\n# Quality\nThe paper is sound and the claims are supported by the findings.\n\n# Clarity\nThe paper is well written and mostly clear. A few recommendations:\n- It would worth explicitly mentioning what does the subtask mean early in the paper\n- For the multi-digit MNIST experiments, Figure 2b shows 2 feature extractor columns. How the multi-digit image is split among those 2 columns? Is it fed to both of them or are the 2 images rendered separately and fed to the encoders individually?\n\n# Significance\nThe findings on the importance of modular encoding is surprising and the community would benefit from such a conclusion. The paper also confirms that the intuitive way of allocating different modules to different semantically meaningful attributes is beneficial. The paper thus is valuable for the community.\n\n# Additional questions\nAn additional experiment that I would like to see is what is the benefit of a modular encoder alone with the rest of the network being monolithic.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a careful investigation into how modularity affects systematic generalization across a number of synthetic datasets. The key contribution is to highlight the role of modularity, especially when introduced at the early stage of the network. 

The most important concern raised by reviewers is that the experimental setting is limited. Neural Module Network, the network that all experiments are based on, is very rarely used in practice, and all the datasets are synthetic. On top of this, ground truth grouping or program layout is provided to the network, which wouldn't be normally available. It is genuinely difficult to tell how the results will translate into performance improvements more broadly. 

On the positive side, the paper is technically sound and the conclusions are interesting, as agreed by most reviewers. With the added experiments in the rebuttal, the experimental data clearly supports that adding modularity to NMNs improves systematic generalization. Based on this, I am happy to support acceptance of the work. I would like to strongly encourage the Authors to broaden their experimental setting in future work, so that these results have real impact on how we train neural networks. Please remember to address all reviewers' comments in the camera ready version. 
Decision: Accept (Poster)