Transformer Module Networks for Systematic Generalization in Visual Question Answering
Authors: Moyuru Yamada
Vanessa D'Amario
Kentaro Takemoto
Xavier Boix
Tomotake Sasaki
Venue: Submitted to ICLR 2023 (2023)
Keywords: Systematic generalization
Neural Module Network
Transformer

Abstract: Transformers achieve great performance on Visual Question Answering (VQA). However, their systematic generalization capabilities, i.e., handling novel combinations of known concepts, is unclear. We reveal that Neural Module Networks (NMNs), i.e., question-specific compositions of modules that tackle a sub-task, achieve better or similar systematic generalization performance than the conventional Transformers, even though NMNs' modules are CNN-based. In order to address this shortcoming of Transformers with respect to NMNs, in this paper we investigate whether and how modularity can bring benefits to Transformers. Namely, we introduce Transformer Module Network (TMN), a novel NMN based on compositions of Transformer modules. TMNs achieve state-of-the-art systematic generalization performance in three VQA datasets, improving more than 30% over standard Transformers for novel compositions of sub-tasks. We show that not only the module composition but also the module specialization for each sub-task are the key of such performance gain.

Under review as a conference paper at ICLR 2023
TRANSFORMER MODULE NETWORKS FOR SYSTEMATIC GENERALIZATION IN VISUAL QUESTION ANSWERING
Anonymous authors Paper under double-blind review
ABSTRACT
Transformers achieve great performance on Visual Question Answering (VQA). However, their systematic generalization capabilities, i.e. handling novel combi- nations
of known concepts, is unclear. We reveal that Neural Module Networks (NMNs), i.e., question-specific compositions modules that tackle sub-task, achieve better similar systematic generalization performance than the conven- tional Transformers, even though NMNs' modules are CNN- based. In order to address this shortcoming of Transformers with respect to NMNs, in this paper we investigate whether and how modularity can bring benefits to Transformers. Namely, introduce Transformer Module Network (TMN), a novel NMN based on compositions of Transformer modules. TMNs achieve state-of-the-an system- atic generalization performance in three VQA datasets, improving more than 30% over standard Transformers for novel compositions of sub-tasks. We show that not only the module composition but also the module specialization for each sub-task are the key of such performance gain.
INTRODUCTION
Visual Question Answering (VQA) (Antol al. 2015) is a fundamental testbed assess the ca- pability of learning machines to perform complex visual reasoning. The compositional structure inherent to visual reasoning is at the core of VQA: Visual reasoning is composition of visual sub- tasks, and also, visual scenes are compositions of objects, which are composed of attributes such as textures, shapes and colors. This compositional structure yields a distribution of image-question pairs of combinatorial size, which cannot be fully reflected in an unbiased way by training distribu- tions.
Systematic generalization is the ability to generalize to novel compositions of known concepts be- yond the training distribution (Lake & Baroni 2018: Bahdanau et al., 2019: Ruis et al., 2020). A learning machine capable of systematic generalization still distant goal, which contrasts with the exquisite ability of current learning machines to generalize in-distribution. In fact, the most successful learning machines, i.e., Transformer-batt
models, have been tremendously effective for VQA when evaluated in-distribution (Tan & Bansal 2019 Chen et al. 2020 Zhhang et al.) 2021). Yet, recent studies stressed the need evaluate systematic generalization instead of in-distribution
generalization (Gontier et al. 2020 Tsarkov et al., 2021 Bergen et al. 2021), as the systematic generalization capabilities of Transformers for VQA are largely unknown.
A recent strand of research for systematic generalization in VQA investigates Neural Module Net- works (NMNs) [Bahdanau al. 2019 2020, D Amario al. 202T). NMNs decompose a question in VQA into sub-tasks, and each sub-task is tackled with shallow neural network called module. Thus, NMNs use a question-specifie composition of modules to answer novel questions. NMNs alleviate the gap between in-distribution generalization and systematic generalization due to its in- herent compositional structure. In our experiments, we found that CNN-based NMNs outperform Transformers systematic generalization novel compositions of sub-tasks. This begs the ques- tion of whether we can combine the strengths of Transformers and NMNs in order to improve the systematic generalization capabilities of learning machines.

---

Under review as a conference paper at ICLR 2023
Example
Example 2
Answer A;
Answer Ag
Library of modules
Module architecture
Module
Module
Module A
Transfomer Encoder
Module
Module
Module
Transformer Encoder
Modull
Madule A
Module C
Module
Module
Module
atorgs)
Image
Program Pg
Sex bask-akk-33 314eg31
Image/s l3
Question
Program P2
Queston Q2
(a) Library and module architecture
(b) Examples of question-speciff networks
Figure 1: Overview of Transformer Module Network (TMN). (a) TMN has a library of neural mod- ules that tackle sub-task, and each module implemented with small stack of Transformer encoders. (b) The modules are composed into question-specific network. This network takes as input an image and a program that corresponds a question. The total number of Transformer layers is the number of Transformer layers of each module (K) x the program length (L).
In this paper, introduce Transformer Module Network (TMN), novel NMN for VQA based compositions of Transformer modules. In this way, we take the best of both worlds: the capabilities of Transformers given by attention mechanisms, and the flexibility of NMNs to adjust to questions based on novel compositions of modules. TMN allows us to investigate whether and how modularity brings benefits Transformers VQA. overview of TMNs depicted in Fig.
To foreshadow the results, we find that TMNs achieve state-of-the- systematic generalization accuracy the following three VQA datasets: CLEVR-CoGenT (Johnson et al.] (017), CLO- SURE (Bahdanau et al.) 2020), and novel test set based on GQA (Hudson & Manning 2019) that we introduce for evaluating systematic generalization performance with natural images, which call GQA-SGL (Systematic Generalization to Linguistic combinations). Remarkably, TMNs im- prove systematic generalization accuracy over standard Transformers more than 30% in the CLO- SURE dataset, i.e., systematic generalization to novel combinations of known linguistic constructs (equivalently, sub-tasks). Our results also show that both module composition and module special- ization a sub-task are key TMN's performance gain.
RELATED WORK
We review previous works systematic generalization in VQA. We first revisit the available bench- marks and then introduce existing approaches.
Benchmarking VQA. Even though systematic generalization capabilities are the crux of VQA, attempts to benchmark these capabilities are only recent. The first VQA datasets evaluated in- distribution generalization, and later ones evaluated generalization under distribution shifts that do not require systematicity. In the following, we review progress made towards benchmarking sys- tematic generalization in VQA:
-n-distrribution generalization: There is plethora of datasets to evaluate in-distribution gener- alization, e.g., VQA-v2 (Goyal et al. 2019) and GQA (Hudson Manning. 2019). It has been reported that these datasets biased and models achieve high accuracy by relying spurious correlations instead of performing visual reasoning (Agrawal et al. 2018 Kervadee et al. (021).
-Out-of-distribuue generalization: VQA-CP (Agrawal et al. 2018 and GQA-OOD (Kervadec) et al. 2021) were proposed to evaluate generalization under shifted distribution of question-anse -nnwwrr pairs. While this requires stronger form of generalization than in-distribution, it does not require tackling the combinatorial nature of visual reasoning, and models can leverage biases in the images and questions.
2

---

Under review as a conference paper at ICLR 2023
Systematic generalization: CLEVR-CoGenT (Johnson al. 2017) and CLOSURE (Bahdanau et al. 2020) are datasets that require systematic generalization as models need to tackle novel com- binations of visual attributes and sub-tasks. Since these datasets include only synthetic images, we introduce GQA-SGI, a novel test set based on GQA to evaluate systematic generalization with
natural images.
Approaches for Systematic Generalization. We now revisit Transformer-base models and NMNs for systematic generalization in VQA as they are the basis of TMNs:
-Trunsformer-Base Models: Currently, most approaches to VQA are based on Transform- (Vaswani et al. 2017); e.g.. Tan & Bansal (2019); Chen et al. (2020); Zhang et al. (202T), and pre-training is at the core of these approaches. These works focus on in-distribution generalization,
and it was not until recently that Transformers for systematic generalization have been investigated.
To the best of our knowledge, in VQA the only related work is MDETR (Kamath et al. (021), which uses novel training approach that captures the long tail of visual concepts and achieves state-of-the- art performance on many ssiionand-language datasets, including CLEVR-CoGenls As we show in the sequel, our approach shows better systematic generalization capabilities CLEVR-CoGenT and CLOSURE without requiring pre-trained Transformer encoders.
-Neural Module Networks (NMNs): They represent a question the form of a program in which each sub-task is implemented with neural module. Thus, modules composed into network specific for the question (Andreas et al, 2016). Some of the most successful modular approaches include NS-VQA (Yi 2018),, which uses symbolic execution engine instead of modules based on neural networks, and the Meta-Module Network (MMN) (Chen et al.] (021). which introduces a module that can adjust to novel sub-tasks. While these approaches are effective for in-distribution generalization, they fall short in terms of systematic generalization. Vector-NMN (Bahdanau et al.) 2020) is the -fate-of-the-art NMN on systematic generalization and outperforms all previous mod- ular approaches. We present evidence that the systematic generalization capabilities of modular approaches can be improved by using the attention mechanisms of Transformer architectures.
TRANSFORMER MODULE NETWORKS (TMNs)
In this section we introduce TMNs, i.e., our novel Transformer-based model for VQA that composes neural modules into qqestion-speeiffe Transformer network. Unlike previous NMNs, TMNs regard the tokens that represent the input image as a workspace shared among the modules, which transform the token representations. This architectural change is not straightforward and thus need to architect NMNs taking into account the philosophy of tokens to leverage the attention mechanism.
Question-speeific Composition of Modules. F provides overview. TMNS a library of modules, and each module tackles a different sub-task we defined (c.g.. FILTER, COUNT). TMNs represent the question in VQA with a program, which is a sequence of sub-tasks each of which is implementable with a module from the library. For instance, a question "How many red squares are there?" can be converted to a program {E {FILTER (square), , FILTER (red), COUNT), } and then the modules corresponding to the sub- tasks are composed to form a network to answer the question.
In TMN, each module is stack of Transformer encoders. Unlike the standard Transformers, which are a stack of fixed number of Transformer encoders (typically 12) regardless of the input question, TMN composes the Transformer modules into a question-specifar network, and thus, the total num- ber of encoder layers in TMN varies according to the question. Let K be the number of Transformer encoder layers in each module, and let L be the number of sub-tasks in the program. Then, TMN has total of K x L layers implement a program of length L.
We follow the same approach in previous works that directly use the program rather than the raw question (Bahdanau al.) 2020, 2019; D Amario et al.) (0221). This allows to analyse systematic generalization in isolation by focusing only on visual reasoning aspects, and omits the performance of the language parser.
Transformer- based Architecture. is preceded by feature extractor and followed by an output classifier, as shown in Fig. aa).
The aforementioned question-specific composition of modules
We use feature extractor to obtain the visual features of the input image. We follow the common procedure in Transformers: The visual features transformed into the visual feature embeddings
3

---

Under review as a conference paper at ICLR 2023
Answer:
Classifier
(Stack)
(Tree)
COUNT Module
Library of modules
null
FILTER Module
Answer
red
FILTER Module
Module (6)
Answer
MM
Square
AND
E Embedding
Madule (4)
Module (6)
E Embedding
Module (3)
AND / OR (5)
Feature extractor
Module (2)
Module (2)
Module (4)
FILTER(SQuare) (rod)
Module (1)
Module (1)
Module (3)
COUNT
Image
P Program
Image
Program
Image
Program Image
Program
TMN Architecture
(b) Stack and Tree modular structures
Figure 2: TMN Architecture and Modular Structures. (a) Given an image and a program related to question, TMNs construct cresstonn-specffe Transformer network as composition of modules that tackle a sub-task. Each module takes as input the representations of the visual features, the head token and the module's argument from the program. A classifier outputs an answer from the final representation of the head token. (b) We investigate two modular structures. Stack structure simply stacks all the modules. Tree structure runs two threads and then merges their outputs with module (AND/OR in this example). The numbers in denotes the order each sub-task in the program.
and then summed with the position embedding, which yields the input visual representations of the first module. The initial head token representation is the average all the visual representations.
Then, the first Transformer module in the question-specific network takes input the sequence of tokens consisting of the representations of the visual features, the head token, and the module's 's ar- gument coming from the program. The Transformer module outputs the transformed representations of the head token and visual features. This output is fed into the next module, which also receives its corresponding argument coming from the program. Unlike Vector-NMN, whose modules always receive the initial visual representations, in TMN only the first module receives the initial visual rep- resentations. Also, due the versatility Transformer architectures, in TMN we can easily change the number of Transformer encoders in each module according to the complexity of the sub-task.
Finally, classifier outputs an answer from the final representation of the head token.
Stack and Tree Modular Structures. We further analyze the effect of modular structures. Some modules with two inputs may be useful for constructing network corresponding program. For instance, a logical sub-task (AND or OR) or sub-tasks that compare two objects. For programs containing such sub-tasks, we investigate two modular structures exemplified in Fig. (b). Stack and tree structures are widely studied in previous works (Andreas et al., [016]] Hu et al.) 2017]. The stack structure simply stacks modules executes the modules in sequence. The tree structure runs two threads in parallel and then merges their outputs with module (AND/OR in this example), which takes two sets representations visual features and a head token as an input and outputs the transformed representations of the first set in our case. We compare these two structures in the experiments.
complex
4 EXPERIMENTAL SETUP
We use three datasets for VQA: two exising datasets and novel test set we built. We compare our TMN with two Transformer-basee baselines and several state-of-the-a models on those datasets.
Datasets. To evaluate the models for systematic generalization to novel combinations of known visual attributes or linguistic constructs, the following three datasets.
4

---

Under review as a conference paper at ICLR 2023
Extractor
Emb
Extractor
Emb
Stack of mransformer Encoders
Stack of Transformer Encoders
Image
Answer
cstor
motor
Emb
reine
Question
Program
(a) Transformer
(b) Transformer w/ PR
Figure 3: Transformer-basel Baseline Methods. (a) Standard Transformer (Transformer) takes an image and question inputs. (b) Standard Transformer with programs (Transformer w/ PR) takes a program as an input instead of the question. Emb denotes visual/word embeddings. These approaches use the same visual inputs and Transformer encoder structure.
CLEVR-CoGenn CLEVR is diagnostic dataset for VQA models (Johnson al. 2017) that consists of synthetic 3D scenes with multiple objects and automatically generated questionss, associ- ated with ground-truth program formed by multiple sub-tasks. This dataset comes with additional splits to test systematic generalization, namely, the Compositional Generalization Test (CLEVR- CoGenT). (LEVR-CoGent is divided in two conditions where objects appear with limited color and shape combinations, that are inverted between training and testing. Using this dataset, we can test systematic generalization novel combinations of visual attributes.
-CLOSURE: This is a test set for models trained with CLEVR, and provides 7 benchmarks (Bah- danau al.) 2020). The benchmarks test the systematic generalization novel combinations of known linguistic constructs, which can be considered as sub-tasks, e.g., "is the same size as" can be sub-task SAME_SI2S CLOSURE uses the same synthetic images in CLEVR but contains questions which require the models to recombine the known sub-tasks in novel way.
-GQA-SGL: GQA is VQA dataset that consists of complex natural images and questions, as- sociated with ground-truth programs (Hudson & Manning, 2019). GQA-SGL. (Systematic Gener- alization to Linguistic combinations) generalization to novel combinations of known linguistic constructs with natural images. We gen- erate new questions with ground-truth programs by combining the existing programs as shown in Fig. 4 in the Appendix A To build the test set, we use the test-dev set from the balanced split. For the train set, in all experiments we always use the balanced split that controls bias. See Appendix A for details.
novel test set we built based on GQA to test systematic
Methods and Settings, We compare TMNs with two Transformer- bascline methods shown in Fig.3) In (a), standard Transformer (Transformer) takes image and question the inputs. In (b), standard Transformer with programs (Transformer w/ PR) takes program as an input instead of the question and treat it as a sequence of tokens. TMNs also take the program an input and executes each module with corresponding arguments as shown in Fig. 2 Thus, comparing Transformer w/ PR with TMNs allow us assess the effectiveness of our Transformer modules and question-specific compositions of the modules. We treat VQA as classification task and train the models with cross- entropy loss across the TMNs and our baselines.
-Archtecturr: We use vanilla Transformer encoder for both standard Transformers and our TMNs for fair comparison. Standard Transformers consist 12 Transformer layers. In TMNs, the number of Transformer layers in each module is 1 for CLEVR-Code ToRGen' and CLOSURE, and 2 for GQA- SGL because the difference in the complexity of their sub-tasks. Transformer encoders in all models have a hidden state size of 768 and 12 attention heads.
-Sub-tasks and programs: For CLEVR-CoGenT and CLOOSRE,, we use the sub-tasks and pro- grams defined in the datasct. For GQA-SGL, follow the definitions proposed in Chen et al. 2021).
-Visual features: For CLEVR-CoGenT and CLOSURE, we use grid features extracted by ResNet- 101 pre-trained with ImageNet to obtain the visual feature of x W x 2048 dimensions, where H and W are the height and the width of the grid feature map. We add positional encod- ing to the grid features and flatten them with linear projection. We treat the flattened features as sequence of tokens and thus the number of the tokens H W (150 these datasets). The
5

---

Under review as a conference paper at ICLR 2023
input images in GQA contain various objects and are much more complex than those in CLOSURE. Therefore, for SQA-SGL, we use the popular regional features (Anderson et al. 2018 extracted by Faster R-CNN (Ren et al. 2015) (with 1esNNet1101 backbone) pre-trained on Visual Genome dataset (Krishna et al. 2016). We consistently keep 36 regions for each image. Each region has lin-siimeniional visual feature and 5-dimensional spatial location feature.
The input representation of the program is a sum of word embeddings, segment embeddings and position embeddings. They represent a sub-task an argument (e.g., FILTER or circle), thread index, and word position, respectively. For the standard Transformer, we adopt BERT tokenizer and the standard sentence embeddings described (Li et al. 2020).
-Embeddings:
-Hyperparameters: We use the Adam optimizer for all cases. Learning rates for standard Trans- formers and TMNs are 2c-5 and le-5 for CLEVR-COGeNT, 2c-5 for CLEVR, and le-5 and 4e-5 for GQA. We search best learning rates for each model. We train all the models with batch size of 128 on Tesla V100 GPUs for 20 epochs except standard Transformer on CLEVR, where we use 30 epochs to achieve convergence. Training of standard Transformers and TMNs finished in 3 and days, respectively. We use this experimental setup and hyperparameter tuning for all experiments.
-Other methods: We also experiment with other state-of-the-art methods, including Vector- NMN (Bahdanau et al.) 2020), MDETR (Kamath et al.] (02T) and MMN (Chen et al. (021)) (LXMERT (Tan & Bansal 2019) in Appendix GD. We use the hyperparamters described in the orig- inal papers. For a fair comparison, we train Vector-NMN and MMN using their publicly available code on the same training
the rest of the methods. We also use the ground-truth programs as in TMN. MDETR was already trained on the same training set as the rest of the methods, and hence, use the publicly available trained model. Also, report the performance of NS-VQA (Yi al., 2018) from its original paper and Akula et al. (021).
5 RESULTS AND ANALYSIS
In this section, we first report the systematic generalization performance to novel combinations of vi- sual attributes linguistic constructs with synthetic images (CLEVR-CoGenl) and CLOSURE) and complex natural images (GQA-SGL). Then, we analyze the effect of module composition, module specialization, and pre-training.
5.1 SYSTEMATIC GENERALIZATION PERFORMANCE
Tables I andd] 2 show the mean and standard deviation accuracy of TMNs and the rest of approaches
for both in-distribution and systematic generalization (see Appendix B D and [ for further de-
tails). In the following, we first introduce the results in the three datasets, and then, we discuss them:
CLEVR-CoGenT. It evaluates the systematic generalization performance to novel combinations of known visual attributes. All models are trained on CLEVR-CoGenT condition A. Our results show that both standard Transformers and TMNs largely outperform the state-of-the-art modular approaches (Vector-NMN and NS-VQA). TMNs' systematic generalization performance also sur- passes MDETR, which is the state-of-the-art Transformer-basse model proposed to capture long- tail visual concepts. TMNs achieve superior performance over the standard Transformer but their performances are slightly lower than the Transformer with programs (Transformer w/ PR). As ex- pected, Transformer w/ PR outperforms standard Transformer as the use of program facilitates visual grounding.
CLOSURE. It evaluates the systematic generalization performance to novel combinations of known linguistic constructs. All models are trained on CLEVR. Our results show that standard Transformers struggle with novel compositions of known linguistic constructs even when the pro- gram is provided (Transformer w/ PR). TMNs achieve much better performance than standard Trans- formers and MDETR, and also outperform the other modular approaches (Vector-NMN and NS- VQA). Remarkably, TMN-Tree improves systematic generalization accuracy standard Trans- formers more than 30% in this dataset. Tree structure seems effective for CLOSURE because the questions often have tree structure.

---

Under review as a conference paper at ICLR 2023
Table 1: Results on systematic generalization novel combinations of known visual attributes and linguistic constructs. Mcan and standard deviation of accuracy (%) across at least three repetitions when possible (only one trained model is provided in NS-VQA and MDETR). All methods are tested on in-distribution FLEVR-CoGent validation condition A and CLEVR validation) and systematic generalization (CLEVR-Cood - validation condition B and CLOSURE, indicated in yellow). We trained and test Vector-NMN CLEVR-CoGenT in our environment while its performances on CLEVR and CLOSURE are cited from the original paper (Bahdanau et al. 2020). "Overall" the right end is the mean value of accuracies CoGenT-B and CLOSURE.
Visual attributes
Linguistic constructs
Overall
Methods
CoGenT-A (In-Dist.)
C-GenT-B
CLEVR
CLOSURE
(Syst. Gen.)
(In-Dist.)
(Syst. Gen.)
(Syst. Gen.)
Transformer
97.5 0.18
78.9 = 0,80
9
57.4 = 1.6
68.2
Transformer w/ PR
97.4 + 0.56
81.7 = 1.1
97.1 + 0.14
64.5 + 2.5
73.1
TMN-Stack (ours)
97.9 + 0.03
80.6 = 0.21
98.0 = 0.03
90.9 = 0.49
85.3
TMN-Tree (ours)
98.0 + 0.02
80.1 = 0.72
97.9 - 0.01
95.4 = 0.20
87.8
Vector-NMN
98.0 0.2
73.2 = 0.2
98.0 1 0.07
94.4
83.8
NS-VQA
99.8
63.9
99.8
76.4
70.2
MDETR
99.7
76.2
99.7
53.3
64.8
Table 2: Results application complex natural images with novel combinations of known linguistic constructs. Mean and standard deviation of accuracy (%) on GQA test-dev and GQA- SGL across at least three repetitions. We pre-trained object detector The numbers in () denote the mean accuracy on only four question types contained in GQA-SGI. (verify, query, choose, and logical). Systematic generalization performance is indicated in yellow.
feature extractor.
GQA
GQA-SGL.
Methods
(In-Distribution)
(Systematic generalization)
Transformer
54.9 = 0.004 (67.6)
47.7 + 2.1 52.2 + 3.2
Transformer w/ PR
54.7 = 0.22 (67.0)
TMN-Stack (ours)
52.8 = 0.10 (64.7)
50.7 + 0.94
TMN-Tree (ours)
53.5 0.24 (65.2)
53.7 + 1.7
MMN
55.6 = 0.12 (67.8)
51.0 + 1.5
GQA-SGL. We evaluate the systematic generalization performance to novel combinations of known linguistic constructs with natural images. All models are trained on GQA. Table [ we evaluate the exact matching between the predicted and ground-truth answers, both GQA and GQA-SGL. TMN-Tree achieves superior performance on systematic generalization with a smaller drop of in-distribution accuracy compared standard Transformers. Although MMN is the recently proposed NMN for GQA and achieves great in-distribution performance, TMN-Tree outperforms MMN on systematic generalization.
Discussion of the Results. Our results in CLOSURE and GQA-SGL show that TMNs (i.e., in- troducing modularity Transformers) improve the systematic generalization performance novel combinations of known linguistic constructs (i.e., novel compositions of sub-tasks) of Transformers and NMNs. The performance gain CLOSURE is relatively larger than that in GQA-SGL tests. One reason to explain this is the complexity of the questions: the maximum program length is 26 in CLOSURE, while it stronger systematic generalization capabilities. Results in CLEVR-CoGeen' show that TMNS do not improve systematic generalization of Transformers for novel compositions of visual attributes. This may be expected as D Amario et al. (2021) demonstrated that the visual feature extractor is the key component for generalization to novel combinations of visual attributes, and Transformer and TMNs have the same visual feature extractor. Finally, we observe that TMNs achieve the best accuracy across all tested methods in CLOSURE and GQA-SGL, and competitive accuracy with the best performing method in CLEVR-CoGenT..
GQA. CLOSURE provides more complex questions which require

---

Under review as a conference paper at ICLR 2023
Table 3: Effect of specialization of modules. Mean and standard deviation of accuracy (%) on CLEVR and CLOSURE across three repetitions with different libraries of modules. "Individual" denotes that module is assigned to a single sub-task. "Semantic group" and "Random group" denote that a module is assigned to group including sub-tasks that are semantically similar or randomly selected, respectively. "Order" denotes that module is selected by the order of sub-task in program.
Library of modules Individual
The number
Specialization of modules
CLEVR
CLOSURE
of modules
(In-Distribution)
(Syst. Gemeralization)
26
98.0 = 0.030
90.9 + 0.49 93.7 + 0.55
Semantic group 12
98.1 + 0.014
Random group
12
97.9 + 0.059
93,0 + 0.29
Order
12
92.2 + 1.5
68.4 + 1.8
Table 4: Ablation analysis. Mean and standard deviation of accuracy (%) on CLEVR and CLO- SURE across three repetitions. We add imitations of properties of TMN, i.e., Variable number of layers (VL) and Split tokens (ST) to the standard Transformer w/ PR. SoM stands for the specializa- tion of the modules.
Properties
CLEVR
CLOSURE
Methods
VI.
ST
SoM
(In-Distributio)
(Syst. Generalization)
Transformer w/ PR
97.1 + 0.14
64.5 t 2.5 60.2 + 2.9
Transformer w/ PR + VL
94.0 + 1.8
Transformer w/ PR + VL ST
V
56.8 + 0.042
43.6 + 0.59
TMN-Stack
98.0 + 0.030
90.9 + 0.49
5.2 ANALYSIS
Behavioral Analysis. We analyze the behavior of our trained modules, i.e., how Transformer modules work, by visualizing attention maps in Appendix H The qualitative results indicate that the modules learn their functions properly. We also analyze the Transformer w/ PR Appendix !
Module Specialization. While the question-specifer composition of modules is a key property of NMNs, recent work reports that degree of modularity of the NMN also has large influence on systematic generalization (D' Amario et al. (2021). Here we also investigate this aspect for TMNs. Table 3 shows the mean accuracy of TMN- with four different librarics of modules, each with different degree of specialization. "Individual" denotes that each module is assigned to a single sub-task (26 sub-tasks defined CLEVR in total). This the library used the previous sections. "Semantic group" denotes that each module tackles group of sub-tasks that semantically sim- ilar. "Random group" denotes that each module tackles a group of sub-tasks randomly selected. "Order" denotes that each module is always placed in the same position of the question-specific network, independently of the sub-task performed in that position. If there are more sub-tasks in the program than modules in the library, the modules are repeated following the same order. this way, we compare the importance of modules tackling sub-tasks with the importance of having cneettionsspciffe network. See Appendix for further details about each library.
Results shows that the systematic generalization accuracy largely drops when the modules are not specialized, even if the number of modules is the same (compare "Order" with "Semantic group" and "Random group"). The degree of modularity (Individual vs Semantic group) and how sub-tasks are grouped (Semantic vs Random) also affects systematic generalization performance. These results suggest that question-specifer composition of modules is not enough and specialization of modules is crucially important. Finally, note that the results of Semantic group and Random group also show that TMN performs better than the standard Transformer (93.7 vs 64.5) even if the total number of parameters that modules have is the same as the number parameters of the standard Transformer (i.e., TMN has 12 modules of one layer each, while Transformers have 12 layers).
Ablation Analysis. We investigate the contribution of TMN's specialization modules (SoM) to TMNs' superior performance over standard Transformer. Besides SoM, TMNs also have variable
8

---

Under review as a conference paper at ICLR 2023
Table 5: Effect of pre-training. Mean accuracy (%) on CLEVR-CoGenT validation condition B and CLOSURE with two different feature extractors, which are an object detector pre-trained on Visual Genome (VG) and a ResNet-101 (without pre-training on VG).
LLEVR-CoGenn
CLOSURE w/ VG
Methods
w/o VG
w/ VG
A
w/o VG
A
Transformer
78.9
87.5
8.6
57.4
62.1
4.7
Transformer w/ PR
81.7
88.7
6,9
64.5
64.5
0.005
TMN-Stac (ours)
80.6
86.9
6.3
90.9
92.7
1.7
TMN-Tree (ours) 80.1
86.4
6.4
95.4
96.4
1.0
number of layers, while standard Transformer have fixed number of layers. We analyze if it is this difference in number of layers or the SoM that is causing TMNs' higher accuracy. To do so, add property named Variable number of layers (VL) the Transformer w/ PR. With the VL, the Transformer w/ PR uses as many encoder layers as the program length (e.g., 18 layers are used output an answer for an input program with a length of 18). VL can be regarded of the guestion-specife compositions of the modules in TMN, but each layer is not specialized for specific sub-tasks. Another key difference between TMNs and Transformers besides SoM that TMN modules takes as argument tokens related to the sub-task indicated by the program, while every layer of standard Transformer takes all input tokens in the program, because Transformers' layers are not specialized to sub-tasks. Splitting the program tokens may help the model steer the computations that should be executed at each layer. To investigate this, we add this property, named as Split Tokens (ST), the Transformer w/ PR.
an imitation
Table shows that any of the aforementioned properties do not improve the systematic general- ization performance without the specialization of modules sub-tasks, i.e., SoM. Even with Split tokens, the performance does not improve because the layers can not be specialized and splitting the token can spoil the flexibly of the standard Transformer.
Pre-training. We analyze how the additional image data from other domains affects systematic generalization novel combinations of visual attributes or linguistic constructs, i.e., CLEVR- CoGenT and CLOSURE, respectively. Table 55 shows the mean accuracy with two different visual features, namely, regional features extracted by an object detector pre-trained on Visual Genome (VG) and grid features extracted by a ResNet-101 without pre-training on VG. Across all the tested models, we observe, as expected, that the performance gains from pre-training the visual feature extractor are larger for generalization to novel visual combinations and smaller for novel linguistic combinations. This highlights the need to control pre-training in order to ensure a fair comparison of different methods for systematic generalization. See Appendix F and for further experiments.
6 CONCLUSIONS
We have compared the systematic generalization capabilities of TMNs with the most promising approaches in the literature, i.e., Transformers and NMNs, on three VQA datasets. TMNs achieve state-of-the-ar systematic generalization performance across all tested benchmarks. We have shown that the question-specific composition of Trransformer modules and the specialization of the modules to each sub-task are the key of the performance gains. Also, our results highlight that a fair compar- ison for systematic generalization requires controlling that all compared methods are pre-trained in the same way. We hope that our investigation unleash the potential of modular approaches for sys- tematic generalization as they motivate promising future work towards creating large-scale modular architectures.
Limitations. TMNs have similar limitations as NMNs, i.e., is challenging to decompose a ques- tion into sub-tasks. We have used the sub-tasks and the programs provided in the datasets generated by humans. This makes it difficult to apply NMNs and TMNs to other datasets that do not contain the programs. Another limitation is that we only investigated two types of systematic generalization (novel compositions of visual attributes and linguistic constructs). These limitations are key open questions that we will tackle in future work.

---

Under review as a conference paper at ICLR 2023
Reproducibility Statement. The source code and new dataset used in this study are included in the supplementary materials for review and will be made publicly available on GitHub.com upon acceptance. URLs of other existing assets (datasets, source code and trained models) used in this study are provided in the reference section. Experimental setup is described in Section
REFERENCES
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proc. of the 31st IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR 2018), pp. 4971 2018.
Arjun Reddy Akula, Varun Jampani, Soravit Changpinyo, and Song-Chun Zhu. Robust visual rea- soning via language guided neural module networks. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021.
Peter Anderson, Xiaodong He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answer- ing. In Proc. of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), pp. 6077-6086, 2018.
Jacob Andreas, Mareus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proc. of the 29th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2016), pp. 39 48, 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zit- nick, and Devi Parikh. VQA: Visual Question Answering. In Proc. the Conference Computer Vision (ICCV 2015), 2015.
the 15th IEEE International
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In Proc. of the 7th International Conference on Learning Representations (ICLR 2019), 2019.
Dzmitry Bahdanau, Harm de Vries, Timothy J. O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron C. Courville. CLOSURE: assessing systematic generalization of CLEVR models. arXiv preprint, 112iv.192..05783v2 2020. The CLOSURE dataset is avaialble
at Iattpa: / zenodo. org/ http////eoo/////////////// code of Vector-NMN is available at httttp:://gtttub.co http:////tg.ttutccocataaa////a/a wtt..///wraarr....oooe.ooo on Sept. 21st, 2022).
3634090# YOYE21TPOUE and the official source (Accessed
Leon Bergen, Timothy J. O'Donnell, and Dzmitry Bahdanau. Systematic generalization with edge transformers. In Advances in Neural Information Processing Systems 34 (NeurIPS 202I), pp- 1390-1402, 2021.
Wenhu Chen, Zhe Gan, Linjie Li, Yu Cheng, William Wang, and Jingjing Liu. Meta module network for compositional visual reasoning. In Proc. of the 202/ IEEE Winter Conference on Applications of Computer Vision (WACV 2021), pp. 655 655-664, 2021. The official source code and trained model of Meta Module Network are available hhttps://githhie com/ wwe.huchen/ /Meta Module e- Network (Accessed on Sept. 21st, 2022).
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European Conference on Computer Vision (ECCV), 2020.
Vanessa D' D'Amrio, Tomotake Sasaki, and Xavier Boix. How modular should neural module net- works be for systematic generalization? In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), pp. 23374-23385, 2021.
Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Pal. Measuring systematic generaliza- tion in neural proof generation with transformers. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 22231-22242, 2020.
10

---

(Accessed on Under review as a conference paper at ICLR 2023
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in visual ques- tion answering. In Proc. of the 32nd IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), pp. 398-414, 2019.
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to reason: End-to-end module networks for visual question answering. In Proc. of the 16th IEEE International Conference Computer Vision (ICCV 2017), pp. 804-813, 2017.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In Proc. of the 32nd IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR 2019), pp. 6700 2019, The GQA dataset is available at htttps:: / cs www.ssaaffr.... edu wtwppepppleee....... wtwllllddoooaaddae gga/ ndex. html Sept. 21st, 2022).
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary vi- sual reasoning. In Proc. of the 30th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2017), pp. 2901-2910, 2017. The CLEVR and CLEVR-CoGenT datasets are available at wtt.ttts:////m //cs. www.ntaaarrrooooo edu httppppppppjjjjjjsspssss clevr/ (Accessed on Sept. 21st, 2022).
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR modulated detection for end-to-end multi-modal understanding. In Proc. of the 18th IEEE/CVF International Conference on Computer Vision (ICCV 2021), pp. 1780- 1790, 2021. The official source code and trained model of MDETR are available https: //github. com/ ashkamathhmnn hannkammat//mdtzr t (Accessed on Sept. 21st, 2022).
Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should VQA expect them to? In Proc. of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), pp. 2776-2785 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David Shamma, Michael Bernstein, and Fei Fei Li. Visual Genome: Connecting language and vision using crowdsourced dense image annotations. Inter- national Journal of Computer Vision (IJC)), 123:32-73, 2016.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to- seequueeet- recurrent networks. In Proc. of the 35th International Conference on Machhne Learning (ICML 2018), pp. 2873-2882, 2018.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, and Ming Zhou. Unicoder- VL: A universal encoder for vision and language by cross-modal pre-training. In Proc. of the 34th AAAI Conference on Artificial Intelligence (AAAT-20), pp. 11336-1 1344, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28 (NIPS 2015), pp. 91-99, 2015.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden Lake. A bench- mark for systematic generalization in grounded language understanding. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), pp. 19861 11861, 2020.
Hao Tan and Mohit Bansal. LXMERT: Learning cross-modality encoder representations from trans- formers. In Proc. of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP 2019), pp. 5100-5111, 2019. The official source code and trained model of LXMERT are avail- able at http:::///ghtuu com/ airs wwwsppaaslaaaaaeaeeeaeaaeeeeeeee ixmert (Accessed on Sept. 21st, 2022).
Dmitry Tsarkov, Tibor Tihon, Nathan Scales, Nikola Momchev, Danila Sinopalnikov. and Nathanael Schärli. *-CFQ: Analyzing the scalability of machine learning on a compositional task. In Proc. of the 35th AAAI Conference on Artificial Intelligence (AAAI-21), pp. 9949-9957,, 2021.

---

Under review as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems 30 (NIPS 2017), pp. 5998-6008 2017.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum. Neural-symbolic VQA: Disentangling reasoning from vision and language understanding. In Advances in Neural Information Processing Systems 32 (NeurIPS 2018), pp. 1039-1050, 2018.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang. Lijuan Wang. Yejin Choi, and Jianfeng Gao. VinVL: Revisiting visual representations in vision-language models. In Proc. of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), pp- 5579-5588, 2021.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A GQA-SGL DATASET
GQA
GQA-SGL
Verify
Does <i> Spetial referring <B... <Alt>>?
Does <A> <Matchingg referring <Attr?
What As -aaching referiig==BB .....?
Query
ih -Spatial referringo
-Satiaa referring> -Spatial referring
< Spalial referring>...
<Ait 1>er <Att 2>?
<A> <Matching referiig>... <B> <C> <D>?
Are there both AA> and <B> ...? there <Attr> +/>or <B> ...?
Logical
Are there both <Atrr 1> ----anndd AAtrr 2 BB>..??
Figure 4: GQA-SGL test dataset for systematic generalization with natural images. GQA-SGL. novel test set built based GQA to evaluate systematic generalization performance. We generate new questions with ground-truth programs that belong to four question types (verify, query, choose, and logical) by combining the existing programs. <>....,D>> and <Attr> denote object (e.g., chair) and an attribute (c.g.. blue). <Spatial referring> and <Matching referring>> are referring expression such as " right of and is the same color ", respectively.
GQA
GQA-SGL
Dees the palm tree to the left the look blue? no
verify
of
What shting looks yellow and black? lupgage
table
open?
colorful yes
What no or the animal to the loft the books the water bottle? oranne
query
What the animal looks orange doing? lying
choose
open?
the device that the same color as the umbrella large small! small
Does the napkin have the same coloras the notebook? no
grinders
lights
photograph? yes
logical
Are there both a straight street blue house this photo? yes
Are there any traffic signs or fire hydrants that are made metal? no
Figure 5: Examples of test questions from GQA and GQA- -GL for the same images.
We generate GQA-SGL, novel GQA split to test Systematic Generalization to the novel Linguistic combinations of models trained on GQA. GQA-SGL contains 200 new questions with their ground- truth programs. 97.4% of the original GQA questions can be divided into four question types: verify, query, choose, and logical. Fig. describes the GQA-SGL data generation process. On the left, we first list out all ground- ground-truth programs that belong these question types. The arguments in parenthesis <.> denote either objects (e.g., A and B), attributes (e.g., Attr), or referring expression. Those can be spatial referring, where object group of objects is selected based on the relation with another object or group of objects (e.g., . the object A which is left to objects B); matching referring where an bbjects/group is selected depending or where either or one between two objects/groups of objects are selected.
matching properties with another object/group; and or
On the right of Fig. 4 generate ground-truth programs by combining attributes and referring expressions which never appeared in GQA. We emphasize the systematic shift between GQA and GQA-SGL linguistic construct by highlighting referring expressions and attributes through different colors.
We generate 50 questions for each question type shown in Fig.
13

---

Under review as a conference paper at ICLR 2023
Table 6: Supplemental results on CLEVR-COGenI. Mean and standard deviation of accuracy (%) for each question type in CLEVR-CoGenT Condition B across at least three repetitions.
Methods
Exist
Count
Transformer
87.8 + 0.69
73.3 + 0.13
Transformer w/ PR
90.4 + 0.90
76.8 = 0.99
TMN-Stack (ours)
88.4 + 0.069
75.1 + 0.27
TMN-Tree (ours)
87.8 + 0.62
74.3 + 0.83
Vector-NMN
84.4 0.35
70.4 + 0.44
MDETR
85.4
75.2
Methods
Compare Integer
Compare Attribute
Query Attribute
Transformer
88.6 + 0.41
85.8 = 0.68
73.2 + 1.7
Transformer w/ PR
89.8 + 0.45
88.3 + 1.1
76.3 + 1.6
TMN-Stack (ours)
88.1 + 0.20
86.3 + 0.22
76.6 + 0.28
TMN-Tree (ours)
89.0 + 0.66
87.8 + 0.69
75.3 + 0.75
Vector-NMN
81.3 = 0.52
78.1 = 0.79
66.5 + 1.05
MDETR
81.3
82.6
68.9
Table 7: Supplemental results on CLOSURE. Mean and standard deviation of accuracy (%) on each CLOSURE test across at least three repetitions.
Methods
and_maa_spa
or_mat
or_mat_spa
Transformer
77.0
19.1 = 0.55
25 + 11
Transformer w/ PR
77.6
39.7 19.7=10 1.9
36.4 + 5.8
TMN-Stack (ours)
98.4 0.19
77.8 1.5
72.7 + 2.6
TMN-Tree (ours)
98.5 + 0.03
89.5 + 0.62
90.1 + 0.8
Vector-NMN
86.3 2.5
91.5 0.77
88.6
MDETR
8.63
34.1
18.4
Methods
embed_spa
embed
compare mat
Transformer
97.0 1.3
55.2 4.4
65.7 2.8
62.7
Transformer w/ PR
59.7 = 2.8
78.1 + 5.0
85.7 + 4.4
74.4 6.6
TMN- Stack (ours)
98.8 = 0.00
92.7 + 0.84
77.8 1.5
72.7 + 2.6
TMN-Tree (ours)
98.7 = 0.16
92.8 = 0.42
99.0 0.03
99.3 + 0.11
Vector-NMN
98.5 + 0.13
98.7 1 0.19
98.5 0.17
98.4 1 0.3
MDETR
99.3
77.4
69.3
66.0
Table 8: Supplemental results GQA-SGL. Mean and standard deviation of accuracy (%) for four question types in GQA-SGL across at least three repetitions. We use pre-trained object detector as feature extractor.
Four question types in GQA-SGL
Methods
verify
query
choose
logical
Transformer
50.0 + 1.6
35.3 + 5.0
50.0 + 7.5
55.3 + 4.1
Transformer w/ PR
51.6 + 7.7
4..6
56.0 6.6
55.6 + 5.4
TMN-Stack (ours)
51.6 + 4.1
5.0
44.8 3.7
56.4 - 7.1
TMN- Tree (ours)
56.0 + 7.3
48.0
47.2 - 3.2
64.0 - 4.6
MMN
46.7 + 4.7
50.7 5.0
47.3 + 1.9
59.3 3.4
14

---

Under review as a conference paper at ICLR 2023
B SUPPLEMENTAL RESULTS ON CLEVR-COGENT
We report the systematic generalization performance for each question type on CLEVR-CoGenT in T All methods are trained on CLEVR-CoGenT condition and tested on CLEVR-CoGenT validation condition B.
C SUPPLEMENTAL RESULTS ON CLOSURE
We report the systematic generalization performance for each test in CLOSURE in Table 7 All methods are trained on CLEVR and tested on CLOSURE tests.
D SUPPLEMENTAL RESULTS ON GQA-SGL
We report the systematic generalization performance for each question type on GQA-SGL in Table g All methods are trained on GQA balanced split.
E LIBRARIES OF MODULES
We design three libraries of modules for TMN, "Individual", "Semantic group", and "Random group", shown in Table Unlike these libraries, module "Order" library is selected by the order of sub-task in a given program and thus in this library the modules are not specialized for specific sub-tasks.
F EFFECT OF REGIONAL FEATURES
We compare the systematic generalization performances on CLEVR-CoGenT with grid features and regional features, as shown in Table TO We use ResNet-101 and region of interest pooling to extract regional features, while we use only the same ResNet-101 extract grid features. case of the standard Transformer and our TMNs, the performance gaps less than 1.5%. In case of the standard Transformer with ground-trum truth programs, the performance gap slightly larger than (3.6%).
G EFFECT OF PRE- TRAINING WITH IMAGE-TEXT PAIRS
Table reports the results of TMN-Tree and state-of-the-art Transformer-based models (MDETR (Kamath et al. 202T) and LXMERT (Tan & Bansal 2019) on GQA and GQA-SGL. The table also indicates the pre-training datasets with image-test text pairs. The methods use different pre-training datasets, and hence, they cannot be compared in a fair manner. Yet, we observe that TMN-Tree strike balance between amount of pre-training data for the Transformer encoders and systematic generalization performance.
15

---

Under review as a conference paper at ICLR 2023
Table 9: Libraries of modules. "Individual" library has 26 modules, while "Semantic group" and "Random group" libraries have 12 modules.
Module number
Individual
Semantic group
Random group
scene
scene count
query.color
count
query-shape
2
exist
exist
union
3
intersect
intersect
same.shape
4
relate
relate
equal_shape
5
union
union
greater_than
6
unique
unique
equalmaterrill
greater.than
filter_shape
7
greater.than
less-than
intersect
equal integer
unique
equal.color
same.naterial
equal.aaterial
filter.aterril
8
less.than
equal_shape
count
equal_size
same-size
filter.color
equal.size
filter.aterrial
query.material
9
equal color
filter shape filter size
scene
equal color
query.coloz query-aaterial
query-size
less.than
10
equal_integer
query_shape
exist
query_size
relate
same color same material
Eilterssize
equal integer
11
equal material
same_shape
filter_color
same.size
same.color
12
equal_shape
13
equal-size
filter.color
15
filter material
16
Wilterrshapp
17
filter_size
18
query.color
19
query.material
20
query-shape
21
query-size
22
same-color
23
same mateeria
24
same same_shape
25
same size
Table 10: Effeett of regional features. Mean and standard deviation of accuracy (%) on CLEVR-
CoGenT validation condition B with grid features and regional features.
Methods Transformer
Grid features
Regional features
78.9 0.80
77.4 0.68
Transformer w/ PR
81.7 + 1.1
78.2 + 0.44
TMN-Stack (ours)
80.6 0.21
79.4 = 0.37
TMN-Tree (ours)
80.1 = 0.72
80.9 = 0.25
16

---

Under review as a conference paper at ICLR 2023
Table 11: Comparison with pre-trained state-of-the-art VQA methods. Mean accuracy (%) on four question types (verify, query, choose, and logical) in GQA test-dev and GQA-SGL. The pre-training refers datasets with image-text pairs used during training besides GQA. All methods except TMN- Tree are pre-trained with large amount of image-text pairs. For the methods with f. we use official trained models. VG, COCO, Fk, GQAu and VQA denote Visual Genome, MS COCO, Flickr30k, GQA unbalanced split, and VQA-v2, respectively.
Methods
Pre-training data
GQA
GQA-SGL
TMN-Tree Tree (ours)
65.2
53.7
MDETR
VG, COCO, Fk, GQAu
73.9
59.0
LXMERT
VG, COCO, VQA
68.9
58.0
Input image
Attention map
Attention map
Filter_shape
filter [atter_naterial (netal)
(sphere)
Filter_col color (yellow)
Program
countt
Classifier
ClassiFter
Answer
?
'yes'
Figure 6: Visualization of TMN's behavior with examples in CLEVR. The left program is {scene, filter shape (sphere), filter.colon (yellow), count] and the right program is (scene,
(metal), exist). Attention maps of modules corresponding to the sub-tasks are shown in grey scale.
H BEHAVIOURAL ANALYSIS OF MODULES
Figure 6 shows visualization of how Transformer modules work, obtained by using exam- ples in CLEVR, i.e., in-distribution case. In the left example, we give TMN-Stack an in- put image with program (scene, filter_shape (sphere), filter_color (yellow), count). Three modules corresponding to the program operate their sub-tasks sequentially and then classifier outputs an answer. We visualize the attention map in each module, where darker grey represents higher attention scores. We average attention scores across all atten- tion heads. We can see that fi Filter shape (sphere) and filter.coleee (yellow) mod- ules highly attend to regions with sphere and yellow objects, respectively. More precisely, the queries of the regions highly attend to the keys of sphere and yellow which are the in- for each module, and the representations of the regions change by adding the values of sphere and yellow. Although filter.color (yellow) module executes its sub-task after filter.shape (sphere), filter.color (yellow) also attends to yellow eube well as yellow spheres. In the right example, we give an another image with a program {scene, filter material (metal), exist). These results indicate that the modules learn their
17

---

Under review as a conference paper at ICLR 2023
Table 12: Effect of model size on Transformers. We compare Transformers and Transformers w/ PR with the different number of encoder layers.
The number of layers
CLEVR (In-Distribution)
CLOSURE
Methods
(Syst. Generalization)
12
97.4 + 0.23
57.4 + 1,6
Transformer
6
96.8 + 0.0068
60.6 + 3.8
3
91.8 + 0.40
56.1 2.0
12
97.1 + 0.14
64.5 = 2.5
Transformer w/ PR
6
95.7 0.40
63.5 + 2.7
functions properly. Figure 7 shows a behavioural analysis on an example in CLOSURE, i.e., novel combinations of known linguistic constructs. In this example, we give TMN-Tree an put image with a program {scene, filter-shappp (cube), scene, filter-size (large), filter_shape (cube), unique, same materiall filter.shape (sphere), union, count). scene represents the beginning of thread and this program contains two threads. TMN-Tree operates the threads individually and merges them with union (OR). For simplicity, we overlay attention maps the input image. The result show that the each module works properly as we expect even though this example much more complex than the ones shown in Fig. 6 and also the sequence of the sub-tasks are unseen in the training.
I BEHAVIOURAL ANALYSIS OF TRANSFORMER W/ PR
Figure and show how Transformer w/ PR works on the same example in CLOSURE shown in Fig. and an example in CLEVR similar to the CLOSURE example. Both examples contain union on (OR), however the CLEVR example is in-distribution while the CLOSURE example requires the model to handle novel combinations of known linguistic constructs, i.e., systematic generalization. We visualize the attention map of the head token for the visual regions and the input sequence of the tokens. The results show that the Transformer w/ PR first attends to the input program then changes where attend in the visual regions.
In Fig. 9 i.e., in-distribution case, the model attends to the token almost from left to right (i.e., order of sub-tasks) and after reaching the last token it stop attend the program. Specifically, the model first attends to the regions with metal cube from layer 2 to 3 and the regions of sphere at layer 5, then attends to the both regions at layer after attending the union on at the previous layer. Finally, the model attends to the count. This result indicates that the Transformer w/ PR learns how to operate the input programs to output the correct answers.
In Fig. g i.e., systematic generalization case, the input program contains came_naterll in addition to union. Both sub-tasks appear in the training while the combination novel. Fig. shows that the model tried to operate the input program in the same manner the CLEVR example shown in Fig. g but failed to lead a correct answer. Specifically, at the same layer as the CLEVR example, the model attends to the union and count, however the attended regions are not correct. After that the model seems to operate same material that should be operated before union. These results indicate that Transformer w/ PR struggles operate the programs that contain the novel combinations of the sub-tasks.
J ABLATION STUDIES FOR TRANSFORMERS
We test Transformers and Transformers w/ PR with the different number of encoder layers to in- vestigate the effect of model size. We train all models in the same way we described in the main paper. Table [2 shows that the performance of Transformers with fewer layers slightly drops except Transformer with 6 layers. These results show that we cannot improve the systematic performance by simply changing the number of layers.
18

---

Under review as a conference paper at ICLR 2023
filter filter_shaper shape(cube)
filter sizeelarge)
filter shape(cube)
same material
filter_stee shape (sphere)
union
count
'3'
Figure 7: Visnalization of TMN-Tree's behavior with an example in CLOSURE. The program (scene, filter.shape (cube), scene, filter.size (large), filter-shape (cube), unique, same material, filter (sphere), (stpeeer) union, count). Attention maps of modules corresponding the some sub-tasks are shown over the input image.
19

---

Under review as a conference paper at ICLR 2023
Layers
Visual attentions
Linguistic attentions
scane
lange filteer_ unique same #ilter. sphere union count shapa
shape
naterialshape shape
11
Prediction: (correct:
Figure Visualization of behavior of Transformer w/ PR with an example CLO- SURE. The program is (scene, filter.shape (cube), scene, filter.size (large), filter_shage (cube), unique, same filter.shape (sphere), union, count).
20

---

Under review as a conference paper at ICLR 2023
Visual attentions
Linguistic attentions
filter_
cube
sphere
count
2
Prediction: (correct)
Figure 9:
Visualization of behavior of Transformer w/ PR with an example in CLEVR. The program is {scene, filter (metal), filter.shape (cube), scene, filter filter_shape (sphere), union, count].
21


Reviews:
[{"review_id": "lZlpQFTggog", "reviewer": "Reviewer_pauq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on the systematic generalization visual question answering where the test set contains novel combinations of training concepts. The authors propose a new modular network whose modules are transformers.", "summary_of_the_review": "The paper proposes a new dataset. But the proposed model is not novel and its effectiveness is not fully verified. Thus I give borderline.", "strength_and_weaknesses": "Strength:\n\nThe paper focuses on the model robustness against VQA bias, which is a common problem in the VQA field.\n\nThe paper proposed a new systematic generalization dataset GQA-SGL based on the GQA. The new dataset can be a new challenge for future work in this field.\n\nWeakness:\n\nThe novelty is limited. The proposed model is a modular network composed of more expressive transformer modules. \n\nThe proposed model also suffers from the weakness of the previous modular network. It needs program supervision and relies on the performance of the question parser. Since all the experiments have a groundtruth program at test time, whether the question parser has the same systematic generalization ability is not verified.\n\nThe improvement on the CLEVR-CoGenT and CLOSURE dataset is not consistent. The proposed model performs worse than the baseline on the CLEVR-CoGenT but improves on the CLOSURE. I suppose the CLEVR-CoGenT and CLOSURE have novel images and questions respectively. However, the TMN can ignore the questions by accessing the groundtruth programs.\nThe authors suggest that the proposed model and baseline share the same visual feature extractor. Thus authors may verify this by training a new extractor from scratch similar to FiLM[1].\n\n[1] Perez E., Strub F., de Vries H., Dumoulin V., Courville, A. FiLM: Visual Reasoning with a General Conditioning Layer. AAAI 2017", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "H0PM1r2Jiy4", "reviewer": "Reviewer_o4YA", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper mainly focuses on combining the strengths of Transformer and NMN by introducing a novel NMN based on compositions of Transformer modules named Transformer Module Network (TMN). The model is evaluated on CLEVR-CoGenT, CLOSURE, and GQA-SGL.", "summary_of_the_review": "The motivation that combining the strengths of the two structures is good. However, directly substituting the modular with TRM seems not novel to me. Besides, there are no experiments to prove that the TRM encoder outperforms all previous NMN modular in the task, thus, it does not convince me of its superiority by directly substituting all NMN modulars. In addition, it lacks comparisons of the SOA methods. And from my knowledge, the performance is not SOA (the last point in the weakness section). Overall, I'm leaning to reject it.", "strength_and_weaknesses": "*[Strength]*\n\nThe motivation that combining the strength of NMN and Transformer is contributing.\n\n*[Weakness]*\n1. In the approach, the paper directly substitutes the module in the NMN with the TRM encoder. However, it lacks ablation experiments and discussion about if the TRM structure outperforms all types of module in NMN-based method. (e.g., in n2mn[1] Table 1, the attention based modules like find, relocate, filter modules and the answer based modules like describe, compare and exist modules)\n2. The Tree modular modular is not novel in NMN domain, [1] uses similar manner in Figure 3.\n3. Lacks of SOA TRM-based and NMN-based model performance comparisons. In addition, the SOA method on GQA dataset is 72.1 from [2], and there is [3] with 69.46 on GQA-dev\n\n\n\n\n[1] Hu, Ronghang, et al. \"Learning to reason: End-to-end module networks for visual question answering.\" Pr oceedings of the IEEE international conference on computer vision. 2017.\n\n[2] Nguyen, Binh X., et al. \"Coarse-to-Fine Reasoning for Visual Question Answering.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[3] Kim, Eun-Sol, et al. \"Hypergraph attention networks for multimodal learning.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "H05O1FWIWAv", "reviewer": "Reviewer_W6Kf", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors reveal that Neural Module Networks (NMNs), i.e., question-specific compositions of modules that tackle a sub-task, achieve better or similar systematic generalization performance than the conventional Transformers, even though NMNs’ modules are CNN-based. \nTo address this shortcoming of Transformers with respect to NMNs, in this paper, the authors investigate whether and how modularity can bring benefits to Transformers.\nNamely, they introduce a Transformer Module Network (TMN), a novel NMN based on compositions of Transformer modules.\n", "summary_of_the_review": "The authors provide interesting observations, and the experiments are solid enough to show the effectiveness of the proposed method.", "strength_and_weaknesses": "Strength\n- The observation about the comparison between the neural module networks and the Transformer models is novel and interesting.\n- The authors provide a comprehensive ablation study to make the final model more solid.\n\n\nWeakness\n- It would be better to add qualitative results to see the behavior of the proposed method and the existing methods. Also, it would be better to discuss why the conventional Transformer models perform worse than the neural module networks by analyzing the samples.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "7_RcyBbQg8C", "reviewer": "Reviewer_zXs6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper addresses the problem of systematic generalization in VQA. It proposes a new model that takes advantage of the learning capability of Transformers and the compositional modeling of Module Networks. The proposed model achieves state-of-the-art performance on three different VQA datasets including standard testbeds for VQA systematic generalization. ", "summary_of_the_review": "The paper is well written and easy to follow. However, it is limited in contribution to advance the systematic generalization task.", "strength_and_weaknesses": "•\tStrengths:\n\n1.\tThe proposed method is technically sound and easy to understand. It is reasonable to have a model that makes use of the known advantages of Transformers and module networks.\n\n2.\tIt shows better performance when compared against existing methods.\n\n•\tWeaknesses:\n\n1.\tWhile the proposed model is straightforward and easy to understand, it appears that the model brings very little insight into the systematic generalization task itself. It is well studied that module networks are a potential approach for systematic generalization but it is extremely brittle and ad-hoc which hinders its application in wider settings. Meanwhile, models with better representation learning such as Transformers are beneficial for all machine learning tasks, not just for systematic generalization. The proposed method does not offer any fundamental contributions toward solving systematic generalization.\n\n2.\tExperiments with original transformers do not sound reasonable as the original design of transformer is for translation tasks with having access to a more generous amount of training data. There is no reason to have to use 12 layers for transformers when applying it to a custom problem. The poor performance of transformers in the experiments might simply be due to overfitting.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject