D3: Data Diversity Design for Systematic Generalization in Visual Question Answering
Authors: Amir Rahimi
Vanessa D'Amario
Moyuru Yamada
Kentaro Takemoto
Tomotake Sasaki
Xavier Boix
Venue: Accepted by TMLR ()

Abstract: Systematic generalization is a crucial aspect of intelligence, which refers to the ability to generalize to novel tasks by combining known subtasks and concepts. One critical factor that has been shown to influence systematic generalization is the diversity of training data. However, diversity can be defined in various ways, as data have many factors of variation. A more granular understanding of how different aspects of data diversity affect systematic generalization is lacking. We present new evidence in the problem of Visual Question Answering (VQA) that reveals that the diversity of simple tasks (i.e. tasks formed by a few subtasks and concepts) plays a key role in achieving systematic generalization. This implies that it may not be essential to gather a large and varied number of complex tasks, which could be costly to obtain. We demonstrate that this result is independent of the similarity between the training and testing data and applies to well-known families of neural network architectures for VQA (i.e. monolithic architectures and neural module networks). Additionally, we observe that neural module networks leverage all forms of data diversity we evaluated, while monolithic architectures require more extensive amounts of data to do so. These findings provide a first step towards understanding the interactions between data diversity design, neural network architectures, and systematic generalization capabilities.

Published in Transactions on Machine Learning Research (07/2024)
D3: Data Diversity Design for Systematic Generalization in Visual Question Answering
Amir Rahimi Massachusetts Institute of Technology (MIT)
anhimi@@iit.cm edu
Vanessa D'Amart
wonessad@mmit.cm edu
MIT / Fujilsu Research of America, Inc.
Moyuru Yamada
gomada.morrrrr@hyytss...mm
Fujitou Limited
Kentaro Takemoto
k. t.takkmmoto@@@jjjsss.ccomm
Fujitsu Limited
Tomotake Sasaki
tomotake. hwwpotkkee.caaliiiiiiiss...o.m
Fujitsu Limited
Xavier Boix
aboizefujitsu.com com
MIT Fujilsu Research of America, Inc.
Reviewed on OpenReview: https: openreview. net/ nettteettnneeeee forum? 2msn13msOp
Abstract
Systematic gencralization is crucial aspect of intelligence, which refers to the ability to generalize to novel tasks by combining known subtasks and concepts. One critical factor that has been shown to influence systematic generalization is the diversity of training data. However, diversity can be defined in various ways, as data have many factors of variation. more granular understanding of how different aspects of data diversity affect systematic generalization is lacking. We present new evidence in the problem of Visual Question Answering (VQA) that reveals that the diversity of simple tasks (i.e. tasks formed by few subtasks and concepts) plays key role in achieving systematic generalization. This implies that it may not be essential to gather large and varied number of complex tasks, which could be costly to obtain. We demonstrate that this result is independent of the similarity between the training and testing data and applies to well-known families of neural network architectures for VQA (i.e. monolithic architectures and neural module networks). Additionally, we observe that neural module networks leverage all forms data diversity we ovaluated, while monolithic architoctures require more extensive amounts of data do so. These findings provide a first step lowards understanding the interactions between data diversity design, neural network architectures, and systematic generalization capabilities.
1 Introduction
Systematic generalization is crucial aspect of human intelligence Lake et al. (2019), allowing to solve novel lasks by combining knowledge gained from previously seen, related subtasks Bahdanau et. al. (2019b); Lake & Baroni (2018); Ruis et al. (2020). Deep learning methods have traditionally struggled to achieve systematic generalization due to their limited ability to generalize beyond the patterns and examples present in the training data, particularly in the presence of complex compositional and structural biases Bergen et al. (2021); Gontier et al. (2020); Hudson & Manning (2018); Tsarkov et al. (2021). ). In response, researchers have

---

Published in Transactions on Machine Learning Research (07 (024))
Medium-narrow
Distribution
Medium--narrow Complex-wide Simple-wide
A
Systematic generalization
Complex-wide Simple-wide
Which training distribution leads better systematic generalization performance?
Question Complexity
Training sets
In-distribution Attribute Compositions
Out-of-distribution Attribute Compositions
Count
Exist
Color
Color
Exist
Shape
In-distribution Length
Out-of-distribution Length
1-Hop
2-Hop
R2
R
R1
Attrs
Attrs
Attrs 1
Attrs 2
Attrs
E.g., Is there yellow objects behind the metal object?
B.g., How many big objects are left side of the red thing front the small yellow aube?
Figure 1: Data diversity and its impact on systematic generalization in VQA. (top) Impact of question complexity distribution on systematic generalization. In our study, question complexity is varied in two aspects: (middle) Attribute composition. In this example, the training set (in-distribution) contains Count and Exist question with different compositions (left) while Count or Exist questions have novel combinations of altributes the test set (ost-oo-distributtin))) (bottom) Length. The in-distribution question has shorter length (different syntactic structure) compared to the ont-of-distribation question.
recently developed various architectures and training regimes to improve systematic gencralization Bahdanau et al. (2019a;b); A'mario et al. (2021); Kamala et al. (2023); Yamada et al. (2023). Yet, studies prove that achieving systematic generalization remains very difficult. Ruis & Lake (2022).
Recent studics have highlighted the critical role of diversity in the training data, it has been shown to impact systematic generalization performance significantly Madan et al. (2022); Ruis & Lake (2022). Madan et al. Madan et al. (2022) demonstrated that increasing training data diversity substantially improves generalization to out-of-distribution (OOD) category-orientation combinations. Similarly, Ruis and Lake Ruis & Lake (2022) showed that augmentation training data and modularity increases systematic generalization performance substantially a natural language compositional generalization task. However, there is still a lack of study on which specific aspects of data diversity are responsible for enhancing systematic gencralization and under what conditions and how they can be applied effectively. This calls for further investigation to unravel the relationships between different types of data diversity and their impact systemic generalization.
Figure 1 illustrates motivating example where given budget of N training questions and test set in Visual Question Answering (VQA) Agrawal et al. (2018); Antol el al. (2015); Kervadec et al. (2021), training on different question complexity distributions results in different systematic generalization performances. To study the impact of question complexity on systematic generalization, we consider two factors that influence question complexity: (i) Attribute composition that focuses on specific combinations of attributes and question types during training. For instance, a model is trained questions that involve only certain combinations,
2

---

Published in Transactions on Machine Learning Research (07/2024)
such Count questions with Color attributes and Exist questions with Shape attributes. However, during testing, the model needs to generalize to novel combinations of questions and attributes, such as Exist questions with Color attributes. (ii) Length that refers to the number of reasoning steps required to answer the question'. In our study for systematic gencralization, we require a model to generalize to novel question lengths. We measure question length by the number of spatial relat ions it includes. Spatial relations refer to relations such as "behind", "left", "right", and "in front of", between entities an image. The hop notation, such as O-Hop, 1-Hop, and 2-Hop, is used to categorize questions based on the nunber of spatial relations they involve. The hop count indicates the number of reasoning steps (analogous to the number of sub-tasks) required to answer question. Higher hop counts imply more advanced spatial reasoning and compositional understanding. An example for length generalization is illustrated in the bottom part of Figure 1. In this paper; we investigate varions factors designing training data that contribute to improving systematic generalization and explore the conditions under which these factors can be effectively applied. By creating datasets in controlled settings in VQA, we analyze the relationship between different factors in training question complexity (e.g., length and compositions of object attributes) and systematic generalization performance. Our empirical analysis reveals that simple diverse questions, i.c., questions that are simple (require less reasoning steps), but cover a diverse range of attributes, are effective for systematic generalization. This can be valuable for practitioners and dataset engineers since collecting simple questions and obtaining their associated answers is more cost-effective than curating a large number of complex questions with many attributes and spatial relations. Moreover, simple questions facilitate the development of models that are less susceptible to overfitting. This is because obtaining diverse representation of attributes through unbiased sampling is considerably more manageable with simple questions as opposed to complex oncs. Our finding is in line with recent research Sorscher et al. (2022) that highlighted the inefficiency of solely relying on the neural scaling law Gordon et al. (2021); Hestness et al. (2017); Hoffmann et al. (2022); Rosenfeld et al. (2020); Zhai et al. (2022) to enhance model performance through data scaling. Instead, it has been shown that designing datasets strategically based on difficulty and dataset size can lead to greater cost efficiency without compromising performance.
2 Data Diversity Benchmark for Systematic Generalization in VQA
In this section, we describe the datasets we generate using the CLEVR Johnson et al. (2017a) repository. We define biased training questions where the biases are formed by limiting the attribute compositions and question lengths in the training sets while we test out-of-distribution questions. Our objective is to gain insights into the models' capabilities and limitations in different aspects of systematic generalization.
We propose our Data Diversity Design, referred to as D3, for each task which involves incorporating diverse set of questions to enhance systomatic gencralization. By doing so, our methodology helps mitigate the impact of biases in the training data, thereby allowing VQA models to generalize effectively to previously unseen question attribute compositions and lengths.
We begin by introducing biases in simple questions about the composition comparison of set of objects using two attributes, focusing compositional generalization. Next, we incorporate more complex questions regarding the spatial relations between multiple sets of objects to incorporate length generalization and explore various aspects of systematic generalization. Finally, we explore the impact of question complexity distribution as an additional aspect of diversity on systematic generalization when full diversity is present.
2.1 Attribute Composition Generalization
Here, we provide detailed description of two biased training sets, sets used for applying D3 (also called D3 sets), and systematic generalization test sets for evaluating attribute composition generalization.
Base training set for compositions of two attributes. This dataset comprises simple elemental questions involving a composition of two attributes as the biased in-distribution (InD) set for training. The images have been generated in a way that three to five objects are present in each image, thereby resulting in a
1 Not be confused with the number words a question in this context
3

---

Published in Transactions on Machine Learning Research (07 (0244
Composition of attributes
Training
Test
Color & Size
Color & Shape
Size
Color & Shape
How many big objects the image?
there big object in the image?
b) Attribute comparison
Training
Test
Equal Size
Equel Color
Equal Size
Equal Color
Material
Size
Color
Shape
Equal Material
Equal Shape
Equal Material
Equal Shape
Shape
Color
Size
Material
Does rubber object the size the shiny object the image?
Does object have the size the green object the image?
c) 2-Hop
Type Compositions
Type Compositions
Test
Color or Size or Left Front
Color or Size Left or Front
Material or Shape or Laft or Front
Color or Size or Right or Behind
Exist
Count
Color Shape or Left Behind
Color Shape or Right Front
Material or Shape or Right Bahind
Material or Shape or Right or Behind
A: How many objects are on the left side of the red thing in front of the small yellow cube?
Is there cube the left side the shiny thing front of the cylinder?
is there big object on the left side of the red thing in front of the
yellow cube?
Specification of different biases for train and test datasets. The gray boxes display the question type, while the valid attributes corresponding each question type shown in white boxes below. A valid example question for each dataset is shown below the boxes.
Figure 2:
simplified dataset. We have used the original O-Hop template from the CLEVR dataset to generate InD and OOD questions for this dataset. Additionally, we have imposed constraints on the types of questions to specify the biased train and test splits.
We have limited the pairs of attributes and question types appearing together in this dataset to only Count and Exist question types, as shown in Figure 2a. Specifically, Count questions are only associated with Color and Size related attributes, while Exist questions are employed only when Color and Shape attributes are present in the questions. For instance, the question "How many big red objects InD question for this dataset as it contains two attributes relating to Color and Size for the Count question type. However, the question "How many rubber objects are there?" will never appear during training in this dataset.
in the image 8 is valid
Test set for composition of two attributes. The systematic generalization test set for this dataset contains questions where the attributes associated with Count and Exist questions are swapped with respect Inl) questions. For instance, in the OOD test set, Color and Size attributes are used in Exist type questions. By designing this test set, understand the elemental attributes and their relationships to different question types.
evaluate if the neural networks have learned to identify and
Base training set for attribute comparison. The dataset for attribute comparison includes four question types: oqual_size, equal_shape,
and equal_ color. Similar to the composition of two attributes dataset, the images are generated such that three to five objects are present in each image. For each question type, a specific attribute of the same type between two different objects is compared. Diffcrent attribute and question lype combinations for the training set of this dataset is shown in Figure 2b. As an example, equal_size size type questions always filter Materials of two objects in input image and compare if they have the same Size. In this case, question like "Does the rubber object have the same size as the shing object in the image?" is considered a valid question. Note that for simplicity, there is no combination of attributes of objects in these lype of questions.
qqual_color.
Test set for attribute comparison. The OOD test set for this datasct constructed in a way that the comparison questions appear with different attributes from the training as shown in Figure 2b. For example, the equal_siiie size questions are about the Color attributes of the objects in the test set..
4

---

Published in Transactions on Machine Learning Research (07/2024)
Applying D3. To apply D3, utilize O-Hop questions that have no spatial relations with constraint that questions from this set contain only single attribute. Our diversified training sets after D3 contain 70% of the original biased questions and 30% of the questions from the respective D3 set. While we have observed promising results with the 70% and 30% configuration, we have also explored other proportions, which will be detailed in Appendix B. Our goal is to investigate the impact of incorporating such simple questions on systematic generalization. We use questions about Color, Size, and Shape attributes for both Exist and Count questions. As an example, the question "Is there a shiny object?" is a valid question for this set.
2.2 Incorporating Length Generalization
To analyze the interplay between different aspects of diversity for systematic generalization, we introduce a set of datasets with varying compositions of attributes and question lengths. The biased training set is based on the 2-Hop template, which includes questions with two spatial relations. This is a more complex setting where spatial relations between objects and attributes come into play and allows us to study how the complexity of the questions and the diversity of attributes can affect the model's ability to generalize systematically. By exploring different combinations of attributes and question lengths, we can gain insights into which factors contribute most to achieving high levels of systematic generalization in VQA models. The images from the original CoGenTT A and CoGenT B splits of CLEVR are used for the training and test sets, respectively. We only consider Count and Exist question types for these sets. Different variations of the sets are illustrated in Figure 2c. Next, we begin by introducing the biased training set, OOD test sets, and how we apply D3.
Base training set for two spatial relations between multiple objects (2-Hop). The biased training set for this datasct is named 2-Hop A, in which only Color, Size, Left, and Front attributes are allowed for Count type questions, and Material, Shape, Right, and Behind altributes are used for Exist quest.ions
Test sets for 2-Hop. We introduce four different test sets to evaluate attribute composition generalization, length generalization, and their combinations using the 2-Hop dataset: - O-Hop: The O-Hop test set is composed of questions regarding a single attribule. It contains all possible combinations of attributes and question types (i.c., Count and Exist questions). We employ this test set to assess the models' ability to generalize to questions with shorter lengths.
2-Hop OOD: This set contains the same set of combinations for both Exist and Count question types. The attribute and spatial relation combinations are carefully chosen such that the possible combinations have an edit distance of two with attribute combinations in 2-Hop A. For example, Material, Shape, Left, and Front can appear with both Count and Exist questions. This set is solely used for evaluating altribute composition generalization on the more complex 2-Hop questions than the previously introduced attribute composition test sots
3-Hop A: This test set comprises questions with three spatial relations and uses only type A altribute compositions. The sole purpose of this test set is to evaluate the network's ability to generalize to longer questions.
H-Hop OOD: This test set is similar to 2-Hop 00D in terms of attribute and question combinations. However, the questions contain three spatial relations (3-Hop), , making it more challenging set that tests both OOD length and OOD attribute composition generalization.
Applying D3. The sets for applying D3 consist of various combinations of attributes and spatial relations, enabling us to systematically analyze the model's ability to generalize to different levels of complexity. As before, we replace 30% of the base training questions with questions from one of the sets below to crcate a new diversified training set:
I-Hop Full: This set includes questions that have only one spatial relations between two objects, hence called 1 H-Hop Full. There is no combinations of attributes for a single set of objects in this D3 set. We do not impose any constraint on the combinations of questions and altribule types for the H-Hop Full set. This set is analogous to the diverse simple D3 set of questions that we used for attribute comparison and composition sets for questions with spatial relations.
H-Hop A: The 1 H-Hop set contains the same combinations of questions and attributes as the 2-Hop A training set, but with only one spatial relation. This allows to study the model's ability to generalize to
5

---

Published in Transactions on Machine Learning Research (07/2024)
Table 1: Accurecy and standard deviations comparing training on the biased dafasets and training after with O-Hop. 30% of the trasning questions from biased datasets are replaced with O-Hop questions for D3.
Training dataset
FiLM
MAC
VectorNMN GT VectorNMNSepStem GT VectorNMN
Two attributes
33.57 + 0.03
46.56 + 0.02
45.39 + 0.01
44.10 + 0.01
42.80 + 0.01
D3 (O-Hop)
64.21 + 0.03
71.09 + 0.04
66.72 + 0.01
64.88 + 0.01
60.15 + 0.01
Comparisons
19.28 + 0.01
53.31 + 0.01
66.27 + 0.04
67.97 + 0.04
58.43 + 0.04
+ D3 (O-Hop)
56.29 0.03 56.58 0.03
83.29 0.02
76.16 + 0.04
73.32 + 0.02
longer questions and analyze its similarity to the test distribution.
1-Hop B: This set is similar to 1-Hop A except that the attribute and spatial relations of Count and Exist Specifically, in 1-HopB, Exist questions can only be paired with Color, Size, Left, and Front attributes. This set serves the same purpose as 1-Hop A, allowing us analyze the model's similarity to the test distribution.
are swapped from the ones in 1-Hop
3-Hop Full: This set comprises 3-Hop questions with no restrictions the combinations of question types and attributes. The main purpose of this set is to evaluate the model's ability to generalize to questions with shorter lengths than the ones seen during training.
2.3 Question Complexity Distribution
In order to explore the impact of question complexity distribution, we introduce the datasets with full diversity. These datasets allow us to examine the systemalic generalization performance by sampling quest from various question types in different proportions, while maintaining a fixed number of questions.
Training sets. We sample different fractions of training questions from O-Hop A, 1-Hop Full, and 2 2-Hop A training sets. 1-Hop Full and 2-Hop A are the same sets as the ones we defined in Section 2.2. The O-Hop A set is similar to 2-Hop introduced in Section 2.2 and Figure 2c except that it does not contain spatial relations. In other words, the Count questions appear with Color Size attributes, and Exist questions appear with Material O-Hop A set. A valid question for this set would be "Is there matte object in the image?"
Shape attributes. Only single attribute can be used in the questions of the
Test sets for question complexity distribution. The lest sets for these wide datasets are identical to the test sets we defined for the 2-Hop dataset in Section 2.2.
Applying D3. For simplicity, we sample different fractions from each training set using only multiples of 1/6 as fraction of questions sampled from each training set. We generated 13 and 10 different sets of fractions for 100k and 600k questions, respectively.
3 Results
In this section, we provide our results and analysis about the effect of dataset diversity systematic general- ization. Our implementation is based on the original implementation of Bahdanau et al. (2019a)². We used MAC Hudson & Manning (2018), FiLM Perez el al. (2018), Vectorized Neural Module Networks Andreas el al. (2016); Bahdanau et al. (2019a) (VectorNMNN) its counterpart with given program generator (VectorNMN GT), and the version introduced in D'Amaoio et al. (2021) with modular image encoders (VetooNMMNSepSeem) To save time and computational resources, we excluded FiLM and VecLorNMMSSpSSeen from our experiments about question complexity distributions due to their inferior performance compared to MAC and VectorNMN. We conducted hyperparameter search using the attribute comparison and composition datasets and identified set of effective hyperparameters that yielded consistent results. Further implementation details and infor- mation on the hyperparameter search can be found in Appendix A. To generate InD and OOD objects in the images, we rely respectively on the CoGen'T condition and CoGen'T condition B splits in CLEVR.
3.1 Effects of Introducing Diversity in Training Data on Systematic Generalization
The results for composition of two attributes and attribute comparison datasets (Section 2.1) are shown in Table 1. We also present the full results for the 2-Hop datasets (Section 2.2) in the form of a heatmap
6

---

Published in Transactions on Machine Learning Research (07 (07/2024)
Test Set
VictorkMMtSapStem
Vector MON
Hlpp Roo
2-lop Rob
Hop Cop
Wap Colp
Srep oog
Arip
R/hpp
21
8.6
-11
-0.5
7
10
-5%
10.0
11
-31
D3 Set for Training
14
3:3
15.5
5.2
11.4
13.3
10.7
8.6
12.5
18.5
17:1
15.2
2olep TULL
10.2
19.7
13.5
14.7
2.2
3
14.7
15.9
23.8
32.3
15.6
Figure 3: Change in accuracy after replacing 30% of base training questions (2-Hop A) with different type of questions as D3 have more diversity. most cases diversity helps systematic generalization significantly.
displayed in Figure 3. The heatmap shows the change in accuracy after applying the respective D3 compared to training the base HoHop dataset. The results the base 2-Hop A dataset are shown in Table 7 in Appendix E. Additinoally, we present the results for a transformer model in Table 5 in Appendix E. As expected, our observations indicate that diversity plays a crucial role in enhancing systematic generalization across various scenarios and D3 with maximum diversity leads to better systematic generalization. However, we note that when the diversity aligns more closely with the specific aspect being tested, we observe improved systematic generalization performance. The key findings are detailed in the following:
D3 with diverse simple questions improves both compositional and length generalization. The results for compositions of two attributes and attribute comparisons presented in Table show that using diverse simple questions of O-Hop type with only a single attribute as D3 can significantly improve attribute composition generalization for both our simple datasets. To specifically examine the impact of diverse simple questions within the D3 framework, we analyze the results of the 2-Hop OOD test set, displayed in the second column of Figure 3, for the respective architectures. Notably, we observe noteworthy improvements in systematic generalization performance when considering both the 1-Hop Full and 1-Hop sets, which demonstrate diversity in attribute composition aspect.
We employ 3-Hop test set to isolate the impact of data diversity on length generalization. This test set maintains the same attribute composition the biased training set 2-Hop A, but with longer questions. Re- markably, we found that introducing diversity through shorter questions with the same attribute compositions, as seen in 1-Hop A, improves OOD generalization ion by incorporating more variations in length. Additionally, we observe that including H-Hop B slightly improves the results. It shows that although it introduces more diversity in attribute compositions along adding diversity in length, the networks may learn to associate questions with type B attribute compositions with shorter questions, making more challenging to generalize to longer questions. On the other hand, since 1-Hop Full contains all forms of compositions and shorter question lengths altogether, the networks learn better strategy to generalize to longer questions.
To examine the influence of length and attribute composition together, we utilize the 3-Hop OOD and O-Hop test sets. In the case of 3-Hop 0OD, D3 with H-Hop A results in a slight decrease in accuracy across all architectures, except for MAC, which demonstrates an improvement leads a significant increase in accuracy. This highlights the importance of diversity both length and attribute composition, as 1-Hop exhibits such diversity while 1-Hop only diversifies in terms of length. Furthermore, in line with previous observations, 1-Hop Full outperforms the other D3 sets on the 3-Hop OOD test set, indicating superior generalization capabilities when full compositional diversity is available. We obscrve that all forms of D3 lead to improvements on the O-Hop datasct (with exclusion of VectorNMN when D3 uses 1-Hop A). For instance, incorporating H-Hop A D3, which divensifies only the length aspect of the questions compared to the base training of 2-Hop A, results noticeable improvements. On the other hand, D3 with 1-Hop B, which is more diverse in terms of both length and attribute composition, outperforms D3 with 1-Hop A. The best performance is achieved by H-Hop Full as it is closer in length the test set and contains full attribute composition diversity. Despite 3-Hop Full having longer question length than
5.7%. Conversely, D3 with 1-Hop B
7

---

Published in Transactions on Machine Learning Research (07/2024)
the base training and thus further distance to the O-Hop test set in terms of length, it still exhibits full attribute diversity and yields significant improvements over the baseline. In conclusion, the diversity of both length and attribute compositions proves beneficial for the O-Hop test set.
D3 with maximum attribute composition diversity leads to better results for attribute com- position generalization. Comparing the performance of 1-Hop and 1-Hop Full, we observe a notable difference when it comes to having the same attribule compositions as the base 2-Hop A training set. Specifically, 1-Hop Full, which shares the attribute compositions with 2-Hop A, demonstrates superior performance. The reason is that H-Hop B introduces swap in attribute compositions for Count and Exist questions compared to 2-Hop A. While this diversification contributes to increased variability in the dataset, it causes the model to struggle to generalize effectively to new attribute compositions, leading to a slightly lower performance compared to 1-Hop Full. Conversely, when focusing on D3 with 1-Hop A, which solcly introduces diversity in terms of question length while keeping attribute compositions coustant, we observe minimal changes in performance.
D3 enhances systematic generalization without getting closer to the target distribution. Since diversity provides a better generalization to OOD questions, one may ask if by applying D3, the training set is getting closer to the target OOD distribution. To answer this question, we provide the following observations:
Shorter questions facilitate length generalization on longer questions, and vice versa: We observe that any form of D3 utilizing Ho variants of questions leads to improved results on H-Hop A, which contains longer questions compared to the ones in the D3 set. Similarly, employing 3-Hop Full D3, which consists of longer questions than the base 2-Hop DOD training set, enhances accuracy on the O-Hop test set, which comprises shorter questions.
D3 with questions that exhibit more diversily lend to yield beller resulls while mainlaining the same dislance in terms of aftribute composition to the OOD test set: This can be observed when comparing D3 with 1 H-Hop A and H-Hop B. Both D3 sets have the same attribute compositions, but they differ in their question types (only Exist and Count questions have been swapped). Consequently, their distance to the 2-Hop 0OD Lest set remains constant since the 2-Hop O0D test set encompasses identical attribute compositions for both Exist and Count questions. However, the resulting set after D3 with H-Hop exhibits more diversity and gains better accuracy.
These observations highlight an important aspect of D3 training: it does not necessarily bring the training data distribution closer to the target test distribution, yet it can still improve systematic generalization.
We also note that it is crucial to consider the alignment between the diversity introduced in the training data and the characteristics of the target test set to obtain optimal performance. For instance, when evaluating the performance on the O-Hop lest set, we observed that the 3-Hop Full D3 set, which consists of longer questions compared to the base HoH training set, results in worse accuracy compared to the 1-Hop Full D3 set. This is because the questions with longer length in the 3-Hop Full set created a greater distance between the training and test distributions in terms of question length, affecting the generalization performance. In contrast, the 1 H-Hop Full D3 set, which encompassed a more similar question length distribution to the target test set, exhibited better accuracy on the O-Hop test set.
Qualitative results. Qualitative results of MAC trained on 1 H-Hop variants of D3 on 2-Hop OOD test set are shown in Figure 4. Consider the top-left question-image pair. The question has the type of Count and it contains Color, Behind, Size, and Right attributes. As we can see in Figure 2, Right and Behind attributes are always present in Exist type of questions Type datasets. This has fooled the models trained on 2-Hop and +D3(1-Hop A) in determining that the question has actually Count type resulting in wrong answers. On the other hand, models trained on +D-HH- B) and +D3(1-Hop FULL) are able to correctly answer the question. We computed the number of questions that the base dataset 2-Hop answers number to questions of type Exist as well as Yes/No answers to Count questions. 2-Hop A makes such mistakes in determining the type of questions in about 48% of the time, while these type of mistakes for +D3 (1-Hop FULL) is reduced to 9%. These type of mistakes also happen in some other image-question pairs in Figure 2. As an example, in the top right image-question pair, 2-Hop A made a mistake in determining the type of question although the number it provided (0) is correct. The bottom right pair shows failure case for all of the datasets.
8

---

Published in Transactions on Machine Learning Research (07 2024)
or Ther is parple thing behind the large parple chings what number of tiny Blue tzings are to the zight ot 1ct Asswee:
Are there any big balls that are In front the large thing in Exont of the Large eylinder? Anave::
prede (2-Hop A, 11-Rop A, +1-Hop B, +1-top (Pul1)); yeu
prade (2-llpp A, +1-Mop N, +2-Hop B, +1-Hop (Pol1)) no no
oyan wany thingg
Frade (2-llop 11-Rop +1-Hop 41-tlop (Pull)):
Peede (2-llop 41-Hop +1-Rop +1-Rop (Pul1)):
Q: lew there any objects bahind the cylinder on the left: side of the yollow objocs? Ansieet you
0: that number of yellow robber abflccta are in front of the objoct that is behind the sed actall thing? Answee:
Preda (2-Hop A, 11-Bop A, 11-2op +1-Hop (Pull)) you yeu you
Peeda (2-Ilpp A, 42-dtop #1-Hap a, +1-Rop (Pull))s
objeett
gray objectt
ball?
Breds (2-Htap +1-Mop +1-Moo
+1-Htop (Pull)):
Feeds (2-lttp #1-Htop +1-Mop
+1-Hop (Pull)):
Figure 4: Qualitative results. The questions and ground truth answers displayed in the top line above each image, while the predictions of MAC using 2-Hop A, D3(+1-Hop A), D3(+1-Hop B), D3(+1-Hop (Full))) are presented from left right in the second line above each image.

---

Published in Transactions on Machine Learning Research (07/ (224))
3.2 Additional Datasets and Models
So far, we have performed our experiments CLEVR dataset using traditional VQA models. The reason for focusing on datasets in controlled settings is that they better ensure that the systematic generalization accuracy is not due to shortcuts in the dataset. It is unclear what biases and shortcuts are present in unconstrained datasets, and hence, unconstrained datasets are limited for gaining an understanding of the systematic generalization capabilities of neural networks. Yet, we reoognize the necessity for additional analysis of the contribution of D3 and its application to more unconstrained datasets and srate-of-the-att models. In the following we provide additional results on non-synthetic datasets. Furthermore, we provide supplementary results on models based on Large Language Models (TLM) that are significantly larger than the traditional models we have experimented with.
TallyQA dataset. TallyQA Acharya et al. (2019) is an open-ended counting dataset for VQA that contains questions generated by humans on real images. We created two splits using the TallyQA training set to experiment with both compositional and length generalization:
TallyQA-Nouns. Our goal in generating the TallyQA-Nouns dataset was to utilize questions containing a certain number of nouns for training, while incorporating diversity in the number of nouns within the questions. We aimed to test for OoD number of nouns in the questions. To identify nouns, employed off-the-shelf part-of-speech tagging Honnibal et al. (2020). Before performing part-of-specch tagging, we preprocessed the questions by removing repetitive phrases such as "In how many of these screenshots..." or "in the photo?" from the beginning or end, as these phrases may introduce additional nouns. We curated two biased datasets: 2Nouns, comprising about 62,000 questions containing two nouns, and Nouns, consisting of about 17,000 questions with three or more nouns. Additionally, we included a set of questions with only one noun, referred to as 1Noun, created additional split using Nouns
as the D3 set for the biased datasets. Furthermore, as the D3 set for the biased 2Nouns dataset. The evaluation was conducted the full test set of the TallyQA datasel, and the results of the MAC model are presented in Table 2. The results demonstrate that incorporating D3 can enhance performance on real-world, in-the-wild questions across all sconarios. Specifically, we observe a 50% relative improvement for 3 Nouns datascts. By comparing the performance of the models trained with 2Nouns + D3 (1Noun) and 2Nouns D3 D3+ Nouns), we validate our hypothesis that simpler questions lead to better generalization within this dataset as well.
2. TallyQA-Cars. The goal of this split is to assess the combination of length and compositional generalization. Below are the specifications of the splits:
Cars-InD: All InD questions are derived from the 2Noun set defined above, with the constraint that questions containing a color word (such as "white", "pink", or "black") are included only if the word "car(s)" " is also present in the question. This ensures that questions about colors are only included when they are associated with cars. All other 2Noun questions that do not. contain any color or car are included in this set.
Cars-D3: The D3 set consists of questions that are 1Noun questions and contain color word. Cars-DOD: The OOD set comprises questions with NoNns that contain color words but not in conjunction with cars.
This dataset is designed to evaluate whether D3 enhances the composition of colors with words other than car/cars in longer questions encountered during training. The results of the MAC evaluation are presented in Table 3. In alignment with previous results, we observe that D3 enhances compositional and length generalization on this split.
GQA Dataset and GPT-2 style model. We devised the following experiment for length generalization on the GQA dataset Hudson Manning (2019). We mcasure length as the number of semantic operations for each question provided in the dataset. We observed that about 47% of the questions in the balanced training set contain three semantic operations. We used these questions as our biased base training set. About 28% of
10

---

Published in Transactions on Machine Learning Research (07 (2742)
Table 2: Accuracy (%) of MAC on TallyQA-Nouns.
Table 3: Accuracy (%) of MAC TallyQA-Cars
Dataset
Test Accuracy
2Nouns
40.67
Datasct
Cars-OOD Accuracy
3 NNuns
22.88
Cars InD
63.94
2Nouns D3 (1Noun)
43.63
+ D3
69.47
2Nouns + D3 Nouns)
40.91
3 Nouns + D3 (1Noum)
34.36
the questions have two semantic operations. We call this the set of short questions. The remaining 25% of the questions contain four to nine semantic operations. We call this the set of long questions. We constructed two different D3 sets by replacing 30% of base questions with short or long questions. Our constructed training sets have total of 450k questions. We trained MAC and GPT-2 style transformer with an additional fixed vision encoder (Resnel-101) on this dataset and lested on out-of-distribution length questions. Results confirm the previous observations. Both MAC and transformer based models got a boost in accuracy when the D3 set contained shorter questions and tested on unseen long questions as shown in Table In addition, report the results on GPT-2 Radford et al. (2019) style transformer, trained from scratch, with fixed 2-Hop datasets in Table All of the results align well with our findings on the CLEVR dataset and traditional models.
vision encoder (pretrained Resnet 101)
Table 4: Change in accurocy (%) for length generalization GQA dataset.
Model, Train Test
Short
Long
MAC, D3 (+short)
+1.91
MAC, D3 (+long)
+0.2
GPT-2, D3 (+short)
+1.35
GPT-2, D3 (+long)
+1.63
Table 5: Results of GPT-2 style transformer 2-Hop datasets.
Train set
O-Hoop
2-Hopp OOD
3-H A
H-H OOD
2-Hop A
12.9
23.7
45.2
20.6
D3 (1-Hop FULL (30%))
29.1
59.7
45.0
44.6
Improvement
+16.2
+36.0
-0.2
+24.0
MiniGPT-vv. MiniGPT-v2 Chen et al. (2023) is a vision language model that aligns a frozen visual encoder, such ViT Dosovitskiy al. (2021), with a Large Language Model (LLM), such as Llama-2 Touvron al. (2023), using linear projection layer. We utilized Llama-2 Chat 7b as the language model in all the experiments. Initially, evaluated the off-tho-shelf trained model chockpoint of MiniGPT-v2 (after stage 3) our 2-Hop OOD dataset. Interestingly, MiniGPT-v2 only achieved accuracy of 34.9%, despite being trained on an extensive number of VQA datasets. Subsequently, we followed their exact instructions to finc-tune MiniGPT-v2 on our 2-Hop datascts. The changes in accuracy results after applying various D3 sets are illustrated in Figure 5. The base results of training on 2 2-Hop A are also presented in Table 7 Appendix F. These results also align with findings from other models. D3 enhances systematic generalization, and diversity plays an important role for MiniGPT-v2. Once again, we observe that in most cases, D)3 improves the results, with D3 combined with 1-Hop FULL achieving the best performance.
3.3 Effect of Question Complexity Distribution on Systematic Generalization
We have shown that using as much diversity possible beneficial for systematic generalization. However, the impact of question complexity distribution, specifically the question sampling strategy, remains unknown. The
11

---

Published in Transactions on Machine Learning Research (07 (07/2024)
MiniGPT-v2
2-Hop OOD
3-Hop o0D
H-H
3-Hop A
1-Hop A
2.3
0.0
1.0
1.0
10
Change in ACC (%)
1-Hop B
7.3
4.2
-0.1
3.3
1-Hop
12.8
4.8
1:0
4.5
-10
H-Hop Full
9.0
4.6
-15
Figure Change in accuracy after applying D3 on 2-Hop dataset using the MiniGPT-v2 model.
conventional method of uniformly sampling questions from different levels of complexity may be suboptimal for systematic gencralization. In the following set of experiments, our goal is to identify the impact question
complexity distribution, the number of training questions, and the behaviour of different neural network architectures in low and large data regimes on systematic generalization.
MAC
VectorNMN
VectorNMN GT
O-Hop Full
2-Hop 00D
3-Hop A
3-Hop 000
O-Hop Full
2-Hop DOD
3-Hpp A
3-Hop GOD
o-Hop Full
2-Hop 00D
3-Hop A
3-Hop DOD
4VG
AVG
AVG
Practions: 333 0.5. 0.187]
100
34
44
82.5
74
85
74
79
77.3
61.2
10.5
60.0
67.5
Tractions: 367 0.5 0,333
3
35.0
80.1
66.7
Tractions: (D 167 333 0,5
4
44.8
62.2
39.0
49.9
80.
63.8
67.7
53
66.3
Practions: (0 333 0.367 0,5
5
46.8
42.8
11.5
35.8
44.2
79.7
60.6
63.0
91.
63.6
79..7
86
67
75.2
ACCURACY (%)
36
45
62.8
84.00
72.6
89
73.1
7..
Fractions: 0.3334 0.33331
00.1
88
75.1
79.7
Fractins: 10.5 0. 0.5)
8
6770
47.4
44
63.1
Factions: 16. 0.5)
9
32.9
00.0
65.1
43.3
48.0
39
53.00
66.0
46
50.9
75.8
34.7
36.7
35.0
45.5
52.4
44.5
42.3
44.5
53.44
82.9
68.7
59.6
59.2
65.1
Fractien::
30.3
31.4
28.1
39.0
38.6
40.7
Fractien:: [L. 0.0.]
45.5
39:4
25.2
39.9
37.5
Fraction:: [0.1.00
13
40.3
31.6
25.9
33:3
40.9
31.4
35.0
35.2
1+Top Full
2-Hop
Figure 6: Training complexity distribution effect. The number of questions kept fixed and is equal to 100k. Fach corresponds to different training set where different fractions of questions from O-Hop A, 1-Hop Full, and 2-Hopp A are selected. For each model, the accuracies on the respective test set are shown each column. The average the four test also shown in the AVG column.
12

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
MAC
80
Accuracy (%)
FiLM
VectorNMN
60
VectorNMN GT
40
101
102
10
Number of training questions x1000
Figure 7: Average accuracy of the four test by varying the number of questions using fractions from dataset 3 in Figure 6.
The results obtained from the experiments conducted with different fractions of diversity presented in Figure 6 for 100k questions. To provide summary of the findings, we calculate the average result across the four test sets for each model and display it in the AVG column.
VectorNMN has superior performance and is more robust to distribution changes in low data regimes. We observe that when limited number training questions are available, MAC performs poorly except for dataset number and shows high sensitivity distribution changes. VectorNMN on the other hand has more consistent results and achieves an overall better performance. The results can be further improved when program generator (VectorNMN GT) is provided.
MAC gains strong systematic generalization performance and robustness to distribution changes when sufficient data is available. To assess the impact of question complexity distribution in large data regimes, we identified the best set of fractions (dataset 3 in Figure 6) and progressively increased the number of questions until reaching saturation accuracy, as depicted in Figure 7. We determined that employing 600k questions would suffice for this dataset to observe the behavior of networks when using large amount of data. Compared to low data case, the patterns alter when expanding the number of questions to 600k, depicted in Figure 8. MAC becomes more robust to distribution changes the amount of data increases and outperforms VectorNMN.
Program generator plays an important. role in systematic generalization. The gap between VectorNMN and VectorNMN GT also increases as the number of training questions grows, indicating that the performance bottleneck of modular networks lies in learning the program generator, specifically in large data scenarios.
Using abundant simple questions can lead strong performance. We also note that the inclusion of distribution with more simple questions proves to be valuable in both data regimes, it results in improved systematic generalization performance. Dataset 1 in Figure 6 and dataset 7 in Figure 8 show that it is not necessary to sample large amount of complex questions to achieve high systematic generalization performance.
Uniform sampling from different complexities can be sub-optimal. MAC's performance on the uniform dataset (7th Figure and 6th in Figure 8) is relatively lower compared to the best results achieved. In contrast, modular networks show similar performance to other full distributions when using the uniform distribution.
4 Discussion
Summary. We introduced Data Diversity Design (D3) for designing datascts with improved systematic generalization performance in VQA. We showed that diverse simple tasks siguificantly enhanced systematic generalization in VQA. We demonstrated that the inclusion of diverse questions, particularly those aligned with the particular systematic generalization aspects, led better overall performance. Additionally, we investigated the impact of different sampling strategies from different question complexities and noted that distributions with enough simple and diverse questions attain better systematic generalization results
13

---

Published in Transactions on Machine Learning Research (07/2024)
MAC
VectorNMM
VectorNMN GT
D-Hop Full
H-Hop 000
3.Hop
3-Hop 000
O-Hop Rull
HoHop 0OD
3-Hop
3-Hop 000
O-Hop Full
H-Hop 000
H-Hop A
3-Hop 000
AVG
AVG
AVG
Factions: 10 0.354 6.856]
100
1
93
60
75.5
29.0
70.2
48.2
98.0
85.0
Fraciooe: 10
0.565 6.534)
2
73
82
98
82
64.
37.3
71.3
59.5
32.0
35.2
82.6
80.7
98.4
81
85.7
3
96
02
87.0
00
66.0
98.9
92.
88.4
80.5
98.0
80
97.6
ACCURACY (96)
Praciions: 167 666 1671
60
5
69.4
54..4
688.5
98.5
877
Factions: to 9 3334 3333]
6
84
97
76.9
87.9
65.8
99.2
93.6
66.5
86.6
79.5
98.0
79.8
86.00
Fradions: (0 354 8.5 6.167)
40
7
89
78
98.5
74.7
65.3
87:
68.5
97.6
95.3
67.1
67.3
80.11
97.5
60.2
866.3
Prection:: 10
40
40.3
45.6
68.4
43.7
42.0
44.1
20
85.0
Fractone: 334
10
85
33
36.0
32
06
39.7
36.0
40.6
50.7
56.0
62.8
0-Hop
1-Hop
2-Hop A
Figure 8: Training complexity distribution effect for 600k questions. Refer to Figure for more details.
than the common uniform sampling practice. Our results showed that modular networks such as VectorNMN are more data efficient and more robust to question complexity distribution variations. On the other hand, the monolithic networks attain their best performance and robustness to distribution changes when sufficient. data is available.
Limitations. While large language models Brown et al. (2020); Chowdhery et al. (2022); Taylor et al. (2022) have shown promise in addressing systematic generalization Anil et al. (2022); Wei et al. (2022a;b), their training data biases are not well-understood. This motivated us focus on datasets created controlled settings isolate and investigate specific factors related data diversity and systematic generalization. However, our studies may not fully capture the complexity and variability of real-world data.
While have observed that diversity contributes to improved systematic generalization, further investigation is needed to fully understand the underlying reasons behind this effect. Our hypothesis is that diversity promotes modularity in neural networks, which in turn enhances systematic generalization. To provide additional insights into the modularity of networks, we present an experiment in Appendix C. Lastly, the evaluation is primarily basod on the VQA task, and it would be valuable to investigate the transfcrability of the observed effects to other domains and tasks.
References
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In AAAI, 2019.
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. Don't just assume; look and answer: Overcoming priors for visual question answering. Proc. of the 31st IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2018), pp. 1971-4980 2018.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 39 48, 2016.
Cem Anil, Yuhuai Wu, Anders Johan Andreassen, Aitor Lewkowyez, Vedant Misra, Vinay Venkatesh Ramasesh, Ambrose Slone, Guy Gur-Ari, Ethan Dyer, and Behnam Neyshabur. Exploring length generalization in large language models. In Advances in Neural Information Processing Systems, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. VQA: Visual Question Answering. In Proc. of the 15th IEEE International Conference on Computer Vision (ICCV 2015), 2015.
14

---

Published in Transactions on Machine Learning Research (07/2024)
Dzmitry Bahdanau, Harm de Vries, Timothy O' O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. CLOSURE: Assessing systematic generalization of CLEVR models. orviv preprinl arXiv:1912.05783, 2019a.
Damitry Bahdanau, Shikhar Murty, Michael Noukhovitch, Thien Hun Nguyen, Harm de Vries, and Aaron Courville. Systematic generalization: What is required and can it be learned? In International Conference on Learning Representalions, 2019b. URL cestps://openeee
Leon Bergen, Timothy O'Donnell, and Dzmitry Bahdanau. Systematic generalization with edge transformers. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), pp- 1390- 14000,142, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Tlya Sutskever, and Dario Amodei. Language models few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL pttts::/proceedingg http:///00000111/444cccdddrerr//
neurips.cc/pep
Paperr .ddf.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechu Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny, Minigpt-v2: large language model unified interface for vision-language multi-task learning. arXie preprint or Xiv::23300.00 094.78, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. Xre preprint arXiv:2204.020 02311,, 2022.
Vanessa D' A'Amario, Tomotake Sasaki, and Xavier Boix. How modular should neural module networks be for systematic generalization? Advances in Neural Information Processing Systems, 34:23374 -23355, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Nicolas Gontier, Koustuv Sinha, Siva Reddy, and Christopher Pal. Measuring systematic generalization in neural proof generation with transformers. In Advances in Newral Information Processing Systems 33 (NeurIPS 2020), pp. 12231-22244 2020.
Mitchell A Gordon, Kevin Duh, and Jared Kaplan. Data and parameter scaling laws for neural machine translation. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing, pp. 5915-5922, 2021.
Jool Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Hcewoo Jun, Hassan Kianinejad, Md Pat- wary, Mostofa Ali, Yang Yang, and Yanqi Zhou. Deep learning scaling predictable, empirically. arXiv preprint ar
2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. Xiv preprint orXiw:2203.15556, 2022.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Language Processing in Python. 2020. doi: 10.5281/zenodo. 1212303.
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In International Conference on Learning Representations (ICLR), 2018
Drew A Hudson and Christopher D Manning. Gqa: A new datasct for compositional question answering. Conference on Computer Vision and Pottern Recognition (CVPR), 2019.
visual reasoning and
15

---

Published in Transactions on Machine Learning Research (07/2024)
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. CLEVR: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2901-2910, 2017a.
Justin Johnson, Bharath Hariharan, Laurens der Maaten, Judy Hoffman, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b.
Yuichi Kamata, Moyuru Yamada, and Takayuki Okatani. Seff-Modularized Transformer: Learn to modularize networks for systematic generalization. In Proceedings of the 18th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications Volume 5: VISAPP, pp- 509, 2023.
Corentin Kervadec, Grigory Antipov, Moez Baccouche, and Christian Wolf. Roses are red, violets are blue... but should VQA expect them to? In Proc. of the 34th IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), pp. 2776-2785, 2021.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequene recurrent networks. In International conference on machine learning, pp. 2873 2882. PMLR, 2018.
Brenden M. Lake, Tal Linzen, and Marco Baroni. Human few-shot learning of compositional instructions. arXiv preprint orXix:1901.000 2019.
Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Pfister, and Xavier Boix. When and how convolutional neural networks generalize out-of-distribution category viewpoint combinations. Nature Machine Intelligence, 4(2):146 153, 2022.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FILM: Visual reasoning with a general conditioning layer. In Proceedings of the 32nd AAAI Conference on Artificial Intelligence, 2018.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
Jonathan S Rosenfeld, Amir Rosenfeld, Yonatan Belinkov, and Nir Shavit. constructive prediction of the gencralization error across scales. In International Conference on Learning Representations, 2020.
Laura Ruis and Brenden Lake. Improving systematic generalization through modularity and augmentation. arXiv preprint arXiv:2202.100 10745, 2022.
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. In Advances in Neural Informalion Processing Systems 33 (NeurIPS 2020), pp. 19861-19872, 2020.
Ben Sorscher, Robert Geirhos, Shashank Shekhar, Surya Ganguli, and Ari Morcos. Beyond neural scaling laws: beating power law scaling via data pruning. Advances in Neural Information Processing Systems, 35: 19523 19536, 2022.
Dídac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. Proceedings of IEFE International Conference on Computer Vision (ICCV), 2023.
Ross Taylor, Marcin Kardas, Guillem Cueurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint ar arXin:2211.00008 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,
16

---

Published in Transactions on Machine Learning Research (07/2024)
Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinct, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungla, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023.
Dmitry Tsarkov, Tibor Tihon, Nathan Scales, Nikola Momchev, Danila Sinopalnikov, and Nathanael Schärli. *- *.CFQ: Analyzing the scalability of machine learning on compositional task. In Proc. of the 35th AAAI Conference Artificial Intelligence (AAAI-21), pp. 9949-9957, 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. Trunsactions on Mochine Learning Research, 2022a.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou, Chain of thought prompting elicits reasoning in large language models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022b. CRL
net/forumiid= /
Moyuru Yamada, Vanessa D' D'Amario, Kentaro Takemoto, Xavier Boix, and Tomotake Sasaki. Transformer Module Networks for systematic generalization in visual question answering. Technical Report CBMM Memo No. 121, Ver.2, Center for Brains, Minds and Machines, 2023.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmect Kohli, and Josh Tenenbaum. Neural-symbolic vqa: Disentangling reasoning from vision and language understanding. Advances in neural information processing systems, 31, 2018.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Paltern Recognition, pp. 12104-12113, 2022.
17

---

Published in Transactions on Machine Learning Research (07 (07/2024) /024)
Appendix
A Implementation Details
We used MAC Hudson & Manning (2018), FILM Perez ct al. (2018), Vectorized Neural Module Networks An- dreas et al. (2016); Bahdanau et al. (2019a) (VectorNMN), its counterpart with given program generator (VectorNMN GT); and the version introduced D'Amario et al. (2021) with modular image encoders (Vec- torNMMSSeSStem). To save time and computational resources, we excluded FiLM and VectorNMNSepSSeem from experiments about question complexity distributions due to their inferior performance compared to MAC and VectorNMN. We conducted hyperparameter search using the attribute comparison and composition datasots and identified a set of effective hyperparancters that yielded consistent results. We scarched for batch size, learning rate, number of iterations, and the way we do D3 for these datasets. We found that with 0.0001 as learning rate and 64 as batch size we get consistent results on the datasets. We tried training with early stopping but found that long training works better. We used 1M iterations for MAC and 500K iterations for the other methods. We also conductod experiments with curriculum learning in which we first training on simple questions from the D3 sets and then train with the other questions but found that training on single mixed dataset achieves better results. Finally, tried MAC and FiLM with ground-trutt program as input instead of question. However, they resulted in poor performance for systematic generalization.
For program generator, we first tried the sequence-lo-sequeen model with attention of CLOSSRE, Bahdanau et al. (2019a) and found that it has poor performance for systematic generalization tasks. Every module in the sqquenco-to-soquenee model is considered as separate instruction. For instance, filter _color [gray] is a separate instruction in the model. We observed that it hard for the model to discriminate different modules when there is compositional biases in the dataset. So, inspired by Madan et al. (2022), we decided to separate the high level modules and their parameters in two stream sequennce-o--sequeeec model. In the two stream sequence-to-sequee example, and the other stream provide its parameter, e-g., gray. This change resulted in more than 40% increase in predicting the correct program for both datasets. All our results for VectorNMN are based on the rwa-stream program generator.
one stream provides the high level instruction, e.g., filter in the above
B Effect of D3 Proportion
Figure 9 shows the effect of amount of proportions in D3 of 1-Hop Full with 2-Hop A and the results on 2-Hop 00D for the VectorNMN GT model. We also compare D3 with 90% and 30% in Table 6 on the 2-Hop datasct. We observe that for VrctorNMNSepStee we have the best result with 90% simple questions. MAC and VectorNMN perform better when D3 with 30% is used. Please refer to Section 3.3 for a more granular effect of different proportions on the final results.
Table 6: Accuracy of D3 with 30% proportion compared 90% on 2-Hop ODD (top) and 3-Hop DOD (bottom).
Training dataset
FiLM
MAC
VectorNMN
VeeporeM
2-Hop A
31.3
29.2
36.7
36.1
+ D3 (1-Hop FULL (30%))
46.9
64.0
61.0
54.8
+ D3 (1-Hop FULL (90%))
50.0
55.5
50.9
67.2
2-Hop A
28.5
27.2
35.9
36.0
+ D3 (1 (1-Hop FULL (30%))
44.1
54.4
51.8
50.7
+ D3 (1-Hop FULL (90%)
48.2
46.5
49.3
56.2
C Testing Modularity
To test the modularity of the trained networks, we devised the following experiments. We chose the O-Hop with single attribute as the test set. We can this test proxy for measuring the degree of modularity
18

---

Published in Transactions on Machine Learning Research (07 (07/202) 2024)
90
80
Accuracy (%)
60
50
40
30
50 Proportion of 1-Hop Full (%)
70
90
Figure 9: Effect of D3 with H-Hop Full proportions accuracy. The base training set is 2-Hop A and the test set is
2-Hop OOD.
FiLM
MAC
VectorNMN GT VectorNMNSepStem GT
+1- Hop full
+1-H +3-Hop full
+1-Hop Hlp-Hop A
+--Hop full + +1 Hop full
+1- +1-Hop
+-HHop B
+--Hop full +1- Hop full HopA
+1-H +1-Hop
+-HHop B
3-HHop full Hop full HopA
+1-H H--oop A
+1-Hop 8
H-HopA
HopA
B
100
blue 19.2
49.5
44.3
42.5
50.0
26
64.7
55.8
29.5
23.9
90.5
44.3
22.2
24.4
6.4
63.0
21.8
88
brown 22.8
48.0
44.5
45.6
45
60.2
56.7
40.5
45
27.5
68.0
44.5
35.0
70.0
6.8
63.2
31
47
79
cyan
4
56
07
80
gray
green
45.4
41.0
45.2
02.0
purple 17.1
48.8
45.0
48.8
52.8
343
59.2
66.3
38.4
39.2
27.1
91.0
45.0
33.7
65.0
7.4
64.3
27.8
66.9
78.9
OHOP ACC (%)
60
red 20.6
46.3
43.8
49.2
47.4
32
63.2
$3.6
41.3
50.8
26.2
67.5
43.8
38.9
59.2
7.6
61.8
23.7
67
68
yellow
45.5
62.7
65.5
45.5
65.3
metal
40
rubber
28.2
14.4
10.0
30.5
57.5
1.6
2.5
22.3
34
60.6
cube 10.8
20.3
8.3
9.0
13.8
23
23.6
86.0
36.3
27.5
29.2
34.8
0.2
7.3
37.3
10.3
16.5
14.4
23
43.
cylinder 8.8
32.4
8.3
11.2
17.5
13
28.7
96.5
32.6
26.9
28.8
32.2
8.3
7.5
32.1
9.6
21.6
12.6
28.7
39
sphere
7.2
large
small-
2.0
10.9
17.5
6.7
26.6
30.2
32.3
20.4
63.9
1.7
0.3
14.8
289
41
Figure 10: Effect of applying D3 on modularity. Modularity is measured by accuracy on the questions with a single attribute from the O-Hop test
since this set contains questions about one attribute of the objects. Figure 10 shows the results of different training regimes defined for the 2-Hop dataset for each attribute. Interestingly, D3 with 1-Hop Full results in better modularity. The color attributes are better learned than the other attributes. There is also a notable accuracy difference between Sphere and other shapes. The reason for this difference is the biases in the images between CoGenT and CoGen'T B, where Cube and Cylinder objects appear with different. colors. We also note that even though VectorNMI GT is using the ground-truth program generator, training 2-Hop results in poor modularity, showing that modules are not performing their expected task when there are biases the dataset. Additionally, we note that D3 with 1-Hop usually improves modularity, although it only adds length diversity.
19

---

Published in Transactions on Machine Learning Research (07/2024)
Table 7: Accurecy of base training on 2-Hop A.
Model
O-Hop
2-Hop OOD
3-Hop A
3- OOD
VectorNMNSepSteen
20.8
36.1
60.4
36.0
VectorNMN
22.3
36.7
57.9
35.9
MAC
20.9
29.2
55.8
27.2
FiLM
13.9
31.3
53.8
28.5
MiniGPT-v2
60.5
70.9
73.7
67.6
Question Generation
The question generation code is based on the original Johnsso et al. (2017a)) and CLEVR-IEP Johnson al. (2017b)* dataset generation frameworks. We added our constraints the code base to generate the biased questions. We have made our question generation code for the CLEVR dataset publicly available
E Base Results of 2-Hop A
Accuracy of base training on 2-Hop dataset for different models is shown in Table 7. The change in accuracy results shown in the heatmaps of Figure 3 and Figure are relative to Table 7.
F Symbolic Methods
VSA Yi et al. (2018). The original NS-VQA framework comprises three components: 1) scene parser, which segments objects in the input image and reconstructs a structural scene representation; 2) a question parser, responsible for converting a question into a program; and 3) a program executor that execules the program on the structural scene represenlation lo derive an answer.
To circumvent. the need for additional annotation of segmentation masks in generated datasets and to mitigate biases inherent in object segmentation models, utilize the original scene graphs used in scene generation. The role of the question parser aligns with the program generator in VeclorNMN, which we have already employed. Subsequently, the program executor can effortlessly execute the generated program on the provided scone graph to produce an answer. Our implementation of NS-VQA can be considered an upper bound compared to the original NS-VQA, as employs ground-truth scene graphs instead of conducting object segmentation. Therefore, the only aspect that can be learned is the program generator, which we have already evaluated for VectorNMN and discussed Appendix A. Nevertheless, for completeness, we adhered to the exact implementation of the sequence-to-sequene model used in NS-VQL NS-VQA and evaluated the complete results the 2-Hop datasets. The results of NS-VQA are provided in Table 8. Similar to the previous results, D3 enhances our implementation of NS-VQA in most of the cases.
the question parser in
Table 8: Results of NS-VQA 2-Hop datasets.
Dataset
O-Hop 2-Hop OOD
3-Hop A
3-Hop OOD
2-Hop
10.77
16.02
37.71
12.12
+1-Hop
11.76
20.97
43.28
17.48
+1-Hop B
35.01
15.88
43.61
12.54
+1-Hop
FULL
18.12
31.70
44.25
19.38
+3-Hop FULL
10.00
99.38
20

---

Published in Transactions on Machine Learning Research (07/2024)
ViperGPT Surís et al. (2023). Viper-GPT relies heavily pretrained models and does not undergo specific training. Therefore, systematic generalization testing cannot. be conducted on Viper-GPT. As pointed out by the authors of Viper-GPP: " ViperGPT may inherit biases from the pretrained models it uses. These biases may be reflected in the outputs generated by our model. It is recommended to consider this potential bios when using ViperGPT and interpreling ils oulpuls." We also note that Modular Networks are similar to symbolic networks with having their modules learned through training.
21


Reviews:
[{"review_id": "ctTEU2Fl3m", "reviewer": "Reviewer_sviB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "<Strengths>\n1. The problem that this work tackles is an important problem – what specific aspects of data diversity can improve the systematic generalization in the VQA task. \n\n2. This work curates a new diversity benchmark on VQA. \n\n3. Experimental results reveal several findings as highlighted with bold fonts in section 3. \n\n<Weaknesses>\n1. Given that this work can be regarded as an empirical study, the performed evaluations are highly limited.\n- Only a single dataset is used – CLEVR. \n- Only two attributes, four different hops, and four question types are considered for the test set. \n- Although it could be a meaningful initial step, it is highly limited to draw dependable conclusions. Moreover, its scope of analyses is relatively narrow as a journal submission.\n\n2. The tested models are outdated. \n- Five models are tested but they are proposed prior to 2020.\n- Thus, the absolute accuracy scores reported in Table 1 are quite low.\n- This work mostly considers module networks based VQA models. Why do not test other categories of VQA model?\n- It would be more interesting to see what results can be obtained on recent powerful LLMs.\n\n3. No qualitative results are presented for the VQA tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work studies the relations between data diversity and models’ systematic generalization in the VQA task. It considers two aspects of data diversity: attribute composition and reasoning hops. It performs experiments using multiple VQA models, including monolithic architectures and neural module networks, on a newly tailored data diversity benchmark based on the CLEVR repository.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weakness above.", "broader_impact_concerns": "No specific ethical concern in this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "MvKfbhq1VC", "reviewer": "Reviewer_onP2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- created a data diversity benchmark - using the proposed D3 (Data Diversity Design)\n- explores model's generalization using attributes composition task\n- tracks the question complexity distribution and length\n- the experiment results show the effect of dataset diversity on generalization  \n\n\nWeakness:\n- the paper's design choices and results are scattered and very hard to follow\n- the paper lacks an overview of context. I don't follow why the proposed design and results are important", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper investigate the role of data, and model architecture for VQA generalization.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Explain why the problem (dataset diversity on generalization for VQA) is worth study? Can it be broke down into subtasks. What is is the impact of this study, and why the findings are useful?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sRMaDNytF7", "reviewer": "Reviewer_8yzK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n1. The paper is interesting and makes sense.\n2. The research question is interesting, and the conclusions also seem to align well with common sense.\n3. The experimental setup is well done.\n\nWeakness\n\n1. As mentioned in the conclusion of the article, the experiment did not provide an analysis of why diverse simple tasks yield better results. It would be beneficial if visualization or other quantitative analyses were presented.\n2. Hope to see more experiments beyond just CLEVR. It would be even better to have more datasets, attributes, and architectures available.\n3. Can comparisons or adjustments be made using large models? For example, MiniGPT4 v2.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper provides new evidence to understand how different diversity affects systematic generalization in VQA, which reveals that the diversity of simple tasks plays an important role in achieving systematic generalization. Moreover, the paper also provides some other findings to understand the interactions between data diversity design, neural network architectures, and systematic generalization capabilities.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The visualization analysis or other quantitative analysis of the experimental results.\n2. More datasets.\n3. Comparison with large models like MiniGPT4 v2.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VfxM31lEE9", "reviewer": "Reviewer_bvDM", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n(+) The paper is unique in that it focuses on the data, not the architecture. It attempts to separate training into different skills and demonstrates that the distribution of these skills in the training data can indeed impact the ability to generalize. \n(+) The presentation and articulation of the results were done well; it was very easy to read through, grasp the main contributions, and understand the conclusions.\n\nWeaknesses: \n(W1) The work should be expanded. At the moment, it provides good proof that there is a need to sample training data better. However, for it to be significant, such a study needs also to be conducted on real image data, not just syntactic. Datasets of VQAv2, TallyQA, and others could provide more significant insights. Ideally, this kind of study should reveal biases in any VQA setup, and it does require more work to understand how nonsynthetic questions can be broken down into different relational-compositional skills. An attempt to incorporate GQA in the appendix, which generates questions syntactically on real images, but it's quite limited compared to the study on CLEVR, and in any case, should not be present only in the appendix.\n(W2) I would also suggest a section for symbolic solutions, such as Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding, and up to the recent study of ViperGPT: Visual Inference via Python Execution for Reasoning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper discusses Visual question-answering tasks and revolves around the effect of different compositional skills in the training data. Specifically, the paper separate skills to the 0-Hop set, which evaluates generalization to shorter, single attribute questions; the 2-Hop OOD set, which assesses attribute composition generalization in more complex questions; the 3-Hop A set, which tests the ability to generalize to longer questions; and the 3-Hop OOD set, evaluating of both out-of-distribution length and attribute composition generalization.\nThe results of the paper show that distributing the training set differently can significantly affect the performance of the same architecture, which is an important conclusion given that most of the literature mostly revolves around architecture and not the training data.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Consider adding a section on real images and non-synthetic questions.\n* It would be helpful if you could discuss symbolic solutions.\n* Including qualitative examples would enrich your paper.\n* Incorporating more experiments from the appendix and improving organization could enhance the paper's clarity.", "broader_impact_concerns": "The Broader Impact Concerns were provided, suggesting that the authors did not use real images in the study due to certain concerns. In my opinion, these concerns are not sufficient reasons to avoid using real data.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides a systematic study to investigate VQA generalization. Coupled with real-data experiments, the experimental evidence is sufficiently convincing. The topic of dataset design is relevant to current VQA research.
Decision: Accept (poster)