Does Data Scaling Lead to Visual Compositional Generalization?
Authors: Arnas Uselis
Andrea Dittadi
Seong Joon Oh
Venue: ICML 2025 poster (2025)
Keywords: ML
compositionality
generalization
OOD generalization
scaling
transfer learning
Primary area: general_machine_learning->representation_learning

Abstract: Compositional understanding is crucial for human intelligence, yet it remains unclear whether contemporary vision models exhibit it. The dominant machine learning paradigm is built on the premise that scaling data and model sizes will improve out-of-distribution performance, including compositional generalization. We test this premise through controlled experiments that systematically vary data scale, concept diversity, and combination coverage. We find that compositional generalization is driven by data diversity, not mere data scale. Increased combinatorial coverage forces models to discover a linearly factored representational structure, where concepts decompose into additive components. We prove this structure is key to efficiency, enabling perfect generalization from few observed combinations. Evaluating pretrained models (DINO, CLIP), we find above-random yet imperfect performance, suggesting partial presence of this structure. Our work motivates stronger emphasis on constructing diverse datasets for compositional generalization, and considering the importance of representational structure that enables efficient compositional learning.

Does Data Scaling Lead to Visual Compositional Generalization?
Arnas Uselis Andrea Dittadi 2345
Seong Joon Oh
Abstract
LAION-400M Indicator Matrix (5,000x5,000 pairs)
Example Pairs (4x4 pairs)
Compositional understanding is crucial for hu- man intelligence, yet it remains unclear whether contemporary vision models exhibit it. The dom- inant machine learning paradigm built on the premise that scaling data and model sizes will improve out-of-distribution performance, includ- ing compositional generalization. We test this premise through controlled experiments that sys- tematically vary data scale, concept diversity, and combination coverage. We find that compositional generalization is driven by data diversity, not mere data scale. Increased combinatorial cover- age forces models to discover a linearly factored representational structure, where concepts decom- pose into additive components. We prove this structure is key to efficiency. enabling perfect generalization from few observed combinations. Evaluating pretrained models (DINO, CLIP), we find above-random yet imperfect performance, suggesting partial presence of this structure. Our work motivates stronger emphasis on construct- ing diverse datasets for compositional generaliza- tion, and considering the importance of represen- tational structure that enables efficient composi- tional learning.
seat
Top 5000 Noung
apple
bed
cheese
tiny liquid melted electric
Top 5000 Adjectives
Pair absent
Pair present
Figure I: Sparse concept combinations in large-scale datasets. Left: An indicator matrix of noun-adjective co-occurrences in LATON-400M shows significant sparsity in concept combinations; majority cells are unobserved (zoomed-in view), demonstrat- ing that even common concepts rarely combine in the dataset. This sparsity biases models toward memorizing frequent combinations rather than learning compositional structure. Right: concrete example of 4x4 matrix of nouns (seat, apple, bed, cheese) and attributes (tiny, liquid, melted, electric). This work investigates how vision models develop compositional attribute-object under- standing in simplified and controlled settings.
rules (Fodor & Fodor, 1975), and neuroscience findings re- inforce this perspective (Dehaene al., 2022). This human proficiency sets high bar for vision models that must un- derstand how visual attributes and objects combine in novel ways. However, recent studies reveal significant limitations in the compositional abilities of state-of-the-ar vision and vision-language models (Rahmanzadehgervi et al., 2024; Tong et al., 2024; Du & Kaclbling, 2024; Yuksekgonul et al., 2023; Zeng al., 2023), raising fundamental ques- tions about whether and when vision models can achieve this capability.
1. Introduction
Compositional understanding is the ability to comprehend novel, complex scenarios by systematically combining sim- pler, known conceptual building blocks. It widely re- garded as a cornerstone of human intelligence. The Lan- guage of Thought hypothesis suggests that cognition arises from fundamental components and structured recombination
The dominant paradigm in machine learning relies scal- ing data and model size to improve model capabilities, with the expectation that this approach will extend to composi- tional understanding. This paradigm, grounded in scaling laws (Kaplan et al., 2020; Hoffmann et al., 2022; Hestness et al., 2017) and demonstrated by the success of large lan- guage models (Brown al., 2020; Touvron et al., 2023) and large-scale vision models (Radford et al., 2021; Dosovitskiy et al., 2021), has driven the creation of massive datasets like LAION-400M (Schuhmann et al., 2021). However, as
'Tübingen AI Center, University of Tübingen Hellmoltz Al "Technical University Munich Munich Center for Machine Learning (MCML) 'Max Planck Institute for Intelligent Systems, Tübingen. Correspondence to: Amas Uselis Carnas.uselis @uni unni tuebingen.de>.
Proceedings of the 4dnd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Does Data Scaling Lead to Visual Compesitional Generalization?
illustrated in Figure I, even LATON-400M exhibits critical sparsity in compositional coverage: many plausible attribute- object combinations are rarely or never observed (e.g. "tiny seat" or "melted apple"). This sparsity reflects combinato- rial explosion: with visual attributes (color, shape, texture) that can combine in vast numbers of ways, most possible combinations will remain underrepresented regardless of dataset size.
random yet imperfect compositional performance.
Our experiments reveal clear principle: compositional generalization is driven by data diversity, not mere data scale. Increased combinatorial coverage forces models to discover a linearly factored representational structure, where concepts decompose into additive components. We prove this structure is not just an artifact but a key to efficiency, enabling perfect generalization from just two examples per concept.
This motivates our central research question:
"Do vision models generalize compositionally, and so, under what conditions?"
2. Related Work
Compositionality, simplicity bias, and generalization. Compositional understanding-tth ability to combine known building blocks into novel representations--is cor- nerstone of human intelligence (Fodor& & Fodor, 1975; De- haene et al., 2022). A central question in machine learning is whether neural networks can achieve this systematic gener- alization. While formalisms for compositionality have been proposed through complexity-basee -base theories (Elmoznino et al., 2024), structural analyses (Lepori et al., 2023), and risk minimization frameworks (Mahajan et al., 2024), mod- els often exhibit simplicity bias (Valle-Pérez al., 2018; Ren & Sutherland, 2024). They favor simple, spuriously cor- related features over more complex, robust ones (Geirhos et al., 2020), a challenge that causal and concept- -ased representation learning aims to address (Rajendran et al., 2024). This bias is especially pronounced when some con- cept combinations are underrepresented, or come from a different domain (Jeong et al., 2025). Our work provides systematic, empirical investigation into the specific data conditions that compel models to overcome this bias and learn a generalizable, compositional latent structure.
Our approach prioritizes controllability to understand when and how vision models can achieve compositional gener- alization. We first train models from scratch on carefully designed datasets to isolate the causal effects of data prop- erties on compositional generalization. This allows us to observe both how generalization performance and represen- tational structure emerge under different data conditions. We then validate whether large-scale pretrained vision mod- els exhibit similar structure and examine how this relates to standard linear probing techniques.
Through this controlled approach, we make five contribu- tions:
(1) Controlled experimental framework ($3): We develop a framework (referred to as (n, k)- ()-ffrmmwoork to systemati- cally study how data scaling impacts compositional gener- alization, varying key factors including training data scale, concept diversity, and combination exposure while focusing on single-object cases to isolate core compositional abilities.
(2) Data diversity over scale ($4.1): We demonstrate that compositional generalization depends critically on data di- versity rather than scale: simply increasing in-distribution training data fails to improve generalization, while increas- ing diversity of data through more concept values and their combinations enhances performance.
Role of data and scaling. The structure of training data is known to be critical for generalization (Madan et al., 2021). Prior work has shown that training on compositionally, struc- tured data improves performance (Stone et al., 2017), and that augmenting data with diverse primitive combinations is beneficial in NL. (Zhou et al., 2023). The broader trend of scaling has led to emergent abilities in large language models (Brown et al., 2020; Bubeck et al., 2023), includ- ing in-context skill composition (He et al., 2024; Arora & Goyal, 2023). However, fundamental limitations remain (Dziri et al., 2023; Zhao et al., 2024; Yu al., 2023), and performance is often tied to concept frequency in the pre- training data (Udandarao et al., 2024; Wiedemer et al., 2025). We contribute to this debate by isolating combinatorial di- versity from raw data quantity, especially when models are trained from scratch, showing that the former is the criti- cal driver for visual compositional generalization, whereas simply increasing the latter is insufficient.
(3) Three-phase feature learning ($4.2): We show that models exhibit three phases of feature learning: (i) spurious features with limited diversity, (ii) discriminative but non- linearly-factored features at moderate diversity, and (iii) linearly factored representations only under high diversity.
(4) Theoretical efficiency of linearly factored structure ($4.3): We prove that when representations exhibit linearly factored structure, observing just two combinations per con- cept value is sufficient for perfect generalization to all un- seen combinations.
(5) Evaluation of pretrained large-scale models ($5): We evaluate whether scrge pretrained models (like DINO and CLIP) exhibit the linearly factored structure identified in our controlled experiments. finding they achieve above-
Structured and linearly factored representations. A
2

---

Does Data Scaling Lead to Visual Compesitional Generalization?
growing body of work finds that large models often ex- hibit structured representations. Specifically,, in large vision- language models, concept embeddings have been observed to sometimes exhibit (to a certain extent) linearity in rep- resentation space, where a composite concept's 's represen- tation is the vector sum of its constituents (Trager et al., 2023; Stein et al., 2024; Park et al., 2024; Andreas, 2019). Theoretical work provides formal conditions under which modularity and abstract representations emerge naturally, for instance as a function of input statistics (Dorrell et al., 2024; Whittington et al., 2022) or when networks are trained to perform multiple tasks (Johnston & Fusi, 2017). However, merely learning structured or disentangled representations does not automatically guarantee compositional generaliza- tion, and the precise conditions under which a compositional structure yields such generalization remain an active area of theoretical inquiry (Lippl & Stachenfeld, 2024; Montero et al., 2022; 2020; Dittadi et al., 2020), particularly for vi- sual attributes (Zhu et al., 2024). Our work provides further investigation under compositional generalization viewpoint, demonstrating the three-phase emergence of this linear struc- ture as a function of data diversity and proving its efficiency for generalization.
introduce our (n, k) experimental framework that allows us to systematically control the complexity of concept spaces and evaluate models' ability to generalize novel concept combinations. Finally, we describe our experimental design, covering both training models from scratch and evaluating pre-trained foundation models.
Our approach is motivated by the question of whether scal- ing data can enable compositional generalization in vision models. To understand the mechanisms behind learning, we also examine whether models develop structured representa- tions in a form of linear factorization, as such structure has been observed to an extent in large pretrained vision models (Stein et al., 2024; Trager et al., 2023).
Data and concept space. We start by formalizing how we represent visual data in terms of concepts. Formally, we consider finite set C = C1 x ... x Ce of concepts repre- senting a factored set of concepts, where each C, contains possible concept values for a particular concept (like shape or color). Each image x E X' is characterized by its position in this concept space through a mapping that assigns it a value from each C. For example, an image of red square could be represented as a point
(2......... cx) E C
where C1 E C1 = {red, blue, green) } represents color and C2 E C2 = (square, circle, triangle] represents shape, and other concepts representing other attributes.
Model-centric approaches and evaluation frameworks. Many works aim to improve compositionality through model-centric solutions, such as specialized architectures (Zahran et al., 2024; ?), object-centric models (Locatello et al., 2020; Wiedemer et al., 2023), soft prompting (Nayak et al., 2023), or feature alignment (Wang et al., 2024a), or algorithmic changes (Ren et al., 2023; 2020). These meth- ods are often studied in zero-shot settings (Atzmon et al., 2020; Xian et al., 2020; Isola et al., 2015; Wang et al., 2023). Concurrently, vision-language models face their own compositional challenges, with debates on whether the hot- tleneck lies in the vision or text encoder (Du & Kaelbling, 2024; Yuksekgonul et al., 2023; Kamath et al., 2023; Vani et al., 2024). In contrast to these model-focused approaches, our work investigates whether compositionality can emerge naturally in standard architectures, isolating the data's 's struc- ture as the primary variable. This requires careful evaluation, and while benchmarks exist for complex reasoning (Zerroug et al., 2022) or specific setups (Madan et al., 2021; Schott et al., 2022; Mamaghan et al., 2024), our (n, k:) framework designed controlled tool. It allows us to make precise causal claims about the data factors that drive generalization.
Definition 3.1 (Concepts and Concept Space). A concept space C = C1 X ... x Ce is a Cartesian product of sets, where each set C, is called a concept and contains all pos- sible values for concept i. Each element G; E C, is called a concept value, and each element c E C represents a unique combination of concept values (ci...., Ce) where e₁ E C,.
Although real-world images typically contain many con- cepts (e.g. color, shape, size, texture), we simplify our study by focusing on pairs of concepts- for example, how models combine colors with shapes in new ways. Even with this simplified setup, we find that models struggle significantly (see Section 4), suggesting that handling more concepts simultancously would be even more difficult.
The (n, k) framework. To systematically study composi- tional generalization, we need a way to control the complex- ity of concept spaces and the diversity of training data. We introduce the (n, k) framework that characterizes concept combination spaces through two key parameters:
3. Approach and experimental framework
: number of distinct values each concept can take A: : number of training examples per concept value
In this section, we establish a systematic framework for studying compositional generalization in visual discrim- inative tasks. We begin by formalizing the compositional generalization through a structured mathematical framework that characterizes how visual concepts combine. We then
Given two concepts with n values each, there are n² pos- sible combinations forming an n x n grid. We observe only l combinations for each concept value during train- ing, testing generalization on the remaining unseen com-
3

---

Does Data Scaling Lead to Visual Compositional Generalization?
Data setting
Model setting
10, A:
-
Train
Concept Concept 2
Concept
DNN
Eval
Correspondence
Train
Scale n
Concept Concept
FM
Eval
MLP
Concept
Train Eval
Figure 2: Investigating compositional learning through concept scaling. The figure illustrates our two main experimental settings. Left (Data setting): Training data consisting of images with corresponding concept combinations shown in the grid, where blue cells indicate observed combinations during training. Right (Model setting): Two approaches-traiinn training models from scratch (Section 4) where we systematically increase the number of possible concept values r while fixing combinations per concept at & = 2, showing examples with 4, n 6. and 10, and evaluating pre-trained foundation models' (FM) compositional abilities by fitting MLP classifier on features (Section S). The grid demonstrates how the concept space expands as we increase ri, creating a larger set unseen combinations for testing generalization.
binations. This framework allows systematic control over both concept complexity (n) and training data diversity (k).
The compositional generalization problem over two con- cepts can be formalized follows:
Given a dataset Duain = {(x,, ci, H Musi of Nurain total images, where each image x; is explicitly labeled with its concept values (c.g., C1 for color, c2 for shape). The training combinations (cf, c2) are drawn from the restricted subset Suain C C1 x Cg. We refer to this as in-distribution (ID) data.
(I) Training:
The figure on the right illus- trates training combinations
k
3
2
Concept 2
for n = 4 concepts with k 3 and k 2 combinations per
concept value. Blue cells indi-
cate the set of training com- binations, which we denote
Concept
Concept
Train
Eval
(2) Testing: Evaluate on combinations from Steat = (Cr x
C2) \ Suain i.c.. concept pairs that never co-occurred during training. We refer to this as out-of- dittr---iittibutionn (OOD) data.
Suain- while orange cells represent the unseen test combina- tions, denoted Stest Each concept value appears in exactly training combinations.
(3) Goal: Learn a model f that accurately predicts both labeled concepts (x), f2())) even for images containing unseen combinations.
Total dataset size. For each of the n k observed training combinations (the blue cells), we generate multiple images to ensure models learn robustly. Specifically, each image varies along several additional unlabeled concept dimen- sions, Crary (like position, orientation, or background). We sample nee examples for each labeled combination, sam- pling uniformly across all possible unlabeled variations. For instance, in a setup with n 4 and 2, there are l = & distinct labeled training combinations. If we in- troduce two unlabeled concepts, such 8 possible positions and 12 possible rotations, the total number of unique images in the training set becomes 8 x 8 x 12 = 768. Concrete examples of the concept space for different values of n and l are shown in Figure 15 in Appendix.
Experimental design. Our experimental approach consists of two main settings as illustrated in Figure 2: (1) training models from scratch, and (2) evaluating pretrained founda- tion models on compositional tasks under our framework. In both cases we systematically vary the (n, k) parameters.
Representation structure and linearity. A key question for understanding compositional generalization is how con- cepts are represented and combined in the learned feature space. We investigate whether concepts combine linearly in the representation space, which would provide a concrete mechanism for efficient compositional generalization (as we show in Section 4.3).
Compositional generalization. Having established our con- cept space framework, we now formalize the specific leam- ing problem we study. Let X be the space of images and (C.Vg-1 1 be the set of possible values for all concepts. While images vary along all concept dimensions, we focus on lcarning and evaluating compositional relationships be- tween two consistently labeled concepts. Specifically, each image x e X our training data is explicitly labeled with a pair of concept values (c1, E Ci x C2, while all other fac- tors of variation (like position, orientation, or background) remain as unlabeled concepts.
Definition 3.2 (Linearly factored embeddings (Trager et al., 2023)). Given a concept space C = C1 x ... x Ce, a collec- tion of vectors (u......, Uc. Je..er... is linearly factored if there exist vectors u., E TRd for all e E C, (i fi=00 1,...,c) .),,,)), which we refer to as concept representations, such that for all c (cI...., Ce):
u Me,
Uc.-
While neural networks are not guaranteed to learn such linearly factored representations, in practice we often ob- serve that these structures emerge during training, as we will

---

Does Data Scaling Lead to Visual Compesitional Generalization?
demonstrate in the following sections. When such linear fac- torizations do emerge, they offer benefits in generalizing compositionally, as we will show in Section 4.3.
cept combinations. We report the average accuracy for the concept pair under consideration.
For representation structure, we consider:
Experimental setup. The guiding principle for our work was to grant models maximally favorable conditions for demonstrating compositional abilities. We do this through several deliberate choices: using oracle model selection rather than validation, fitting multiple classification heads simultaneously to encourage feature reuse, and partitioning concept combinations to create clear train (ID) / test (OOD) evaluation splits.
(i) Seccodabiityyffollovve Kirichenko et al. (2023); Uselis & Oh (2025), we train linear probes on balanced data and report average accuracy across concepts, indicating if fea- tures capture concept information; that is, we merge the training and testing sets, and use held-out dataset covering all concept combinations for measuring decoded accuracy.
(ii) Linearity-we compute the coefficient of determina- tion (R²) between joint representations f(x) and their reconstruction from individual concept representations
Model selection and metrics. For model selection, we use the average accuracy across all concepts at each epoch. We perform oracle model selection by directly evaluating mod- els on the test set to select the best performing checkpoint (Gulrajani & Lopez-Paz, 2020). This allows to focus the fundamental capabilities of models rather than validation strategies.
Σ Ule where R2 = Σx [|f(x) - ",,1I² with
Σ. [[((x)-
= D ExED f(x) measures how well representations
1
follow linear structure. Here, represents the mean repre- sentation across all samples.
(iii) Orthogonality we measure the mean cosine similarity p Ei, cos(u (uef 444 between concept representations to assess if concepts are encoded in orthogonal subspaces, sometimes found in pretrained models (Stein et al., 2024; Wang et al., 2024b).
(I) Training from scratch: We use RESNET-50 (He et al., 2015) with linear classification heads; we found that using transformer backbone (ViT) did not improve generalization performance (see Appendix C.5). The model outputs two predictions fx) = (xi, fx())) where fa X + C2 predicts the value of concept j using a shared backbone followed by separate linear heads. Unlike CLIP which uses language embeddings for classification, we learn fixed clas- sification heads directly from visual data to provide an op- timistic setting for compositional learning through feature reuse.
We report the representation structure metrics only for from- scratch models; this is due to the fact that pretrained models may encode other information other than the target concepts.
4. Does compositional generalization emerge with data scale?
(2) Pre-trained models: We evaluate RESNET5O- IMAGENETIK (He et al.. 2015), RESNET5O-DINOW (Caron et al., 2021), DINOV2-VIT-L/VE (Oquab et al., 2024), and LLIP-VIT-L/14 (Radford et al., 2021). For these models, we pick the best probe architectures on the frozen pre-trained features: a direct linear probe (no hidden layers), an MLP with one hidden layer of size 512, an MLP with two hidden layers of size [512, 512] with ReLU activations; we found these to provide the best performance, and more complex architectures lead to diminishing returns (results in Appendix C.4).
Building on our formal framework, we systematically in- vestigate how neural networks learn compositional under- standing as we vary both data quantity and concept diversity, Our (n, k) framework allows us to precisely control which concept combinations models see during training, enabling us to isolate how different factors affect compositional gen- eralization. Through controlled experiments, we investigate several key questions:
1. Can models generalize compositionally under basic settings? We find that compositional generalization remains challenging with accuracy drops of 27-95% on unseen combinations.
Datasets. We use DSPRITES (Matthey et al., 2017) (using only heart shape to avoid symmetries), 3DSHAPESS (Kim & Mnih, 2019). PUG (Bordes et al., 2023), COLORED- MNIST (Arjovsky et al., 2020), and dataset we introduce of perceptuully--caallenngggg shapes without symmetries to which we refer as FSPRITES. Details in Appendix D.
2. Does increasing ID data quantity improve composi- tional generalization? We show that simply scaling ID data quantity is insufficient.
3. Can neural networks achieve compositional general- ization under any conditions? Yes, but only with suffi- ciently diverse training data.
Metrics. To evaluate compositional generalization and ana- lyze the learned representations, we use two sets of metrics.
4. What kind of structure do representations exhibit when models generalize well? We find that models that gener- alize well exhibit a highly linear and orthogonal struc- ture in their feature space.
For generalization, we report the zero-shot accuracy on Spst: measuring the model's 's ability to classify unseen con-
5

---

Does Data Scaling Lead to Visual Compasitional Generalization?
(a)
(b)
(c)
Shipssap
1.0
-175--27%
Random
-1216-2%%
ID
195%--3%
OOD
-9%
-78%
1.00
Shopes3D
CMNIS
0.75
1.00
Pup pri
Accuracy
0.6
0.75
0.50
DSprites
FSprites
0.50
0.4
scale
Scale
0:25
Random
PSprits
0.00
Shapes3D
DSprites
FSprites
CMNIST
0.7
0.8
Normalized Number of Classes (ningge)
Normalized Num. Combinations
Figure 3: Compositional generalization emerges through different forms of concept diversity. (a) In basic settings with limited diversity, models show substantial accuracy drops unseen combinations (brown) compared to seen combinations (ycllow), demonstrating the inherent difficulty of compositional generalization. (b) When increasing the number of target classes (n) while keeping dataset size and diagonal training combinations fixed (k = n 1), models show improved generalization, suggesting that target space diversity drives compositional learning. (c) With fixed maximum target classes, increasing the number of training combinations (k) also improves performance, showing that exposure to more concept combinations enhances generalization ability, even if the target size is the same.
5. What are the theoretical benefits of such structure for compositional generalization? We show that this lin- ear structure enables perfect generalization to unseen combinations with just two combinations per concept value.
Shapes3D
DSprites
Mean
CMNIST
0.5
ESprites
0.4
0.3
7.5
15
30
60
120
Dataset Size (x10*)
4.1. Compositional generalization is difficult but diverse data helps
Figure 4: Increasing ID training data quantity does not solve compositional generalization. Despite training with significantly more in-distribution samples, models still struggle generalize to unseen concept combinations. The gap between ID and OOD performance remains large across all datasets, suggesting that the challenge of compositional generalization cannot be solved simply by scaling up training data within the same distribution.
Models struggle with basic compositional generalization. In Figure 3(a), in basic compositional setting with 3 concept values and k 2 seen combinations per concept value, while all models achieve strong ID accuracy (near 100%, yellow bars), their performance drops significantly when evaluated on unseen combinations of concepts (brown bars). For example, MNIST digit recognition accuracy drops by around 78% in the OOD setting. Interestingly. in all datasets, least one concept shows relatively small degra- dation, ranging from only 3% drop (orientation in FSprites) to 17% drop (object-hue in Shapes3D), while other concepts in the same datasets show much larger performance gaps.
As shown in Figure 4, despite increasing the training data by 4x, the gap between ID and OOD performance remains large across all datasets: models still show accuracy drops of 60-80% on unseen combinations. This suggests that sim- ply scaling up training data within the same distribution is insufficient for achieving compositional generalization.
Takeaway $4.1: Compositional generalization remains chal- lenging across all datasets, with accuracy drops of 60-80% on unseen combinations despite perfect in-distribution per- formance. While increasing target diversity and combination exposure improves gencralization, scaling dataset size pro- vides limited improvement. Some concepts show relatively small degradation (3-17% drops) while others in the same datasets show much larger gaps. Both target space diversity and exposure to more concept combinations enhance composi- tional learning, but increasing training data quantity (up to 4x) only helps reduce the large ID-OOD performance gap without fully solving the problem.
Increasing concept diversity improves generalization. Figure (bbcc shows that generalization improves both when increasing the number of target classes (n) with fixed di- agonal training combinations (k- creasing training combinations (k) with fixed maximum target classes. This suggests that both target space diver- sity and exposure to more concept combinations enhance compositional learning, even when the target size remains constant.
1), and when in-
Dataset size alone provides limited improvement for gen- eralization. We experimented with RESNET5O trained from scratch using = 3, k= = 1 and three different training set sizes: 7,500, 15,000, and 30,000 samples for SHAPEs3D and CMNIST (the maximum number of unique samples possible with these combinations), and up to 120,000 sam- ples for DSPRITES and FSPRITES. We excluded PUG from this analysis since with n = there were too few unique samples available effectively train the model from scratch.
4.2. Three-phase behavior in feature learning
To understand why models struggle with compositional gen- cralization, we investigate two potential explanations moti- vated by prior work on shortcut learning and distributional robustness (Geirhos et al., 2020; Sagawa et al., 2020). First, the learned features could be spurious, failing to capture meaningful concept information. Second, novel concept
6

---

Does Data Scaling Lead to Visual Compasitional Generalization?
(a)
Decoded accuracy
Zero-shot accuracy
(b)
Linearity R-
Orthogonality cos(Us. Uz)
(c)
8% combs.
42% combs.
98% combs.
1.0
Bg.=1
C
0.8
0.6
0.2
0.6
PC2
0.2
3.4
0.1
Bp -0
50
Charm1
0.0
100
50
100
50
100
PC1
PC1 PC1
of Combinations
95 of Combinations
N: Combinations
55 of Combinations
FSprites
CMNIST
PUG-Animal
3DShapes
DSprites
Figure 5: Linearity emerges with data diversity, while feature discriminability alone does not imply linear structure. (a) Feature discriminability emerges early but does not imply compositional structure, (b) Linear concept representations only emerge with increased mraniigg diversity, as shown through R scores and orthogonality measures, (c) PCA visualizations confirm evolution from entangled to linear feature organization as training diversity increases. X-axis represents percentage of training combinations k/n, with n being the maximum number of concept values.
combinations may produce "misplaced" representations that the classifier fails on. We analyze these possibilities by ex- amining the structure of feature spaces using the linearity and orthogonality metrics defined in Section 3, measuring both the quality of individual concept representations and how predictably they combine. Using& a balanced dataset with all concept combinations (including unseen ones) and 100 samples per combination, we evaluate models trained in the previous section across multiple datasets (MNIST, ESPRITES, SHAPES3D, PUG).
These results indicate a link between training diversity and representation structure in NNs. While models can learn to discriminate individual concepts with limited data (at around 25%), lincarity in representations emerges only with extensive concept diversity. Empirically, linearity and zero- shot accuracy appear to be directly related, suggesting an explanation of previous work showing that decodable fea- tures can be re-aligned to support generalization in large systems like CLIP (Koishigarina et al., 2025).
Takeaway $4.2: Neural networks exhibit three phases: (1)
Our analysis reveals two key findings about how neural networks learn to represent concepts (Figure 5). First, we find that linearity in representations emerges naturally as models are exposed to more diverse training combinations. As shown in Figure 5(b), both the linear separability (R²² scores) and orthogonality (cosine similarity) of concept di- mensions improve with increased training diversity. This emergence of linear structure is accompanied by improved zero-shot generalization Figuue 5(a) shows that zero-shot accuracy on unseen combinations steadily increases as train- ing diversity grows.
With limited diversity (<10% combinations). models learn spurious features and fail at basic concept discrimination; (2) At moderate diversity (10-75% combinations), models gain discriminative ability but lack linear structure; (3) Only with high diversity (>75% of combinations) does true composi- tional structure emerge, with highly linear (It² > 0.8) and orthogonal (cosine similarity < 0.2) concept dimensions. This progression shows that concept diversity is necessary for mod- els to learn structured and generalizable representations.
4.3. Benefits of linear factorization
The benefit of a linear feature structure becomes apparent when contrasted with the weaker property of decodability. While features are often decodable, this alone insufficient for generalization to unseen combinations. Generalizing through decodability may require exposure all possible concept pairings, which is infeasible. As illustrated in Fig- ure 7 (center), while adaptation can compensate for unstruc- tured representations, this approach demands a balanced dataset of all combinations, which is impractical at scale. In contrast, a linear feature structure enables generaliza- tion without exhaustive supervision. As shown in Figure 7 (right), when representations are organized linearly, models can correctly classify novel combinations, overcoming the limitations of mere decodability.
Second, we observe that this progression occurs in three dis- tinct phases: (i) With limited concept combinations (0-10%),
models learn spurious features with poor discrimination (de-
coded accuracy <80%) and random-level zero-shot perfor- mance, as shown by entangled representations in Figure 5(c) at 8%.
(ii) At moderate diversity (25-75%), linearity and orthogo- nality begin emerging (Figure 5(b)), with features becom- ing decodable (100% accuracy) and zero-shot performance reaching 60-80%.
(iii) At high diversity (75-100%), while discriminability plateaus, representations become strongly linear (/²² > 0.8) and orthogonal (cosine similarity <0.1), enabling zero-shot accuracy above 90% on the majority of the datasets. The PCA visualizations in Figure 5(c) qualitatively confirm this progression from entangled to linear factorization.
Motivated by our observation that models achieving strong compositional generalization exhibit highly linear concept representations, we now investigate the theoretical bene- fits of such structure. In this idealized case, how many
7

---

Does Data Scaling Lead to Visual Compositional Generalization?
Shapes3D (object-hue, scale)
DSprites (scale, orientation)
FSprites (color, Shape)
CMNIST
PUG Animal
(color, digit)
(character-namm world-name)
Accuracy (%)
100
80
98
86
30
80
100
100
75
70
64
38
38
25
22
0
ResNet50
INIK
DINOV1
ResNet50
DINOv2
VIT-L
CLIP
VIT-L
ResNet50
INIK
DINOv1
ResNet50
DINOV2
VIT-L
CLIP
VIT-L
ResNet50
INIK
DINOV1
ResNet50
DINOv2
ViT-L
CLIP
VIT-L
ResNet50 INIK
DINOv1
ResNet50
DINOV2
ViT-L
CLIP
VIT-L
ResNet50
INIK
DINOV1
ResNet50
DINOv2
VIT-L
CLIP
VIT-L
Training
Testing
Random
Figure 6: Compositional generalization capabilities of pre-trained models under assumed linear factorization. Bar plots show both
training (transparent) and testing (solid) accuracy across different datasets (DSPRITES, SHAPES3D, CMNIST, PUG-ANIMAL) when using minimal training data (k: combinations per concept) learn lincar concept representations for each concept. Dashed lines indicate random baseline performance. Following Proposition 4.1, we identified the factored representations Uey and Ucy for cach concept value using & = 2 combinations per concept value. While perfect generalization predicted by the proposition would require ideal linear compositionality,
empirical results show strong performance on certain concepts (e.g.. 90% accuracy on color, orientation, digit, and background concepts for either CLIPP DINOv2 models), with varying effectiveness suggesting that pre-trained representations exhibit partial linearity their representations.
different concept types and models,
Decodable
Linear
tional representations: with just two examples per concept value, perfect generalization is possible if the feature space is linearly factorized. We view this as starting point-whii the assumption of linearly independent factors is often sat- isfied in both from-scratch and pre-trained models, it can break down the number of values grows, making joint linear independence impossible e.g., such factors may oc- cupy low-dimensional subspaces (Sonthalia et al., 2025). We expect that this assumption can be relaxed, and that a more complete understanding the setting is possible in future work.
Shape
Shape
Shape
Color
Color
Color
(3) Correct zero-shot Figure 7: Importance of linear feature structure for compo- sitional generalization. We illustrate a schematic for shape and color classification using linear models in 2-dimensional feature space, comparing zcro-shot and adapted cases with frozen fcature extractor. (1) If the feature space lacks a linear structure, the model misclassifies the orange square in zero-shot inference. (2) Adaptation by adding orange square samples allows cor- rect classification. (3) A lincarly structured feature space enables correct zero-shot generalization without adaptation. The decision boundaries linear in all cases, but only the features in the right- most panel enable zero-shot generalization.
Incorrect zero-shot
(2) Correct decoded
Takeaway $4.3: When linear factorization is present, perfect compositional generalization possible with just two combi- nations per concept value.
5. Do large pre-trained models generalize compositionally?
concept combinations would model with perfectly linear representations need to observe to generalize to all unseen combinations? We answer this questions in the following proposition.
Our analysis of models trained from scratch revealed that lin- ear structure emerges naturally when models are exposed to diverse concept combinations. This finding raises a question: Have large-scale pretrained models already learned such lin- ear structure through their pretraining? To investigate this, we evaluate pretrained models using two complementary ap- proaches. We first test for the ideal linear structure from our theoretical framework (Proposition 4.1), which would able perfect generalization. This reveals how close existing models are to this optimal linear structure. Second, we use (non linear probing to assess general concept accessibility in the feature space. Comparing these approaches allows us to distinguish between models that simply encode concept information and those that represent it in structured, linear manner.
Proposition 4.1 (Minimal Compositional Learning). Let f X Rd be a feature extractor with linearly fac- tored concept embeddings over C. Let {ue...... Ucy and fuet..... ueg] be the concept vectors for the first and sec- ond concepts respectively, where their joint span has dimen- sion 2n 1. Suppose we only observe joint representations for concept combinations Cj (1,....,1)) Then k: = 2 combinations per concept value suffice to learn linear classifier that perfectly generalizes to all (n k) unseen combinations.
This proposition illustrates the benefit of perfectly composi-
8

---

Does Data Scaling Lead to Visual Compositional Generalization?
5.1. Evaluating via linear factorization
Performance Acrass Datasets
Normalized Accuracy
Measuring linearity. Building on our earlier findings show- ing the natural emergence of linearly factored representa- tions, test how well the recovered concept value repre- sentations (detailed algorithm in Appendix 1) can be used to classify novel concept combinations. Classification of a new input x can then be performed by projecting the repre- sentation fxx onto the u and v values to acquire labels for both concepts.
1.00
0.75
0.50
0.25
0.00
0.0
0.2
0.4
0.6
0.8
1.0
% Combinations
We calculate accuracy for each concept using this approach and illustrate the results in Figure 6. Certain concept pairs show strong amenability to linear representation across all models. On PUG-ANIMAL, all models achieve exception- ally high accuracy (>90%) on WORLD-NAME concept, sug- gesting more linear representations. The best model consis- tently exceeds 90% accuracy on some concept classification across all datasets. Additionally, models show clear special- ization: CLIP excels at color-based tasks (highest accuracy on CMNIST color-digit and SHAPES3D object-hue), while DINOV2 performs best on shape-based concepts (c.g. on scale, shape, orientation, and character).
ResNet50 from-scratch
DINOv1 ResNet50
CLIP VIT-L
ResNet50 INIK
DINO VIT-L
Figure 8: Even with pretraining, models struggle with composi- tional generalization. Despite the benefits of pretraining, models still face challenges in generalizing to unseen concept combina- tions. While larger models like CLIP and DINO VIT-L show the strongest performance, the persistent gap between pretrained and from-scratch models indicates that current pretraining approaches do not generalize compositionally well.
Takeaway $5: Pre-training is not substitute for data diversity. While large models like CI.IP and DINO VIT-I develop partially linear representations, our analysis shows they only generalize reliably after training a downstream model on a diverse set of concept combinations.
While no model achieves the perfect generalization pre- dicted by our theoretical analysis for ideally linear represen- tations, these results demonstrate that pre-trained models exhibit partial linearity in their representations, varying in strength across concept types. Strong performance on some concept pairs supports our hypothesis that linear representa- tion organization facilitates compositional generalization.
6. Conclusion
In this work, we systematically investigated the conditions under which vision models achieve compositional general- ization, focusing on the distinct roles of data scale versus data diversity. Our findings reveal that merely increasing the volume of training data is insufficient for generalization to novel concept combinations. Instead, data diversity the critical factor. We identified a three-phase learning dynamics where models transition from learning spurious correlations to discriminative features, and finally linearly struc- tured representation space only when trained with sufficient combinatorial diversity. We provide theoretical evidence for the power of this structure, proving that such linear factor- ization allows for perfect generalization from a minimal number of training examples in an idealized setting. When we evaluated large-scale pretrained models through this lens, found they exhibit some of this compositional structure but remain far from perfect, achieving mixed results that highlight their limitations.
5.2. Evaluating generalization via probing
While the linear factorization analysis tests for an ideal com- positional structure, we also employ a more direct test of generalization: probing. In this approach, we train a simple classifier non-linear probe) on the model's features for the seen concept combinations from our training set and evalu- ate it the unseen combinations. This directly measures whether a consistent mapping from features to concepts can be learned and transferred. For each model and dataset, we compute the average accuracy for a given value, keep- ing = 7mmax- To enable fair comparison across datasets, we normalize each model's performance by its maximum accuracy and aggregate the results, shown in Figure 8.
All pre-trained models consistently outperform the from- scratch RESNET50, showing that pre-training provides significant advantage. However, is not a complete solution, as all models improve as the diversity of training combina- tions increases. Full results are in Appendix C.2.
Ultimately, our work suggests that while current scaling paradigms are beneficial, they do not automatically confer robust compositional abilities due the inherent combina- torial sparsity of large-scale datasets. Achieving composi- tional generalization will likely require more deliberate focus on structured data diversity to induce the necessary representational geometry in vision models.
9

---

Does Data Scaling Lead to Visual Compositional Generalization?
Acknowledgments
Dehaene, S., Al Roumi, F., Lakretz, Y., Planton, S., and Sablé-Meyer, M. Symbols and mental programs: A hy- pothesis about human singularity. Trends in Cognitive
We thank the anonymous reviewers for their valuable feed- back, Yujin Jeong, Simon Buchholz, Yi Ren, Samuel Lippl. Ankit Sonthalia, Alexander Rubinstein, and Martin Gubri for helpful discussions, and the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Arnas Uselis. This work was supported by the Tübingen AI Center.
Sciences, 26:751-766, 2022. 1,2
Dittadi, A., Träuble, F., Locatello, F., Wüthrich, M.. Agrawal, V., Winther, O., Bauer, S., and Schölkopf, B. On the transfer of disentangled representations in realistic settings. arXiv preprint (rxxi:20100144000 2020. 3
Dorrell, W., Hsu, K., Hollingsworth, L., Lee, J. H., Wu, J.,
Impact statement
Finn, C., Latham, P. E., Behrens, T. E., and Whittington, J. C. Don't t cut corners: Exact conditions for modularity in biologically inspired representations. arXiv preprint arXiv:2410.062. 262.222,,, 2024.
This work advances understanding of compositional learn- ing vision models, which could enable more data-efficient and reliable Al systems. We release our code and datasets publicly to promote reproducible research and responsible development of these capabilities.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale, 2021.
References
Andreas, J. Measuring Compositionality in Representation
Learning, 2019. 3
Du, Y. and Kaelbling, L. Compositional Generative Mod- eling: A Single Model is Not All You Need, 2024. 1, 3
Arjovsky, M., Bottou, I.., Gulrajani, I., and Lopez-Paz, D.
Invariant Risk Minimization, 2020. 5
Dziri, N., Lu, X.,, Sclar, M., Li, X. L., Jiang, L., Lin, B. Y.. West, P., Bhagavatula, C., Bras, R. L., Hwang, D., Sanyal, S., Welleck, S., Ren, X., Ettinger, A., Harchaoui, Z. and Choi, Y. Faith and Fate: Limits of Transformers on Compositionality, 2023. 2
Arora, S. and Goyal, plex Skills in Language Models, 2023. 2
Theory for Emergence of Com-
Atzmon, Y., Kreuk, F., Shalit, U., and Chechik, G. A causal view of compositional zero-shot recognition, 2020.
Bordes, F. Shekhar, S., Ibrahim, M., Bouchacourt, D., Vin- cent, P., and Morcos, A. S. PUG: Photorealistic and Semantically Controllable Synthetic Data for Representa- tion Learning, 2023. 5
Elmoznino, E., Jiralerspong, T., Bengio, Y., and Lajoie, G. A complexity-based theory of compositionality. arXiv preprint arXiv:2410. 14817, 2024. 2
Fodor, A. and Fodor, J. A. The Language of Thought. The Language and Thought Series. Crowell, New York, NY, 1975. 1,2
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert-Vosss A., Krueger, G.,
Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Bren- del, W., Bethge, M., and Wichmann, F. A. Shorteut Learning in Deep Neural Networks. Nature Machine Intelligence, 2:665-673, 2020. 2,
Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, volume 33, pp. 1877 1901. Curran Associates, Inc., 2020. 1
Gondal, M. W., Wuthrich, M., Miladinovic, D., Locatello, F., Breidt, M., Volchkov, V., Akpo, J., Bachem, O.. Schölkopf, B., and Bauer, S. On the transfer of inductive bias from simulation to the real world: new disentangle- ment dataset. Advances in Neural Information Processing Systems, 32, 2019. 21, 22
Bubeck. S.. Chandrasckaran, V., Eldan, R., Gehrke, J.,
Horvitz, E., Kamar, E., Lee, P., Lee, Y. T., Li, Y., Lund- berg, S., Nori, H., Palangi, H., Ribeiro, M. T., and Zhang, Y. Sparks of Artificial General Intelligence: Early experi- ments with GPT-4, 2023.7 2
Gulrajani, and Lopez-Paz, D. In Search of Lost Domain Generalization, 2020. 5
Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging Properties in Self-Supervised Vision Transformers, 2021. 5
He, K., Zhang, X., Ren, S., and Sun, J. Deep Residual Learning for Image Recognition, 2015.
10

---

Does Data Scaling Lead to Visual Compesitional Generalization?
He, T., Doshi, D., Das, A., and Gromov, A. Learning to grok: Emergence of in-context learning and skill composition in modular arithmetic tasks, 2024. 2
Lippl, S. and Stachenfeld, K. When does compositional structure yield compositional generalization? kernel theory, 2024. 3
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., Patwary, M. M. A., Yang, Y., and Zhou, Y. Deep learning scaling is predictable, empirically, arXiv preprint arXiv: arXiv:1712.004099 2017.
Locatello, F., Weissenborn, D., Unterthiner, T., Mahendran, A., Heigold, G., Uszkoreit, J., Dosovitskiy, A., and Kipf, T. Object-Centric Learning with Slot Attention, 2020. 3
Madan, S., Henry, T., Dozier, J., Ho, H., Bhandari, N., Sasaki, T., Durand, F., Pfister, H., and Boix, X. When and how CNNs generalize to out-of-distribution category- viewpoint combinations, 2021. 3,
Hoffmann, J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., Casas, D. d. L., Hendricks, L. A., Welbl, J., Clark, A., et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.15555, 2022.
Mahajan, D., Pezeshki, M., Armal, C., Mitliagkas, I., Ahuja, K., and Vincent, P. Compositional risk minimization, 2024. 2
Isola, P., Lim, J.. and Adelson, H. Discovering states and transformations in image collections. In 2015 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1383-1391, Boston, MA, USA, 2015. IEEE. 3
Mamaghan, A. M. K., Papa, S., Johansson, K. H., Bauer, S., and Dittadi, A. Exploring the Effectiveness of Object- Centric Representations in Visual Question Answering: Comparative Insights with Foundation Models, 2024. 3
Jeong, Y., Uselis, A., Oh, S. J., and Rohrbach, A. Diffusion classifiers understand compositionality, but conditions apply. arXiv preprint arXiv:2505. /5t.::5500./99555, /7955, 2025. 2
Matthey, L.. Higgins, I., Hassabis, D., and Lerchner, A.
dSprites: Disentanglement testing sprites dataset, 2017. 5
Montero, M. L.., Ludwig, C. J., Costa, R. P., Malhotra, G., and Bowers, J. The role of Disentanglement in Gen- eralisation. International Conference on Learning Representations, 2020.
Johnston, S. and Fusi, S. Abstract representations emerge naturally in neural networks trained to perform multiple tasks. PLOS Computational Biology, 13:e 2017. 3
Montero, M. L., Bowers, J. S., Costa, R. P., Ludwig, C. H., and Malhotra, G. Lost in Latent Space: Disentangled Models and the Challenge of Combinatorial Generalisa- tion, 2022. 3
Kamath, A., Hessel, J., and Chang, K.-W. Text encoders bot- tleneck compositionality in contrastive vision- ssiionlan models, 2023. 3
Kaplan, J., McCandlish, S., Henighan, T., Brown, T. B., Chess, B., Child, R., Gray, S., Radford, A., Wu, J., and Amodei, D. Scaling Laws for Neural Language Models,
Nayak, N. V., Yu, P., and Bach, S. H. Learning to Com- pose Soft Prompts for Compositional Zero-Shot Learning, 2023.
2020. 1
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Assran, M., Ballas, N., Galuba, W., Howes, R., Huang, Y., Li, S.WW., Misra, I., Rabbat, M., Sharma, V., Synnaeve, G., Xu, H., Jegou, H., Mairal, J., Labatut,
Kim, H. and Mnih, A. Disentangling by Factorising, 2019.
Kingma, D. P. and Ba, J. Adam: A Method for Stochastic Optimization, 2017. 14
P., Joulin, A., and Bojanowski, P. DINOv2: Learning Robust Visual Features without Supervision, 2024.
Kirichenko, P., Izmailov, P., and Wilson, A. G. Last Layer Re-Training is Sufficient for Robustness to Spurious Cor- relations, 2023. 5
Park, K., Choe, J.J, Jiang, Y., and Veitch, V. The Geom- etry of Categorical and Hierarchical Concepts in Large Language Models, 2024. 3
Koishigarina, D., Uselis, A., and Oh, S. J. Clip behaves like bag-of-words model cross-modally but not uni-modally. arXiv preprint arXiv:2502.03566 2025. 7
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning Transferable Visual Models From Natural Language Supervision, 2021.
Lepori, M. A., Serre, T., and Pavlick, E. Break It Down: Evidence for Structural Compositionality in Neural Net-
works, 2023. 2
1.5 5
Rahmanzadehgervi. P., Bolton, L., Taesiri, M. R., and Nguyen, A. T. Vision language models are blind, 2024. 1

---

Does Data Scaling Lead to Visual Compesitional Generalization?
Rajendran, G., Buchholz, S., Aragam, B., Schölkopf, B., and Ravikumar, P. From causal to concept-based representa- tion learning. In The Twelfth International Conference on Learning Representations (ICLR), 2024. 2
Xu, P., Yan, Z.,, Zarov, I., Zhang, Y., Fan, A., Kambadur, M., Narang, S., Rodriguez, A., Stojnic, R., Edunov, S., and Scialom, T. Llama 2: Open Foundation and Fine- Tuned Chat Models, 2023.
Ren, Y. and Sutherland, D. J. Understanding Simplicity Bias towards Compositional Mappings via Learning Dy- namics, 2024. 2
Trager, M., Perera, P., Zancato, L., Achille, A., Bhatia, P., and Soatto, S. Linear Spaces of Meanings: Compositional Structures in Vision-Language Models, 2023. 3, 4, 15
Ren, Y., Guo, S., Labeau, M., Cohen, S. B., and Kirby, S. Compositional Languages Emerge in Neural Iterated Learning Model, 2020.
Udandarao, V., Prabhu, A., Ghosh, A., Sharma, Y., Torr, P. H. S., Bibi, A., Albanie, S., and Bethge, M. No "Zero- Shot" Without Exponential Data: Pretraining Concept Frequency Determines Multimodal Model Performance, 2024. 2
Ren, Y., Lavoie, S., Galkin, M., Sutherland, D. J., and Courville, A. Improving Compositional Generalization using Iterated Learning and Simplicial Embeddings. 2023.
Uselis, A. and Oh, S. J. Intermediate layer classifiers for ood generalization, 2025. 5
Sagawa, S., Koh, P. W., Hashimoto, T. B., and Liang, P. Dis- tributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Gen- eralization, 2020. 6
Valle-Pérez, G., Camargo, C. Q., and Louis, A. A. Deep learning generalizes because the parameter- function map is biased towards simple functions. https://arxiv.orr/abb////// 08552255..2225 2018.
Schott, L.., von Kügelgen, J., Träuble, F., Gehler, P., Russell, C.. Bethhe, M., Schölkopf, B., Locatello, F., and Brendel, W. Visual Representation Learning Does Not Generalize Strongly Within the Same Domain, 2022. 3
Vani, A., Nguyen, B., Lavoie, S., Krishna, R., and Courville, A. SPARO: Selective Attention for Robust and Composi- tional Transformer Encodings for Vision, 2024. 3
Wang, H., Si, H., Shao, H., and Zhao, H. Enhancing Compo- sitional Generalization via Compositional Feature Align- ment, 2024a. 3
Schuhmann, C., Vencu, R., Beaumont, R., Kaczmarczyk,
R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. LAION-400M: Open Dataset of CLIP- Filtered 400 Million Image-Tex: Text Pairs, 2021.
Wang, Q., Liu, I.,, Jing, C., Chen, H., Liang, G., Wang, P., and Shen, C. Learning Conditional Attributes for Compositional Zero-Shot Learning. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pp. 11197-11206, Vancouver, BC, Canada, 2023. IEEE. 3
Sonthalia, A., Uselis, A., and Oh, S. On the rankability of visual embeddings, 2025. 8
Stein, A., Naik, A., Wu, Y., Naik, M., and Wong, E. Towards Compositionality in Concept Learning, 2024. 3,
Wang, Z., Gui, L., Negrea, J., and Veitch, V. Concept alge- bra for (score-based) text-controlled generative models, 2024b. 5
Stone, A., Wang, H. Stark, M., Liu, Y., Phoenix, D. S., and George, D. Teaching Compositionality to CNNs, 2017. 2
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs. 2024.
Whittington, J. C., Dorrell, W., Ganguli, S., and Behrens, T. E. Disentanglement with biological constraints: A theory of functional cell types. arXiv preprint arXiv:2210.01766 2022.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi,
A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W. Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshorn, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V., Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A.. Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet, X., Mihaylov, T., Mishra, P., Molybog, I. Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K. Schelten, A., Silva, R., Smith, M., Subramanian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X.,
Wiedemer, T., Brady, J., Panfilov, A., Juhos, A., Bethge, M., and Brendel, W. Provable Compositional Generalization for Object-Centric Learning, 2023. 3
Wiedemer, T., Sharma, Y., Prabhu, A., Bethge, M., and Bren- del, W. Pretraining Frequency Predicts Compositional Generalization of CLIP on
Tasks. 2025. 2
Xian, Y., Lampert, C. H., Schiele, B., and Akata, Z. Zero- Shot Learning A Comprehensive Evaluation of the Good, the Bad and the Ugly, 2020. 3
12

---

Does Data Scaling Lead to Visual Compositional Generalization?
Yu, D., Kaur, S., Gupta, A., Brown-Cohen, J., Goyal, A., and Arora, S. Skill-Mix: A Flexible and Expandable Family of Evaluations for AI models, 2023. 2
Yuksekgonul, M., Bianchi, F., Kalluri, P., Jurafsky, D., and Zou, When and why vision-language models behave like bags-of-words, and what to do about it?, 2023. 1,3
Zahran, Y., Burghouts, G., and Eisma, Y. B. Anticipating Future Object Compositions without Forgetting, 2024.
Zeng, Y., Huang, Y., Zhang, J., Jie, 7., Chai, Z., and Wang, L. Investigating compositional challenges in vision- language models for visual grounding. 2023.
Zerroug, A., Vaishnav, M., Colin, J., Musslick, S., and Serre, T. A benchmark for compositional visual reasoning. In Advances in Neural Information Processing Systems, 2022. 3
Zhao, H., Kaur, S., Yu, D., Goyal, A., and Arora, S. Can Models Learn Skill Composition from Examples?, 2024.
2
Zhou, C., Chen, P., Liu, B., Li, X., Zhang, C., and Huang, H. Data factors for better compositional generalization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023. 2
Zhu, W. Y., Ye, K., Ke, J., Yu, J., Guibas, L., Milanfar, P., and Yang, F. ArtLLM: Attribute Recognition Through Vision-Based Prefix Language Modeling, 2024. 3
13

---

Does Data Scaling Lead to Visual Compesitional Generalization?
A. Appendix
A. Experimental setup and implementation
A.I. Implementation details
In this section we provide additional details on the implementation of the experiments.
Optimization. All models trained using the Adam (Kingma & Ba, 2017) optimizer. Based initial grid search, use learning rate of 10 for ResNet training from scratch and 10 for probing pre-trained features. All models trained for 100 epochs with batch size of 64.
Train/Test splits. For each concept value i, we observe combinations with values j where (i j j + n) mod n < k, and evaluate on all other combinations. This creates clear distinction between combinations seen during training and those requiring compositional generalization.
The key idea is creating a training set that balanced such that each concept value is observed with equal frequency. For each concept value E {O,....,h) 1}, we observe exactly k combinations during training, defining training and test sets as:
Curain U ((i, (i+ mod n)) j E (O...., 111 [m0l
(2)
Gess (C1 C2) Crain.
This construction ensures that: (1) each concept value appears in exactly & training combinations, (2) the test set contains (n k) novel combinations, and (3) the split is deterministic and reproducible across experiments.
Concept value selection. For each experiment with parameters n and k, we select n values for each of our two target concepts that are maximally spread across their respective concept spaces. Specifically, if concept has [Cmax] possible values, we select values at indices {i [[Cmax//n] / ensure even coverage.
Sampling procedure. Within each valid training combination (each "cell" in our concept grid), sample neeel examples uniformly from all possible variations of the remaining unlabeled concepts Cvary (like position, orientation, background, etc.). This uniform sampling across |Cvary| possible variations ensures balanced representation of each concept combination across different visual contexts.
B. Proofs
In this section provide the proofs for our main theoretical results.
Notation. We summarize the notation used throughout the proofs, though we reintroduce each term where appropriate.
Spaces and mappings:
X' represents the input space (images)
- C = C x C2 x ... C. represents the concept space
- C₁ is the i-th concept dimension (e.g., color, shape)
c: - C is the mapping from images to concept values
cxx) (c1....,cy Ce) gives the concept values for image
C; denotes the value of the i-th concept
Framework parameters:
is the number of concept values per dimension in the (n, k) framework
- is the number of training combinations per concept value
- c is the total number of concept dimensions
Feature representations:
f(xx the feature extractor output for image
f T ExeD fx) is the global mean embedding
Up represents the true concept vector for value &
u', the recovered centred concept vector for value C;
14

---

Does Data Scaling Lead to Visual Compesitional Generalization?
u e,e,, denotes the pairwise joint embedding for values G3 Cj
Datasets:
D represents dataset of image-concept pairs
De is the dataset over all possible concept combinations
Duain and Dist are the training and test datasets with limited and unseen combinations, respectively
Da, is the subset of DD containing concept value ç;
- Dan.ey contains both values & and Cj
Training constructs:
-irain is the set of observed concept combinations during training
- fij represents the mean embedding for combination (i,j)
Let X' denote the imput space and C C1 x C2 ... C. represent the concept space. We assume mapping X - C that identifies for each image X' its corresponding concept values c(x) = (e) EC
We denote De as the dataset over all possible concept combinations. In practise, we only observe limited combinations, as discussed in Section A. We denote such a dataset as Drain and Diest for the training and test sets, respectively.
We also restate the linear factorization definition from the main text:
Definition B.1 (Linearly factored embeddings (Trager et al., 2023)), Given concept space C = C1 ... x Ce collection of vectors (ue)eec is linearly factored if there exist vectors Uc, E Rd for all E C2 (i = 1,..., ,,), which we refer to as concept representations, such that for all c Ce):
U Usy + +... +..44 Use
(3)
Assuming linear factorization,
f(x) = Σ [oz(x))
and given dataset D = {(x,, o,)}i=1 with 8 s II [.), with image-concept pairs, can recover representation (up to global shift shared by all factors) for each concept value by averaging feature vectors across all combinations that contain that value (Trager et al., 2023). Formally, for a value e; E C, let
Σ [s(x) ff, xED
f
Σ f(x), XED
(4)
D
D
Thus u', is the conditional mean feature vector, centred by the global mean f.
We first describe the relationship between the ground truth factors u,, and the recovered ones n',- These relationships only hold for the case when the contstructed factors are recovered from the full dataset.
Lemma B.2 (Relation to ground truth concept vectors). Let Uc, denote the true concept vector for value oi and u', the recovered one from (4). Over the full dataset,
U
Ca
EEE;
Proof.
Start from the definition (4) and substitute the linear factorisation f(x) Ei=1 Mex())
u,,
[f(x)
f)
De.l
xEDr;
Uee(x) - f.
(I)
Dey
xEDe,
t=1
Interchange sums in (I). For term with i each X Da. contributes Uny hence
Uc:
Ue:-
Dr. xED
15

---

Does Data Scaling Lead to Visual Compositional Generalization?
For any l each value de Ce occurs equally often inside Dey, namely |Perl/Ccl times. Therefore
1
Σ
(x)
Ce
xED
CfECe
Summing these contributions and using the explicit formula for the global mean
f
Σ
f(x)
ueyi
Cel
XED
CFEC
it follows that
U/a Uc
+
4 - f
C.
eCEC:
as claimed.
also follows that this construction of factors Ue, leads recovery of the sum of factored embeddings up a global mean. Importantly, if full dataset De is available, normalizing the mean of the embeddings (i.e. setting f := 0) is possible.
Lemma B.3 (Reconstruction of centred embedding). For any x with concept values (ci (x)),..,,cc() cx(x))
f(x) =f Σ (x)))))
Proof. Using Lemma B.2 we have for every concept value Ci
u', Ue
1
Σ u
|C,
CEc.
Applying this identity to the particular values (x() of the sample and summing over 1,..., ., k yieldss
Σ u f(x) -f.
deEc.
where the last equality uses f(x) = E, (xx(x) and the definition of the global mean f.
In what follows study compositional settings where the concept space may include many factors, but only two factors, C₁ and C2, are observed; the remaining factors C, are unobserved. Importantly, factors C₁ and C2 exhibit a correlation due to the (n, k) framework.
Next, we establish a convenient property of the factored representations. Lemma B.4 (Zero-sum embeddings). For any concept dimension i E {l,...,
Σ
n
ECEC.
Proof. Let f =
Σ f(x) be the global mean. For each value C; E C, set
[D]
XED
De, == (xE (x =cb,
m: := [Pal
(same for every ct),
16

---

Does Data Scaling Lead to Visual Compositional Generalization?
Summing over c gives
Σ
U
Σ
Σ xEDs
f(x) -f
(5)
GEC;
1
Σ
f(x)
|c,
(6)
OzEC: XEDr,
1 Σ f(x) - C.
XED
|P| f |c,l f (ID) = [C,1
[c,| f |C 0.
In practice, we often only observe subset of concept combinations. To accomodate such a constraint, we formalize through pairwise joint embeddings:
Definition B.5 (Pairwise joint embedding). Given concept space C C1 x... xCe, the pairwise joint embedding for factors # j and values c Ci. Cj Cj is
Σ xEDoLies
[f(x) f), De,e, == fx c(x); =G, c(x); = cg..
(10)
(Desoos
Lemma B.6 (Additivity of joint embeddings). Under linear factorisation f(x) = Ei=1 (x) holds,
u u/, u/y.
(11)
Proof. Define
Dar,e x E c(x); = Cis c(x); =c
Ne,,e (Densyl-
Substituting the centred decomposition f(x) = f Ei=l u' ex(x) from Lemma B.3 to Definition B.5 gives
u/
[f(x) -f
(12)
Neces xE xEDegieg
+
U' -
(13)
Neses xEDegigg
1=1
= Nea.y xEDeiee t=1
[ee(x))
(14)
For every x E Delee we have (x) and (x) Cy. Hence the terms with l = and L contribute exactly u', and uLy, respectively.
For any l & (i,j) each value &e C2 occurs equally often inside Des,e.. Therefore
u 0, by Lemma B.4.
No
C
xEDegig
Efc
Collecting all contributions we obtain the desired identity
en u/, + u'y
17

---

Does Data Scaling Lead to Visual Compesitional Generalization?
We establish our main theoretical result the minimal data requirements for compositional generalization. The derivations from the Lemmas above appropriate under the assumption of balanced training set. Due to the unlikely nature of certain concept combinations (as described in the (n, k) framework), the main challenge is identifying the factors under such a setting.
Proposition B.7 (Minimal compositional learning). Let J X - IR be a feature extractor with linearly factored concept embeddings over C. Let {uct..... uey} and (uet...... Mey } be the concept vectors for the first and second concepts respectively, where their joint span has dimension 2n - L. Suppose we only observe joint representations for concept combinations Ca. Cj E {I....,m.] n.). Then & = combinations per concept value suffice to learn a linear classifier that perfectly generalizes to all (n k) n unseen combinations.
Proof. The proof proceeds three steps: (1) showing that joint factored embeddings are identifiable from training data, (2) showing that the system of linear equations has full rank with 2n equations and 2n unknowns, and (3) showing that optimal classifiers can be constructed via orthogonal projections.
Part 1: Identifying joint factored embeddings uci.c!"
We assume for simplicity, but the same applies for higher k. First, note that observe the following combinations:
Crain {(i, :ie in]} U ((iiiiii1))001 e [n 1}} u ((n 1)} 1, n)) ((n,
(15)
= ((1, (1,1)) (2,)), (n, U (1, (2,3), (n
(16)
with CCrainl 2n total combinations. This dataset restricted the combinations in Cuain. but varies in other concepts. We denote this dataset Drain = (2, (2),0) Cuain, X
We aim to show that the average embedding over the training set, Unain equals the global mean embedding f (as defined in the proof of Lemma B.4). Let Di C. Drain be the subset of training samples for the specific concept combination (i,j). To see the importance of this, note that
cj.c = u'g + n'g.
(17)
By Definition B.5, given some observations of concept values of and ei the pairwise joint embedding u', di.ef of the embeddings of the training samples for the combination (i,j) shifted by the global mean embedding f. Consider the mean embedding over the training set
is the average
Ocrain
Σ f(x).
(18)
Derain
x
We now show that f = Ugain-
Under the assumption a balanced training set where each combination (i,J) E Craain has the same number of samples, we can define the mean embedding for each combination as:
Σ f(x). xED
Dia
18

---

Does Data Scaling Lead to Visual Compositional Generalization?
The overall training mean then:
Otaai
Σ
[(x)
(19)
Deriin
xc Dean
n1
n1
f,1 f
(20)
2n
n-1
(f + u'y
+ uug
Σ
(f + u'y + u( + (f + u/p + uuuy)
(21)
2n
2
i=1
2nf +
u +
(22)
2m
i=1
1
(2nf + +2.0+2.0)
(by Lemma B.4)
(23)
2n
(24)
Thus, we can identify the factored representations "of.og for each concept value combination i, E [xi] from the training data since the average representation over the training data under our training dataset is the global mean embedding f. With this, we can compute u' for 2n combinations.
c.c
Part 2: Identifying the individual factored representations u,f and ueg for each concept value E [n].
Consider a training set with exactly combinations per concept value. By the linear factorization property, for any combination (i,j) in training set, we have: u', of u' u', where of denotes value for the first concept and oj denotes value for the second concept.
Let U1 U2 E Rdxa be matrices whose columns are the unknown factored representations nny and u'g respectively for i [n]. Let V E RR be the matrix of observed pairwise joint embeddings u for the 2n training combinations. The
system of equations can be written as:
0
0
1
0
4.
1
0
u/p.eg
0
0
(25)
0
1
4.0
0
0
U1
U2
We note that this system is full rank, the design matrix has linearly independent rows. The first block of rows corresponds to the diagonal combinations i). while the second block corresponds to cyclic combinations (i, + 1) (with wraparound from to 1). These form distinct patterns that ensure linear independence.
Given this full rank system with 2n equations and 2n unknowns (the factored representations u, and u', for each concept value), can uniquely solve for the factored representations. For k 2 combinations per concept value, get more equations while maintaining the same number of unknowns, making the system overdetermined and the solution more robust.
19

---

Does Data Scaling Lead to Visual Compesitional Generalization?
Once we recover these factored representations, compute u' of uny +
ung for any combination (i,j), including
the (n- 2)n unseen ones.
Part 3: Optimality of classifiers. To show that construct classifiers that provable generalize to novel combinations, we simply note that by assumption no concept representation is within the span of remaining representations. As such, given U1 = span(fu', ).e)). and U2 == span( (u'a ( such that dim(U1) C1l and dim(U2) |C2] 1 and U1 n U2 {0}. any vector w in their joint span can be uniquely decomposed as = μj + where E U1, U2 E U2 and - U2. This allows to construct projection matrices Pv, and Pus onto these orthogonal subspaces, which can then
be used to build optimal classifiers by projecting input features onto the respective concept subspaces.
B.1. Algorithmic recovery of factored representations
We provide a constructive algorithm for recovering factored concept representations from limited available training combinations in Algorithm
Algorithm Recovering factored concept representations for concepts
Require: Training dataset Dinin where each individual concept appears in least 2 different combinations (k 2 2)
Require: Feature extractor f -> IR'
Ensure: Factored concept representations {u's 1i {u'g Jim
1: Compute global mean embedding: [a [Pmnll ΣxeDim [xx,a for each dimension
2: for d = 1 to d do
3: Initialize design matrix A 22xx2 based on observed combinations
4: Initialize E _ store joint embeddings for dimension
5: row 1-1
6: for cach combination (i,j) in training set do Σx =x fxx) fa
7: u
c (1x=e(x)==,e((222)
8: Store u "f in position row of v
9: Update row of A with indicators for concepts and j
10: row
end
12: Solve system A
v for dimension d
13: Store solutions {u He., [u'g Fim at dimension
end for
15:
return (u's Hi (u (u4))=
C. Additional experimental results
This section presents supplementary experimental findings.
C.1. From-scratch model performance
Figure summarizes how aaftof-distribution accuracy varies with the number concept classes and the number of training combinations per class across four datasets. In all cases, increasing concept diversity (number of classes) is associated with higher compositional generalization performance, when the number of training combinations per class is held fixed.
20

---

Does Data Scaling Lead to Visual Compositional Generalization?
Shapes3D
DSprites
FSprites
CMNIST
Num. Combs,
100
93
64
Performance increases
75
72
75
70
44
41
43
Num. Classes
Num. Classes
Num. Classes
Num. Classes
Performance decreases
Figure 9: Performance scaling with concept diversity. OOD accuracies across four datasets: Shapes3D, dSprites, FSprites, and Colored- MNIST. Each heatmap shows performance for different combinations concept values (n) and seen combinations (k) per concept value. Increasing concept diversity (higher rt) consistently improves generalization performance across all datasets, even when the number of training combinations per concept remains fixed.
C.2. Pre-trained model probing results
Shapes3D object-hux hue
DSprites scale
FSprites shape
CMNIST color
PUG Animal character-name
Accuracy
100
100
100
100
100
50
90
50
0
orientation
orientation
digit
world
Accuracy
100
100
100
100
50
25
62
81
100
37
68
84
100
11
77
100
18
38
79
100
26 Combinations
100
% Combinations
% Combinations
% Combinations
%
Combinations
%
ResNet50 from scratch
ResNet5o
DINOv1 ResNet50
Figure 10: Compositional generalization in pre-trained models. Hcatmaps show out-of-distribation accuracy for different combinations of (concept values) and k (training combinations) across datasets, Darker colors indicate higher accuracy. Pre-trained models exhibit improved generalization with increased concept diversity, mirroring the pattern observed in from-scratch training.
To systematically probe compositional generalization in pre-trained vision models, we evaluated range of architectures, including ResNet50 (from scratch and ImageNet pre-trained), DINOv1, DINO ViT-L, and CLIP ViT-L across several datasets and concept axes, as shown in Figure 10.
C.3. MPI3D dataset results
To validate our findings on real-world datasets, we conduct experiments on the MP13D dataset (Gondal et al., 2019), which contains photographs of 3D scenes with systematic concept variations.
21

---

Does Data Scaling Lead to Visual Compesitional Generalization?
Color:
Color: Shape: 2
Color: Shape: 0
Color: 3
Color: Shape: 3
Shape:
Shape: 2
Color:
Color: 0
Color: 2 Shape: 1
Color: 0
Color: 2 Shape:
Shape:
Shape: 5
Shape: 5
Figure 11: Sample images from the MPI3D dataset (Gondal et al., 2019). The dataset contains real- world images of objects with varying properties like color, shape, size and camera viewpoint. Examples from the testing set of 6, k are shown.
-1%
-94%
1.0
Accuracy
0.8
object-shape
Random
object-color
ID
0.6
OOD
0.4
0.2
0.0
MPI3D
Figure Accuracy comparison n 3, k using ResNet-50. shown the main text, compositional generalization difficult: the model struggles to generalize to the object-shape concept.
Accuracy
0.50
MPI3D
OOD Accuracy
0.05
0.45
0.00
0.55
Random
0.35
30
120
0.5
0.6
0.7
0.8
0.9
1.0
Dataset (x10%)
Normalized Number Classes (n/0max)
66
OOD Accuracy
MP13DI
62
0.60
0.55
Normalized Num. Combinations
Figure Compositional generalization only improves data diversity, not data quantity. Top left: Under few training combinations k3 k= 2), compositional generalization does not benefit from more data. The remaining plots show compositional generalization improving with more diverse training combinations: when the number of classes increases (top right), and when the number of training combinations increases (bottom left)
These results provide strong evidence that compositional generalization benefits specifically from diversity in concept
22

---

Does Data Scaling Lead to Visual Compositional Generalization?
combinations rather than mere quantity of training data.
MPI3D
(object-color, object-shape)
100
92
MPI3D object-color
Accuracy (%)
73
Accuracy
100
75
object-shape
DINOVI ResNet50
DINO VIT-L
54
CLIP ViT-L
ResNet50 IN1K
25
25
Random
0
37
52
68
84
100
37
52
68
84
100
ResNet50 INIK
DINOv1
ResNet50
DINOV2
VIT-L
CLIP
VIT-L
% Combinations
% Combinations
ResNet50 IN1K
DINOVI ResNet50
DINO VIT-L
VIT-L
VIT-L
Figure Evaluating pre-trained vision models on MPI3D. Left: Accuracy comparison for classifiers constructed under linear factorization. All models show near-perfect accuracy on the color concept, while shape concept performance is worse. Right: Probing results using linear and non-linear probes.
C.4. Comparison of diffeernt probe configurations
We present detailed comparison of probe results across different model architectures and probe types. Table reports the accuracy of both linear and non-linear (two-layer MLP) probes the FSprites dataset for several pre-trane mre-trained models. Notably, non-linear probes generally yield higher accuracy than linear probes, especially for models like DINO ResNet-50 and DINO ViT-Large, indicating that some compositional information is not linearly accessible in the representations. However, for ResNet-50 and CLIP ViT-Large, the difference between linear and non-linear probe performance is smaller, suggesting that their representations are more linearly separable for the evaluated concepts.
Table Linear and non-linear probing results. Comparison between linear probes and two-layer MLPs [512, 5121 as the observed percentage of combinations on FSprites dataset. Results show the accuracy in the form of linear non-linear probing for different pre-trained models.
Model
25%
50 %
75 %
93%
ResNet-50 ImageNet
0.59 / 0.55
0.67 0.65
0.75 0.75
0.79 / 0.82
DINO ResNet-50
0.60 / 0.67
0.71 / 0.80
0.76 0.88
0.80 / 0.92
DINO ViT-Large
0.68 / 0.70
0.78 / 0.83
0.84/
0.86 / 0.95
CLIP ViT-Large
0.61/0.5
0.70 0.74
0.75 0.79
0.76 / 0.84
C.5. Architecture comparisons
We provide detailed comparisons between different neural architectures validate our choice of ResNet-50 the primary baseline.
A comprehensive hyper-parameter sweep was conducted for the vision transformer (ViT), varying patch size (E (8, 16)), depth (E {4,0 6, 8)), width (e (384, 512}), number of heads (E {8, 12)), MLP width (E {384, 512)), and learning rate (E (3x10- 1x10-4 3x10 1xx0011 31). Across all configurations, VIT does not outperform a scratch-train generalisation. Both models achieve comparable in-distribution accuracy (99.7%), but the ResNet-50 baseline consistently yields higher OOD performance across datasets and diversity regimes. Table summarises these results.
ResNet-50 in OOD
23

---

Does Data Scaling Lead to Visual Compositional Generalization?
Table 2: Accuracy of ResNet50 and VIT models trained scratch.
Dataset
Model
% Combinations
k/n
Training samples (x10*)
OOD Acc.
ResNet-50
80
/ 10
60
95.1
ViT
80
8 / 10
60
94.5
CMNIST
ResNet-50
40
4/ 10
60
66.0
VIT
4/ 10
71.0
NesNe
92
13 14
60
80.1
FunnySprites
VIT
92
13/ 14
60
66.0
ViT
13/ 14
120
57.3
D. Dataset details and examples
This section provides comprehensive information about all datasets used in our experiments, including detailed descriptions and visual examples.
Table 3: Overview of experimental datasets. Each dataset provides controlled variations along two primary concept dimensions, enabling systematic study of compositional generalization.
Dataset
Primary Concepts (C1, C2)
Concept Values
PUG
Animal type, Background type
each
Shapes3D
Scale, Object hue Scale, Orientation
each
dSprites
6 each
FunnySprites
Shape, Color
14 each
ColOred-MNIST
Digit, Color
10 each
MPI3D
Object shape, Object color
Variable
24

---

Does Data Scaling Lead to Visual Compositional Generalization?
x
++
x
4
2
99
Figure 15: FunnySprites dataset examples. Shape and orientation variations for 14 concept values with 2 training combinations.
Each sprite generated by connecting traced points to unique geometric shapes, providing a challenging test for compositional generalization.
2
3
4
S
6
7
8
9
0
2
3
4
5
6
7
9
9
O
2
3
4
S
6
7
8
9
2
3
4
5
6
7
8
9
0
2
3
4
S
6
7
8
9
O
/
2
3
4
5
6
7
8
0
2
3
4
5
6
7
8
9
O
/
2
3
4
S
6
7
8
9
0
/
>
3
4
5
6
7
8
9
0
/
2
3
4
5
6
7
8
9
Figure 16: Colored-MNIST examples. Digit and color combinations for 10 values with A: training combinations. This dataset combines the MNIST digits with color variations to test compositional understanding of shape and color altributes.
We introduce the Funny Sprites dataset, OOD dataset designed test models' ability to generalize to previously unseen shape combinations. The dataset consists of sprites traced from 5-15 points 128x128 pixel grid, creating diverse set of abstract geometric shapes.
25


Reviews:
[{"review_id": "Qz6fdn2bQl", "reviewer": "Reviewer_FpUd", "summary": "Compositional generalization is a fundamental aspect of human intelligence, yet its emergence in machine learning, particularly in vision models, remains unclear. This paper investigates whether increasing data scale contributes to compositional generalization in vision models by systematically isolating the effects of dataset size, diversity, and compositional structure. Through behavioral evaluations and probing latent representations of both pretrained and from-scratch models, the authors find that data diversity—specifically, the ratio of exhaustive compositional samples—plays a crucial role in generalization, rather than sheer data quantity. Their findings suggest that compositional representations emerge only under conditions of sufficient data diversity, where representations become more structured and linearly organized in latent space, facilitating efficient generalization.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We will add clarifications to the updated manuscript.\n\n**Clarify of definitions**\n\nIn our terminology, “concepts” refer to interpretable properties of the data (e.g., colour, shape, size), while “features” refer to the latent representations extracted by the model. \n\n**Investigating non-uniform distributions**\n\nThe goal in our setup was to ensure uniform coverage over observed combinations to avoid introducing imbalances across concept values. While certainly interesting, we are careful in complexifying the data setting in a way that makes the problem harder.\n\n**Evaluating language models and VLM.**\n\nExtending to language or vision-language models is challenging. Autoregressive models lack comparable representations. For non-autoregressive ones like CLIP, we probed the vision encoder; probing the text encoder is harder, as it requires generating many prompts to describe concept values. Nonetheless, we consider this an important direction for future work.\n\n**Transformer-based vision models.**\n\nDue to space constraints in this response, we couldn’t include the results for the vision transformer here. Another reviewer also expressed interest in the ViT’s performance--please refer to the table in the response to reviewer `TQRD`.\n\n**Related study by Dorrell et al. (2024).**\n\nThank you for pointing us to this excellent reference--we will be sure to discuss it in more detail. In short, this work, as well as the work it builds on (Whittington et al., 2023), characterises when the minimal-energy non-negative solution is modular, i.e when each neuron responds to a single task factor. We believe both this framework and ours can be viewed in a similar light. The key distinction lies in the goal: while their focus is on the structure of solutions over the observed data, our work studies whether representations can be predicted from their constituent concept values under novel combinations in the test set, and how data factors influence the structure of representations.\n\n**Simplicity bias and spurious correlations, underrepresented concepts**\n\nOur reasoning is based on prior observations in the debiasing literature. When training data exhibits underspecification--i.e., when multiple hypotheses are consistent with the data--a model may favour simpler hypotheses. In such cases, it may learn to generalize to majority groups and memorize the underrepresented groups [a]. In compositional generalization settings, this happens when some combinations are missing entirely (as in OOD splits), and models default to entangled solutions instead of learning disentangling concepts in their representation. Concepts, especially in the real world, are highly co-varying in some combinations and not in others, even if the observed concept pairs are not entangled. Because of this sampling bias, models may exploit such spurious correlations.\n\n[a] Sagawa, Shiori, et al. \"An investigation of why overparameterization exacerbates spurious correlations.\" ICLR 2020.\n\n**In Section 3.1, formula for unique images**\n\nIn Section 3.1, we were referring specifically to the number of training combinations observed in the $(n,k)$ framework, but we will clarify this distinction in the revised text.\n\n**Figure 3: boundary value of $k$**\n\nIf we understood your question correctly: The boundary value of $k$ corresponds to $n-1$, that is, training on all but one combination per concept value. \n\n**Figure 3: For maximum target classes, does the improvement hold for lower $n$?**\n\nYes, and we believe Figure 10 in the appendix demonstrates this: for fixed $k$, increasing $n$ lowers generalization performance; conversely, for fixed $n$, increasing $k$ improves it.\n\n**Figure 4: What is the $y$-axis measuring? Also, why does FSprites exhibit lower values in general?**\n\nThe $y$-axis in Figure 4 shows OOD accuracy. FSprites performs worse than other datasets, likely due to its more complex and visually similar shapes, which are harder to disentangle. Also, orientation may not align well with shape features, encouraging memorisation of joint shape-orientation co-variations instead of separate concept learning.\n\n**Figure 7: Does the gap indicate that the retrained model demonstrates better compositional generalization performance than a from-scratch model, or does it reflect the limitations of the pretrained model?**\n\nIt indicates both. The retrained model benefits from better compositional generalization due to exposure to more diverse or better-structured training data. However, the presence of a gap also reflects the limitations of the pretrained models to generalize compositionally.\n\n**Linear Probing in Section 5: What is the $k$ value used in the linear probing analysis? How does it compare to the threshold required for compositional representations to emerge?**\n\nWe varied $k$ from 2 to $n-1$, showing only aggregated results across datasets. While the exact threshold varies, $k=2$ is theoretically the minimum needed per dataset.", "claims_and_evidence": "The claims presented in the paper are well-supported by experimental results and clearly articulated. The authors systematically control for dataset characteristics and demonstrate that compositional generalization is not merely a function of dataset size but rather of diversity in compositional coverage. Their analysis of pretrained models further substantiates their claims, revealing that large-scale vision models do not inherently develop compositional generalization unless trained on sufficiently diverse data.", "methods_and_evaluation_criteria": "The methods employed in the study are appropriate for assessing compositional generalization in vision models. The authors introduce a systematic protocol that varies dataset size, concept values, and training combinations while keeping other factors controlled. Their use of synthetic datasets allows precise manipulation of compositional factors, ensuring that their results are not confounded by spurious correlations. The evaluation criteria, including out-of-distribution (OOD) accuracy and probing latent representations, align well with the problem at hand and provide clear insights into model generalization.", "theoretical_claims": "The paper does not focus on proving new theoretical results but instead builds on existing theoretical insights into compositional learning. The experimental findings align with prior work on neural network biases and structure in latent space, supporting claims about the importance of data diversity for compositional generalization. The authors also introduce a theoretical analysis showing that only a small number of compositional samples are required for perfect generalization if the feature space is structured compositionally, which is empirically validated in their experiments.", "experimental_designs_or_analyses": "The experimental design is rigorous and well-structured, effectively isolating key variables affecting compositional generalization. The study employs controlled datasets where specific combinations of concepts are systematically included or excluded from training, allowing precise analysis of generalization behavior. The comparison between from-scratch models and large-scale pretrained models provides valuable insights into the role of pretraining. However, it would be beneficial to extend the analysis to other modalities, such as language models or vision-language models, to examine whether similar principles apply across domains.", "relation_to_broader_scientific_literature": "The paper is highly relevant to researchers studying generalization in machine learning, particularly in vision and compositionality. It also holds significance for neuroscience and cognitive science communities, as it provides insights into how structured learning environments influence compositional learning. The findings can inform experimental design in cognitive science by emphasizing the importance of exhaustive combinatorial sampling. Additionally, the study is valuable for researchers working on dataset construction and training strategies, as it suggests guidelines for optimizing training data diversity to improve generalization.", "essential_references_not_discussed": "A related study by Dorrell et al. (2024) suggests that modularity in latent representations arises from certain input distribution characteristics. This work may provide additional theoretical grounding for the claim that data diversity enhances compositional generalization by promoting structured latent spaces. Including a discussion on how their findings align or contrast with Dorrell et al.’s results would strengthen the paper’s positioning within the literature.\nReference: Dorrell, Will, et al. \"Don't Cut Corners: Exact Conditions for Modularity in Biologically Inspired Representations.\" arXiv preprint arXiv:2410.06232 (2024).", "other_strengths_and_weaknesses": "The paper is well-written and clearly structured, making it an enjoyable read. The authors conduct an extensive set of experiments that provide strong empirical support for their claims. A notable strength is their systematic approach to controlling dataset characteristics, which allows them to make precise claims about the role of data diversity. However, one potential limitation is the exclusive focus on vision models—extending the study to multimodal or language models could provide a more comprehensive understanding of compositional generalization. Additionally, it would be interesting to explore whether the findings hold for different architectural choices, such as transformer-based vision models.", "other_comments_or_suggestions": "1. Clarify Definitions: The paper would benefit from a clearer distinction between \"features\" and \"concepts.\" Are features referring to learned latent representations, while concepts are explicit or implicit properties of the stimuli?\n2. Further Exploration of Takeaway 4.1: Consider investigating non-uniform distributions of n−k concept combinations to determine whether concepts that exhibit lower degradation in uniform settings maintain robustness under more naturalistic data distributions.\n3. Additional Experimentation: If space allows, evaluating whether the results extend to language models or vision-language models would strengthen the paper's impact.", "questions_for_authors": "1. **Compositionality and Simplicity Bias**  \n   - In Section 2, why does the simplicity bias lead to model preference for spurious correlations? And why is this issue more pronounced for underrepresented concepts?\n\n2. **Dataset Combinatorial Calculations**  \n   - In Section 3.1, when calculating the number of possible unique images, should the formula be \\( n \\times k + k \\times (n-k) \\) instead?  \n   - Since we consider \\( n \\times k \\) for the first dimension while the remaining \\( n-k \\) concept values for the second dimension should also be combined with possible \\( k \\) values from the first dimension.\n\n3. **Figure 3 Analysis**  \n   - What is the boundary value of \\( k \\) for achieving the observed compositional generalization improvements?  \n   - Additionally, for fixed maximum target classes, does the improvement hold for lower \\( n \\) values?\n\n4. **Figure 4 Clarification**  \n   - What is the y-axis measuring—OOD accuracy or drop in performance?  \n   - Also, why does FSprites exhibit lower values in general? What factors make it more challenging compared to other datasets?\n\n5. **Figure 7 Interpretation**  \n   - Does the gap indicate that the retrained model demonstrates better compositional generalization performance than a from-scratch model, or does it reflect the limitations of the pretrained model?\n\n6. **Linear Probing in Section 5**  \n   - What is the \\( k \\) value used in the linear probing analysis?  \n   - How does it compare to the threshold required for compositional representations to emerge?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AM5hl47yI4", "reviewer": "Reviewer_hh3A", "summary": "This paper examines whether data scaling improves compositional generalization in vision models, emphasizing the role of data diversity. Through controlled experiments with synthetic datasets, the authors show that models can achieve compositional generalization, but this ability depends on training data diversity rather than quantity.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We will clarify the raised points in the revised manuscript. \n\n**L223: What is the significance of reporting average accuracy per epoch? Does it effectively reflect the model's final compositional generalization capability?**\n\nThank you for pointing this out. We should have stated this more clearly. The metric reported in Line 223 corresponds to the best-performing epoch (i.e., oracle model selection), not an average over epochs. Specifically, we track the average accuracy across both concepts at each epoch and report the highest value observed during training. This provides an estimate of the *upper bound* on achievable performance under ideal model selection. We chose this evaluation to emphasize that even with perfect model selection, the performance gap between ID and OOD is present.\n\n**The use of two independent classification heads to predict each concept in a composition is debatable (Line 230)**\n\nIf we understood your question correctly, by similarity-based, do you mean a CLIP-like objective? Our current design mirrors the supervision used in CLIP and standard classification settings, up to norm-rescaling of the features and weight vectors. The main difference is that our setup assumes a closed-vocabulary setting, and we believe that using two independent classifiers is appropriate since the concepts are independent. \n\nWe would be happy to hear more about alternative formulations or suggestions on how to frame this classification problem!\n\n**The authors claim to have evaluated three probe architectures (Line 255), but these are not mentioned in the experimental results.**\n\nOur main goal was to evaluate the pre-trained models' representations as charitably as possible. We did evaluate multiple probe architectures, but our main text only reported the best-performing configuration.\n\nIn response to this comment, we now include the average testing accuracies on both concepts across all datasets across 4 settings of data diversity (% of combinations in training), comparing linear probes and MLPs. MLPs most often performs better, but not universally across the datasets. The first number is the linear probe and the second is the MLP ([512, 512]):\n\n| Model | 25.0%| 50.0%| 75.0% | 93% |\n|----|---|---|----|----|\n| ResNet50 IN1K | 0.59 / 0.55 | 0.67 / 0.65 | 0.75 / 0.75 | 0.79 / 0.82 |\n| DINOv1 ResNet50 | 0.60 / 0.67 | 0.71 / 0.80 | 0.76 / 0.88 | 0.80 / 0.92 |\n| DINO ViT-L | 0.68 / 0.70 | 0.78 / 0.83 | 0.84 / 0.91 | 0.86 / 0.95 |\n| CLIP ViT-L | 0.61 / 0.64 | 0.70 / 0.74 | 0.75 / 0.79 | 0.76 / 0.84 |\n\n**Additionally, the relevance of evaluating architectures with only minor differences (e.g., one hidden layer or activation function) remains debatable.**\n\nWhile the architectural variations may appear small, we found that going beyond two hidden layers didn't result in substantial increase in performance. Below, we provide comparisons using the FSprites dataset (again for 4 diversity settings); we used these results as justification for focusing on simpler configurations in this work.\n\n| Model / Probe | 25%  | 50%  | 75%  | 93%  |\n|--------------------|------|------|------|------|\n| ResNet50 IN1K (Lin)| 0.38 | 0.52 | 0.57 | 0.66 |\n| ResNet50 IN1K (MLP)| 0.35 | 0.47 | 0.55 | 0.64 |\n| ResNet50 IN1K (Deep)|0.34 | 0.46 | 0.55 | 0.63 |\n| DINO R50 (Lin) | 0.40 | 0.56 | 0.64 | 0.74 |\n| DINO R50 (MLP) | 0.41 | 0.62 | 0.72 | 0.79 |\n| DINO R50 (Deep) | 0.41 | 0.60 | 0.71 | 0.79 |\n| DINO ViT-L (Lin) | 0.44 | 0.63 | 0.78 | 0.81 |\n| DINO ViT-L (MLP) | 0.46 | 0.65 | 0.81 | 0.90 |\n| DINO ViT-L (Deep) | 0.45 | 0.64 | 0.81 | 0.90 |\n| CLIP ViT-L (Lin)   | 0.41 | 0.56 | 0.68 | 0.74 |\n| CLIP ViT-L (MLP) | 0.43 | 0.60 | 0.73 | 0.85 |\n| CLIP ViT-L (Deep) | 0.42 | 0.59 | 0.72 | 0.82 |\n(Lin = linear probe, MLP = [512, 512], Deep = [512, 512, 512])\n\n**Does the number of training samples remain constant while increasing training data diversity? (Sec. 4.1)**\n\nYes, we do control for this factor, although we acknowledge that it was not made sufficiently clear in the text. All experiments in Section 4.1 (except for Figure 4) use a fixed dataset size of 60,000 samples where available; for Shapes3D and CMNIST we used 30,000 samples.\n\n**Does the authors consider conducting experiments on real-world datasets?**\n\nWe considered real-world datasets with known generative factors, but they are currently limited in compositional complexity. Moreover, real-world datasets offer sparse coverage of concept values, making controlled compositional generalization experiments intractable. We are open to suggestions for real-world datasets that offer controllability.\n\nFinally, although our datasets are synthetic, Section 5 evaluates pre-trained models used in real-world practice and tests how pre-trained vision models cope in compositional settings.\n\n**Line 318, what does performance specifically refer to?**\n\nBy performance, we refer to the mean classification accuracy over the considered concept values.", "claims_and_evidence": "Basically yes.", "methods_and_evaluation_criteria": "The experimental setup is somewhat unconvincing:\n1) The authors report the average accuracy across all concepts at each epoch (Line 223). What is the significance of reporting average accuracy per epoch? Does it effectively reflect the model's final compositional generalization capability?\n2) The use of two independent classification heads to predict each concept in a composition for evaluating compositional generalization is debatable (Line 230). Other setups, such as a similarity-based approach, should be considered to improve comprehensiveness of evaluation.\n3) The authors claim to have evaluated three probe architectures (Line 255), but these are not mentioned in the experimental results. Additionally, the relevance of evaluating architectures with only minor differences (e.g., one hidden layer or activation function) remains debatable.\n4) Does the authors isolate a single variable in their experiments to ensure a controlled comparison? For instance, Does the number of traing samples kept constant while increasing training data diversity? (Sec. 4.1)\n5) Does the authors consider conducting experiments on real-world datasets?", "theoretical_claims": "Yes.", "experimental_designs_or_analyses": "Yes, please see \"Methods And Evaluation Criteria\"", "relation_to_broader_scientific_literature": "This paper explores the impact of data on the compositional generalization ability of vision models, a topic of broad relevance. As deep learning increasingly depends on large-scale data pretraining, optimizing data collection and filtering with compositional generalization in mind can significantly enhance training efficiency and model performance.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "As mentioned above, this paper's topic has both theoretical and practical significance, but the experimental setup needs improvement.\nAdditionally, some parts of the writing require clarification. For example, on line 318, what does performance specifically refer to?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See `Methods And Evaluation Criteria` and `Other comments Or Suggestions`.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N8TrNFJJeR", "reviewer": "Reviewer_Nv93", "summary": "This paper studies whether compositional generalization emerges in vision models trained from scratch and large-scale datasets. The authors design simple experiments where two factors, i.e., shape and color, control the dataset. The authors explore different levels of compositions in the training set and offer some discussions based on the results.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thorough review. We will incorporate these suggestions in the final version of the manuscript.\n\n**\"large pretrained models\" can be very misleading**\n\nWe agree about emphasizing the vision aspect. However, our models are large within the vision domain—for example, DINO ViT-L contains 300M parameters. Would it be clearer if we added a footnote explaining our criterion for \"large-scale models\" in the vision context?\n\n**L90 is not very clear what it means**\n\nWhat we meant by this is that prior work usually considers fixed compositionality settings where the scale of compositions is fixed. \n\n**It is challenging to see the relation between this work and others mentioned in related work, especially in the last four paragraphs.**\n\nWe aimed to cover prior work relevant to (compositional) generalization and vision, and highlight challenges. In brief:\n- *Compositionality and vision models:* We discuss existing limitations in vision models, e.g. under spurious correlations and how it affects generalization. \n- *Scaling and emergent abilities:* Prior work shows scaling can improve performance in (compositional) generalization tasks. We build on this by systematically varying data and model scale across multiple axes.\n- *Improving compositionality:* While others introduce architectural or loss-based modifications, we evaluate whether modern scaling practices alone suffice for compositional generalization.\n- *Representation learning:* Our approach focuses on the structure of learned representations--specifically, how they support generalization to unseen compositions.\n\nWe will revise the text to more clearly explain how our work relates to prior work.\n\n**\"ID data\" is not explained.**\n\nWe agree that this terminology is confusing without an explicit mention of it. By ID data we are referring to the data sampled from the training region of combinations defined by the $(n, k)$ framework we introduced.\n\n**L186: it is unclear why authors introduce more variants to the compositions while claiming that the paper limits its compositionality as 2 for simplicity.** & **Is there any reference to follow and justify whether the simplification is reasonable and effective for its extension?**\n\nWe wanted to emphasize that the total dataset size is not restricted by the two concepts we study. While the specific pair of concepts we focus on is limited in size, other concepts in the data can vary freely. This allows us to study a more realistic setting for compositionality. For example, in the real world, when we restrict the pair (color, animal), we know that blue and green pandas aren't common in datasets. However, the pairs that are present (e.g., white pandas) can still vary along many other dimensions, such as size, pose, location, and other attributes.\n\nAdditionally, we focus on the two-concept case because we believe it is the simplest compositional setup that still presents challenges for vision models. Given that the total number of concepts is greater than 2, applying further restrictions would only make the task harder. Therefore, we view the two-concept focus as a reasonable simplification.\n\n**Zerroug et al. (2022), Zhou et al. (2023), and Stone et al. (2017) are essential related works not discussed.** & **It is unclear what the difference is in the empirical findings from existing work**\n\nThank you for these references. We briefly summarize how these works relate to ours:\n\n- Zerroug et al. focus on visual reasoning and data-efficiency, combining representation learning and reasoning over them with repeatedly sampled (i.e. the train and test set has overlap), or procedurally-generated tasks (visual primitives aren't overlapping between training and testing). In contrast, our work centers on classification tasks that rely on strong vision representations and examines generalization when seen concept values at test time are combined in unseen ways.\n- Zhou et al. study data compositionality in NLP using primitive-level data augmentation to enhance generalization. While we find a similar conclusion that diverse training data is beneficial, our focus on the vision domain introduces challenges not encountered in language tasks (e.g. making primitive-level augmentations is simple in language, but not in vision; and scaling ID data quantity isn't sufficient to improve generalization).\n- Stone et al. modify architectures and losses with object masks to promote compositionality, which relies on injecting priors and using dataset-specific annotations. Our approach, however, investigates compositional generalization from a data-scaling perspective.\n\nWe believe our work isolates a specific and underexplored aspect of compositional generalization in vision. Our work, to our knowledge, is the first to consider the scale of compositions across multiple dimensions: types of concepts, number of concept values, number of datapoints used in training, types of datasets, and densities of compositions in a controlled manner.", "claims_and_evidence": "- “large pretrained models” can be very misleading. I would suggest the authors always emphasize “vision models pretrained with large-scale datasets”.\n- L90 “settings where no restrictions on the scale of compositions are imposed” is not very clear what it means exactly.\n- It is challenging to see the relation between this work and others mentioned in related work, especially in the last four paragraphs. This further makes it difficult to judge the contribution of this work.", "methods_and_evaluation_criteria": "- “ID data” is not explained at all in the paper and is used many times.", "theoretical_claims": "yes", "experimental_designs_or_analyses": "- L186, “contains additional concept dimensions (like position, orientation, or background) with |Ci| possible values each. For instance, in a dataset with two additional dimensions having |C1| = 8 and |C2| = 12 possible values respectively”, it is unclear why authors introduce more variants to the compositions while claiming that the paper limits its compositionality as 2 for simplicity.", "relation_to_broader_scientific_literature": "The key contributions of this paper could facilitate compositionality learning, a core component of human-like AI.", "essential_references_not_discussed": "- Zerroug et al., A Benchmark for Compositional Visual Reasoning, NeurIPS 2022, shared a highly similar topic.\n- Zhou et al., Data Factors for Better Compositional Generalization, EMNLP 2023. Although the paper focuses on NLP domain, it also discusses compositionality from a data perspective.\n- Stone et al., Teaching Compositionality to CNNs, CVPR 2017 has demonstrated that training from scratch offers better compositionality generalization.", "other_strengths_and_weaknesses": "**Strengths**\n\n- The introduction is well-written and easy to follow.\n- The studied topic is very important to human-like AI development.\n\n**Weakness:**\n- “in this work, we focus specifically on pairwise compositions” Is there any reference to follow and justify whether the simplification is reasonable and effective for its extension?\n\nMy major concerns are:\n 1. It is unclear what the difference is in the empirical findings from existing work.\n 2. It is unclear whether the simple experimental setting in this paper can scale up to compositions with various compositionality dimensions in the real world.", "other_comments_or_suggestions": "L95, “Vision-language models face specific compositional challenge”, would be great to explain a bit more in texts.\n\nL129, “face their own limitations in handling complex visual compositions.”, it would be great to explain a bit more in texts.\n\nL215, \"We study this through two main sets of experiments: pre-training, where we train models from scratch (Section 4), and evaluating pre-trained foundation models’ (FM) compositional abilities…” is very confusing and is probably just a typo? I suggest just rephrasing “pre-training, where we train models from scratch” as “trained from scratch”.", "questions_for_authors": "Please see comments and suggestions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5wUz472N5i", "reviewer": "Reviewer_TQRD", "summary": "The article investigates how the compositional generalization capacity of vision models is related to the scale and quality of the training dataset. The authors main contribution is the proposal of a systematic way to manipulate training and testing dataset splits in order to evaluate how data diversity promotes generalization. Additionally, they introduce a novel dataset composed on abstract shapes called Funny-Sprites which provides a more comprehensive benchmark than the more standard dSprites. The authors perform a thorough evaluation using both pre-trained models and models trained from scratch. Their findings seem to indicate that data diversity is crucial for compositional generalization, yet cannot achieve perfect compositionally at the scales they investigated.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your thorough review. We will incorporate the feedback into the updated manuscript.\n\n**They do not test a large variety of models when training from scratch (only ResNet-50 is reported), but they claim to have used other baselines in the text. It would be good to include some of these results, even if they are not used in all the conditions. One way to justify not including all test conditions could be to start with simple ones and, should they fare worse than ResNet-50, exclude them from subsequent evaluations.**\n\nThank you for this suggestion. We agree that including results from additional models would strengthen the completeness of our empirical analysis. We include some of the results of vision transformers and compare them with ResNet50 here by varying the % of combinations for CMNIST (since the model achieves comparable results with 80% of combinations); additionally, we highlight that even under increased number of training points, vision transformer still underperforms (all experiments assumed 60,000 datapoints, except for the last row):\n\n| Dataset       | Model     | % Combinations ($k/n$) | OOD Accuracy |                                   |\n|--------------|-----------|------------------|--------------|-------------------------------------------|\n| CMNIST       | ResNet50  | 80% (8/10)        | 95.1         |                                           |\n|              | ViT       | 80% (8/10)        | 94.5         |                  |\n|              | ResNet50  | 40% (4/10)        | 66.0         |      |\n|              | ViT       | 40% (4/10)        | 71.0         |      |\n| FunnySprites | ResNet50  | 92% (13/14)       | 80.1         |     \n|              | ViT       | 92% (13/14)       | 66.0         |                                           |\n|              | ViT (120K data)       | 92% (13/14)       | 57.3         |  |\n\nWe attempted to be as charitable to ViT architectures as possible: we varied the patch size (8 and 16), ViT depth (4, 6, 8), width of each layer (384, 512), number of heads (8, 12), MLP width (384, 512), and learning rate ($1e{-}5$, $1e{-}4$, $1e{-}3$). In the table above, we report the best-performing configuration for each setting. All the models were able to achieve 99.7% accuracy on the ID data, but ViT typically underperformed on the testing data.\n\n**My main suggestion is regarding the use of the term compositionality.**\n\nThank you for raising this point. We agree that the term “compositionality” may not have been the most precise description for the notion we are exploring. Our motivation for using it stemmed from the natural emergence of linear factorization in the from-scratch setting with increased data diversity. However, we acknowledge that compositional structure does not have to be linear. Instead, we propose to use the term “linear factorization” as a specific instantiation of compositionality. We will clarify this in the text.\n\n**In fact, the authors’ own findings contradict the claim that these are compositional representations, since they fail to generalize systematically to unseen combinations. [...] Thus the representations must either be unstable or such a representation is not enough to ensure compositionality.**\n\nOur goal is not to claim that the representations are fully compositional, but rather to study to which extent they are compositional, instantiated in our case as exhibiting a linear factorization. The failure to generalize to unseen combinations highlights the limitations of this structure in the models we study. If our writing gave the impression that we claim full compositionality, we’d be happy to clarify--please do let us know if a particular phrasing seemed misleading.\n\n**What happens if in the analysis in Section 4.2, instead of training the decoder and determining PCA directions with both training and testing combinations, the authors only use the training ones and project the test combinations with the resulting model/PCA? Do the test combinations clearly separate from the training ones? Is the nice lattice structure observed in the projections disrupted?**\n\nThe displayed points in Section 4.2 are from the OOD data, i.e., unseen combinations. Specifically, we sample a 2×2 grid of combinations that share concept values (i.e., (i, j), (i, k), (l, j), (l, k)), all from the test set. We will clarify this procedure in text. \n\nRegarding the suggestion to perform PCA only on the training data and project the test points using the resulting model: in this case, the lattice structure is not preserved. Intuitively, this is because combinations of concept values can span up to a $2n - 1$-dimensional space. Even if the training data admits a linear factorization, projecting unseen combinations using a PCA fit on the training data alone does not guarantee that the structure will be preserved since the model is trained to classify concepts. This is the reason why we used a smaller set of combinations for visualization purposes.", "claims_and_evidence": "In general, the authors make a really good effort at substantiating their claims with solid empirical evidence. They evaluate several variations of their test protocol, varying the number of values each concept can take, and the number of combinations each value can appear in. This allows them to manipulate both the diversity of the data and, potentially, the qualitative nature of the missing combinations (though they don't really explore this angle too much).\n\nThey do not test a large variety of models when training from scratch (only ResNet-50 is reported), but they claim to have used other baselines in the text. It would  be good to include some of these results, even if they are not used in all the conditions. One way to justify not including all test conditions could be to start with simple ones and, should they fare worse than ResNet-50, exclude them from subsequent evaluations.", "methods_and_evaluation_criteria": "For the most part, yes. The datasets used are standard in the compositional generalization literature. Additionally, they introduced a new dataset called Funny-Sprites which contains abstract, non-regular shapes. They also analyze the learned representations using several metrics to understand how they are structures (i.e. are they more linear, can concept values be decoded, do they lead to better generalization).", "theoretical_claims": "I only skimmed through them since they are in the appendix, but looked fine. The claims at least make intuitive sense.", "experimental_designs_or_analyses": "Yes, I did. They appear correct to me.", "relation_to_broader_scientific_literature": "The paper is well situated within the literature. The authors make a substantial effort to cover the related literature and identify where their contribution lies, and in my opinion they succeed at this.", "essential_references_not_discussed": "No that I can remember. The authors are very thorough in their discussion of related work.", "other_strengths_and_weaknesses": "The authors explore an important question: how does data diversity influence compositionality, which is important since one popular claim is that scaling data is enough to achieve strong generalization in AI systems. These findings suggest that this is in fact not the case.", "other_comments_or_suggestions": "My main suggestion is regarding the use of the term compositionally. Throughout the paper the authors use it as equivalent to orthogonal/linear representations, yet this is not exactly correct. Rather, having a representation as the one in equation 25 is one way, but not the only way, to implement a compositional representation. Indeed, it is a very popular view of compositionally in DL. However, it is perfectly valid to imagine a representation that does not conform to such notions of linearity/orthogonality, but is still compositional. The only requirement is that whatever mechanism is doing the composition can handle such representation. Thus, I would suggest that the authors make it clear that this is the implementation of compositionally they are exploring. Moreover, the main issue is whether such a representation is stable in the models being studied. \n\nIn fact, the author's own findings contradict that claim that these are compositional representations, since they fail to generalize systematically to unseen combinations, a fact that authors point to briefly when they state that they are not \"ideal\". Note that in the Philosophy of Mind literature compositionality implies systematicity; thus if a model is not systematic it cannot be compositional (or \"ideally compositional\", see Fodor and Pylyshyn). Thus the representations must either be unstable (they deviate from the expected position in latent space for that combination) or such a representation is not enough to ensure compositionality.", "questions_for_authors": "What happens if in the analysis in section 4.2, instead of training the decoder and determining PCA directions with both training and testing combinations the authors only se the training ones and predict/project the test combinations with the resulting model/PCA? Do the test combinations clearly separate from the training ones? Is the nice lattice structure clearly observed break? I would like to see this analysis.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: ## Overall
This is an interesting work. Reviewers were initially on the borderline (5, 2, 2, 4), one of the reasons being the lack of large-scale datasets, however the authors added MPI3D and scores raised to (5, 3, 3, 4) during reviewer discussion. I think the authors did a great job on their rebuttal and this work should be accepted.

## Summary
The authors find that data diversity plays a more important role in compositional generalization than data quantity. The authors achieve these conclusions on existing datasets and a new proposed one both on pre-trained models and training from scratch.

## Reviews
This work was initially assessed as borderline (3, 2, 3). Reviewers were asked to engage in discussion to reach a final agreement before April 14th AoE. To reach a clear consensus, they were instructed to discuss the following points and attempt to arrive at a clear accept/reject score:

ySH5 maintained that the technical contribution was limited due to the similarity to HyperFormer. The reviewer was asked to read the authors' latest reply, which argued that the Complementary Learning Systems inspiration and the resulting design choices (task dictionary, hypernetwork as a soft connection) offered a novel perspective for continual learning. The reviewer was then asked if they saw any merit in this argument and whether the overall system, even if built upon existing components, was greater than the sum of its parts in addressing the continual learning problem.

tFmB pointed out that block-wise separation was not a substantial conceptual breakthrough. Given the experimental results demonstrating improved performance with the block-wise design, the reviewer was asked if they believed the empirical gains could justify the complexity introduced.

eK4Y was asked for their opinion on the two points above and whether they thought there was enough contribution to grant acceptance.
Decision: Accept (poster)