Title: Scaling can lead to compositional generalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: compositionality

compositional generalization

scale

systematicity

generalization
Abstract: Can neural networks systematically capture discrete, compositional task structure despite their continuous, distributed nature?
The impressive capabilities of large-scale neural networks suggest that the answer to this question is yes.
However, even for the most capable models, there are still frequent failure cases that raise doubts about their compositionality.
Here, we seek to understand what it takes for a standard neural network to generalize over tasks that share compositional structure.
We find that simply scaling data and model size leads to compositional generalization.
We show that this holds across different task encodings as long as the training distribution sufficiently covers the task space.
In line with this finding, we prove that standard multilayer perceptrons can approximate a general class of compositional task families to arbitrary precision using only a linear number of neurons with respect to the number of task modules.
Finally, we uncover that if networks successfully compositionally generalize, the constituents of a task can be linearly decoded from their hidden activations.
We show that this metric correlates with failures of text-to-image generation models to compose known concepts.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates whether standard neural networks can achieve compositional generalization through scaling data and model size. It presents theoretical results suggesting that MLPs can approximate compositional task families with linear scaling in the number of neurons relative to task modules. Empirically, it demonstrates that scaling data and model size improves compositional generalization across various task encodings, provided the training distribution sufficiently covers the task space. The paper also shows that task constituents can be linearly decoded from hidden activations in models that successfully generalize compositionally, correlating this decodability with the success of image generation models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper offers a formal definition of compositional tasks.\n2. The authors conduct experiments across different task encodings and model sizes. \n\nWeaknesses:\n\n1. This paper lacks novelty. The problem it explores—whether scale can bring about compositional generalization—is not of significant importance to the industry. Moreover, we cannot gain any insights from it.\n2. The writing quality of this paper is very poor. The figures are poorly made, and the typesetting and writing are subpar, making it difficult to follow.\n3. The motivation of this paper is not clear enough. The abstract mentions that “there are still frequent failure cases that raise doubts,” but there is no more specific support in the introduction section provided. Moreover, it is not clear why this would lead to a focus on the issue of scale.\n4. The scale of the experiments in this paper is too small and lacks a clear experimental setting. The experiments are not extensive enough and are not convincing.\n5. The writing of the related work section is also unclear, making it difficult to understand the positioning of this work within the field.\n\nQUESTIONS:\n1. The paper mentions that underrepresentation of certain modules leads to failures. Could the authors provide more detailed analysis on the types of failures observed and potential strategies to mitigate them beyond ensuring sufficient training data coverage?\n2. How do the findings translate to real-world applications where task specifications are not fully known or are ambiguous? \n3. I believe that these conclusions should be verified in more popular scenarios, such as in language models, to see if given certain capabilities, a small model cannot solve a task but scaling up can. I suggest referring to the experiments in \"Compositional task representations for large language models\" to provide a more in-depth analysis.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nthe paper focuses on compositional generalization -- a whether a model can generalize correctly to new combinations of previously learned components.\n\nThe paper makes a very bold statement: just giving a model larger scale in terms of parameters and data is sufficient to learn compositional tasks -- without any explicit priors (say architectural priors) to learn compositionality.\n\nThe main contributions are:\n\n- Empirical results that compositional generalization emerges naturally as standard MLPs grow larger, provided the training data covers the compositional space of the task family.\n\n- A theoretical result showing that MLPs can approximate compositional task families efficiently, with neuron count growing linearly rather than exponentially with number of modules.\n\n- introduction of \"linear decodability,\" a metric that quantifies the extent to which compositional structure can be recovered linearly from internal activations\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n+ The experiments are carefully designed, showing that scaling standard MLPs indeed leads to compositional generalization under some conditions. \n+ A theoretical contribution showing that the number of neurons required scales only linearly with number of modules.\n- Rather artificial/synthetic compositional tasks. Not convincing that the main result of the paper would have more broad applicability.\n- The theoretical result is derived under a single hidden layer assumption. Very unclear if this would capture well what's happening with deeper or more complex architectures such as transformers.\n\nClarity\n+ The paper is well organized, and for the most part easy to follow. \n+ very good figures and other visualizations.\n- I would prefer to see more clear/formal definitions early on about key concepts such as \"compositional support\"\n- it seems that the paper assumes that all readers are familiar with Kolmogorov complexity\n\nSignificance\n+ The paper challenges a very prevalent assumption about the necessity of specialized architectures or explicit inductive biases for compositional generalization.\n- is it true however? I am not convinced because both the theoretical result and the empirical results are for very simplified settings. So it may be that the main \"message\" of this paper is challenged by others in the next 1-2 years. Of course this is not necessarily bad -- this is how research progresses.\n- the paper could contrast its insights to recent theoretical frameworks (such as the Coverage Principle or the Random Hierarchy Model -- see references below) that define conditions for compositional generalization. \n\nOriginality\n+ If the main result is true, the paper is definitely making a very original contribution: \"scale is all you need\" for compositional generalization\n- There are a couple of highly relevant papers that are not cited and discussed by the paper:\nhttps://arxiv.org/pdf/2307.02129 (the random hierarchy model)\nhttps://arxiv.org/pdf/2505.20278 (the coverage principle)\n\nQUESTIONS:\n1) I am not a mathematician but it seems to me that the theoretical contribution is rather informal. It appears less precise compared to say the analysis in Schug et al. where explicit theoretical conditions (connected and compositional support) and identifiability conditions were  formally proven. Do you agree? If so, can you clarify/refine the theoretical contribution?\n\n2) As I wrote earlier, a major concern is that the tasks are rather synthetic and the results only applicable to MLPs. Can you at least provide some insights -- why do you think that the same major result would be true say for transformers and for much more complex real-world tasks? \n\n3) I like what you propose about \"linear decodability\". However, it is unclear what linear decodability exactly captures about the structure learned by the model—particularly how it relates or contrasts with previously introduced measures such as \"coverage\" or \"synonymic invariance\" (see the earlier links I gave you)\n\n4) You suggest that scale along is enough for compositional generalization. However the paper that you cite (Schug 2024) shows that too much overparameterization can affect negatively compositional generalization. I cannot quite reconcile these two results/observations. Can you help?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates whether MLP neural networks can achieve compositional generalization through scaling alone, without compositionality enforced in the architecture. Using synthetic \"hyperteacher\" tasks where networks must compose modules to solve problems, the authors show that simply scaling data and model size leads to compositional generalization. The authors claim that the number of neurons needed for the generalizing solution scales linearly with task modules, making it asymptotically more efficient than memorization. They also find that compositional generalization emerges across different task encodings (identity, orthogonal, language, few-shot examples) and illustrate different shapes of the training data which are required to support compositional generalization. Finally, the authors find that models which successfully generalize can linearly decode task constituents from their hidden activations. The authors validate this on real image generation models, showing that diffusion models with better compositional abilities have more linearly decodable task representations.\n\nSTRENGTHS AND WEAKNESSES:\nS1: This paper addresses an important problem: Compositional Generalization. And the paper discusses this topic all the way from theory, to synthetic experiments to real experiments.\nS2: The paper is well formatted and easy to follow with well curated figures.\nS3: The results on compositional support are well illustrated.\nS4: The linear decodability result is often assumed but it's useful to see this tested both in synthetic and real image data.\n\nW1: **This is my main concern.** It is hard to determine whether, as the authors claim, “scale” alone leads to compositional generalization. Recently, several papers (albeit in different topics (e.g. https://arxiv.org/abs/2412.01003, https://arxiv.org/abs/2410.16531) has pointed out that it is often misleading that a scale-dependent effect is claimed when networks are trained with same number of steps, since wider networks are known in practice and in theory to learn faster. Would the authors clarify if all experiments are ran in equal flops settings? (This does not require more data for smaller models, but might need more epochs)\n\nW2: Another concern is that together with W1, it seems like the title is too general, (kind of the “attention is all you need” vibe), as it’s more the case that compositional support, data diversity, enough optimization *and* scale is needed for compositional generalization.\n\nW3: It would be much better if the experimental data is more explicitly described with examples.\n\nW4: The results are definitively useful and are well illustrated, but it is unclear to what degree these results are unexpected, surprising or actionable.\n\nQUESTIONS:\nQ1: Are the runs for different model sizes FLOPs matched? (equal compute).\n\nQ2: do you have any flops to loss plots which demonstrate how saturated the training is?\n\nQ3: Figure 3 is very interesting! Is Compositional support and Connected support both R^2 ~ 1 ?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the impact of data and model scale on compositional generalization by considering a teacher-student setup. Specifically, they consider compositional tasks and a hypernetwork teacher (that receives as input the specific compositional task identity). The student receives some function of the compositional task identity as well as the given input. The authors find that with increasing model size and data scale, the network improves in its generalization to novel combinations of task. They tie this to an argument that MLPs can efficiently represent such compositional functions and therefore prefer learning such a solution over memorizing solutions for each task combination. They then suggest that MLPs that can learn such compositional functions will have representations that enable decoding the task identity. Notably, they replicate this phenomenon in image generation models.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis paper is a neat demonstration of how large data and model size can enable compositional generalization. This is particularly intriguing as the authors consider a very similar scenario to the one investigated by Schug et al., who demonstrated that modularity can be sufficient for generalization --- thus demonstrating that modularity is not necessary with sufficient data. I also appreciated that the authors provided theoretical arguments for why this might be the case. While this study focuses on relatively simple networks, I think such a simplified scenario can be a strength in itself as it can provide a useful case study. Finally, I thought the paper was well written and easy to follow.\n\n**Weaknesses**\n\nTheorem 3.1 does not actually show that neural networks in practice learn a compositionally generalizing solution; rather it demonstrates that this is an efficiently learnable solution. This is an important gap between the theory and the practical scenario. That said, theoretically understanding deep neural networks trained with backpropagation is often quite complicated and so in my mind, this is an acceptable gap.  However, it would be nice to provide some justification for why Theorem 3.1 might be useful for understanding networks trained in practice, e.g. by drawing on the literature showing that even overparameterized neural networks often end up learning a solution with a sparse hidden layer (indicating that they might be more likely to learn a solution that can be parameterized with few hidden neurons) [e.g 1-3].\n\nNotably, all of the results related to scale are based on simple MLPs and relatively simple teacher networks. While I think this is an interesting scenario to investigate (as noted above), I do think the authors should more strongly qualify their findings based on this limitation, in particular by making the title more specific to the scenario they are investigating.\n\n**References**\n1. https://proceedings.mlr.press/v125/chizat20a.html\n2. https://proceedings.mlr.press/v99/savarese19a.html\n3. https://proceedings.neurips.cc/paper_files/paper/2022/hash/7eeb9af3eb1f48e29c05e8dd3342b286-Abstract-Conference.html\n\nQUESTIONS:\n- It is quite surprising to me that the linear decoder of the task identity doesn't work. Given that the hidden layer of the neural network is fairly large, I would imagine that even a random nonlinear embedding would allow for memorizing the different tasks with a linear decoder. Can you give me an intuition for why this might happen? Is it related to the fairly large regularization parameter associated with the decoder? Does the neural network actually end up collapsing the representation?\n- I think it would be important to scale back the generality of the claims (as noted in my comments above). An example title would be \"Scale leads to compositional generalization: a teacher-student perspective\" --- more broadly, just giving a sense of the specific setup in the title would be good. There are many ways to study compositional generalization and the impact of scale and it would be helpful to be specific about what your contribution here is.\n\nLIMITATIONS:\nYes (though see comments about scope of the work above).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}