Title: Scalable Evaluation and Neural Models for Compositional Generalization
Venue: NeurIPS 2025
Accepted: yes
Keywords: compositional generalization

compositionality

disentanglement

representation learning

computer vision
Abstract: Compositional generalization—a key open challenge in modern machine learning—requires models to predict unknown combinations of known concepts. However, assessing compositional generalization remains a fundamental challenge due to the lack of standardized evaluation protocols and the limitations of current benchmarks, which often favor efficiency over rigor. At the same time, general-purpose vision architectures lack the necessary inductive biases, and existing approaches to endow them compromise scalability. As a remedy, this paper introduces: 1) a rigorous evaluation framework that unifies and extends previous approaches while reducing computational requirements from combinatorial to constant; 2) an extensive and modern evaluation on the status of compositional generalization in supervised vision backbones, training more than 5000 models; 3) Attribute Invariant Networks, a class of models establishing a new Pareto frontier in compositional generalization, achieving a 23.43% accuracy improvement over baselines while reducing parameter overhead from 600% to 16% compared to fully disentangled counterparts.


Peer reviews:
comments: SUMMARY:
This paper unifies and extends existing compositional generalization evaluation protocols. The authors introduce a Compositional Similarity Index to measure the difficulty of generalization and use it to construct more principled dataset splits. They conduct extensive and comprehensive experiments to assess the compositional generalization capabilities of current visual backbones and further propose a new model.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The authors address key limitations in existing compositional generalization benchmarks and propose a unified evaluation methodology. This provides a more principled and practical way to assess existing approaches, which can significantly influence future research in this area.
2. The experimental evaluation is thorough and convincing, covering a wide range of model architectures and settings.

Weaknesses:
There are no major weaknesses.

QUESTIONS:
I have a question regarding line 97 of the paper. If the degree of freedom c=1, then test samples should share at most one attribute with the training set. However, in the example provided (Figure 2 a, top-left and bottom-left images), the two test images seem to share two factors. Doesn't this imply that c=2, rather than c=1? Could the authors clarify this?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors formalise visual compositional generalisation through a well-formulated dataset curation technique to construct a standard in visual compositional generalisation. The main idea is constructing a high-dimensional 'orthotopic' geometry composed of dataset partitions, then partitioned based on the complexity level of the measured compositional generalisation capabilities. The authors emphasise the efficiency of using orthotopic evaluation and its disentangled nature. After building this benchmark, the authors evaluate different architectures, pre-trained models, and more, training more than 5000 models. The authors conclude with constructing a new architecture, specifically endowed with inductive biases to improve visual compositional generalisation capabilities, and show its good performance and efficiency compared to its counterparts.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n1. Nice experimental setup, ablation studies, and case studies\n2. The flow of the paper is remarkable, understandable examples, showing clear attention to details\n3. Training 5000 models and exploring their compositional generalisation skills with a well-crafted dataset is an important contribution to the literature\n4. The newly introduced architecture is valuable in terms of its performance, efficiency, and increasing the number of works intersted in disentangled representations\n\n### **Weaknesses**\n\n1. The main contribution of the paper is already achieved in an uncited paper published in NeurIPS 2023 [1]. This paper already explores the idea of concept distance, which is equivalent to the compositional complexity proposed in this paper. Although the authors add a few novel details on top of the already present idea, the novelty of the paper is reduced.\n\n\n#### **References**\n\n1. Maya Okawa, Ekdeep Singh Lubana, Robert P. Dick, and Hidenori Tanaka. 2023. Compositional abilities emerge multiplicatively: exploring diffusion models on a synthetic task. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NIPS '23). Curran Associates Inc., Red Hook, NY, USA, Article 2182, 50173–50195.\n\nQUESTIONS:\n- I am willing to raise my score if the authors can convince me regarding the novelty of their work, or highlight a different aspect that I might have missed.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of compositional generalization in supervised vision models by introducing a formal compositional similarity index **c** i.e. the maximum number of shared factors between any training and test sample. Authors propose an orthotopic evaluation procedure that generates compositional OOD splits via iterative projection and pruning. The authors validate this framework by training models across different datasets and empirically confirm a “difficulty ladder” from extrapolation (c=0) to in-distribution (c=I). They also introduce Attribute Invariant Networks (AINs), which enforces invariance to irrelevant attribute transformations. Overall, this work offers an interesting analysis and simple approach to improve compositional generalization in vision models.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n1. **Interesting Formulation** - The paper provides a rigorous, unified formalism for compositional evaluation by defining 𝑐 and offering an orthotopic evaluation algorithm that projects and prunes data across all 𝑐-dimensional subspaces, it avoids the combinatorial blow-up.\n2. **Scale of Experimentation** - Over 5,000 training experiments on diverse architectures and six benchmarks lends strong statistical support to the paper’s findings.\n3. **Insightful Findings** - The empirical results verify that no model generalizes at c=0 (extrapolation) but nearly all succeed at c=I (in-distribution), with a monotonic accuracy ascent in-between, validating the proposed “difficulty ladder”.\n4. **Proposed Model** - The AIN architecture is a simple approach that enforces attribute-wise invariance in gradients, it promotes disentanglement without the expensive parameter cost of ED models. Moreover, AIN achieves substantial OOD gains over monolithic backbones with minimal parameter overhead.\n\n## Weaknesses\n\n1. **Disconnect between analysis and method design** - The paper's first part’s detailed empirical study of compositional difficulty does not clearly motivate specific choices in AIN’s architecture. It's unclear which insights from the analysis or the “difficulty ladder” directly informed the design choices in AINs.\n2. **Synthetic datasets only** - All evaluations use controlled, synthetic benchmarks (dSprites, I-RAVEN, Shapes3D, CLEVR, Cars3D), leaving the question open whether orthotopic splits and AINs transfer to complex natural-image data (e.g., COCO, ImageNet) with richer variability and no annotations for individual factors.\n3. **Fully supervised assumption** - Relies on known, clean generative‐factor labels. In many domains these are latent or noisy, and the paper does not address how to estimate or adapt c under such cases.\n4. **Scalability of attribute count** - Although AINs have far lower overhead than ED models, parameter and computational costs still depend on the number of attributes, which may become prohibitively large for real-world images involving many factors.\n\nQUESTIONS:\n1. Can the authors show that orthotopic splits and AINs can work on natural‐image datasets (e.g., COCO, ImageNet) without explicit factor annotations, and how well do they transfer to richer, uncontrolled data?\n2. How can your framework and AINs be adapted when generative‐factor labels are noisy, incomplete, or unavailable. Can c be estimated or optimized under such weak or semi-supervised settings?\n3. How did the empirical study of the compositional similarity index **c** and the “difficulty ladder” inform your specific architectural choices in AIN?\n\nLIMITATIONS:\nNo, the authors don't explicitly discuss the limitation of their analysis and method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nCompositional generalization is a key aspect for improving the generalization capability of AI models, but it remains underexplored in how to design principled and scalable evaluation protocols. This makes it challenging to develop models that can systematically generalize to unseen combinations of known concepts. To address these challenges, this paper introduces a unified framework for evaluating compositional generalization, featuring a compositional similarity index and the orthotopic split generation algorithm, which enables efficient and tractable creation of OOD splits with controllable difficulty. Extensive experiments on various vision tasks demonstrate the effectiveness of this evaluation framework. In addition, the paper proposes Attribute Invariant Network (AIN) — a new neural architecture that enforces attribute invariance by design, achieving strong compositional generalization with significantly less parameter overhead compared to fully disentangled models.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\n- **S1.** The paper clearly motivates the practical importance of compositional generalization and identifies concrete shortcomings in existing evaluation protocols.\n- **S2.** It introduces the compositional similarity index to systematically control task difficulty, along with an efficient orthotopic split generation algorithm that enables scalable and controllable evaluation settings.\n- **S3.** The experimental results demonstrate that the proposed evaluation framework is well-designed and aligns closely with the authors’ stated motivation.\n- **S4.** The proposed AIN architecture achieves strong compositional generalization while requiring significantly fewer additional parameters compared to fully disentangled models.\n\n### **Weaknesses**\n\n- **W1.** The paper assumes that the task-relevant generative factors are fully known in advance. However, in real-world scenarios, it is often difficult to identify these factors, and AINs require labeled generative factors for training. This may significantly limit their practical applicability.\n- **W2.** There has been a growing body of research on compositional generalization in vision, including recent work on object-centric representation learning that also considers attribute disentanglement [1, 2]. A more thorough discussion comparing AIN to these related directions would strengthen the paper.\n- **W3.** ED and AIN adopt a modular network structure that implicitly assumes each module specializes in a specific generative factor. Analyzing how well each module corresponds to the specific generative factors in practice would provide valuable insights and help clarify how AIN achieves its compositional behavior.\n- **W4.** Although the paper mentions training over 5,000 state-of-the-art models multiple times, the detailed results and concrete outcomes of these large-scale experiments are not clearly presented.\n\n[1] Neural Systematic Binder, ICLR 2023\n\n[2] Neural Language of Thought Models, ICLR 2024\n\nQUESTIONS:\n- **Q1.** In Algorithm 1, is the ${X_\\text{train}}$ in line 5 meant to be ${X_\\text{train}^{C}}$?\n- **Q2.** How might the orthotopic evaluation ladder and the compositional similarity index extend to domains where task-relevant generative factors are latent or only partially observable? Would the proposed framework remain tractable and robust under these more realistic conditions?\n\nLIMITATIONS:\nSome of the limitations are discussed in the Weaknesses and Questions sections.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}