Interaction Asymmetry: A General Principle for Learning Composable Abstractions

Authors: Jack Brady, Julius von Kügelgen, Sebastien Lachapelle, Simon Buchholz, Thomas Kipf, Wieland Brendel
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: disentanglement, compositional generalization, representation learning, object-centric learning, identifiability, unsupervised learning, out-of-domain generalization

Abstract:
Learning disentangled representations of concepts and re-composing them in unseen ways is crucial for generalizing to out-of-domain situations. However, the underlying properties of concepts that enable such disentanglement and compositional generalization remain poorly understood. In this work, we propose the principle of interaction asymmetry which states: "Parts of the same concept have more complex interactions than parts of different concepts". We formalize this via block diagonality conditions on the $(n+1)$th order derivatives of the generator mapping concepts to observed data, where different orders of "complexity" correspond to different $n$. Using this formalism, we prove that interaction asymmetry enables both disentanglement and compositional generalization. Our results unify recent theoretical results for learning concepts of objects, which we show are recovered as special cases with $n=0$ or $1$. We provide results for up to $n=2$, thus extending these prior works to more flexible generator functions, and conjecture that the same proof strategies generalize to larger $n$. Practically, our theory suggests that, to disentangle concepts, an autoencoder should penalize its latent capacity and the interactions between concepts during decoding. We propose an implementation of these criteria using a flexible Transformer-based VAE, with a novel regularizer on the attention weights of the decoder. On synthetic image datasets consisting of objects, we provide evidence that this model can achieve comparable object disentanglement to existing models that use more explicit object-centric priors.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1729071998757

thread_type:
official_review

<thread_object>
creator: reviewer_a7gp

<content>
summary:
This paper introduces the interaction asymmetry principle as a way to measure disentanglement and compositional generalisation, based on decomposing the latent space into blocks, and requiring non-zero partial derivatives between elements of the same block, but not between those of different blocks. The authors prove, in the appendix, that functions meeting this condition are disentangled, and discuss how this framework generalises some existing characterisations of disentanglement. Then they describe how this idea can be implemented as an additional loss term in a vision transformer VAE, and present results quantifying the degree of disentanglement using two metrics they devise, and compared to a slot attention decoder and a spatial broadcast decoder.

strengths:
The problem of measuring disentanglement and compositional generalisation is interesting.

The intro and high-level background is generally well-motivated.

The general principle of interaction asymmetry is, to my knowledge, creative in that it is unlike previous approaches to this problem. It is convincing and put to good use.

Eq 4.2 is a nice expression of how a function can, in some sense be decomposed into a disentangled and entangled portion.

weaknesses:
The discussion on the relationship between disentanglement and compositional generalisation could be improved by engaging with further existing work. While Montero et al (2020, 2022) claim to find no relationship between disentanglement and compositional generalisation, Higgins et al. (2016) and Esmaili et al. (2019) claim that their disentangled methods also enabled compositional generalisation, and Mahon (2024) claims to find an empirical correlation between disentanglement and compositional generalisation across multiple models and datasets. In fact, the position of Montero seems somewhat incongruent with your argument that they both share an underlying mechanism of interaction asymmetry.

I am slightly confused by some terminology: does ‘latent’ mean latent dimension or latent vector? e.g. on 197 "group of latents from the same or different slots" sounds like the former, but on line 409, it's in bold, suggesting a vector; does 'block' mean the same thing as 'slot'?

In Section 4.2, I don’t follow why $f$ and $\hat{f} \circ h$ being ‘sufficiently predictable’, in having all partial derivatives be fixed-degree polynomials, means that they are both equal on $Z_{cpe}$? It seems like there are three notions here, the interaction asymmetry condition, 'sufficiently predictable' and compositional generalisation.

The description of ARI on lines 478-9 is not the standard one as used e.g. in clustering (Vinh et al; 2009). Please state what the measure is exactly.

Comparing the different ablation settings in Table 1, it looks like having $\beta > 0$ makes a bigger difference than $\alpha > 0$, did you try a range of values for $\beta$? it could be that if you have the correct value of $\beta$, then $alpha>0$ makes no difference. It’s a bit odd to just report the value of these scaling parameters as > 0, it would be more insightful to report the exact values used.

Some information from Appendix I should be in the main paper, at least the number of dimensions in each slot.

**Refs**

Montero et al. (2020) "On the Role of Disentanglement in Generalisation".

Montero et al. (2022) "Lost in Latent Space: Examining failures of disentangled models at combinatorial generalisation"

Higgins et al. (2016) "beta-vae: Learning basic visual concepts with a constrained variational framework."

Esmaeli et al. (2019) "Structured Disentangled Representations"

Mahon et al. (2024) "Correcting Flaws in Common Disentanglement Metrics"

Vinh et al (2009). "Information Theoretic Measures for Clustering Comparison: Is a Correction for Chance Necessary?"

questions:
What does your work imply about the debate in the literature as to whether disentanglement and compositional generalisation are related?

How does  ‘sufficiently predictable’ imply compositional generalisation?

What are the exact values of $\alpha$ and $\beta$ in Table 1?

Also, a general point: This formulation of the interaction principle is binary, either the required derivatives are zero everywhere or they are not. Disentanglement, on the other hand, is normally quantified to a matter of degree, giving a particular model on a particular dataset a score in [0,1]. How easy would it be to extend your formulation to provide a score in [0,1], rather than {0,1}?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730359764793

thread_type:
official_review

<thread_object>
creator: reviewer_c6zh

<content>
summary:
This paper introduces a novel principle for learning composable representations, termed “interaction asymmetry.”, stating that interactions between distinct concepts are less complex than those within sub-parts of each concept. By formalizing these interactions as high-order derivatives of each concept to output images, the authors theoretically demonstrate that interaction asymmetry facilitates disentanglement and compositional generalization (though the latter is not empirically validated). Notably, their theoretical findings encompass previous approaches for achieving compositional generalization as specific cases (e.g., assumptions of at most 0th or 1st order interactions) and extend to accommodate more complex scenarios with higher-order interactions.To implement these theoretical insights, the authors propose regularizing cross-attention layers in standard transformer architectures. Empirical evaluations on two synthetic datasets show that the proposed method effectively captures distinct objects. The main contributions of the paper are as follows:
- The introduction of interaction asymmetry as a general principle for compositional representation learning, supported by a theoretical framework that links interaction asymmetry to disentanglement and compositional generalization.
- A practical method to enforce interaction asymmetry within transformers by regularizing cross-attention layers, with empirical results on synthetic datasets that demonstrate the method’s ability to capture distinct objects.

strengths:
- The paper is clearly written and easy to follow, with a well-motivated presentation that effectively derives theorems and implementations.
- The theoretical foundation of the paper is solid and sound, offering meaningful insights that contribute to our understanding of compositional representation learning.
- It is interesting and valuable that the proposed approach unifies existing work while also extending to more complex scenarios, e.g., including high-order interactions between concepts, highlighting its versatility and potential impact.

weaknesses:
- Although the theoretical foundation is solid, the experimental support is relatively limited. The empirical results are based solely on the simple Sprite and CLEVR6 datasets, raising questions about the generalizability of the findings to more complex scenarios.
- Theorem 4.3 states that achieving disentanglement requires ensuring at most $n$-th order interaction between slots. 
While theoretically sound, implementing this in practice as minimizing interaction without knowing the ground truth value of $n$, could be a limitation. For instance, each dataset (such as Sprite or CLEVR6) may have interactions of different orders, implying that the regularizer's weight to control interaction strength should be carefully adjusted for each case. If the Sprite dataset exhibits 0th order interaction, a stronger regularizer may be necessary compared to that in CLEVR6. Additional interpretation or obervations from the authors on this aspect would make the empirical support for their theory more comprehensive. 
- To implement Theorem 4.3 in a feasible way, the authors propose to minimize interactions between distinct slots through an attention regularizer in the transformer. However, it is unclear if this regularizer provides any advantage over the existing spatial exclusiveness inductive bias in slot attention encoders, which already minimizes interaction through slot-wise competition. Clarifying the unique benefits of the proposed regularizer compared to slot attention’s inherent mechanism would enhance the paper's contributions.
- Additionally, the authors claim that their generalized transformer structure with regularization is more flexible than slot attention-based methods (e.g., slot attention encoder + transformer decoder in L460–463). However, this claim is less convincing due to the following reasons: (1) To enforce invertibility, an additional KL term is required in the objective function, as shown in Table 1 (the value suggests that this term is crucial). 
Therefore, the proposed approach requires two additional terms (KL term and the regularizer term) and it may introduces additional burden on finding proper hyper-parameters. It is hard to tell if these inductive biases are favorable than architectural bias in slot attention methods.
(2) If my understanding is correct, the transformer decoder lacks a self-attention layer, using only cross-attention to avoid slot interactions. This specific design differs from the conventional transformer architecture, potentially limiting scalability. In contrast, slot attention-based methods, they can resume off-the-shelf models such as off-the-shelf encoder + slot attention encoder + off-the-shelf decoder (as in DINOSAUR [1], SLATE [2], LSD [3]), which might be more flexible in real-world applications.
- Finally, the proposed J-ARI metric may be misleading. The Jacobian of the decoder identifies regions most affected by each slot, but due to slot interactions, it may fail to segment objects accurately. For example, in Figure 3(B), the metallic sphere reflects the blue cuboid, causing the normalized slot-wise Jacobian to activate for the blue cuboid in the reflection area. However, this does not imply that the slot for the blue cuboid genuinely contains information about the metallic sphere. To assess whether composable abstractions are effectively captured, a downstream task like property prediction, as in LSD, may be necessary.

[1] Seitzer et al., bridging the gap to real-world object- centric learning, in ICLR 23. 

[2] Singh et al., Illiterate dall-e learns to compose, in ICLR 22. 

[3] Jiang et al., Object-centric slot diffusion, in NeurIPS 23.

questions:
- In L519–521, can the trade-off between interaction and model expressivity be effectively controlled through the parameter $\alpha$? For example, when increasing $\alpha$, does it lead to increased J-ARI but prevent reflections in mirrored objects in output images?
- As the Sprite dataset inherently exhibits 0th order interaction between objects, wouldn’t a spatial broadcast decoder (though the final alpha-blending layer does permit some interaction) serve as an architectural bias enforcing 0th order interaction? If so, should this setup represent the upper bound performance? Why does the proposed method outperform this approach?
- Although not a critical question, why did the authors choose to use learned queries instead of randomly sampled queries, which are typically used in Slot Attention?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730465651949

thread_type:
official_review

<thread_object>
creator: reviewer_qqbq

<content>
summary:
The paper titled "Interaction Asymmetry: A General Principle for Learning Composable Abstractions" presents the principle of "interaction asymmetry" to explain why some latent structures enable better generalization and disentanglement in representation learning. The core idea is that parts of the same concept interact more complexly compared to parts of different concepts, and the authors formalize this using block diagonality in higher-order derivatives of a generator function.

The main contributions of the paper include:

Theory: A formal proof that the principle of interaction asymmetry enables both disentanglement and compositional generalization. The authors extend previous results to more general generator functions, generalizing beyond earlier specific cases.
Method: An autoencoder-based approach that uses a Transformer architecture with an attention-based regularizer to limit interactions across slots, following the interaction asymmetry principle.
Empirical Evaluation: Demonstration on synthetic datasets that the proposed model can achieve disentanglement comparable to other models while allowing for more flexible interactions among the latent representations.
The paper unifies prior work by presenting a more general theoretical framework and proposes a practical implementation that achieves strong results in object disentanglement tasks.

strengths:
**Originality**:  
The paper introduces the principle of **interaction asymmetry** to advance disentanglement and compositional generalization, providing a unified theoretical foundation that applies to broader conditions. The use of block diagonality in higher-order derivatives is creative and extends representation learning. The combination of theoretical insights with a practical Transformer-based VAE implementation is also notable.

**Quality**:  
The theoretical contributions are well-developed, extending previous findings to more complex settings. The regularized Transformer-based VAE is supported by empirical evidence, showing comparable performance to state-of-the-art methods. The use of multiple datasets (Sprites and CLEVR6) allows for a thorough assessment of the model's capacity for disentanglement.

**Clarity**:  
The paper is well-written and accessible. The authors clearly define theoretical constructs like **interaction asymmetry** and **nth order interaction**, using visual aids to illustrate complex concepts effectively. The logical structure and provided diagrams help convey the ideas intuitively.

**Significance**:  
The principle of interaction asymmetry offers a general framework for **disentanglement** and **compositional generalization**, extending the applicability of models to more complex generative processes. The flexible and scalable Transformer-based model positions this work as a significant step toward learning useful representations for real-world tasks.

weaknesses:
1. **Limited Empirical Evaluation**: The empirical evaluation, while thorough for the presented datasets, is somewhat limited in scope. The paper only evaluates on synthetic datasets (Sprites and CLEVR6), which may not fully demonstrate the model's potential for real-world applications. Including experiments on more challenging, real-world datasets would strengthen the evidence for the model's practical utility and scalability.

2. **Theoretical Assumptions**: The theoretical results rely on several assumptions (e.g., sufficient independence, specific conditions on the latent distribution). These assumptions may be restrictive in practical scenarios, and their implications are not fully explored in the context of real-world data. It would be beneficial to discuss how these assumptions could impact the applicability of the model and whether there are potential relaxations that could make the theory more practical.

3. **Connection with Information Bottleneck**: The paper does not explicitly connect its disentanglement framework with the **Information Bottleneck** principle, which has been discussed in several works as an effective approach to disentanglement. Incorporating a discussion on how interaction asymmetry relates to information bottleneck theory could strengthen the theoretical underpinnings and offer insights into how limiting mutual information might aid in disentangling the latent representations. Including references to relevant literature, such as Tishby et al. (2000), Alemi et al. (2017)  and Meo et al. (2024), could provide additional context and depth to the discussion.

https://arxiv.org/abs/physics/0004057
https://arxiv.org/abs/1612.00410
https://openreview.net/pdf?id=ptXo0epLQo

questions:
1. Are there plans to extend the empirical evaluation to include more challenging, real-world datasets? This would help establish the practical utility of the model in diverse settings.

2. Given the theoretical assumptions made in the paper, could you discuss how they might be relaxed to make the model more applicable to real-world scenarios? Are there any potential modifications to the theory that could broaden its applicability?

3. How does the interaction asymmetry principle relate to the Information Bottleneck theory? A discussion on this connection could provide additional insights into the theoretical foundations of the proposed approach and highlight its relevance to existing literature on disentanglement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690938724

thread_type:
official_review

<thread_object>
creator: reviewer_k8sm

<content>
summary:
This paper investigates the challenge of learning disentangled and composable representations, crucial for robust generalization in machine learning. It introduces the principle of "interaction asymmetry," positing that parts of the same concept interact more complexly than parts of different concepts. This principle is formalized through block-diagonality conditions on higher-order derivatives of the generator function mapping concepts to observed data, with the order of the derivative reflecting the "complexity" of interaction. The authors theoretically demonstrate that interaction asymmetry enables both disentanglement and compositional generalization, unifying and extending prior work focused specifically on visual objects. They show that existing results for object-centric learning emerge as special cases of their framework, corresponding to lower-order interactions (0th and 1st order). The theory suggests practical criteria for disentanglement: enforcing invertibility with minimal latent dimensions and penalizing interactions across concept slots during decoding. The authors propose an attention-regularized Transformer-based VAE, leveraging the VAE framework for capacity control and incorporating a novel regularizer on the decoder's attention weights to minimize cross-slot interactions. Experiments on synthetic image datasets with objects demonstrate the effectiveness of this approach, showing that the proposed model learns disentangled object representations, achieving performance comparable to existing methods that rely on more explicit object-centric priors.

strengths:
- the principle of interaction asymmetry, offers a novel and insightful perspective on disentanglement and compositionality. While previous works have explored related ideas, the formalization through higher-order derivatives and the connection to the complexity of interaction is original and provides a unifying framework for prior results in object-centric learning. The proposed regularizer on the Transformer's attention weights is a creative application to a scalable architecture.
- The theoretical results are rigorously presented and appear sound. The connection to existing works is carefully established, demonstrating the generality of the proposed framework. The experiments, while conducted on synthetic datasets, offer convincing evidence for the effectiveness of the proposed method and regularizer.
- The paper is generally well-written and easy to follow. The authors clearly motivate their work, provide intuitive explanations of the core ideas, and present the theoretical results in a structured and accessible manner. The figures effectively illustrate the key concepts and experimental results.

weaknesses:
- the aligned-connectedness condition on $Z_{supp}$ seems to be restrictive and its practical implications are not fully explored.
- The proposed regularizer, while efficient, only approximately enforces minimal interactions. Analyzing the gap between this approximation and the theoretical ideal, and exploring alternative regularization strategies that more directly address higher-order interactions, could further enhance the method's effectiveness.
- The experiments primarily focus on synthetic datasets and object-centric learning. While this provides a controlled setting for evaluating disentanglement, exploring the effectiveness of the proposed method on more complex, real-world datasets and diverse concept types (e.g., attributes, events) is crucial for demonstrating its broader applicability.

questions:
Please see above comments.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732289003455

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Reply to All Reviewers

comment:
We thank reviewers for their valuable feedback, and their assessment of our work as “a significant step toward learning useful representations for real-world tasks“ (`qqbq`) and “offering meaningful insights that contribute to our understanding of compositional representation learning” (`c6zh`). We appreciate that the reviewers found our theory “interesting and valuable” (`c6zh`) and “creative” (`A7GP`, `qqbq`), and our experiments as “convincing” (`K8sM`) and “thorough” (`qqbq`). We were also pleased that all reviewers rated the soundness, contribution, and presentation of our work as good or excellent.

We are grateful for the constructive feedback given by all reviewers. Based on this feedback, we have made several additions to the manuscript which are highlighted in yellow in the revised version. We briefly summarize the most significant additions below:

**Experiments on More Complex Data**

Reviewers `K8sM`, `qqbq`, and `c6zh` each highlighted that our contribution would be further strengthened by experiments on more complex data. To this end, we would first like to note that we believe that our methodological and empirical contributions are already more extensive than other theoretical studies of disentanglement and compositional generalization (e.g, [1, 2, 3, 4] ). With this being said, we agree with the reviewers that experiments on more complex data are important for better understanding the utility of our proposed method. 

We have thus run additional experiments on the CLEVRTex dataset [5]. This dataset is a significant step up in complexity from CLEVR6, and has been shown to be very challenging for popular object-centric models [5, 6]. We evaluate three models on this dataset: 

- Our regularized Transformer autoencoder, ($\alpha=.1, \beta=.1$)
- An unregularized variant of this model ($\alpha=0, \beta=0$)
- An unregularized variant in which the Transformer encoder is replaced with Slot Attention.

We train all models using the same protocol as on Sprites and CLEVR6, with the one key difference in that we do not reconstruct the original image but instead reconstruct a representation of the image given by a pre-trained Vision Transformer. This training method, referred to as DINOSAUR [7], has been shown to be helpful for scaling object-centric models to complex data. We note that by testing our method with DINOSAUR, we are also better able to understand its applicability to more scalable training pipelines.

We report results in the table below after training for 300,000 iterations across 3 seeds for each model:

| Encoder | Loss | J-ARI | JIS |
|------------------|------------------|------------------|------------------|
| Transformer | $\alpha=0, \beta=0$ | $81.4 \pm 3.7$ | $50.9 \pm 2.8$ |
| Slot Attention | $\alpha=0, \beta=0$ | $94.2 \pm 0.2$ | $54.4 \pm 0.3$ |
| Transformer | $\alpha=0.1, \beta=0.1$ **(Ours)** | $\mathbf{95.9} \pm \mathbf{0.06}$ | $\mathbf{65.4} \pm \mathbf{0.6}$

Similar to Sprites and CLEVR6, we find that **the regularized Transformer achieves superior object disentanglement** in terms of J-ARI and JIS compared to these baseline models. We also visually corroborate these results by including plots of the Jacobian of the decoder for each model in Fig 7. in App J.3. We have added a dedicated section (Section I), discussing these results in more detail. We will conduct experiments for additional baselines and ablations, in the camera-ready version of this work, and then will add these results to the main text.

**Additional Discussion of Theoretical Assumptions**

Reviewers `K8sM` and `qqbq` suggested that our work would benefit from additional discussions on the implications of our theoretical assumptions and whether relaxations are possible. To address this, we have added a new section, App. H.1, that includes several such discussions. We elaborate further on the content of this section in our individual replies.

**Discussion on Scalability of Method and Relationship to Slot Attention**

Reviewer `c6zh` raised several points related to the scalability of our proposed method as well as its relationship to the object-centric method, Slot Attention. We have included several additional discussion paragraphs in App H.2, addressing these points in detail. This is elaborated upon further in our individual reply.

[1] Brady et al., 2023, Provably Learning Object-Centric Representations

[2] Lachapelle et al., 2024, Additive Decoders for Latent Variables Identification and Cartesian Product Extrapolation

[3] Wiedemer et al., 2023, Compositional Generalization from First Principles

[4] Wiedemer et al., 2024, Provable Compositional Generalization for Object-Centric Learning

[5] Karazija et al., 2021, ClevrTex: A Texture-Rich Benchmark for Unsupervised Multi-Object Segmentation

[6] Biza et al., 2023, Invariant Slot Attention: Object Discovery with Slot-Centric Reference Frames

[7] Seitzer et al, 2023 Bridging the Gap to Real-World Object-Centric Learning
</content>
</thread_object>

</thread>


Metareview:
metareview:
# Summary
This paper investigates the conditions for disentangling concept attributes. It
proposes a theory of interaction asymmetry principle for learning disentangled representation, and introduces metrics to evaluate disentanglement. Implementation of theory and experiments are conducted to support the claims.

# strengths
+ The principle of interaction asymmetry, offers a novel and insightful perspective on disentanglement and compositionally.
+ It presents a solid theoretical foundation.
+ It unifies existing work while also extending to more complex scenarios.
+ The paper is well-written and easy to follow.

# weaknesses
- experimental support is limited: need to conduct experiments on more complex data.
- need additional discussion on theoretical assumptions.
- need to discuss the relationship between disentanglement and compositional generalization further.
- need to discuss scalability of method and relationship to Slot Attention.

# reasons for decision
The paper provides insights for better understanding compositional representation learning, and it makes a significant step toward learning useful representations for real-world tasks.
The combination of theoretical insights with a practical Transformer-based VAE implementation is also notable. I recommend accept.