Learning Robust Intervention Representations with Delta Embeddings
Authors: Panagiotis Alimisis
Christos Diou
Venue: ICLR 2026 Poster (2026)
Keywords: action representation
causal representation learning
interventions
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: Causal representation learning has attracted significant research interest during the past few years, as a means for improving model generalization and robustness. Causal representations of interventional image pairs (also called ``actionable counterfactuals'' in the literature), have the property that only variables corresponding to scene elements affected by the intervention / action are changed between the start state and the end state. While most work in this area has focused on identifying and representing the variables of the scene under a causal model, fewer efforts have focused on representations of the interventions themselves. In this work, we show that an effective strategy for improving out of distribution (OOD) robustness is to focus on the representation of actionable counterfactuals in the latent space.  Specifically, we propose that an intervention can be represented by a Causal Delta Embedding that is invariant to the visual scene and sparse in terms of the causal variables it affects. Leveraging this insight, we propose a method for learning causal representations from image pairs, without any additional supervision. Experiments in the Causal Triplet challenge demonstrate that Causal Delta Embeddings are highly effective in OOD settings, significantly exceeding baseline performance in both synthetic and real-world benchmarks.

Published as a conference paper at ICLR 2026
LEARNING ROBUST INTERVENTION REPRESENTA- TIONS WITH DELTA EMBEDDINGS
Panagiotis Alimisis & Christos Diou
Department of Informatics and Telematics Harokopio University of Athens Athens, Greece
(palimisis) cdiou @hua gr
ABSTRACT
Causal representation learning has attracted significant research interest during the past few years, means for improving model generalization and robustness. Causal representations of interventional image pairs (also called "actionable coun- terfactuals" in the literature), have the property that only variables corresponding to scene elements affected by the intervention / action are changed between the start state and the end state. While most work in this area has focused on iden- tifying and representing the variables of the scene under causal model, fewer efforts have focused on representations of the interventions themselves. work, we show that effective strategy for improving out of distribution (OOD) robustness is to focus on the representation of actionable counterfactuals in the latent space. Specifically, Causal Delta Embedding that is invariant to the visual scene and sparse terms of the causal variables it affects. Leveraging this insight, propose method for learning causal representations from image pairs, without any additional su- pervision. Experiments in the Causal Triplet challenge demonstrate that Causal Delta Embeddings are highly effective in OOD settings, significantly exceeding bascline performance in both synthetic and worldworld benchmarks. Our code publicly available
this
propose that an intervention can be represented by a
INTRODUCTION
Understanding how the world changes in response to actions and external interventions is fundamen- tal for artificial intelligence agents, especially those operating dynamic environments. Although deep learning models are highly successful at capturing complex patterns from data, they often fail to generalize to new situations where the underlying data distribution changes, which is a critical limitation for real world deployment [endrryck et al. (2021); Geirhos et al. (2020). To overcome this, agents must recover the underlying mechanisms that generate and transform data, enabling causal reasoning and robust generalization (Pearl 2009).
This fundamental problem falls within the scope of Causal Representation Learning (CRL) (Schölkopf et al. 2021), which seeks to disentangle the causal variables of system (Khe-
makhem et al. 2020a).
Despite its importance in practical applications such as robotics or health-
care (Gupta et 2024 Hellström 202T [aachhez al. 2022] "TeaadaLLapuerta et al.) 2023), the challenge of learning disentangled and generalizable representations of the causal variables remains unresolved. Addressing this challenge requires accurate modelling of the underlying data generation process, a task which is guided by two fundamental assumptions within CRL. First, the Indepen- dent Causal Mechanisms (ICM) assumption, which posits that the distribution's 's generative process can be decomposed into autonomous and independent modules, each representing distinct causal mechanism (Peters al. 2017 7 Schölkopf et al.) 2021). Second, the Sparse Mechanism Shift (SMS) assumption, which suggests that an intervention typically affects only small, localized subset of
Projecct page: https: im wis.isssaaa thub. lnw.bb/ig//eaaringnnnegeeo ontariggenobbtt--nttteeeeeetooo Intervention--ee resentat with- kmbeddi iggg/

---

Published as a conference paper at ICLR 2026
Entangled Representations
Consistent Representations
drawer
ô oppen
Safe
Ô Oopen
Oppen
Open
ddrawer
safe
Adrawer
(a) Baseline Model (ERM). Action represen- tations depend on the object and seene fea- tures.
(b) Causal Delta Embeddings. The action representation óepen is invariant to the object and scene context.
Drawer: Open
Safe: Open
(a) Before
(b) After
(c) Before
(d) After
(c) Examples of intervention pairs from our dataset, showing pre- and post-intervention states for various actions and objects.
Figure 1: Visualizing Causal Delta Embeddings. Unlike baseline model that produces entan- gled representations action vectors (left), our model learns object invariant action representations (right), that generalize well to out of distribution samples. The model is trained on intervention pairs like those shown at the bottom.
these causal mechanisms (Schölkopf et al. 202 While foundational methods focused on identi- fying these mechanisms from observational or weak supervision (Brehmer al. 2022 Lippe et al. 2022 Buchholz et al. 2023). recent advances have successfully leveraged interventional data to establish stronger identifiability guarantees, utilizing score-based methods (Kulkarni et al. 2025 unknown multi-node interventions (Varicı et al.) 2024), invariance principles (Yao et al. 2024 However, the primary focus of these approaches remains on recovering the latent state variables. Fewer methods have focused learning generalizable representations of actions (interventions), which can be equally important in predicting the outcome of interventions, especially when faced with novel situations.
In this paper, we introduce Causal Delta Embedding (CDE), a novel framework for learning robust representations of interventions from image pairs. Using CDEs the intervention can be effectively isolated and represented as the vector difference between the latent representations of pre- and post- intervention states if satisfies the properties of (a) independence to causally irrelevant elements of the scene, in accordance to the ICM assumption (b) sparsity, in accordance to the SMS assumption and (c) object invariance. i.e., that the representation remains the same across objects. Using these properties guide, learning strategy is proposed for learning CDEs from interventional image pairs.
We evaluate CDE on the Causal Triplet challenge (Liu et al. 2023), which encompasses 3 in- creasingly complex settings: single-object synthetic data, multi-object synthetic data and real world scenes from Epic Kitchens (Damen al. 2022). Our experiments demonstrate that CDE establishes a new state of the art in OOD generalization for this challenge. Beyond quantitative performance, analysis reveals that CDE learns semantically meaningful representations in the intervention space, autonomously discovering anti- paralealll relationships between opposing actions (c.g., open al ose) without any explicit supervision.
2

---

Published as a conference paper at ICLR 2026
Our main contributions are follows:
We introduce Causal Delta Embedding (CDE), a novel approach for learning generalizable representations of interventions disentangled latent space.
We propose multi-objective loss function, designed to learn well separated, sparse and object invariant causal representations directly from visual data.
We perform an extensive quantitative evaluation showing that our approach achieves state- of-the-art results in the Causal Triplet challenge.
We show that our model discovers the semantic structure of the intervention space, includ- ing fundamental anti- anti-parallel relationships between opposing actions, without any explicit supervision.
RELATED WORK
Causal Representation Learning Research on CRL spans multiple directions. One line of work
focuses on identifying latent causal variables from high-dimensional observations (Khemakhem Øt al. 2020a). These methods established identifiability conditions for nonlinear ICA and demon-
strated causal factor recovery under specific assumptions (Wendong et al. [0233] Monti et al. 2020
Khemakhem et al.] 2020b). Recent theoretical advances leverage score- based
methods ac hhhieve
identifiability even with unknown multi-node interventions (Varicı et al.] 2024 Varici et al. 202.3)
while practical applications demonstrate use cases in roboties (Kulkarni et al. 2025) and single-
cell genomics (Lopez ct al. 2023). Another body of work focuses causal disentanglement (Yang
cct al.
2021;
Shen et al. [2200] Locatello et al. 2020a). often extending the VAE framework (Kingma
et al.
2013
Higgins
al.) 2017) and leveraging interventional data (Brehmer al. 2022 Lippe
et al.
2022.
Squires et al. 2023] Lippe et al. 2023 Ahuja et al., 2022] 2023] Buchholz et al.)
2023)
while other methods focus on object-centric representations in order to disentangle visual scenes
into manipulable objects (Locatello et al., 2020b) Seitzer et al. 2022). The invariance principle has emerged unifying framework (Yao et al.) 2024). showing that many CRL methods exploit distri- butional symmetries created by interventions rather than requiring explicit causal semantics. While most of these methods focus on recovering causal variables and their relationships, our work takes a fundamentally different approach: we learn embeddings that represent the interventions themselves, capturing how mechanisms change rather than identifying which variables exist.
Visual Action Recognition and OOD Generalization Traditional action recognition methods rely on spatiotemporal patterns and achieve strong performance under IID conditions (Carreira & Zisserman 2017| Feichtenhofer et al. 2019: Arna et al., (002). However, these correlation-
approaches struggle with distribution shifts Geirhos et al. 2020)) and often rely on spurious corre- lations (Wang & Jordan| 2024). Recent work has explored domain adaptation (Chen et 2019
Munro & Damen 2020) and causal approaches (Magliacane et al. 2018 Wang et al.) 2023) for ro- bust action understanding. Another category of methods uses large Vision Language Action (VLA)
models (Kim et al. 2024 Zitkovich et al. 2023; Ma et al. 2024) to enable agents to perform actions in challenging environments. These models typically depend on large-scale pre-training on diverse data, yet generalization to unseen tasks remains an open challenge (Sapkota et al.) (025). Unlike these approaches, our method learns causal representations of interventions that satisfy propertics resulting from CRL assumptions, generalizing to novel object-action combinations without fine- tuning.
Contrastive Learning and Sparse Representations Contrastive learning effectively learns mean- ingful representations by contrasting similar and dissimilar examples (Chen et al.) 2020 Khosla et al. 2020) He et al., 20200 Grill al. 2020). Recent theoretical work establishes deep connections
between contrastive learning and causal structure discovery. Multi-view contrastive methods handle partial observability by learning shared representations across modalities (Yao et al. 2023 Federici
et al. 2020; Tian et al., 2020). When data augmentations correspond to causal interventions, con- trastive learning can provably disentangle causal factors (Von Kügelgenn et al. 2021 Zimmermann et al.; 2021). However, these methods contrast individual samples augmented views, whereas our approach contrasts relationships between pre- and iost-intervention pairs aiming at learning embeddings of the transformations themselves rather than sample level representations. The sparse
3

---

Published as a conference paper at ICLR 2026
Causal Graph for a pair of observations (x, before and after an action a, proposed by Liu et al. (023). The data generating process is described by set of latent factors, including global scene level factors 2, and local object level factors zi which are dependent due to confounders c. The action is assumed influence only few object level causal factors in the scene and the effect of that influence is captured by žn- The red dashed line indicates the structural equation assumed by CDE approach.
Figure 2:
mechanism shift (SMS) principle (Schölkopf al., 2021; Peters et al. 2017) suggests interventions
affect only small subsets of causal mechanisms. This sparsity assumption enables identifiability in various settings, such as mechanism sparsity regularization for nonlinear ICA (Lachapelle et al.) 2022 2024) or instance dependent partial observability (Xu et al., 2024). Recent work demon- strates that combining sparsity with appropriate regularization improves both identifiability (Phistr & Peters 2022 Buchholz et al.) 2023) and generalization (Lopez et al. 2023 Layne et al.] 2024). While adversarial training with sparsity constraints has been explored (Liu et al., 2023). it often yields poor OOD performance when confounders remain in the scene, motivating our stricter causal assumptions and explicit intervention embedding approach.
PROBLEM FORMULATION
The central challenge this paper addresses is the development of CRL framework that can ro- bustly infer actions interventions from high-dimensional observations, particularly under distribu- tion shifts.
We formalize this challenge within the framework of the Structural Causal Model presented by Liu al. (2023) (Figure Let us consider a set of causal variables e C RR. representing the state of the underlying data generating mechanisms. These variables have dependencies that are defind through a set structural equations. Assuming an additive noise model, these equations are of the form
2, = (pa(2:)) + c, i=
where pa(z;) denotes the set of causal parents of variable and the G; are mutually independent stochastic noise terms representing unmodeled exogenous factors. The high-dimensional visual observation E X C Rd is rendered from these latent variables via a complex generative function g: 2 X. such that æ = g(=).
We assume the latent variable z can be partitioned into scene-level variables z, (e.g., illumination, camera pose) and set of object-level variables 2. ( N.K ==1.k=1> corresponding the k-th property of the n-th object. An agent performs action A, which applies intervention on the system. This intervention transforms the pre-intervention state into post-intervention state E. Unobserved confounders (c) create spurious correlations and training-testing distribu- tion mismatch Pirsin = 2, a) * (Z = 2, a). Following the Independent Causal Mecha- nisms principle (Schölkopf al. 2012: Peters al. 2017). assume the causal mechanism P(2. ān 2n. Za. a) invariant this shift. Therefore, robust model must learn this invariant mechanism instead of non-stationary correlations.
We investigate challenging types of OOD shifts: Compositional Shifts: Training and test sets share the same object classes, O = O 1ests but disjoint sets of object-action pairs. (Amii x Ourain) n (Arest Orea) 0.
4

---

Published as a conference paper at ICLR 2026
0. Notice also that the exogenous noise may differ across states 2 and 2, especially in real-world data, where there is no control over the environment and data acquisition conditions. For this rea- son, they are referred to as "actionable counterfactuals" in Liu et al. (2023) (as opposed to perfect counterfactuals).
Systematic Shifts: The training and test sets of object classes are disjoint, Ouain DuainnOur nO
Objective Given dataset of paired observations D = {(z, ă, ر where and ă are the pre- and post-intervention images respectively, and is the corresponding action label, our objective is to learn function F X' x X' + A. This function must predict the action a by learning representation that isolates the invariant causal signature of the intervention, thereby achieving high performance OOD test data characterized by the compositional and systematic shifts defined
above.
4 CAUSAL DELTA EMBEDDINGS
Consider an Encoder, Φ X' + Z that maps nigh-ddmeesiioal observation æ E X' point in the latent space Z. A Delta Embedding is defined as follows.
Definition (Delta Embedding) Given pair of observations ã)) corresponding the state of the world before and after an intervention a E A, the Delta Embedding, 6a. is the vector difference
8. = ((T) 6(x)
Assuming identical noise across observations (i.e., perfect counterfactuals, the actionable counter- factual case will be discussed the following section) and that 2, in the data generating process of Figure iis identifiable by the encoder, for the Delta Embedding have
8, =
0)
where z, is the dimension (or subset dimensions) of object n that is affected by action accor- dance with the notation used in the model of Figure 2 From equation I we observe the following properties 8..
1. Independence. scene properties and objects not affected by a, i.e., it not influenced or informed by them.
Under the model of Figure 2 an action's 's representation is independent of
2. Sparsity. If the assumption of Sparse Mechanism Shifts Schölkopf et al. (2021): 1: Liu et al.
(2023) holds, then the action a will affect only a few underlying causal factors of the sys- tem, and the representation of the change, 8,, will be sparse.
To generalize novel compositions of actions and objects, the action representation must satisfy additional properties. Specifically, even the Independence and Sparsity properties are satisfied, the action a affects different objects in different way, learning system will not be able predict how the action will modify the representations of unseen objects, or even seen objects but without any examples of these objects with a in the training set.
We therefore introduce an additional requirement on the action representation, namely that it remains similar when applied to different objects, e.g., that the representation of action open fundamen- tally the same, regardless of whether it is door or box that is being opened. We therefore introduce an additional property:
3. Invariance. The action representation 8, should not vary across different objects. One way to formalize this is through the variance of the delta embeddings across samples, i.e.,
Pa R
Definition (Causal Delta Embedding) A Causal Delta Embedding (CDE) is a Delta Embedding that satisfies the properties of Independence, Sparsity and Invariance.
5

---

Published as a conference paper at ICLR 2026
Encoder
CLS
Contrastive Sparse
Action Clas- sifer
Crees Entropy Loes
A) Global Causal Delta Embedding Model
Causal Pro-
jector (MLP)
CDE
Patch-Wise Embedding Model
Top-K Ag- gregation
After
jector (MLP)
CDE
F..)
Figure 3: Model architecture. Model A (top) computes global causal delta from CLS tokens.
Model B (bottom) computes patch-wise deltas, aggregated to a causal delta. Both feed into a com- mon action classifier.
In terms of the SCM Figure structural equation of 2, as
2 Causal Delta Embeddings can be implemented by defining the
2o f(za, a, E) Za +
where is the value of zero mean, independent exogenous noise. When the only change before and after the intervention is the effect action a (perfect counterfactual), e = 0. The following section uses this definition to develop strategy for learning Causal Delta Embeddings.
APPROACH
5.1
THE GLOBAL CAUSAL DELTA EMBEDDING MODEL
5.1.1
MODEL ARCHITECTURE
We first introduce global model, i.c., model that learns single causal representation from the entire image. The model consists of three main components, as illustrated Figure (A).
The Encoder (o): The encoder is responsible for mapping input image into the Z. It is composed of two sub-modules. (i) A Pre-trained Vision Backbone: We use powerful Vision al. 2020), specifically
pre-trained with the DINO self- supervision algorithm (Caron et al.] 2021). The backbone processes the input image and outputs high-dimensional feature vector. We use the output corresponding the '[CLS]' token as the global image representation. (ii) A Causal Projector: The feature vector from the backbone is then passed through a small multi-layer perceptron (MIP). This projector's role is to transform the general- general-purpose purpose features into an 1-dimensional representation satisfying the Causal Delta Embed- ding properties.
Transformer (ViT) (Dosovitskiy
Delta Computation and the action classifier h: We compute the CDE according to Definition] I
Given the latent vectors for the pre-intervention image (2 Φ(x)) and post-intervention image (2 o, the delta is calculated via simple, element-wise subtraction, 8 = غ - This vector is the sole imput to a final classification head, which an MLP that takes the I-dimensional delta and outputs logits for the different action classes in A.

---

Published as a conference paper at ICLR 2026
5.1.2 IMPLEMENTATION THE LEARNING OBJECTIVE
To learn CDES that satisfy the properties outlined in the Section 4 we combine three loss functions. (i) Cross-Entropy Loss: The primary objective is to ensure the delta embedding is useful for the downstream task. We use a standard Cross-Entropy loss, LcE between the predicted action logits h();) and the one-hot ground-truth action label aç. (ii) Supervised Contrastive Loss: To learn embeddings that are clustered together for the same action (Property 3, Invariance), we use the Supervised Contrastive Loss, Luumtrast (Khosla et al.) 2020). For batch of B delta embeddings, the loss for each embedding ó; (the "anchor") encourages it to be closer to other embeddings of the same class ("positives") than to all other embeddings in the batch.
exp(sim(6,, 5p)/TT
conntrast
log
P())
PEP()) Σj exp((im(pp.)... 6,)/T)
where P(i) the of all positive samples for anchor in the batch, sim(:.-) denotes the cosine similarity, and т is scalar temperature hyperparameter. This loss component is also consistent with the structural equation [] Finally, we introduce a (iii) Sparsity Regularizer: To encourage minimal representation in line with the sparse mechanism shift hypothesis (Property 2. Sparsity), we apply an & regularization penalty. This loss penalizes the sum of the absolute values of the embedding dimensions, promoting solutions where most dimensions are zero.
1 B
Cquariity
Σ
6
16
(5)
The final training objective is weighted of these three components:
Liotal LCF + Occomrast Ceontrest Osparsity Spparsity
(6)
where Ocenntrstt and Osparsity are scalar hyperparameters that balance the influence of each loss com- ponent. Note that the network trained end-to-end, i.e., the encoder, Φ, is also updated during training.
5.1.3 ACTIONABLE COUNTERFACTUAL CASE
Notice that although no loss component explicitly enforces Property 1, this property is directly satisfied by the use of the Delta Embeddings and image pairs, where the observed changes are only due to This does not hold under the actionable counterfactual case (Liu et al.] 2023), where the exogenous noise may change across observations, leading to E * in equation 3 and non-zero elements in the delta embedding of equation I In the Appendix (Section Eb show that if is zero mean independent noise, then the classifier trained with LC not affected by these non-zero values. This observation is also supported by empirical findings presented in Section 6 where the delta embeddings lead to effectiveness improvements in real-world OOD data.
5.2 SPATIAL EXTENSION: THE PATCI- WISE MODEL
In complex scenes with multiple objects or significant background noise, an action may only affect a small, localized region of the image. A global embedding risks 'averaging out' this important local change, making it difficult to detect. To address this, we developed patch-wise extension of our model.
5.2.1 ARCHITECTURE
The Patch-Wize model adapts the core architecture to operate local regions, as shown in Figure (B).
The architecture includes (i) Patch-wist Feature Extraction: We use ViT backbone, but instead of taking the global '[CLS]' token, we retain the output feature vectors for each individual image patch. This gives sequence of patch features for both the before and after images, (ii) Patch-wise Delta Computation: A shared Causal Projector and the subtraction operation are applied independently to each corresponding pair of patch features. This yields set of delta embeddings, (5p). one for
7

---

Published as a conference paper at ICLR 2026
Table I: Single-object ProCTHOR results. Our Global Delta Embedding model significantly im-
proves OOD generalization under both compositional and systematic shifts. (R: ResNet-16 18, V:
ViT-DINO)
Method
IID Acc.
OOD Comp.
OOD Syst.
Gap Syst. (1)
Vanilla-R
0.96.0.00
0.36
0.48
0.48
Vanilla-V V
0.9540.0
0.34
0.47
0.48
ICM-R
0.9549.01
20.1
0.50+0.09
0.45
ICM-V
0.95+0.01
0.38
0.49
0.46
SMS-R
0.96+a.01
0.47
0.54++0.07
0.42
SMS-V
0.95
0.34
0.39+0.04
0.56
OUTSWFELIPP)
0.97
0,91
0.72+
0.25
OUTSWEEDINO
0.96
0.91 0.02
0.75 0.02
0.21
OUrS(WEMAB)
0.96 0.1
0.95 0.01
0.7 + 0.02
0.25
each spatial patch location p. (iii) Top-K selection: We assume that the action's 's primary effect is localized to a small number patches. We therefore identify the & patches with the largest change by measuring the L2 norm of their delta vectors (115p122. The loss Lioul of equationnoja then applied to each of the delta vectors, leading one loss cli for each vector. The final loss is then the average of the individual patch losses, L == Σ wial
EXPERIMENTS
This section evaluates the effectiveness of our CDE framework. We first describe our experimental setup, then present the main quantitative results demonstrating CDE's effectiveness in OOD gener- alization, followed by qualitative and ablation analyses that provide deeper insights into its learned representations and design choices.
6.1 EXPERIMENTAL SETUP
Our evaluation is conducted on the Causal Triplet benchmark (Liu et al. 2023), specifically de- signed for intervention-centric causal representation learning. This benchmark features three dis- tinct settings of increasing visual complexity: single-object synthetic scenes, multi-object syn- thetic scenes (both from ProcTHOR (Deitke et al., 2022)), ) and challenging real-world scenes from Epic-Kitchens (Damen et al.) (022). In all settings models are trained on pairs of pre- and post- intervention images with action labels and are evaluated for their ability to infer the action. Further details on the datasets and data filtering procedures are provided in the Appendix.
We follow the Causal Triplet protocol, evaluating models on both IID and OOD test sets. The OOD splits test two forms of generalization: Compositional Distribution Shifts, where the model encoun- ters unseen combinations actions and objects (e.g., open (drawer) when only open (door) and close (drawer) where seen during training); and Systematic Distribution Shifts, where gen- cralization to entirely unseen object classes required. Visualizations these distribution shifts available in the Appendix. All reported quantitative results mean accuracies and standard deviations average over 3 random seeds. We set Orcomtrast = 2.0 and Ospursiity 1.0 for all experiments (see the Appendix for more details).
We compare our two proposed models against the baselines from the Causal Triplet paper (Li et al, (023), including vanilla ResNets (He et al. (2066), methods incorporating causal regularization (ICM, SMS), and object-centric approaches (Slot Attention (Locatello et al. 2020b), Group ViT (Xu
al. 2022
6.2
MAIN QUANTITATIVE RESULTS
Our CDE framework consistently delivers substantial improvements in OOD accuracy across all evaluation settings, establishing a new state of the art. For single-object scenes, our global CDE model cuts the generalization gap from 0.56 to 0.21 while matching IID accuracy (Table T. In
8

---

Published as a conference paper at ICLR 2026
Table Results across multi-object ProcTHOR and Epic-Kitchens (systematic shift). Oracle-mask utilizes ground truth mask to isolate the intervened object. Conversely, other approaches must infer the action and object without this supervision.
Dataset
Method
IID Acc.
OOD Acc.
Gap
ProcTHOR
ResNet
0.83
0.30
0.53
Oracle-mask
0.90
0,42
0.48
Slot-avg
4.49
0.15
0.34
Slot-dense
0.50
0.19
0.32
Slot-match
0.66
0.21
0.45
0.91
0.30
0.61
OTTS(VI--IIN)
0.92
0.45
0.47
(
0.94. 0.00
0.48 10.07
0.46
Epic-Kitchenn
ResNet
0.42+0.03
0.17
0.25
CLIP
0.45+0.02
0.24
0.21
Group-avg
0.47
0.24
0.23
Group-dense
0.50
0.26
0.24
Group-token
0.52+0.03
0.27 20.03
0.25
(TT
0.50+0.02
0.30
0.20
Ours 0.54
0.33 10.00
0.21
OurS(ViECPP)
0.59 0.03
0.34 1. 0.01
0.25
challenging multi-object and real-world settings (Table 2 our Patch-Wap Wise model outperforms all baselines, including oracle methods that ground-truth segmentation masks. Notice that the ResNet18 encoder performs significantly worse than the rest of the encoders, possibly indicating that this backbone cannot disentangle latent representations that depend on the actions from the rest of the scene as required by the model of Figure
6.3 ACTION RELATIONSHIPS CAUSAL DELTA SPACE
To study the semantic structure of the learned delta space, we investigated whether the model could discover fundamental relationships between actions on its own. We computed the pairwise cosine similarity between all learned action representations. The result is visualized in the appendix (Fig- ure 7.. The analysis reveals that the model has learned a perfect anti-parallel relationship for op- posite actions. The cosine similarity between the representations for open and close, for dirty and clean, well as for turn work not only separates the action concepts but also discovers opposing relationships between them, organizing the representations in meaningful way. similar pattern is observed in the more chal- lenging real-world dataset where the model learns the anti-parallel representations for the open and close action pair, as well as for the fold and stretch pair (see Figure g in the Appendix for details).
and turn off, is -1.0. This demonstrates that frame-
In summary, the combination of strong predictive properties and consistent semantic structure demonstrates that our CDE framework learns meaningful representations of interventions. For fur- ther geometric analysis of the delta space, including UMAP projections, refer the Appendix.
6.4 ABLATION STUDY
To understand the contribution of each component of CDE framework, also conducted series of ablation studies, by analyzing the impact of each major loss component on the performance of our primary model with ViT-DINO backbone. Table 3 presents the results, comparing full model against versions where each loss component is removed, and a baseline trained only with standard CE loss.
The results demonstrate the effectiveness of our approach. Our full model achieves an OOD ac- curacy of 75.0%, +8 point improvement over delta embedding representation trained solely with

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study of our method's components on the ViT-DINO model. Results for the single-object systematic shift setting, showing the impact on OOD accuracy when each core component removed.
Model Configuration
IID Acc. (%)
OOD Acc. (%)
Full Model
0.96
0.75
Ablations
w/o Sparsity Loss
0.96
0.73
w/o Contrastive Loss
0.95
0.68
CE Loss only
0.94
0.67
Baseline (Liu al., 2023)
0.95
0.47
a CE objective, validating that explicitly structuring the representation space leads generalization improvements. Removing the supervised contrastive loss component causes 7-point drop in OOD accuracy. Removing the sparsity loss term causes 2-point drop. Please refer to the Appendix for further ablation experiments.
7 CONCLUSION
This paper introduces the Causal Delta Embedding (CDE) framework, simple yet effective ap- proach to interventional causal representation learning. By explicitly modeling interventions as delta vectors in a structured latent space, CDE inherently satisfies the properties of independence, sparsity and invariance, leading to improved gencralization. Our empirical validation on the Causal Triplet challenge demonstrates that CDE achieves state-of-the-an OOD generalization, outperform- ing prior methods across synthetic and real world datasets. Beyond quantitative gains, we show that CDE learns semantically meaningful representations without supervision, where opposing actions have anti-parallel representations. Despite the promising results, we acknowledge that limitations remain for real-world data, since both IID and OOD accuracies still low for real world deploy- ment, and also the use of universal delta embeddings for each action limits its ability to capture context-dependent visual transformations of actions. Future research directions includes the devel- opment of mechanisms to increase the robustness of the method in real-world scenarios involving increased noise or occlusions, extending the framework video streams for modeling causal dy- namics in temporal sequences, and investigating compositional properties of delta embeddings to enable multi-step interventions and generalization to novel action sequences.
REPRODUCIBILITY STATEMENT
The previous sections have outlined the main building blocks of the proposed method, well as the approach followed in the experiments, with the Appendix providing additional information and results. The code to reproduce the experiments attached as supplementary material (without any identifiable information of authors) and will be made publicly availablc upon acceptance. Finally, all experiments were carried out by strictly following the Causal Triplet benchmark (Liu al., (023) evaluation protocols, which relies on the publicly available ProcTHOR (Deitke et al.) 2022 and Epic-Kitchens (Damen et al., 2022) datasets.
ACKNOWLEDGMENTT
The work leading these results has been funded by the European Union under Grant Agreement No. 101057821 (project RELEVIUM). Views and opinions expressed are however those of the au- thors and do not necessarily reflect those of the European Union the granting authority (HaDEA). Neither the European Union nor the granting authority can be held responsible for them.
REFERENCES
Kartik Ahuja, Jason S Hartford, and Yoshua Bengio. Weakly supervised representation learning with sparse perturbations. Advances in Neural Information Processing Systems, 35:15516-15522
10

---

Published as a conference paper at ICLR 2026
2022.
Kartik Ahuja, Divyat Mahajan, Yixin Wang, and Yoshua Bengio. Interventional causal representä- tion learning. In International conference on machine learning, pp. 372-407. PMLR, 2023,
Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6836-6846, 2021.
Johann Brehmer, Pim De Haan, Phillip Lippe, and Taco Cohen. Weakly supervised causal repre- sentation learning. Advances in Neural Information Processing Systems, 35:38319-38333 2022.
Simon Buchholz, Goutham Rajendran, Flan Rosenfeld, Bryon Aragam, Bernhard Schölkopf, and Pradeep Ravikumar. Learning linear causal representations from interventions under general non- linear mixing. Advances in Neural Information Processing Systems, 36:45419-45544 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng. Tempo- ral attentive alignment for large-scale video domain adaptation. Proceedings of the IEEE/CVF international conference on computer vision, pp. 6321-6330, 2019.
Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Evangelos Kazakos, Jian Ma, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, et al. Rescaling egocen- tric vision: Collection, pipeline and challenges for epie-kitchens-100. International Journal of Computer Vision, pp. 1-23, 2022.
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances Neural Information Processing Systems, 35:5982- 5994, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922 2020.
Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi-view information bottleneck, 2020. URL http::ts///xxi.oottttt 3/2002.0700
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6202-6211,
Robert Geirhos, Jörn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11):665- 673, 2020.
Saan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Beraardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 33:21271-21288 2020.

---

Published as a conference paper at ICLR 2026
Tarun Gupta, Wenbo Gong, Chao Ma, Nick Pawlowski, Agrin Hilmkil, Meyer Scetbon, Marc Rigter, Ade Famoti, Ashley Juan Llorens, Jianfeng Gao, et al. The essential role of causality foundation world models for embodied ai. arXiv preprint arXhv:2402.0666. 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning, 2020. /RLp::tt///////.orrrg// 05122
Thomas Hellström. The relevance of causation in robotics: A review, categorization, and analysis. Paladyn, Journal of Behavioral Robotics, 12(1):238 255, 2021.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustmess: A criti- cal analysis of out-of-distribution generalization. Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021.
2021.
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with constrained variational framework. In International conference on learning representations, 2017.
Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational autoen- coders and nonlinear ica: A unifying framework. In International conference on artificial intelli- gence and statistics, pp. 2207-2217. PMLR, 2020a.
Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Identifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 33:12768-127788 2020b.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18666 13:18661-18673, 2020.
Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, et al. Openvla: An open-source Aision-language-action model. arXiv preprint M.Xi::24066092466 2024.
Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
Pranamya Kulkarni, Puranjay Datta, Burak Varieı, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. Ropes: Robotic pose estimation via score-based causal representation learning. arXiv preprint wrXhv:2510.208855 2025.
Sébastien Lachapelle, Pau Rodriguez, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, and Simon Lacoste-Julien. Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ica. Conference on Causal Learning and Reasoning, pp. 428-484. PMLR, 2022.
Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexan- dre Lacoste, and Simon Lacoste-Julien Nonparametric partial disentanglement via mecha- nism sparsity: Sparse actions, interventions and sparse temporal dependencies. arXiv preprint arXiv:2401.04890 2024.
Elliot Layne, Jason Hartford, Sébastien Lachapelle, Mathieu Blanchette, and Dhanya Sridhar. Sparsity regularization via tree-structured environments for disentangled representations. arXiv preprint 2XXx:2445.204883 2024.
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Stratis Gavves. Citris: Causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, pp. 13557-13603. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2026
Phillip Lippe, Sara Magliacane, Sindy Löwe, Yuki M Asano, Taco Cohen, and Efstratios Gavves. Biscuit: Causal representation learning from binary interactions. In Uncertainty in Artificial Intelligence, pp. 1273- PMLR, 2023.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set set object detection. In European Conference Computer Vision, pp. 38-55. Springer, 2024.
Yuejiang Liu, Alexandre Alahi, Chris Russell, Max Horn, Dominik Zietlow, Bernhard Schölkogpf, and Francesco Locatello. Causal triplet: An open challenge for intervention-centric causal rep- resentation learning. In Conference on Causal Learning and Reasoning, pp. 553-573. PMLR, 2023.
Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly-supervised disentanglement without compromises. In International confer- on machine learning, pp. 6348 6359. PMLR, 2020a.
Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alcxey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot atten- tion. Advances in neural information processing systems, 33:11525-11538 2020b.
Romain Lopez, Natasa Tagasovska, Stephen Ra, Kyunghyun Cho, Jonathan Pritchard, and Aviv Regev. Learning causal representations of single cells via sparse mechanism shift modeling. In Conference on Causal Learning and Reasoning, pp. 662-691. PMLR, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
17Xiv:1111.05101, 2017.
Yueen Ma. Zixing Song, Yuzheng Zhuang, Jianye Hao, and Irwin King. survey on vision- language-action models for embodied ai. arXiv preprint arXiv:2405. 14093, 2024.
Sara Magliacane, Thijs Van Ommen, Tom Claassen, Stephan Bongers, Philip Versteeg, and Joris M Mooij. Domain adaptation by using causal inference to predict invariant conditional distributions. Advances in neural information processing systems, 31, 2018.
Ricardo Pio Monti, Kun Zhang, and Aapo Hyvärimen. Causal discovery with general non-linear relationships using non-linear ica. In Uncertainty in artificial intelligence, pp. 186-195. PMLR, 2020.
Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recog- nition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 122-132, 2020.
Judea Pearl. Causality. Cambridge university press, 2009.
Jonas Peters, Dominik Janzing, and Bernhard Schölkopf. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Niklas Pfister and Jonas Peters. Identifiability of sparse causal effects using instrumental variables. In Uncertainty Artificial Intelligence, pp. 1613-1622. PMLR, 2022.
Pedro Sanchez, Jeremy Voisey, Tian Xia, Hannah Watson, Alison Q O'Neil, and Sotirios A Tsaftaris. Causal machine learning for healthcare and precision medicine. Royal Society Open Science, 98))220638, 2022.
Ranjan Sapkota, Yang Cao, Konstantinos Roumeliotis, and Manoj Karkee. Vision-languuge-action models: Concepts, progress, applications and challenges. arXiv preprint arXiv:2505.04770 047669 2025.
Bernhard Schölkopf, Dominik Janzing, Jonas Peters, Eleni Sgouritsa, Kun Zhang, and Joris Mooij. On causal and anticausal learning. arXiv preprint arXiv:/206.644: /206.6471, 2012
Bernhard Schölkopf. Francesco Locatello, Stefan Bauer, Nan Rosemary Ke. Nal Kalchbrenner, Anirudh Goyal, and Yoshua Bengio. Toward causal representation learning. Proceedings of the IEEE, 109(5)6612-66 2021.
13

---

Published as a conference paper at ICLR 2026
Maximilian Seitzer, Max Horn, Andrii Zadaianchuk, Dominik Zietlow, Tianjun Xiao, Carl-Johann Gimon-Gabriel, Tong He, Zheng Zhang, Bernhard Schölkopf, Thomas Brox, et al. Bridging the gap real-world object-centric learning. arXiv preprint arXiv:2209.148800 2022.
Xinwei Shen, Furui Liu, Hanze Dong, Qing Lian, Zhitang Chen, and Tong Zhang. Disentangled
generative causal representation learning, 2020.
Chandler Squires, Anna Seigal, Salil S Bhate, and Caroline Uhler. Linear causal disentanglement via interventions. In International conference on machine learning, pp. 32540-32560. PMLR, 2023.
Alejandro
Paul Bertin, Stefan Bauer, Hananch Alice, Yoshua Bengio, and Fabian J. Theis. Causal machine learning for single-cell genomics, 2023. URL htt.htttps:/aaaee abs/ /2e//2100.. 14935
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding, 2020. URI http s: 5://arxiv.. wwwpr/ii..rrgaa////9900 05849
Burak Varici, Emre Acarturk, Karthikeyan Shanmugam, Abhishek Kumar, and Ali Tajer. Score- based causal representation learning with interventions. arXiv preprint arXiv:2301.0822 2023.
Burak Varıeı, Emre Acartürk, Karthikeyan Shanmugam, and Ali Tajer. Linear causal representation learning from unknown multi-node interventions. Advances in Neural Information Processing Systems, 37:111614-1111
2024.
Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 34:16451-16467, 2021.
Shanshan Wang, Yiyang Chen, Zhenwei He, Xun Yang, Mengzhu Wang, Quanzeng You, and Xingyi Zhang. Disentangled representation learning with causality for unsupervised domain adaptation. In Proceedings of the s1 ACM international conference on multimedia, pp. 19-2992., 2023.
Yixin Wang and Michael Jordan. Desiderata for representation learning: A causal perspective. Journal of Machine Learning Research, 25(275): 2024.
Liang Wendong, Armin Kekić, Julius von Kügelgen, Simon Buchholz, Michel Besserve, Luigi Gre- sele, and Bernhard Schölkopf. Causal component analysis. Advances in Neural Information Processing Systems, 36:32481-32522 2023.
Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius Von Kügelgen, Francesco Locatello, and Sara Magliacane. A sparsity principle for partially observable causal representation learning. arXiv preprint wrXiv:2403.08335 2024.
Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 98134-18144, 2022.
Mengyue Yang, Furui Liu, Zhitang Chen, Xinwei Shen, Jianye Hao, and Jun Wang. Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2021.
Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, and Francesco Locatello. Multi-view causal representation learning with partial observability. arXiv preprint arXivv2311.04055 2023.
Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, and Francesco Locatello. Unifying causal representation learning with the invariance principle. arXiv preprint arXiv:2409.02772. 2024.
Roland S Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, and Wieland Brendel. Contrastive learning inverts the data generating process. In International conference on machine learning, pp. 12979-12990. PMLR, 2021.
14

---

Published as a conference paper at ICLR 2026
Brianna Zitkovich, Tianhe Yu, Sichun Xu, Peng Xu, Ted Xiao, Fei Xia, Jialin Wu, Paul Wohlhart, Stefan Welker, Ayzaan Wahid, et al. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In Conference Robot Learning, pp. 2165-2183 PMLR, 2023.
15

---

Published as a conference paper at ICLR 2026
A DATASET DETAILS
This section provides further details on the datasets used in our evaluation.
ProcTHOR The ProcTHOR dataset Deitke al. (2022) provides synthetic indoor scenes. For our single-object scenes, each scene contais one manipulated object, ensuring a clear focus on the intervention. multi-object scenes, multiple objects are present, increasing the visual complexity of the scene, although only one object is again manipulated. We follow the dataset generation and filtering procedures as described in Liu et al. (2023) to ensure consistency with the Causal Triplet benchmark.
Spic-Kitchems The SpiccKitchens dataset Damen et al. (0222) comprises real world egocentric videos of diverse kitchen activities. From this, extract pre- and post-intervention image pairs. Unlike synthetic environments, Epic-Kitchens introduces significant real world challenges such as camera motion, varying lighting conditions, occlusions and dynamic backgrounds, making the task of isolating interventions particularly challenging. To ensure dataset quality, a two-stage filtering process using Grounding DINO Liu et al., (2024) for zero-shot object detection is applied. For each extracted pair, the pipeline verifies that the target object appears clearly in both frames with detection confidence above set threshold 0.45. This automated filtering removes cases with poor object visibility excessive motion blur.
A.1 VISUALIZATIONS OF OOD SHIFTS
Figures and 6 visually illustrate the compositional and systematic distribution shifts utilized in
the Causal Triplet benchmark.
break
clean-
action class
close
dirty
open
turnoff
turnon
CellPhone
Desktop
Laptop
Television
Bed
Bowl
Cloth
Cup
Mug
Pan
Plate
Pot
Book
Box
Cabinet
Drawer
Fridge
LaundryHamper
Microwave
Safe
Toilet
Faucet
FloorLamp
object class
Figure 4: Compositional Distribution Shift the ProcThor dataset. Blue boxes indicate IID data, while red boxes indicate novel OOD action-object combinations.
B GEOMETRIC ANALYSIS OF CAUSAL DELTA EMBEDDINGS
This section provides additional analysis of the geometric properties the learned Causal Delta Embeddings, complementing the insights presented along with the experimental results.
B. ACTION REPRESENTATION RELATIONSHIPS LEARNED FROM REAL- WORLD DATASETS
Figure 7 fillustrates the the pairwise cosine similarities between the embeddings learned for all ac- tions in the ProcTHOR dataset, while Figure presents the same information for the more challeng-
16

---

Published as a conference paper at ICLR 2026
break
clean
action class
close
dirty
open
turnoff
turnon
CellPhone
Desktop
Laptop
Television
Bed
Bowl
Cloth
Cup
Mug
Pan
Plate
Pot
Book
Box
Cabinet
Drawer
Fridge
Lampery
Microwave
Safe
Toilet
Faucet
FloorLamp
object class
Figure 5: Systematic Distribution Shift the ProcThor dataset. Blue boxes indicate IID data, while
red boxes indicate novel OOD objects that the model has not encountered during training.
empty
clsss
nil
flip
fold
int
srretch
scuce
broccol
bag
nottle
box
bread
conta ner
cupboaar
donireeie en crawer
frige
auborgine
becon
carro
mnderr
oilin
paper
pearh
pesper
pizz
mutato
salee
tomelo
bowl
cup
cless
cattle
rice
sink
moon food
pate
on
sauic
atesses
peas
object
Figure 6: Systematic Distribution Shift in the EpicKitchens dataset. Blue boxes indicate IID data, while boxes indicate novel OOD objects that the model has encountered during training.
17

---

Published as a conference paper at ICLR 2026
100
dry
1,00
3.00
0.03
0.01
0.01
9.02
0.02
0.75
dean
2.00
1.00
0..2
-0.3
0,50
0.25
om
-1
0,00
turTon
0.01
9,22
0.03
0.07
1.00
1.00
0.22
-0.25
-0.50
-0.75
dose
0.02
0.03
-3.03
0.07
002
0.02
1.00
-1.00
deon
Lamon
bemot
Figure 7: Heatmap of pairwise cosine similarities between all learned delta embeddings. The strong blue squares (similarity near -1.0) reveal a near-perfeet perfect anti-pare mntilparallel relationship for opposite action pairs, which discovered entirely from the data.
1.00
1.00
-3.00
40.12
40:27
0.11
-3.29
012
0.15
00.17
.0.3
0.04
0.37
20.0
40.03
5.13
0.06
0.35
dese
1:00
100
0.13
0.28
0.10
0.27
0.12
0.15
0.16
0.03
9.03
0.17
0.06
0.2
3.12
0.05
05
0.13
empty
peel
-.01
0.10
0.06
0.02
1.00
0.09
0.60
0.00
0.00
0.17
0.19
0.37
0.05
0.03
9.21
0.33
0.33
0.50
00.-00.2
0.7
0.28
0.10
0.09
1.00
027
0.74
0.06
0.03
0.73
0.18
0.14
0.01
0.75
0.15
072
0.12
4.11
40:0
0.85
3.00
0.00
0125
4.09
0.00
0.08
0.77
0.25
0.27
0.80
0.00
0.009
2.00
0.23
0.00
0.25
0.00
Ienovo - 0.03 0.03
0.56
0.19
0.37
0.03
0.25
0.48
0.22
1.00
0.27
0.36
0.19
0.22
9.25
0.41
0.39
break 0.04 4.03
0.02
0.24
0.19
5.75
0.04
0.01
0:3
0.7
7.00
0.81
0025
0.19
0.11
0.30
0.38
2.00
dratch
40.08
40.03
8.08
0.22
0.07
3.00
2.13
4.17
0.77
0:00
021
0.23
011
0.59
0.45
026 4.11
0.30
6.11 013
1.00
0.50
0.2
Ift
0.00
C.05
0.54
0.35
1145
0.25 841
0.36
0.25 029
J.00
0325
11.00
Figure 8: Heatmap of pairwise cosine similarities between all learned action prototypes for the EpicKitchens dataset.
ing real-world Epic Kitchens dataset. We observe that both cases the learned relationships for certain opposing actions such open and close well as fold and stretch are antiparal- lel in the embedding space. Note, however, that the representation does not capture all antiparallel relationships (e.g., add and remove have similarity of 0.1)..
18

---

Published as a conference paper at ICLR 2026
open
break
dirty
clean
UMAP Dimension 2
turnon
close
turnoff
UMAP Dimension 1
Figure 9: UMAP projection of individual delta embeddings from the IID test set. Embeddings shaped by their ground-truth action. The plot reveals strong global separation between different action clusters.
B.2 ANALYSIS OF LEARNED ACTION REPRESENTATIONS
To study the properties of the action representations resulting from our method, we first tested the resulting delta embeddings could reliably predict the outcome of an intervention. To do this, for each sample in the OOD test set, we took the "before' state embedding (2) and added the corresponding average action vector (Paction) that was computed using the training set samples. We then measured the cosine similarity between this predicted 'after' state and the ground-truth 'after' state (2). Our framework showed remarkable predictive power, achieving an average cosine similarity of 0.98 in the single object systematic shift setting. This near perfect score confirms that the learned action prototypes function as true, generalizable transformation vectors, providing strong evidence that our model has learned the underlying mechanies of interventions.
B.3 UMAP PROJECTION
Figures 9 and 10 present the UMAP projection of individual delta embeddings from the IID and OOD test set of the single-object environment respectively. The delta embeddings in the IID set- ting achieve a clear separation between each action, leading to a near perfect III) accuracy as was presented by experiments. On the other hand, while strong intra-class cohesion is visible, the global separation between these action clusters is not always visually distinct in the OOD setting. This suggests that while representations remain locally coherent, action representations are as clearly discriminated compared to the IID setting. It is worth mentioning, however, that the 2D projection may not fully capture the features of the high high-dimensional latent space.
C ABLATION STUDIES
In order to understand the effectiveness each component our method, we conducted a series of ablation studies to evaluate the impact of different backbone architectures, the impact of loss
19

---

Published as a conference paper at ICLR 2026
open
dirty
close
turnoff
UMAP Dimension 2
clean
turnon
break
UMAP Dimension 1
Figure 10: UMAP projection of individual delta embeddings from the OOD test set. Embeddings shaped by their ground-trus truth action. The plot reveals strong local cohesion (points of the same shape cluster together) but shows lack of clear global separation between the different action clusters.
hyperparameters a and the impact the hyperparameter & in the Top-K selection procedure for our the single-object systematic shifts setting,
Patch-Wise model. All the subsequent experiments except for the Top-K ablation study which ran on the multi-object systematic shifts setting. Also these experiments
performed in smaller subset of the dataset.
Table compares the OOD performance of the best configuration for each backbone against the benchmark's 's state-of-the-art ResNet-18 result. Our final ViT-based model significantly outperforms the best ResNet-based model, demonstrating that while the richer features from VIT enhance per- formance, the substantial gains are primarily driven by our proposed CDE learning framework.
Table 4: Comparison of OOD performance with differ- ent backbone architectures the single-object system- atic shift benchmark.
Backbone
Method
OOD Aec. (%)
ResNet-18
Liu et al.
2023
0.54
ResNet-18
Ours'
0.45
ViT-DINO
Ours (CE Only)
0.67 0.75
VIT-DINO
Ours (Full Model)
Best ResNet performance from experiments was with Con Loss.
C.I IMPACT OF BACKBONE ARCHITECTURE
To isolate the contribution of our CDE framework from the choice of feature extractor, conducted a controlled comparison between our ViT-DINO backbone and the ResNet- 18 backbone used the original Causal Triplet benchmark.
20

---

Published as a conference paper at ICLR 2026
C.2 IMPACT OF LOss HYPERPARAMETERS
In order to select values for Occontrast and Osparsity, we conducted an ablation study comparing various values and combinations between them. Table 5 compares
of the combinations of the val- ues that we experimented with. Selecting a larger value for Olcontrast: rather than Sppuriity helps the model learn better representations, thus achieving better OOD accuracy. We set Cloonrratt 2.0 and Asparsity = 1.0 in all our main experiments.
Table 5: Comparison of various hyperparameter values for Okguntrast and Ossurrityy on the single-object systematic shift benchmark.
Ccentrast
Cispariity
OOD Acc. (%)
0.0
0.0
0.20
0.1
1.0
0.27
1.0
0.1
0.28 0.28
0.5
0.5
0.29
1.0
2.0
0.31
2.0
1.0
0.33
C.3 TOP-K SELECTION
In order to select the hyperparameter l: in multi-object and real world data settings, we executed an ablation study understand the sensitivity of our method to this parameter. presented in T we can see that OOD accuracy increases as & increases too. This observation makes sense, since bigger objects (e.g. Fridge, Bed) would need more patches for their representations in order to be captured effectively. Thus, we set the value of k = across all our multi-object and real world experiments.
Table 6: Comparison of OOD performance with k values for the patch selection process in multi- object settings.
ProcTHOR
EpicKitchens
0.42
0.12
k: 0.45
0.13
0.47
0.13
0.48+0.04
0.15
D EXPERIMENTAL DETAILS
D.1 HYPERPARAMETERS
Table summarizes the key hyperparameters used across all experiments. These values were lected based on ablation studies and remained consistent across different experimental settings un- less otherwise noted.
D.2 EXECUTION ENVIRONMENT
All experiments were run on NVIDIA A100 GPU with the Slurm Workload Manager. The code was implemented in Python, using the Pytorch library. Each run takes approximately hour to complete for the ProcTHOR and two hours for the Epic-Kitchens dataset.
D.3 IMAGE AUGMENTATIONS
We do not apply any augmentations to the images, since do not want to modify the interventional nature the pairs. Augmentation in this problem could harm our assumptions. For example, a rota- tion could affect equation 1 and eliminate the faithfulness of the encoder. We leave it future work
21

---

Published as a conference paper at ICLR 2026
Table 7: Summary of hyperparameters used across all experiments.
Parameter
Value
Learning Rate
x 10
Backbone LR
1 x 10-6
Batch Size
128
Epochs
50 (100 for Epic-Kitchens)
Weight Decay
0.05
Ccontrast
2.0
Osparsity
1.0
Temperature (7)
0.07
Top-K (k)
4
Embedding Dim. (1)
256 (512 for Epic-Kitchens)
Input Resolution
224 x 224
to investigate whether augmentations can boost OOD performance under different assumptions. We only resize images 224 224 pixels and apply zero-mean normalization with unit variance.
D.4 OPTIMIZATION
We use batch size of 128 and an AdamW Loshchilov & Hutter (2017) optimizer with cosine annealing learning scheduler for 50 epochs. the real world setting, we instead train for 100 epochs. The VIT feature extractor is not frozen but fine-tuned with a reduced learning rate of 10% of the network's base learning rate, which is set 10-4 The weight decay parameter is 0.05. All reported results include standard deviations computed over three independent runs with different random seeds.
D.5 MODEL ARCHITECTURES
To ensure reproducibility and clarify the pre- vre-training objectives used in our experiments, Table tails the specifications for the vision backbones used in our Causal Delta Embedding (CDE) frame- work and the comparative baselines. We utilize the implementations provided by the i library.
Table 8: Comparison of Vision Backbones utilized in experiments.
Model
Architecture
Pre-training Objective
Dataset
Params
VIT-DINO
ViT-S/16
Self-Supervised (Distillation)
ImageNet-1k
21.7M
ViT-MAE
ViT-B/16
Self-Supervised (Reconstruction)
ImageNet-Ik
85.8M
CLIP
ViT-B/16
Weakly-Superviss (Contrastive)
Web-400M
866
ResNet-18
R
Supervised (Classification)
ImageNet-Ik
11.7M
E ACTIONABLE COUNTERFACTUALE AND DELTA EMBEDDINGS
we relax the requirement for identical noise across observations (E * 0 in equation 3) then the delta embedding of equation | becomes (assuming column vectors)
8
= [E1
62,
3. 2a +
where is the number of vector dimensions and c,, C2 etc are assumed to be zero-mean noise vari- ables that do not depend on or ب 2n We use the following representation for convenience 5f [u. n where u, the part of the representation that does not depend on and u, part that does. Consider that this representation is used with a binary logistic regression model with parameters wr [ w!' The gradient of the binary cross-entropy loss with respect to w.
[Cce] E. [[a (w? u, + w." u, b) a) uz]
22

---

Published as a conference paper at ICLR 2026
Note that at w. this becomes
Ww. [CoE] lw.=0 E [(ar (wa2 u. +b) a) u
Given that u. is independent of a, ua and b then this equation can be factorized as
Vw. [CoR] ww Eug.a [(or (w" u. + b) -a) Eu, [u.] =
since ....... are zero mean variables. This means that 0 is stationary point of LCE. Since Lee is convex for logistic regression, for any given Wa and b. w. 0 is a global minimum of the loss. This implies that under imperfect interventions (actionable counterfactuals) under the zero mean independent noise assumption, the cross-entropy loss is minimized where the nonzero noise variables are ignored. Similar arguments apply in the multiclass case.
F ADAPTATION OF OBJECT- CENERTCC MODELS FOR ACTION PREDICTION
To leverage the latent structure of object-centric representations for downstream action reasoning, we adapt the implicit Slot Attention framework (Locatello et al. 2020b). The model decomposes each input scene into set of N spatially and semantically related regions (slots), each characterized by its own feature vector. Given pair of pre- and post-intervention images, we explore three distinct aggregation strategies bridge the object-centric slots with the action encoder.
This baseline approach performs average-pooling over the N slots for each im- age independently. The resulting single feature vectors for the pre- and post-intervention images are concatenated and passed to the action encoder, effectively treating the aggre- gated slots as a distributed representation.
Slot-Avg:
Slot-Dense: This strategy densely pairs every slot from the pre-intervention image with every slot from the inseinntevennnonn image, resulting in N x N combinations. All pairs are processed by the action encoder to generate relation embeddings, which are then aggregated via average-pooling to form the final action representation.
Slot-Match:
This method selectively pairs slots across the two images based on cosine similarity. Only the matched pairs are passed the action encoder. The resulting N relation embeddings are aggregated using max-pooling to capture the most significant latent changes corresponding the intervention.
23


Reviews:
[{"review_id": "y2c6X2fWQZ", "reviewer": "Reviewer_Xc8e", "summary": "This paper tackles the task of learning robust representations via interventional data. It proposes Causal Delta Embeddings (CDE) for representing interventions as latent-space deltas between pre/post-states. Results on synthetic and real benchmarks show gains in triplet evaluations and effective identification of changed semantics.", "questions": "* how robust is CDE to noise or partial observability in interventions?\n* does sparsity regularization risk collapsing subtle but real effects?\n* how does the performance change when using different pretrained models? MAE, VQ-VAE's encoder, CLIP  would be very interesting", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "* tackles a very important problem of learning robust and interpretable representations \n* leverages pretrained vision encoders in a good way and moves away from toy-dataset-only evaluations. \n* clear conceptual framing of intervention representation problem\n* strong quantitative OOD gains; well-executed ablations\n* visualization & semantic structure analysis support claims", "weaknesses": "* only evaluates one vit backbone\n* requires heavy supervision that is only possible with synthetic data\n* empirical gains limited in real-world \n* lacks exploration of confounding effects or imperfect interventions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7ttLyAYd9g", "reviewer": "Reviewer_RYXt", "summary": "The paper introduces the concept of delta embeddings to represent atomic causal interventions between two images (e.g. open or close a drawer on the image) and proposes a simple method for classifying intervention actions. The proposed method first separately generates embeddings of a pair of images distant by a single intervention using a backbone Vit coupled with a MLP head tasked to disentangle the embeddings into delta embeddings. Then, the difference between the two delta embeddings is computed and sent as an input to an action classifier. The pipeline is trained end-to-end and regularized with contrastive and sparsity losses to improve the creation of robust representations that hold out-of-distribution. The method achieves improved o.o.d performance compared to baselines and generates embeddings with meaningful relationships between classes of actions.", "questions": "1. How realistic is the assumption of independence of latent factors in the data generative process? Indeed, the presence or absence of an object can have an effect on the lighting of the scene or on objects on top of it.\n2. Must the set of possible actions $\\mathcal{A}$ be known in advance or can the approach generalize to new actions, e.g. compositions of actions?\n3. Have you conducted experiments with additional backbone models, e.g. ResNet-18, for a fair comparison with other baselines?\n4. How do you interpret the drop in performance in o.o.d in Table 2 for the procTHOR multi-object setting?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper tackles a challenging problem in causal representation learning, namely the disentanglement of interventions, using a an original approach. The geometry of Delta embeddings could potentially convey very meaningful information, as hinted by the experiments and visualizations in the appendix.\n2. The paper is well-written and easy to understand. The theoretical section complements well the description of the approach, justifying it accurately.\n3. The experiments on out-of-distribution splits are particularly useful for assessing the generalization of the proposed delta embeddings.", "weaknesses": "Experiments are conducted on a single benchmark (causal Triplet), which limits the generalizability of the findings (although the experiments in o.o.d settings mitigate this issue). Using larger backbone models on more datasets would further strenghten the contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hw9sPf6b2C", "reviewer": "Reviewer_AZZ2", "summary": "This paper proposes to learn representations corresponding to interventions, in contrast to learning representations for the underlying variables. The motivation behind this is to learn reusable representations/embeddings for interventions invariant of the object on which the intervention acts. From the paper, I gather that such representations are useful in Visual Language Action (VLA) models.", "questions": "**Q1. Related works on CRL from interventional data**: In related works, some works on causal representation learning (CRL) that use interventional data are mentioned. However, these works are not mentioned in the introduction when the story is built around the existing state of CRL. I think it is important to highlight the current advances in CRL using interventional data, and how this work is different from them. Also, I would suggest adding more recent works in CRL using interventional data. See [A1-3] and the references therein. [A1] is a contemporary work. The authors need not include it, but I shared it here as a source for recent works.\n\n**Q2. Related works on contrastive learning**: There are two sentences on contrastive learning -- one listing two general contrastive learning works, and another providing a slightly unfair comparison w.r.t. this work. While contrastive learning only compares individual samples, it also does not require any pre- and post-intervention sample pairs, like this work. Another thing is that several works that link contrastive learning to CRL are missing. Some of them are [A4-5] and related works in [A6].\n\n**Q3. Related works on SMS**: Again, a few important works on SMS are missing from your literature survey. See [A7-9].\n\n**Q4. Minor writing comments**:\n\n**Q4. (a)** I suggest rewording the sentence in line 121, starting with \"While previous methods...\" to clarify what part of the interventional mechanism is captured by CDEs and what invariance (across contexts) is targeted by CDE.\n\n**Q4. (b)** Line 154: \"a set of causal variables $Z\\in\\mathcal{Z}\\subset\\mathbb{R}^l$...\" Which variable is the set here?\n\n**Q4. (c)** Line 161: \"a complex, **non-invertible** generative function...\" How can you learn actions if the variables on which the action works are not retrievable?\n\n**Q4. (d)** How did eq. (3) come about? Is $f$ in eq. (3) same as $f$ in line 157? How did the second equality in eq. (3) come about? Are the non-zero indices of $\\delta_a$ aligned with the changes in $z_a$ due to the action?\n\n**Q4. (e)** Can you clarify what the sentence in line 323 starting with \"If, however, these...\" means?\n\n**Q4. (f)** I find lines 425-427 to be slightly misleading. In Fig. 8, there are indeed some anti-parallel representations learned. But there are also some nearly-anti-parallel representations for action pairs that do not make sense. For example, cut-break, move-pull and move-eat have around -0.75 in Fig. 8.\n\n**Q4. (g)** Do any of the numbers in Tables 5 and 6 appear anywhere in the main results?\n\n**References**\n\n[A1] Pranamya Kulkarni, Puranjay Datta, Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, Ali Tajer, \"ROPES: Robotic Pose Estimation via Score-Based Causal Representation Learning\", ArXiv 2025.\n\n[A2] Burak Varıcı, Emre Acartürk, Karthikeyan Shanmugam, Ali Tajer, \"Linear Causal Representation Learning from Unknown Multi-node Interventions\", NeurIPS 2024.\n\n[A3] Dingling Yao, Dario Rancati, Riccardo Cadei, Marco Fumero, Francesco Locatello, \"Unifying Causal Representation Learning with the Invariance Principle\", ICLR 2025.\n\n[A4] Julius von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, Francesco Locatello, \"Self-Supervised Learning with Data Augmentations Provably Isolates Content from Style\", NeurIPS 2021.\n\n[A5] Roland S. Zimmermann, Yash Sharma, Steffen Schneider, Matthias Bethge, Wieland Brendel, \"Contrastive Learning Inverts the Data Generating Process\", ICML 2021.\n\n[A6] Dingling Yao, Danru Xu, Sébastien Lachapelle, Sara Magliacane, Perouz Taslakian, Georg Martius, Julius von Kügelgen, Francesco Locatello, \"Multi-View Causal Representation Learning with Partial Observability\", ICLR 2024.\n\n[A7] Elliot Layne, Jason Hartford, Sébastien Lachapelle, Mathieu Blanchette, Dhanya Sridhar, \"Sparsity regularization via tree-structured environments for disentangled representations\", ArXiv 2024.\n\n[A8] Sébastien Lachapelle, Pau Rodríguez López, Yash Sharma, Katie Everett, Rémi Le Priol, Alexandre Lacoste, Simon Lacoste-Julien, \"Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies\", ArXiv 2024\n\n[A9] Danru Xu, Dingling Yao, Sébastien Lachapelle, Perouz Taslakian, Julius von Kügelgen, Francesco Locatello, Sara Magliacane, \"A Sparsity Principle for Partially Observable Causal Representation Learning\", ICML 2024.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of learning representations for actions is interesting, and it is a reasonable choice to learn these representations using interventional data. I believe Causal Delta Embeddings (CDEs) will have practical applications in robotics and other interactive domains.", "weaknesses": "I include my major concerns under \"weaknesses\" and minor concerns (mainly related to writing) under \"questions.\" My most important concerns are W1, W2, and W5 (d, e). Most weaknesses/questions can be answered without experiments. I will raise my score if my concerns are addressed.\n\n**W1. Interventional vs counterfactual**: A major confusion I have is whether CDE requires interventional or counterfactual data, the latter being a stricter requirement. In lines 194-197, the goal is stated to learn some function using a dataset of pre- and post-interventional data. If these data samples are obtained with the same exogenous noise, they are counterfactual samples. The equation in lines 157-158 mentions the exogenous noise $\\epsilon$, but its role in the training dataset is unclear. \"Identical noise\" is assumed in lines 211, which implies the requirement is counterfactual data. To satisfy eq. (1) using interventional data, the encoder $\\phi$ must disregard the exogenous noise completely. Can the authors clarify what the real requirements are?\n\n**W2. Should the underlying representation be identifiable?** Even if counterfactual data is provided, eq. (1) can be satisfied only using an identifiable $\\phi$. Is my understanding correct? If that's the case, will such an identifiable encoder $\\phi$ be automatically learned while learning CDE, or is it a starting requirement? If my understanding is wrong, please provide a counterexample and include it in the text. I have a related concern on line 216. It is written \"an action's representation is independent of the causally irrelevant...\" Does \"independent\" here mean statistically independent, or that it is not affected by the \"causally irrelevant elements?\"\n\n**W3. Will the aggregation module allow patch-level CDE?** In Sec. 5.2.1, the loss function is applied only over the aggregated embeddings. It is possible for patch-level embeddings to not follow Def. 2 even when the aggregated embedding satisfies Def. 2. Is that true? Are patch-level CDEs not required to follow Def. 2?\n\n**W4. Comparison to BISCUIT**: BISCUIT (Lippe et al., 2023) also encodes the action variable responsible for interventions. See Fig. 7 in (Lippe et al., 2023). Is it possible to compare CDE against BISCUIT?\n\n**W5. Questions on experiments**:\n\n**W5. (a)** What are the baseline models trained on? Lines 257-259 say that the encoder is a DINO-pretrained vision backbone. Are ResNets and ViT in Tables 1 and 2 also DINO-trained? Why is it compared against a CLIP in Table 2, instead of another DINO? What exactly is the oracle-mask approach?\n\n**W5. (b)** How are object-centric models, such as Slot Attention, adapted to predict actions?\n\n**W5. (c)** Why is OOD Comp. accuracy smaller than OOD Syst. accuracy for all baselines, except CDE in Table 1? OOD Syst. seems to be a more difficult task.\n\n**W5. (d)** How is CDE able to achieve that much accuracy in OOD Syst. in Table 1? The manifestations of actions in images are linked to the object on which it acts. So how can the model foresee what the action will look like on an unseen object? I can think of a possibility: CDE works only in scenarios where the action manifestation on the unseen object was seen during training, and that can also maybe explain the difference between OOD Comp. and OOD Syst. accuracies. Can the authors explain why CDE works on OOD Syst.?\n\n**W5. (e)** The CE-only model in Table 3 beats all baselines in Table 1. I thought vanilla-R and vanilla-V were also trained with just CE. So does CDE benefit from something beyond its loss functions?\n\n**W5. (f)** What is Fig. 10 supposed to convey? If the point was to show that CDEs are suitable for k-NN, then it must be compared with other baselines or other variants of CDE with fewer losses (like in Table 3). Although I would appreciate answering this question with experiments, I will not reduce my score if the experiments are not provided, as it is not a main experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VRk3rHXRe4", "reviewer": "Reviewer_jtzb", "summary": "The paper proposes Causal Delta Embeddings (CDE), a method to learn robust representations of interventions from image pairs. The key idea is to represent an intervention as the difference between pre- and post-intervention latent embeddings. The method combines cross-entropy, supervised contrastive, and sparsity losses to enforce independence, sparsity, and invariance. Results on the Causal Triplet benchmark show strong out-of-distribution generalization and meaningful representation structure.", "questions": "See weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear motivation and solid theoretical grounding.\n- Simple but elegant framework using delta embeddings to capture interventions.\n- Strong experimental performance.\n- Well-written and well-structured paper, with convincing ablations and visualizations.", "weaknesses": "- The method is trained with three losses. It is unclear how sensitive the performance is to the weighting between them. Although the appendix provides implementation details, it is not clear how one would set these weights in practice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)