Sanity Checking Causal Representation Learning on a Simple Real-World System
Authors: Juan L. Gamella
Simon Bing
Jakob Runge
Venue: ICML 2025 oral (2025)
Keywords: causal representation learning
benchmarks
causality
Primary area: general_machine_learning->representation_learning
TL;DR: We provide a sanity test for CRL methods and their underlying theory, based on a carefully designed, real, physical system whose data-generating process matches the core assumptions of CRL, and where these methods are expected to work.

Abstract: We evaluate methods for causal representation learning (CRL) on a simple, real-world system where these methods are expected to work. The system consists of a controlled optical experiment specifically built for this purpose, which satisfies the core assumptions of CRL and where the underlying causal factors---the inputs to the experiment---are known, providing a ground truth. We select methods representative of different approaches to CRL and find that they all fail to recover the underlying causal factors. To understand the failure modes of the evaluated algorithms, we perform an ablation on the data by substituting the real data-generating process with a simpler synthetic equivalent. The results reveal a reproducibility problem, as most methods already fail on this synthetic ablation despite its simple data-generating process. Additionally, we observe that common assumptions on the mixing function are crucial for the performance of some of the methods but do not hold in the real data. Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application. We hope the benchmark serves as a simple, real-world sanity check to further develop and validate methodology, bridging the gap towards CRL methods that work in practice. We make all code and datasets publicly available at <anonymized>.

Sanity Checking Causal Representation Learning on a Simple Real-World System
Juan L. Gamella" "1
Simon Bing
23 Jakob Runge 43
Abstract
theoretical treatise (Comon, 1994; Hyvärinen & Pajunen, 1999; Bengio et al., 2013; LeCun et al., 2015; Lake et al., 2017), as well as practical applications (Johansson et al., 2022; Tibau et al., 2022; Lopez et al., 2023). Causal repre- sentation learning is among the newest approaches this line of work, and its enticing promise lies in recovering the causal latent ground-truth model from high-level obser- vations of system. While this problem is provably hard (Locatello et al., 2019), there has been a considerable effort to advance the understanding of the theoretical constraints within which causal representation learning may be success- fully applied (Hyvärinen & Morioka, 2016; Khemakhem et al., 2020; Gresele al., 2021; Brehmer et al.. 2022; Ahuja et al., 2023; Zhang et al., 2023; Lippe et al., 2022: 2023a;b; Lachapelle et al., 2023; Vanci et al., 2023; Saengkyongam et al., 2024; von Kügelgen et al., 2024; Yao et al., 2024b), Applying established causal inference methods to unstruc- tured, high- nigh-dimensional data, addressing the inherent limi- tations of machine learning methods in out-of -ffto-ii-ittuutioon scenarios, or learning mechanistic models of the world are among the multitude of promises that drive research in the field of causal representation learning (Schölkopf et al., 2021).
We evaluate methods for causal representation learning (CRL) on simple, real-world system where these methods are expected to work. The system consists of a controlled optical experi- ment specifically built for this purpose, which satisfies the core assumptions of CRL and where the underlying causal factors-th the inputs to the experiment-and are known, providing a ground truth. We select methods representative of different ap- proaches to CRL and find that they all fail recover the underlying causal factors. To under- stand the failure modes of the evaluated algo- rithms, we perform an ablation on the data by substituting the real data-generating process with a simpler synthetic equivalent. The results reveal a reproducibility problem, as most methods al- ready fail this synthetic ablation despite its simple data-generating process. Additionally, we observe that common assumptions on the mix- ing function are crucial for the performance of some of the methods but do not hold in the real data. Our efforts highlight the contrast between the theoretical promise of the state of the art and the challenges in its application. We hope the benchmark serves simple, real-world sanity check to further develop and validate methodol- ogy, bridging the gap towards CRL methods that work in practice.
A challenge that obfuscates the progress towards these goals is the lack of meaningful real-world benchmarks, to evalu- ate methods and identify theoretical approaches that hold potential for application. By and large, new methods are evaluated on synthetic datasets generated according their underlying assumptions (Brehmer al., 2022; Ahuja et al., 2023; Lippe et al., 2022; 2023a;b; Squires et al., 2023; Lachapelle et al., 2023; 2024; Liang et al., 2023; Buch- holz et al., 2023; Varieı et al., 2023; Bing et al., 2024; von Küggllgnn et al., 2021; von Kügelgen et al., 2024; Yao et al., 2024c; Xu et al., 2024). This practice provides further vali- dation for the theoretical foundations of these methods but yields limited insight into their applicability to real-world problems. More sophisticated synthetic benchmarks have been proposed (Lippe et al., 2022; 2023a; Ahmed et al., 2021; von Kügglggnn et al., 2021; Liu et al., 2023), e.g., using visualizations of computer games or renderings of three-dimensional scenes. Because these do not cater to the assumptions of any particular method, they are extremely valuable for the standardized evaluation of CRL methods.
1. Introduction
Uncovering the underlying factors that determine the be- havior of system is a problem with a long history, both in
'Equal contribution 'Seminar for Statistics, ETH Zurich "Technische Universität Berlin 'Department of Computer Science, University of Potsdam 'ScaDS.AI Dresden/Leipzig. TU Dresden. Correspondence to: Juan Gamella <juangamella@ aacillll@@mmil...m gmail.com>. Simon Bing <bing @ campus.tuuu tu bebberinn.de>.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Sanity Checking Causal Representation Learning a Simple Real-World System
J Vi Light sensor
0 Angle sensor
0g Angle
sensor
Light source
Linear polartzer
Linear polertzer
Camera output
Light sonsor I2 V22
Light sensor fg, V/3
Im
Red, green and blue levels
Motor setting
Molor setting 82
R, G, Drawn electric current
0₁
C
Figure 1: The light tunnel (left) and a simplified schematic (right) showing its main components and variables. The tunnel consists of controllable light source, linear polarizers mounted on rotating frames, a camera, and sensors to measure light intensity different wavelengths and positions. The inputs to the system (R, G, B, 02) are displayed in bold math print in this figure. The system's 'ss outputs-image data and numeric sensor measurements are denoted by a tilde.
However, their synthetic nature means they are limited testbeds to discuss which CRL approaches-ann and their un- derlying theory-are applicable in the real world. While real-world datasets are often used to illustrate potential ap- plications of new methods (Zhang et al., 2023; Chen et al., 2024; Yao al., 2024a), the absence of ground truth also precludes any in-depth analysis into this matter.
As first application of our sanity check, evaluate representative method from three different families of ap- proaches: CRL methods requiring interventional (or coun- terfactual) data (Brehmer et al., 2022; Ahuja et al., 2023; Zhang et al., 2023; Squires et al., 2023; Buchholz et al., 2023; VarıeI et al., 2023; Liang et al., 2023; Bing et al., 2024; von Küggelgen al., 2024), CRL considering multiple views (Gresele* et al., 2019; Locatello al., 2020; von Kügelggnn et al., 2021; Daunhawer et al., 2023; Ahuja et al., 2024; Xu et al., 2024; Yao et al., 2024c), and CRL based on time-series data (Yao et al., 2021; 2022; Lippe et al., 2022; 2023a;b; Lachapelle et al., 2024). The results and background needed understand them are given for each method Section 3.
As orthogonal contribution to existing synthetic bench- marks, we propose a real-world sanity check to test the applicability of CRL methods and their underlying theory. The test based on a carefully designed, real, physical system whose data-generating process matches the core assumptions of causal representation learning (Section 2). Furthermore, the relationship between latent factors and observations is far simpler than in the synthetic benchmarks mentioned in the previous paragraph. Thus, we base our sanity check on the premise that generic CRL method designed to work in various settings-shoule also work on this simple but real system.
To our knowledge, this sanity check is the first of its kind, and we hope that it can serve as standardized benchmark to evaluate new methods and test the assumptions that underlie theoretical work. However, we find it important to clearly delimit the conclusions that can be drawn from this bench- mark. First, our current setup is geared towards evaluating the assumptions concerning the mixing function, they underpin the identifiability results of many methods, and they are the most straightforward to test. This means that the applicability of other assumptions, for example, regard- ing modeling the latent causal structure using structural causal model (SCM, Peters et al., 2017, Chapter 3), cannot readily assessed using the datasets we currently provide. We further discuss this point in Section 2. Second, as with any test, only negative result is truly informative: method
The physical system consists of unleesodddto optical experiment involving the polarization of light. The sys- tem was first introduced by Gamella et al. (2025) and is described in detail in Section 2. The control inputs to the experiment constitute the underlying causal factors, and its outputs-image and sensor data core-core to the entan- gled observations. Because the control inputs are known, we have ground truth directly evaluate the performance of the tested methods in recovering the underlying causal factors.
2

---

Sanity Checking Causal Representation Learning on a Simple Real- World System
that fails this simple and tightly controlled testbed can be expected to fail in other, more complex, real-world scenar- ios. However, the converse does not hold. While our setup is representative of other systems that produce data with digital sensors, a method that succeeds on our benchmark is not guaranteed to do so on all other real-world systems that are the target of application.
B
E
real
R:128 G:12 B:25
R:134
G:227 B:89
R:64 G:112 B:170
81:73.9 02:54.6
61.76.4 02:53
01:3.5 02:-14.5
C
F
synthetic
The physical system used for our benchmark is one of the two Causal Chambers first introduced by Gamella et al. (2025). Our contribution is to leverage this sys- tem to develop a meaningful sanity check for CRL algorithms. We make the novel datasets and their data-collection procedures publicly available in the rkk_v dataset at github. com/ juangame chamber. The code repro- duce the results of this paper can be found at github. com/s a momb /tg//RLSanniee tycaackk The determin- istic simulators developed for the synthetic ablation (Sec- tion 2) are documented in Appendix C, and we provide a Python implementation in the causa nausalchamber package.
R:128 G:0 B:128 61:0 62:0
R:128
G:0 B:128
81:90
82:0
8. = 6 polarizer effect
4 = 0g
Figure 2: (A-D): Real images collected from the light tun- nel, with the corresponding control inputs overlaid in white. The light source color (R, G, B) is the same for images C and D, but the first polarizer is shifted by 90 degrees in image D, showing the effect of the linear polarizers angles (01, 0z). (F,, F): Comparison between (F) a real image (at low resolution) and (F) the synthetic counterpart produced by simple multi-layer perceptron given the same inputs.
2. Experimental Setup
Our physical system is light tunnel like the one introduced by Gamella et al. (2025). We provide schematic of its main components and relevant variables in Figure 1. The tunnel is an elongated chamber with a controllable light source at one end, two linear polarizers mounted on rotating frames, a camera, and sensors to measure different physical quantities. As control inputs, the system takes the brightness level of the red, green, and blue LEDs of the light source (R, G, B) and the polarizer positions (01, 02). Its outputs are the im- ages captured by the camera (Figure 2) and readings of the infrared (J,, T2, Ta) and visible (Vi, Vz, Vi)) light intensity at different positions, the electrical current drawn by the light source (C), and additional noisy measurements of the polarizer angles (0,, 02). A detailed description of all the variables can be found in Gamella et al. (2025, Appendix II).
A Weal-World Mixing Function. The light tunnel pro- vides an ideal testbed to evaluate assumptions concerning the mixing function that transforms the ground-trut -truth causal factors into observations. In the tunnel, the process that produces the images is far simpler than those in common synthetic benchmarks, which arise, for example, from vi- sualizations of computer games (Lippe et al., 2022; 2023a) or renderings of physical simulator (Ahmed al., 2021; von Kügelgen al., 2021). Evidence of this simplicity is that the process can be simulated to a high degree of visual realism with a simple multi-layer perceptron (MLP) (Fig- ure 2, E/F). Furthermore, the relationship between the light source setting (R, G. B) and the pixel intensities is approxi- mately linear up to the overexposure of the camera sensor, as happens in the central region of the images. The polarizer angles (01, 02) scale the light intensity reaching the camera by factor of cos2 (01 02). dictated by Malus' law (Col- lett, 2005). This dimming affects each color wavelength differently, resulting in distortion of the color noticeable at relative angles close to 90 (see D/E in Figure 2). These effects are thoroughly documented in Gamella et al. (2025, Appendix III). The images also contain some other subtle artifacts, introduced by the chromatic aberration of the lens, the reflections of the polarizer frames, and the processor on- board the camera. Different inputs always result in different images.
As a data-generating process, the light tunnel satisfies the core assumptions of causal representation learning. In par- ticular, it transforms some underlying causal factors the control inputs- into observations consisting of images and numeric sensor measurements. Because we control the inputs to the system, bution or causal structure, as we do in the experiments of Section 3. However, this means that the underlying causal structure is synthetic, and the real-world applicability of the corresponding assumptions cannot be evaluated. Therefore, using the current setup we can only test the assumptions corresponding to the mixing function, but not those placed on the latent. causal model generating the data.
sample them from any distri-
A crucial difference with commonly used synthetic bench- marks is the noise present in the images and sensor measure- ments, which naturally arises from several processes. The
3

---

Sanity Checking Causal Representation Learning on a Simple Real- System
first is the fluctuation of the light source, which modulates its brightness by switching on and off at frequency of 2KHz, interacting with the photodiodes on the camera and the sensors in a complex way. The second is the measure- ment noise introduced by the sensors themselves, which is to be expected from any real system. For the camera and the third light sensor, which are placed behind the polarizers, the relative strength of these two noise sources is modulated by the polarizer positions 0j and 0g. As result, the "mixing function" of the light tunnel is not deterministic process. Furthermore, this stochasticity is not captured by an additive noise model.
our results, ran preliminary tests on synthetic datasets replicating the settings used in the respective original works (Buchholz et al., 2023; Yao et al., 2024c; Lippe et al., 2022). Our experimental pipeline for each method is described in detail in Appendix A.
3.1. Contrastive CRL
The most prevalent class of CRL methods are those that assume access to data from different environments, stem- ming from interventions (or counterfactual observations) on some of the underlying causal factors (Brehmer et al., 2022;
Ahuja et al., 2023; Zhang et al., 2023; Squires et al., 2023;
Buchholz et al., 2023; Vaniı et al., 2023; Liang et al., 2023;
Synthetic Ablation. To explore the effect that these subtle artifacts have on the performance of causal representation learning methods, we perform additional experiments where we substitute the real data-generating process of the light tunnel with a deterministic simulator that produces synthetic images and sensor measurements given the same inputs (R, G, B, 02). The simulator produces synthetic mea- surements via mechanistic model and synthetic images (Figure 2F) via a simple MLP trained on a separate dataset. Further details about the simulator and its construction can be found in Appendix C.
Bing et al., 2024; von Kügelgen et al., 2024). We choose the method of Buchholz et al. (2023) as representative of this family of models. In the remainder of the text, we will refer to it as Contrastive CRL (CCRL).
Background. As an underlying causal model, CCRL as- sumes linear structural causal model (SCM) with addi- tive Gaussian noise (Buchholz et al., 2023, Assumption 2). The underlying causal factors are transformed into observa- tions through a nonlinear and deterministic mixing function (Buchholz et al., 2023, Assumption 1); while in theory their results can be extended to stochastic mixing functions with additive noise, both their experiments and implementation assume deterministic mixing function. Provided with data from environments that correspond to single-node inter- vention on each causal variable, the goal of the method is to recover the ground-truth causal factors and the graph en- coding the causal structure between them (Buchholz et al., 2023, Theorem I).
3. Results
We apply our sanity check to methods representative of three different families of approaches: contrastive CRL (Section 3),,, multiview CRL (Section 3.2), and CRL from temporal intervened sequences (Section 3.3). As a result, the methods differ greatly in their setup and assumptions on the latent causal structure, as well in their goals and the metrics to evaluate them. This makes a direct comparison between them difficult, and we instead perform our analysis method-by- by-tthoo,, introducing the necessary background together with the results.
Data Generation. In line with the assumptions made by the method, we sample the values for the ground-truth factors-the tunnel inputs R, G, B, 01 and fr-from a lin- ear SCM with additive Gaussian noise; the corresponding ground-truth graph is shown in Figure 3. To generate the interventional data, we perform interventions that shift the mean of their target, closely following the assumptions de- scribed in Buchholz et al. (2023, Assumption 3). We collect 10K observations per environment, constituting total of 60K images provided as input to the method. A detailed exposition of the data-generating process is available in Appendix A.I.I.
In our efforts, we encounter a first challenge in the applica- tion of causal representation learning methods to real-world problems: a recurrent lack of public, well-documented, and tested code. As a result, our choice of methods is biased towards those with public code, or for which the authors provided code upon request.
We find that pre-processing steps and implementation secisions-such as network architectures and training procedures have a drastic effect on the performance of some of the methods. This also highlights potential diffi- culty in re-implementing CRL methods to reproduce their results. To minimize potential points of failure, we used the original implementations as much as possible. We thank the authors of the selected methods for their assistance, which was instrumental in getting them to run for this benchmark. To further ensure that bug in our pipeline does not distort
Results. In Figure 3, we provide a summary of the results of applying CCRL to the real images and to the synthetic images (e.g., Figure 2F) produced by the deterministic sim- ulator of the light tunnel described in Section 2 and Ap- pendix C. We report the same metrics used in the original experiments of the method: the commonly used mean cor- relation coefficient (MCC, Hyvärinen & Morioka, 2016;
4

---

Sanity Checking Causal Representation Learning on a Simple Real-Worke System
02
02
MCC f
SHD +
R
01
R
01
R
01
Real
0.285 + 0.054
7.600 + 0.894
Synth. ablation 0.891 + 0.005
2.000 + 0.000
G B
G+I-B
Ground truth
Synthetic ablation
Real data
Figure 3: Experimental results for the Contrastive CRL method applied to the real data and the data from the synthetic ablation described in Section 2. Left: MCC score of predicting the ground-truth factors, and SHD score between the estimated latent graph and the ground-truth graph (shown the left). We provide the average and standard deviation (+) of the scores over five random initializations of the method. Right: Ground-truth graph and a summary of the estimated latent graphs produced by five random initializations of the method. The edges are shaded and labeled according to the frequency they appear in the estimates obtained in the five runs the method, with darker edges appearing more often. The method performs well on the data from the synthetic ablation, where a deterministic simulator (Appendix C) substitutes the gettraeenrating process of the light tunnel. The method fails to produce meaningful results based on the real data.
Khemakhem et al., 2020), which is measure of how well the method recovers the latent factors, and the structural Hamming distance (SHD, Tsamardinos et al., 2006) that measures the recovery of the latent causal graph.
Im
View
View
View 4
(R, G, B)
01
02
02
The method obtains fairly good metrics on the data from the deterministic simulator, indicating that this is a setting where most necessary assumptions for CCRL are met, al- lowing the method recover the latent variables and graph reasonably well. However, the method exhibits stark drop in performance when applied to the real data, failing to recover the latent variables and returning vastly different latent graphs in each of its five random initializations. Since the main difference between these two settings is the deter- minism of the mixing function (see Section 2), we conclude that CCRL is highly sensitive noise in the mixing process. A possible explanation is that CCRL relies on detecting in- terventions in the latent space, a sensitive statistical problem for which it may lack power in the case of noisy mixing process.
01
View 3
Vi
iz
V/2
Figure 4: Ground-truth graph relating the underlying causal factors, shown in bold print, to the different views employed in the Multiview CRL experiment. The views are disjoint sets of the output variables produced by the tunnel. The separate factors (R, G, B) are shown as a tuple to avoid drawing additional edges. The graph is subgraph of the complete causal ground-truth graph for the light tunnel, described in Gamella et al. (2025, Figure 3b).
3.2. Multiview CRL
continuous distribution with a positive density almost ev- erywhere is allowed (Yao et al., 2024c, Assumption 2.1). The underlying factors are transformed into observations via multiple diffeomorphic mixing functions that share (sub- sets of) these factors as their inputs, resulting in multiple views on the underlying causal model. For our testbed, we group the images and sensor measurements produced by the light tunnel into different views, as shown in Figure 4. Given set of views, the goal of the method is to recover the underlying factors in the intersection of their inputs. These shared factors are referred to as the "content block" of set of views, whereas those not in the intersection of inputs are called the "style block". Regarding identifiability, the the- ory guarantees partial (or block-) identifiability (Lachapelle
The multiview approach to CRL (Gresele* et al., 2019; Locatello et al., 2020; von Kügelgen et al., 2021; Daunhawer et al., 2023; Ahuja et al., 2024; Xu et al., 2024; Yao et al., 2024c) relies on having different sets of observations, or views, that arise from mixtures of different subsets of the underlying causal factors. We choose the method introduced by Yao et al. (2024c) as representative of this family of approaches.
Background. As opposed to CCRL, the method from Yao et al. (2024c) places more flexible assumptions on the distri- bution of the underlying causal factors, where any smooth,
5

---

Sanity Checking Causal Representation Learning on a Simple System
view
view
view
view 3
view (im)
view 4
(fm) vs. (C, I, V1 Iz.
V2)
(Itu) vs. (01)
vs. (42)
Ground-truth latent vs. view 4
AVV.. R
A
B
C
D
view
0.5
0²
0
real
4
synth
1
1
R² = 0.999
R
G
B
01
02
R
62 content style
R
G
B
01
02
62 latent
content
style
style
style
content
Figure 5: Experimental results for the multiview CRL method over five random initializations. (A, B, C): Average R2-score of predicting the ground-truth factors from the learned representation across different pairs of views. Success is indicated by simultaneously attaining a score near one for the factors in the content block and a significantly lower score for those in the style block. While content variables are consistently predicted better than style variables. the model fails to disentangle 01 and 02, evident in panel B & C. (D): Scatter plot of the ground-truth factor 0g vs. the corresponding sensor measurement in view 4 (02). Even though this view and the ground-truth factor are almost perfectly correlated, the model fails to recover the underlying factor 0z.
& Lacoste-Julien, 2022) of the ground-truth factors in the content block up to an invertible transformation. The infor- mation about the factors in the style block is discarded as the model learns "[...] all and only information about [the content block]" (Yao et al., 2024c, Definition 2.3).
Results. In Figure 5, we display the results of applying the multiview CRL method (Yao et al., 2024c) on both the real data and the synthetic ablation described in Section 2. We report the same metric used in the original paper: the R² score (Wright, 1921) of predicting each ground-truth factor using non-linear estimator, fit on the encodings returned by the method for each view, and averaged for each pair of views shown in panels A-C Figure 5. Further details about the metric can be found in Appendix A.2.3.
Data Generation. To generate the ground-truth causal factors, we employ the same sampling procedure as for CCRL. Due to its flexible constraints on the underlying factors, the distribution entailed by the SCM and the in- terventions falls under the method's assumptions, and we pool the observations from the different interventional envi- ronments. See Appendix 1. for an in-depth description of the data-generating process. The input to the method consists of 60K observations, whose variables are grouped into the four views shown in Figure 4. The first view is the images (Im) captured by the camera, which depends on all ground-truth factors (R, G, B, B,B, 02). The second view comprises the combined measurements of the current drawn by the light source (C) and the light intensity at the first two sensors (I,, Vi, I2, 122); these depend only on the light-source setting (R. G, B). The third and fourth views are the measurements of the two angle sensors 01 and 0g. These are simple linear functions (e.g., Figure 5D) of the polarizer positions 0, and 0z, respectively. In line with the experiments in the original paper (Yao et al., 2024c, Section 5.3), during training we rely on the ground-truth content selection of the shared encodings, as opposed to learning it. More details can be found in Appendix 1.2.1.
Success is indicated by simultaneously attaining high score (near one) for the underlying factors shared by both views (the content block) and a score significantly lower than one for those factors that are not (the style block). In the case where the underlying factors are independent, the score for the style is expected to be near zero, but given the correla- tion between variables induced by the underlying SCM, this is not the case here. While the method consistently recovers the variables in the content block better than those in the style block, the difference in scores is not large, as is partic- ularly evident for the intersections of view I with views 3 and 4, where this difference is barely noticeable (Figure 5 B and C, respectively). This implics that the method fails to separate the information between the content and style vari- ables, which is one of the method's goals (Yao et al., 2024c, Definition 2.3). It is noteworthy that there is prominent decrease in performance from the synthetic to the noisy real data. A hypothesis for this phenomenon lies in the subtlety of the trace of the polarizer angle information (0) 02) in the image data and the model's failure to learn this pattern in the image encoder. Given the fact that image data is captured
6

---

Sanity Checking Causal Representation Learning on a Simple System
Real data
Synth. ablation
1.0
R
-0.02
00.00
0.05
0:21
0.07
-0.00
0.02
Learned latents
0.8
0.21
0.11
-0.02
-0.01
0.14
0.01
0.10
0.00
0.02
R² diag f
R² sep
B 0.00
-0.00
-0.01
-0.01
0.00
00.00
-1.01
-002
-0.00
00000
Real
0.092 + 0.056
0.620 + 0.091
01 8.15
0.24
0.30
0.17
0.29
0.35
0.45
0.28
0.35
0.20
004
Synth. ablation
0.120 + 0.068
0.636 + 0.107
02
0.31 0.32
0.44
0.49
0.15
0.18
0.13
0.37
0.52
0.51
-0.
NA
0.00
-0.01
-0.02
-0.00
-0.00
-0.00
-0.01
-002
-0.00
-0.00
R
G
B
0,
02
R
G
B
0 02
avg. R²
Ground-truth causal factors
Figure 6: Results for the CITRIS method on real and and the synthetic ablation as described in Section 2. Left: R² correlation metric over five random initializations. We report the average and (+)) the standard deviation. The diag columns refer to the average value of the diagonal of the R² correlation matrix (shown on the left) while the sep columns denote the maximum off-diagonal value. Right: Average R² correlation matrices for real and synthetic ablation datasets. NA refers to the group of learned latents that are not assigned to any ground-trun causal factor during training (Lippe et al., 2022, Section 3.4). While there is minor drop in performance from synthetic to the real data, this is negligible and the method fails to recover the ground-truth factors in both cases, as indicated by the low scores along the diagonal and (relatively) high off- off-diagonal scores, respectively.
in distinct red, green and blue channels, it is unsurprising the ground-truth factors (R, G, B) are more easily learned by the image encoder than the polarizer angles. However, as we show in Appendix B the information of the angles is indeed present in the images, giving us no reason to believe this task to be fundamentally ill-posed.
Data Generation. We generate the ground-truth causal factors by sampling from the temporal causal process de- scribed in Appendix 1.3.1. As in the experiments of the original paper (Lippe et al., 2022, Appendix C.2), at each time step we randomly intervene on one of the factors (also including the possibility intervening on none), and record the vector of the intervention targets at each time step. The original paper uses an additional loss metric (the "triplet" loss, Lippe et al., 2022, Section 6.1) for model selection during training, which requires access to the ground-truth factors and requires the collection of an additional dedicated dataset. Since collecting data on a physical system is costly, and assuming access to the ground-truth factors during train- ing is unrealistic, we forego the use of this metrie. We use the CITRIS-VA VAE variant of this method (Lippe et al., 2022, Section 4.1) for all experiments.
3.3. CRL from Temporal Intervened Sequences
As a third group of CRL approaches, we consider meth- ods that exploit the causal temporal structure of a process evolving time (Yao et al., 2021: 2022; Lippe al., 2022: 2023a;b; Lachapelle et al., 2024). To represent this group of methods, we select CITRIS (Lippe et al., 2022), for which well-documented code is publicly available.
As in the original paper (Lippe et al., 2022, Section 6.1), we generate an additional dataset from independently sampled underlying factors to calculate the final metrics: the R² (Wright, 1921) and Spearman's rank correlation coefficients (Spearman, 1904), which show how well the model can predict the ground-truth factors from the latents learned during training. Please see Appendix A.3.3 for details on these metrics.
Background. CITRIS assumes that the underlying causal factors follow first-order Markov, stationary dynamic Bayesian network (DBN, Dean & Kanazawa, 1989; Mur- phy, 2002) without instantaneous causal effects (Lippe et al., 2022, Section 2). Furthermore. the method assumes to have access to information about which underlying causal fac- tor has undergone an intervention at each time step (Lippe et al., 2022, Section 3.1). In contrast to most other methods, CITRIS allows for multidimensional causal ground-truth variables. It explicitly models observations as variables with noise in its theoretical assumptions (Lippe et al., 2022, Section 3).
Results. We present the results for the R² score in Fig- ure 6; the results for the Spearman's 's rank coefficient are provided in Appendix A.3.4. Both metrics metrics-empiiipepee employed the original paper-produce correlation matrix between
eecerocseeepooedd in
7

---

Sanity Checking Causal Representation Learning on a Simple Real- System
the learned latents and the ground-truth factors. An op- timal model displays values close one on the diagonal and zeros everywhere else (see Appendix A.3.3 for details), The results indicate that CITRIS fails catastrophically in recovering the ground-truth factors, both from the real data and the ablation with synthetic images and measurements. The metries along the diagonal display a low score, while the off-diagonal values are relatively higher. This suggests that the model does not learn encoding that separates the ground-truth causal factors nor a correct mapping between latents and causal factors.
failure is catastrophic, and the output of the methods has a very weak or no association with the ground-truth causal factors. To better understand the reasons for this failure, we performed a synthetic ablation on the data by constructing a deterministic simulator to substitute the real-world data- generating process of the light tunnel. However, save for the CCRL method, performance on this simpler synthetic data was not significantly better than on the real data.
To better understand the failure modes of the algorithms, it is useful to draw the distinction between failure due to misspecification and failure due to optimization. To prove identifiability, CRL algorithms assume a data-generating process with different assumptions surrounding distribu- tions, number and type of interventions, and the functional form of mechanisms, among others. Given a generative process that follows these assumptions, methods typically show that optimizing a certain objective-under the assump- tion of infinite data-recovers the data-generating causal variables up to accepaable indeterminacies such as permuta- tion and element wise rescaling of variables. Thus, failures in practice can be attributed either to a misspecification of the assumed data-generating process or to issues in the optimization procedure.
CITRIS is a complex method with many sub-modules that interact during training. We hypothesize that a failure in one of these modules, such as the encoder struggling to extract information about one of the latents, can compound and drive down the method's ' performance. Although CITRIS and its variants have shown favorable performance on sev- eral visually complex synthetic datasets (Lippe et al., 2022; 2023a;b), the exact reason for the failure in this setting re- mains elusive. Further analysis to pinpoint where things go wrong would require an in-depth study and additional ablations on the underlying data-generating process, e.g., by considering a simpler benchmark with only R, G, and B as ground-trute truth factors. We believe this to be beyond the scope of this work.
To control for misspecification failures, we exactly adhere to each method's 's assumptions surrounding the -ata-gene process, leaving only the real-world mixing function as a source of misspecification. As a negative control for its effect, we carry out synthetic ablation experiments, using deterministic simulators as well-specified synthetic surro- gates of the real-world mixing function. The CCRL method succeeds in recovering the latent variables from the synthetic ablation data and fails on the real data, providing evidence that the failure mode for this method can be traced back to a misspecification of the real-world mixing function. The same conclusion cannot be readily drawn for the multiview CRL method and CITRIS, as they fail on both the real and synthetic ablation data. We must conclude that their failure likely lies in combination of both misspecification and optimization failure.
4. Discussion
We have constructed testbed for causal representation learning using real and tightly controlled physical system built around well-understood optical experiment. The data-generating process of this system satisfies the core as- sumptions of causal representation learning, and the straight- forward mixing process is described by simple physical principles. Furthermore, we have access to ground-truth la- bels of the underlying causal factors. Therefore, the testbed serves as a sanity check where a CRL method that is ex- pected to work on variety of real-world scenarios should also succeed. A failure on this testbed indicates potential failure on other, more complex real-world systems. How- ever, the opposite does not hold, as success on this simple and controlled testbed is not guaranteed to carry over to more complex scenarios.
To exclude key potential failures in the optimization process, we check for non- -on-decraginng losses and we perform a large hyperparameter search for all methods (cf. Appendix D). Additionally, we make a substantial effort exclude bugs and other issues by performing preliminary checks pipeline, reproducing the synthetic experiments in the origi- nal papers, and consulting the respective authors to advise us where possible. Exhaustively covering the vast space of all factors surrounding optimization, such as alternative model architectures and training regimes, is beyond the scope of this work, which is sanity check for existing methods as they would be available to a practitioner. For the multiview CRL method and CITRIS, we cannot exclude
As a first application of our testbed, we evaluated three methods representative of different approaches to causal representation learning: contrastive CRL, multiview CRL, and CRL from temporal intervened sequences. Due to the different setups, assumptions, and goals of each method, we performed our analysis on a method-by by-thoo basis. How- ever, some general patterns emerge. First, all methods failed to attain their goals in recovering-up to their correspond- ing theoretical constraints- the ground-truth causal factors. Except for single sub-result of the multiview method, the
8

---

Sanity Checking Causal Representation Learning on a Simple Real-Words System
that architecture choices, as well as finite sample issues, are not responsible for their failure.
Acknowledgements
We thank Francesco Locatello for valuable discussions, and Simon Buchholz and Dingling Yao for providing support for their respective model implementations. We also wish to thank the anonymous reviewers for their comments and feedback. S.B. and J.R. received funding for this project from the European Research Council (ERC) Starting Grant CausalEarth under the European Union's 's Horizon 2020 research and innovation program (Grant Agreement No. 948112) 12). S.B. received support from the German Academic Scholarship Foundation. This work used resources of the Deutsches Klimarechenzzentrum (DKRZ) granted by its Sci- entific Steering Committee (WLA) under project ID 1083.
This opaque understanding of failure modes highlights deeper problem, underscoring the fragility of these methods and their dependence on pre-processing steps and other im- plementation decisions, such as model architectures, train- ing regimes, and hyperparameter values. This constitutes significant challenge for reproducibility and, ultimately, the application of these methods to real-world problems.
4.1. Outlook
Although we advocate for the of real-world data, we believe that synthetic data is crucial for the development and validation of new methods. However, there is inher- ent conflict of interest if the validation of new method is limited to synthetic data produced by its authors. Method- agnostic benchmarks have been catalysts of progress in various other fields (LeCun et al., 1998; Deng et al., 2009; Rajpurkar et al., 2016; Runge al., 2019; Lin al., 2022), especially if they include or resemble data that may find in the real world. Thus, we applaud existing efforts in this regard, including the synthetic benchmarks men- tioned throughout the manuscript (Lippe et al., 2022; 2023a; Ahmed et al., 2021; von Kügelgen et al., 2021; Liu et al., 2023).
Impact Statement
This paper provides a sanity check to validate advances in causal representation learning. Because our contribution pertains to fundamental and theoretical research in machine learning, the potential societal consequences are difficult to ascertain and, in any case, not immediate. Thus, we do not feel any ethical concerns must be specifically highlighted here.
While the community acknowledges that applications of CRL to real-world problems lacking, the response has largely been to push the theoretical frontier and relax the assumptions needed to establish identifiability results. We argue that, in many regards, the theory in the field is already quite advanced and is, therefore, not the only bottleneck. It equally important to make real effort to implement and apply the existing theory in a robust, reproducible, and effi- cient manner. Without pursuing the engineering endeavor to develop actionable algorithms from the existing theory, we cannot hope to know where the true obstacles to application lie.
Unless we abandon the ultimate goal of applying causal representation learning real-worids problems, we cannot be satisfied with validating our methods on purely synthetic data. Our goal with this physical testbed is to provide re- searchers in the field with a stepping stone toward more complex real-world scenarios, with the hope of breaking down this hard problem into manageable steps.
The datasets we provide constitute a small fraction of all experiments possible with the light tunnel, which offers additional control inputs not described here that be used to construct more complex tasks. We are open suggestions for additional experiments that may prove useful-pleat reach out to the corresponding authors.

---

Sanity Checking Causal Representation Learning on a Simple Real- System
References
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vi- sion and Pattern Recognition, pp. 248-255 2009. doi: 10.1 1090C/RR.2009955550
Ahmed, O., Träuble, F., Goyal, A., Neitz, A., Wuthrich, M., Bengio, Y., Schölkopf, B., and Bauer, S. CausalWorld A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning. International Conference on Learning Representations, 2021.
Gamella, J. L.., Peters, J., and Bühlmann, P. Causal cham- bers as real-world physical testbed for Al methodol- ogy. Nature Machine Intelligence, 2025, doi: 10.1038/ -42256-024-00964-
Ahuja, K., Mahajan, D., Wang. Y., and Bengio, Interven- tional Causal Representation Learning, In Proceedings of the 40th International Conference Machine Learning, pp. 372 407, 2023.
Gresele*, L.,, Rubenstein P. K., Mehrjou, A., Locatello, F,, and Schölkopf, B. The incomplete rosetta stone problem: Identifiability results for multi-view nonlinear ICA. In Proceedings of the 35th Conference on Uncertainty in Artificial Intelligence, volume 115, pp- 217-227, 2019. *equal contribution.
Ahuja, K., Mansouri, A., and Wang, Y. Multi-domain causal representation learning via weak distributional invariances. In Proceedings of the 27th International Conference on Artificial Intelligence and Statistics, pp. 865-873. 2024.
Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transac- tions Pattern Analysis and Machine Intelligence, 35 (8):199-1882) 2013. doi: 10. 101.100.M...01.....
Gresele, L.,, Kügelgen, J. V., Stimper, V., Schölkopf, B., and Besserve, M. Independent mechanism analysis, new concept? In Advances in Neural Information Processing Systems, 34, pp. 28233-28248 2021.
Bing, S., Ninad, U.. Wahl, J., and Runge, J. Identifying Linearly-Mise Mixed Causal Representations from Multi-Node Interventions. Proceedings of the Third Conference on Causal Learning and Reasoning, pp. 843-867, 2024.
Hyvärinen, A. and Morioka, H. Unsupervised feature ex- traction by time-contrastive learning and nonlinear ICA. In Advances in Neural Information Processing Systems, 29, pp. 3772-3780, 2016.
Brehmer, J., de Haan, P., Lippe. P., and Cohen, S. Weakly supervised causal representation learning. In Advances in Neural Information Processing Systems, 35, pp. 38319- 38331, 2022.
Hyvärinen, and Pajunen, P. Nonlinear independent com- ponent analysis: Existence and uniqueness results. Neural Networks, 12(3):429-433 439, 1999.
Johansson, D., Shalit, U., Kallus, N., and Sontag, D. Generalization bounds and representation learning for estimation of potential outcomes and causal effects. The Journal of Machine Learning Research, 23(1): 166:7538, 2022.
Buchholz, S., Rajendran, G., Rosenfeld, E., Aragam, B., Schölkopf, B., and Ravikumar, P. Learning linear causal representations from interventions under general nonlin- ear mixing. In Advances Neural Information Prucess- ing Systems, volume 36. pp. 45419-45462, 2023.
Khemakhem, I., Kingma, D., Monti, R., and Hyvärinen, A. Variational Autoencoders and Nonlinear ICA: A Unifying Framework. In Proceedings of The 23rd International Conference on Artificial Intelligence and Statistics, pp. 2207-2217, 2020.
Chen, T., Bello, K., Locatello, F., Aragam, B., and Raviku- mar, P. Identifying general mechanism shifts in linear causal representations. arXiv preprint 2rXiv:2410.24059, 2024.
Collett, E. Field guide to polarization. International society for optics and photonics, 2005.
Kingma, D. P. and Ba, J. L. Adam: A method for stochastic gradient descent. In International Conference on Learn- ing Representations, 2015.
Comon, P. Independent component analysis, A new con- cept? Signal Processing, 36(3):287-314, 1994.
Lachapelle, S. and Lacoste-Julien, S. Partial Disen-
Daunhawer, I., Bizeul, A., Palumbo. E., Marx, A., and Vogt, J. F. Identifiability results for multimodal contrastive learning. In International Conference on Learning Rep- resentations, 2023.
tanglement via Mechanism Sparsity. arXiv preprint arXi::2207.07732. 2022.
Lachapelle, S., Deleu, T., Mahajan, D., Mitliagkas, I., Ben- gio. Y., Lacoste-Julien, S., and Bertrand, Q. Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learning. In Proceed- ings of the 40th International Conference on Machine Learning, pp. 18171-18200 2023.
Dean, T. and Kanazawa, K. A model for reasoning about persistence and causation. Computational Intelligence, 5(2):142-150, 1989. doi: 10.11111/.1467-8640.1999 tb00324.x.
10

---

Sanity Checking Causal Representation Learning on a Simple Real-World System
Lachapelle, S., López, P. R., Sharma, Y., Everett, K., Priol, R. I.., Lacoste, A., and Lacoste-Julien, S. Nonparametric Partial Disentanglement via Mechanism Sparsity: Sparse Actions, Interventions and Sparse Temporal Dependen- cies. arXiv preprint arXiv:2401.04900 2024.
Locatello, F,, Poole, B., Rätsch, G., Schölkopf, B., Bachem, O., and Tschannen, M. Weakly- disentangle- ment without compromises. In Proceedings of the 37th International Conference on Machine Learning, pp. 6348 6359, 2020.
Lake, B. M., Ullman, T. D., Tenenbaum, J. B., and Gersh- man, S. J. Building machines that learn and think like people. Behavioral and Brain Sciences, 40:e253, 2017.
Lopez, R., Tagasovska, N., Ra, S., Cho, K., Pritchard, J., and Regev, A. Learning Causal Representations of Single Cells via Sparse Mechanism Shift Modeling. In Proceed- ings of the Second Conference on Causal Learning and Reasoning, pp. 662-691 2023.
doi: 10.
LeCun, Y., Bottou, I.., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 86(1 ():2772--222)) 22244, 1998.
Murphy, K. Dynamic Bayesian networks: representa- tion, inference and learning. University of California, Berkeley, 2002.
LeCun, Y., Bengio, Y., and Hinton, G. Deep learning, Na- lure, 521(7533):436-444 2015.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.,, Desmaison, A., Köpf, A., Yang. E.,, De Vito, Z.., Rai- son, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., and Chintala, S. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, 32, pp. 8026- 8037, 2019.
Liang, W., Kekić, A., von Kügelgen, J., Buchholz, S., Besserve, M., Gresele, L., and Schölkopf, B. Causal component analysis. In Advances in Neural Information Processing Systems, volume 36, 2023.
Lin, S., Hilton, J., and Evans, O. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pp. 3214- 3252, 2022.
Peters, J., Janzing, D., and Schölkopf, B. Elements of causal inference: foundations and learning algorithms. The MIT Press, 2017.
Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., and Gavves, CITRIS: Causal identifiability from tem- poral intervened sequences. In Proceedings of the 39th International Conference on Machine Learning. Proceed- ings of Machine Learning Research, pp. 13557-13603, 2022.
Rajpurkar, P., Zhang. J., Lopyrev, K., and Liang, Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natural Language Processing, pp- 2383- 2392, 2016
Runge, J., Bathiany, S., Bollt, E.,, Camps- Valls, G., Coumou, D., Deyle, E., Glymour, C., Kretschmer, M., Mahecha, M. D., Muñoz-Marí, J., et al. Inferring causation from time series in earth system sciences. Nature communica- tions, 10(1):1-13, 2019.
Lippe. P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., and Gavves, F. Causal representation learning for instantancous and temporal effects in interactive systems. In International Conference on Learning Representations, 2023a.
Saengkyongam, S., Rosenfeld, E., Ravikumar, P. K., Pfister, N., and Peters, J. Identifying representations for inter- vention extrapolation. In International Conference on Learning Representations, 2024.
Lippe, P., Magliacane, S., Löwe, S., Asano, Y. M., Cohen, T., and Gavves, E. BISCUIT: Causal Representation Learning from Binary Interactions. In Proceedings of the 39th Conference on Uncertainty in Artificial Intelligence, 2023b.
pp.
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward Causal Representation Learning. Proceedings of the IEEE. 109 (5):612-634, 2021.
Liu, Y., Alahi, A., Russell, C., Horn, M., Zietlow, D., Schölkopf, B., and Locatello, Causal triplet: An open challenge for intervention-centric causal representation learning. In Proceedings of the Second Conference on Causal Learning and Reasoning, pp. 553-573, 2023.
Spearman, C. The proof and measurement of association between two things. The American Journal of Psychology, 15(1):72-101, 1904.
Locatello, F., Bauer, S., Lucic, M., Rätsch G., Gelly, S., Schölkopf, B., and Bachem, O. Challenging Common As- sumptions in the Unsupervised Learning of Disentangled Representations. In Proceedings of the 36th International Conference on Machine Learning, pp. 41 4124 2019.
Squires, C., Seigal, A., Bhate, S. S., and Uhler, C. Linear Causal Disentanglement via Interventions. In Proceed- ings of the 40th International Conference on Machine Learning, pp. -3550032560 2023.

---

Sanity Checking Causal Representation Learning on a Simple Real-World System
Tibau, X.-A., Reimers, C., Gerhardus, A., Denzler, J., Eyring, V., and Runge, J. A spatiotemporal stochastic cli- mate model for benchmarking causal discovery methods for teleconnections. Environmental Data Science, 1:e12, 2022.
Information Processing Systems, 35, pp. 26492-26503, 2022.
Zbontar, J., Jing, L.,, Misra, L., LeCun, Y., and Deny, S. Barlow twins: Self-supervised learning via redundancy reduction. In Proceedings of the 38th International Con- ference on Machine Learning, pp. 12310-12320, 2021.
Tsamardinos, I., Brown, L. E.,, and Aliferis, C. F. The max- min hill-climbing Bayesian network structure learning algorithm. Machine learning, 65:31-78, 2006.
Zhang, J., Greenewald, K., Squires, C., Srivastava, A., Shan- mugam, K., and Uhler, C. Identifiability Guarantees for Causal Disentanglement from Soft Interventions. In
Varieı, B., Acartürk, E., Shanmugam, K., Kumar, A., and Tajer, A. Score-based Causal Representation Learning with Interventions. arXiv preprint 0000000000.082300 2023.
Advances in Neural Information Processing Systems, vol- ume 36, pp. 50254-50292, 2023.
von Kügelgen, J., Sharma, Y., Gresele, L.,, Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F. Self- Supervised Learning with Data Augmentations Provably Isolates Content from Style. In Advances in Neural Infor- mation Processing Systems, volume 34, pp. 16451-16467, 2021.
von Kügelgen, J., Besserve, M., Wendong, L., Gresele, L., Kekić, A., Bareinboim, E., Blei, D., and Schölkopf, B. Nonparametric identifiability of causal representations from unknown interventions. In Advances in Neural Information Processing Systems, volume 36, 2024.
Wright, S. Correlation and causation. Journal of agricul- tural research, 20(7):557, 1921.
Xu, D., Yao, D., Lachapelle, S., Taslakian, P., von Kügelgen, J., Locatello, F., and Magliacane, S. A sparsity principle for partially observable causal representation learning. In Proceedings of the 41st International Conference Machine Learning, pp. 55389-55433, 2024.
Yao, D., Muller, C.,, and Locatello, F. Marrying causal rep- resentation learning with dynamical systems for science. arXiv preprint arXiv:2405. 13888, 2024a.
Yao, D., Rancati, D., Cadei, R., Fumero, M., and Locatello, F. Unifying causal representation learning with the invari- ance principle. arXiv preprint arXiv:2409.02777 2024b.
Yao, D., Xu, D., Lachapelle, S., Magliacane, S., Taslakian, P. Martius, G., von Küiglggen, J., and Locatello, F. Multi- view causal representation learning with partial observ- ability. In International Conference on Learning Repre- sentations, 2024c.
Yao, W., Sun, Y., Ho, A., Sun, C., and Zhang, K. Learn- ing Temporally Causal Latent Processes from General Temporal Data. In International Conference on Learning Representations, 2021.
Yao, W., Chen, G., and Zhang, K. Temporally Disentan- gled Representation Learning. In Advances Neural
12

---

Sanity Checking Causal Representation Learning Simple Real-Worke System
A. Experimental Details
In this section we provide details on all data generation procedures, hyperparameters, training procedures, and minor adjustments made the original implementations required to reproduce the results of this work. Our general approach was use each method's existing code as out-of-the-box as possible, and almost all hyperparameters and settings are the same reported in the original papers. However, slight modifications were inevitable in making the code for our data, and we thank the authors of the methods for their guidance. All implementations use the PyTTrrh machine learning library (Paszke et al., 2019). The data used in this work is available through the 1t st_crl___ www.rrrwwncaaaeaeeeeeee v1 dataset at gi thub. com/ juangame 1awph/aameellaaaoao causal whtpa//aaaaaaaaaaaaaaeaeeeee chamber and the code, including all implementations of the considered www.aaggg/aSannntnnaaaaa
methods, is available at https:: /ttp:///gghthbbbu com/
Computational Resources and Compute Time. All experiments were run on a high high-performance cluster with NVIDIA
A100 GPUs. The total compute time required reproduce all experiments in this work is approximately 100 GPU hours.
02
R
0
G
B
Figure 7: The ground -ruth graph for the Contrastive CRL dataset.
A.1. Contrastive CRL
A. DATA GENERATION
We define linear SCM with additive Gaussian noise between the variables (R, G, B, (,,,,))) with causal relations encoded by the graph shown in Figure 7. The independent Gaussian noise variables all have zero mean and a variance sampled from (0.0011, 0.02]). To generate interventional data, we consider interventions that shift the mean of their target by adding a factor 17. We sample the value of the shift from (([1,))) 2]) for all interventions. Each variable intervened upon individually and we collect n = 10000 samples from each intervention, as well as the observational distribution. The full dataset is then split into train, validation, and test subsets according to the ratios (80/10/10) while ensuring that each subset contains the same fraction of samples from each environment.
1.1.2. TRAINING DETAILS
For our experiments with the CCRL method, we rely the code kindly provided by the authors of the original work. As the original implementation only included shallow convolutional encoder for images, we found it useful to extend the image encoder to a slightly deeper architecture, which we report in Table 1. Otherwise, we use the exact same architecture and implementation as the original paper. We report the hyperparameters used during training Table
Our initial experiments revealed it was crucial that the noise terms of the underlying SCM have mean zero. Attempts with non-zero mean led to the failure of the method.
A. METRICS
Mean Correlation Coefficient. The Mean Correlation Coeefficent (MCC, Hyvärinen & Morioka, 2016; Khemakhem
et al., 2020) measures the pairwise correlation between the learned representation and the ground truth latents. If C E R the Pearson correlation matrix between the ground truth variables and the learned embeddings and G, denotes the set of depermutations, the MCC score is formally defined as
MCC = max S d
Σ
C G-
13

---

Sanity Checking Causal Representation Learning Simple Real-Worke System
Structural Hamming Distance. The Structural Hamming Distance (SHD, Tsamardinos et al., 2006) measures how well method recovers the ground-truth causal graph by comparing the adjacency matrices of the ground-truth graph A E {0, 1jadd and the learned graph A E {0, 1jdxd and counting the number of edges that are different. Incorrectly oriented edges count as two errors. We follow the convention that for an adjacency matrix A, a nonzero entry Ai, # O implies an edge + j. Since the Contrastive CRL method outputs a continuous estimate the adjacency matrix, it must the thresholded to obtain binary entries. We follow the heuristic in the original paper and choose the thresholding that results in same number of edges as the ground truth A. Naturally, this not possible when the ground-truth graph not known. The SHD then defined as
SHD AA A.j.--
Table 1: Convolutional encoder architecture for Contrastive CRL..
Layer
Activation function
Conv (c_in c_out 64, kernel 3, stride 2)
GroupNorm-SiLU
Conv (c_in 64, c_out 64, kernel 3, stride 1)
GroupNorm+Shl
Conv in 64, out 64, kernel = 3, stride 2)
GroupNorm-SSi
Conv (c_in 64, c_out 64, kernel 3, stride 1)
GroupNorm+SiLU
Reshape (1 X 1 X (16 16 64))
Linear ((16 16 64) x 1024)
SayerNorm+SiLU
Linear (1024 x 5)
Table 2: Contrastive CRL hyperparameters.
Parameter
Value
Model latent dimension
Learning rate
5e-4
Optimizer
Adam (Kingma & Ba, 2015)
Epochs
100
Batch size
512
Learning rate scheduler
Plateau Scheduler
K
0.1
T1 (Buchholz al., 2023, Eq. 9)
le-5
(Buchholz al., 2023, Eq. 9)
le44
14

---

Sanity Checking Causal Representation Learning Simple Real-Worl System
View 2
Č
Im
View
View
(R, G, B)
01
22
0
11
01
View
V1
i2
V2
Figure 8: Ground-truth graph relating the latent causal factors, shown in bold print, to the different views employed in the Multiview CRL experiment. The views are disjoint sets of the output variables produced by the tunnel. The graph is a subgraph of the complete causal ground-truth graph for the light tunnel, described in Gamella et al. (2025, Figure 3b).
A.2. Multiview CRL
1. DATA GENERATION
The underlying causal model used in our experiments for the Multiview CRL method is identical the one used for the Contrastive CRL method described in Appendix A.1.1. In addition to first view given by the images (Im)-which depend on (R, G, B, 11 02)-we we consider three additional views. The second view consists of the electrical current drawn by the light source and the infrared and visible light intensities from sensors one and two, i.e., (C, J1, V1. Iz, Vz). These depend on the underlying causal factors (R, G, B). The third and fourth views consist of the angle measurements of the first (01) polarizer and second polarizers, respectively. In turn, they depend on the corresponding polarizer angles (01) and (02). graphical overview of the data-generating process is shown in Figure 8. These views consist of either real images and sensor measurements collected from the tunnel, their synthetic and deterministic counterparts produced by the simulator given in Appendix C. To increase the overall sample size, we include observational samples, well as all interventional samples described in Appendix A.1.1, arriving a dataset with 60000 samples. The data is randomly split into train, validation, and test sets according to 80/10/10 ratio.
A.2.2. TRAINING DETAILS
We use the publicly available https:: //github/ Com/Causaal (aaasaaaaarrnnngalll htt.//muttiiiewwememmemmm the Multiview CRL method from For our experiments, we use the variant of this implementation that relies on BarlowTwins (Zbontar et al., 2021), in accordance with the experiments that use image data in the original paper. The hyperparameter settings used in our experiments are reported in Table 3. The number of training steps is chosen to obtain approximately the same number of epochs given our chosen batch size as the original paper uses for their image data experiments.
implementation
Table 3: Multiview CRL hyperparameters.
Parameter
Value
Model latent dimension
5
Learning rate
le-4
Optimizer
Adam (Kingma & Ba, 2015)
Nr. of training steps
28550
Batch size
512
Similarity metric
Cosine similarity
A.2.3. METRICS
The theory of the Multiview CRL method states that the model achieves block-identffiabiity (Yao et al., 2024c, Definition 2.3), meaning that the information of the ground-truth variables belonging to the content block of a set of views perfectly
15

---

Sanity Checking Causal Representation Learning on a Simple Real- System
recovered by the model, while the information of the style variables belonging to this set of views is ignored. The authors measure recovery by looking at the R2 score of estimator trained to predict the values of the individual ground-truth latents from specific subsets of the learned representation. For each view in a given set, the view-specific encoder returns an encoded latent vector 2 E R2. Let [d] (1....,d d) denote the set of integers from one to d. Then, a subset of the view-specific encoding vector 25, 5 S [d] is selected as input to the estimator that predicts the learned latent variables. This subset differs between distinct sets of views and corresponds to what the model learns as the content-blec of that respective set of views. For each latent variable and view in given set, we record the R2 score (Wright, 1921) of the estimator fit according to the aforementioned procedure, averaging the scores over the encodings obtained by different view-specific encoders in given set. According to the theory, if the model achieves block identifiability, this metrie will be close to one for the variables belonging to the content block in set of views, while being significantly lower for variables that belong to the style partition.
A.3. CITRIS
A.3.1. DATA GENERATION
We generate the data for the CITRIS experiments by defining a time-evolving process given by the following time-series graph and assignments:
R+ t ff {( +ER
GO
B
65
0
R² -G
G +
G +
EG
1
1
2
G B
G++1
B++1
0ct1
B++1 t ff
B² B²+
+ +EB
4
0ct1 t 9 (0) +e)
Rd - R++1
(09
( 00
18-00
02
8 t 9 + SSR', ((RR,B'))
4 c2
where the random innovations are sampled as é ta, ~ Unif[ 50).e1 el ied Uni 10). and el iid Unif[- 5)... Furthermore, the functions S. f and g are defined as
Lid.
{1
Bf > Rd -1 B' < R f(x)
-I
if x < 0
-180 x if < -90
S(R', B¹)
and g(x)
510 - x
if 2 255
180 x if 2 90
The purpose of f and g is to ensure that the values of ....... 02 remain within the bounds of the control inputs for the light tunnel ((0, 255] for R, G, B and [-90, 90] for 02).
Intuitively, the process can be understood as follows: the variables R and 0, follow random walks. G + 1 tries to approach R' in each step, and B' variables from converging to their targets. Finally, 02 tries to exploit the color-shifting effect of the polarizers to maximize the ratio of the blue to red pixels in the resulting images. It does so by approaching 0 aligning the polarizer axes-ii B' > R', and receding if R' 2 R'. The behavior is regulated by the function S(.) above. At each time step, we perform no intervention with probability 0.3, or we select an intervention target at random from {R', G B", H, 88), setting it to a value sampled uniformly at random from its valid range. We encode the intervention target at each time step in one-hot vector, with a vector of zeroes indicating that no variable is intervened upon. Python implementation of the process is provided in github. com/ juangame aaacaaaaaaaalaaaaaaaaaaaaa watissallchabbbrreeeeee /blbe//r//bb/b///bbee ma in datasatttsss/s///s/ _|t_crr___ cwt/tiiittsss/ .py.
approaches G in the same fashion. However, the innovations cn, CG. and cp prevent these
We collect a sequence of n = 100000 samples from this process and use the first Thrain = 80000 samples as the training dataset and the following nva = 10000 and nhest = 10000 as validation and test datasets. Following the experiments in the original paper, to evaluate the final metrics we collect an additional dataset with n = 1000 samples, where the latent variables are sampled uniformly at random from their valid range.
16

---

Sanity Checking Causal Representation Learning Simple Real-Worke System
A.3.2. TRAINING DETAILS
For the CITRIS experiments we rely on the code that is publicly available sttps://gitthis
com/phLippe/CIRREE. The hyperparameter settings used in experiments shown in Table 4. As mention in the main text, opted not use the method's triplct loss for model selection, this relies on accessing ground-truth latent values during training, which we find highly unrealistic. We replaced this validation loss with the same loss function used during training. Calculating the correlation metrics at intermediate steps during training does not suggest that this leads to a suboptimal model selection in terms of the final metric.
During initial experiments, we found that a number of different choices did not result in any improvement of the final results. We tried using the auto-regressive prior that does not rely on normalizing flows (as opposed the one that does), setting the latent dimension of the model to match that of the ground truth, fixing the assignment of the model latents the causal variables instead of learning it, as well as training for 500 instead of 250 epochs. None of these choices resulted in an improved final score.
Table 4: CITRIS hyperparameters.
Parameter
Value
Model latent dimension
16
Learning rate
le-4
Optimizer
Adam (Kingma & Ba, 2015)
Epochs
250
Batch size
512
Learning rate scheduler
Cosine Warmup (25 steps)
A.3.3. METRICS
Similar to the MCC score, the authors of CITRIS report the correlation between the learned representation and the ground- truth variables. However, since CITRIS allows for multiple learned latents describe single ground-truth variable, their model first learns mapping that assigns latent dimensions a ground-truth factor. To compute the final metries, MLP is trained to predict each ground-truth trrth variable from the learned groupings of model latents. Each ground-truth factor is predicted from each respective grouping, resulting in matrix. The authors then report the R² (Wright, 1921) as well as Spearman's 's rank correlation coefficient (Spearman, 1904) for this matrix. As summary metric, the diagonal of the resulting matrices is examined, considering its average value, which should be close to one in the optimal case, as well as the maximal off-diagonal value, which is expected to be close to zero.
A.3.4. ADDITIONAL FIGURES
We present additional results of the CITRIS experiments (cf. Section 3.3) using Spearman's rank correlation coefficient in Figure 9. We see that this metric echoes the findings presented in Figure 6.
17

---

Sanity Checking Causal Representation Learning Simple Real-World System
Real data
Synth. ablation
1.0
R-
0.05
-0.00
0.00
0.03
Leamndd latents
C
0:39
0.33
0.30
0.00
0.07
0.36
0.18
0.25
0.04
0.11
Spearman diag f
Spearman sep +
B-
0.05
0.30
0.00
0.00
0.03
0.5
Real
0.230 + 0.058
0.801 + 0.052
0] 0.33
0.47
0.52
0.30
0.51
0.60
0.47
0.54
0.2
0.41
Synth. ablation
0.245 = 0.055
0.809 = 0.072
02
0.53
0.63
0.38
-0.2
NA
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
0.00
R
G
1
0:
00
R
G 81
02
avg.
0.0
Ground-truth causal factors
Spearman
Figure 9: Results for the CITRIS method on real and and the synthetic ablation as described in Section 2. Left: Spearman's s rank correlation coefficient over five random initializations. We report the average and (++) the standard deviation. The diag columns refer to the average value of the diagonal of the Spearman correlation matrix while the sep columns denote the maximum off-diagonal value. Right: Average Spearman's 's rank correlation matrices for real and synthetic ablation datasets. NA refers to the group of learned latents that are not assigned to any ground-trute truth causal factor during training (Lippe al., 2022, Section 3.4). Analogously the R2, the results here show that the method fails fails recover the ground-truth factors both for real and synthetic data, indicated by the low scores along the diagonal and (relatively) high off-diagonal scores, respectively.
B. Supervised Sanity Check
To ensure that the information of the latent variables (R, G, 01, 02) can indeed be recovered from the image data, we train small MLP (architecture ef. Table 6) in a supervised fashion to predict the ground-trus truth causal factors from images held-out test set. We train on subset of 5000 samples and test on 500 samples. Our results (cf. Table 5) indicate that the latent variables do indeed leave a detectable trace in the data. We stress that this is merely a sanity check to exclude that our data collected in contrived way. The near-perfie serr-perfect performance of this supervised model should not be compared with the unsupervised methods we investigate in the main paper.
Table Test score of supervised model trained to predict latents from images for the respective datasets.
CCRL & Multiview CRL Dataset
CITRIS Dataset
R22
0.976
0.914
Table 6: Supervised MLP network architecture.
Layer
Activation function
Linear ((3 64) x 64)
Leak)
Linear (64 x 256)
LeakyReLU(O.0))
Linear (256 256)
LeakyReLU(0.00)
Linear (256 64)
Lea.0Re
Linear (64 x 5)
18

---

Sanity Checking Causal Representation Learning Simple Real- Real-World System
C. Deterministic Simulator of the Light Tunnel
this section, we describe the deterministic simulator used for the synthetic ablations of the light-tunnel data in Section The simulator consists of components: mechanistic model of the sensors (Appendix C.I), derived from first principles and the technical datasheets of the sensors, and neural simulator produce synthetic images (Appendix C.2). An implementation of both components, using Numpy and Pytorch, is provided by the causal chamber package; simulators 1t. Dete i iisiiiic i and It. www.derrimgnieeeeee in the simulator index for the source code and Jupyter notebooks with examples on how to run them.
For completeness, we provide a diagram the light tunnel in Figure 10, including additional variables used by the simulator that are not shown Figure 1. A detailed description of all involved variables given in Table 7, and the simulator parameters are described Table
V. TY DI. Light sensor
0, R Angle sensor
0g. R₂ Angle sensor
Light source
Linear polarizer
Linear polarizer
B
Light sensor
Light sensor
Red, green and blue levels
Motor
Motor setting 02
R, G, B
setting
Ig V22
Ig. V TV T. DY
Drawn electric current C, R
0
T T
D2. DY
Figure 10: Diagram of the light tunnel with the variables involved in the deterministic simulator, where bold symbols correspond to the inputs to the simulator, and its outputs are denoted by tilde. The inputs include variables not shown in Figure 1, which control the behavior the current sensor (Rc), the light-intensity sensors (D', DV ,T TV, ), and the angle sensors (Ri, R2). All variables are described detail Table 7 below, and a complete description of the light tunnel can be found in Gamella et al. (2025).
Table 7: Description of the light tunnel variables involved in the deterministic simulator, including their symbol, value range, and the corresponding column name in the dataset files.
Variable
Type
Data column
Description
i, i, (0....,210.
Output
ir_1
The uncalibrated measurement of the light- intensity sensors in the infrared part of the spec- trum, placed before (f1), between (1z), and after (Ig) the polarizers, in reference to the light source.
ir_2 2 ir_3
Vi, V2, V/ {0, ... 216 1}
Output
vis 1
The uncalibrated measurement of the light- intensity sensors in the visible part of the spectrum, placed before (V1), between (V2), and after (V²) the polarizers, about the light source.
vis 2 vis_3
19

---

Sanity Checking Causal Representation Learning Simple Real-World System
Table 7: Description of the light tunnel variables involved in the deterministic simulator, including their symbol, value range. and the corresponding column name in the dataset files.
Variable
Type
Data column
Description
Č [0, 1023]
Output
current
The uncalibrated measurement of the electric cur- rent drawn by the light source. The calibrated measurement (in amperes) is given by
Rc
Ö
x 2.5.
1023 X 5
81. 02 E [0, 1023]
Output
angle_1 angle_2
The position of the polarizers is encoded into voltage using a rotary potentiometer, which is then read by the control computer to produce the mea- surements 0 and 02. Given these measurements, the calibrated angle measurement is given as
720
vref( degrees,
2g)
1023
where R3 is the reference voltage of the corre- sponding sensor, and Zi 507, Zz = 512 are the readings at angles 0N = 02 and reference voltages R Ra = 5.
R. R.G.Be ..... }
Input
red
The brightmess setting of the red, green, and blue LEDs on the tunnel's light source. Higher values correspond to higher brightness.
green
blue
01,0c {-180, 179.99
180}
Input
pol_1
Position of the tunnel's polarizers, in degrees.
pol_2 diode ir_1 diode_ir, diode ir_3
D4. e 10,1,2)
Input
The size of the photodiode used by the light sensors to take infrared measurements, corresponding the small (Df 0), medium (DJ 1) and large (Dj =) photodiodes onboard.
DY,DY Dý E 10
Input
[lode__is__1 di _id___vis_2 diode vis_3
The size of the photodiode used by the light sensors to take visible-light measurements, corresponding to the small (D) = 0), medium (D) = 1) and large (D) 2) photodiodes onboard.
TH, TV. Ty Ty E (0,1,,,3)
Input
t_ir_1/2/3 t t_v13_1/2/3
The exposure time of the photodiode during light- intensity measurement. Higher values correspond to longer exposure times.
Rc, R1, Rg E (1.1, 2.56,5
v_c Input
The reference voltage, in volts, of the sensors used to measure the current (C) and polarizer angles (0,,02), respectively.
v v_angle_1
v v_angle_2
C.1. Simulating the Sensor Measurements
We derive a mechanistic model from first principles and the details provided in the technical datasheets of the involved compo- nents. The datasheets are provided at git github. com/ juangame 1 causal-chamber / /////m/n/////////mmamm//aa/ hardwarr/ datasheets. The expressions for the model, with color-coded variables to differentiate between the inputs, outputs, and parameters of the simulator, are given below. A comparison of the resulting synthetic measurements and the true
20

---

Sanity Checking Causal Representation Learning Simple Real-World System
measurements collected from the light tunnel provided in Figure 11.
Synth. measurement
71
V1
12
R2 = 0.93
R² 0.94
R2 = 0.92
Synth. measurement
V2
V/3
R² 0.89
R² 0.93
R2 0.86
Synth. mcasurement
Ö
0,
02
R² 0.91
R² 1.00
R² 1.00
Real measurement
Real measurement
measurement
Figure 11: Comparison between the real measurements (x-axis) and the corresponding synthetic measurements (y-axis) produced by the deterministic simulator for the same inputs. Each dot, corresponding to a real/synthetic pair, is colored according to the light-source setting (R, G, B). The diagonal black line corresponds to the deterministic output of the simulator, and we report the R² score between the real and synthetic measurements.
Light-Intensity Sensors. We model the sensor response linear combination of the intensity of each light-source color (R, G, B). with coefficients encoded in the matrix S. To account for the decrease in intensity as we move away from the light source, we apply the inverse-square square law using the distances d, d2, dg each sensor to the light source. The effect of the polarizer angles (01, 02) on the third sensor (Ig, V/) modeled through Malus' law (Collett, 2005). Following the specifications of the se we model the effect of the photodiode size (D(, DV and exposure time (T), TV) as scaling the sensor response by an exponential factor.
See pages 42/43 the 1i1 datasheet.
21

---

Sanity Checking Causal Representation Learning on Simple Real-Worls System
R
i
2
2f
Vi
=
0
2 +DF
S
G
B
R
12
2
P!+T
d1
s
T
G
1/
0
2 +I
B
2
d1
s
diag ((Tp - TT) cos - -02) + IT)
G
Vi
0
2D +
Current Sensor. The drawn current Ö is modeled as a linear combination of the brightness setting (R, G, B) of each light-souree color, encoded the matrix of coefficients Q. We include an intercept Co for the base current drawn even when the light source is tumedd off (R G B 0) and account for the scaling induced by the reference voltage Re-
o==
Q
G
Co
Rc
Angle Sensor. The relationship between the polarizer positions (01, and the voltage measured by the angle sensors (0),0)) is also affine, and is encoded in the coefficient and the reference points d1, 0g. As for the current sensor, we account for the scaling induced by the reference voltages R1, Rg. To model the saturation of the sensors at lower reference voltages (see Gamella et al., 2025, Supplementary Figure 9), we take the minimum between the synthetic measurement and the maximum possible value (1023).
5 R f 0g =
81 == min 1023, (A01 + a1)
:-mm min 1023, (A0g + a2) Ro
The parameters of the simulator, displayed in orange in the above equations, are described in Table below. For the synthetic observations used in the experiments of Section 3, the parameters are either taken from the datasheets the corresponding sensors or fit separate calibration dataset. See the tutorial notebook for the actual values and example of running the Python implementation of the simulator.
Table Description of the parameters of the simulator for the sensor measurements, including their range and the name they take in the Python implementation (see simulator 1 It.Determiniee Dei.smmmisstt.c...
Parameter
Name in code
Description
S E 22x3
S
Encodes the linear response of the photodiodes to the light of different colors. The first
row 11, gives the response of the smallest IR photodiode (Di = 0) to the red, green, and blue light; the second row corresponds to the smallest visible photodiode (DY = 0).
d1, d1,d2,d5 d3 E
d1, d2, d3
The distance, in millimeters, from the tunnel's light source to the first (d1), second (d2) and third (dg) light sensors.
22

---

Sanity Checking Causal Representation Learning Simple Real-World System
Table 8: Description of the parameters of the simulator for the sensor measurements, including their range and the name they take in the Python implementation (see simulator 1 .t..eeermiiitiic)
Parameter
Name in code
Description
T. [],
Ts
The transmission rate of the first polarizer for each color wavelength.
Tp, To E [0, 1
Tp, Te
The joint transmission rates of the two polarizers for each color wavelength when their polarization axes are parallel (Tp) or orthogonal (Te) each other.
Q E +
12x
Q
Encodes the linear response of the current sensor to the brightness setting of each color.
Co E IR+
c0
The current drawn by the light source when it is turned off, i.e., R G B 0. The value corresponds to the uncalibrated measurement when Rc 5.
A R
A
The linear coefficient relating the change in polarizer position the change in the voltage reading of the angle sensor, when the reference voltage is R2 for
2, respectively.
01,02 E R
al, a2
The zero point of the angle sensor, i.e., the voltage reading when the polarizer positions
are 0 and 02 = 0, respectively, and the reference voltages are R₁ R2 =
C.2. Simulating the Images
To simulate the images produced by the light tunnel, we train a simple neural network (Table to generate synthetic images given the inputs R, G, B, 0, and 02. comparison of the real and synthetic images provided Figure 12. A Python imple- mentation with pre-trained model is available through the causal chamber package (simulator 1t DecoderSimple)) see the tutorial notebook for example code.
A
real
B
real
C
real
D
real
E
real
F
real
R:166 G:9 B:16 81:-63.0 82:50.6
R:213 G:42
R:150
B:8
R:209 G:192 B:217
R:213 B:217
61:-88.7 82-72.11
61:44.5 82:56.7
61:-6.33 02:61.8
61:12.6 02-59.5
61:64.4 02-4.8
synthetic
synthetic
synthetic
synthetic
synthetic
synthetic
Figure 12: Comparison between the real and synthetic images produced by the deterministic simulator. The simulator does capture the noise present in the images, and smoothens out some of the finer details produced by the reflection of light on the polarizer frames, which form the ring around the center of the image.
23

---

Sanity Checking Causal Representation Learning Simple Real-Worke System
Table 9: Architecture and training parameters of the neural image simulator. The network consists of a stack of 4 fully connected linear layers with ReLu activations. The training loss is the mean-squared error over the pixel intensities. The network is trained on 108900 images from the It__amerr__" v1 and it_camera__e _awa_wlllal ks_vl datasets, which can be found github com/ aaaangamllaacausal-ooeeeee found in this notebook.
The code reproduce the training procedure can be
Layers
Parameter
Value
Linear (5)
Optimizier
Adam (Kingma & Ba, 2015)
Linear (64 x 64)
Learning rate
10-3
Linear (64 x 64)
Weight decay
10-5
Linear (64 x 64 3)
Epochs
100
Batch size
4096
D. Hyperparameter Search
Here we present tables of the results of hyperparameter search experiments for CCRL, multiview CRL and CITRIS. For all models we see that performance is not particularly sensitive to the choice of hyperparameters.
24

---

Sanity Checking Causal Representation Learning Simple Real-Worke System
Table 10: Average R
random seeds) for different hyperparameter settings of the Contrastive CRL method.
Ir
le -
5e 4
R
T2
0.01
0.1
1.0
0.01
0.1
1.0
le
0.303 0.070
0.294 0.021
0.333 + 0.056
0.240
0.038
0.182
0.092
0.330 + 0.052
1e-5
5
0.324 0.063
0.285 + 0.067
0.312 + 0.081
0.278
0.143
0.296 0.032
0.322 + 0.034
le 4
0.327 = 0.040
0,246 = 0.063
0.360 = 0.042
0.262 + 0.066
0.194 + 0.088
0.357 = 0.028
le 6
0.223 0.044
0.240 0.1 16
0.354 + 0.046
0.336 + 0.052
0.249 + 0.034
0.330 + 0.016
le - 4
le
0.307 0.087
0.261
0.033
0.341 + 0.050
0.278
0.046
0.287
0.019
0.327 = 0.064
le 4
0.302 = 0.071
0,267 = 0.033
0.269 = 0.037
0.262 = 0.075
0,230 + 0.070
0.337 = 0.038
1e 6
0.328 = 0.091
0.275 + 0.025
0.311 + 0.088
0.310 = 0.040
0.295 + 0.051
0.268 = 0.056
le- 3
le 0.261 0.045
0.221 0.094
0.345 = 0.038
0.337 0.071
0.313 0.116
0.283 = 0.059
0.322 0.046
0.252 0.016
0.290 + 0.065
0.243 0.084
0.247 0.027
0.305 + 0.095
Table 11: Average SHD (3 random seeds) for different hyperparameter settings of the Contrastive CRL method.
Ir
1c 4
5c 4
K
0.01
0.1
1.0
0.01
0.1
1.0
le
7.333
1.155
6.667 1.155
8.000 + 2.000
7.333 1.155
6.667 1.155
8.667 + 2.309
le
7.333
1.155
6.667
1.155
8.667 + 2.309
7.333 3.055
7.333
1.155
8.000 : 2.000
1c 4
7.333 + 1.155
6.000 + 2.000
7.333 + 3.055
6.667 + 1.155
6.667 + 1.155
8.000 + 2.000
1c 6
7.333 + 1.155
6.667 + 1.155
8.667 + 2.300
7.333 +: 2.309
6.000 +:: 2.000
8.000 + 3.464
le - 4
1c
7.333 3.055
6.667 1.155
8.667 + 2.309
8.000 2.000
6.667 1.155
8.667 + 2.309
le 4
7.333 1.155
7.333 + 1.155
8.000 + 3.464
6.000 2.000
6.667 + 1.155
8.667 + 2.309
1c 6
7.333 + 1.155
8.000 + 2.000
7.333 + 1.155
7.333 + 1.155
8.000 + 3.464
6.667 + 1.155
le 3
1e-5
6.667
1.155
7.333 3.055
8.667 + 1.155
6.667 1.155
7.333
1.155
8.667 + 2.309
16
6.667 1.155
8.000 0.000
7.333 + 1.155
7.333 1.155
8.667 1.155
7.333 + 1.155
25

---

Sanity Checking Causal Coweesenttttnn
Searning on a A
Simple Real- World System
200000
210112
100000
0.778 1e10
1007S
5.38 210000
20000
10000
0.37 1
20012
1
$0.128
1ILLNO 0001 2007 m
0,200 +0114
$40025
0.60 1
+
20.182
$004
0.40 10,000
9.40 1.0000
0,30 +
$40000
1.0000
comtent
02
3.74
3,500
0.855 0.0000 100000 0.004 &000 0,027 0.0LGG 0.005 0.364 = 0.177 0.002 0.07 0000 = 2.11 0.000 #0.05 6.48 0.003 0.500
0.300
0.304
GO0 0.42
0-71
0.20
0.00
0.58
0.01
0.218
3.2
0.30
0.41
3.40
9.40
2002
2001
2005
0.187 0.00 0 0.200 1000 6.08 2000
0.002 LOOT
DO0S 2005
G011 6011- 00me
0.045 2000
0.00 20.00
6,08 2.0
0,118 6.020 = 0.000
0.000 2002
6,00 0
6.0T 2000
0,200 n,12 6.08 0,025
0.200 0.188 6.000 = 0.01
201772 0.000 0.011
6,02 =0.15
6.088 0,000
6.09 0.000
view (Im) vs. view 4 (6x)
0.06
0.30
0.000
6.28
0.000 S0M0S
0201 =00100010
030 100057
10:174
0,080 2000
0.220 10.12
0.30 210170
+
0.11110.122
10022
20,300
0,200 112214
0.000
0403 0,18
0.20 001 0:398 2 9.229 0 0.308 1.3315
0.11 0.500
B
6.m
0
0
0,201
0,300
0.308
0402
0.48
0
style
WOSE 01000 0.000
0.9 0:200 D
0.03
D.10
1.
0.000
0.10r 0.106
0.18
0.005
0.2g
0.088
0.10
0.28
0.417 0.004
0.18
0
0.299 & c.om 5 0.150 0,187 0 00 car car==011 2014 0.887 2014
G
0.00
0.138
0.02
0:005
0.129
0,000
0,187
0.134
0.022
=
6,578
0:224
0:225
0.288
0.206 + i. n,002 0.161 0222 0.1188 0.10 -.11-00.05 0.002 8.181
Table 12: Average R2 (3 random seeds) for the Multiview CRL hyperarammeerrr search.
000
2014 0,023 000 coo 0,000
0,118 200 0006 0
n.312 o... 0.008 100. o.ca 0.000
Lo 0o0 0018
nanss 0.006 c.n -.020
0.00 0.00 D.00 0.057
CANY 2015
0.000 a 0.000 0.008 0.063
0,06 0.,00
0.076
0.02 0.000
n.347
0.055
0.107 a.00
car2 0.18
0.376 2 0,204 a 9.100 0210 0 c. 0.000
0.00
0:200
0.86
0.74
0.130 O O
LOOT
0.000
0.00
0.122
0.120 1.120
0.085
0,20 0.353
0.08
0.19
0.376 o 136 0.383 a 1 0,306 = 152 20
02
style
0,000 OF 20.10 0,012
0.100
0,001
0.00
0.178
0,100
0.147
m1275
0,157
0,213
0:009 0.122
1011
10:101 0201 10.17
0,003 = 9.139
0,300 0.007
n,00 0.122
n.200 -220000.10 0.11
0.121
0.125
0,428 a: 0.046
0.088
00116
0.t00
nato 0.148
3,000
0,300 0.158
6,15 0.185
consent
118=00.0
0,402
view 1 (Im) vs. view 3 (0.)
0,893
0.348
0.305
0.000 0,208
0,000
6.416
6,00
0,10
1:0000
0.035 COT
1.0000
0.00
0.075
1.000
1:177
nARI +.000
1.005
0.0.0
0.00m
0.092
1:0000
0.088
0.227 & 0.228
0,210 0.002
0.281 a 0.000
0.180 &
0.186
0.127
0,210
0.044
0.100
0.170
0.121
0:305
0.006
0.150
0.005
0,237
0.128
0.351
0.022 OOO
0.10
0.000
0.00
nMm +00,44
0.00
0.00
0.0088
thnan 0.000
1.18 +-0 14
2.0.1111 1iG
1 340
0.300
2 24D
0.108 0.00
0:2
0
n2242 1.1111
0.343 0.000
9.00 0:334 2
G
style
G710 0.00 1000 022 1001 0000 00 220 1et0e 0.101
0:03 1
0,103
0.015
0.119
0,080
0.121
0.007
0.001
0.076
0.200
1.7777 0.397
100000
0.000 20000
0.991 400002
0.1 1.0.55
10.005
0,152 +
11118 10.135
a 0.163 2.005
0.074 0.070
10 n.Ors
3.168 $
+0001
6,140
0,000
2..115
9.17 6.188
0.168 0.011
1.000
R
0.14
0.201
9,226
0.14
9.191
1.777
0.023 CO
COG
0.70 OO
000 10
0.515 1000 0.866 10.000
0.632 100011
+.0.
100000
0,838 2.000
0,823 & 0.000
+.0000
0.00
0004
0,000 0.0LL
0.000
0,036 coclo
0,000 c.0lin
0,001
0.2 1AM0 0.03
0,02
0,86
0,038
0,50
0,011
0,006
0.017
view 1 (Imi) vs. view 2 (C, Ti, R. V3)
style
0.301 0.000
O 2001
20013
0,30 20017
0310 10000
0.314 2.000
0.005
200117
#-0122
0.005
1.0000
+0.15
8.0007
1.,000
0.000 0.035
0.091 2.000
0.108 0.0024
20.014
101 2001
I.
01
0,200
060
0.34
0.811
220
0.065
0.048
2.2
0.007
D.851 0.0002
DI 0,000
0000
1000
0.822 0.002 0710 0.00 0.801 1,000
57770 10001
0.800 n.00s
0.879 0.005
0.001
0,002
0.002
0,000
0,000
1.015
0,002
0,000
0.811 0.012
0.748 0,000
0.720 0,001
0.521 0.011
0.005
ra 1.000
15
0.000
064 = 0.534
0,80
01800
0.000
0.85T
0,500
0.806
0.80
0.70
0.708
0.78G
0.000
00.00
0O09 10.001
0.000
0.000
0,000
6,706 0.002
BANT
0.007
0,698 0.005
6,700 0.005
0.003
0.00
0.001
0.009
content
0.00
0,700
0.000
0.001 0,700
0,000 0,700
momn 0.000
0,718
1c 05-5 5 0.500 0002 0.006
0,704
0,000
0,701
0,000
6,704
0,700
0.004
0.000
0000
0.000 DAMI
0,700 0.000
0,511 0.002
0080 10000
0.001
$00000
0.402 2.17
0.4 0.003
le 5 0.728 0.022
o.sia 2.015
0.477 2..014
le- (0-5 5 0.000 0.125
0.400 0,012
00001000001
0.011
0.782
0
0
0.70
0.764
0.780
0.740
16 5 0.778
0.312
0.71
0.000
5
5
16-5 5
16- 44-4
5e 4
1e 5
5
4
4
1e -4 4
5e 4
le 4
5e 4
le -4
5e 44
16
5c
1e-
le
5c
le-
Se
16
le-
de
le-
Se
batch size
256
512
1024
256
512
1024
256
512
1024
0.1
1.0
10.0
26

---

Sanity Checking Causal Representation Learning Simple Real-Worls System
Table 13: CITRIS hyperparameter search (3 random seeds). Empty cells indicate failed runs due to training diverging.
lat. dim.
batch size
Ir
R² diag f
R2 sep +
Spearman diag 1
Spearman sep +
le 4
0.106 + 0.028
0.241 + 0.020
0.222 + 0.025
0.406 + 0.032
128
3
0.072 0.076
0.308 + 0.095
0.225 + 0.110
0.464 0.092
le - 2
le - 4
0.193 + 0.024
0.207 + 0.064
0.360 + 0.024
0.398 + 0.085
16
512
le 3
0.117 0.058
0.276 + 0.043
0.271 + 0.063
0.440 + 0.042
le - 2
le - 4
0.157 = 0.038
0.236 + 0.025
0.370 + 0.072
0.464 + 0.043
2048
le 3
0.142 0.076
0.236 + 0.028
0.301 + 0.082
0.410 + 0.056
1e
0.112 0.059
0.251 + 0.026
0.260 + 0.086
0.407 + 0.018
le
0.051 = 0.073
0.204 + 0.042
0.255 + 0.080
0.423 = 0.038
128
le - 3
0.084 + 0.110
0.325 + 0.081
0.249 + 0.087
0.496 + 0.089
1e 2
1e 4
0.065 0.054
0.179 + 0.009
0.222 + 0.047
0.385 + 0.013
32
512
le 3
0.002 0.023
0.307 + 0.006
0.189 + 0.019
0.482 + 0.012
le - 2
le 4
0.101 + 0.047
0.221 + 0.059
0.306 + 0.038
0.438 + 0.057
2048
le 3
0.055 + 0.021
0.242 + 0.036
0.228 + 0.022
0.441 + 0.069
1c - 2
0.016 0.007
0.215 + 0.005
0.168 + 0.006
0.404 + 0.026
27


Reviews:
[{"review_id": "FK2fELzOL0", "reviewer": "Reviewer_DQvb", "summary": "The authors evaluate existing methods for causal representation learning (CRL) on a real-world system that appears to meet the assumptions of such methods, and yet they show that the methods almost entirely fail to recover a valid causal model of the system. Specifically, they test methods under assumptions about the mixing function that transforms the inputs (“causal factors”) into observations, focusing on cases in which such assumptions underpin the identifiability results of the methods.\n\n## Update after Rebuttal\n\nI remain convinced that this paper should be accepted. It is an impressive and useful contribution to the very difficult problem of empirical evaluation of methods for causal representation learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review!\n\nWe wanted to be cautious about making unsubstantiated claims regarding the failure points of the algorithms. Due to their complexity (especially Multiview CRL and CITRIS), understanding the exact source of failure may be very difficult, and it may not be possible to pin it down to a single assumption or implementation choice.\n\nHowever, following your feedback and that of reviewer 55Kj, we have performed additional experiments (hyperparameter searches) for all considered methods and, if accepted, we will add further discussion using the extra page. You can see a draft of this discussion and the supporting experiments in our answer to reviewer 55Kj below (we don’t repeat it here for lack of space).\n\nThank you for raising the above point; we agree that such a discussion can improve the quality of the paper.", "claims_and_evidence": "One of the primary benefits of empirical evaluation is that it can evaluate whether the assumptions of a given method are realistic enough to provide useful results. As the authors clearly state: “This practice [evaluating on synthetic data] provides further validation for the theoretical foundations of these methods but yields limited insight into their applicability to real-world problems.” This point seems to be lost on many researchers, and the authors of this paper place it front and center. Bravo.", "methods_and_evaluation_criteria": "Given the diversity of the three methods being evaluated, the methods and evaluation criteria are similarly diverse. The authors are clear about this, but it leads to some difficulty for readers who are not familiar with each of the methods. That said, the authors deserve credit for “meeting the methods on their own ground” rather than attempting to shoe-horn all the methods into a common evaluation framework.", "theoretical_claims": "The theoretical claims are those made by the original papers, rather than the ones made by this paper.", "experimental_designs_or_analyses": "The experiments apply each of the three methods to data drawn from the physical system (a light tunnel) that forms the centerpiece of this work. As noted above, the diversity of the methods evaluated virtually necessitates a diverse set of experiments, and the authors fairly clearly describe the different approaches used. \n\nOne of the few disappointing aspects of this paper is that it offers only hypotheses, rather than strong evidence, for why the differences in data (between synthetic and real) produce differences in performance. The most promising case is contrastive CRL (CCRL), in which the substitution of highly similar, but synthetic, data produces far better results than for data drawn from the actual system. Even in this case, however, the authors end on a hypothetical note, saying: \"A possible explanation is that CCRL relies on detecting interventions in the latent space, a sensitive statistical problem for which it may lack power in the case of a noisy mixing process.” In the two other cases, however, the authors offer even less clear explanations. For example, in the case of CRL from temporal intervened sequences, the authors state that “…the exact reason for the failure in this setting remains elusive.”\n\nWhile I don’t think strong experimental evidence about the cause of the observed performance difference is mandatory, it would provide greater confidence that the failure of each CRL method is based on a limitation of the method itself rather than some error in the way the authors configured or evaluated the method. Despite this, the paper is sufficiently interesting, detailed, methodologically solid, and well-written that it deserves publication.", "relation_to_broader_scientific_literature": "The paper lies at the frontier of work on causal representation learning, a vitally important research area, given the problems with current, largely associational models that learn representations. The key issue that this paper explores is whether the identifiability assumptions of current methods are practical. While this is only one case, it still represents an important step forward in evaluating methods for CRL.", "essential_references_not_discussed": "I don’t know of references missing, though I am not an expert in this area. The authors cite a large number of apparently relevant work.", "other_strengths_and_weaknesses": "The authors cleverly call their real-world system a “sanity check” rather than a “benchmark” or other name that implies a domain of realistic complexity. Instead, this domain is remarkably simple (as the authors emphasize), and thus the failure of current state-of-the-art methods is even more striking.\n\nThe claims are clearly delineated, and the authors take some pains to make clear what their \"sanity check\" *cannot* do (second-to-last paragraph of Section 1). They also clearly outline the different sources of noise. The authors also take substantial pains to test for bugs in their analysis pipeline (see the last paragraph before section 3.1).", "other_comments_or_suggestions": "(None)", "questions_for_authors": "(None)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yVwSM5xjDZ", "reviewer": "Reviewer_KaPR", "summary": "The paper benchmarks 3 representative causal representation learning methods on real data produced by a simple controlled physical system with known ground truth. CRL models have underlying causal factors that are \"mixed\" into observed variables. In this benchmark, the underlying causal model is simulated (Line 145), but the mixing is done by the physical system. In addition to releasing public benchmarking data and evaluating key CRL methods, the paper highlights some methodological and reproducibility issues in CRL research.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your positive review! We answer your points below.\n\n**Re: bayesian:** We have fixed this in an updated version of the manuscript, thank you.\n\n**Re: Contributions w.r.t.. Gamella et al., 2025:** Indeed, Gamella et al., 2025 introduce the light tunnel. In our work we leverage it to construct a meaningful benchmark for CRL. This required an in-depth analysis of the experimental setup in light of CRL assumptions (section 2), designing new experiments and collecting the data, and developing the deterministic simulators for the synthetic ablation. None of these existed in Gamella et al., 2025. The ICA experiment in Gamella et al., 2025 could also be used as a first test for CRL, but since the latent factors are all independent this didn’t seem very interesting. We will add a paragraph in section 2 to clearly separate our contributions.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes, a reasonable balance is struct between method-specific evaluations and method-agnostic comparisons.", "theoretical_claims": "N/a", "experimental_designs_or_analyses": "Yes, I checked the experimental design, data generation process, and the analyses. I'm not familiar with such physical systems, but the process seems quite reasonable and the data is highly desirable and useful for CRL evaluation. It's too bad the causal model has to be simulated, but anyway the mixing (which is done by the physical system) is what's crucial for CRL and the current benchmark data is nevertheless challenging and interesting enough.", "relation_to_broader_scientific_literature": "Details of the physical system and other real datasets from it are recently published in Nature (Gamella et al. 2025). This complements that with CRL-specific datasets, benchmarking, and methodological discussion.\n\nAlso a nice complement to the the Sachs et al. datasets, which have been widely used in causal discovery benchmarking but are not suitable for CRL.", "essential_references_not_discussed": "I would find it helpful if the paper more clearly delineated its contributions---and especially the physical system configuration and resulting datasets---compared to the Gamella et al. (2025).", "other_strengths_and_weaknesses": "- extremely polished and well written\n- expected to be a common benchmark for much of the future work in CRL", "other_comments_or_suggestions": "- fix \"bayesian\" (Line 506R and 557L) in references and check for any other similar mistakes", "questions_for_authors": "N/a", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zRf4Y48yDa", "reviewer": "Reviewer_55Kj", "summary": "The authors test three representative causal representation learning (CRL) algorithms on real-world data generated from Causal Chambers, which is a small light tunnel that takes in 5 controllable inputs (factors) and outputs numerical sensor and imaging data. The authors treat these measurement processes as a noisy mixing function of the factors and evaluate whether CRL algorithms are capable of recovering them up to   indeterminacies as promised by their respective claims of identifiability. For each representative CRL algorithm, the factors are sampled accordingly to satisfy the respective latent assumptions. As an ablation, synthetic versions of the measurements are obtained by simulating the known physical laws (for the sensors) or by mimicking supervised examples with an MLP (for the images). The results show that, despite latent assumptions being perfectly satisfied, none of the CRL algorithms are able to recover the true factors up to the desired indeterminacies from this noisy real-world mixing function. The same conclusion holds for the synthetic ablation, with the exception of the contrastive CRL method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful review and constructive feedback.\n\n**Re: Gamella et al., 2025:** Thank you for raising this point. As other reviewers (KaPR, uuRR) also pointed this out, we will add a paragraph text in section 2 to clearly separate our contributions from Gamella et al., 2025. Please see our response to reviewer KaPR below for the details of our contribution w.r.t. those of Gamella et al., 2025.\n\n**Re: additional methods:** We agree in principle that running our sanity check on more methods would be insightful. However, our approach of meeting methods in their own ground (as praised by Reviewer DQvb)—using the original code and evaluation techniques—meant that getting a method to work takes a significant amount of time and effort. We can use the extra page for this, but given how difficult it was to get the current methods to run (incl. communicating with the authors + tests to ensure no bugs) we don’t want to promise that we can do this for Zhang et al., 2023 or Ahuja et al., 2023. It is definitely not feasible before the rebuttals are due. We hope others will use our work to investigate individual methods in full detail in future works.\n\n**Re: optimization:** Thank you for raising this point; we find your categorization of failure modes (misspecification vs. optimization) very useful. We fully agree that the latter is often overlooked and that an additional discussion will improve the paper. To support it:\n\n1. We have now performed additional experiments doing a large hyperparameter search for each method on the real data. The results ([see here](https://blue-gina-78.tiiny.site/)) show that the hyperparameter choice does not significantly affect the final metrics, across all models. We will repeat the experiments on the synthetic data for the camera-ready version as time constraints did not permit doing so for the rebuttal.\n\n2. We will add the training curves of all methods ([see here](https://maroon-agneta-85.tiiny.site/)) to the appendix. From them, we conclude that training does converge (although noisy for some methods, cf. Multiview CRL), not raising immediate concerns about optimization failing in some catastrophic way.\n\nWith the above we hope to address (and exclude) some of the key potential failure modes regarding optimization, i.e., losses not going down during training or hyperparameter choices greatly affecting the outcome. Our hyperparameter search was possible because we have access to ground truth labels to compute the final metrics and perform tuning. However, in a practical setting without ground truth, model selection of unsupervised models is still a great challenge, and finding the modes of failure w.r.t. optimization would be even more difficult.\n\nIn addition, we put in a substantial effort to exclude bugs and other issues by performing preliminary checks to reproduce the synthetic experiments in the original  papers using our pipeline, as well as consulting the authors of the methods to advise us, where possible.\n\nRegarding a misspecification in the generative process: to isolate the mixing transformation as the only source of mismatched assumptions, we generated the data closely following the assumptions for each method. We do not merely follow the general assumptions, we also closely replicated the more subtle details of the respective methods’ data generating processes, such as no. of latent variables, noise mean and variances, intervention strengths etc.\n\nTo summarize:\n- CCRL appears to fail because of a misspecification of the generative process (noiseless vs. noisy mixing function), since the method works well for the synthetic ablation but fails on the real data.\n\n- Because the other methods fail on both real and synthetic data, it seems like failure is instead also due to optimization issues. Because the training curves converge, and the performance does not change given different hyperparameters, we argue that there are either architecture choices that need deeper investigation, or finite sample issues that should be better understood. All identifiability results in CRL rely on the unrealistic assumption of infinite data, but how much data is enough in practice remains elusive. We collected as many samples for each method as was possible given the scope of this study, but because there are actual costs to collecting real data samples, these still fall short of the enormous sample sizes in the original papers: CCRL 10k vs 25k (per env.), Multiview 60k vs 145k, CITRIS 100k vs 100-250k.\n\nWe will add all the above as additional discussion using the extra page in the final version. This will be framed (as you suggest) by an explanation for general audiences of how the theoretical guarantees of CRL methods work and how the two possible modes of failure result from this.", "claims_and_evidence": "The submission introduces a new benchmark for CRL. It claims to be the first sanity check for CRL based on data from a physical process, to my knowledge this is true. I agree with the authors that this would be a much preferable benchmark for practical purposes than what is currently in the literature. The submission convinces me that existing CRL algorithms, at least out of the box, are inadequate for practical use. However, I'm a bit unsure about whether the experiments convincingly capture the failure modes: see the next section for details.", "methods_and_evaluation_criteria": "I think the authors are missing a crucial discussion in their evaluation: optimization. \n\nThe authors state that the set-up is \"geared towards evaluating the assumptions concerning the mixing function\" (l. 97r). I think this amounts to saying that the generative process assumed by CRL methods are mis-specified for the true physical process here. While I don't doubt that this could be true, the evaluation here does not rule out bad local optima, which I think is the other possible failure mode here. For example the authors note that results differ drastically over the five random initializations in all methods. \n\nI find it confusing that the authors emphasized using the original implementations, as hyperparameters used in their synthetic experiments for example should certainly not be expected to perform well in a new setting without any tuning. I think the experiments would be much more convincing if more effort were made to ensure that the algorithm at least can reach some sort of stable optima. Of course, if this is simply not possible given the loss landscape, then that itself is a failure mode as well and should be discussed (maybe we should advocate for simpler models then for CRL).", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The data generation part of the design is great, I really enjoyed how the authors mimicked the latent processes as per the original papers before passing it to the physical simulator. The synthetic ablation is a nice touch. As mentioned above I think they could have done more to investigate the optimization landscape. \n\nI also think taking only one representative from the classification of CRL methods may not be sufficient. Currently they appear to be based on the constraints under which they obtain identifiability (which, fair enough, this determines how the data should be generated), but they can actually be algorithmically speaking quite different. As an example for interventional CRL, Zhang et al., 2023 or Ahuja et al., 2023 use an autoencoder setup instead of the contrastive approach. It would be interesting to see how these differ in practice (Ahuja et al. 2023 also consider do-interventions which are qualitatively different but easy to implement in this setting).", "relation_to_broader_scientific_literature": "CRL is crucial for causal reasoning over unstructured data, which is of great importance in many scientific areas, e.g., biology, physics, medicine. To date most advancements in this area have been theoretical (in particular focused on identifiability), so it is very important to design benchmarks that support the design of CRL algorithms for practical uses.", "essential_references_not_discussed": "I believe this is the first benchmarking of CRL algorithms on data generated by a physical process. It includes all the necessary references to understand the contribution.", "other_strengths_and_weaknesses": "### Strengths\n\n- The paper is wonderful to read as someone already familiar with the theoretical literature.\n- The study is extremely important at this stage of the literature for CRL. Continued experimentation and progress based on a benchmark like this can possibly determine whether CRL becomes a practical tool or remains a theoretical exercise in the future.\n\n### Weaknesses\n\n- There may be some novelty overlap with the original causal chambers paper (Gamella et al., 2025), which seems to include an ICA experiment. If the focus here is on strictly causal settings maybe it would be good to mention the difference.", "other_comments_or_suggestions": "### Suggestions\n\n- For general audiences, I think the paper could do a better job explaining what the theoretical guarantees, and thus resulting failure modes, of the CRL algorithms are. Usually, it is based on minimizing some objective relative to the true generative process which is assumed to have certain properties, and based on minimizing this objective exactly, saying that the resulting representations are related to the ground truth in some sense. Given the theory, the only failure modes that I can think of when transitioning to practice is to do with 1) mis-specification of the generative process, and 2) optimization issues. As mentioned above, I think the paper explains 1), but not 2).", "questions_for_authors": "I don't have any additional questions, but I will just summarize the two points I tried to make which I think would improve the paper. As-is my score is \"Weak Accept\" which is primarily based on clarity and the importance of the problem. \n\n1. Discussion of failure modes of CRL relative to the theory, in particular discussing failed optimization in addition to unmet assumptions. Empirically, attempting to improve optimization in the experiments or showing that it is infeasible for a given algorithm (see \"Methods And Evaluation Criteria\" and \"Suggestions\"). \n\n2. Choosing representatives for CRL algorithms to evaluate not only based on their assumptions for identifiability but also practical implementation (e.g., autoencoder vs contrastive, see \"Experimental Designs Or Analyses\"). \n\nI would be happy to argue for acceptance if the authors could satisfactorily discuss aspects of (1). If additional experiments could eventually be included in the paper corresponding to (1) and (2) I would lean towards strong acceptance.\n\n\nPOST REBUTTAL COMMENT:\n\nThank you for responding to my questions. As mentioned, the discussion about optimization is what moves the needle most for me. The author's response on architecture, sample sizes, etc. should be included/expanded upon in the paper. Therefore I will raise my score to 4. I understand it is unfeasible to ensure a high quality for new implementations within the rebuttal period but I still feel the experiments could have been more comprehensive from the start, thus I cannot justify a 5.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "32ZxH7ffbs", "reviewer": "Reviewer_uuRR", "summary": "This work evaluates causal representation learning (CRL) methods on a simple, real-world optical system designed as a sanity check for CRL assumptions. The authors argue that while many CRL methods show theoretical promise, they fail when applied to this controlled real-world system due to critical issues such as noise sensitivity and unrealistic assumptions about the mixing function. To investigate further, they perform a synthetic ablation using a deterministic simulator for the optical experiment, revealing that many methods also fail on simplified synthetic data. They evaluate representative methods from contrastive, multiview, and time-series CRL approaches, highlighting reproducibility challenges and performance gaps. Extensive experiments on this benchmark reveal that existing CRL techniques struggle to recover the underlying causal factors, underscoring the gap between theory and real-world applicability.\n\n## update after rebuttal\n\nAfter the discussion with the authors, it seems the key message and motivation of this work are not clear, making the implications of the results in this work limited. Therefore, I am leaning not to accept this work.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We respond to your points below.\n\n**Motivation & differences to other benchmarks:** As all other reviewers highlight and we stress in the paper, the primary distinction of our sanity check is that it stems from a real, physical system to evaluate the practical validity of assumptions:\n- Rev. DQvb: \"[evaluating on synth. data] provides further validation for the theoretical foundations of [CRL] methods but yields limited insight into their applicability to real-world problems. This point seems to be lost on many researchers, and the authors of this paper place it front and center. Bravo.\"\n- Rev. KaPR: \"the data is highly desirable and useful for CRL evaluation\"\n-Rev. 55Kr : “I believe this is the first benchmarking of CRL algorithms on data generated by a physical process.” and “would be a much preferable benchmark for practical purposes than what is currently in the literature”\n\n**Key messages:** You say “It would be better to formulate the underlying research problem and the concerned factors that this benchmark is designed to study and control the other factors more rigorously.” We believe we have clearly explained the factor we study (the mixing function), we rigorously control the other factors, e.g., generating the latents according to each model’s assumptions, and made very clear what conclusions can be drawn. Citing Rev. DQvb: “The claims are clearly delineated, and the authors take some pains to make clear what their \"sanity check\" cannot do”\n\nRegarding failure in the optimization routines, we have performed additional experiments to study this mode of failure. See the answer to reviewer Rev. 55Kj for a draft of what will be included in the final version.\n\n**MCC:** As Rev. DQvb points out, we aim to “meet each method on their own grounds”, using the same validation metrics as in the original papers. As we stress, this is a sanity check, and our focus is not to compare methods to each other (difficult given their different goals). We agree that the MCC is not an ideal metric for CRL, but it is the most commonly used one and we are not aware of a metric that directly addresses its shortcomings. It is well beyond the scope of this study to derive a novel metric.\n\n**Systematicity:**\n- Measurement error: The work you reference considers measurement errors for causal discovery, which we do not directly see being applicable to CRL. While CCRL does not explicitly consider measurement noise, this is no reason to exclude it from the sanity check. Measurement noise is inherent to any real-world data; the failure of this method points to an important issue in realistic scenarios that should perhaps be considered by future works.\n- Other failure modes: As you point out, many things can go wrong—beyond the mixing function—when transferring CRL from theory to practice. However, we argue that focusing precisely on this single factor (as opposed to considering everything at once) is what gives our study systematicity (see answer to Rev. 55Kj). As Rev. 55Kj points out, we meet all other assumptions of each method to isolate and better understand the effect of misspecified assumptions regarding the mixing function. Given the subsequent failure of methods after only this single (!) misspecification, *we believe our setting already highlights a problem worth discussing*. Including more factors of analysis is of course always possible, but (1) it is beyond the scope of such a paper and (2) it dilutes the point of focusing (and understanding) the failure mode in such a simple setting.\n- As you point out, this work is a sanity check and *not* a benchmark that aims to exhaustively study any possible mode of failure. We stress this throughout, as other reviewers (DQvb) also acknowledge. CRL is far from being practically applicable, and our sanity check is meant to highlight the “remarkable failure of SOTA systems in such a simple setting” (Rev. DQvb), rather than comparing working approaches.\n\n**Method choice:** Any choice will exclude other methods, we don’t claim exhaustiveness, nor do we draw conclusions from these representatives to their whole family of methods. Our results highlight their failures and that this is worth looking into. Citing Rev. 55Kj: “[it] convinces me that existing CRL algorithms, at least out of the box, are inadequate for practical use”\n\nWe chose our representative methods for the following (often pragmatic) reasons:\n- CCRL: allows for general nonlinear mixing functions (cf. Ahuja et al. 2023, Zhang et al. 2023 with polynomial mixing, Squires et al. 2023 with linear mixing) and uses a contrastive loss, which has proven superior to an encoder-decoder in other representation learning domains.\n- Multiview CRL: multiview approach that permits more than two views and considers nonlinear mixing (on images), with accessible code.\n- CITRIS: considers complex nonlinear mixing experiments (images), very well maintained code base.\n\n**Technical novelty:** See our answer to Rev. KaPR.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "MCC may not be a stable evaluation metric for non-linear correlations. It would be nice to extend to more evaluation metrics to robusitify the evaluation results.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n\n(+) This paper considers a critical point in CRL, which has been neglected by the advance of theories in this field.\n\n(+) The authors conduct extensive and thorough discussion about their experimental settings, and provide interesting results;\n\n## Weakness\n\n(-) The study lacks systematicity:\n- From the design of testbed, it considers only a limited part of the factors in CRL such as the mixing function:\n  - In theory, especially, there are already some works working on measurement errors such as [1]. The focus of CCRL seems not to resolve the measurement error introduced in the mixing functions. \n  - Moreover, there are also other factors such as hidden confounders, missing data. What factors do the authors choose to study when constructing the benchmark and why? Are they sufficiently representative?\n  - In implementation, it has already been mentioned that network architectures and optimization could also affect CRL. However, they are not sufficiently discussed and studied.\n- From the methods: there are also other CRL works open sourcing codes. Why do the authors choose the current three methods instead of the others?\n- In addition, it is worthwhile to clearly highlight the contributions of the benchmark from the existing CRL benchmarks.\n\n(-) The motivation and the key messages are not clear:\n- In the analysis of CCRL, the noises in mixing function seem to be an critical issue for the failure of CCRL, while it is not the focus of CCRL;\n- In the analysis of Multiveiw CRL, noises in mixing function seem not be influential anymore, while the separation of latent variables seem to be a critical influencing factor;\n- In the analysis of CITRIS, it is even confusing that the implementation of the submodules may be the reason for its failure. \n- It would be better to formulate the underlying research problem and the concerned factors that this benchmark is designed to study and control the other factors more rigorously. Otherwise, it is quite confusing to see what messages this work would like to convey through benchmarking.\n\n(-) The technical novelty may be limited since it is directly applied from Gamella et al. 2025.\n\n\n**References**\n\n[1] Causal Discovery with Linear Non-Gaussian Models under Measurement Error: Structural Identifiability Results", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please find my questions in the previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The main purpose of this submission is to benchmark 3 CRL methods using a real physical system from Gamella et al (2025). This is an important problem given the primarily theoretical leaning of CRL papers. As with any real-world experiments, there are necessarily some limitations: Several reviewers pointed out a blind spot with respect to how the authors control the optimizer, as well as significant overlap with Gamella et al (2025). The authors have promised to address both these points, which should be easy. Another reviewer pointed out a misspecification issue, which the authors did not respond to but should acknowledge and discuss in the camera ready. 

Although the experiments have some limitations, this is easily the most thorough and realistic sanity check of CRL available to date. Given the timeliness of this submission and overall positive reception from reviewers, I recommend accept.
Decision: Accept (oral)