Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions
Authors: Hidde Fokkema
Tim van Erven
Sara Magliacane
Venue: NeurIPS 2025 poster ()
Keywords: Interpretability
Concepts
Causal Representation Learning
Primary area: social_and_economic_aspects_of_machine_learning
TL;DR: We describe a framework that provides theoretical guarantees on the correctness of learning concepts from data and on the number of required labels.

Abstract: Machine learning is a vital part of many real-world systems, but several concerns
remain about the lack of interpretability, explainability and robustness of black-box
AI systems. Concept Bottleneck Models (CBM) address some of these challenges
by learning interpretable concepts from high-dimensional data, e.g. images, which
are used to predict labels. An important issue in CBMs are spurious correlation
between concepts, which effectively lead to learning “wrong” concepts. Current
mitigating strategies have strong assumptions, e.g., they assume that the concepts
are statistically independent of each other, or require substantial interaction in
terms of both interventions and labels provided by annotators. In this paper, we
describe a framework that provides theoretical guarantees on the correctness of
the learned concepts and on the number of required labels, without requiring any
interventions. Our framework leverages causal representation learning (CRL)
methods to learn latent causal variables from high-dimensional observations in
a unsupervised way, and then learns to align these variables with interpretable
concepts with few concept labels. We propose a linear and a non-parametric
estimator for this mapping, providing a finite-sample high probability result in the
linear case and an asymptotic consistency result for the non-parametric estimator.
We evaluate our framework in synthetic and image benchmarks, showing that the
learned concepts have less impurities and are often more accurate than other CBMs,
even in settings with strong correlations between concepts.

Sample-efficient Learning of Concepts with Theoretical Guarantees: from Data to Concepts without Interventions
Hidde Fokkema
Korteweg-de Vries Institute for Mathematics
University of Amsterdam
j fokkema@uva....
Tim van Erven
Sara Magliacane"
Korteweg-de Vries Institute for Mathematics
Informatics Institute University of Amsterdam
University of Amsterdam
iim@timvanerven.cr
magliacama @uva n1
Abstract
Machine learning is a vital part of many real-world systems, but concerns remain about the lack of interpretability, explainability and robustness of black-box Al systems. Concept Bottleneck Models (CBM) address some of these challenges by learning interpretable concepts from nigh-dimensional data, e.g. images, which are used to predict labels. An important issue in CBMs are spurious correlations between concepts, which effectively lead to learning "wrong" concepts. Current mitigating strategies have strong assumptions, e.g., they assume that the concepts are statistically independent of each other, or require substantial interaction terms of both interventions and labels provided by annotators. In this paper, we describe framework that provides theoretical guarantees on correctness the learned concepts and the number of required labels, without requiring any interventions. Our framework leverages causal representation learning (CRL) methods to learn latent causal variables from high-dimensiooal observations in a unsupervised way, and then learns to align these variables with interpretable concepts with few concept labels. We propose linear and -onpparaettiic estimator for this mapping, providing a finite-sample high probability result in the linear case and asymptotic consistency result for the non- -aramaraectiic estimator. We evaluate our framework in synthetic and image benchmarks, showing that the learned concepts have less impurities and often more accurate than other CBMs, even in settings with strong correlations between concepts.
1 Introduction
Machine learning is a vital part many real-world systems, but concerns remain about the lack of interpretability, robustness and safety of current systems [9]. These issues might be exacerbated by the lack of guarantees in explaining the behavior of AI systems in terms of interpretable, high- level concepts. The field of interpretable machine learning and explainable AI has developed many techniques to interpret models and explain their predictions [44], either by extracting known concepts from the internals of black-box models [25, 15, 16. 36), or by building the explicit use of concepts into the internals of these systems, e.g. as in concept bottleneck models (CBM) [28, 21, 39, 57].
Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CRL on unlabelled data
Learning aligaments with concept lahels
Label predictor
A/1
Cole object
G
C
Rot. object
X
Ma
Ca
Rot. spotliphi
M
Y
C
Figure I: Left: An overview of our framework: we learn the alignment function that maps causal representations M, learned on cheap unlabelled data by causal representation learning (CRL) encoder, to interpretable concepts C3 using only few concept labels. As in standard CBMs, these concepts are used in downstream task like regression or classification process, where G are the latent causal variables, X is observation, M are the representations learned by model Iv. C are the interpretable concepts and Y is the final label.
Y. Right: Data generating
An advantage of CBMs is that they can in many cases provide similar accuracy in terms of prediction compared to black-box methods, while also ensuring interpretability by construction [28, 21, 57], as opposed to post-hoc methods [8]. Users may also interact with the models by verifying and, necessary, correcting the detected concepts at test time [28]. However, existing methods suffer from two important limitations. First, spurious correlations in the training data may lead learned concepts that do not capture the intended semanties of the concept [15, 41]. encode more information than intended [38, 58], even with concept-level supervision. Recent works propose mitigation strategies, but implicitly assume that the concepts are statistically independent [39, 50] or require interventions on the data generating process with labels [40]. Secondly, in addition to labels required for the learning task at hand, CBMs require many expensive concept labels that may not always be available. Attempts at working around this issue have so far focused on obtaining concept labels from additional sources like GPT-3 [45] using multi-model information [54].
In this paper, we propose a framework that provides theoretical guarantees on the correctness of the learned concepts and on the number of required labels without requiring any interventions. We assume that the concepts that we want to learn correspond to latent variables in a causal system and allow arbitrary dependences between them. As illustrated in Fig. 1 (left), we propose two- step pipeline: (i) leverage out-of-the-box causal representation learning (CRL) methods learn provably disentangled representations of concepts M even in case of spurious correlations; (ii) learn an alignment map a between these representations and the interpretable concepts with thcoretical guarantees and as few concept labels possible. In setting, the map consists of permutation and element-wise wise transformations. Unlike standard CBMs, our pipeline only uses concept labels in the second stage to disambiguate the relation between the representations learned by CRL and the concepts. This requires much fewer concept labels, addressing the second limitation for CBMs.
We introduce two principled estimators to learn based on variants of the group lasso [56] and weighted matching procedure. The first estimator is based on a linear model with an optional feature map, and comes with a high probability finite-sample result on its correctness (Thm. 4.2). This result provides explicit guidance for tuning the regularization hyperparameter, and shows how the required number of concept labels scales with problem parameters like the number of concepts and the dimension of the feature map. Our second estimator is based on a kernelized procedure, allowing for more flexibility, but comes only with asymptotic guarantees (Thm. 5.2).
We evaluate our framework in synthetic and image benchmarks with both real-valued and binary concepts. The experiments show that the learned concepts have less impurities, measured in terms of OIS and NIS [58], and are often more accurate than other CBMs. The experimental results additionally indicate that our estimator performs well beyond the assumptions of our theorems. For example, our estimators are able to find the correct permutation, even in settings with strong correlations between the concepts.

---

2 Related Work
Extracting high-level concepts from the inner workings of machine learning models has gained traction over the last years [3, 14, 42]. These concepts can used to create interpretable explanations. In concept bottleneck models, the concepts are hard-coded into the structure of the model [28, 39, 21, 57]. This approach has been quite successful, with multiple variants, c.g. probabilistic CBMs [26] or energy-based CBMs [53]. One drawback of CBMs is that the learned concepts may not correctly capture the intended semantics [41] and that their representations can encode impurities, e.g. more information than intended, [38, 58]. Havasi et al. [17] address this problem by allowing a more expressive autoregressive concept predictor that can take advantage of the correlations between concepts. On the other hand, the issue is exacerbated when there are spurious correlations among concepts in the training data. In this case, Marconato et al. [39] encourage disentangled representations of the concepts by adding a regularization term in the concept predictor. Similarly, Sheth and Kahou [50] introduce a concept orthogonal loss that encourages separation between concept representations. Both approaches implicitly assume the concepts be statistically independent, which means that they can fail to learn the correct concepts in settings in which there are dependences.
We instead propose leverage causal representation learning (CRL) [49] the first step of our pipeline. CRL aims at recovering potentially dependent latent variables from observations. Identifying these latent variables is often only possible up permutation and element-wiie transformation. Many CRL methods exist with different assumptions on the available data, e.g. the availability of interventional, counterfactual or temporal data, or parametric assumptions about the underlying system and observations, e.g., [20, 24, 51, 31, 33, 2, 30, 55] and many others. Most methods focus on the infinite sample setting, with the exception of [1], which provides sample complexity results for the CRL task with interventions. Our framework is agnostic to the CRL method used and focuses on the supervised learning task of aligning concepts efficiently to the causal representations.
Recent work relates identifiable representation learning, including causal representation learning, with concept-based models, but in an unsupervised concept discovery setting, opposed to our supervised setting. Leemann et al. [32] propose an identifiable concept discovery method that uses classic meth- ods like PCA and ICA for independent concepts, but also introduces novel disentanglement methods for dependent concepts, based on disjoint mechanisms or independent mechanisms. Rajendran et al. [46] argue that the identifiability of CRL methods often requires too many assumptions, e.g. access to interventional data, and hence relaxes the identifiability of concepts by defining them as affine subspaces of the latent space of the causal variables. Given set of concept conditional datasets, they prove they can recover these concepts up to linear transformations.
The most closely related work is by Marconato al. [40], who describe theoretical framework that shows how that disentangled representations can be used in CBMs to avoid concept leakage. Their theoretical framework suggests that to align learned embeddings and concepts one might require interventions on the data generating process and labels. Instead, we do not require interventions. Moreover, we provide a theoretical analysis of the alignment function, proving guarantees about error bounds on the learned concepts or the required number of labels.
Framework and Main Definitions
Our setting takes inspiration from causal representation learning (CRL) [49] and the connections between CRL and CBMs described by Marconato et al. [39, 40). We assume that the interpretable concepts are related to causal variables in an unknown data generating process.
As illustrated in Fig. (right), we assume a causal system with latent causal variables G (GI,..., Ga) E G C Rdd, which can potentially have causal relations between them. The ob- servation is denoted by X E X C _ and is generated by unobserved invertible mixing function f:g 9 + X' as X = f()). The goal of CRL is to recover the causal variables by learning a function Go: X + d that approximates f-1, This requires specific assumptions and we cannot identify the ground-truth variables exactly, but only up to an equivalence class. A common notion of identifiability is up to permutation, scaling and translation [19, 24, 31], which means that the learned representa- tion satisfies (X) PAG + 6 for permutation matrix P. an invertible diagonal matrix A and
3

---

vector b. A more general notion is identifiability up to permutation and element-wist transformations in which, instead of diagonal matrix, we consider a diffeomorphism, which we formalize as: .d + fl....,d] df be a permutation of the variable indices. Let P E TRdxd be the permutation matrix associated with x, meaning that Pix(1) and otherwise, and To R + Rd map. A representation Z identifies the ground causal variables up to a permutation and element- wise transformation if:
Definition 3.1. Let w: fl.....d]
PT( =
Ta(ay (4))] = [G1,...Gal = G
There many more notions of identifiability in the literature, e.g., in some cases the causal variables can be multidimensional [33] or identified up to a block, i.e. a group of causal variables [2, 51, 55, 30]. In the main paper we focus mostly the single- migle-dimensionll case in which we identify each individual variable, but we provide extensions to the multidimensional case in App. A.
We denote the learned causal variables by M = (M Ma) E R For simplicity of exposition, in the main paper we will assume that the interpretable concepts C' = .E.....,C.)" E F correspond to the ground-truth causal variables GI...., ... G. up to permutation and element-wise transformations; in other words, Per shown in Fig. 1 (right) also identifies ground-truth causal variables up to the same identifiability class. In App. A and Sec. we extend this to allow each of the concepts to be transformation of group of causal variables. Our goal is to learn the alignment map that transforms the learned representations M to the concepts C' efficiently and accurately.
Note that in our theoretical results we assume that both concepts and causal variables are continuous- valued. Our motivation for this choice is technical: most current CRL methods assume that the underlying causal variables are continuous-valueed and only very few methods identify discrete- valued causal variables under very specific assumptions, e.g., in cases where different causal variables always affect different parts of an observation [29]. Since in concept- based models it is common to work with binary concepts, we also test these cases in our experiments and show that our estimators still provide good results, but we leave the extension of the theoretical results to discrete causal variables and concepts for future work.
For our theoretical analysis, we assume that we are given a function Pv from a CRL method that correctly identifies the ground-truth causal variables up some identifiability class and that the interpretable concepts correctly identify the causal variables up to the same class.
Assumption 3.2. Let M = 9u (X) ) be the causal representations learned by a pretrained CRL method and let C yor (X) be the interpretable concepts annotated from the observation X. We assume that M and C' identify G up to the same identifiability class. This implies that
PT(M) (I+c)) ... TT(a) (M++a)) [CI,... Ca] C. =C.
(I)
While this assumption is needed for our thcoretical analysis, we will show that empirically our framework works even when the CRL methods do not fully identify the causal variables. Finding the alignment function a in Fig. 1 reduces to learning the permutation ı and a separate regression per concept C, to learn the transformation from learned causal representation Ma(e) to interpretable concept C.. The main difficulty here is identifying ı from observational data, i.e. without performing interventions, and with few samples. In the following we introduce two estimators for this setting, one assuming the clement-wise transformation is linear, e.g. is the case in some CRL methods like [20, 24], for which we will be able to provide finite sample results based on a tunable parameters, and second, non-parametric method based on kernel methods that allows for arbitrary invertible element-wise transformations, and can hence be applied to most CRL methods. After recovering the interpretable concepts, we can use them as in CBMs as inputs to a label predictor for Y.
Linear Regression Alignment Learning with the Group Lasso
In this section, we describe a linear regression approach based on the Group Lasso to learn the permutation π and transformation T in (1). We will prove that this method simultaneously provides accurate regression estimates for T and identifies и correctly with high probability. To simplify the exposition, we focus here on the case of scalar variables. Proofs are in App. A, which also contains discussion of the assumptions, pseudo-code description Alg. and generalization to block variables. The proof combines techniques from high-dimensional statisties [10, 35].

---

Method. Linear regression can describe non-linear relations by transforming covariates using feature map yp : + RR. In this section, we assume that T; can be expressed as linear function of *p( ()))))) The choice of yp therefore gives precise control to trade off interpretability with expressive power for T. For instance, the simplest and most easily interpretable case, yp can be the identity function, so that p 1 and C₁ and (i) are related by scaling. In more challenging settings, richer functional relations may be needed, e.g. splines or random Fourier features. We apply the same feature map to all machine variables in M, for which we write ((MM = [(((M) .....)((MM)) " Then each C2 is modeled as a linear function of the transformed variables:
C ( +
(2)
where 3 E Rdd is unknown parameter vector, and e; (V( is Gaussian noise. By assumption, C; only depends on Mats) and not on any of the other variables, so (3, is sparse: only the coefficients for ((M)(() are non-zero. To express this formally, let G = {(j ..p......p' be the indices that belong to variable Mj and, for any B E Read define /3 = (3x k E Gý) to be the corresponding coefficients. Then (3 is non-zero only for j = )i).
We assume we are given a data set D {(C(), M( that contains n independent samples of corresponding pairs C (c{e) "c ) and M = (M ..... M()). We stack the ( into matrix C E Raxd and the feature vectors yp (M(() into + Rxpppd This leads the relation C = 2 +
where C, is the i-th column of C and the noise vector E, consists of n independently drawn N NV o²) variables. To estimate 87, use the Group Lasso with parameter À > 0:
8, -ar min I|.. 0.312 \/ B11,,
1 Σ 1
BE
The (2, 1)-mix norm |2, 1 in (3) encourages group-wise sparsity. applies the Euclidean norm ||Sو|| to each group j separately, and sums the results over groups, as defined below. We also define the (2, oc) -mix norm as || 2,00 = maxj=1,...dd ||
Theoretical Analysis. We denote the full covariance matrix by Σ = T कb. For the group of columns of & that correspond to srM,) we write $ = .,.- Also let Enjy = 1 Φ6 denote the covariance matrix between groups j and j', and abbreviate E, Ezy. Then, w.l.o.g., we can assume that the data within each group have been centered and decorrelated:
1 T = 0 and 12
Σ₁= =1
for all j = 1......, d.
This can be achieved by pre-procsssing: subtract the empirical mean of Φ2 and multiply it from the right by the inverse square root of the empirical covariance matrix. Preprocessing is allowed in our theoretical results, because they apply to the fixed design setting, so probabilities refer to the randomness in C conditional on already having observed $. If sp(M)) and (p(M)) are completely correlated, then 33 is not uniquely identifiable, no matter how much data we have. To rule out this possibility, we make the following assumption, which limits the amount of correlation. This is standard assumption when analyzing the Group Lasso [35, 34].
Assumption 4.1. There exists > 1 s.t. for all j + j',
(Eir)al
1
(Eiy) )arl S
1
max LE (E
max 14a (l...p))
14ap"
Theorem 4.2. Suppose the data have been pre-processed to satisfy (4) and let Assump. (4.1) hold. Take any E (0, 1) and set X 2 4Ag, where
20 nn
8 Blog(d (log(d/5)) 8log(d/8)
A
1
P
p
and set c = (1+ 7(a-1))
24 ) Then, any solution B, the
the Group Lasso objective (3) satisfies
113, - -3,11,,000 I ctvP
(5)
5

---

with probability at least 1 - 6 d' If. in addition, (1 > pc then (5) implies that J..
arg max
[ estimates Tii) correctly.
Theorem 4.2 gives us an explicit relation between the parameters n. p, d. å of the learning task, the tuning of the hyperparameter X, and the estimation errors for ( and T(i). For example, if we set À 4Au and let - oo, then À + 0 and J, estimates the correct index T(i) with probability 8= == 3, n2 tending to 1. So, regardless of the true parameter magnitude [[());) ) the estimator is consistent given sufficient amount of data. Another way to express this is to ask about sample complexity: which sample size n do we need to reach accuracy E > 0? Setting À = 4Ao and solving for n large enough that /\vp= S E, we see that
6 v 8p (plog(//))) + llog(d/))
E
is sufficient. For estimating the permutation (i) correctly, the required accuracy is E S (1())) so the larger the true parameters, the easier this task becomes.
The estimation is performed separately for each concept C₁ and, if J, is correct for all i simultane- ously, we can construct valid estimate of the permutation by (t)) Jr. However, this estimate is not robust to estimation errors and may even produce functions т that are not permutations if some Jj are incorrect. The actual estimator of the permutation, म. therefore optimizes a weighted matching problem, which leads to the same estimate as गे if the J. together produce a valid permutation, but forces ā to be a valid permutation even they do not:
19 1.
arg max
(6)
Here, IT the set of all permutations. This assignment can be solved without cycling through all permutations, with cubic runtime in the dimension d. By a union bound over i, it follows from Theorem 4.2 that ā estimates the true permutation correctly with high probability:
Corollary 4.3. Assume the same setting as Theorem 4.2 such that for each = 1,.... d, (13))) 2cA 2/p and consider the estimator ß as defined in (6). Then के with probability at least 5.
5 Kernelized Alignment Learning
The previous section describes how learn functions with knite-dimensional represetttioonssss We now extend the estimator to use general functions from reproducing kernel Hilbert space (RKHS) [18]. This may be interpreted as a (typically unfiite-dimmnsionall feature map & that maps to the RKHS. However, using a representer theorem, all computations can be performed on finiteddimensional representations. We summarize the method in Alg. in Appendix B.
Method. Define again M = (M MM), where we now allow each machine variable M, to take values in an abstract space Zz. Let Z1 X ... Za. We then generalize (2) to
C. = 8 (M) + Eir
where B1 E H is function from 2 to IR, and 8; NN o2). The space of possible functions H will be an RKHS containing functions of the form B(M) = 3 (M,). where each / is an element of an RKHS H, that captures the effect of variable M, on C.. The assumption that C, depends only on Mr(s) means that (M) = (8)) (M Each H. can be freely chosen, and is typically specified indirectly by the choice of positive definite kernel K. 2, x Z, - IR [18]. This kernel defines measure of similarity between inputs: K. (M,. (P, ()) (M,))) where
Py : 2, - Hl is the corresponding feature map. See 34 for examples.
Given data D = {(C(), (()) let M E zn denote the matrix with the machine variables (M stacked as rows. If we further define ((M)= = [3(M()), ((M())((())))) then the Group Lasso objective (3) generalizes to
3, arg min LIC,, ((M) + A Σ || 14
BER

---

where || is the norm associated with Hy To optimize the objective in (7), we need a finite dimensional objective give Group Lasso solver. We provide version the Representer Theorem showing that the solution of (7) lives in subspace of H. that can be described by finite- dimensional parameters .., E :
Theorem 5.1. Let ; 4d the feature maps associated with Hi....; Hd. Then there exist , ad E IR" such that the optimization problem in (7) has solution 3r with each 3r of the form
B
Σ
(M
Substitution of this form into (7) gives that Ct...., will be the minimizers of the following finie-dimennional optimization problem:
III. Σ K,,211+ Σ | |e
min ERR
(8)
j=1
where K2 E R=**" with (Ky)es = K2 ky(a)) ) and ||'||,,, = V Kjo.
This procedure is performed for i = I,..dd d. The permutation is estimated as in the linear case:
= arg max
1c,
1K400-
REll
Theoretical Analysis. Using result by Bach [4], prove that our estimator for is consistent under suitable conditions, discussed in App. B. This holds for random design, so for the joint randomness of DD
Theorem 5.2. Assume (A-D) in App. B.2. Then, for any sequence of regularization parameters An such that An + 0 and vnin + too when Do, the estimated permutation से converges in probability to æ.
To use a standard Group Lasso solver we need to reparametrize the optimization problem in (8), because the scaled norms || ||,, We can do this with a Cholesky decomposition: Lemma 5.3. For each j = l..... d let L3 be the Cholesky decomposition of the Gramm matrix Kj, then 2 (155)199 if the parameters 4/1 1---- 3/ minimizers of
Implementation.
d
min HER
[||C,
Σ 1, A117
(9)
j=1
Experiments
We perform two types of experiments on four different datasets to evaluate our estimators. In the first type of experiment, we evaluate how well the estimator can learn the permutations and recover the original concepts, which can be either continuous or binary. In the second type of experiment we assess the usefulness of our estimator to perform a downstream classification task, where the target label is binary. For this task, concepts are binarized by setting the concept to 0, if the value is lower than the midpoint selecting sub-selection of concepts and checking if some of those columns are 1. This creates random classification task for each seed. While the thcoretical guarantees for our estimator do not translate directly to setting where the concepts are binary, can use the logistic Group Lasso [43] and we see empirically that this variant also performs well.
range that concept, and otherwise. The label is randomly generated by
The first dataset, called "Toy dataset", is synthetic. Here the concepts are generated using either linear combination of features (which we call the wellspecified case) or diffeomorphisms (which we call the misspecified case) of the representations. The concepts are then permuted. We evaluate on datasets common CRL: Action and Temporal Sparsity [31] ] and Temporal Causal3Dident [33]. We
train several CRL methods as a first step of our pipeline: DMS-VAE [31], CITRIS-VAF [33], iVAE
[24] and TCVAE [11]. In the downstream task experiments, we compare to CBM [28], CEM [57]
and HardCBM [17]. Details are in App. D.
7

---

degress
degrees 8 knote & 3 dagreos
knoto & daggeus
group
Spenrmon
6-100
4-100
A-0.01 01
2=0.01 xpaemfale we 1250
specmtrus 1250
specmfidonn am1250
specttrue no 1250
Pormitatio Firor
1.0
1.0
0.8
0.8
0.6
0.6
0.4
0.4
0.2
0.0
0.0
10
10
10
10
20
40
60
80
100
20
40
100
Regularization (A)
Regularizatinn (A)
Dianension (d)
Dimensson (d)
4-00 A-0.01
d-00 A-0.01
d-000 A-001
d-800 A-0.01
sper-true n=-220
spec-falre n-1250
p-O spec-true
p- spoc-Talls
Permittion Eiror
1.0
1.0
0.5
0.5
0.6
0.4
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
10a
10a
Correlation (p)
Correlation (p)
Nr. data points (n)
Nr. data points (m)
Figure 2: Permutation error rate for spline features. From top left bottom right we vary: (i) the regularization parameter, (ii) the number of dimensions, (iii) the correlation of the variables and (iv) the number of labels. The first plot of each pair shows the wellspecified and the second the misspecified case. We average over 10 seeds and shade the 25-75th percentile.
Performance Metrics. To assess our estimator in terms of concept reconstruction capabilities we report the mean error in the learned permutation of the variables: MPE 1 Ei= { + x())-- In the experiments where we test its capabilities for downstream tasks such as classification, we report the accuracy the final label and the Oracle Impurity Score (OIS) [58]. The OIS measures how much information each concept is contained in other concepts. For concept interpretability. it is desirable that the OIS should be as small possible. We report extended discussion of our results in Appendix E, with additional metrics, e.g. the NIS [57], and computation times.
Toy Dataset. The synthetic data experiments consist of 4 settings, each using a different set of features to perform the regression: linear, splines, random Fourier features (RFF) and kernels. The M variables are distributed according A (1 p) + pl1), where denotes a matrix filled with only I's. The p E (0. 1) parameter controls the amount of correlation between the marginal variables. We sample n data points, on which we perform 80/ 80/20 train/test data split. In the wellspecified case, we generate the M variables using the features from that setting. For each dimension j 1, 1....,d, d, we draw a random weight vector 83 E IR." such that ||,3]|| [16λo, 32Ao] uniformly. A permutation [l....,d}} d} is sampled uniformly from all possible permutations. Finally, with independent e; (2, noise variables, we get C, ((M+()) (*())) + In the misspecified case, the M variables are still sampled the same as before, but the variables are generated by sampling diffeomorphism for each dimension. These outcomes are then permuted using a random permutation again. We cover a large range of values of p.d, and as shown in App. D.I.
:: .
Action/Temm Temporal Sparsity Datasets. We use the synthetic datasets from [31] that represent the action and temporal sparsity settings in a time series. These settings have 10 causal variables -1,-.., Z10 E [-2, with causal structure. The mixing function an invertible neural network with Gaussian random weights, after which the columns in the linear layers are orthogonalized to ensure injectivity. To recover the ground-truth permutation, we follow Lachapelle al. [31] and use the test set to calculate a permutation based on Pearson correlations.
Temporal Cddsa.3DIdent. We evaluate our methods on an image benchmark, Temporal- Causal3DIdent [33]. The dataset consists of images of 3D objects, rendered in different positions, rotation and lighting. The causal variables are the position (ær y, 2) [-2, 2/2, the object rotation with two dimensions (a: 3] [0, 2,, the huc, the background and spotlight in j0, 2T). The object

---

Table 1: Label Accuracy and the OIS-metric on our downstream task. The (n) indicates the number of train and test points used in each column. We averaged the results over 10 seeds and report the mean and standard deviation. The best result for each n is written in bold.
Label Acc. 1 (n)
OIS + (n)
Model
Method
20
100
1000
10000
20
100
1000
10000
Action Sparsity Dataset
Linear 0.77 = 0.03 0.81 = 0.01 0.83 + 0.01 0.84 + 0.01
0.63 0.01 0.40
0.00 0.15 = 0.00
0.12 + 0.00
DMS-VAE
Spline 0.72 1 0.03 0.83 1: 0.02 0.85 1 0.01 0.86 + 0.01
0.63 0.01
0.40
0:00
0.15 = 0.00
0.12 1:2::000 0.00
RFF
0.77 = 0.02 0.82 = 0.02 0.84 + 0.01 0.85 + 0.01
0.64
0.39
1 0.00 0.14 0.00
0.12 0.00
Linear
0.73 = 0.03 0.79 0.02 081 = 0.01 0.83 = 0.01
0.03
0.40
029 = 0.00
iVAE
Spline
0.69 = 0.02. 0.76 =.02 081 = 0.01 0.83 + 0.01
0.63 0.01
0.40
+ 0.28 0.00
RFF
0.59 0.04 0.70 i: 0.02 0.78 1: 0.81
1
0.04
0.39 0.00 0.28 0.00
0.00
CBM [28]
0.60 0.03 0.60 0.02 0.73 1 0.88
1
0.00 0.43 0.01
0.11
0.00 0.07 0.00
CEM [57]
0.66 0.02 0.69 = 0.03 0.82 + 0.88
+
0.03
0.47
0.01
0.40
0.01
0.57 -
HardCBM [17]
0.56 0.03 0.61 + 0.01 0.68 + 0.01 0.89 + 0.00
0.92 + 0.02
0.46 0.01
0.12
0.00
0.06
0.00
Temporal Cansal3DIdent Dataset
Linear 0.74 = 0.06 0.75 = 0.06 0.80 + 0.04 0.81 = 0.04
0.69 0.02 0.44 0.01
0.19 = 0.00
0.16 - 0.00
CITRISVAR
Spline 0.74 0.06 0.80 :: 0.04 0.82 1: 0.03 0.83 + 0.03
0.69 0:02 0.01 0.13 0.00
0.00
RFF 0.70 0.06 0.76 0.5 0.79
0.81 +
0.65 0.01
0.43
0.16
0.13
Linear
0.74 0.06 0.76 1: 0.78
0.80 1
0.69 0.02 0.44 0:01 0.17 0.00
0,14 0.00
IVAE
Spline
0.73 0.06 0.78 0.04 0.79
0.04 0.81 + 0.04
0.02
0.44 0.17
=00
0.13 0.00
RFF
0.69 = 0.06 0.75 = 0.05 0.78 0.04 0.80
+ 0.04
0.65 0.01 0.42 = 0,02 0.16 0.00
0.13 + 0.00
CBM [28]
0.57 = 0.02 0.53 = 0.03 0.68
: 0.04 0.78
+ 0.05
100 0.48 0.02 0.25 0.00
0.18 + 0.00
CEM [57]
0.64 = 0.04 0.67 = 0.03 0,72 0.05 0.78
+ 0.05
0.97 0.05 0.51
= 0.02 0.36
0.01
0.49 0.01
HardCBM [17]
0.57 a: 0.05 0.53 1: 0.02 0.63 1: 0.03 0.77 1 0.05
0.96 1 0.03
0.47 1: 0.02 0.24 0.00
0.16 1 0.00
shape is a categorical variable. We use a pretrained CITRIS-VAL VAE encoder, which outputs 32 dimen- sional latent space and a grouping of which dimensions relate to which causal variables. Although CITRIS-VAE provides the correct permutation of the groups, we ignore it and perform random permutation on the variables. We train an MLP for each of the 32 dimensions that predicts all causal variables. Based on the R scores of these regressions, we learn the group assignments.
Results. We show a set of representative results in Fig. 2 for the Toy Dataset, where we compare as baselines to the permutations one can learn with Pearson or Spearman correlation, and linear sum assignment. Our estimator is able to reconstruct the correct permutation perfectly with only small number of features. performs well for a broad range of regularization parameters. In the wellspecified case we calculate the Å parameter and see that the estimator performs well around this value. The dimension dependence is almost negligible, which was predicted by the dimension appearing only in the log factor in Thm. 4.2. The estimator works well with few data points and even better than theory predicts, as it has a low error even with high correlation, as shown Fig. 2. Spearman correlation also performs well in some simple settings, which suggests that the relation is indeed monotonic in these settings. On the other hand, there are no theoretical guarantees for this baseline and it cannot be used for multi-dimeesionall representations, for which our estimator still provides a principled approach. We report results with similar trends for RFF and the kernelized approach in App. 1. To test the robustness of our estimation to the case in which CRL methods do not provide a complete, but only partial disentanglement, we also show an ablation where we provide mixtures of pairs of causal variables input to algorithm, showing that it still recovers the correct permutation for each pair. Similar trends apply also to the image dataset, as reported in App. E.
A selection of results for the downstream classification task for the action sparsity dataset and temporal Causal3DIdent dataset are in Tab. 1. Our estimator consistently scores high in terms of Label Acc. and OIS, while requiring less labels. For image data, our estimator beats the baseline for every number of datapoints. In App. E.2 we report the complete results, which show that the baselines perform well with regards to the mean concept accuracy in the Action/Temport sparsity datasets, but our estimator still performs better with the smallest number of labels and with less computation power to calculate. Our estimators consistently achieve a perfect NIS [58] score, which measures the impurities distributed across the concept representations.
Finally, in Fig. 3 we show the execution times of our estimator compared to several baseline methods. We report the times in both the continuous experiments and binary classification experiments conducted with the Temporal Causal3DIdent dataset. The baseline in the continuous setting is given

---

Linere
Spline
REF
Luplacian
NN
Liner
RFP
CRM CRM
CRM AR
Lapiacinn
10a
10y
10a
Exocution
10a
109
10-1
10a
10
data points (w)
points
[11
Figure 3: Execution times for our estimators and several baseline models on the Temporal Causal3Dident dataset. Left: The causal variables are continuous-valued. The baseline is given by training several neural networks and the matching based on the RC-scores. We report the time needed to estimate the matching and train the neural networks. Right: The causal variables are binarized and a downstream classification task is added. We report the time needed to estimate the matching and learn the classification task the time needed to train the concept-based models.
by training a neural network for each causal variable. Each neural network learns how to predict the causal variable from the learned encodings, and the Rt"-scores used to construct matching. The time that is reported is the time that is needed to estimate the matching and the training, if needed. In the binary classification experiments, the baselines are given by the CBM. CEM, and HardCBM models. The time that is reported is the time needed to either estimate the matching and learn the classification or to train one of the concept-based models. We see that all versions of our estimator require significantly less computation time than the baselines.
7 Conclusions and Discussion
We propose a framework that provides thcoretical guarantees on learning of concepts by leveraging causal representation learning (CRL). We provide two estimators that able to learn the alignment between the learned representations and the concepts: a linear estimator with finite sample guarantees and nan-amaaettiic kernelized estimator with asymptotic guarantees. We test our methods on CRL benchmarks and show they perform even better than the theory predicted.
While our work proposes promising research direction, several limitations remain. For example, we assume that human concepts coincide with the causal variables recovered by the CRL methods, which limits the framework's ' applicability. While our framework can also consider settings in which only blocks of causal variables are identifiable, as opposed to each individual variable, then the alignment can be only learned between these blocks and the corresponding blocks of concepts. To generalize framework to different levels of coarse- coarse-grained human concepts, it would be interesting to incorporate ideas from causal abstraction [47, 13, 7] to extend this analysis cases in which human concepts are abstractions of the underlying causal system. Moreover, while in the current framework we focus on CRL methods in order get theoretical guarantees the identifiability of latent variables from high-dimensional observations, we do not use any causal semantics, so in principle our framework could be applied also other types of methods that provide similar theoretical guarantees.
theoretical limitation of our analysis is the assumption low correlation between the learned representations. In our experiments find that our methods still work with significantly stronger correlations than assumed in our thcoretical results, so we believe there may be room here to strengthen the theory. This might also allow us relax assumption that CRL methods achieve perfect disentanglement although consider discrete concepts in evaluation, do not provide theoretical analysis for them, because current CRL methods cannot identify discrete variables, or can only do so specific settings [29]. A possible way to extend our theoretical results this setting would be to consider previous results for the Group Lasso for logistic regression [43].
the causal variables, which is not realistic in many domains. Finally,
10

---

Acknowledgments and Disclosure of Funding
We thank SURFsara for the support in using the Snellius Compute Cluster and Robert Jan Schlimbach for the technical support in parallelizing most of the experiments. Van Erven was supported by the Netherlands Organization for Scientific Research (NWO) under grant numbers VI. 1idi. and 61w.///....166111/11// 0tt.:////000.11616/////0.05.....
References
[1] Acartürk, E., Vanieı, B., Shanmugam, K., and Tajer, A. (2024). Sample complexity of inter- ventional causal representation learning. Advances Neural Information Processing Systems, NeurIPS.
[2] Ahuja, K., Mahajan, D., Wang, Y., and Bengio, Y. (2023). Interventional causal representation learning. In International Conference on Machine Learning, ICML, Proceedings Machine Learning Research. PMLR.
[3] Alain, G. and Bengio, (2017). Understanding intermediate layers using linear classifier probes. In International Conference on Learning Representations, ICLR.
[4] Bach, F. R. (2008). Consistency of the group lasso and multiple kernel learning. Journal of Machine Learning Research, ((40):1179-13)
[5] Bach, F. R., Jenatton, R., Mairal, J., and Obozinski, G. (2012). Optimization with sparsity- inducing penalties. Foundations and Trends in Machine Learning, pages 1-106.
[6] Baker, C. R. (1973). Joint measures and cross-covariance operators. Transactions of the American Mathematical Society.
[7] Beckers, S., Eberhardt, and Halpern, Y. (2020). Approximate causal abstractions. In Uncertainty in Artificial Intelligence UAI. PMLR.
[8] Belinkov, Y. (2022). Probing classifiers: Promises, shortcomings, and advances. Computational Linguistics.
[9] Bengio, Y., Mindermann, S., Privitera, D., Besiroglu, T., Bommasani, R., Casper, S., Choi, Y., Fox, P., Garfinkel, B., Goldfarb, D., Heidari, H., Ho, A., Kapoor, S., Khalathari, L., al. (2025). International ai safety report.
(10) Bühlmann, P. and Van De Geer, S. (2011). Statistics for high-dimensional data: methods, theory and applications. Springer Science & Business Media.
[11] Chen, R. T,, Li, X., Grosse, R. B., and Duvenaud, D. K. (2018). Isolating sources of dis- entanglement in variational autoencoders. Advances in neural information processing systems, NeurlPS.
[12] Gallego-Posada, J.. Ramirez, J., and Erraqabi, A. (2021). Flexible learning of sparse neural networks via constrained Top regularization. In NeurlPS 202/ Workshop LatinX Al.
[13] Geiger, A., Lu, H., Icard, T., and Potts, C. (2021). Causal abstractions of neural networks.
[14] Ghorbani, A., Wexler, J., Zou, J. Y., and Kim, B. (2019). Towards automatic concept-based explanations. Advances in Neural Information Processing Systems, NeurIPS.
[15] Goyal, Y., Feder, A., Shalit, U., and Kim, B. (2019). Explaining classifiers with causal concept effect (CaCE). arXiv preprint
[16] Graziani, M., Nguyen, A.-p., O' Mahony, I.,, Müller, H., and Andrearczyk, V. (2023). Concept discovery and dataset exploration with singular value decomposition. In ICLR 2023 Workshop on Pitfalls of limited data and computation for Trustworthy ML.
[17] Havasi, M., Parbhoo, S., and Doshi- Velei, F. (2022). Addressing leakage concept bottleneck models. In Advances in Neural Information Processing Systems, NeurIPS.

---

[18] Hofmann, T., Schölkopf, B., and Smola, J. (2008). Kernel methods in machine learning, The Annals of Statisties.
[19] Hyvärinen, A., Khemakhem, I., and Monti, R. P. (2024). Identifiability of latent-variable and structural-equation models: from linear nonlinear. Annals of the Institute of Statistical Mathematics.
[20] Hyvärinen, A., Sasaki, H., and Turnrr, R. E. (2019). Nonlinear ICA using auxiliary variables and generalized contrastive learning. In International Conference on Artificial Intelligence and Statistics, AISTATS, Proceedings of Machine Learning Research, PMLR.
[21] Ismail, A. A., Adebayo, J., Bravo, H. C., Ra, S., and Cho, K. (2023). Concept bottleneck generative models. In International Conference on Learning Representations, ICLR.
[22] Jang, E., Gu, S., and Poole, B. (2017). Categorical reparameterization with gumbel-softmax. In International Conference on Learning Representations, ICLR.
[23] Khajenezhad, A., Madani, H., and Beigy, H. (2021). Masked autoencoder for distribution estimation on small structured data sets. IEEE Transactions Neural Networks Learning Systems.
[24] Khemakhem. I., Kingma, D. P., Monti, R. P., and Hyvärinen, A. (2020). Variational autoencoders and nonlinear ICA: A unifying framework. In International Conference on Artificial Intelligncce and Statistics, AISTATS, Proceedings of Machine Learning Research. PMLR.
[25] Kim, B., Wattenberg, M., Gilmer, J., Cai, C., Wexler, J., Viegas, F., et al. (2018). Interpretability beyond feature attribution: Quantitative testing with concept activation vectors (TCAV). In Inter- national Conference on Machine Learning, ICML, Proceedings of Machine Learning Research. PMLR.
[26] Kim, E., Jung, D., Park, S., Kim, S., and Yoon, S. (2023). Probabilistic concept bottleneck models. arXiv preprint arX1::2306.0157.
[27] Kingma, D. P. and Ba, J. (2015). Adam: A method for stochastic optimization. In International Conference on Learning Representations, ICLR.
[28] Koh, P. W., Nguyen, T., Tang, Y. Mussmann, S., Pierson, E.,, Kim, B., and Liang, P. (2020). Concept bottleneck models. In International Conference on Machine Learning, ICML., Proceedings of Machine Learning Research. PMLR.
[29] Kong, L., Chen, G., Huang. B., Xing, E., Chi, Y., and Zhang, K. (2024). Learning discrete concepts in latent hierarchical models. In Advances in Neural Information Processing Systems, NeurIPS.
[30] Lachapelle, S., López, P. R., Sharma, Y., Everett, K., Priol, R. L.,, Lacoste, A., and Lacoste- Julien, S. (2024). Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, Niv:22001004900
interventions and sparse temporal dependencies. arXiv preprint
[31] Lachapelle, S., Rodriguez, P., Sharma, Y. Everett, K. E., PRIOL, R. L., Lacoste, A., and Lacoste- Julien, S. (2022). Disentanglement via mechanism sparsity regularization: A new principle for nonlinear ICA. In First Conference on Causal Learning and Reasoning.
[32] Leemann, T., Kirchhof, M., Rong, Y., Kasneci, E.,, and Kasneci, G. (2023). When are post-hoc conceptual explanations identifiable? In Uncertainty in Artificial Intelligence, UAL
[33] Lippe, P., Magliacane, S., Löwe, S., Asano, M., Cohen, T., and Gavves, S. (2022). CITRIS: causal identifiability from temporal intervened sequences. In International Conference on Machine Learning, ICML, Proceedings of Machine Learning Research. PMLR.
[34] Lounici, K., Pontil, M., Tsybakov, A. B., and van de Geer, S. A. (2009). Taking advantage of sparsity in multi-task learning. In Conference Learning Theory, COLT.
[35] Lounici, K., Pontil, M., Van De Geer, S., and Tsybakov, A. B. (2011). Oracle inequalities and optimal inference under group sparsity. The Annals of Statistics.
12

---

[36] Lovering, C. and Pavlick, E. (2022). Unit testing for concepts in neural networks. Transactions of the Association for Computational Linguisties.
[37] Maddison, C. J., Mnih, A., and Teh, Y. W. (2017). The concrete distribution: A continuous relaxation of discrete random variables. In International Conference on Learning Representations, ICLR.
[38] Mahinpei, A., Clark, J., Lage, I., Doshi-Velez, F., and Pan, W. (2021). Promises and pitfalls of black-box concept learning models. arXiv preprint arXiv:2106. 13314.
[39] Marconato, E., Passerini, A., and Teso, S. (2022). Glancenets: Interpretable, leak-proof concept-based models. In Neural Information Processing Systems. NeurIPS.
[40] Marconato, E., Passerini, A., and Teso, S. (2023). Interpretability is in the mind of the beholder: causal framework for cuman-interpretable representation learning. Entropy.
[41] Margeloiu, A., Ashman, M., Bhatt, U., Chen, Y., Jamnik, M., and Weller, A. (2021). Do concept bottleneck models learn as intended? arXiv preprint arXiv:2106. 13314.
[42] McGrath, T., Kapishnikov, A., Tomašev, N., Pearce, A., Wattenberg, M., Hassabis, D., Kim, B., Paquet, U., and Kramnik, V. (2022). Acquisition of chess knowledge in alphazero. Proceedings of the National Academy of Sciences.
[43] Meier, L.,, Van De Geer, S., and Bühlmann, P. (2008). The group lasso for logistic regression. Journal of the Royal Statistical Society Series B: Statistical Methodology.
[44] Molnar, C. (2022), Interpretable Machine Learning. 2 edition.
[45] Oikarinen, T., Das, S., Nguyen, L. M., and Weng, T.-W. (2023). Label-free concept bottleneck models. In International Conference on Learning Representations, ICLR.
[46] Rajendran, G., Buchholz, S., Aragam, B., Schölkopf, B., and Ravikumar, P. (2024). From causal to concept-basee based representation learning. In Advances in Neural Information Processing Systems, NeurIPS.
[47] Rubenstein, P. K., Weichwald, S., Bongers, S., Mooij, J. M., Janzing, D., Grosse Grosse-Wentrup. M., and Schölkopf, B. (2017). Causal consistency of struetural equation models. In Conference on Uncertainty Artificial Intelligence, UAI. AUAI Press.
[48] Schölkopf, B., Herbrich, R., and Smola, A. J. (2001). A generalized representer theorem. In International conference on computational learning theory. Springer.
[49] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., and Bengio, Y. (2021). Toward causal representation learning. Proceedings of the IEEE.
[50] Sheth, I. and Kahou, S. E. (2023). Auxiliary losses for learning generalizable concept-based based models. In Advances in Neural Information Processing Systems, NeurIPS.
[51] von Kügelgen, J., Sharma, Y., Gresele, L., Brendel, W., Schölkopf, B., Besserve, M., and Locatello, F. (2021). Self-supervised learning with data augmentations provably isolates content from style. In Advances in Neural Information Processing Systems, NeurIPS.
[52] Williams, C. and Seeger, M. (2000). Using the nyström method speed up kernel machines. In Advances in Neural Information Processing Systems, NeurlPS, volume 13. MIT Press.
[53] Xu, X., Qin, Y., Mi, L., Wang, H., and Li, X. (2024). Energy-based concept bottleneck models: Unifying prediction, concept intervention, and probabilistic interpretations. In International Conference on Learning Representations, ICLR.
[54] Yang, Y., Panagopoulou, A., Zhou, S., Jin, D., Callison-Burch, C., and Yatskar, M. (2023). Language in bottle: Language model guided concept bottlenecks for interpretable image classifi- cation. In Proceedings of the IEFE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 19187-19197.
13

---

[55] Yao, D., Xu, D., Lachapelle, S., Magliacane, S., Taslakian, P., Martius, G., von Kügelgen, J.. and Locatello, (2024). Multi-view causal representation learning with partial observability. In International Conference on Learning Representations, ICLR.
[56] Yuan, M. and Lin, Y. (2006). Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society Series B: Statistical Methodooogy.
[57] Zarlenga, M. E., Barbiero, P., Ciravegna, G., Marra, G., Giannini, F., Diligenti, M., Shams, Z., Precioso, F., Melacci, S., Weller, A., et al. (2022). Concept embedding models: beyond the accuracy-expaannablii trade-off. In Advances in Neural Information Processing Systems, NeurIPS.
[58] Zarlenga, M. E., Barbiero, P., Shams, Z., Kazhdan, D., Bhatt, U., Weller, A., and Jamnik, M. (2023). Towards robust metrics for concept representation evaluation. In Proceedings of the AAAI Conference on Artificial Intelligence.
[59] Zimmermann, S., Sharma, Y., Schneider, S., Bethge, M., and Brendel, W. (2021). Contrastive learning inverts the data generating process. In International Conference Machine Learning, ICML, Proceedings of Machine Learning Research. PMLR.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All claims made the abstract and introduction either have formal proofs or empirical evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The limitations of the theoretical claims are discussed by pointing out which assumptions are realistic and which assumptions do not appear be as important through empirical evidence. The limitations of the empirical experiments in terms of how realistic they are is also discussed.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
Each theoretical result has proof and its assumptions clearly stated
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: For all experiments, the settings, architectures and parameters are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code will be uploaded with the Supplemental Materials. The data is either generated or publicly available and instructions to access it are provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: /ttcps://nii /ttp///ii../ee/eee//// cc/
publ a / ggggudde quides/// / Codeeuuom s.s oooooooolio lyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / /nips. cc/ publi he/guiides Code Submi ss i conPllccy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
All the specific settings are provided the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Each experiment is repeated 10 50 times, with different seeds, and report the mean and the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute hardware is reported for the experiments and timing indications are given as well.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /tw/riuripppp cc
Answer: [Yes]
Justification: In this paper, we offer theoretical framework and new estimator to create concept- based models with fewer concepts labels. As such, no human subjects were involved, all data is either generated or publicly available, and no mitigating measures are needed. The negative societal impact are minimal. Thus, we feel that we conform to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
18

---

Justification:
The potential use of our framework to build understandable and interpretable concept based models is discussed positive consequence our work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper offers framework to build new concept based models and intro- duces a new statistical estimator. Both these pose no significant risk to society.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators of the code and datasets used in this paper are properly credited, and the licenses will be reported in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswa wwteerswwtccdeee com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Aside from our framework and the estimator proposed in this paper, no new assets are introduced.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper doesn't 't involve any LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:: //l::///neee //nurriip wttsscc/oooooeeeeeeeeeeeeeeeeeee /2tnnncceeees2// 2025/ LLM) for what should should not be described.
21

---

A Proofs for Linear Regression Alignment Learning
In this section we provide proof details and the algorithm for Section 4. As discussed there, we extend Theorem 4.2 to more general result Theorem A.5 that allows for blocks of causal variables corresponding to a single concept. To state the general result we will redefine our model and introduce additional notation. The variables C₁ I and Mj E TR now live in potentially different spaces.
Let C = -C (Ci,..., Ca) E Id and = (M1...., Ma) Rt, where = Σÿ=ık ky 2 d. Let Kj- = - kip + blb (1...., }j}} denote the subset of indices M that correspond to the block Mj. The permutation that we want to recover is и. Each dimension in the M variable can be transformed through a separate feature map fo:: R -> TRPt that can be different for each I,....
We will denote the total feature vector by
((M) = (P(
(Mx))
The grouped features will be denoted by ((M) (M. | K)). Define the average feature set size as p = Ei=1 Pt. The model described by
C = ((M) + E B E (0.
For the actual regression task, we can define data matrices again. The matrix will be defined as in the main and & now becomes With E; denoting n independently draw N (0, σ²) variables, this results in the relation
kp matrix, in which all feature vectors sp( (((()) are stacked.
Ei. -
The 3 again has sparse structure, because only the parameters corresponding to ((M) should be non-zero. Let the indices of these parameters be denoted by Gy. Alternatively, this G, is defined through yp( (MM), = ( (MM). Thus, in this setting we again have groups all denoted by Gj. To ease notation we again B ((8e), / E Gý). The definitions of the norm ||- 1||2,00 and covariance matrices, En = (MM) ((M)G _ by, are altered in accordance with these groups As the groups can now be of different size, we have to change the definition of the || Ill.i-norm to take the different group sizes into account,
d
1
Σ 1
where p² = -IEK, Pt. The loss function that we want to optimize to estimate 3r, has the same form as before
8 arg min I||C 2.311 AllB|12,1 arg min I|C,C 1.31 + BER
Σ | 1VV... (10)
3ER
Algorithm Estimating the permutation using linear regression with Group Lasso regularization
Input: regularization parameter À
2: Data: {(C(0) (9)
3: for = 1,....d d do
4:
B + arg min C. +. + > SER
5: end for
(1
6: क +- arg max
22

---

The optimality conditions for any solution, B,, for this convex optimization problem are given by
[ -3,3) AvP p
B
if 0 + O,
(11)
2
||
A 2
# $ (C, - 32)1 -
if
8
(12)
To ensure that our results hold even in the case where n < pd, we introduce a standard assumption on the data from the high-dimensional statistics literature. Intuitively, this assumption ensures that the data is "variable enough" in the directions that matter.
Assumption A.I. The Restricted Eigen Value (RE(1)) satisfied by the data matrix & Ruxkp
if there exists a > 0 such that for all A R& {0} and j L...., d with Σ | S
3 | V pd it holds that
| AJ
12 k.
This property is satisfied for any A E R (0) if E = 1 Φ + has positive minimal eigenvalue.
Let Amin > 0 be the minimal eigenvalue of E. then
| 1 A T 4Λ = nA ΣΛ 2 n A A n1min ||
Now divide by and take the square root on both sides. This gives
| vn
Amin
Amin
All
v
Amin
Σ A
min
|
The second inequality follows from an application of Jensen's 's inequality. Dividing both sides by ||Al||
|| gives the desired result.
The matrix Σ is the empirical covariance matrix and will be positive definite almost surely whenever kp and hence RE(1) will be satisfied if n 2 kp.
Finally, define Pmin = minn==1...dl and Pmax = max ma...1 p² The following theorems and proofs adapted from Chapter 8 in Bühlmann and Van De Geer [10] and sections and 5 in Lounici et al. [35].
Theorem A.2. Assume that for all l 1,...... n, e. ~ NO, o²) independently, a² > O, the RE(1)
condition is satisfied with K >> O and consider the Group Lasso estimator
B. arg min I1 - 1 + A|18112,1,
RER
where X 2 4Ag with
2a
8 (og((//)
8 log(d/))
Ao
Vn
Pmin
Puin
Then, for any 8 E (0, 1). with probability at least 1
8
4'
2=12
1 n ||. 3/ + x113 - 11112,1 S
(13)
x²2
|55(3 8'1
Av n
for all j I....,d
(14)
7()
|| 31112,1 S
(15)
R²
This theorem offers us several things. Equation 13 gives us a bound on the true prediction error. The last two equations, (14, 15), are needed to prove that we find accurate parameter values using the Group Lasso approach. The fact that the last equation gives a bound in the (2, 1 1o-mmm allows us to use duality argument later on to provide a bound on the (2, nnorm of the difference between the learned and true parameter. Knowing that only one of the groups has to be non-zero combined with this uniform bound enables us to conclude that the correct group has been identified in the proof of Theorem A.5.
23

---

Proof. First let define for every j = ,,..., d the random events A, 1 E اول and their intersection A = Nブ=1 j= Aj. Most importantly, we see from Lemma C.2 that this event has probability at least z. We get the 1/d factor by using 5 8= 6 in Lemma C.2 and noticing that this d
only adds factor of 2 in the log terms. The first assertion (13) is true the event A and follows from the proof of Theorem 8.1 in [10] and noting that in our setting their oracle parameter is given by 3 and that fo T
Moving on towards (14), by the optimality condition (11) and (12) we have for each j = ......e. d
[ - +,311 v Avpl
Let us rewrite the expression (14) into
|| - 3 #1 1 (43 1)13)>111
Substituting $ = C; into this expression gives
[|({(B - S |I|( ( (0 c.)) ( EI
AVP AVP =>P= .VP. 2
2
Note that this inequality only holds on A.
The final assertion is direct consequence of the first
24)2p=(e)
1| 37112,1 S =11 -,)1 + AII 321 S
x2
||3, 31112,1 24
R²
To state and prove the general version of Theorem 4.2 we also need to generalize Assumption 4.1. Assumption A.3. There exists some constant > 1 such that for any * j'. it holds that
max 1stSmin(ye) yort (5ifg) )ael S
Pmin
(16)
14a V
Pmax
and
(Ezz) Yeal S
Pmin
max IStEN,Ige Ait
(17)
14a
Pmax Vpip"
The previous assumption is stronger than the RE(1) property, as shown by the following lemma: Lemma A.4. Let Assumption A.3 be satisfied. Then RE(1) is satisfied with k = V 1/a.
Proof. This Lemma B.3 [35].
The following theorem a modification of Theorem 5.1 by Lounici al. [35], where some adaptations are made to adjust the result to our setting.
Theorem A.5. Let Assumption (A.3) hold, for Ł = 1,...,d, of _N((0 independently. o² > 0, and with 8 E (0,1) set 2 42o, where
2x
Blog(d/0)
8 log(d/0)
Ao
Pmin
Pmin
Furthermore, set c
24 7(a
Then, for any E 1), with probability at least 1
8 any
1)
solution 3, of (10) satisfies
113; 33 12,00 I cd cPaxp
(18)
24

---

If. in addition, (33))=(0)| > 2cd 2c> Pumxx. then (18) implies that |
J, = arg max
;=1...,d
estimates (i) correctly.
Proof. Most of the proof is similar to the proof of Theorem 5.1 in Lounici al. [35]. We supply full proof for completeness and because setting is slightly different. We will need more notation to prove this statement. Set Pae maxis <d p³ and define the extended covariance matrices Σny of
size Peo x Poo as
Σ
0
if j and Σوg
pp xp
if j j'
We also define for any = 1.....d d and A E R the vector A E Re such that
A
A
Now set A = 33 B and bound
|| 1| (E- (xpxkpp ) |EAl12.00 + +I(
<pp All2.o)
The first term is controlled by (14) from Lemma A.2. The latter term can be bounded by noticing that only the off-diagonal elements will contribute to the norm. We can bound using Cauchy- -chwarz:: ||(S
TE
2,.00 $=1...,d
max |1((S- [KEXRF)
1/2
1/2
Σ
Eg Da Dz
max jmm yood
Pmleml
1/2
27 1/2
< max j=
Eg ) D!
max j=
Σ
E )ae A
We now bound both terms separately. The first term can be bounded using an application of Assumption A.3 and then Minkowski's inequality. The Minkowski's inequality is true for L norms and tells
| + +upp S |001p + llvllp
In our case this generalises to
1/2
d
d
A
Σ
|
p
A
1
2.11
Pmii
v Punin
t=1
Ø=1
j=1
==1
Combining Assumption A.3 with the above inequality gives us
27
1/2
27 1/2
d Σ
Psc
Pmin
Σ
A
max j= j=ly..d
14a
Pmax
5==
3'
Pmin
12,1
14a V
Pmax
Pmin
||
1
14a V
Pmax
25

---

The second term can now be bounded by another application of Schwarz:
1/2
Σ
(g DH A..
max j=l,odd
10=1,00
1/2
1
Pmin
A
14a
Praxx y= b....dd
max
P
Vp" p
1
Pmin
d
A
14a
Punxx
11=1==1
1
Pmin
1
12,1
14a
V
Pmnxx
Pmin
1
A
14a
Pmax
The (2, 1)-morm term is now bounded using (15). Putting everything together get
1 - 57 12,000 $ ||5(
122 + +1((
(3) 11,
2
24
< X Pmax +
14a
Pmax
x
< 1 1+
A/Pmaxx
7k
To satisfy both assumptions (A.1, 4.1), we need to set ax² = (a as per Lemma A.4.
Finally, to prove the final claim, note that (18) combined with sparsity assumption on the true parameters implies that for j' * (i) must be that 1| || = 3 - (3;)²1 e We
will show that for =(i) it must be that [| cd Pmax- Hence, the estimator gets the correct index with high probability. Indeed, |[())=(00] (1) > 2cA /Pmax we get
( || (((8))(()) (+)) ((87)a()) 3- 2 (((8))=(001 - (1((B))x()) )1))
> 2cd Pinxx
V cd Pimax
= cd Paax-
Let us restate the specific version of Theorem 4.2 again for clarity. This theorem is now corollary of Theorem A.5.
Theorem 4.2. Suppose the data have been pre-processed to satisfy (4) and let Assump. (4.1) hold. Take any E (0, 1) and set X 2 4Ag, where
20
8 (og(d/5))
(lg((/8))
Ao
1
vn
P
p
and set (1+ T[-11))
24
Then, any solution B, of the Group Lasso objective (3) satisfies
11 8 12,000 S ScAVP
(5)
with probability at least 1 If. in addition, |1(3))0(01| > 2c>/p, then (5) implies that J. arg max;=1..... [|3] estimates at mi) correctly.
26

---

Proof. The result follows from Theorem A.5, where in this case & d. and p = for all j 1....,d. d.
Corollary A.6. Assume the same setting Theorem 4.2 such that for each 1..... d. (1,)))(() > 2cx/p and consider the estimator fn as defined in (6). Then ß with probability least 6.
Proof. Consider the following estimators
8 arg min
||C
T.311 A/Pl18
BE
J, arg max j
|
::
id J4.
We will first show that गे estimates ग with probability at least 1 - 6. Afterwards, we will show that the event on which is correct, is contained in the event that स estimates x correctly, implying lower bound on the requested probability.
We apply a union bound
T(==x) P(vi ,............... d... x())
= =1 P(3: = 1
d x(1)
> 1 d
s=1 = =1-4 8.
We proceed the second step. If गे estimates correctly, then स. is already valid permutation and = Indeed, if गे is correct then that means that || = [| the maximum norm for each i. Coincidentally, by π being correct permutation, मे describes correct matching with largest values, which means that (1) = (i) for each = l.....d and
iP π PP 1-
B Proofs and Implementation Details for Kernelized Alignment Learning
In this section we provide proof details and the algorithm for Section 5. We will make one adjustment to the Group Lasso regularization in the optimization problem compared to (7), which is that we square regularization term. This form theoretically more appealing, but is still equivalent to
Algorithm 2 Estimating the permutation using kernels
1: Input: reg. parameter À > 0, kernels K)
2: Data: {(C(() (9)]
3: for j=1 j= d do
4:
(Ky) < Ky (M
(M), M())
5: Lj t #=ChotssyDeccompsttionnkk
6: end for
7: for i
d do
8: 24 arg min RER"p
IC,
Σ j=1
9: end for
+- arg max
27

---

standard formulation: as Bach [4] argues, the two versions of the optimization problem have the same sets of solutions when varying the regularization parameters. For μ > 0, the objective given by
d C| - (( + l Σ
inf fdd
11 11
(19)
Let , 3 be the solutions of the above optimization problem. The translation between regular- ization parameters that give the same solutions for (7) and (19) is given by À = 1 (Σn 11HH)
B.1 Representer Theorem
The squared version of the optimization problem allows to prove the Representer theorem from the main text:
Theorem 5.1. Let f1,...,pP ,d be the feature maps associated with Hi...., Hd. Then there exist e....., Cf E IR" such that the optimization problem in (7) has solution 3: with each 3i of the form
B
(M, )(())e
Proof. First we state the following result about a variational equality for positive numbers
||
Σ
1
inf MEAR
(20)
proof of this statement can be found in section 1.5 of [5]. Using (20) and switching the squared version of (7) we rewrite (19) as
inf - |C, 3(M) - 1
11,39 1|
9 Ba.....a n
|| ||4
inf inf 1 |IC, ((M] + /4 neda Pa... n
?
(OPT)).
We can rewrite this expression further, using the reproducing property of the RKHSs Hj, which gives (M,,) = (37,,,((M,)) Furthermore, defining 3i and
P2 =
T we rewrite
|
(OPT;) inf inf C
B (1,
yeAy 91.....aa
n
E=1
d Σ
d +
= inf
inf
c((0
Vn bo (M
B
11
mEAd B1.....f Te
E=1
==1
inf
inf
c(())
Σ
Br
1.39 1244
yEAd Bi.....
n
t=1
= (OPT2).
28

---

This final expression should be recognized as the feature representation of the Representer theorem [48] applied to the kernel described by
x(n)(M, M'M, Σ (F(M))
Σ Ti (1)(M),)
Σ (M,, Mj).
==1
The Representer theorem then gives us that the solution of the inner optimization problem in (OPT2) can be described by
33
Bi Σ (M)) = 8=1
VTB Σ (M) = B 8'= 8=1 1=1
",, (n)())
Th
with R" and 7 E Ad. Alternatively interpreted this says that there exist c'...., ed R" such that 30 = Ei= (M," (c)),. We invoke again the equivalence between the squared and un-squared versions of the optimization problem using the translation of regularization parameters - - j l11 and conclude that the solutions of (7) are of the same form.
To get the finite-dimension optimization problem stated in (8) substitute the correct forms of B back into the original optimization problem. Define the Gramm matrices (K₃)eж ky k,(M) Mb}) (M) <M)(M))) and observe
inf i Bq.....a
c
Σ
+
1
|
l|x,
j=1
j=1
inf e dR.. n
cl1l)
E(KYo())
1
(o
j=1
d
inf dh...rre Enn
I.C. - Σ k2dd2 + Σ
|ella,,
5=1
j=1
B.2 Estimator consistency
As stated in the main text, the assumptions in Theorem 5.2 are explained in this section. The assumptions stated in (A-D) ensure that the RKHSs that we work with are nice enough and that the function we want to estimate is not too miss specified. For more complete discussion on the assumptions, we refer Bach [4]. To remind ourselves, we are given random variables M = (MI...., Ma). where each random variable lives in Zj. and d RKHSs Hi,..., Ha associated with d kernels KI,....Ky. The cross-covariance operator, Σij for H, to H, is defined such that for all (8) E H4 Hl..
(8'E)18)) = (M.) - ElS (M.) (M2)).
(21)
The bounded correlation operators Pij defined through the decomposition Eij Ell Pis E3 [6].
(A) For each j 1, ...... ... associated with kernel K, and the random variables k, M,)) are not constant and have finite fourth-order moments.
the Hilbert space H. is separable reproducing kernel Hilbert space
(B) For all i.j 1,.... d, the cross correlation operators are compact Psy and the joint correla- tion operator is invertible.
29

---

(C) For each i = 1....,d, d, there exist functions 57 ....,,"" E , Ha. by R and function f. M such that
C Σ (M,) + Eis
where Els; M = 0 and min < Els? M] < m with B[Si(M)?) < oc, [M)] -
and [[f (.a))) (My) = 0] for all j = 1.... d. We define (i) to be the one index for which 87 () + 0.
(D) For all i.j ,,...,de there exists of E Hj such that 87 = D1/2 g.
For each function, (()()) hat is non-zero we will require the following condition
E1/² PPato Dgx(o)
max (+
H4 <1,
(22)
Where is a block diagonal operator where each block consists of the operators | 11 IH
Condition (B) can be seen as an analogue the correlation assumption Assumptions 4.1 and A.3, as it ensures that the variables are not too dependent.
Before we prove Theorem 5.2, we will first prove that each individual index mii) can be estimated consistently. This follows from an asymptotic result by Bach [4].
Theorem 5.2. Assume (A-D) in App. B.2. Then, for any sequence of regularization parameters An such that An + 0 and VnAnn + 400 when n + 0o, the estimated permutation के converges in probability to æ.
Proof. To prove this result first define the estimator of each individual index (i)) as J arg max
[| 1|,,
(23)
j=1.....d
Theorem 11 in [4] gives consistency for the estimated parameters 3/ and estimated index Jr. However, his result is stated for the squared version of the Group Lasso and has assumption that the pnn regularization parameters have the property that μ4, - oo and VTiue + +oo as the number of data points n + co. The translation factor between regularization parameters A, = Mn (E/ j=1 |1 ll,,) convergence to a constant in probability, by the consistancy of the estimated parameters. This shows that the scalings for \n and μín are the same asymptotically. We conclude that J, estimates t(i) with probability tending to
Remember that the norms of 8 and é are the same through 11,/ = |l@,ll,.. This means that we have consistency for estimators of (i) that are based on || [K, as well. For each i = .,...,d we can repeat the above argumentation to get consistency for each Jr separately. To combine the conclusions, we apply the same argument as in the finite dimensional case and a union bound that finishes our proof,
P(# = π) 2 P(3i = 1....,d
dd x())
2 1 - Σ P(J. # =(()) +
B.3 Implementation
Lemma B.I. For each = = dy...,dlu let Lj be the Cholesky decomposition of the Gramm matrix Kj,
then 2 = (Z,5) (Z.5)-14 -11g if the parameters "/.... 2d are minimizers of
min 21
I|
Σ
1, 111-2112.11
1=1
30

---

Proof. Applying the Cholesky decomposition Kj = L for each j = 1,...,d .. and substituting this into (9) gives
inf ERER
I|
Σ
, +
|
inf c ER
I| Σ
A
1
elll
j=1
j=1
j=1
inf ? 7 ER
1 C
Σ j=1
1 x
Σ |
The number of parameters now scales with the number of data samples. Computationally, this quickly becomes unwieldy. We apply Nyström style approximation by sub-sampling m < n columns of each K, Gramm matrix and using those to approximate the full Gramm matrix [52].
C Probability Results
Lemma C.I. For == 1,.... d and o² O let a be independent chi-square distributed random variables with rd degrees of freedom for ,...., d. Then, with ő E (0, 1) and for
20
Hog(d/8)
4 Alog(d/))
A
1+
Pmin
Puin
we have
X5
Au
P
max
2 1-8. 8.
13y
np2
2
Proof. This Lemma 8.1 in [10] and substituting x = log(1/6).
The previous lemma is the general version of a concentration inequality that needed in the proof of Theorem A.5. The concentration inequality that we want to use is the following.
Lemma C.2. Let o² and assume that e(1) (1),...(()) are independently (000 o²) distributed, & as in Appendix A, and with 8 E (0, 1) set X 2 4do for
2o vn
4 log(d//)
4 llg(dd/))
Ao
1
Pmin
Pmin
Then, P(A) 2 ō, 6, where A n 1==1 Ai with the events A, ( 1 < AV for all j = 1, 1,...d and = (s( ((n]]
Proof. By assumption, I xp
E, - 1 and the fact that ~V Inxn), we first 1 oy (DE)'
oyR D,
NO n _ Inxa 2b) N00 Ipxx")
This shows us that ا [ 1 has a chi-squared distribution. We can now apply Lemma C.I by noticing that it holds also holds for X 2 4)p Au and that
A
max ISi
- /np In
1
1
(18)))
S
A
j=1
31

---

2
0
2
0
2
3
Figure 4: The different types of diffeomorphisms used in the misspecified case.
D Experimental Details
All the code to reproduce the experiments and figures in this paper is GitHub
provided
repository at https: / com/ ddeFok? We perform synthetic experiments
sample sample-en on 192-core CPU node (AMD Genoa) with 336GB of RAM. We perform the image benchmark experiments on single 18-core GPU node (NVIDIA A A10 GPU and Intel XEON CPU) with 120GB of RAM.
D.1 Toy Dataset Experiments
The synthetic experiments can be subdivided into sets, based which features mapping is used. These features mappings are linear features, spline features. random Fourier features and kernels. Here, describe the data generatation and the hyperparameters of the features and kernels.
D.1.1 Data Generation
For the synthetic experiment sample the C Rd variables from (N P) + pl) distribution, where 1 denotes a matrix filled with only I's. The p E (0, 1) parameter controls the amount of correlation between the variables. For the experiments with continuous concepts, sample n data points, on which we perform 80/20 train/test data split. For the binary concept experiments, we perform 50/50 train/test split, to in the test set. We also keep the correlation coefficient in the test set at 0, while varying the correlation the train set.
that both labels of each concept appear
We now simulate the variables directly from the C' variables, instead of using a causal representa- tion learning method that would learn them from the observations X. This allows different sources of noise more carefully. There are 2 settings in which we generate the M Rd variables, well-specified (following all of our assumptions) and misspecified (settings in which consider more general mixing functions beyond our thcoretical assumptions.)
control the
Wellspecified. In the wellspecified setting we generate the M variables by applying a map con- sisting of the features and kernels used to estimate the permutation. This setting is sanity check to see our estimator works setting that satisfies all the required assumptions. For each dimension j = 1,... d, a random weight vector 83 E IR is sampled, such that [|8] || E [16λg, 32 uniformly. A permutation n: {L.... d} + {1, ...,, d} is uniformly sampled from all possible permutations. Finally, with independent E; ~ (O, o²) noise variables we get
C, = ((MM()) Bi(e) + Eq.
Misspecified. In the misspecified setting we generate the M variables by first sampling diffeo- morphisms uniformly, {f:: IR + R dd from a of pre-specified diffeomorphisms. The set of possible functions is plotted in Figure 4. Each function gets a random scaling w; uniformly in 2, Finally, we get
C; = Wn(t) (((() (Ma(e) + Ei.
32

---

with Eg (V o again independent noise variables.
Target Label. In the experiment where we need binary concepts, the concept labels were created by thresholding each variable at the midpoint. The underlying continuous score was generated using the misspecified setting from above. In this case the midpoint O and we get
C [[; 0].
The M variables are kept continuous and a logistic regression used map M C. To create the target labels, a subset of size & min(3, [ ] is sampled from {l,...; d). without replacement. To determine if the target label is 0 or 1, we sum the binary concepts in the k selected variables and check if the sum at least max(1, [)
Metrics. For each of the experiments with continuous variables we report the the mean enror in the learned permutation of the variables w.r.f to the true permutation π of the variables, defined as MPE = / Σ (i) # T()). Moreover, we report the R score and execution time. To evaluate different settings of our estimator, we compare with the MPE, R² and execution time of using only the purely linear version of our estimator that particular data setting. We also report the MPE and execution times of assigning the variables based on Pearson or Spearman correlations as naive baselines. In the binary experiments, we report the MPE, the mean concept accuracy, the label accuracy, the OIS-metric and the NIS-metric by Zarlenga et al. [58].
Parameters of the experiments. We vary the following parameters: the regularization parameter X, the dimension d, the correlation and the number data points n. As stated before, in each of the continuous experiment we look at the wellspecified and misspecified case. The settings in the continuous case are:
The regularization parameter varies in À {0.001, 0.005, 0.01, 0.05, 0.5, 1}. The other settings are set to d E {20, 60, 100), p = and n = 1250.
The dimension is varied in d (5, 30, 60, s0, 100}. The other settings are set À {0.001, 0.01, 0.1), p 0 and = 1250.
The correlation parameter varies in p {0, 0.2, 0.4, 0.6, 0.8, 0.95, 0.99}. The other settings are set {0.001, 0.01, 0.1), d = 60 and 1250.
The total number of data points is varied in E {65, 125, 1250, 2500, 5000). The other settings are set to À E (0.001, 0.1,,
60 and p = O.
In the case of the binary concepts, we have fewer combinations, because we focus more on the downstream task and comparing the estimator to baselines:
The dimension is varied in d E {5, 10, 15, 20, 303. The other settings are set to A e {0.001, 0.01, 0.1), p = 0.5 and n 2000.
The total number of data points is varied in {100, 200,2000, 4000, 10000). The other settings are set to À E (0.001, 0.01,001), d = 20 and p = 0.5.
D.1.2 Feature and Kernel setting
We now describe the types of models we consider in our evaluation the synthetic data.
Linear features In the linear case no transformation applied to the M variables.
Spline features In the spline features case perform the regression using spline basis trans- formation, either piecewise linear or cubic splines. We expect this method to work especially well, because the cubic splines form a dense subset in the space of twice-difierentiable functions, of which the diffcommrphisms are a subset. To calculate these features we use the SplineTrans former class of the scikit- -eikiitleeann package. The total number of feature parameters is calculated as p = ni + nu - 1, where nı is the number of knots and na is the degree of each spline. each the toy dataset experiments the number of knots was n, E {4, 8} and the degrees were n. E (1,3).
Random Fourier features For the random Fourier features we use a varying number of random features. We sample random features that approximate the RBF kernel. To sample these features
33

---

we use the RBF Sampler class of the scikit-learn package. The total number of feature parameters in this case the number of random Fourier features. The number of features the toy dataset experiments is E (2,4,6,8)
Kernels For the kernel experiments we perform the experiments for several kernels: the polynomial kernel, the RBF kernel, the Brownian kernel and Sobolev kernel. These kernels are given by
KSpal (I; y) = (1 + (x,, y))
KRBB (I, 9) (x-
KiLap (Ir, v) = jr-yyl
Koos (I, 2) cos((1, (1),()))))
D.2 Action/Temporal Datasets Experiments
We consider a synthetic data benchmark from the causal representation learning literature [31]. The data generation settings, model architectures and training hyperparameters are taken from the origi- nal paper [31]. The implementation can be found at https:://iithi com/ s Lachapell le/ disentang ement via mechanichhnincceecc sm sm_ sn__sparittyttree//// main, and is available under the Apache License 2.0.
D.2.1 Dataset Generation Details
The benchmark consists of temporal data sequences, {(X*, ,", a²) where X R20 is the observed data, a' E 10:00 is an action, which seen as an auxiliary variable in the ICA framework developed in [24], and z' E IR is the latent causal variable. The ground-truth mixing function f is a random neural network with three hidden layers of 20 units with Leaky-ReLLU activations with negative slope of 0.2. The weight matrices are sampled independently according to N 1) and the weight matrices are then orthogonalized to ensure invertability of the mixing function. The observational noise in each dimension is sampled according A/O. 10-4) and is added to f(="). The transitions from (2--1, a²=1) to z are sampled according to ( 10 10- The mean function μ will be different between the Action Sparsity dataset and the Temporal Sparsity datasets.
Action Sparsity Dataset In this case, the sequences have length T 1 and the mean function is given by
paz-11 a ( = sin( 2 a + (i 1)) + sin( 2+ a + (2 ())1)),
where the index i = -1 is periodically identified with i = 10.
Temporal Sparsity Dataset In this case, the sequences have length T' = 2 and the mean function given by
4(2l-1, (1al-1,a'-1)x a'-1))== = 2 1-1 + 0.5
sin 2+d t-1 + (i 1)).
Target Labels We employ the same tactic as before to create the target labels. The true latent vari- ables 2f are binarized by looking the empirical range over the whole dataset, and then thresholded at the midpoint. The target label is then determined by sampling 3 dimensions in {1,.... 10) and setting the label to at least of the latent variables labels are
In both datasets we sample 106 points and split the data 80/20 for the train/test split.
D.2.2 Model Architectures
As the first step of our pipeline we compare three models: TCVAE [11], iVAE [24] and DMS-VE VAE [31]. The same encoder and decoder architecture is used for all models: an MLP with 6 layers of
512 units with LeakyReLU activations with negative slope 0.2. The encoder fend 0) outputs the
mean and standard deviation of q0 1t x²), which are the densities of normal distributions. The
latent transition distribution Px (z) 2 a <t), where <t ) and a (a )=)) is also
34

---

Table 2: Architecture details for the encoder and decoder used in the temporal and action sparsity dataset experiments.
Layer
Hidden Size
Activation Function
Linear 512
LeakyReLU(0.2))
Linear 512
Lea)yReLLU(0.2)
Linear
512
LeakyReLU(0.2))
Encoder
Lincar
512
LeakyReLJU00.2)
Lincar 512
LcakyReL_U0.22)
Linear 512
LeakyReL.U00.2))
Lincar
2.10
Linear 512
LeakyReLUU0.2))
Linear 512
LeakyReLU(0.2)
Linear
512
LeakyReLUU(0.2)
Decoder Linear
512
LcakyRel.U000.2)
Linear
512
LeakyRel.U(0.2)
Linear 512
IcakyRel..(00.2)
Linear 20
learned by fully connected neural network. The decoder faec(2; v) tries to reconstruct the original data from the learned encodings. A minibatch size of 1024 is used for training. See Table for detailed description.
The differences between the three methods come from the loss function that is optimized. The common term in each of the optimizations is the Evidence Lower Bound (ELBO) objective, which is given by
ELBO(O, A)
E Topp (-(at))rt)
[log
|0)]
(24)
E
KL(Co(=" (x') Px ((,t"1),,,
(25)
where KL(- || -) is the Kullback-Leiller divergence.
TCVAE We use the implementation of TCVAE [11] by Lachapelle et al. [31]. The loss function consists of the same components the ELBO objective, but they decompose it into 3 terms and add a weight parameter to each the terms. The hyperparameters for the training procedure can be found in Table 3.
iVAE We use the implementation iVAE [24] by Lachapelle et al. [31]. The loss function here is similar to the ELBO objective, but it adds one parameter 3 to the KL-term in the objective. The hyperparameters for the training procedure can be found in Table 3.
DMS-VAE Lachapelle et al. [31] introduce a sparsity regularization the ELBO objective to prove the identifiability in temporal and action settings. The new objective is given by
ELBO(8, v/, A) (=I|G= |lo a.llGallo
The variable G. is a learned matrix, representing the relations between latent variables between two time steps. The variable G, a learned matrix representing the relations between the actions and the latent variables. The norm Ilo counts the number of non-zero terms. This is discrete objective and can transformed into a continuous objective using the Gumbel-Soffmax trick [37, 22].
Instead of using the regularized objective, the authors also propose constraint based optimization procedure on the ELBO objective, where the constraint is determined by the number of edges in the learned graph. We use their constrained optimization method, which provides optimization schedule that we set by enabling the -oconst ra nt int_scchedu flag. The other hyperparameters for the training procedure be found in Table 3.
35

---

(a)
(b)
(c)
(d)
(e)
Figure 5: Examples the 7 shapes in the Temporal Causal3DIdent dataset. From left to right: teapot, armadillow, bunny, cow, dragon, head and horse.
D.3 Temporal Causal3DIdent
We consider an image data benchmark from the causal representation learning literature [33]. The dataset, model architectures and training hyper parameters are taken from [33]. The implementation can be found at https: //gittttb com/ whwppph//ppppppppppppppppppppppp httplpppe/IIIII/EEEEEEE /main, which is available under the BSD 3-Clause Clear License.
D.3.1 Dataset Details
The data comes from a setting which is referred as Temporal Intervened Sequences. The assumption is that there are d causal variables (GI...., Ga) and a corresponding causal graph G (V, E) where each node
represents a causal variable G.. The variables can real-valued or vector-valued and each edge (i,5) E represents relation between G; and Gj. We assume that there are T time steps and for every 1..... the causal variables follow a stochastic process. We therefore have sequence {(G .... It= where only the causal variables in t are the parents of the causal variables in time step t. The observations X at each time step are created through a mixing function X f(G G2 e),, where g' is i.i.d. noise variable. Additionally, d-dimensional binary vector each time step I E {0, 1] that tells us which causal variables have been intervened on, but not with which value.
have access
The causal variables that are used in the data generating process are the following:
The object position (pos_o) is modelled in 3 dimensions (æ, y, 2) E [- The values are forced to be in this interval to ensure that the object does not disappear from the image, becomes too small or covers the whole image.
The object rotation (rot_o) is modelled in dimensions (a, 3) [0, 2m) Distances for angles are calculated in periodic fashion, ensuring that angles close to 0 and 2x are close together.
The spotlight rotation (rot_s) the positioning of the spotlight that shines on the object.
The value range is [0, 2n), where distances are again calculated in periodic fashion.
The spotlight hue (hue__) is the color of the spotlight. The range of the value is [0, 2m). where corresponds red.
The background hue (hue_b) is the color of the background. The value range is [0, 2m)
with 0 corresponding to red again.
Table 3: The hyperparameters used for the training of the OMS-VAE, TCVAE and iVAE for the action and temporal sparsity datasets
Hyperparameter
Value
Batch Size
1024
Adam [27] and
Optimizer
Cooper [12]
Learning rate
5e-4 (DMS-VAE), 1e-4 (iVAB), 1e-3 (TCVAE)
KL divergence factor 3
1.0
Number of latents
20
Number of epochs
500
Gumbel Softmax temperature
1.0
36

---

The object hue (hue__ _oue_oo is the color of the object, with value range is [0, 2x) and again with 0 representing red.
Lippe et al. [33] generate the data using Blender, a setup inspired by von Kügelgen et al. [51] and using code provided by Zimmermann
[59]. They generate the dataset by starting with an initial random set of causal variables. They then sample the causal variables in each subsequent time step by following specific conditional distribution, which is given by the set of equations in (26).
f(a, b) a a-b
pos_ [os_xtt1 ʃ(1.5 15 sin(rot_ pos_x', e
pos r f(1.5 sin(rot a²), pos pos_y's y'; y' :gy)
2os z ++ f(1.5 sin(rot_ a²), pos_pt foo__2,,_0 ,
rot_a ++ f(hne_l rot_a', (aa)
(26)
rot
g g++ f(hue_o' rot pot_Bt,E3)
rot +11 ([atan2(pos_<> x², pos_y', rot rottoossscee et,)
huc s+1 f(2x huc bb, huc_s s chaa)
hue b 2+1 hue_ + +
hue 6 =f(g(i), hue o', cho)
All the noise variables, E, are independently W(0, 10 () distributed for the position and (OO, (0.15)3) distributed for the angles. The g function in the final line maps the object shapes to specific values detailed in Table 4.
The object shape is changed in each time step with probability of 0.05. If it is changed, new shape sampled uniformly over the shapes.
They then sample for each time step the intervention targets 1 Bernoull(OO1).. If causal variable is intervened on, it is replaced with a random sample from U-2, 2) for the position values or (0, 2r) for the angles. For the object shape uniform distribution over the shapes is used. They run this simulation for 250, 000 steps, which is the full dataset.
We use the already generated dataset downloaded from 6637749# YgcWCCVBxCA, which is available under the Creative Commons Attribution 4.0 Inter- national license.
lereendooorg//oo records/
Target Labels We employ the same tactic as before to create the target labels. The true latent variables G,, are binarized by looking the empirical range over the whole dataset, and then thresholded at the midpoint. The target label is then determined by sampling three dimensions in (1....,77 7) and setting the label to if at least two of the three latent variables labels are
Table 4: Output of the g function for each object shape. The avg function for angles is defined as sin(o) +sin(A) (os(a))
avg(a: (a, 8) atan2
2
2
Object shape
Object hue goal
Teapot Size
Armadillo
2x
Hare
avg (Shueess hue_b")
Cow
4s
5
Dragon
π + avg(hue_s* ss Lue_b'.
Head
6a
g
Horse
37

---

Table 5: Architecture details for the encoder and decoder used in the Temporal Cadeal3DIdent experiments.
Layer
Feature Dimension (H x W C)
Kernel
Stride
Activation Function
Conv
32 x 32 x 64
3
2
BatchNorm-SSLL
Conv
32 x 32 x 64
3
1
BatchNorm
Conv
16 16 x 64
3
2
BatchNom-SSiLU
Conv
x 16 x
3
1
BatchNorm-SiLU
Conv
8 8 x
3
2
BatchNom--SiLU
Encoder
Conv
8 x 8 x 64
3
1
BatchNorm-SiLU
Conv
4 4 x 64
3
2
BatchNorm-SiLU
Conv
4 x 4 x 64
3
1
BatchNorm-SiL.U
Reshape
x x 1024
Linear
x X 256
LayerNomm-SLLU
Linear
X X -num -aum__atents latents
Linear
X 256
LayerNom++Sl.U
Linear
X 1024
Reshape
x x 1024
Upsample
8 x x 64
ResidualBlock
8 x 64
3
Upsample
16 16 x 64
ResidualBlock
16 x 16 x
3
Decoder
Upsample
32 x 32 x 64
ResidualBlock
32 x 32 x 64
3
Upsample
64 64 x 64
ResidualBlock
64
x
Pre-Activations
64 x 64 x 64
BatchNorm-SiLU BatchNorm+SiLU Tanh
Conv
64 x 64 x 64
1
1
Conv
64 64 x
D.3.2 Model Architectures
In this setting
the first part of our pipeline both CITRIS [33], in particular CITRIS-VAL VAE, and iVAE [24]. In both models, the encoder and decoder architecture are set to be the same. The parameters per latent variable. These
encoder convolutional neural network, which outputs will be the mean and the log of the standard deviation for the normal distribution that models the latent variable. The decoder The full architecture described in Table 5.
bilinear upsampling and residual blocks to reconstruct the image.
As an additional optimization step, autoencoder is pre-trained to map the high-dimensional images to nower-dimensional feature vectors, but without enforcing disentanglement. This is done separately from the main training procedure, as Lippe et al. [33] mention that this improves performance. During training a small amount of Gaussian noise is added to the encodings to prevent a collapse of the encoding distribution. No prior is enforced for this encoder. This autoencoder has 2 ResidualBlocks instead per resolution in the decoder part. The training hyperparameters are described in Table and the autoencoder is trained using the MSE reconstruction loss.
CITRIS-VAE CITRIS [33] allows for multidimensional causal variables. The number of latent variables allowed to be bigger than the number of causal variables d' but the model subdivides the latent variables into d possibly uneven blocks that get mapped the causal variables. This means that an assignment t: (1....,k) }} (1..... ......d)) d} is learned between the learned latent variables and the learned causal variables, that multiple latent variables represent causal variables. This is done by assuming that each (i) follows Gumbel-Softmax distribution and we learn the continuous parameters that govern these distributions. CITRIS- VAE optimizes the following objective: ELBO(8, (p,7) ol 1(ll (=1a 1od113 12)
E
[log Pa (x f+1
2++]]]
E (-1a") *~GS(7)
Σ 1=1
KL (2 1+1 | | II ps (=tt)) 1 (211-11))
38

---

Table 7: The hyperparameters used for the training of both the CITRIS- SERRIS-VAESE and iVAE models.
Table 6: The hyperparameters used for the training of autoencoder used as preprocessing by both the CITRIS-VAE and iVAE.
Hyperparameter
Value
Hyperparameter
Value
Batch Size Optimizer
512
Adam [27]
Batch Size
512
Learning rate
le-3
Optimizer
Adam [27]
Cosine Warmup
Learning rate
le-3
Learning rate scheduler
(100 steps)
Cosine Warmup
Learning rate scheduler
KL divergence factor B
1.0
(100 steps)
KL divergence factor du (A)
0.01
Number of latents
32
Number of latents
Gaussian noise
0.05
Number of epochs
600
Number of epochs
1000
Target classifier weight
2.0
Gumbe Softmax temperature
2.0
Here Pe models the encoder, qe the decoder, Po (2 |², T--1) the transition prior and GS is the Gumbel-Softmax distribution of the causal variables between time steps given the intervention targets. Finally, is the target assignment between learned encoding variables and the causal variables. During training a latent-to-causal variable assignment is sampled, while during testing the argmax used. The transition prior P.p is learned by an autoregressive model, which for each at+1 takes *v(()
z², 1 and .++1 as inputs and outputs a Gaussian random variable. The autoregressive model follows a MADE architecture [23], with 16 neurons per layer for each encoding, and the input to these neurons are the features of all previous encodings. The prior layers deep, and uses SiLU activation functions. Finally, small network is trained to predict the intervention targets, given 2' and (+1 for each i =1,..... 1. (()
IVAE
To adapt the iVAE model [24] for this setting, the auxiliary variable u is given by the previous observation z' and intervention targets f++1 Another alteration that is made, is that the prior with the
iVAE model only conditions on (x* [ The main difference between iVAE and the CITRIS-VAL
the structure the prior p(= ++1 z ((t)). Another difference is that no target assignment is learned during the training, but only after. For iVAE a 2-layer MLP with hidden dimensionality of 128 is used for the transition prior.
All the hyperparameters for training CITRIS -VEE and iVAE are reported in Table 7.
D.4 Concept-Basse Models
Concept-bottlennck models were introduced and popularized by Koh et al. [28]. They offer deep learning architecture that inherently interpretable, by letting the input go through concept layer. Each node in that layer indicates either the value of predefined concept when using continuous concepts if the concept is present or not when the concepts are binary. Then, linear layer from these concepts to the final target layer is applied. Formally, given the observational data {[z,, Ci M= where V; E X, 6 E G., y/ E IR, a concept encoder (e x)) and a label predictor Pp c) are learned. The total prediction is then performed by combining these 2 probabilities. Different choices are possible for this combination, which lead to different concept-bass based models. We will discuss the Concept-Bottlened Model (CBM) [28], the Embeedtg Model (CEM) [57] and the HardCBM [17], which are all three used the binary versions of our experiments. For all three models, use the same encoders in the VAE models, to make the comparison fair.
The linear layer allows the user to construct explanations for each prediction. When a prediction is made, we can look at the concept activations, é; and the corresponding weights in the final layer tb. An explanation is given by the set
(V,, a) (L).
The interpretation is that f/s tells us how important that concept in general, and é tells us how active that concept is for this particular prediction.
39

---

CBM In the Concept-Bottlend Model (CBM) [28] the output of the concept layer will be denoted by 9 and the function from the concepts to the labels will be f. The output of g are the logits of that concept being 1, meaning (g(x)) po(c; = a:) for = 1,..., d, where a is the sigmoid function. The function f is a linear combination of the probabilities for all the concepts so together f((((())) vo a (g(z)) Pe.v (y r).
CEM Concept-Fimbedding Model (CEM) [57] show that the performance of CBMs can be improved by letting the concepts be encoded in a higher dimensional space, i.e., é, E R instead of & E IR. These embeddings are then concatenated and fed to the label predictor. For each concept, a function 8; is learned that predicts the concept label from these embeddings. These predictions can then be used for explanations, but the embeddings are passed to the label predictor.
HardCBM CBMs can also be susceptible to concept leakage, when unrelated information is encoded in each concept. HardCBM [17] propose way address this issue. In particular, to discourage information from other sources leaking through the concepts, the HardCBM label predictor only takes binary values from the concept layer input. To make prediction, we have to marginalize over the possible concepts, po, (y xr) = (eepooteaa) [y c)]. To model possible dependencies between the concepts in the distribution of X C, the conditional distribution is decomposed as
d
Pa(c x) = II p(c: x, .C.....--.).
i=1
In the HardCBM such decomposition can be achieved by using autoregressive architecture.
In the experiments, each model is trained for 100 epochs, with batch sizes of 256 and optimised by Adam with learning rate of 1e-3.
D.5 Performance metrics
To assess the performance of our estimator we report the mean permutation error of the estimated permutation (MPE), together with the execution time. Whenever the concepts are continuous, we also report the R2 R2-score. Some of the alignment learning baselines are created by regressing every input variable onto every output variable and using the individual RR"-scores to extract an alignment. In those cases, we first match the encodings and ground truth causal variables according to the highest Rco-rc The -so-e that is reported is the average of the Rt--scores that are chosen to be matched. This is also referred to as the R2 score on the diagonal. This gives
C. a, (M (Mar) 12
n
n2
C - d
C,
1
8=1
Here, a and ने are the estimated alignment map and estimated permutation. The alignment map is applied to the whole vector My. As in the main text, C and M are matrices where the rows are the data samples and the columns the concepts and encodings respectively.
In the downstream classification experiments we also report various other metrics. The mean accuracy of all the concepts and the accuracy of the downstream label prediction is reported. Furthermore, two additional metrics for concept-basse based models are reported, the OIS and NIS metric. The exact definitions can be found in Zarlenga al. [58]. The OIS metric measures how well concept used to reconstruct concept j. The NIS metric measures how much information about concept i is contained in the other concepts jointly. A low OIS metric, but high NIS metric indicates that it is not possible to reconstruct one concept from any of the others individually, but combining the concept embeddings would allow to reconstruct the concept from the others.
be
Estimator settings We use various settings of our estimator, to assess if there are particular advantages for certain versions. The versions that we use are
Linear, feature map is applied.
Random Fourier Features, we sample random Fourier features from the RBF kernel.
Spline, we calculate cubic spline features with 4 knots.
40

---

Laplacian, we use the Laplacian kernel with min(n, 20} components. If the number of components less than the number of datapoints a Nyström sampling procedure is used.
Two stage apply
stage approach, where use 20% of the data estimate the permutation using no additional features and then use the rest of the data to perform ridge regression with cubic spline features using knots.
We define an array of regularization parameters and report the results the best choice for each n. The parameters that we consider are À E {0.0001, 0.0005, 0.001, 0.005, 0.01, 0.05, 0.1,0.2) 0.2).
Alignment Learning Baselines. We estimate the permutations using the Pearson and Spearman correlations. We also add another baseline for the experiments with the encodings learned by VAE, This baseline is given by training multiple neural networks. Each neural network takes an individual encoding as input and tries to predict all the causal variables. The individual -coree are used to construct a matching again. This neural network is a 2-layer MLP with 128 hidden nodes in each layer and tanh is used as an activation function. The network is trained for 100 epochs with Adam and a learning rate of 4e-3. We considered using 32 and 64 hidden nodes per layer, but concluded that 128 nodes per layer offers the best baseline.
In the Temporal Causat3DIdent dataset experiments, both VAE models learn groups latent variables that are matched with a causal variable. To use the Pearson and Spearman correlations in this case, we first sum the latent variables in the groups and then calculate the correlation coefficients.
The baselines in the binary versions of the experiment in the two DL datasets are given by the three concept-basee models discussed in Section D.4.
E
Additional Results
In this section, we show all the results obtained in the experiments with the Toy Dataset, Ac- tion/Temporal Sparsity datasets and Temporal Causal3DIdent dataset.
E.1 Toy Dataset
Continuous Concepts. We provide plots for experiments performed with linear features, spline features, random Fourier fcatures and kernels. The concepts are continuous in these plots. All results are depicted in Figures 6-14.
In each version of our estimator we see that the MPEs are good, especially in the misspecified setting. It is interesting to note that the estimator does not perform well in the wellspecified case, when the regularization parameter is not tuned correctly. This can be explained by the non-invertability of the functions in this setting. This makes the identification of each matching more noisy and difficult, which can also be noted by the fact that the Pearson and Spearman correlation approaches are not able to find the correct permutation, while estimator is able do it correctly.
The non-invertibility of the functions in the wellspecified setting also explains why the R² scores in that setting are worse than the R2-score in the misspecified setting. Another reason for that observation is that the norms of the true parameters have to be quite large, to strictly adhere to the assumptions of our theoretical results. This increases the variability of the output data by large amount and makes regression more difficult.
Finally, we also see that our estimator does work well, even when the correlation is high, but the regularization parameter has to be tuned correctly. This does come at the cost of an increased computation time.
Binary Concepts Ablation. We also perform an ablation study, with binary concepts. How the continuous scores are binarized is explained in Appendix D.1 and sampling the diffeomorphisms from the functions depicted in Figure In these experiments we added the CBM [28], CEM [57] and HardCBM [17] as baselines. We report the concept accuracy, label accuracy, OIS and NIS scores [58] and execution times. The results are depicted in Figures 15-19. The ablation comes from the fact that the correlation in the training set differs from the correlation in the test set; in the former it p = 0.5 and in the latter p 0. This is done to mimic the situation where a classifier could pick up spurious correlations in the train set that are not present in the test set. We see that our
41

---

estimator performs well in this case in terms of concept accuracy and label accuracy, and better than the concept based-modee whenever the regularization parameter is set be not too large. Our estimator performs similarly as the concept-based models in terms of OIS, but consistently scores better when considering the NIS metric.
It is interesting to note that CEM [57] performs the worst in terms of the NIS and OIS metrics. An explanation for this is that it has the capacity store more information about the other concepts in the embeddings. CBM [28] and HardCBM [17] suffer less from this problem, as the concepts are stored as one-dimensional objects. All concept- models perform similarly in terms of label and concept accuracy.
One caveat is that the training times of our estimator do increase faster than the training times of the concept-basee models for increasing number of dimensions. We think this can be attributed to the fact that to perform the logistic alignment learning procedure, loop over all dimensions has to be done. In each step one logistic Group Lasso regression is performed. In the continuous case, this was not necessary, as we could see the output as multi-dimensional, and all regressions could be done in parallel. In principle this should be possible for the logistic Group Lasso as well, but implementing this parallel computation was beyond the scope of this paper. The implementation of the logistic Group Lasso we use, does not have this feature for multiple labels.
Mixtures of Causal Variables. We conducted an ablation with the synthetic toy dataset to evaluate performance when we have mixtures of causal variables as inputs to the alignment, which simulates the case in which CRL methods might not provide complete disentanglement, but only partial disentanglement. In these experiments, created d/2 pairs of M, variables. For each pair (M, M,),, we then created a new mixture variable Mj = (1 M, + a. where represents the mixing parameter. We then used these Mj variables as imput to the alignment estimation step. We ran 10 random repetitions for each setting and show the permutation error for the original variables in Figure 20. We additionally show paired mean permutation error, where we consider the estimated permutation correct if the matching identifies the pairs correctly. These results are shown in Figure 21.
set the dimension d to even number and then randomly
What we see is that our estimator keeps performing perfectly up until slightly below 1/2, but afterwards does not get the permutation right. However, we do see that the pairs are matched correctly for almost all values of a.
E.2 Action/Temporal Sparsity Datasets and Temporal Causal3DIdent Dataset
Continuous Concepts. Here, we report all results obtained in the Action/Temporal Sparsity datasets and Temporal Causal3DIdent dataset. The MPEs are reported in Table 8, the R2-scores in Table and the execution times are plotted in Figure 22.
In terms of MPEs, we always see that version of our estimator performs the best or as good as the best in each of the datasets for each number of training data samples considered. For the R2 scoces,, we see that we perform well in some cases the low data regime, but when using all the data available, the neural network often performs the best. This does come at computational cost, where the neural network approach requires two to three orders of magnitude more computation time.
It is interesting to note that the estimator typically works better for the more advanced models developed. This can be explained by the fact that these models achieve better disentanglement, which should make it easier to find the correct matching between the encodings and the causal variables.
Binary Concepts. We create binary concepts and labels in similar fashion detailed in Ap- pendix D. The number of active columns selected is now fixed at = and at least two of them have to be non-zero for the label to be one. The permutation errors are also reported in this case and can be found in Table 10. The baselines are CBM [28], CEM [57] and HardCBM [17]. The concept accuracy and label accuracy are reported in Table 11, and the OIS- and NIS-metries in Table 12. The execution times are plotted in Figure 23.
We see that the standard sancept-based models perform worse with respect to all metrics with only a few labels. However, in the action and temporal datasets, they quickly perform well in terms of concept accuracy and OIS. Our estimator performs especially well when using non-tabular data, such
42

---

as in the Temporal Cadsal3Dident dataset. This shows the added benefit of the CRL phase of our framework.
4 knots & 1 degrees
knots & degrees
Linear group
Pearson Spearman
4 knota & 3 degrees
knota degrees
2-20 pod
2-20 AoD Epec_.
d:00 pull
du00 p-ll
du163 puu spec
du100 p-D
spec-.troe
spw..
spac.
1.0
1.0
1.0
Poroulation Bevin
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
Regulariontonn (A)
Repulariontion (A)
Regularinaiin (A)
Regolarization (A)
Regnlariiation (A)
Reg segrllarinatio
L00 <pe-tmp 2-1220
Lnn que Eale 1250
A.n a.D <pec- 1250
spec Dden 12.150
1.0
Pesniaatinn Beeu
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.0
0.2
0.2
0.2
0.0
0.0
0.0
20
10
60
80
100
20
80
60
80
100
20
40
60
80
100
20
40
100
20
40
60
80
100
20
40
60
100
Dimeenson
Dimemiio (4)
<qei- den E350
spec Fale 1u01500
spec..tme 1201
spei Ealle nu. 1259
operiil daee 1201
1.0
Persuiation Einut
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
08
10
0.0
0.2
0.4
088
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0 0.2
0.4 0.6
1.0 0.0
0.2 0.4
0.6
1.0
Commlation (P)
Complation (e)
Correlation Apl
Correlation (p)
Correlation (6)
Comelation (e)
2.m
dam A.000
d_00
1.0
1.0
1.0
Pormriatton Banot
0.8
0.8
0.8
0.8
0.6
0.4
0.4
10
107
10P
102
109
Nr: data pointe (n)
Nr. data points (n)
Nr. data points (m)
Nr. datta pointss (m)
Nr. data points (n)
Nr. data points (n)
Figure 6: Permutation Error using Spline Features for all parameters considered in the experiments. Each plot is paired, where the left is wellspecified case and the right is the misspecified case.
43

---

4 knots & 1 degrees
knots & degrees
Linear group
Pearson
knots & 3 degrees
knots & degrees
Spearman
pud
2-300 pwp
4:00 p-l
Au00 pm)
d (سم
4_100 (-)
spocmfdlac
apocmtous
apec-lese 1-120
spocmdaac
1.0
1.1
1.0
0.8
0.5
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
10
2
RogullsSention (A)
Regulaetontoon (2)
Regularizatio (A)
Regulariation (A)
Regulaz (A)
Rog Seg (A)
A-RL00 ym1 opse-tese 1250
A-p.00 pm1
1-0.00 yrD apocmtnus 1210
A- ymB
A-0i1 ym1 apec 1210
A-0.1 p00 spocrdaae
apac Gula 1110
spec lale n- 1230
1.01
1.0
0.8
0.8
0.8
0.6
0.6
0.0
0.4
0.1
0.6
0.2
0.2
0.2
0.0
0.0
0.0
20
40
60
90
100
20
40
60
50
100
20
40
60
80
100
20
40
80
100
20
40
60
100
20
40
60
100
1llnousion
Diumossion
Dimmasoo
Dimession
Diusenion (4)
Dimseniin (4)
2-000
2-10 A--00::
2-00) apecittru zim1220
d-00
apocmieon am 1230
spec Bade 1100
specntrus mm1254
apee Jaber n- 1250
spocrdaae mm12200
1.01
0.8
0.6
0.6
0.6
0.1
0.4
0.2
0.2
0.2
0.0
0.0-
0.0
0.0
0.2
0.4
0.5
0.8
1.0
1.0
0.2
04
0.5
08 1.0
0.0
0.2
0.4
0.6
11
1.
0.0
0.2
0.4
0.
088
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.2
0.4
0.5
0.8
1.0
Cherelation (e)
Coerelation (e)
Correuatin (p]
Correiation [p]
Correationn
(e)
Corveation (p)
20-01003
drin 20-0
2-10
d-00 A-0001
4-00
falls
specn Iales
1.0
1.0
1.0
0.8
0.8
Recure
0.6
0.6
0.4
0.2
0.0
112
10at
102
10a
10
111
10P
10a
102
10a
102
10a
No. dala paints (n)
Nr. dula points (14
Nr. daii points (x)
Nr. daaia points (n)
Nr. dats points (x)
Nr. data points (n)
Figure R2-score on the diagonal using Spline Features for all parameters considered in the experiments. Each plot is paired, where the left is wellspecified case and the right is the misspecified case.
44

---

4 knots & degrees knota degrees
knots & degrees
Linear group
Pearson
knots degrees
Spearman
specelrte am
speowfill 1500
sprowtrat 1250
speew Jabi nee 250
spo- k=1300
sposnill sm 135)
10F
102
10P
10.
Time
10
100
10-1-
10
10-1
10-22
10
10-0
10
10
10
10
10
10
100
10
10
10
10
10
10
100
10
Begulattariin
Regulatiaaionn
Regulatiation
Regularizatinn
Regulariation (A)
Begalartaation
Amd.001
1-001
syaa falar 1200
apcomtrac 12-1250
apte-abbe rw 250
spacmiiae 1290
10a
10P
102
10a
Thea
10/
107
100
10
10
10
Liinonsion (4)
Dimmako (d)
Dimarioo (4)
Dimnason
Dimnssio
LZmommiion (i)
duu00 Aud D011 specwittie
AwED Swb001
dudt A-d D1 aprowdrrar
d= 3w0
duul0 Awdh speo
d-00 J
spec
qpee
10a
10
102
10a
Time
100
10
10
10
10
0.0
0.6 Corrolction (x)
0.4 Corrriation (p)
0.4
1.0
0.8
0.2
0.6
Correlation (P)
Correlation (p)
Currelation (p)
Conreation (x)
deott J-d 001
dweb Am0000
dwot) J-d0 DI
d-t0 2w00
d-00 2ort
10a
10
10¹
301
10
Thm
10
10
10
10-5
10
10
10"
102
10
107
10a
100
10P
10P
10F
107
107
Nr. data points (x)
(v)
points (6)
polats
polats
Nr. data points (n)
Figure 8: Execution times using Spline Features for all parameters considered in the experiments. Each plot paired, where the left wellspecified case and the right is the misspecified case.
45

---

2 Fourier features Fourier
Fourier features features
Linear group
Pearson Spearman
2-20 umb
2-20 pmo
2-03 mm
ded0 pmù
d-100 pm)
4m:00 pm)
1.11
1.00
1.0
Persulsiiv Bevnte
01.8
0.5
0.8
0.6
0.6
0.6
0.4
0.4
5
RegulasSention (A)
RegulasSontin (A)
Regularization (A)
Regularizatinn (A)
Regulttttntionn (A)
Rog toatton (A)
A-10000 gmn
A-0.00 pm1 Spec
2-0.01 ymD specntng a.0
A-0.0 ymn
A-0.1 pmm
quenadroo 1350
tuii 1350
1.01
Persralaiive Breer
0,8
0.8
0.6
0.6
0.6
0.4
0.4
0.0
0.2
0.2
0.2
0.0
0.0
0.0
10
60
90
100
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
20
40
60
90
100
Ilimousion
Diumossion
Dimmsion
Dimession
Diusenion
Dimomiion (4)
speiwitron oo dun 1350
spec felen mue 550
specnttue aud Soa
speii fallo a- 1359
sper Num11100
sperie Pone 1200
0.4
0.4
0.2
0.2
0.0
0.0
0.0
0.0
0.2
04
08
10
00
02
04
05
08
100
0.0
0.2
0.4
0.6
0.8
1.0 00
02
0.4
0.6
0.8
1.0
0.2
044
0.6
0.0
0.2
0.4
0.6
10
(e)
(e)
Correution (p]
Correatton (P]
Corrdation
Corclation (p)
4-00
d-tn falle
6-00
A-000:
6-00
4-00
spec
1.0
1.0
1.0
0.8
0.88
0.8
0.6
102
102
10
109
Nr. data poincs (n)
Nr. data points (n)
Nr. data points (m)
Nr.
data points (m)
Nr. data points (n)
Nr. data points (n)
Figure 9: Permutation Errors using Random Fourier Features for all parameters considered in the experiments. Each plot is paired, where the left is wellspecified case and the right is the misspecified case.
46

---

2 Fourier features
Fourier features
Linear group
Pearson Spearman
4 Fourier features
Fourier features
pud
2-30 pwr spocmfdaa
4:00 p-l
A:50 pal)
d (سم
4_100 (-)
apocntous
1.0
1.0
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.2
0.2
10
10
10-2
10
10-2
10 1
2
RogullsSention (A)
Regulaetonnion (A)
Regularizatin (A)
Regulariation (A)
Regulaziont (A)
Rog Sogtltattaation (A)
A-11000 ym1 opsc-tese 1230
A-p.00 ym1
1-0.00 yrm apac-ttns 1210
0- ymp
A-0i1 ym1 apec -lrse 1210
A-0.L p00 spocrdaae
Galad 1m10
Iale n- 1230
1.01
1.0
0.8
0.8
0.8
0.6
0.6
0.0
0.4
0.1
0.6
0.2
0.2
0.2
0.0
0.0
20
40
60
S0
100
20
40
60
100
20
40
80
100
20
40
80
100
20
40
60
100
20
40
60
100
1llnousion
Diumession
Dimmasoo
Dimession
Diusenion (4)
Dimsenion (4)
2-000
2-00 A-000:
2-00 X--0.12
2-00) apecittuu zim1230
apocmiton aum 1230
spec faleo 1102
specntrus m11254
apee Jabe n- 1250
spocrdaae 2112204
1.01
0.8
0.8
0.6
0.6
0.6
0.1
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0.0
0.2
0.4
0.5
0.8
1.0 1.0
0.2
04
0.5
08 1.0
0.0
0.2
1.4
0.
1.
0.0
0.2
0.4
0.
0.8
1.0
0.0
0.2
0.4
0.5
0.5
1.0
0.2
0.4
0.6
0.8
1.0
Cherelation
(e)
Coerelation (e)
Carreuatin (p]
Correiation [P]
Correationn
(e)
Correation (p)
drim 20-0
2-10
4-00 A-0001
4-00
spocitrrr
falas
specn balas
1.0
1.0
1.0
0.8
0.8
Racure
0.6
0.6
0.4
0.2
0.0
0.0
102
10at
102
10
10
111
10P
10
102
10a
10a
10a
No.
dala paints (x)
Nr.
dula prints (1)
Nr. daii points (x)
Nr. datia points (n)
Nr. dats points (x)
Nr. data points (n)
Figure 10: R² score on the diagonal using Random Fourier Features for all parameters considered in the experiments. Each plot is paired, where the left is wellspecified case and the right is the misspecified case.
47

---

Fourier features Fourier features
Fourier features
Linear group
Pearson Spearman
Fourier features
1-000 and)
de20 ped
dwol pnd
4-00 pwd)
d=100 pad
d=100 pat
speowfale
10"
107
10
10a
10.
10:
The
10-1-
10-1
10-1
10
10
10
10
10
10
10"
10
10
10
10
10
10
10
10
11
111
10
10
100
10
Begulatiaccon (A)
Regulariaation
Roguiatiaaiio
Reguloriantion (A)
Regulariantion
Begalartaatin
ap4s Masem am 1200
falef 1200
apsombrac tym1250
apsemaale labe 250
sparmion num 1200
spormible 1219
10a
10P
102
10a
The (4)
107
10a
100
10
10
16
10
10
10
10-3
100-1
20
40
100
20
100
60
80
100
20
40
60
100
20
40
60
100
20
40
100
Diamusion
DEmsmiio
duid
dur0 1w0
(Gult Awdh specstrue san 1290
du00 Ju0 spec aon 1353
specwirrte 1500
spec 1200
aprooiirna 120
spec Sake mo 250
10a
10F
102
10P
Thme
100
10a
100
10
10
10
10
02
0.2
Correlation
Correlation (p)
Correlation (p)
Correlation (p)
Currelation (p)
Conrelation
slw00 2w00.00 pod appcmbrac
dwe0 3w0001
doo0 1-0.01 pmb spacmlese
3-60 2w0.01 pm spocnfale
dedl Aw0.1 pnd specmieue
doo0 J apan
pmdl apeeeiabe
1.0
10a
10
102
10'
10
The
16a
10
10
10
10
10
10
10-15
10P
10a
10a
10a
100
107
10"
10
Nr.
data points (n)
No. data pointa (1)
No: data points (1)
21. datta polass (1)
Nr. data polats (x)
Nr. data points (n)
Figure 11: Execution times using Random Fourier Features for all parameters considered in the experiments. Each plot is paired, where the left wellspecified case and the right is the misspecified case.
48

---

cosine
polynomial
Linear group
Pearson Spearman
laplacian
rbf
0-40 <pec_tmu 124120
ded0 Qr 12-1
qpec-troe du 1350
apec-trre 14-1250
spera o.
1.0
1.0
Peraraaloo Exvus
0.4
0.4
0.2
0.2
0.0
0.0
0.0
10
10
10
10
10
10
10
10
10
10
100 10
10
10
10'
10
10
10
10
10"
Regnlarrission
Regulariontinn
Regularication (A)
Regulariration (A)
Regullariiation (A)
Regolarinntion (A)
2-00000
2-0.0 1.1250
1--0.1 Sea
1350
Ialio 1850
1.01200
1.0
1.0
1.0
Port Beeur
0.5
0.8
0.8
0.6
0.0
Dimeension (d)
Dimennson (d)
Dimmmsion (6)
Dimmmsio (d)
Dimnension (d)
Dimeensin (d)
2-m 200 qpenwtroee
6-m 0.03
6-0 A-0 epe
dum A-001
2_00 Audi agen
A0.00 Audi
sprie
1.0
1.0
Ponsulliion Brroit
0.8
0.5
0.5
0.6
0.6
0.6
0.6
0.4
0.4
0.2
0.2
0.2
0.2
0.2
1.0
0.0
0.6
0.0
0.4
Commelatin (P)
Cormllaion (p)
Currelation
Currelation
Correlation
Comelation
4-:0 Am0 (w-D opecutrae
Aum A ((o) qpeceallin
4-00 Amd puol quceatoee
4400 Aud pwD specddikee
Aa403 Andi poofl qprotccrion
du00 Aud puol spece fides
Nemusation Biror
0.8
0.8
0.6
0.6
0.4
0.4
0.4
0.2
0.2
01.2
0.0
0.0
0.0
10a
10
102
10
102
10a
102
10a
109
109
pointe (n)
pointe (n)
data points
data points
points (n)
(n)
Figure 12: Permutation Errors using Kernels
49

---

cosine
polynomial
Linear group
Pearson
laplacian
rbf
Spearman
pm0
pm)
4-100
d-100 spocmaaae 2202
apre-ies: 1250
xpaeefiaa 1m12
apaemtrue 1-12
aptc-lale fabe n-125
apre- 1210
1.0
1.0
1.0
0.8
0.8
0.6
0.6
0.0
0.0
10
10-2
10
10 10
10
10
10
10-2
10
10 10-3
10
10
10
10
10-2
10-1
10a 10
10
10
Regula Regulactantion (A)
Rostllstoation (A)
Rogularization (A)
Hogularization (A)
Rogeltsiintton (X)
Regaltatsattion (A)
2-00000 umn
A-10000 um1
2-0.0 prn
1-000 pmn
1-0.1 u00
T-1r um0
1.11
1.0
1.0
0.8
0.5
0.8
0.6-
0.6
0.6
0.4
0.4
0.2
0.0
0.0
100
Diineusion (4)
Diuzesiio (4)
Dimmaton (d)
Dimmason (d)
Diusousion (4)
Dimonsion (4)
2--00 2--0.033
2-10 20-0.025
2-00 A-000:
2-00 A-0001
2-00 A-m0. apec 12m0
d-00 Amo.a
<pocmizoe 1250
spacmBiad 1m1220
spac
apac-fale n- 1250
spocriaae 2200
1.01
1.0
0.8
0.8
0.8
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
0..0
0.2
044
0.8
0.8
1.0
0.0
0.2
044
0.5
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
Cocvelation
Coevelation
Corredation
Corredation
Conrelation
Conelation
2--0 p000 apacntres
gio specn fills
prod aocembtue
prott spec
pio spec Tileo
0.6
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
102
10at
102
10at
10P
110
10P
10a
102
10a
102
10a
points (x)
points (v)
Nr. dati points (x)
Nr. dacla points (x)
data points (x)
data pointa (n)
Figure 13: R²-scom the diagonal using Kernels for all parameters considered in the experiments. Each plot is paired, where the left is wellspecified case and the right is the misspecified case.
50

---

cosine
polynomial
Linear group
Pearson
laplacian
rbf
Spearman
specelete 1500
spccwfille 150
sprowtrat t=1220
spee labe ne. 250
spos no
spocmiile so 1352
10a
10F
102
10P
10.
Time
100
10a
100
10-1-
10
10-1
10-22
10
10-0
10
10
10
10
10
10t
101
10
10
10
10
10
100
10
Begulattaricn
Regulatiaation
Regulatiation
Regulurizaiinn
Regulariation (A)
Begalattaation
Amd.001
1-001
spasmboue 1200
syau Spx--ii 1200
apcomtrac 12-1250
atte-labe ro 250
spacettoe nom 1290
sparmiilee 1290
10a
10F
10a
10a
Thea
106
107
100
10
10
10
Liinonsion (4)
Dimmako (d)
Dimarioa (4)
Dimnason
Dimnssio
duu0 Aud D011 specwirtie
AwED Swb001
dudt A-d D1 aprowdrrar
d= Jw0
duul0 Awdh specwtrte
d-00 J
spec
spee
10a
10
102
10a
10
Time
10a
10
10
10
10
10
0.0
0.6
0.4
0.4
1.0
0.8
0.2
0.6
Corrolction (x)
Corrriation (p)
Correlation (P)
Correlation (p)
Currelation (p)
Conreltion (p)
doott J-d 001
dweb Am0000
dwot) J--0
d--0 2w00
d-00 2
10a
10
10'
301
101
Thm
10
10
10
10-5
10
10
107
102
107
107
107
10a
100
10P
10P
109
107
102
Nr. data points (x)
points (v)
points (1)
polats
potats
Nr.. data points (e)
Figure 14: Execution times using kernels for all parameters considered in the experiments. Each plot paired, where the left wellspecified case and the right is the misspecified case.
51

---

linear
knots 3 degrees
CBM CEM
HardCBM
Fourier features
Japlacian
A=0.001 =000
0.001 n=2000
0= ==2000
1.0
Concept Acc.
0.8
0.6
0.4
0.2
0.0
5
10
15
20
25
30
5
10
15
20
25
30
10
15
20
25
30
Dimension
(d)
Dimension (d)
Dimension (d)
0.0001 d=20
x=0.1 d-20
A= 2-
1.0
Concept Acc.
0.8
0.6
0.4
0.2
0.0
10a
103
10a
10a
103
10
102
10a
10a
Nr. data points (n)
Nr. data points (n)
Nr. data points (n)
Figure 15: Concept accuracy on the Toy Dataset when the training correlation is 0.5 and the test correlation is 0. Several versions of our estimator are compared to the performance of several concept-based models.
linear Pourler features
knots 3 degrees
CBM
HardCDM
Japlacian
CEM
A=0.001
20200
>=0.01 2=2000
>=0.1 x=2000
1.0
0.8
0.6
Label
0.4
0.2
0.0
5
10
15
20
25
30
5
10
15
20
25
30
10
15
20
25
30
Dimension
(d)
Dimension (d)
Dimension
(d)
>=0.001 d=20
A=0.01 d=20
A=0.1 d=20
1.0
Label Aec.
0.8
0.6
0.4
0.2
0.0
10a
103
10a
10a
10a
10a
10a
10a
10
Nr. data points (n)
Nr. data points (7))
Nr. data points (n)
Figure 16: Label accuracy on the Toy Dataset when the training correlation 0.5 and the test correlation is 0. Several versions of our estimator are compared to the performance of several coneept-based models.
52

---

linear
knots 3 degrees
CBM CEM
HardCBM
Fourier features
Japlacian
A=0.001 ==2000
00001 ==0000
A= ==2000
1.0
0.8
0.6
0.4
0.2
0.0
10
15
20
25
30
5
10
15
20
25
30
10
15
20
25
30
Dimension (d)
Dimension (d)
Dimension (d)
>=0.001 d=20
A=0.01 d=20
A= d=
0.8
O19
0.6
0.4
0.2
10a
10
10a
10
10
10
10
10a
10
Nr. data points (n)
Nr. data points (n)
Nr. data points (n)
Figure 17: OIS metric on the Toy Dataset when the training correlation and the test correlation is O. Several versions of our estimator compared to the performance of several concept- based models.
linear Pourler features
knots 3 degrees
CBM
HardCDM
Japlacian
CEM
A=0.001 ==2000
0=0.01 202000
A=0.1 x=2000
1.0
0.9
NIS
0.8
0.7
0.6
0.5
10
15
20
25
30
5
10
15
20
25
30
10
15
20
25
30
Dimension (d)
Dimension (d)
Dimension
(d)
>=00001 d=20
>=0.01 d=20
A=0.1 d=20
1.0
0.9
NIS
0.8
0.7
0.6
0.5
10a
103
10a
10a
103
10
10a
10a
10
Nr. data points (1)
Nr. data points (7)
Nr. data points (n)
Figure 18: NIS metric on the Toy Dataset when the training correlation 0.5 and the test correlation is O. Several versions of our estimator compared to the performance of several concept-based models.
53

---

linear
degrees
HardCRM
Fourier Tentarss
Japlacian
CEM
A=0.001 2=0000
0=0.01 2000
λ=0.1 n=2000
10
10a
Time
10
10"
5
10
15
20
25
30
10
15
20
25
30
5
10
15
20
25
30
Dimension
(d)
Dimension (d)
Dimension (d)
0=00001 d=20
A=0.01 d=20
A=0.1 d=20
10a
Time (s)
102
10a
10a
10a
103
10
10a
10
104
10a
10a
10
Nr. data points (n)
Nr. data points (n)
Nr. data points (n)
Figure 19: Execution times the Toy Dataset when the training correlation is 0.5 and the test correlation is O. Several versions of our estimator are compared to the performance of several concept-based models.
54

---

Poutier features
knote
degrees degrees
laplacian polyuomial
linear
guit03 pedt quoculiii
s012000 pul quoallie
puizi palopeaPPale
Judi s0113) aoliipmllle
and sqeuRRi
1.0
Forndatls Baoe
0.4
0.0
0.5
04-
0.0
6.0
0.4
02
0.5
10 6.0
D. 10
Mornng
-0-00 J.EAEL metzte
dvi0 And D01 mott33
2-00 2-d 60 1-1300
4055 2-d m-1t00
<000 x-83 1-1330
whit And wo3350
0.8
0.5
0.6
06-
0.6
0.2
010
0M0
0.9
6.1
Mining panzacko
Mising parmunter (x)
Midia jesmmtte (4)
Micing paramate (x)
Milin parmacies
parmaccee (c)
Figure 20: Mean permutation errors on the Toy Dataset when the causal variables are mixed according to parameter a.
Fourier features
knots degreee degrees
laplacian polynomial
linear
1225 pnb apac- Tra
1213 pil geecniaaa
1220 pi0 gece/hne
peb PadappaPal Pale
Dipece Then
and spec specilieee
1-129
Polod PS
0.8
0.0
0,6-
08-
0.4 02
06
0.
1.
0.0
06
0.5 1.0
Aud awilt33
Jbot Jud s40100 spece/ine
Suit maitt0t PapornPaal
Jud 1401100
Jedt 1000
0.8
Poled PE
M
0.6
06-
0.6
080
0.9
Miking poncneter
Miking permmdter (s)
Mistig persmoter (4)
Mixlag parameter (x)
Milin parameter
Mixing panometer
Figure 21: Mean paired permutation errors on the Toy Dataset when the causal variables are mixed according to a parameter
55

---

Table Permutation Errors for the encodings learned in the Action/ Temiooal sparsity datasets and the Temporal Cadeat datasets. The (n) indicates the number of train and test points used in each column. We report the mean and standard deviation over 50 random seeds and in each column indicate the best method by bold.
Permutation Error (n)
Model
Method
5
10
20
100
1000
10000
Action Sparsity Dataset
NN
0.18
0.03
0.01
0.00
0.00
0.00
Pearson
0.31 # 0.2
4 o.00
0.00
+ 0.00
0.00
= 0.00
0.00
+ 0.00
0.00
0.00
Spearman
0.28 = 0.03
0.04 0:00
0.00
0.00
0.00
= 0.00
0.00
:: 0.00
0.00
= 0.00
DMS-VAE
Linear
0.25 0.3
0.03 1 0.0
0.00
+: 0.00
0.00
0.00
0.00
+ 0.00
0.00
0,00
Spline
0.36
0.28 0.03
0.02
0.00
0.00
0.00
RFF
0.47 =
0.11
0.02 +
0.00
0.00
0.00
Laplacian
= n.as
0.28 + 0.03
0.03 + 0.01
0.00 = 0.00
0.00 + 0.00
0.00 = 0.00
Two Stage
0.46 n:03
0.06 + 0.02
0.02 + 0.01
0.00 0.00
0.00 + 0.00
0.00 + 11,00
NN
0.82
0.72 0.02
0.53
0.38
0.24
0.20
Pearson
0.76 = 6:02
0.56 +
0.39 + 0.35
0.16 +
0.14
Spearman
= 0.02
0.64 + 0.22
0.50 + 0.02
0.31
= 0:03
0.18 + 0.03
0.10 = 0.00
iVAE
Linear
0.78 6,02
0.56 + 0.02
0.41 + 0.0
0.20
0,83
0.02 + 0,11
0.00 = 11.00
Spline
0.84 0.02
0.77
002
0.61 0.02
0.32
0,02
0.12 + 0.02
0.00 +: 0.00
RFF
0.83
0.76
0,63
0.37
0.19
0.18
Laplacian
0.02
0.78 4 0.2
+ 0.02
0.34 + 0.02
0.03 + 0.01
0.00 0.00
Two Stage
0.78 = 0.02
0.65 + 0.02
0.54 + 0.02
0.32 0.03
0.21 + 0.02
0.22 + 0.03
Temporal Sparsity
Dataset
NN
0.78 0.02
0.70 4 0.02
0.48 + 0.02
0.46 :: 0.01
0.20 + 0.00
0.00
:: 0.00
Pearson
0.68 0,02
0.40 0.02
0.23
0.0
0.20
0.03
0.03 0.02
0.00 0.00
Spearman
0.72
0.53 0.02
0.40 0.00
0.11
0.02
0.00
0.00
0.00
DMS-VAL -AE
Linear
0.67 = u,2
0.41
+
0.20 + 0,03
0.07 = 0.02
0.00 = 0.00
Spline
0.72 = 0.02
0.70 1 0.02
0.47 + 0.03
0.17 = 0.03
0.03 = 0.1
0.00 = 0.00
RFF
0.76 naz2
0.59
+ 0022
0.37 + 0.08
0.10 0:02
0.03 + 0.02
0.01 + 0.01
Laplacian
0.80 0,02
0.67
0.2
0.45 0.02
0.15 0,03
0.01
0.00 0.00
Two Stage
0.78
0.59 +
0.44 +
0.15
0.01 +
0.00
= 0.02
+
0.65 + 0.02
0.33 = 0:01
0,25 + 0.02
0.60 + 0.00
Pcarson
0.74 0.02
0.54 + 0.03
0.30 + 0,03
0.36 0:03
0.24 + 0.03
0.23 = 0.02
Spearman
0.77 0.02
0.61 0.0
0.47 0.03
0.28
0:03
0.14 0.03
0.01 0.01
TCVAE
Lincar
0.76 = 0.50 +
0.36 +
0.26
=
0.18 +
0.19 -
Spline
=: 0.02
1 0.02
0.48 1: 0.02
0.32
0.03
0.24 1: 0.03
0.15 0.02
RFF
0.81 :: 0,02
0.60
4: 0.02
0.44 +: 0.00
0.30
=: 0.03
0.29 +: 0.03
0.23 +: 0,03
Laplacian
0.83 0.02
0.70
0.02
0.53
0.00
0.32
0.03
0.22 0.03
0.12 0.03
Two Stage
0.80 0.02
0.64 0.02
0.52 0.02
0.32
0.22
0.08
Temporal
Causal3DI Dataset
NN
0.72
0.37
0.11
0,00
0.00
0.00
Pearson
0.78 :: 0.03
0.64 1 0.03
0.39 1: 0.02
0.30 :: 0.02
0.14 :: 0.00
0.01
:: 0.01
Spearman
0.73 = 0.3
+ 0.00
0.56 + 0.04
0.25
= 0.04
0.09 - 0.03
0.00
= 0.00
CITRIS VAE
Linear
0.56 uла
0.41
+ 0.0
0.27 = a.
0.01
0:00
0.00 : 0.00
0.00 = 1100
Spline
0.69 0.02
0.59 0.00
0.33 0.03
0.00
0.00
0.00
0.00
RFF
0.79
0.63
0.42 +
0.10
0.01 + 0.00
Laplacian
0.71 + 0.02
0.56 + 0.02
0.35 + 0.02
0.11
+ 0:01
0.00 + 0.00
0.00
0.00
Two Stage
0.79 + 0,02
0.66 4 0.03
0.59 + 0.04
0.25 0:05
0.09 + 0.03
0.00
+ 0.00
NN
0.73 0.03
0,46
0.8
0.29
+ 0.2
0,00
0,00
0.00 + 0.00
0.00
0.00
Pearson
0.63
0.47
+
0.26
+
0.24
0.00
0.00
Spearman
= 0.05
0.50
+ 0.0
0.39
+ 0.02
0.11
+ 0:02
0.00 + 0.00
0.00
+ 0.00
iVAE
Linear
0.70 = 0,03
0.45
1 0.03
0.24 1 0,83
0.00 :: 0,00
0.00 :1 0.00
0.00
:: 0.00
Spline
0.69 0.03
0.62 1 0.0
0.41 1: 0.00
0.02 0.01
0.00 : 0.00
0.00
0.00
RFF
0.73
0.54 0.03
0.39
0.02
0.00
0.00
Laplacian
0.69 0M8
0.49 +
0.32 +
0.01 = 0.01
0.00 + 0.00
Two Stage
= 0.03
0.55 + oma
0.36 + 0.02
0.08 = 0.02
0.00
+ 0.00
0.00 + 0100
56

---

Table 9: R scores on the diagonal for the encodings learned in the Action/Teneen Temoprrl sparsity datasets and the Temporal Causal3DIdent datasets. The (1) indicates the number of train and test points used each column. We report the mean and standard deviation over 50 random seeds and each column indicate the best method by bold. If score was below 100, we indicate this with
R² score on the diagonal 1 (n)
Model
Method
20
100
1000
10000
Action Sparsity Dataset
0.21
0.03
0,56
0.01
= 0.00
0.00
0.79 + 0.00
4 0.00
Pearson
Spearman
Linear
-0.29
0,56
0.72
0.77
0.77
0.77 0.00
DMS-VAL
Spline
0.08
0.53 =
0.78
0.78 + 0.79
RFF
-0.69 = 0.12
0.24 = 0a2
0.59 = 0.11
0.77 # 0.00
0.78 + 0.00 0.78 + 0.00
Laplacian
-0.72 0.88
-0.03 = 0.2 0.33
0.01
0.64
= 0.1
0.70
= 0.00 0.70 + 0.00
Two Stage
11.25
141 0,53 0.08
0.77 0,00
0.79
0,00 0.79 1: 0.00
NN
-0.36
-0.14
0.10 = 0.1
0.29
0.31
0.31
Pearson
Spcarman
Linear
-0.86 0.11
-0.10
0.02 0.14 = 0.1
0.25
0,00 0.27
1 0,00 0.28 1: 0.00
iVAE
Spline
-19224
-0.18
0,05
0.24
0.27
0.27
RFF
-0.93
0.10
-0.22
à: 0.02
-0.02
0.01
0.19
: 0.00
0,25
: 0.00
:: 0.00
Laplacian
-0.83 = 0.06
-0.23
= 0.00 -0.05
0.0 0.12
= 0.00
0.17
= 0.00
0.17 1 0.00
Two Stage
-1.49
0.19 -0.28
0.03 0.19
0.11
0.30 + 0.00 0.31 + 0.00
Temporal Sparsity Dataset
NN
-278
0.30
-0.26 0.00 0.17 0.01 0.38 0,00 0.42 +: 0,00 0.43 1: 0.00
Pearson
Spearman
RMS-VAE
Linear
-0.94 0.11
0.11
= 002
0.32 = 8.1
0.41
= 0.00
0.43 = 0.00 0.43 1 0.00
Spline
-6.88 +: 4.8
-0.16 + am 0.17 = 0.1
0.40
# 0.00
0.42 # 0.0 0.43 4 0.00
RFF
-1.00 0.11
-0.10
a. 0.21
0.11
0.39
0,00
0.41
0.0
0.42 0.00
Laplacian
-1.11
-0.22
+ 0.04 = 0.24
0.30
0.31
Two Stage
-1.24
0.19
0.05
0.32 = 0.01 0.43
0.00
0.43 0.00
NN
-3.02 0.24
-0.44
0.4
0.05 0.01
0.28
0.00
0.33 + 0.00
0.34 + 0.00
Pearson
Spearman
TCVAE
Linear
-1.00 =: 0.15
0.00
0.03
0.23 = 0.01
0.32
: 0.00
0.33 : 0.00
0.33 1: 0.00
Spline
-0.18
:: 0.0
0.10
: 0.01
0,30
: 0.00
0.33
: 0.00
0,34 1: 0.00
RFF
-0.84 0.08
-0.13
0.0 0.15
0.0
0.31
0.0
0.33
: 0.00
0.33 1 0.00
Laplacian
-0.92 0.10
-0.21
= -0.01
0.17
0.1
0.23
0.00
0.23 0.00
Two Stage
-1.54 = -0.25
0.21
0.33 - 0.00
0.34
Temporal Causal3DIdent Dataset
4.11 0.32
-0.17 i: 0.00 0.22 = 0.02
: 0.01
0.59 1 0.00 0.65 : 0.00
Pearson
Spearman
CITRIS- -VE
Linear
0..2 0.15
-0.17
004 0.20
081
0.44
0.00 0.49
0.0 0.50 1
Spline
-0.21
0.12
0.44
0.60
0.62
RFF
-0.86 +: 0.15
-0.26
+: 0.0 -0.02 +00 0.27
+ 0.00 0.46
+ 0.1 0,50 4
Laplacian
+0.80
0.14
-0.24
+ 0. -0.02 + 0.1
0,36 + 0.0 0.54 4 0.00 0,57 4 0.00
Two Stage
-3.59
= 1.13 -0.81 20n +0.00
nas 0.52 : 0.01 0.62 1 0.00
NN
4.29
-0.23
0.13
0.62
0.71
0.74
Pearson
Spearman
iVAE
Linear
-0.88
0.16
-0.20
043
0.14 = 601
0.43 : 0.00
0.47 : 0.00 0.48 1: 0.00
Spline
-0.41
0.15
0.01 -00 0.43
0.00 0.55
0.56 0.00
RFF
-0.84
-0.26
-0.01 =0 0.36
am 0.52 - 0.54 +
Laplacian
-0.81 = 0.14
-0.24
= 0.
0.00 =
= 0.49 + 0.00
+ 0.00
Two Stage
-0.74 +
-0.50 + 0a6 0.54 + 0.00
0,56 + 0.00
57

---

Table 10: Permutation errors when using binary concepts created based on the ground truth latent variables in the Action/Temporal sparsity datasets and the Temporal Causal3Dldent datasets. The (n) indicates the number train and test points used in each column. We report the mean and standard deviation over 10 random seeds in each column and we indicate the best method by bold.
Permutation Errors + (n)
Model
Method
20
100
1000
10000
Action Sparsity Dataset
Linear
0.08 = 0.00 0.00
0.00
0.00 0.00
Spline
0.08 =
0.00
0.00
0.00
0.00
DMS-VAR
RFF
+ 0.03 0.00
0.00
0.00
+: 0.00
0.00
0.00
Laplacian
0.08 = 0.03 0.00
0.00
0.00
+ 0.00 0.00
0.00
Two Stage
0.08 0.03 0.04
0.0 0.00
+: 0.00
0.00
0.00
Linear
0.65 + 0.38 0.06 0.10 + 0.05
0.00
Spline
0.73 = 0.04 0.41 = 0.05 0.14 + 0.05
0.00
0.00
iVAE
RFF
0.75 = 0.03 0.45 = 0.04 0.27 + 0.04
0.09 0.05
Laplacian
0.73 0.02 0.52 0.04 0.25 0.05
0.06 0.04
Two Stage
0.65 1: 0.65 0.04 0.31
0.00
Temporal Sparsity Dataset
Lincar
0.37 + 0.13
0.00
0.00
Spline
0.58 = 0.04 0.16 0.07 0.00 + 0.00 0.00 0.00
DMS- VAE
RFF
0.56 1 0.05 0.20 i: 0.06 0.02 i 0.02
0.00 0.00
Laplacian
0.60 1 o.us 0.30 0,7 0.02 1 0.02 0.00
0,00
Two Stage
0.46 1 0.47 0.06 0.06
0.00
Linear
0.54 1 0.38 0.06 0.32 1 0.04 0.20 0.00
Spline
0.66 1: 0.0s 0.36 i: 0.07 0,21 1: 0.04 0.18 ::: 0.05
TCVAE
RFF
0.63 1 0.04 0.36 0.04 0.21 1 0.07 0.12 0.08
Laplacian
0.69 1 0.43 0.06 0.18
0.26
Two Stage
0.63 = 0.51 0.08 0.31 + 0.06
0.08
Temporal Causal3DIdent Dataset
Linear
0.10 1 0.06 0.04 0.00 1 0.00
0.00
Spline
0.07 1 0.03 0.00 i 0.00 0.00 1 0.00 0.00
0.00
CITRISVAE
RFF
0.33 = 0.06 0.10 = 0.04
0.00
+ 0.00 0.00
0,00
Laplacian
0.17
0.00
0.00
0.00
0.00 0.00
Two Stage
0.27 = 0.34
0.05 0.09
+ 0.02 0.10 0.02
Linear
0.06 = 0.04 0.00 = 0.00 0,00
+ 0.00 0,00 0.00
Spline
0.07 = 0.05 0.03 = 0.00
+ 0.00 0.00 0,00
iVAE
RFF
0.17 = 0.03
0.03 0.00
0.00
Laplacian
0.10 + 0.03
0.03
0.00
0.00
Two Stage
+ 0.04
0.14 0.05
0.00
+ 0.00
0.00 0.00
58

---

Table 11: Concept Accuracy and Label accuracy when using binary concepts and binary downstream labels created based on the ground truth latent variables in the Action/Temponn sparsity datasets and the Temporal Cidett datasets. The (n) indicates the number of train and test points used in each column. We report the mean and standard deviation over 10 random seeds in each column and we indicate the best method by bold.
Concept Acc (v)
Label (n))
Model
Method
20
100
1000
10000
20
100
1000
10000
Action Sparsity Dataset
Lincar
0.83 = 0.01
0.86 = 0.00 0.85 : 0.00 0.86 + 0.00
0.77 = 0.03 0.01 0.33 0.01
0.84
0.01
Spline
0.83 :: 0.11 0.86 0.00 0.85 :: 0.00 0.86 0.00
0.72 4 0.0 0.83 0.02 0.85 0.00
0.86 0.01
DMS- DMS-VAE
RFF
0.82 1: 0.00
0.85 0.00
0.85 1: 0.00 0.86 0.00
0.77 i: 0.2 0.82 0.02
0.84 0.01
0.85 0.01
Laplacian
0.81 = 0.01 0.85 = 0.00
0.85 = 0.86 0.00
0.75 + 0.82 0.4
0001
Two Stage
0.83 +: 0.85
0.86 +: 0.86
0.72
0.80
0.02
Linear
0.66
0.68
0.00 0.70 0.00
0.73 4 0.03 0.79 0.81 0.01 0.83 0.01
Spline
0.63 1: 0.01
0.68
0.70 1:001:0000 0.70
0.69 0.02 0.76 0.81 0.01 0.83
IVAE
RFF
0.59 1: 0.01
0.66 0.01
0.68 0.00 0.69 0.00
0.59 0.44 0.70 1: 0.22 0.78 0.00 0.81 0.01
Laplacian
0.59 = 0.01
0.65 0.01 0.68 0.00 0.68 + 0.00
0.60 100:00 0.70 0.02 0.78 0.00 0.79 0.01
Two Stage
0.64 + 0.01
0.65 0.00 0.69 + 0.00 0.70 0.00
0.67 0.03 0.70 0.02 0.79 0.01 0.82 + 0.01
CBM [28]
0.64 = 0:01 0.73 0.01 0.85 0.00 0.91 = 0.00
0.60 0.60 = 0.73
0.01 0.88 = 0.01
CEM [57]
0.64 1 0.02
0.72 1 0.01 0.86 1: 0.00 0.91 1. 0.00
0.66 1 0.02 0.69 1 0.03 0.82 1 0.01
0.88 1. 0.01
HardCBM [17]
0.66 = 0.01 0.73 = 0.01 0.85 + 0.00 0,91 + 0.00
0.56 + 0.03 0.61 + 0.01 0.68 + 0.01 0.89 - 0.00
Temporal Sparsity Dataset
Lincar
0.69 1 0:01 0.73 0.00 0.74 1: 0.00 0.74 0.00
0.75 0.03 0.79 0.01 0.84 0.02 0.85 0.02
Spline
0.64 = 0.01 0.73 0.01 0.74 0.00 0.74 = 0.00
0.69 0.0 0.79 = 0.02 0.84 0.01 0.85 0.01
DMS- DMS-VAE
VAE
RFF
0.66 + 0.01
0.72 0.00 0.74 : 0.00 0.74 + 0.00
0.72 0.79 0.00 0.82 0.01
0.84 0.0
Laplacian
0.64 1: 0.01
0.71 = 0.00 0.73 1: 0.00 0.73 +: 0.00
0.69 0.02 0.02
0.75 + 0.22 0.80 0.00 0.83 : 0.01
Two Stage
0.67 # 0:01 0.69 = 0.00 0.74 + 0.00 0.74 0.00
0.66 + 0.03 0,71 0.00 0.84 0.02
Lincar
0.67 = 0.01
0.70 = 0.01 0.71 0.00 0.71 = 0.00
0.72 0.03 0.03
084 0.02
0.84 0.01
Spline
0.63 1: 0:01 0.70 = 0.01 0,71 0.00 0.71 0.00
0.67 0.0 0.77 0.01 0.83 0.02 0.84 + 0.01
TOVAE
RFF
0.63 1: 0.01 0.69 = 0.01 0,71 0:00 0.00 0.71 0.00
0.68 1: 0.03 0,79 0.0 0,22 0.01 0.84 0.01
Laplacian
0.61
0.69
0.70 0:00 0.70
0.74
0.80
0.01
Two Stage
0.63 0:01 0.67
0.71
0.71
0.73 0.72
0.80 0.02 0.84
CBM [28]
0.64 : 0:01 0.74 0.01
0.86
0.92
0.53
0,60
0.75
0.89 0.01
CEM [S7]
0.66 0.11 0.75 0.01 0.87 0.00 0.92 0.00
0.62 0.04 0.68 0.0
0.82
0.01
0.89 0.01
HardCBM [17]
0.64 1 0.01 0.72 0.01 0.86 1 0.00 0.92 0.00
0.50 0.04
0.58 1: 0.02 0.71 0.02 0.90 0.01
Temporal Causal3Dldent Dataset
Linear
0.81 :
0.85
0.85 0.00
0.00
0.74 0.06 0:75
0.80
0.04
Spline
0.80
0.87
0.88 0.89
0.74
0.80
0.82
0.83 0.03
CITRISVAE
RFF
0.71
0.81
0.84
0.85 0.01
0.70 0.06 0.75 0.79 0.04 0.81
Laplacian
0.76 : 0.01 0.84 0.01 0.87 0.00 0.88 0.00
0.72 0.06 0.78 0.04 0.80 0.00 0.82 0.03
Two Stage
0.76 + 0.01
0.77 0.02 0.86 0.1 0.86 0.00
0.72 0.5 0.72 0.06 0.77 0.04 0.79 = 0.04
Lincar
0.81 1: 0.01
0.84 0.00 0.85 1 0.00 0.85 0.00
0.74 0.06 0.76 0.05 0.78 0.05 0.80 1 0.04
Spline
0.77 = 0.01
0.84 0.00 0.86 0.00 0.87 0.00
0.73 0.06 0.78 0.04 0.09 0.81 = 0.04
iVAE
RFF
0.75 +: 0.01
(0.1 0.01 0.85 0.00 0.86 + 0.00
0.69 0.06 0.75
0.05 0.78 0.04 0.80 + 0.04
Laplacian
0.74 1: 0.01
0.81 = 0.00 0.85 0:000 0.86 1: 0.00
0.69 1 0.7 0.76
0766:000 0.0 0.78
0.00 0.80 + 0.04
Two Stage
0.78 1: 0.01
0.82 : 0.01 0.86 1. 0.00 0.87 1 0.00
0.77 0.06 0.75 1 0.05 0.75 0.05 0.78 0.05
CBM [28]
0.58 1: 0.01 0.65 = 0.01 0.73 0.00 0.82 0.00
0.57 1 0.22 0.53 0:00 0.68 0.044 0.78 0.05
CEM [57]
0.61 + 0.01
0.64
= 0.01 0 0.82 0.00
0,64 004 0.67
00770000
0.72 0.78 0.05
HardCBM [17]
: 0.01 0.65 = 0.01 0.74 0.83
0.57 0.0 0.02
0.03
0.77 0.05
59

---

Table 12: NIS and OIS scores when using binary concepts created based on the ground truth latent variables in the Action/Temporal sparsity datasets and the Temporal Causal3Dldent datasets. The (n1) indicates the number of train and test points used in each column. We report the mean and standard deviation over 10 random seeds in each column and we indicate the best method by bold.
OIS-score +. (n)
NIS-score 4 (n)
Model
Method
20
100
1000
10000
20
100
1000
10000
Action Sparsity Dataset
Lincar
0.63 1 0.01 0.40 1 0.00 0,15 0.00 0.12 1 0.00
0.50 0.00
0.50
1 0.00
0.50
0.00
0.50
0.00
Spline
0.63 0:01 0.40 = 0.00 0.15 0.00 0,12 0.00
0.46
0,50
1:50:000
0.50
+:00
0.00
DMS VAE
RFF
0.64
0.39
0.14
0.12
0.47
0.50
0.50
0.00
0.00
Laplacian
0.64
0.40
0.14
0.11
0.47
002
0,50 1:50::00
0,50
Two Stage
0.84 1: 0.02 0.42 0.1
0.15 0.00 0.13 0.00
0.47 0.03 0.57 0.02 0.64 0.00 0.65 0.00
Linear
0.63 0:01 01.40 = 0.00 0.29 1:999:00 0.00 0.26 0.00
0.50
0.00 0.00 0.50 0.00
Spline
0.63 + 0.01 0.40 0.00 0.28 + 0.00 0.26 + 0.00
0.50 0.00 0.50 0.00 0.50
0:00 0.50 0.00
IVAE
RFF
0.64 = 0.01 0.39 0.00 0.28 = 0.00 0.26 0.00
0,45 = 0.02 0.50 0.00 0.50
0.00 0.50 0.00
Laplacian
0.64 1: 0.01 0.40 :: 0.00 0.27 1: 0.00 0.25 1: 0.00
0.47 1: 0.22 0.50 0.00 0.00 0.50 0.00 0.00 0.50 1.500.10 0.00
Two Stage
0.92 + 0.02 0.47 = 0.01 0.29 + 0.00 0.27 + 0.00
0.49 - 0.22 0.63 0.01 0.78 0.01 0.81 + 0.00
CBM [28]
0.92 : 0.01 0.43 = 0.01 0.11 1: 0.00 0.07 + 0.00
0,56 + 0.03 0.03 0.63 : 0.01 0.63
0.00 0.59 0.00
CEM [S7]
0.90 = 0.03 0.47 = 0.01 0.40 : 0.01 0.57 0.01
0.59 0.02 0.64
0.01 0.71
0.01
0.81 = 0.01
HardCBM [17]
0.92 1 0.02 0.46 0.01 0.12 1: 0.00 0.06 0.00
0.57 0.22 0.63
0.01 0.64
0.00 0.59 1. 0.00
Temporal Sparsity Dataset
Linear
0.62 = 0.01 0.39 0.00 0.26 0.00 0.23 = 0.00
0.50 0.00 0.50 0.00 0.50
0.00
0.50
0.00
Spline
0.62 1: 0.01 0.39 0.00 0.26 1: 0.00 0.23 1: 0.00
0.47 1 0.03 0.50 0.00 0.50
1:
0.50 0.00
DMS-VAE VAE
RFF
0.63 + 0:01 0.38 = 0.00 0.25 :: 0.00 0.23 + 0.00
0,50 + 0.47 0.1
0,50
0,50 0.00
Laplacian
0.64 +: 0.01 0.39 = 0.00 0.26 :: 0.00 0.22 + 0.00
0.49 + 0. 0.50 0.00 0.50
+:00000000000000000000000000
0.50 0.00
Two Stage
0.88 +: 0.02 0.45 = 0.02 0.26 +: 0.00 0.23 0.00
0,52 003 0.62 0:0 0.76 0:00
0.80 0.00
Lincar
0.62 1 0:01 0.39 : 0.00 0.27 1: 0.00 0.25 0.00
0,50 0.00 0,50 1
0,50 0.00 0.50 0.00
Spline
0.62
0.39
0.27 1: 0.25
0.50
0.00
0,50
1:500:0.000
0.00
TCVAE
RFF
0.63 0:01 0.38
0.27
0.25
0,50 0,50
0.50
0,50
Laplacian
0.64 0.01 0.39 0.00 0.27 0.00 0.24 0.00
0.00 0.50 0.00 0.00 0.50
Two Stage
0.91 1 0.02 0.47 0.01 0.28 1 0.00 0.24 0.00
0.49 0.00 0.63 0.01 0.79
0.00 0.82 1 0.01
CBM [28]
0.96 0.01
0.44 0.02
0.11 1: 0.00 0.05 0.00
0.55 0.20 0.63
0.01 0.61
0.00 0.57 0.00
CEM [57]
0.95 : 0.02
0.46 0.01 0.38
+ 0.01 0.54 0.01
0.53 000 0.65 0.00
0.71
0.01
0.81 = 0.01
HardCBM [17)
0.93 = 0.01 0.44 0.01 0.12 1:12000 0.00 0.05 0.00
0.54 A.44 0.62
0.62 0.00 0.57 0.00
Temporal Causal3DLdent Dataset
Lincar
0.69 : 0.02 0.44
0.19
0.16 0.00
0,46 0,50
0.50
0.50
0.00
Spline
0.69 0.02 0.44 0.00 0.09 0.00
0.41 0.03 0.50
0.00
0.50
CITRISVAE
RFF
0.65 1 0.01 0.43 0.00
0.00
0.13 0.01
0.41 0.00 0.50 0.50
10000.000000
0.50
Laplacian
0.64 = 0.01 0.40 0.02 0.00 0.09 + 0.00
0.44 0.50
0.00
0.50
Two Stage
0.88 +: 0.03 01.66 0.02 0.16 0.01
0.14 + 0.01
0.46 0.0 0.0
0.00
0.59 0.00
Linear
0.69 + 0.02 01.44 0.01 0.07 0,14 0.00
0.47 +.0 0.50
0,50
0.30
Spline
0.69 + 0.02 0.43 = 0.04 0,17
0.13 0.00
0.42 0.03 0,50
0.50=00000
0,50
0.50 0.00
iVAE
RFF
0.65 1: 0.01 0.42 = 0.02 0.00 0.13 0.00
0.47 0.0 0.50
0,50
1::00:00000 0.50 0.00
Laplacian
0.64 + 0.01 0.43 = 0.02 0.00 0.13 0.00
0.45 0.0Z 0,50
0.50
0.50 0.00
Two Stage
0.89 + 0.46 = 0.03 0.00 0.13 0.00
0.46 0.00
0,55
0.01
CBM [28]
1.00 = 0.03 0.48 0.02 0.25 0.00 0.18
0.50 0.01
0,58
00.00
0.00
CEM [57]
0.97 0.51
0.36 0:01 0.49
0.50 1:03 0.59 0:00
HMrdCBM [17]
0.96 0.47
0.24
0.16
0.51 000
0.55
0.58
0.57 0.00
60

---

Linear
Spline
RFF
Laplacian
Two Stage
10a
10a
Execution (x)
10
10
100
10-2
10
10a
10a
10a
10
10a
10a
10*
Nr: data points (n)
Nr. data points (1)
Figure 22: Execution times of the baseline and multiple versions of our estimator the causal variables and encodings learned based on eetion/Temporal Sparsity Dataset and the Temporal DIdent dataset
Linear Spline
RFF
Two Stage
CEM
Laplacian
CBM
CBM AR
10a
Exocution (s)
10a
10a
10a
10a
10
10
10-2
10a
10a
10a
10a
10a
10'
points (n)
points (n)
Figure 23: Exccution times of the concept-based models and multiple versions of our estimator on the classification downstream tasks based the Action/Tempont Sparsity Dataset and the Temporal Cidsat dataset
61


Reviews:
[{"review_id": "j6vk7W8vOx", "reviewer": "Reviewer_rz7E", "summary": "This work studies the problem of extracting interpretable concepts from data. This problem is broken into two steps. The first step is to extract latent causal generative factors from data, which is abstracted away as causal representation learning and is not addressed in this work. The paper focuses on the second step, which is to recover interpretable concepts from these latent factors. For this latter step, two approaches are proposed with theoretical guarantees. The first one is to use group regularized linear regression and the work presents sample complexity guarantees (w.h.p). The second one is to kernelize this procedure using basic ideas from RKHS theory and the paper obtains asymptotic guarantees borrowing results from high dimensional statistics. Experiments on synthetic and image benchmarks show the validity of their results.", "questions": "None", "limitations": "Limitations have been discussed.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths:\n\n- While the field of causal representation learning is rapidly advancing, sample complexity guarantees are far less studied, so this work is a relevant and important direction in progressing the field.\n\n- The main claims seem to be supported well with proofs, and the experimental details are well hashed out, and the paper also probed various aspects of the experimental settings. I also appreciate the detail in the proofs which make them easy to follow, as well as the extensive plots and tables.\n\n## Weaknesses:\n\n- The assumption that the ground truth causal variables correspond to the concepts that we would like to learn about (even the minor modifications in appendix A that blocks of causal variables correspond to a concept), seems very strong. It's not clear this will be the case and indeed, the field of causal representation learning aims to recover these causal factors regardless of their interpretability.\n\n- Similarly, assuming that the mixing function can be recovered by CRL techniques (assumption 3.2) is abstracting away the difficulty of the problem at hand. It's this precise step that requires various assumptions such as availability of interventional data, therefore this limits the technical contributions of the results of this paper.\n\n- The experimental results validate the theory to some extent, however the main drawback is that they're still somewhat simple and limited -- they mostly work with toy data (including misspecification) and semi-synthetic setups like causal3DIdent. However, this is in line with most prior works in this direction.", "was_revised": true, "final_justification": "I read the other reviews and the authors' responses. The authors clarified my questions satisfactorily.", "rebuttal": "We thank the reviewer for their thoughtful and encouraging feedback and discuss the weaknesses mentioned in the review.\n\n> The assumption that the ground truth causal variables correspond to the concepts that we would like to learn about (even the minor modifications in appendix A that blocks of causal variables correspond to a concept), seems very strong. It's not clear this will be the case and indeed, the field of causal representation learning aims to recover these causal factors regardless of their interpretability.\n\nWe agree that this is one of our main limitations and, as mentioned in the Conclusions, in future work we would like to generalize it to the setting in which the concepts are abstractions of the ground truth variables, which could cover the case in which we have a very fine-grained ground truth variable (e.g. the position of an object) and a high-level interpretable concept (e.g. “top left”).\n\nIn general, we need this assumption mostly to have theoretical guarantees on the fact that the learned representation will be disentangled and we do not use any of the causal semantics in the learned representations. So in principle, if in the future new “non-causal” methods that provide the same theoretical guarantees for dependent variables are developed, we could also use them in this pipeline.\n\n> Similarly, assuming that the mixing function can be recovered by CRL techniques (assumption 3.2) is abstracting away the difficulty of the problem at hand. It's this precise step that requires various assumptions such as availability of interventional data, therefore this limits the technical contributions of the results of this paper.\n\nThis is indeed the main technical challenge in CRL methods, but in our case the technical challenge is the combination of techniques from many different fields. In particular, our main goal is to bridge the results from CRL, XAI and high-dimensional statistics to provide theoretical guarantees to concept-based models.  Especially the fields of CRL and XAI have often developed similar ideas and facing similar or complementary challenges, but seem to have few interactions. For CRL, this would provide a notable downstream task with real-world applicability. For XAI, we hope to provide a principled approach to CBMs that would not suffer from spurious correlations between concepts and depend on a large number of labelled data.\n\nMoreover, our empirical results seem to confirm that our methods could work also in case of large correlations between the learned representations, e.g. if the perfect disentanglement is not achieved by the CRL methods. This suggests that our theory might be currently too conservative, so in future work we plan to extend it and test it also on the settings in which Assumption 3.2 doesn’t hold.\n\nA small side note regarding the availability of interventional data, in principle our pipeline works also with CRL methods that do not need interventions, but for example require parametric assumptions, e.g. iVAE in settings for conditional exponential priors and assuming an auxiliary variable $u$ for which $z_i$ and $z_j$ are conditionally independent, or settings with polynomial mixing functions or independent support (see also Table 1 in [Ahuja et al. 2023]), or multi-view settings [Yao et al. 2024].\n\n> The experimental results validate the theory to some extent, however the main drawback is that they're still somewhat simple and limited -- they mostly work with toy data (including misspecification) and semi-synthetic setups like causal3DIdent. However, this is in line with most prior works in this direction.\n\nThis is a fair point and is indeed in line with most prior works in CRL. As mentioned in the Conclusions, it would be interesting to use real-world datasets from CBM, but since in this dataset the concepts are discrete, we would need to substantially change our setup, adapt our theory to this setting and evaluate with discrete CRL methods, which are currently quite limited to very specific assumptions, e.g. the assumption that different variables always affect different parts of an observation [Kong et al. 2024].\n\n[Kong et al. 2024] Kong, Lingjing, et al. \"Learning discrete concepts in latent hierarchical models.\" Advances in Neural Information Processing Systems 37 (2024): 36938-36975."}, {"review_id": "HPCPJ7wy3K", "reviewer": "Reviewer_KPGk", "summary": "The paper proposes a two-stage framework for *sample-efficient concept learning* that couples recent Causal Representation Learning (CRL) encoders with an *alignment stage* that maps latent causal variables to user-defined concepts.  Stage 1 leverages any off-the-shelf CRL method to obtain latent variables that – under suitable identifiability conditions – recover the ground-truth causal factors up to permutation and element-wise transformations.  Stage 2 estimates this unknown permutation and the per-variable transforms with only a small number of concept labels.  Two estimators are introduced:\n\n* **Linear estimator** based on a Group-Lasso regression that comes with finite-sample, high-probability guarantees on both parameter error and exact permutation recovery (Thm 4.2) and an explicit sample-complexity bound.\n* **Kernelized estimator** that removes linearity restrictions and remains consistent under general invertible per-variable diffeomorphisms (Thm 5.2).\n\nExperiments on synthetic toy data, Action/Temporal-Sparsity time-series benchmarks, and the image dataset *Temporal-Causal3DIdent* show accurate permutation recovery and competitive downstream classification accuracy with substantially fewer concept labels than Concept Bottleneck Models (CBMs) and their variants.", "questions": "1. Can you provide quantitative evidence (or theory) for how permutation recovery degrades when CRL variables contain mixtures of causal factors?  \n2. Given that empirical $\\rho$ exceeds Assump. 4.1, do you observe systematic failures when correlations approach 0.9–0.99?  Could a debiased estimator or decorrelation pre-processing tighten the finite-sample guarantees?\n3. Have you considered extending the alignment to *sets* of latent variables (blocks) mapping to a single concept?  \n4. Would the method scale to CUB or CheXpert concepts if CRL encoders were available?  \n5. Do you foresee extending Theorem 4.2 to the logistic case, or providing a finite-sample excess-risk bound for binary concepts?", "limitations": "Yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "#### Strengths\n\nThe work addresses an important bottleneck of CBMs – heavy dependence on concept supervision – by showing how unsupervised CRL can substitute for most concept labels.  Theoretical analysis is unusually thorough: the linear estimator enjoys *finite-sample* guarantees with an explicit tuning rule for the regulariser, while the kernelised version attains asymptotic consistency.  The presentation of assumptions, proof sketches, and ablation studies on the effect of regularisation, dimension, correlation, and data size is clear and technically solid.  Empirically, the method achieves lower Oracle Impurity Scores and higher label accuracy than strong CBM baselines when concept labels are scarce.  The code-free alignment step is light-weight and could be dropped into existing CRL pipelines, potentially broadening the impact of CBMs in domains where labels are expensive.\n\n#### Weaknesses\n\n1. **Over-idealised core assumption.** All theoretical guarantees hinge on the *pre-trained CRL achieving almost perfect disentanglement*, i.e. recovering each latent $G_i$ up to permutation + one-dimensional invertible transform.  In practice, available CRL methods on complex images or temporal data often yield *partially* disentangled variables and are sensitive to permutation/grouping instability; once this fails, the subsequent Group-Lasso or kernel step may learn mis-aligned coefficients, yet the paper provides no error bounds or robustness analysis beyond a brief empirical remark that “it still works”.\n2. **Correlation assumptions mis-match practice.** Finite-sample theory requires the *weak-correlation* condition (Assump. 4.1) for Group-Lasso. The authors admit that the empirical $\\rho$ in experiments “far exceeds” the theoretical regime, but no new convergence analysis is given, leaving the most central finite-sample bound without guarantees in realistic high-collinearity regimes.\n3. **Equating human concepts with causal variables is too strong.** The framework *assumes* that target “concepts” coincide with the data-generating latent causes (or monotone transforms).  In many domains (medical imaging, NLP) human-level concepts are abstractions over multiple causal factors.  While future work on “causal abstraction” is mentioned, the present proofs, estimator design and all experiments rely on the stronger assumption.\n4. **Narrow experimental coverage.** Three out of four datasets are synthetic and the sole real-data benchmark (Temporal-Causal3DIdent) is a rendered scene dataset.  There is no evaluation on natural images, text, or genuine human-annotated concept benchmarks, so interpretability remains at the metric-level (OIS/NIS) rather than user utility.\n5. **Discrete concepts lack theory.** Although the paper states that the method “also handles binary concepts” via a logistic Group-Lasso variant, there is *no* consistency or sample-complexity analysis for discrete or categorical variables; multi-class concepts such as “object shape” in Causal3DIdent are ignored in practice, leaving an important gap for real-world deployments.", "was_revised": true, "final_justification": "I am inclined toward a borderline accept.", "rebuttal": "We thank the reviewer for an insightful evaluation of our paper.\n\nWe acknowledge the listed ``weaknesses\", but we view them as current limitations, which can potentially be addressed in future work. In theoretical work, there is often a gap between what can be mathematically proven and what happens in practice. Ideally this gap is closed over time, but even if it isn’t then theory is still helpful: it points out which aspects play an important role, it provides a sanity check that the approach works as advertised at least in simple cases, and it makes very precise and general claims that do not depend on the specific data set under consideration.\n\n## Over-idealised core assumption\n\nFirst, a clarification: our framework, including the theory, already covers **partially disentangled** variables in the sense of “block-identifiability”, if we know which factors are grouped together, as shown in the Appendix. \n\nAs mentioned in the Conclusions, extending our theoretical results to imperfect disentanglement or larger correlations between concepts is interesting future work. Note that in the benchmarks that we consider, which come from CRL literature, existing CRL methods already provide a good disentanglement and work well enough for our approach to beat existing CBM methods, as also evidenced by the low OIS scores. \n\n> Can you provide quantitative evidence (or theory) for how permutation recovery degrades when CRL variables contain mixtures of causal factors?\n\nWe performed a new experiment on the synthetic data, where we sample $d/2$ pairs for the $M_j$ variables and during the alignment step, we feed mixed versions of these causal variables to the estimator, $M_j’= (1-a)M_i + a M_j$. We see that the permutation error goes to 1 whenever $a>0.5$, but we remain almost errorless when we consider the permutation correct if the matching identifies the pairs correctly. The columns indicate the mixing parameter values, the experiments were run over 10 seeds and we provide the mean and the standard deviation.\n\n### Mean Permutation Error Table\n|          Feature map          |     .00     |    0.1      |    0.22     |     0.33    |     0.44    |    0.56     |    0.67     |    0.78     |    0.89     |    1.0      |\n|:-----------------------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n| Linear                        | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |\n| RFF(p=8)                      | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.04 ± 0.01 | 0.99 ± 0.01 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |\n| Splines (n_knots=4, degree=3) | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |\n| Kernel (RBF)                  | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 | 1.00 ± 0.00 |\n\n### Paired Mean Permutation Error table:\n|          Feature map          |    0.00     |    0.11     |    0.22     |    0.33     |    0.44     |    0.56     |    0.67     |    0.78     |    0.89     |    1.00     |\n|:-----------------------------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|:-----------:|\n| Linear                        | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |\n| RFF(p=8)                      | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.03 ± 0.01 | 0.03 ± 0.01 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |\n| Splines (n_knots=4, degree=3) | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |\n| Kernel (RBF)                  | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 | 0.00 ± 0.00 |\n\n\n## Correlation assumptions mis-match practice\n\nIt is clear that strongly correlated concepts cannot be distinguished with a limited amount of data using any method, so some assumptions are unavoidable. But there indeed remains a gap here between what we can prove and what we would like to prove. We suspect that this empiric robustness to correlations comes from the matching procedure, but we have not been able to prove it yet.\n\n> Do you observe systematic failures when correlations approach 0.9–0.99?\n\nYes, we do, but if the correlations between the concepts are that high, e.g. 0.99, then the concepts are (almost) deterministically related. If this is due to the underlying causal variables, even most CRL methods would not be able to disentangle them, since they often assume faithfulness (which implies no deterministic relations). If this is due to the imperfect disentanglement, then we might need to assume that the true factors have a substantially higher correlation with the correct concept than the spurious ones, and this might increase the necessary sample size for the concept labels.\n\n## Equating human concepts with causal variables is too strong\n\nWe agree with this point. As mentioned in the Conclusions, we intend to actively pursue a causal abstraction approach, but this would be too much in a single paper.\n\n## Narrow experimental coverage\n\nWe have focussed here on data sets from the CRL literature with a clear ground truth. Moving to more natural data would indeed be an important threshold to cross in follow-up work.\n\n> Would the method scale to CUB or CheXpert concepts if CRL encoders were available?\n\nThis would first require developing the theory for binary/discrete concepts, and using CRL methods for discrete causal variables, which are currently quite limited to very specific assumptions, e.g. the assumption that different variables always affect different parts of an observation [Kong et al. 2024].\n\n[Kong et al. 2024] Kong, Lingjing, et al. \"Learning discrete concepts in latent hierarchical models.\" Advances in Neural Information Processing Systems 37 (2024): 36938-36975.\n\n## Discrete concepts lack theory\n\nBesides the points regarding the lack of general purpose discrete CRL methods as described above, in general we believe there is no fundamental obstacle to extending our theoretical results to the logistic Group Lasso. Indeed, the results in [Meier, vd Geer, Bühlmann, 2008] would allow us to prove that the permutation can be recovered from a logistic regression on binary labels with the group lasso regularization. They also comment that their results can be extended to generalized linear models, which indicates we can apply the result to the feature maps we consider.\n\nMeier, van der Geer, Bühlmann, The Group Lasso for Logistic Regression, Journal of the Royal Statistical Society Series B: Statistical Methodology 70.1 (2008): 53-71.\n\n> “Do you foresee extending Theorem 4.2 to the logistic case, or providing a finite-sample excess-risk bound for binary concepts?”\n\nYes, as we said in the previous results. Consistency and variable selection results exist for logistic regression with the group lasso regularization, which would allow us to prove a similar result as in the Appendix, but for the logistic loss, which is indeed the intention in future work. However, there were some non-trivial steps involved in proving this that it was beyond the scope of one article. \n\n## Could a debiased estimator or decorrelation pre-processing tighten the finite-sample guarantees?\n\nDebiasing might indeed be useful to avoid underestimating the norm of the parameter vector. For the theory, it is known that the resulting estimates are asymptotically normally distributed, which might provide improved asymptotic guarantees, but there is some doubt if the method is reliable for finite samples [Wüthrich, Zhu, 2023].\n\nDecorrelation would not resolve this issue, as identification of the correct permutation relies on the sparse structure of the true parameter in the linear model. Adding a decorrelation step would affect the true parameter as well and in particular it would destroy this sparse structure of the true parameter and permutation identification remains infeasible. On top of the loss of sparsity. Decorrelating would involve inverting a correlation matix, which will be close to non-invertible if the correlation is high. So this procedure would face numerical instabilities in this regime.\n\n[Wüthrich, Zhu, 2023] K. Wüthrich, Y. Zhu (2023). Omitted Variable Bias of Lasso-Based Inference Methods: A Finite Sample Analysis. The Review of Economics and Statistics, 105 (4): 982–997.\n\n## Extending the alignment to sets of latent variables (blocks) mapping to a single concept\n\nYes, the most general version of our theory, which we report in the appendix for readability, can already handle blocks of causal variables matching to one concept."}, {"review_id": "mHGXlj9Q7K", "reviewer": "Reviewer_G2in", "summary": "This paper proposes using Lasso regression to map from a pretrained representation $M$, stemming from some exisitng VAE-based model that has been trained to discover disentangled causal factors, into a predefined set of concepts $C$, via a sparse mapping $\\alpha$. Sparsity is very important in this task, since each $M_i$ only becomes semantically interpretable if it maps to a single $C_j$. This is specifically formalized using Group Lasso, in order to allow for multidimensional factors in $M$ to be active together (for instance, 3D position as a factor).\nThe results on several synthetic datasets, where the underlying factors are known by design, show the advantage of the proposed approach with respect to regular concept bottlenecks.", "questions": "My main request for the authors would be a clear delineation of where the claimed novelty lies, including a title that clearly states it (or at least an abstract).\nIn addition, I would also like to see the response of the authors to the raised weakness points.", "limitations": "Some of the main limitations are mentioned in the conclusions section, specifically related to the fact that an actual alignment between the real causal factors and human-understandable concepts may not exist. It is also mentioned that the method is not meant to work with correlated factors, although it is claimed it works better than expected in this setting. The one limitation that is not mentioned is the fact that human-annotated concepts are generally hard to obtain. In addition, all datasets used are synthetic, which raises questions about the usability of the approach on real data.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The main strength of this submission lies in the numerical results, which show the clear advantage of using disentangled features for downstream tasks.\n\nOn the weaknesses side:\n1. I found it tricky to understand what the contribution is. This may be due to my low familiarity with some of the elements in the paper. In the beginning, based on the title and abstract, I thought that the main contribution would be around CRL, but the it turns out that the CRL component is taken as a pretrained representation. This means that the novel aspect should lie with the learning of $\\alpha$, for which two variants of Group Lasso are used. I apologize if I misunderstood this aspect, but it would be helpful to state more clearly which element is novel. Is it the application of Group Lasso to this specific task that is novel? Are some of the provided proofs novel?\n2. This issue is reflected in the title itself, where the three lines suggest three different elements where the main contribution is (sample efficiency? Theoretical guarantees? No need for interventions?). By reading the rest of the paper, I could not make up my mind about this.\n3. As noted by the authors, CITRIS-VAE already recovers interpretable factors. What is then the contribution of the proposed approach with respect to that?\n4. The numerical comparisons are against several CBMs. However, there are no comparisons against the raw factors discovered by the various VAEs used. Does the proposed matching between $M$ and $C$ lead to differences in the performances?\n5. I guess the use of Group Lasso is due to the existence of structured, multidimensional causal factors. Is this so? I could not find a clear motivation in the paper.\n6. The authors claim that their method performs “better than the theory predicted”. I would like to seem some insights into how this is the case, since I could not find any in the discussion section.", "was_revised": true, "final_justification": "The authors have clarified most of the doubts I had. Since several reviewers are positive with this paper, I don't want to stand on the way, since I'm not too familiar with several of its main elements.", "rebuttal": "We thank the reviewer for their careful reading of our paper. To answer the points raised:\n\n## Contribution and Title \n\nThe main contribution of the paper is the pipeline presented in Figure 1(Left), where we leverage a pretrained CRL method (potentially trained on large numbers of unlabelled data) to propose a Concept Bottleneck Model approach that has theoretical guarantees, e.g. no concept leakage (a guarantee that we get by using the CRL method as a first step) and error bounds on the coefficients of the alignment function that extracts the learned concepts given a small number of labelled samples. \n\nThe task itself is novel, since there are no other methods that (i) provide **theoretical guarantees** on the learned concepts in terms of error bounds, (ii) separate the CBM approach in two steps (of which one is unlabelled) and hence can use only a very small number of labelled data as opposed to standard approaches, hence being **sample efficient** (iii) provide **theoretical results** on how many labels would be needed for a certain accuracy. At **no point in this pipeline we require an intervention** on the concepts to learn the alignment with the concepts, since we have an already disentangled representation from CRL.\n\nThe use of the GroupLasso is the basis for our theoretical results on the error bounds and sample complexity, but in principle other methods could be used. All of the proofs are novel, since this is a novel setting also in statistical learning theory, i.e., learning a permutation and an element-wise transformation from the learned representations to the concepts. Since we use Group lasso, we leverage some of the results in high-dimensional and non-parametric statistics as a starting point.\n\n## CITRIS-VAE does not recover interpretable factors\n\nAs other CRL methods, CITRIS-VAE is an unsupervised representation learning method and it cannot recover interpretable factors, since the learned representations in the latent space are not semantically meaningful (i.e., they are embeddings of the ground truth variables). So we still need to learn an alignment function between these representations and the concepts by using concept labels, and part of our contribution is how to do this efficiently in terms of concept labeled samples and with theoretical guarantees. As opposed to other CRL methods, CITRIS-VAE also recovers the correct permutation, which simplifies learning this alignment, but at the cost of having knowledge of which variable is intervened upon at each timestep in the dataset for the pre-training phase. Since knowing which interventions are performed or having interventions at all in the data is in general not true for other CRL methods, in our evaluation we have ignored this permutation and compared CITRIS-VAE with the other methods on the same setting.\n\n## Additional comparisons against raw factors would not make sense\n\nWe are mostly interested in the quality of the learned concepts and the raw factors do not have the right permutation or any semantic meaning, so it wouldn’t make sense to compare them directly to how accurately they predict the concept labels, except through learning a mapping from them to the labels with supervision, which is exactly the point of our paper. \n\nThe CRL literature also uses labels in the evaluation, but in that case the mappings are learned ad hoc through NNs and with hundreds of thousands of labels. We instead provide a principled approach with theoretical guarantees for this task.\n\n## Why Group Lasso: non-linear estimation\n\nIn the main paper, we first use the GroupLasso so we can use $p$-dimensional feature maps $\\phi(M_j)$ (e.g. polynomials or splines) for each causal variable $M_j$ with the linear estimator, allowing us to model some non-linear relationships in that setting as well. All the outputs of the feature maps for a specific $M_j$ represent a group that the Group Lasso will consider together, e.g. by considering all $p$ corresponding coefficients together. The kernelized estimator uses the same idea, but it generalizes it to non-parametric settings.\n\nFor the same reason, The Group Lasso is also useful for the more general case in which the $M_j$ are multi-dimensional, which we consider in the appendix.\n\n## Results better than the theory predicted\n\nOur theoretical results assume an upper bound on the correlation between the $M_j$, but in our experiments we observe that this upper bound can be significantly relaxed. We believe the reason is that the weighted matching in Equation $6$ is very robust to estimation errors in the $\\widehat{\\beta}_i^j$. For instance, it will find the correct matching if, for all $i$, $||\\widehat{\\beta}_i^{\\pi(i)}|| > ||\\widehat{\\beta}_i^j||$ for $j \\not= \\pi(i)$. This is why we write in Section 7 that we believe there may be room to strengthen the theory. We will add this discussion to the paper.\n\n## Limitations: human-annotated concepts are hard to obtain\n\nThis is true, but we believe this issue actually points to a strength of our framework, because we require much fewer concept annotations than existing concept-bottleneck methods by leveraging unsupervised CRL as part of our framework. The sample complexity we provide also informs practitioners how many labels are at least needed, which can help in reducing the cost of labelling, as not too many labels are gathered."}, {"review_id": "kznr8BS8yD", "reviewer": "Reviewer_8oyR", "summary": "This work utilizes causal representation learning methods to first learn causal variables related to a given task, and then maps those causal variables into interpretable concepts for use in a CBM. By first learning causal variables, the authors propose that with few concept labels they are able to train a CBM with less impurity and higher accuracy than prior work (CBM, CEM, HardCBM).", "questions": "- Could the authors explain Figure 20 in the appendix as well as the general computation cost in more detail? From what I understand, the proposed approach is a two-stage approach, yet is more computationally efficient than CBM which should naively learn concepts from annotated data. Similarly, if I utilize a more powerful backbone for CBM, can I outperform the proposed approach without needing to perform the additional complexity?\n- What do the authors mean by 'few concept labels' throughout the paper? Are fewer concepts used? I don't see this claim substantiated in the main results, as it seems each method is tested over a different number of training points.\n- Could the authors explain lines 295-297: \"Although CITRIS-VAE provides the correct permutation of the groups, we ignore it and perform a random permutation on the variables. We train an MLP for each of the 32 dimensions that predicts all causal variables. Based on the R2 scores of these regressions, we learn the group assignments.\" From what I understand the authors are saying the CRL method perfectly learns the true factors of variation within the dataset, is this the case?", "limitations": "Yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Concept leakage is a very important problem in CBMs, and learning disentangled concept representations is a very interesting idea for ensuring concepts mean what they say.\n- Results are strong on both synthetic datasets evaluated. \n\nWeaknesses\n- Both datasets are synthetically created and fairly toy problems. Do the causal representation learning methods just not scale? Why can the authors not evaluate on a classic dataset such as CUB-200? If the theoretical analysis is grounded in the fact that the CRL approach can learn to disentangle the factors of variation within the dataset, then how is learning the causal variables not enough already? Current CBMs offer a solution for when the learning task is too complicated for simpler methods to work effectively, but if this approach is limited to small-scale datasets then how could it be useful in practice?\n- From what I understand, the approach has many hyperparameters compared to typical concept bottleneck models. Is performance resistant to the selection of these hyperparameters? How were the parameters in Table 3 of the Appendix selected?", "was_revised": true, "final_justification": "Thanks for addressing my questions! While I am not very familiar with the CRL literature, I believe this paper offers an interesting connection between concept-based models and CRL approaches and I am happy to raise my score, even if it is unclear whether the CRL methods which the approach is built upon will be able to scale beyond toy problems in the future.", "rebuttal": "Thank you for your careful review, we will address each point in the following.\n\n> Both datasets are synthetically created and fairly toy problems. Do the causal representation learning methods just not scale? \n\nThe datasets we considered are from the CRL literature. For CRL methods it is fairly typical that they consider 10-20 causal variables to identify, since indeed they have issues scaling.\n\n> Why can the authors not evaluate on a classic dataset such as CUB-200? \n\nWe cannot evaluate on such a dataset, since the concepts are binary, and the very few discrete CRL methods are currently limited to very specific assumptions, e.g. the assumption that different variables always affect different parts of an observation [Kong et al. 2024], which generally don’t hold in this dataset.\n\n[Kong et al. 2024] Kong, Lingjing, et al. \"Learning discrete concepts in latent hierarchical models.\" Advances in Neural Information Processing Systems 37 (2024): 36938-36975.\n\n> If the theoretical analysis is grounded in the fact that the CRL approach can learn to disentangle the factors of variation within the dataset, then how is learning the causal variables not enough already? \n\nCRL methods are unsupervised, and can therefore only identify the factors up to an equivalence class of permutation and element-wise transformation. So even if the CRL method learns to to disentangle the factors of variation,  it will remain unclear what each learned variable means semantically. To actually learn the concepts it is crucial to also learn the alignment map with concept labels (see second stage of Figure 1). \n\n> Current CBMs offer a solution for when the learning task is too complicated for simpler methods to work effectively, but if this approach is limited to small-scale datasets then how could it be useful in practice?\n\nThe limitation is not the size of the data, but rather the number of latent dimensions/concepts because of the current CRL methods. Our main concerns are to address the two main problems with CMBs: a) reliability of the learned concepts, which should avoid spurious correlations and provide theoretical guarantees on the accuracy; and b) reducing the number of required concept labels by leveraging unsupervised CRL methods.\n\n> Number of hyperparameters in Table 3 and robustness to their selection:\nFor each CRL method, we select the same hyperparameters as the original CRL papers, also since we are evaluating on their original benchmarks. In particular Table 3 reports all of the hyperparameters from the DMSVAE paper. \n\n> Could the authors explain Figure 20 in the appendix as well as the general computation cost in more detail? From what I understand, the proposed approach is a two-stage approach, yet is more computationally efficient than CBM which should naively learn concepts from annotated data. \n\nIn Figure 20 we did not consider the CRL phase, since we assume that we get a pre-trained CRL encoder. If this phase was considered the total time would be 8+ hours $\\approx$ 29k seconds, since we can use hundreds of thousands of unlabelled data for the unsupervised CRL methods to have a good disentangled representation. On the other hand, the second step of our approach is generally more computationally efficient than the CBM approaches, as we only need to perform a regularized regression, instead of training a whole network.\n\n> Similarly, if I utilize a more powerful backbone for CBM, can I outperform the proposed approach without needing to perform the additional complexity?\n\nA more powerful backbone, e.g. different network architecture, for CBM would not help, because it would not address spurious correlations and it cannot significantly reduce the number of required concept labels, which are used for the whole pipeline in standard CBMs. \n\n> What do the authors mean by 'few concept labels' throughout the paper? Are fewer concepts used? I don't see this claim substantiated in the main results, as it seems each method is tested over a different number of training points.\n\nStandard CBMs use fully supervised learning to learn the concepts $C_i$. In addition to labels for the desired prediction target $Y$, CBMs therefore require labels for all concepts to be provided with the training data. In contrast, we use unsupervised CRL to identify the concepts up to an equivalence class, and then we only need $n$ concept labels to learn the alignment map $\\alpha$ (see Figure 1), which requires much fewer labels for the concepts $C_i$ to be provided with the training data. Table 1 shows how the methods perform for different values of the $n$ concept labels.\n\n> Could the authors explain lines 295-297: \"Although CITRIS-VAE provides the correct permutation of the groups, we ignore it and perform a random permutation on the variables. We train an MLP for each of the 32 dimensions that predicts all causal variables. Based on the R2 scores of these regressions, we learn the group assignments.\" From what I understand the authors are saying the CRL method perfectly learns the true factors of variation within the dataset, is this the case?\n\nNo, this is not the case. As other CRL methods, CITRIS-VAE is an unsupervised representation learning method and it cannot recover interpretable factors, since the learned representations in the latent space are not semantically meaningful (i.e., they are embeddings of the ground truth variables). So we still need to learn an alignment function between these representations and the concepts by using concept labels, and how to do this efficiently in the samples and with theoretical guarantees is part of our contribution.\n\nAs opposed to other CRL methods, CITRIS-VAE also recovers the correct permutation, which simplifies learning this alignment, but at the cost of having knowledge of which variable is intervened upon at each timestep. Since this is in general not true for other CRL methods, in our evaluation we have ignored this permutation and compared CITRIS-VAE with the other methods on the same setting."}]

Author rebuttal:

Meta-review: This paper explores the sample complexity of learning concepts in a limited setting where a perfect pre-trained decoder is available (i.e. perfect disentanglement), and the problem reduces to extracting concepts from the decoder. They consider both linear (finite-sample) and nonlinear (asymptotic) settings, and develop theoretical results in both settings. Experiments validate the theory.

Initially, reviews were somewhat mixed, but after discussion, all concerns were addressed and all reviewers support accepting the paper. Despite its limitations, the paper tackles an important problem, which will hopefully stimulate further extensions down the line.
Decision: Accept (poster)