Causal Abstraction Learning based on the Semantic Embedding Principle
Authors: Gabriele D'Acunto
Fabio Massimo Zennaro
Yorgos Felekis
Paolo Di Lorenzo
Venue: ICML 2025 poster (2025)
Keywords: structural causal models
causal abstraction
semantic embedding principle
Stiefel manifold
Riemannian optimization
Primary area: general_machine_learning->causality

Abstract: Structural causal models (SCMs) allow us to investigate complex systems at multiple levels of resolution.
The causal abstraction (CA) framework formalizes the mapping between high- and low-level SCMs. 
We address CA learning in a challenging and realistic setting, where SCMs are inaccessible, interventional data is unavailable, and sample data is misaligned.
A key principle of our framework is *semantic embedding*, formalized as the high-level distribution lying on a subspace of the low-level one. 
This principle naturally links linear CA to the geometry of the *Stiefel manifold*.
We present a category-theoretic approach to SCMs that enables the learning of a CA by finding a morphism between the low- and high-level probability measures, adhering to the semantic embedding principle.
Consequently, we formulate a general CA learning problem.
As an application, we solve the latter problem for linear CA; considering Gaussian measures and the Kullback-Leibler divergence as an objective.
Given the nonconvexity of the learning task, we develop three algorithms building upon existing paradigms for Riemannian optimization.
We demonstrate that the proposed methods succeed on both synthetic and real-world brain data with different degrees of prior information about the structure of CA.

Causal Abstraction Learning based on the Semantic Embedding Principle
Gabriele D'Acunto 12
Fabio Massimo Zennaro
Yorgos Felekis Paolo Di Lorenzo 12
Abstract
The Semantic Embedding Principle (SEP)
Structural causal models (SCMs) allow us in- vestigate complex systems multiple levels of resolution. The causal abstraction (CA) frame- work formalizes the mapping between high- and low-level SCMs. We address CA learning in a challenging and realistic setting, where SCMs are inaccessible, interventional data is unavailable, and sample data is misaligned. key principle our framework is semantic embedding, formalized the high-level distribution lying subspace of the low-level one. This principle naturally links linear CA to the geometry of the Stiefel mani- fold. We present a category-theorete approach to SCMs that enables the learning of a CA by find- ing morphism between the low- and high-level probability measures, adhering the semantic embedding principle. Consequently, we formu- late general CA learning problem. As appli- cation, we solve the latter problem for linear CA; considering Gaussian measures and the Kullback- Leibler divergence as an objective. Given the non- convexity of the learning task, we develop three algorithms building upon existing paradigms for Riemannian optimization. We demonstrate that the proposed methods succeed on both synthetic and real-world brain data with different degrees of prior information about the structure of CA. Code: https: / //1 //g//tttuh thub. con http//SAACOMM/C....c
Causal Abstractions must preserve high-leve causal knowledge when embedded in the low-level.
Re
V
Rixt
St(E, h)
RA
VoVT
(xx)
ld
Figure 1: Pictorial representation of SEP for linear CA. A linear map V belonging the Stiefel manifold embeds high-level causal knowledge x" into a low-level one, viz. X" identifying an embedded causal knowledge V (x"). Then, a linear CA V abstracts up (x*), yielding causal knowledge identical to x Notice that the arrow Id,^ underlines that commutativity holds only in one direction, that is, SEP does not imply to (xx) X
a widely adopted framework for causal reasoning (Pearl, 2009). While canonical causal theory focuses on a single SCM, scientific research often requires multiple represen- tations of the same system different levels of resolution. For example, biological processes can be studied at the molecular level (e.g., gene expression), cellular level (e.g.. metabolic pathways), or organism level (e.g., physiological responses), each offering a different view of the same under- lying system. Causal abstraction (CA) theory (Rubenstein et al., 2017; Beckers Halpern, 2019) formalizes mappings between SCMs different abstraction levels, enforcing rigorous consistency requirements. This makes CA a power- ful tool for transitioning between resolutions, synthesizing causal evidence, and selecting the most parsimonious repre- sentation for a given task. However, CAs are unknown in practice, underscoring the need for advancing CA learning from data (Zennaro et al., 2023).
1. Introduction
Causal modeling and reasoning are key to trustworthy and responsible AT (Ganguly al., 2023; Rawal al., 2024; Qi et al., 2024). Structural causal models (SCMs) provide
'Department of Information Engineering, Electronies and Telecommunications, Sapienza University, Rome, Italy "National Inter-University Consortium for Telecommunications (CNIT), Parma, Italy 'Department of Informatics, University of Bergen, Bergen, Norway 'Department of Computer Science, University of Warwick, Coventry, UK. Correspondence to: Gabriele Acunto <gaariele. uniromal.its.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Related works. Seminal works on CA have focused on defining and assessing given CA maps (Rubenstein al., 2017; Beckers & Halpern, 2019). Our approach builds the a-abstraction category- thhegory- theoretic framework introduced by (Rischel, 2020), which neatly separates the structural and functional components of the CA. From a learning per- spective, several methods have been proposed which rely on restrictive assumptions. In our work, we transform them into non-assumptions (NA). (Zennaro al., 2023) addresses the learning problem under (NA1) complete specification of SCMs, which, in reality, is rarely available. (Felekis et al., 2024) assumes (NA2) knowledge of causal DAGs, which are often unknown in many applications. (Dyer et al., 2024) relies on the (NA3) availability of interventional data, which may be infeasible or unethical obtain. (Kekić et al., 2024:
Regarding the latter, CRL treats low-level variables (e.g. pixels) as noncausal observations generated by latent high- level causal concepts, and aims to recover those concepts and their causal graph to improve interpretability and perfor- mance robustness (Schölkopf et al., 2021; Yang et al., 2021; Komanduri et al., 2022). This goal also aligns with recent developments that integrate causal reasoning into communi- cations. leveraging functorial formalism as well (Thomas et al., 2023; Thomas & Saad, 2023; 2024). By contrast, CA learning focuses mappings between SCMs, where the low- and high-level variables are causal and known, to enable causal knowledge transfer across abstraction levels.
Contributions. First, we introduce the semantic embed- ding principle (SEP) for CA, informally stating that in a well-behaved CA, embedding the high-level (coarser) causal knowledge into the low-level (finer) one and then abstract- ing it back enables perfect reconstruction of the high-level causal knowledge. Second, to formalize SEP categorically, we present an alternative CA, which allows us to focus on the semantic layer of an SCM. Third, we formulate a general CA learning problem based on SEP and (AI). Fourth, we tackle the linear CA case, showing that SEP naturally links the linear CA to the geometry of the Stiefel manifold, shaping the learn- ing process as a Riemannian optimization problem. As application, we consider the Gaussian setting with the Kullback- (KL) divergence as a measure of align- ment between the low- and high-level SCMs. Fifth, we for- malize and solve nonsmooth and smooth learning problems for linear CAs in this setting. For the former, we present the LinSEPAL-ADMM and LinSEPAL-PG methods; for the latter, the CLinSEPAL one. Our experiments on synthetic and brain data, across different levels of prior knowledge, confirm good performance of the proposed methods.
Massidda et al., 2024) make (NA4) functional assumptions on the SCMs, such as linearity. (Massidda et al., 2024) implicitly assumes (NA5) alignment between data gener- ated by two models, which requires tight coordination in sample collection. Conversely, we work under the realistic and pragmatic assumption that (A1) at least partial prior knowledge of the structure of a CA is available. (A1) is met in different application domains, such as neuroscience. For instance, consider the learning of a CA between two brain SCMs, the first referring to some brain region of inter- est (ROIs). the second to the brain lobes. A map between ROIS and brain lobes is implicitly defined by the location of ROIS, and so it would be natural to try to exploit such prior knowledge when learning the CA. Finally, we build on top of different continuous optimization frameworks, working in both the Euclidean and Riemannian spaces. Specifically, when dealing with a nonsmooth Riemannian problem, we leverage the manifold alternating direction method of multi- pliers (MADMM, Kovnatsky et al., 2016) and the manifold proximal gradient (ManPG, Chen et al., 2020). They are the Riemannian counterparts of the ADMM (Boyd et al., 2011) and PG (Parikh et al., 2014). Additionally, when deal- ing with a smooth, constrained, Riemannian problem, our solution combines the splitting of orlogonality constraints (SOC, Lai & Osher, 2014), the ADMM, and the successive convex approximation (SCA, Nedić et al., 2018) methods.
-tteorrrtt framework for
Our work is a first step to bridging the gap between CA learning methods and real-world applications.
2. Background on Causality and Abstraction
This section provides the notation and key concepts related to causal modeling and abstraction theory.
Related to CA learning are mechanistic interpretability via causal abstraction (Bereska & Gavves, 2024) and causal representation learning (Schölkopf et al., 2021). Regarding the former, Geiger et al. (2021; 2024) use CA to explain DNNs by treating the DNN as a low-level - to be read as black-box - SCM and aligning it via interchange in- tervention training (ITT) with a high-level - to be read as unnnrtt-dderst - SCM, built from theoretical and empirical modeling work. This setting where two SCMs share structure but differ only in semantics would not con- stitute valid macro abstraction under our framework, nor can IIT objectives be directly adapted due to (NA3).
The set of integers from 1 to is [n]. The vectors of zeros and ones of size n are O, and 1, The identity matrix of size n x n is I,,- The Frobenius norm is l The set of positive definite matrices over Rax* is S7 The Hadamard product @. Function composition is c. The domain of a function is DD] and its kernel ker. Let M(X")) be the set of Borel measures over X C R". Given a measure jr" E MM(2") and a measurable map
Notation.
VV, X G " 2 V E X"',, we denote by xX (n²) = 1 (* (x)) the pushforward measure p.' E M(
2

---

Causal Abstraction Learning based on the Semantic Embedding Principle
We now present the standard definition of SCM.
Ind
Prob
Definition 2.1 (SCM, Pearl, 2009). A (Markovian) struc- tural causal model (SCM) M is a tuple (X, (F where (i) X = [X x, } is a set of n endogenous ran- dom variables; (ii) 2- = (Z,,...
auue, Eech
ar, Ee
(4h Eeen 5*))
} is a set of n exoge- nous variables; (iti) F is a set of n functional assignments such that X, = fP. 2.). A E [m]. with P; S X {X:}: (iv) ç is a product probability measure over independent exogenous variables x Ie S² where Ç² P(Z.)
M
M
(Ve, E a(vn, Sym)
(2h, ,inn,
A Markovian SCM induces a directed acyclic graph (DAG) 9M" where the nodes represent the variables X' and the edges are determined by the structural functions F; P: con- stitutes then the parent set for X, Furthermore, we can recursively rewrite the set of structural function F as a set of mixing functions M dependent only on the exogenous variables (ef. App. C). A key feature for studying causality is the possibility of defining interventions on the model:
Figure 2: An abstraction natural transformation, that is, a set of commuting arrows in Prob (dashed black) from M (purple) to M* (cyan).
RRxx (Massidda al., 2024). App. C provides formal definitions for interventional consistency, linear and constructive abstraction.
a V=
Definition 2.2 (Hard intervention, Pearl, 2009). Given SCM M" = 2, ) a (hard) intervention L = do(X' x" C X. is an operator that generates a new post- intervention SCM M, (N, Z.F. by replacing each function ft for X; E X with the constant at x'. Graph- ically, an intervention mutilates GM- by removing all the incoming edges of the variables in X
3. Category-theory Formalization
Standard category-theoretic formalization of CA (Rischel, 2020; Otsuka & Saigo, 2022) are based on a functorial semantics (Jacobs al., 2019) approach mapping the graph- ical structure of causal models (syntax) onto the discrete distributions of individual variables (semantics). Because of our non non-assumption (NA2), no knowledge of the structure of an SCM is available in our setting; thus, we propose a formalization mapping a dyadic structure (syntax) onto the exogenous and the endogenous probability measures implied by an SCM (semantics).
Given multiple SCMs describing the same system at differ- ent levels of granularity, CA provides the definition of an a-abstraction map to relate these SCMs:
Definition 2.3 (a-abstraction, Rischel, 2020). Given low- level M and high-level M SCMs, an a-abstraction is triple a = (R. ot), where (i) R C X is a subset of relevant variables in Mc; (ii) m : R + X is a surjective function between the relevant variables of M and the en- dogenous variables of M*; (iii) a D[R] + D[x"] is a modular function a= Siejnl Ox made up by surjective functions xxp D(m (x?)] + [[x]] from the outcome of low-level variables m 1 (X() E X onto outcomes of the high-level variables X E 4
A crucial role in our modelling is that of the mixing func- tions M, which express the data generation process as a recursive process from the exogenous functions. This allows us to define an SCM M" in measure- tuple made up of the probability space of exogenous vari- ables (U, Eu, {), the probability space of the endogenous variables (V, Σv, x), and of measurable functions M given by the mixing functions (cf. App. C).
terms as a
Notice that an a-abstraction simultaneously maps variables via the function m and values through the function a. The definition itself does not place any constraint on these func- tions, although a common requirement in the literature is for the abstraction to satisfy interventional consistency (Rubenstein et al., 2017; Rischel, 2020; Beckers Halpern, 2019). An important class of such well-behaved abstrac- tions is constructive linear abstraction, for which the fol- lowing properties hold. By constructivity, (i) a is interven- tionally consistent; (ii) all low-level variables are relevant
We can now rely on this representation to interpret an SCM as a thhegocy functor from simple index category Ind, made up only of a source and a sink object and an edge between them, to the category of probability spaces Prob, where objects (X, Ex. p) are probability spaces and morphisms yp are measurable maps:
Definition 3.1 Category-ttheoeeee SCM). An SCM is a functor M : Ind - Prob, mapping the source node of Ind to (U, Exe, <). the sink node of Ind to (V, Σv, x). and the edge of Ind to the collection M of measurable maps.
R- = Xt: (iii) in addition to the map a between endoge- nous variables, there exists a map ay between exogenous variables satisfying interventional consistency (Beckers & Halpern, 2019; Schooltink & Zennaro, 2024). By linearity,
App. B presents basic chtegory concepts, whereas App. C.5 deepens Def. 3.1. CA can now be expressed as
3

---

Causal Abstraction Learning based on the Semantic Embedding Principle
natural transformation between two SCMs, as shown in Fig. 2. This formulation has two important features. First, it highlights the role of exogenous variables in a construc- tive abstraction showing the commutativity of the paths M ) and ((vh,
Second, because of the non-assumption (NA3) only obser- vational data is available. Thus, we can not explicitly use interventional consistency information to drive our learning, Only if we identify the true constructive abstraction, we are guaranteed interventional consistency. In trying to learn the abstraction, we leverage (A1), which is met in application domains as discussed in Sec. 1.
(v)) o.M'. ).M Second, morphisms in Prob relates measure spaces, viz. sets equipped with sigma algebras. Consequently, the natural transformation components measurable maps with dimensionality deter- mined by the cardinality of x and X², To ease the notation, we will denote (u))) by az and (vr,, E.n) by axx Then, we can formally recast the a-abstraction in Prob.
Third, to learn a CA, we look for a distance function quanti- fying the misalignment between the probability measures x and x", given axx Since the probability measures be- long to spaces of different dimensionality, specifically R and IR*, we leverage the approach proposed in (Cai & Lim, 2022) to compute the misalignment through an embedding as D (x* 1P ax (x").. where D is an information-theoreti metric (e.g., p-Wasserstein) or é-divergence (e.g., Kullback- Leibler). Please refer to App. F for more details. We can now pose the following general learning problem:
Definition 3.2 (a-abstraction Prob). Given low-level M and high-level M SCMs, an abstraction a = (R. 2. m, oe) is tuple, where: (i) R is the same as in Def. 2.3; (ii) Q S Zz is set of relevant exogenous variables given by the union of the set of exogenous corresponding to the en- dogenous in R and those corresponding to their ancestors; (iā) m = (m z, mxx is a pair of surjective functions map- ping sets, mg Q - z and mxx : R + X respectively; (iv) a = (az, 0xx) is a natural transformation made by mea- surable functions mapping probability spaces, az for the exogenous and xx for the endogenous, respectively.
Problem 1. (SEP-based CA Learning)
Input: (i) probability measures x and x"; (ii) prior information about m,x, and (üi) a distance function
D (xx* 1P axx (x)).
Goal: learn measurable map o²x such that (i) it belongs to ker (x". :6 (x)) (ii) it complies with SEP in Def. 4.1, and (iii) it agrees with the prior information about mxx
As Def. 2.3, Def. 3.2 makes no reference to interventional consistency. App. D explains how intervened SCMs and interventional consistency can be represented categorically.
4. Problem Formulation
The zeroing of the distance function implies X* = P# (x"),
Within our category-theoretie framework, CA learning amounts to finding the endogenous components m.x and ox from data. We start by formulating a general learning problem working under the non-assumption (NA1)-(NA5), and then decline it to the case of linear CA.
which, together with Eq. (1), yields 8#
axoox (x")
ae (x'). However, despite solving Prob. 1, there is no guar- antee that a' coincides with the ground truth CA. In other words, the optimal solution is not unique. For a linear con- structive CA, we express mxx and axx as B E {0, 1),x and VT E R respectively. In accordance with con- structivity, each row of B has single nonzero entry, and each column has at least one nonzero entry. Importantly, for linear CA, simple yet principled way to satisfy SEP is via the geometry of the Stiefel manifold:
F#
Our problem formulation relies upon three key ingredients. First, we assume that the data generated by a constructive abstraction adheres to the semantic embedding principle. This principle requires that the CA component ox admits right-inverse measurable map.
Definition 4.1 (Semantic embedding principle, SEP). Given an a-abstraction as in Def. 3.2, the semantic embedding principle states that axx has a right-inverse measurable map 3x, such that xx 3x = Id (ymh, Xx)x))) Hence, it holds
St(C, = (V E R | VV-
(2)
The Stiefel manifold (see App. E for details), is conve- nient choice for the following reasons: (i) differently from a generic pseudo-inverse matrix, the orthogonality of V guar- antees that the geometry of the high-level space is preserved; (ii) the transpose eases the formulation and ensures numeri- cal stability in optimization. Consequently, we restate SEP for the linear case as follows.
x = Φ Bxx (x").
(1)
The SEP implies that going from the high-level model M* to the low-level model Mf and then abstracting back to M al- lows for perfect reconstruction. Notice that SEP only holds in one direction, as suggested by the word embedding: thus, identity on the left inverse is not guaranteed, meaning that the abstraction from the low level to the high level can still shed information, as we would expect in CA.
Definition 4.2 (Semantic embedding principle, linear case). Given the linear constructive CA, viz. VT. SEP implies that V E St h). From Eq. (1) we get x = y (x*).
VeV
4

---

Causal Abstraction Learning based on the Semantic Embedding Principle
A pictorial representation of Def. 4.2 is provided in Fig. Def. 4.2 shapes our methodology for CA learning, posing it as Riemannian optimization problem (Boumal, 2023).
Theorem 4.3. Let x N(0 ). Xa N( ) where E' E S4++ and Ehh E S Denote by 0 < X S
... S de the eigenvalues of E', and by 0 < Ki .... ... S Kn. those of E". If linear CA V E St(f, h) complying with SEP from x to x" exists, then
As an application, in the sequel, we will tackle an implemen- tation of Prob. for the linear constructive case ox = VT, where (i) x" ~ (O,,, Σ and x ~ (O,, )); and (ii) D (x* 1 (x9) DKL (x*1l0 (x9) where DKL stands for KL divergence. Specifically,
S S Aite-Ri
Vi [k].
(4)
Proof. See App. G.
DV - - '=n] + log der (V VV (3)
dei
We now turn to formulating the learning problem. We in- vestigate two approaches for injecting the prior information about mxx encoded in the matrix of prior knowledge B, into our problem. Please notice that in case B is not fully specified, it might not comply with the row and column con- straints discussed above. These formulations translate into non-smooth and smooth Riemannian learning problems.
where C' C=- log det { Σ² h is a constant term. Ad- ditionally, from Eq. (3) it is immediate see that both V and V belong to ker D Such application is highly relevant as it is common to deal in practice with Gaussian measures (or quasi) (D' Acunto et al., 2024); also, in causal- ity, such a measure casily arises from the prominent family of linear models (Bollen, 1989; Shimizu et al., 2006) and is investigated in the CA literature (Kekić al., 2024; Mas- sidda et al., 2024). KL divergence is a common choice in ML and statistics, but notice that any distance vanishes when evaluated at the ground truth.
Nonsmooth problem. In the nonsmooth problem we in- troduce B as a penalty term in the objective function. The rationale is to penalize entries in V corresponding zeros in B. Let D (Lexh B). The problem reads follows:
Problem 2. Given Σ E EStt, Σ E St D D (0. 1)
Spectral properties entailed by SEP. In the setting of zero-mean Gaussian measures, all causal information is encoded in the covariance matrices of x and x², which may be geometrically represented as ellipsoids in IR and R². Each covariance Σ admits an eigendecomposition Σ UAUT, where the columns of U specify the ellipsoid's principal axes and the square roots of the diagonal entries of A give the corresponding axis lengths. Thus, for the low- level distribution N (0,, E",, one obtains an é-dimensional ellipsoid with axes Ut and lengths VA, i E [f]; similarly, (O,, ) defines an h-dimensional ellipsoid with axes Thh and lengths V E [h].
and R4, the CA is the transpose of
V = arg f(V) + D VESt(E,))
(5)
h(V)
Here, f(V) follows Eq. (3), omitting the constant C.
Please notice that, although appealing in its form, Eq. (5) does not guarantee the constructiveness of the learned CA. Moreover, the penalty term introduces bias in the learned V in the case of partial prior knowledge.
Smooth problem. In the smooth problem, we introduce B directly in the objective function f()). The CA is now defined as the Hadamard product of V and the support (B o S) integrating prior B and learned S knowledge. This formulation is particularly convenient as it enables us to jointly optimize for V E Rtxh and matrix S E [0, 1 However, we also need to introduce three constraints: (i) by SEP, B 3 S O V must belong to the Stiefel manifold; (ii) by functionality, the columns of the support (B @ S) must sum up to one, meaning that they lie on sphere, defined
When projecting the low-level ellipsoid into an h-dimensional subspace via an orthonormal map V E St(e, the resulting covariance VVV VTU'AU V = AQ, with Q = U' V, yields projected ellipsoid whose axes are linear combinations of those in Us. Because V is contractive by SEP, no projection can increase variance. Thus, the length of each axis of the projected ellipsoid lies between the minimum and maximum length of those of the f-dimensional ellipsoid. Specifically,, by the Ostrowski's theorem for rectangular V (cf. Th. 3.2 in Higham & Cheng, 1998), the i-th largest axis length of the projected ellipsoid falls between the i-th and (i ++ 8- h)-th largest lengths of the E-dimensional ellipsoid. Consequently, for the optimal CA case where the projected ellipsoid exactly matches that of hh, the projected axis align with U and the previous interlacing inequalities provide necessary spectral conditions for the existence of a linear CA from N (0g, E to N(O,, )
Sp (h. = { (A E 1)
||a, l12 = 1 and
h Σ ai = -. E 101 i=1
(6)
(iii) by surjectivity, the rows of the support (B + S)T must contain at least a one. The problem reads as:
5

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Problem 3. Given E E Shا Eh Sh and B E {0. 1)exh, the linear constructive CA is given by the trans- pose of the product B 3 S @ V, where
5.1. Solution of the nonsmooth learning problem
Leveraging Proposition 5.1, we have that Eq. (5) is consti- tuted by smooth yet nonconvex term, f(V), and non- smooth one, h(V). Hence we solve Prob. through two different optimization paradigms for nonsmooth Rieman- nian optimization: MADMM and ManPG. We term the proposed methods LinSEPAL-ADMM and LinSEPAL-PO, where LinSEPAL stands for Linear Semantic Embedding Principle Abstraction Learner. Next we provide sketch of the solution and provide the full mathematical derivation in App. H and App.
arg min f(V.S):
VeR SE
subject to
(1) B O V St(C,h,
(7)
(i) (B S) E (h))
(iti) 1, (B () 1, On:
and
LinSEPAL-ADMM. The MADMM framework appeals to our setting given the objective function separating into smooth and nonsmooth terms. To derive the LinSEPAL-ADMM iterative algorithm, we proceed as follows. First, the nonsmooth term h(V) is associated with a splitting variable Y to be optimized over R obtaining an equivalent problem formulation (cf. Eq. (P2)). LinSEPAL-ADMM proceeds by iteratively minimizing the augmented Lagrangian with respect to the primal variables V and Y, while maximizing w.r.t. the scaled dual variable. Specifically, LinSEPAL-ADMM, solves the subproblem for V (cf. Eq. (36)) through standard techniques for smooth optimization on the Stiefel manifold (e.g., conjugate gradient, Edelman et al., 1998). This is the most complex update in the LinSEPAL-ADDMI iterative procedure due to the nonconvex objective and the Stiefel manifold. Next, LinSEPAL-ADMM updates Y in closed form through the element-wint soft-thresholding operator (cf. Eq. (37)). Finally, the scaled dual variable is updated by adding the primal residual evaluated at the current solution (cf. Eq. (R1)). The stopping criteria for LinSEPAL-ADMM, are established according to primal and dual feasibility optimality conditions (Boyd et al., 2011, ef. App. H). To the best of our knowledge, the convergence guarantee for MADMM in the Riemannian space has not been proven. Consequently, the same holds for LinSEPAL-ADMM. Algorithm 1 summarizes the method.
[(V, ((V (VV.))-T((BBBB +log det (B
Constraints (ii) and (iii) further underscore the role of S: it enables learning the support of CA while guaranteeing its constructiveness. Notice that the matrix S does not need to be a logical matrix; it is the product B c S which must be logical. Also, if B provides full prior knowledge about the structure, we have = B and we do not need to learn S. This approach guarantees the ground-truth structure for the learned CA. The full prior problem formulation is provided in App. J.6.
Unfortunately, both the Stiefel manifold in Eq. (2) and DKT. in Eq. (3) are nonconvex in V. In the next section we devise methods suitable for this setting.
Remank 1. Learning linear CAs remains challenging even with full prior structural knowledge and jointly sampled data (NA5). Although one could decompose the task into h independent linear regressions, enforcing SEP requires each coefficient vector to satisfy a unitary fn-norm constraint, rendering the problem nonconvex.
5. Problem Solution
To solve the nonsmooth and smooth Riemannian problems in Sec. 4, we leverage the following:
LinSEPAL-PG. Our LinSEPAL-PG is an iterative algo- rithm alternating two updates (cf. Eq. (R2)). The first up- date is the proximal mapping providing proximal gradient direction G onto the tangent space to the Stiefel manifold, using the first-order approximation of the objective around the k-th estimate. The second is the update for V which exploits the canonical retraction (cf. Eq. (23)) technique for projecting back V + G from the tangent space to the manifold. The hardest step in the LinSEPAL-PG algo- rithm is the proximal update (cf. Eq. (42)). We solve it by declining the regularized semi-smooth Newton method (Xiao et al., 2018) to our application (cf. App. D). Follow- ing the rationale in (Si et al., 2024), differently from the
Proposition 5.1. Consider the function
((A)=TI((A E'A) 'd
+ +log det AA A'A.. (9)
Eq. (9) is smooth for A E ht) Additionally, define A = (ATE ATTA -1 The grudient of (A) is
VAI 2 2E -
(10)
Proof. See App. G.
6

---

Causal Abstraction Learning based on the Semantic Embedding Principle
original ManPG method which uses the parameterization of
to the closest orthogonal approximation problem (Fan & Hoffman, 1955; Higham, 1986), whose solution is obtained in closed form via polar decomposition. Subsequently, the subproblem for X is solved in closed-form according to Lemma J.3. Finally, the scaled dual variables are updated with the corresponding primal residuals. Empirical con- vergence for CLinSEPAL is established when the norms of primal (cf. Eq. (97)) and dual (cf. Eq. (98) residuals vanish, in accordance with absolute and relative tolerances (cf. Eq. (99)). Algorithm 3 summarizes the method. Addi- tionally, App. J.6 details the solution in the special case of full prior knowledge.
the tangent space, we constrain G* to the tangent space by exploiting the basis of the normal space to the manifold (cf. Eq. (44)). This way, we ease the mathematical solution, with benefits from the computational perspective (cf. Si et al., 2024). Next, LinSEPAL-PG updates V in closed form (cf. Eq. (69) by applying the re-ettaccion, employing an Armijo line-search procedure to determine the stepsize. The optimization stops either when maximum number of iterations is reached, or when V is below threshold y ,L R 0. LinSEPAL-PGG inherits the global convergence of the ManPG framework, established in (Chen et al., 2020). Algorithm 2 summarizes the method.
6. Empirical Assessment on Synthetic Data
5.2. Solution of the smooth learning problem
This section provides the empirical assessment of LinSEPAL-DDMM, LinSEPAL-PG and CLinSEPAL with different degrees of prior knowledge, from full (p) to partial (pp). We monitor four metrics to evaluate the learned CA DKL (i) constructiveness, as required by Def. 4.2; (ii) V
We provide a sketch of the solution below and the full math- ematical derivation in App. J. In this case, we want to jointly optimize S and V, both being components of the linear CA,
viz. (B 2 S V)T. Hence, unlike the nonsmooth case, we constrain to the Stiefel manifold the product (B @ S @ V). To solve Prob. 3, we combine the SOC, ADMM, and SCA methods. According to the rationale behind SOC, we add two splitting variables, namely Y1 and Y2 in St(e, h) (cf. Eq. (72)), to separate the nonconvexity of the objective func- tion from that induced by the manifold. The reason why we have two splitting variables is that we need to take into account the bilinear form of the first constraint in Eq. (7). Additionally, to manage the second constraint in Eq. (7), we introduce another splitting variable X E Sp (h, E). Start- ing from the equivalent problem formulation (cf. Eq. (73)), we write the (nonconvex) scaled augmented Lagrangian (cf. Eq. (74)), thus arriving at the update recursion for our proposed method (ef. Eq. (75)). We term the latter CLin- SEPAL (Constructive LinSEPAL) to highlight that it returns constructive support for CA. CLinSEPAL proceeds by it- eratively minimizing the augmented Lagrangian w.r.t the primal variables V, S, Y1. Y2, and X; and maximizing it w.r.t. the scaled dual ones. In the subproblems for V (ef. Eq. (76)) and S (cf. Eq. (84)), we adopt the SCA paradigm to manage the nonconvexity of ((V.S) and f(V++), .S), respectively. By exploiting the smoothness of f(V, S) (cf. Corollary J. the strongly convex surrogates around the current solution (cf. Eqs. (77) and (85)). CLin- SEPAL solves the strongly convex surrogate subproblems (cf. Eqs. (78) and (88)) exactly. Due to the presence of the inequality constraints, the subproblem for S is a constrained quadratic programming problem. CLinSEPAL solves via standard techniques (e.g.. Stellato al., 2020). These two steps in CLinSEPAL can be seen as instance of the lin- earized ADMM framework (Alg.1 in Lu al., 2021) where each internal update is solved exactly. Next, CLinSEPAL solves in closed-form the updates for the three splitting variables. Indeed, the subproblems for Y, and Y2 amount
evaluating the alignment between P V (x') and x"; (iii) the
Frobenius distance between the absolute value of V and that of the ground truth V*, normalized by ||V*Il to make the settings comparable; (iv) the Fl score computed using the support of the learned CAs and that of V evaluate structural interventional consistency. App. K provides the definition for the above metrics and the haper-parameters values used in the experiments.
Full prior knowledge. In the fp case, we investigate three different settings h2) E {(12, (12, (12, 6)}, corre- sponding to the cases of high, medium-high, and medium coarse-graine graining. We do not consider the case where h > 2/2 since the abstraction for - f/2 nodes of the low-level model would be fully specified due to the availability of full prior knowledge. For each setting, we instantiate S = 30 ground truth abstractions V*, and for each sim- ulation E [S] we run all the methods R = 50 times, with different initializations. Then, for each s and method, we retain the V minimizing the objective DKL
Fig. shows the performance of the tested methods. All the methods provide constructive CAs V E [S), and reach a good level of alignment in terms of DKL Recall that, while CLinSEPAL and LinSEPAL-ADMM stop the learning pro- cedure according to primal and dual residuals convergence, LinSEPAL-PG exits when DKL. is below a certain threshold
derived
V
KL, (in the experiments Kk, 10-4). The Frobenius abso- lute distance shows comparable performances for the three methods, although CLinSEPAL and LinSEPAL-ADMM outperform in case (e, h) = (12, 4). This metric tells us that, h increases, the learned V tends (in absolute terms) to the ground truth. Interestingly, when h2, = (12,2) we observe a high distance from V although the learned V
7

---

Causal Abstraction Learning based on the Semantic Embedding Principle
leamed
dierggeoee
nappiint
theegente
101
140
10
0.8-
10
10
04-
1.00
0.-
0.38
11
10"11 0.99
10
011
Froberics
Froberio aboclate dicarce
Fi score
mithed
3.1-
CLASSEPAL
INSEPAL ATMM
0.-
8
9
1.03-
0.4
8
8
0.7-
1055
8
0
182
02-
0
04
(12.2 7)
[12.0 0)
[12.
[12
(12 8)
0.25 Fraction labealable prioe
(0.A)
(6.00
Fraction arovolibb knowllgg
Figure 3: Synthetic fp results for all settings h) and methods: (i) fraction of learned CAs that are construc- tive, (ii) DKL, (ii)) normalized absolute Frobenius dis- tance from V*, and (iv) F1 score.
Figure 4: Synthetic pp results for setting (4,4) = (4, 2), all methods, and prior knowledge amounting to the cor- reet structural mapping for 25%, 50%, or 75% of the nodes. All plots and color legend as in Fig. 3.
has the correct structure (ef. F1 score). This suggests that under a high grassegraining. the size of ker RLL grows, and it is more difficult for our methods to estimate V un- der (NAI)-(NAS) Finally, the Fl score confirms that the methods guarantee the true CA structure of V, for all the settings. To sum up, CLinSEPAL and LinSEPAL- are slightly better choices than LinSEPAL-POR in case of full prior knowledge our experimental setting.
fraction of constructive CAs. Finally, with the highest drop (75%) CLinSEPAL succeeds in learning constructive CA and lowering DKL even if the Frobenius absolute distance slightly increases. To sum up, for the pp setting only CLin- SEPAL guarantees a constructive abstraction.
7. Causal Abstraction of Brain Networks
Partial prior knowledge. In the pp case, consider the setting (c )44 and simulate partial prior knowl- edge by setting to the entries of the 25%. 50%. and 75% of the rows in B. For each setting, we instantiate S = 30 ground truth abstractions V*, and for each simu- lation s E [S] we run all the methods Rt = 30 times, with different initializations.
To show the practical relevance of our approach, we apply CLinSEPAL to resting-state functional magnetic resonance imaging (rs-fMRI data, using the dataset from (D' Acunto et al., 2024) (refer to the paper for details on the dataset). The data, publicly released as part of the Human Connec- tome Project (Smith et al., 2013), comprises recordings from 100 healthy adults with a parcellation scheme that divides the brain into 89 regions of interest (ROIs), K 44 for each hemisphere plus the shared vermis region.
In Fig. 4, the first plot immediately shows that only CLin- SEPAL consistently returns constructive linear CA, as guaranteed by its formulation in Prob. 3. We decided to consider methods performing under a threshold of 90% to be unreliable in returning constructive CAs and not to re- port their remaining metrics. For completeness, Fig. 7 in App. L provides the results where the threshold on con- structiveness is removed. In the case of limited drop of prior knowledge (25%) all methods perform well, similarly to the fp case, with CLinSEPAL. and LinSEPAL--DMMI slightly outperforming LinSEPAL-PG. With higher drop (50%), LinSEPAL-PG fails to achieve our constructiveness threshold, while CLinSEPAL and LinSEPAL-ADMMN still perform well, although LinSEPAL-ADMM provides lower
We simulate first investigating team of neuroscientists taking zero-mean stationary time series for the left hemi- sphere of the first adult in the dataset. They estimate the data covariance matrix using Gaussian mixture probability model, viz. Σ E RéXE with é = K + 1, and interpret it as generated by underlying, unknown, low-level SCM.
In a first fp scenario, we imagine a second investigating team that has collected data according to their causal network specified on a coarser parcellation of the same brain in h = 14 macro ROIS. We generate the data for the second team using a ground truth linear CA B, V E St(45, 14)
8

---

Causal Abstraction Learning based on the Semantic Embedding Principle
based on the structural mapping in (D' Acunto al., 2024), and use the data for estimating the covariance matrix Σd E Raxh, In this scenario it is realistic to assume knowledge of B defining how macro ROIa are mapped to ROIs. Then, to align their models, the two groups run CLinSEPAL to recover the abstraction given E E and B. Fig. 8 (in App. M) shows that CLinSEPAL recovers V*.
the constructiveness of the CA, we developed CLinSEPAL. Our empirical assessment on synthetic data confirmed the effectiveness of our methods, and the application to brain data showcased the potential in real-world problems.
Our work paves the way for several exciting research direc- tions. First, as it emerges from our Gaussian application, linear CAs with different probability measures deserve care- ful investigation. Second, studying the nonlinear case is a compelling avenue. We believe that deep and reinforcement learning paradigms, such as incoding-decoding and actor- critic architectures, hold promise for modeling nonlinear CA maps. Lastly, we view our work as a foundational step toward observational causal abstraction learning, bridging the gap between CA learning and causal discovery (Spirtes & Zhang, 2016). Our calegory-theorete framework under- scores the pivotal role of exogenous variables, drawing path to translate SCM identifiabillty results into CA iden- rifiability results. This suggests that, in some cases, inter- ventional consistency may be achieved without relying on interventional data.
In a second pp scenario, we imagine that the second in- vestigating team has collected data according to causal network aggregating ROI time series into h = 8 brain func- tional networks related to different activities (e.g., motor, visual, default mode). Data is generated again through a ground truth linear CA B, V E St(45, 8) based on group- ings in (D' Acunto et al., 2024) and the covariance matrix h E RR computed. In this scenario, knowledge of B is debatable as different studies in the literature suggest differ- ent relations between ROIs and functions; we then express this partial information via uncertainty over B, meaning that some rows of B have more than one entry equal to one. Fig. 9 in App. M shows the B matrix provided as input to CLinSEPAL, as well as the ground truth. The two groups now run CLinSEPAL using Σ n and an uncertain B; partial knowledge compounds on an already challenging learning problem due to the high coarse-graining. Fig. 10 and Fig. 11 show results with different levels of uncertainty. For low uncertainty, CLinSEPAL correctly retrieves the structure of the CA, although we observe some variation in the colors w.r.t. V*; additionally, D and the Frobe- nius absolute distance in Fig. 11 show that misalignment is minimized and V very close to V*. For medium and high uncertainty, CLinSEPAL makes some mistakes in terms of structural mapping, but Fig. 11 shows that insights from the method are still valuable.
Impact Statement
Our work is foundational, aiming advancing the field of causal abstraction. Our proposed methods can be applied to different application domains, such as neuroscience. As demonstrated by our empirical assessment, the information resulting from their application is high-level and useful for a better understanding. Hence, we believe that the risks associated with improper usage of our techniques are low.
A Acknowledgements
The work of Gabriele D' A'uunto and Paolo Di Lorenzo was supported by the SNS JU project 6G-GOALS (Stri- nati et al., 2024) under the EU's Horizon program Grant Agreement No 101 139232. The work of Gabriele D' Acunto was also supported by the European Union under the Italian National Recovery and Resilience Plan (NRRP) of NextGen- erationEU, partnership on Telecommunication of the Fu- ture" (PE0000000) program " RESTART"). The work of Yorgos Felekis was supported by the Onassis Foundation Scholarship ID: F 7R 062-1/2021-2022.
8. Conclusion and Future Works
In this work, we addressed the challenge of CA learning in realistic scenarios, abandoning restrictive assumptions (NA1)-(NA5) that limit the applicability of existing methods. We proposed an alternative for SCM and CA, and introduced the semantic embedding principle to learn CAs that meaningfully preserve infor- mation. We formulated general CA learning problem grounded in SEP, under a mild assumption of partial prior knowledge about the structure of CA. For the linear CA setting, we showed how SEP links CA to the geometry of the Stiefel manifold; as application, we tackled the im- portant case of Gaussian measures, with the KL divergence as a measure of alignment between the low- and high-level SCMs. We pursued two different formulations. For the first, a nonsmooth Riemannian learning problem, we devised the LinSEPAL-ADMM and LinSEPAL-PC methods. For the second, a smooth Riemannian learning problem ensuring
erregory-theorete framework
References
Absil, P.-A., Mahony, R., and Sepulchre, R. Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2008
Beckers, S. and Halpern, J. Y. Abstracting causal mod- els. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 2678 2685-2685, 2019.

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Bereska, L. and Gavves, S. Mechanistic interpretability for ai safety-a review. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL https:// Wpenreview.net/Eorree forum? Pd=ePUVetPKus.
Fan, K. and Hoffman, A. J. Some metric inequalities in the space of matrices. Proceedings of the American Mathe- matical Society, 6(1):111-116 1955.
Felekis, Y., Zennaro, F. M., Branchini, N., and Damoulas, T. Causal optimal transport of abstractions. In Causal Learning and Reasoning, pp. 462-498. PMLR, 2024.
Bollen, K. A. Structural equations with latent variables, volume 210. John Wiley & Sons, 1989.
Boumal, N. An introduction to optimization smooth manifolds. Cambridge University Press, 2023.
Ganguly, N., Fazlija, D., Badar, M., Fisichella, M., Sikdar, S., Schrader, J., Wallat, J., Rudra, K., Koubarakis, M., Patro, G. K., et al. A review of the role of causality developing trustworthy Al systems. arXiv preprint arXiv:2302.06975, 2023.
Boumal, N., Mishra, B., Absil, P.-A., and Sepulchre, R.
Manopt, Matlab toolbox for optimization manifolds. Journal of Machine Learning Research, 15(1):1455-1455) 2014.
Geiger, A., Lu, H., Icard, T., and Potts, C. Causal abstrac- tions of neural networks. Advances in Neural Information Processing Systems, 34:9574 9586, 2021.
Boyd, S. and Vandenberghe, L. Convex Optimization. Cam- bridge University Press, 2004.
Boyd, S., Parikh, N., Chu, E.,, Peleato, B., Eckstein, J., al. Distributed optimization and statistical learning via the alternating direction method of multipliers. Foundations and Trends@ in Machine learming, 3(1):1-122, 2011.
Geiger, A., Wu, Z., Potts, C., Icard, T., and Goodman, N.
Finding alignments between interpretable causal variables and distributed neural representations. In Causal Learn- ing and Reasoning, pp. 160-187. PMLR, 2024.
Brookes, M.
The matrix reference manual. http: / wwwwwwww...ooo ee www.iiae..accc...mmmmmaam..mm www.eerrialaa..uhhggg.. /hp/ staff/ dmb/ comb/mmtriin itatmaatiintttteee html, 2020. Ac-
Higham, N. J. Computing the polar decomposition-wiit applications. SIAM Journal Scientific and Statistical Computing,
1986.
cessed: 2024-01-10.
Cai, Y. and Lim, L.-H. Distances between probability distri- butions of different dimensions. IEEE Transactions on Information Theory, 68(6), 2022.
Higham, N. J. and Cheng, S. H. Modifying the inertia of matrices arising in optimization. Linear Algebra and its Applications, 275:261-22 2729, 1998.
Chen, S., Ma. S., Man-Cho So, A.. and Zhang, T. Proximal gradient method for nonsmooth optimization over the Stiefel manifold. SIAM Journal on Optimization, (1):210-239, 2020. doi: 10.1137/18M122457X. URL https: //t:::///oiiie/ org 10t.111111112222 010//1188222777
Jacobs, B., Kissinger, A., and Zanasi, F. Causal inference by string diagram surgery. In International Conference on Foundations of Software Science and Computation Structures, pp. 313-329. Springer, 2019.
Kekić, A., Schölkopf, B., and Besserve, M. Targeted re- duction causal models. In Uncertainty in Artificial Intelligence, pp. 1953-1980. PMLR, 2024.
D' D'Acunto, G., Bonchi, F,, Morales, D. F., and Petri, G. Extracting the multiscale causal backbone of brain dynamics. In Causal Learning and Reasoning, pp. 265- 295. PMLR, 2024.
Komanduri, A., Wu, Y., Huang, W., Chen, F., and Wu, X. SCM-VAB Learning identifiable causal representa- tions via structural knowledge. In 2022 IEEE Interna- tional Conference Big Data (Big Data), pp. 1014 1023. IEEE, 2022.
Diamond, and Boyd, CVXPY: A python-embedded modeling language for convex optimization. Journal of Machine Learning Research, 17(83):1-5, 2016.
Dyer, J., Bishop, N. G.. Felekis, Y., Zennaro. F. M., Cali- nescu, A., Damoulas, T. and Wooldridge, M. J. Inter- ventionally consistent surrogates for complex simulation models. In The eiihty-eighth Annual Conference on Neu- ral Information Processing Systems, 2024. URL https: //openreview net forum d=Ut TMDTTF..
Kovnatsky, A., Glashoff, K., and Bronstein, M. M. MADMM: A generic algorithm for non-smooth optimiza- tion on manifolds. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, Oc- tober 11-14, 2016, Proceedings, Part V 14, pp. 680-696. Springer, 2016.
Edelman, A., Arias. T. A., and Smith. S. T. The geometry of algorithms with orthogonality constraints. SIAM Jour- nal on Matrix Analysis and Applications, 20(2):303-35 1998.
Lai, R. and Osher, S. A splitting method for orthogonality constrained problems. Journal of Scientific Computing, 58:431-4493 2014.
10

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Lu, S., J. D., Razaviyayn, M., and Hong. M. Linearized ADMM converges to second-order stationary points for non-convex problems. IEEE Transactions Signal Pro- cessing, 69:4859-449 4874, 2021.
Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalch- brenner, N., Goyal, A., and Bengio, Y. Toward causal representation learning. Proceedings of the IEEE, 109(5); 612-634, 2021.
Mac Lane, S. Categories for the Working Mathematician, volume 5. Springer Science & Business Media, 2013.
Schooltink, W. and Zennaro, F. M. Aligning graphi- cal and functional causal abstractions. arXiv preprint arXiv:2412.1700 2024.
Marzouk, Y., Moselhy, T., Parno, M., and Spantini, A. Sampling via measure transport: An intro- duction, pp. 1-41. Springer International Publish- ing, 2016. ISBN 9783319112596. doi: 10.1007/ 978-3-319-11259-6622--1 URL http://dk /dx doi. org/10.
Shimizu, S., Hoyer, P. O., Hyvärinen, A., Kerminen, A., and Jordan, M. A linear non nan-Gasssian acyclic model for causal discovery. Journal of Machine Learning Research, 7(10), 2006.
311----11199111---911 23-1
Si, W., Absil, P.-A., Huang, W. Jiang, R., and Vary, A Riemannian proximal Newton method. SIAM Journal on Optimization, 34(1):654-681, 2024.
Massidda, R., Magliacane, S., and Bacciu, D. Learning causal abstractions of linear structural causal models. In Uncertainty in Artificial Intelligence, pp. 2486-2515.
Smith, S. M., Beckmann, C. F., Andersson, J., Auerbach, J., Bijsterbosch, J., Douaud, G., Duff, E., Feinberg,
PMLR, 2024.
D. A., Griffanti, L., Harms, M. P., et al. Resting-state fMRI in the Human Connectome Project. Neurolmage. 80: 1444168, 2013.
Nedié, A., Pang. J.-S., Scutari, G., Sun, Y., Scutari, G., and Sun, Y. Parallel and distributed successive convex approximation methods for big-data optimization. Multi- Agent Optimization: Cetraro, Italy 2014, pp. 141-308, 2018.
Spirtes, P. and Zhang, K. Causal discovery and inference: Concepts and recent methodological advances. In Applied Informatics, volume 3, pp. 1-28. Springer, 2016.
Otsuka, J. and Saigo, H. On the equivalence of causal models: A category-theoreti approach. In Conference on Causal Learning and Reasoning, pp. 634-646. PMLR, 2022.
Stellato, B., Banjac, G., Goulart, P., Bemporad, A., and Boyd, S. OSQP: An operator splitting solver for quadratic programs. Mathematical Programming Computation, 12(4))637-672, 2020. doi: 10.1007/ $12532-000-001179-2 URL https: /t///ii.org//////
Parikh, N., Boyd, S., et al. Proximal algorithms. Founda- fions and Trends@ in Optimization, 1(3):127-239, 2014.
Pearl, J. Causality. Cambridge University Press, 2009.
Strinati, E. C., Di Lorenzo, P., Sciancalepore, V., Aijaz, A., Kountouris, M., Gündüiz, D., Popovski, P., Sana, M.. Stavrou, P. A., Soret, B., et al. Goal-oriented and se- mantic communication in 6G Al-native networks: The 6G-GOALS approach. In 2024 Joint European Confer- ence on Networks and Communications & 6G Summit (EuCNC/6G Summit), pp. 1-6. IEEE, 2024.
Perrone, P. Starting Category Theory. World Scientific, 2024.
Qi, Y., Schölkopf, B., and Jin, Z. Causal responsibility attribution for Auman-Al collaboration. arXiv preprint arXiv:2411.03275, 2024.
Thomas, C. K. and Saad, W. Neuro-symbolic causal reason- ing meets signaling game for emergent semantic commu- nications. IEEE Trunsactions Wireless Communica- tions, 23(5):5546-466, 2023.
Rawal, A., Raglin, A., Rawat, D. B., Sadler, B. M., and McCoy, J. Causality for trustworthy artificial intelligence: Status, challenges and perspectives. ACM Computing Surveys, 2024.
Thomas, C. K. and Saad, W. Symbolic logic and category theory for reasoning erasledng-enabled semantic communications, In 2024 58th Asilomar Conference on Signals, Systems, and Computers, pp. 121-125. IEEE, 2024.
Rischel, E. F. The category theory of causal models. Mas- ter's thesis, University of Copenhagen, 2020.
Rubenstein, P. K., Weichwald, S., Bongers, S., Mooij, J. M., Janzing, D., Grosse- Grosse-Wentrup. M., and SchöIkopf, B. Causal consistency of structural equation models. In 33rd Conference on Uncertainty in Artificial Intelligence (UAI 2017). pp. 807. Curran Associates, Inc., 2017.
Thomas, C. K., Saad, W., and Xiao, Y. Causal semantic communication for digital twins: A generalizable imita- tion learning approach. IEEE Journal on Selected Areas Information Theory, 4:698-711 2023.

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Xiao, X., Li, Y., Wen, Z., and Zhang. L. A regularized semi-smooth Newton method with projection steps for composite convex programs. Journal of Scientific Com- puting, 76:364-389, 2018.
Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., and Wang, Causal Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9593-9602, 2021.
Zennaro, F. M., Drávuez, M., Apachitei, G., Widanage, W. D., and Damoulas, T. Jointly learning consistent causal abstractions over multiple interventional distri- butions. In 2nd Conference on Causal Learning and Reasoning, 2023.
Zhang, K. and Hyvärinen, A. On the identifiability of the post-nonlinear causal model. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial In- telligence, pp. 647-655, 2009.
12

---

Causal Abstraction Learning based on the Semantic Embedding Principle
A. Extended Notation for the Appendix
Below is the notation used throughout the appendices. The set of integers from to is [n]. The vectors zeros and is I,. The entry indexed by row and column á is asj Ayy diag(a) (a) is the diagonal matrix having as diagonal the vector a, while diag(A) is the diagonal of the matrix A. The Frobenious norm is || The set of positive definite matrices over R=** is S That of symmetric ones as Sym(p). The column-wise vectorization of matrix is vec (). The Hadamard product is @. Function composition is c.
ones of size are O and 1n- The identity matrix of size
Let M( be the set of Borel measures over n C RR Given a measure pa E _M(²") and measurable map v X V x Xm we denote by 9 (p*) = n² (='V (x)) the pushforward measure n²' E EM(x."). The proximal mapping of h at A is p() (A) arg minly h(V) + 1/(22) |V - Al X e IR+. The Euclidean gradient of smooth f is V f. while the Riemannian one h. The Euclidean subgradient of nonsmooth h is dh, the Riemannian instead Oh.
B. Category Theory Essentials
Below are fundamental definitions and examples that are instrumental in providing the necessary background on category theory to understand our work. For comprehensive overview of category theory see resources such as Mac Lane (2013); Perrone (2024).
Definition B.1 (Category). A category consists of
A collection of objects, viz. X in C, A collection of morphisms, viz. f X - Y
such that:
Each morphism f has assigned two objects of the category called source and target, respectively,
Each object X has identity morphism idx X X,
Given f X + Y and g + Z, than the composition exists, g h: X-- -
These structures satisfy the following axioms:
(Unitality) Vf + Y, f idx = f and idy o f=
(Associativityy) Given f. 9. and h such that the compositions hold, then h ogo = -
Example The following are some notable examples categories:
Indicate with Poset a partial order set. Poset can be viewed as the category whose objects are the elements p and morphisms are order relations p'. Notice that there is at most morphism between two objects;
Vect& is the category whose objects are real vector spaces and morphisms are linear maps;
Prob is the category whose objects are probability measure spaces and morphisms measurable maps.
Arrows between categories are called functors, defined as follows:
Definition B.2 (Functor). Consider and D categories. functor F: C + D consists the following data:
For each object X in an object F(X) in D; For each object morphism f X Y in C, morphism F(J): F(Y) in D;
such that the following axioms hold:
(Unitality) VX in C, F(idx)=idl = =dd In other words, the identity in C is mapped into the identity in D. (Compositionalitt) Vf and in such that the composition defined, then F'g o f) = F(g) o F(J). other words, the composition in C is mapped into the composition in D.
To ease the notation, in the sequel, we use FX and Fl denote F(X) and F(J), respectively. Finally, we can have arrows
13

---

Causal Abstraction Learning based the Semantic Embedding Principle
between functors well, called natural transformations:
Definition B.3 (Natural transformation). Consider two categories C and D, and two functors between them, namely F: C - D and G C + D. A natural transformation F S G consists of the following data:
For each object X in C, morphism ax px + G. in D called the component of a at X;
For each morphism f X + X' in C, the following diagram commutes:
Fx
F
FX
(1I)
0x
0x
G
G
G
natural transformation can thought of as a consistent system of arrows between two functors, invariant with respect to maps between the images of two functors.
C. Causality and Causal Abstraction
This section provides additional definitions and examples related to SCMs and the CA framework.
C.1. Mixing functions
A set of structural function in Markovian SCM can be reduced to set of mixing functions dependent only the exogenous variables.
Given an SCM M", recall that F is a set of n functional assignments which define the values X; = f((P,, 2.). Vie [n].
with P; C X' \ {X.). Denote by AA S Z \ {Z,} the set of exogenous variables corresponding to the ancestors of X;, where A; S [n] \ (i]. According to F, we can identify set of mixing functions M {m1,.... m,,) such that the values of the endogenous random variables are equivalently expressed as z; = m, ((2)))e.4 vi. [n].
also characterize the product probability measure implied by the SCM purely in terms of the exogenous variables, viz. x [[Em()) P (X, Z.).
Further,
As an example, consider a causal relation # + T2. In the linear SCM with additive noise (Bollen, 1989; Shimizu et al., 2006) setting we have
(12)
22 = 2, + 52 = 02,1 + 52
Again, for the post-nonlinear model (Zhang & Hyvärinen, 2009), we get
I1 f1,1(2)) mi m2)))
32 (2,((221 + 22)
(13)
12,(f2,1 o fi,1 + 22) mz(21,
C.2. Interventional consistency
A typical requirement imposed on CA maps is that they act in a consistent way with respect to interventions (Rischel, 2020).
Definition C.1 (Interventional consistency). Given an a-aassraction between M and M and a set I of hard interventions on X C Xh, the abstraction is interventionally consistent if, for any intervention in and for every set of target variable y C 2 Xt the following diagram commutes:
14

---

Causal Abstraction Learning based on the Semantic Embedding Principle
d((2))
D[
DD
a x
0
D
D
P() do(2)))
or equivalently.
dy (PP)))do(22))) P() (do(22))),
(14)
where X2
mmmmmma 1mmmm123 (2t) and 1/- = m (00)
Essentially, commutativity suggests that we obtain equivalent intervention outcomes in two different ways: (i) either by intervening the low-level model and then abstracting or, (ii) by abstracting to the high-level model and then intervening in equivalent fashion.
C.3. Linear abstraction
The class of abstractions may be restricted by assumption of the form of the abstraction map (Massidda et al., 2024): Definition C.2 (Linear abstraction). Given a-abstraction (R, m, a) from M to M the abstraction is linear if = V E Rhxe.
C.4. Constructive abstraction
particularly well-behaved form of abstraction is constructive abstraction. In the context of the 7-abstraction framework (Beckers & Halpern, 2019), constructive abstraction is an abstraction such that: (i) the variable mapping defines clustering of the low-level variables (constructivity); (ii) consistency holds for all high-level interventions (strongness); (iii) the value map is surjective and it implies a map between exogenous values and between interventions (r-abstraction). the a-framework few of these properties hold by construction; thus, we define a constructive abstraction as (Schooltink Zennaro, 2024):
Definition C.3 (Constructive abstraction). Given an a-abstraction a m, from M to M*, the abstraction is constructive if the abstraction is interventionally consistent and implies the existence of map ay 2 + 2h between exogenous variables.
C.5. Measure-theoretic definition of an SCM
Any SCM can be defined in terms of the probability measure spaces underlying it: Definition C.4 tMeasure-theoreeicc SCM). A (Markovian) SCM M" is triple ((44 Eu, C) (V, Ev, x) ,M) where:
(M; Eu, () is probability space associated with exogenous variables. Specifically, it consists of the product probability measure Ç = Ci X x S. on the product measurable space (4, Ex)) where it = 21 x x Ur is a product set and Ey = Es, ....& Lu, a product -lagebra. The probability measure is such that, for each W1 E Ex....., Wn. E Σu.. we have
Si x ... (M x ... x W.) (MW) ... x (WW.):
(15)
(V, Σy, x) is a probability space associated with endogenous variables consisting of joint probability measure on the product measurable space (V, Σy) = (V1 x... x Vn, Σv, 8...@ @ Σv.);
M is a set of n mixing measurable maps T (cf. Def. 2.1) such that the joint probability measure x factorizes as
X
X
(us (Ue x
(16)
1=1
where 411 je Uy. and, denoting by Ene; = space (1: x U4, Exe, 39 Eua.).
x jEA, Exyy Pi is probability measure on the product measurable
15

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Ind
Prob
(u, E)
M
(V. Σv.x)
Figure 5: An SCM is a functor (purple arrows) from Ind (left) to Prob (right).
D. Category-theoretic Formalization
This section extends the category-theore formalization introduced in the main paper to intervened models and abstraction.
Recall the category-theoree definition from the main paper:
Definition D.1 Category-theoretio SCM). An SCM is functor M Ind + Prob, mapping the source node of Ind to the probability space associated with the exogenous variables (U, Eu, S), the sink node of Ind to the probability space associated with the endogenous variables (V, Σy, x), and the only edge of Ind to the measurable map induced by the set JF of functional assignments.
Fig. 5 offers a depiction of an SCM as a functor.
In the same vein, directly the post-interventional model M, as in Def. D.I, we will adopt representation that is closer to the intervention operator itself. First, notice that, whenever the domains of the variables of an SCM are continuous, we can represent an intervention as a measurable map by relying on the truncation formula (Pearl, 2009):
have functorial representation for intervened SCMs as well. However, instead of representing
Lemma D.2. Given continuous Markovian SCM M = ((4, Eu, C)2 (V, 2 and an intervention e. on M there exists a measurable map p. from the probability space of endogenous variables of the pre-interventional SCM (V, Ev, x) to the probability space of endogenous variables of the post- interventional SCM (V., Ev,:
Proof. Given a Markovian SCM M", the probability measure X over the measure space endogenous variables (V, Σv, x) can be expressed by through the factorization over the endogenous variables X IIke(m] P (X:lP;, Zi). with X P (X, Z,). Given intervention i = do(X'' = x') on M", the new post-interventional measure x" can be computed through the truncation formula (Pearl, 2009):
M P(X,|,, Z,)
if if x'
x
(17)
We can now define measurable map ø' connecting (V, Σp,x) and (V, Σx,x') such that s (x) = X². Specifically, for each X, E 2², (XX,) = rij thus guaranteeing the distribution on the second line of Eq. (17); for each X, & X", denoting by x = PP, Z,) evaluated at X' = x' as in the first line of Eq. (17), we solve a measure transport problem (Marzouk et al., 2016) from Xi to X which, in the continuous case, guarantees a transport map over the domains that satisfies the distribution on the first line of Eq. (17).
We can then encode an intervened model as follows:
Definition D.3 (Category-theoretie post-interventional SCM). A post-interventional SCM is functor M Ind - Prob, where the functor maps the source node of Ind to the probability space associated with the endogenous variables of the pre-interventional SCM (V, Σy, x). the sink node of Ind to the probability space associated with the endogenous variables of the postvinnervlntionnal SCM (V., Σv,, X.), and the only edge of Ind to the function 42, encoding the intervention
This construction gives rise to the structure in Fig. 6 and an immediate category-theory expression of abstraction equivalent to Def.2.3: Lemma D.4. An interventionally consistent abstraction a singular natural transformation a, that is, a morphism av in Prob, that, for all intervention guarantees the commutativity of the diagrams constructed from from Fig. 2.
16

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Ind
Prob
M (ut, (9) M (Ve, Eve,x
(V,, Eye, x.)
Olighh
ayy
(t Men, {h) Mh (vh, Vyn, x²)
M (2
Viph X.x
Figure 6: Representation of M (blue), Mg (cyan), M (red), MC (orange) as functors. An abstraction is just natural fransformation, that is, set of commuting arrows in Prob (dashed black). Notice two commuting diagrams in Prob: the first observational one rooted on the exogenous variables (M* aiee aya o MM), the second interventional one connecting observational and interventional model (M) o Crph = Cyr o M.).
Proof. Recall the definition of interventional consistency in Def. C.I:
ay (PPO(ddo(X())) P() (d) (do(
(18)
Let us relate this definition to our categorical notation. First, Cryz and ox2 are components of the abstraction map a in the categorical notation, this map correspond to oy. The probability distribution P() do(x)) is distribution in the low-level model; with no loss of generality, assuming 2 to encompass all the non-intervened variables, this distribution correspond to the measure x furthermore, the interventional measure X can be obtained through the pushforward of the observational measure x via the interventional mixing functions M as by Lemma D.4. Finally, the probability distribution P()% xx (d)()))) is distribution the high-level model; again, with no loss generality, assuming > encompass all the non- non-intervened variables, this distribution correspond the measure X where is the abstraction of the terms in 4. Also, as before, the interventional measure x can be obtained through the pushforward of the observational measure x" via the interventional mixing functions M thanks to Lemma D.4. We then obtain rewriting of abstraction as:
ay o M MM aa.
(19)
corresponding to the commutativity of the right diagram in Fig. 6, for all interventions.
E. Stiefel Manifold
We now provide a short review of the Stiefel manifold, referring the interested reader to (Absil et al., 2008; Boumal, 2023)
for a comprehensive discussion.
Given f, h N, h the Stiefel manifold is the of l x matrices with orthonormal columns, mathematically
St(e, (V R V VTV= [b}-
(20)
Consider the function g R + Sym(h), 9 (VV) V V Ih is well-known that g is generating function for St(E, h), thus making an embedded submanifold of Réxh, with dimension dim Réxh dim Sym(h) = th lihh + 1)/2. Given point of the manifold V, the tangent space to St(E, h) can be defined implicitly as the kernel of the differential of at V,
TySt(E,h) = (G e fx G
(21)
We consider the Riemannian metric as the restriction of the Eucliden product between two matrices in REXA to St h). Accordingly, given A, B E St, ht,, we have (A, B Tr A B. The tangent space linearizes the manifold around V, then, we move away from V along the directions in SySt(e, h). However, to make such movement smooth along the manifold, we employ the retraction map Ry (:) St h) + St(E, h). The retraction has to satisfy the following conditions
|Ry (G) - (V + G) = 0.
(i) Ry (0Exh) = V, and (i)
lim
(22)
G G-00xh
|G||p
17

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Among the canonical retractions, we have
R (G) = qf(V + G), [QR retraction]
RV (G) (V + G - VV
Polar
[Polar retraction]
(23)
R (G) (I' W(G)) (I' W(G))V;
Caley
[Caley retraction]
where of indicates the Q factor of the QR decomposition, and W(G) = -VT=VVVTGVT - -V - VVVT).
Finally, the normal space to the manifold has the following explicit form
NySt(e, h) = (VS) {Sym(h))
(24)
Starting from Eq. (24), the orthogonal projection to TySt(L, h), namely Projv. has to be such that G - Projy G lies onto NySt(e,h),
G P-ooojy VS.
(25)
Plugging Eq. (25) into Eq. (21), it can be derived that
Projy (I' V G V
(26)
2
Finally, for St(e,h) h) (and in general for Riemannian submanifolds) the Riemannian gradient of V is the orthogonal projection of Vvf to TySt(E, h). Mathematically, starting from Eq. (26), have
Vvʃ = Projy Vvs.
(27)
F. information-theoreiie Distance on Spaces of Different Dimensionality
Two types of distances can be defined as follows using an affine map yp V.b. (Cai & Lim, 2022).
Definition F.1 (Embedding and projection distances). Let f, h E N with h < l, and let V. =
+ be an
affine map with V St(e, h) and b e R For any measures x E M(RR)) and x E M(RR), the set of embeddings of x" into R the set of of Ê-dimensional measures, defined as follows:
x+ (x = fa MMRMM((R)))) :
94 (a) x
(28)
Similarly, the set of projections of x into R² is the set of h-dimensional measures defined as:
Φ 0x( = (B @ x (x9) 33
(29)
Now, for any given distance measure D(. defined in M(R)), we can define the embedding distance D (x"
=
infaeo+ (x*.0) .ry D(a, v) and the projection distance D D- (x"xx") = inf (x.M) D 3)).
Embedding and projection distances can measure distances between probability measures of different dimensions. Addition- ally, Theorem 1.2 (Cai Lim, 2022) states that the former two distances are equivalent, that is, for number of different distance metries and d-divergences, D+ (x*,x² D- (x", x D(x x²), implying that computing the embedding distance the projection distance yields the same result.
G. Proofs
Theorem 4.3. Les Lexx ~ N(O,, E¹, x N(O,,, E"), where E' E St_ and Eh E Sh. Denote by < S Se the eigenvalues of L and by < K1 S x" exists, then
S Kn. those of Eh. If linear CA V E St(E, h) complying with SEP from x to
X S Ki S Aite-h
Vie [M].
(4)
18

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Proof. a linear CA V St(L, exists, then x" p (x"). This implies that the kernel of any cnnemmtiionttheoreeee
metric or (-divergence is nonempty, and also V E = D". The latter determines the eigenvalues of V Σ V are those of Σe. By the Ostrowski's ' theorem for rectangular V - cf. Th.3.2 in (Higham & Cheng, 1998) we have
K4 = 0,p4; with E [M],
(30)
where
d S us S Aite-Ai
(31)
and
eigvls(V (ool(VVVV) V) S S eigyls(V
(32)
Since V E St.C, k). by Eq. (32) 9; = for each E [b] Substituting the latter into Eq. (30), we get Ri = 14; thus obtaining Eq. (4) by Eq. (31).
Proposition 5.1. Consider the function
((A)= [()= -1 " + log del Σ
AATAAAAAAAAA AA
(9)
Eq. (9) is smooth for A St(E, h). Additionally, define A = (A E'A)) -1
The gradient of f (A) is
Va 2 (I- E²
(10)
is well-defined and smooth in case L is positive definite (Boyd & Vandenberghe, 2004). If AT A S for all E RRh, y j 02 holds T A T Ay > By defining RC 3 z = Ay, this is equivalent to say 2. Σ' > 0, vy * On. Since Σ' E S by assumption, we have to prove that + Og, Vy + O₁ Consider that exists * O, such that ă Aý Og. This means that
Proof. Consider the first term Tr T (ATE'A)
-1 Σ² '''' in Eq. (9). We have that Tr (ATE'A) Σい
On A A A Aÿ = = シ 0n:
(33)
which contradiction. Hence A T A Sh and Tr (A A' >d} is smooth over St(e, h). Consider now log det A A' in Eq. (8). Since A T A Sh also this latter term is well-defined and smooth.
Let A (A (AT A) -1 The gradient Eq. (10) follows from the application of the following rules of matrix calculus (Brookes, 2020),
(i) (AT(AA'' " 'x" --2E A Ā and (ii) V dog A AĀ,
(34)
H. LinSEPAL-DDMM
Let us recall below the nonsmooth Riemannian problem have to solve.
Problem 2. Given E S Ed Sh DE (0, 1jexh, and R4, the CA is the transpose of
V arg min f(V) + D O
(5)
VESt(E,A)
h(V)
Here,
Eq. (3), omitting the constant C.
The structure of the objective in (5), separating into smooth (cf. Proposition 5.1) and nonsmooth terms, makes the alternating direction method of multipliers (ADMM, Boyd et al., 2011) an appealing optimization framework for deriving a solution.
This is the rationale behind the general framework manifold ADMM (Kovnatsky et al., 2016), that we decline to our setting in the following, thus obtaining the LinSEPAL-ADDM algorithm.
19

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Starting from (5), add splitting variable Y IRexh be optimized over the Euclidean space handle the non-smooth term h(V);
The ['VV - >="} log det d E'V) + 1I
min VESEA subject to
(P2)
Y D D V 0exh-
At this point, following (Boyd et al., 1), by denoting by U Rfxh the scaled dual variable, and by p R+ the ADMM stepsize, the scaled augmented Lagrangian reads as
Lp YV = TT TT (VTE'V) "' Σ²
log det VVVV + X||I|| + I D -
(35)
Starting from Eq. (35), the ADMM updates at the k-th iteration are
Vk+1_
arg arg min Lp (V, (V,Y', U*) VESt(E,A)
y arg min Lp (Vk+1 Y, U*), YEREXA
(RI)
U = U +DO V y
Solution for Vk+1
The update for Vk+1 in (R1) reads
V arg
arg min Tr (V ' + log det ( V E'v) + D - Y + U
(36)
VESt(E.))
Eq. (36) is standard smooth optimization problem over the Stiefel manifold, and it can be solved by standard techniques such as those in (Boumal, 2023). Newton and conjugate gradient methods for the Stiefel manifold are discussed in (Edelman et al., 1998). In our experiments, we use the conjugate gradient implementation in (Boumal et al., 2014).
Solution for Y The update for Yk+1 (RI) reads as
Yk+1 arg min AI Y
2 L - - 2
YEReXE
(37)
= Sxpp (D vk+1 + U*);
where Sz(x) = sign(x) max([al
6, 0) the element-wise soft-thresholding operator (Parikh et al., 2014).
Stopping criteria. The empirical convergence of LinSEPAL-ADMM, is established according primal and dual feasibility optimality conditions (Boyd et al., 2011). The primal residual, associated with the equality constraint in Eq. (P2), is
R k+1 = k+1 D o V++1
(38)
The dual residual, which can be obtained from the stationarity condition, is
R++1 6++1 = p 3 (Y -Yk)
(39)
As + oo, the norm of the primal and dual residuals should vanish. Hence, the stopping criterion can in terms of the norms
(i) d
R +| and (ii) db R ||
(40)
Specifically, given absolute and relative tolerance, namely T and -" in R4, respectively, convergence in practice is established following Boyd et al. (2011) when
(i) dp S 7 Vth + max (| I|. Do v*+1llp, and (i) da S " D
(41)
20

---

Causal Abstraction Learning based on the Semantic Embedding Principle
The LinSEPAL-ADMM algorithm is summarized in Algorithm 1.
Algorithm 1 LinSEPAL-ADMN
1: Input: E', Σn, D, A, p. T"
2: Initialize: V St(E, h), Y Rexh, U + D v Y
3: repeat
4: V <- Solve Eq. (36) via tef-the-shelf method for smooth Riemannian problems
5: Yk+1 +- Sx/p (D @ Vk+1 + U*)
6: Uk+1 <- U D vk+1 Y
7: until Eq. (41) is satisfied
8: Output: V, Y, U
I. LinSEPAL-PG
This method based upon the manifold proximal gradient (Chen al., 2020) framework, which generalizes the proximal gradient framework defined in the Euclidean space the Stiefel manifold. Following Chen et al. (2020), denoting by V the iterate at the step k; the updates recursion for solving (5) reads as
k² arg min
(Vf (V*). G) + 2p
|G| + AID O (V* +G]l
G
(R2)
Vl+1= = Ryv (G*)
In (R2), the first update is the proximal mapping providing a proximal gradient direction G onto the tangent space to the Stiefel manifold, using the first-order approximation of the objective around the k-tth estimate. The second is the update for Vk+1, which exploits the canonical retraction (cf. Eq. (23)) technique for projecting back V + G from the tangent space to the manifold. Global convergence of the ManPG method has been established in Chen et al. (2020).
Solution for G Chen al. (2020) shows that the first update can be efficiently solved using the regularized semi-smooth Newton method in Xiao et al. (2018). Specifically, according Eq. (21), the feasible set St(S, h) translates into linear constraint. By defining A (G) T V G, the update
G arg min
( (V*). G) +
1 2p
|GIl
+ (V* +G]
GeR
(42)
subject to A (G) 0axh
However, following the rationale in Si et al. (2024), we can force G* E Stx, h) by exploiting the basis Byx of the normal space to the manifold, namely St,E h). To find such By:, recall the explicit form of Ny h) in Eq. (24).
The basis of Sym(h), having dimension hah 1)/2, is
Eij {0, hxxh
Eij has eig Eji 1, 0 elsewhere, isisis hh)
(43)
follows from Eqs. (24) and (43) that
By = [ V'E,, isisish]
(44)
At this point, the membership to St h) can be expressed as
BB G) =0, 0, visigish.
(45)
Hence, (42) reads as
1
G arg min
(V)( G) + | + D (v* +G
GeRex
2p
(46)
subject to
B G) = 0, visisish.
21

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Consider h (V* G) Lagrangian for (46) is
[M11 (h11:/12.--- M12.-..:111 Mejo.... :Mj Panl with 1 S S j h. The | (V* G) and Ra /a
p (VF (v*). G) + 2p G + 2h (V* + G)
E
Piy ( G).
(47)
1sisish
Let us define now the matrix R*xth B |vec (Bf)) (B ..... hh)) IT, where vec (Bij) E R We can compactly express the equality constraints
Bevec (G) 0,.
(48)
Thus, the Frrush-Kuhn-Teee (KKT) conditions of Eq. (42) reads as
(i) Otxc E 0GLp (G, M)) and (ii) vec (G) = 0,.
(49)
From the stationarity condition get
Otxh G (V(VV) E By
xxpgg (V* +G)
(50)
ISISIsh
At this point, recalling the inclusion property of proximal operators, viz. prox, - B P Og(P), we have
Oxx E +G
V*- - P (S(VV) Σ
Big
+xpdch (V- + G
(51)
ISiSiSh
B(4)
from which get
G(p) proxx (B (H1) Vk
(52)
At this point, proxapM ph-) can be computed element-wise as
(H)( SAp(bis(14))
if d, 0, otherwise
Prox (bis (44)
(53)
Substituting Eq. (52) into Eq. (49), we have
ve (G( 02
(54)
Here the r-th entry of vec (G(H)) corresponds to the entry of G(u) at row u (r - 1) mod E + 1, and column v - 1)/E] + 1,re [eh].
At this point, can use the regularized semi-smooth Newton method (Xiao al., 2018) to solve Eq. (54). Our target function is
F(u) v (G(μ)) R + R
(55)
By the chain rule of calculus, using Eq. (52), the generalized Jacobian matrix
OF(u) Ovec (G(H))
paxa 3 J = Ivec (G(1))
Öμ Oproxxpao pn)() (vee (B(μ)))
(56)
Ovec (B(p)) 0p
Be
Ovec (B(u))
22

---

Causal Abstraction Learning based the Semantic Embedding Principle
The proximal-related term is diagonal matrix M E (Rthxeh, where
1 if vec (D), 0 or (vee (D), and |b,| Ap 0)
Myr 0
(57)
otherwise.
Additionally, starting from
[ wo ))
(14 = V P Vs(vi)) E
Mij
ISiSiSh
(58)
(V* (Vs((it) - b
-b(.. 11)
Hence, get
R 3 0b,((1)
b
(59)
0μ
Consequently, starting from Eq. (56), using Eqs. (57) and (59), we finally have
11
by
J == MC, with g C=
(60)
Eh
Following Xiao al. (2018), denoting with n* | 12 a R-R. we define
, (J=I df pk-1
(61)
At each iteration we want find the step d by solving Eq. (61) inexactly, such that
||^ S t min (1, +-1 | 100- d 122) T (0,
(62)
obtaining a trial point
u = p + d
(63)
Let 50 = 1F44)) and E (0.1) 1). If F(uu)))) )12 I 7 then we set
1 = n gh = =|F(("))l2> and a = k-1
[Newton step]
(64)
Otherwise, let
<h -F(u))
(65)
|
Select 0 < Φ1 Φ2 < anddd 1 < th 22. Hence, make safeguard step follows
c* 2 Φ1 and
F(v*)
F( F
[projection step]
Wk
g* Φ1 and
[F(v*)
|| [fixed-point step]
(66)
k-1
if s* Φi, unsuccessful step
where
F(u²) T u
F(u*), w =-r-11 - oF(H*), de (0, =
rk
n 8-1
(67)
F (1*))]]
23

---

Causal Abstraction Learning based the Semantic Embedding Principle
where w E (0, 1]. Finally, denoting R+ O, the parameters 3h+1 and a updated
(ā, (ā,ah=1);
if g 2 d2
Bh = Bh-1 a E [a--1 11114-11, (viak-11 12204-1],
if Φ1 S Φ2 otherwise.
(68)
At this point, we set G (1**)) according to Eq. (52).
Solution for V Given V + G Stv, h). we have to project the point onto the manifold. This can be accomplished via the canonical retractions in Eq. (23). However, as suggested by Chen al. (2020), our LinSEPAL-PG implementation performs an Armijo line-search procedure determine the stepsize a. Hence, the update
V
QR (aG*)
(69)
Stopping criteria. Empirical convergence the LinSEPAL-PO algorithm established either when maximum number of iterations K is reached, when the DKL,, is below a certain threshold , O. The LinSEPAL-PG algorithm is summarized in Algorithm 2.
Algorithm 2 LinSEPAL-PG
Input: Σ', Σh, D, X, p. 7 E (0, 1), KL K
2: Initialize: v St(E, h). Y Rexh U Réxh
3: repeat
4: G* +- Solve Eq. (46) via the regularized semi-smooth Newton method
5: a a+1
6: repeat
7:
a= 7/a
8:
V R (a G
a||G*|/
9: until DII. > DKL.
VE
2p
10: Vi+1_ < V
11: until k > K Vki1 < KL.
12: Output: V
J. CLinSEPAL
The problem we want solve is:
Problem 3. Given E St,,, Wdh E St and B e {0, 1x
xh, the linear constructive CA is given by the transpose of the
product B S V, where
V*,S*= arg min
f(V.S);
VeR S
subject to
B
(7)
(ii (BoS) (p))
(iii) - 3 8) 1.00;
and
((V.)) HV (BOS k
(8)
+log det (B
Prob. makes it explicit that the abstraction morphism is given by three key ingredients: (i) the given partial, structural prior information represented by B; (ii) the structural component to learned, such that the resulting causal abstraction
24

---

Causal Abstraction Learning based on the Semantic Embedding Principle
constructive; and (iii) the abstraction coefficients in V determining the linear functional forms of the causal abstraction, which have to be learned well. Specifically, "partial" means that some rows of B have more than one entry equal one.
Unfortunately, Prob. 3 is nonconvex because of the objective function and the Stiefel manifold. Additionally, in this case, the CA results in bilinear form B @ S @ V, which is not jointly convex in S and V. Consequently, the constraint B L@S S O V E St(e, h) has to be carefully handled.
Regarding the nonconvexity of the objective in Eq. (8), we proceed by leveraging its smoothness. Specifically, we have the following result.
The function S(V,S) S) in Eg. (8) is smooth. Additionally, define A The partial derivatives w.E. V and S are
Corollary J.1. The
B
V) and À (ATE'A) -1
Vv/=2(BoS) ((E'AA) (I--AAA)
(70)
and
vsf= T (EEAA) (I1-
(71)
Proof. Smoothness directly follows from Proposition 5.1 by defining A (B o S @ V). which is constrained to St(e, h) as given in Eq. (7). The partial derivatives in Eqs. (70) and (71) follow from the application of Eq. (34), together with the chain rule for derivatives.
At this point, we leverage Corollary J.1 to provide a solution which combines ADMM (Boyd al., 2011) and SCA (Nedić et al., 2018). Specifically, ADMM is suitable to isolate and consequently tackle the nonconvexity in different subproblems. To manage the bilinear form within the first constraint in Eq. (7), we introduce two splitting variables, namely Y1 and Y, in St h), and the corresponding equality constraints
Y1
B
S V O
Orxh and Y2- Y Vi+1 @ S = Ozxh, respectively.
(72)
In this way, given the solution at iteration & within the ADMM framework, we optimize separately over V and S while always tracking St(E, h). Please notice that we use V since when optimizing over S, V has already been updated. The rationale behind the usage of the splitting variable for handling the Stiefel manifold is the same as the splitring of orthogonality constraints method (SOC, Lai & Osher, 2014). Additionally, to handle (B @ S) T E Sp (h, E), we introduce
another splitting variable X E Spa E), and the corresponding equality constraint X (B @ S) from Eq. (7), we get the following equivalent minimization problem
Oxxe- Thus, starting
v', Yi, Y2
arg min Verexi 1je0 YLESt(r,A)
f(V.S):
Y2ESt(E,h)
XespA
subject to
Y1 B S* O V 0txh
(73)
Y, B vh+1 @S = 0zxh
X (B G 0
1 (B の S) 1, < 0g
Starting from Eq. (73), considering the penalty E R4, we introduce the scaled dual variables U, and U, in Réxh, and W E Rixe and write the scaled augmented Lagrangian
L (V,S,Y, YYS.YY,YY,,X.... X,U, U, W) -f(V BOS ④ V Y, + U
(74)
B OS
- Y2 U2 2 + - (OO)) - x w/
25

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Now, we can apply ADMM iterative procedure, getting the recursion for updating the primal and scaled dual variables. In detail, denote by IN the current iteration. We have
vk+ arg min Lp (V, St Y Y x Uf Ug w*); VeRtxh
Sk+1 arg min Lp (V++1 S, Y Y x U U. w*). SEjU,1jxXA
subject to
1
(B の S) 1. 0p:
Y1= = arg min YIESt(E,A)
4p
[v gk+1 11, Y x U U w*);
Y arg min Y2ESt(E,))
Txp [v++1 S r Y, x U Ug. w
(75)
X arg min Lp (vk+1 sk+1 y Y x, U U w*) (h,)*(h,))
U - + (B Vk+! - -Y
+ - (B V O Skt y
W w + (Bo S -x*+1
Similarly to SOC, we isolate the objective nonconvexity into the first and second (nonconvex) subproblems; and the nonconvexity of the manifold into the third and fourth (nonconvex) ones. Notably, the first and second subproblems can be managed through SCA. Additionally, the third and fourth nonconvex subproblems admit closed-fone solutions since they boil down to the closest orthogonal approximation problems (Fan & Hoffman, 1955; Higham, 1986). Thus, the latter nonconvexity is somehow resolved. Finally, we solve the subproblem for x in closed form as well.
J.1. Update for V
Starting from Eqs. (74) and (75), the subproblem we have to solve is
V
arg min
J(V,S*)+ BOS [ LoV-Y- Yf + +f
(76)
VER(xA
Eq. (76) is nonconvex due to the inherent nonconvexity of S(V,S*). However, the latter function is smooth and differentiable w.r.t. V, as given in Corollary J.1. Hence, we apply the SCA framework. In detail, denote by q the SCA iteration and set y 4 for q 0. We derive strongly convex surrogate f(V; V1,S*) around the point V i... the solution the iterate g exploiting Eq. (70):
VVV = IV(VVITVV.s) (V (V- + V - |
(77)
It is immediate check that Eq. (77) is proper surrogate satisfying the stationarity condition Vv Vvjlvo
Therefore, at each SCA iteration g, solve strongly convex problem in closed-form and then apply the usual smoothing operation by using a diminishing stepsize 7 E R4. Specifically,
Ve+1_
arg min
J(V; V (9,
B Vv-
Y
U
(Strongly convex problem)
VERXX
(78)
V = V9 (yet1_ V9). (Smoothing)
The solution of the strongly-convex problem is given element-wise wise in Lemma J.2.
Lemma J.2. The update for V can be computed wlement-wise as
.+01 1
pbisotioi pbssttyuii + +
(79)
+ bigsh
26

---

Causal Abstraction Learning based the Semantic Embedding Principle
Proof. The proof follows by imposing the stationarity condition
Otxh Vvfl(vess + +
+
B BOO
V Y Uū),
(80)
and solving for V.
Additionally, the diminishing stepsize 7 has to satisfy the classical stochastic approximation conditions (Nedić et al., 2018),
Σ 7 oo and
(ii) Σ (79) <..
(81)
In our experiments, use the decaying rule
7 = 7" (1- E (0,
(82)
The SCA framework is guaranteed to converge stationary points of the original nonconvex problem in Eq. (76) (Nedié et al., 2018). Accordingly, we establish convergence for the update when
| V ST, ==
(83)
and set Vk+1 V
J.2. Update for S*+1
Starting from Eqs. (74) and (75), the subproblem we have to solve is
gh+1 arg min f(V*, 5) + f vk+ os Y + U - (B -x* + w F 1e(0,
(84)
subject to 1 (B e S) 1, 0
The subproblem above is nonconvex and constrained. Similarly to App. J.1, we apply the SCA framework. Denote by q the SCA iteration and S S* for 0. Here, the strongly convex surrogate of f(V++1, .S) reads as
F(S: v++1 (4) = Tr Vsf (ver (S (S--S)
|S 1
(85)
which satisfies Vs/l(ve- VsJl(vet1 .8y). At each SCA iteration we solve a constrained quadratic programming (QP) problem and apply the smoothing step by using the stepsize 7" E R4 complying with the conditions in Eq. (81). In detail, let vec (A) be the column-wise vectorization of a given matrix A and define
Q rI + pdiag ((vec (B) @ vec (V++1)) @ (vec (B) @ vec (V*+1))) + pdiag (vec (B) O vec (B))
vec (Vsfl(ve+,sa.)) - voc (St) - p vec (Y U O vec (B) @ vec (Vk+1) vec (B) O vec -wh)
(86)
Additionally, recall that vec (AC) (C* La) vee (A). with A E Rixe and C Rtxm Hence, denoting with Ktth the commutation matrix, the inequality constraint can be rewritten as
1 vec (B 2) 1,) = 1I - (17 & I.) vec (B + S)
1 (1 @ In) K h vec (B @ S)
1n (1) @ In) K Mivee (diag (vec (B)) vec (S))
(87)
= 1 (1] & I, K diag (vec (B)) vec (S) S 0n.
At this point, starting from Eq. (84) and exploiting Eqs. (86) and (87), we can pose the SCA recursion:
vec (S) T Qvec (S) + e vec (S), (QP problem) vec (S)++1= arg min 1j
(88)
subject to 1 Gvec (S) S O
vec (S) = vec (S) + (vec (S)++1 vec ($)8) (Smoothing)
27

---

Causal Abstraction Learning based the Semantic Embedding Principle
The QP problem in Eq. (88) can be solved through off-the-shelf quadratic programming solvers. In our experiments,
the OSQP (Stellato et al., 2020) implementation available in cvxpy (Diamond & Boyd, 2016). Since the quadratic form involves diagonal, positive definite matrix Q, in case a solution exists in the feasible set determined by the inequality constraint, it is also unique. Regarding the smoothing step. 7 follows Eq. (82). Similarly to App. J.1, we determine convergence when
|| vee (S) - vec (5) STa 0
(89)
and set S Sh+1, where S is the reshaping vec (S) in matrix form.
J.3. Update for yk+1 and Y
Starting from Eqs. (74) and (75), the subproblem to solve is
* arg min Y,ES((E,))
B C vh+l - Yr + U
+1+Ull vk+1 + U
2
(90)
Y1
with Y = B sk+
proxs((t,))
The evaluation of proxsu((.A) (1) in Eq. (90) is equivalent to the (unique) solution of the closest orthogonal approximation problem (Fan Hoffman, 1955; Higham, 1986). Specifically, equal to the Up, factor of the polar decomposition of the matrix Y,= = Up P Pp., namely
Y
=Up Up p1
(91)
Similarly, defining Y2 = B + gh+1 @ Vk+1
U and considering the polar decomposition Y; UU..P p2, we have
Y U p2
k
(92)
2
J.4. Update for X
Starting from Eqs. (74) and (75), the subproblem reads
x k+1
arg min (1,))(h,))
6 (B (B - x w
(
(93)
(BBo + w+)
proxspA(h.0)
The following result gives the solution. Lemma J.3. Consider
(A (0, 10, |||y/|| = 1 and Σ
0g = 1, Vj 11]
(94)
1=1
and E BRxxf, The proximal operator
proxsp (h,9) (A) = arg min
A - -X/lpp
(95)
XeRix
the matrix X* such that
1,
ifa = arg min; aaj - 1/.
Vj [[], z =
(96)
0, otherwise.
Proof. To belong to Sp (h, E). X* must have only a single nonzero entry equal to one for each column j E [é]. Consequently. the objective in Eq. (95) is minimized by setting, for each column j E [6]. ztz = in correspondence of the element a.j whose absolute distance from one is minimum.
X*
28

---

Causal Abstraction Learning based on the Semantic Embedding Principle
J.S. Stopping criteria
The empirical convergence of the proposed method is established according to primal and dual feasibility optimality conditions. In this case, the primal residuals associated with the equality constraints in Eq. (73) are
R k+1 P,1 = Y - B о S C Vk+1;
R
Rk+1
Y B v O Sk+1
(97)
p:2
4+1 p,3 =
x
[B o S*S++
Additionally, the dual residuals obtained from the stationarity condition are
R = B o s @ (Y yk+1_ - Y1
R p V (Y2 y Y2))
(98)
Rk+1 = B O X k+l x d,3
Following (Boyd et al., 2011), denoting with 7 and T" in R4 the absolute and relative tolerances, respectively, the stopping criteria be satisfied for empirical convergence are
R k+1
k+1 S STVehh T" max Y k+1
B S o 11))
R
Rk+1
<T"Vhh + 7" max k+1 n2
B vk+ @ s 11)))
p.2
Rk41
d+1 ~-\/hh + T" max
B B k I
p3
R+41
dtl S /"Vth + 7' B
S* U
(99)
R
d+1 -- Vih + + B ++
U 1
d2
F
R
k+|
d+1 < B 3 W w
d,3
The CLinSEPAL method is summarized in Algorithm 3
Algorithm 3 CLinSEPAL
1: Input: Σ hh B. P, T", -².7"
2: Initialize: y E Rtxh 5 = B. Y E St,, vg E AU X = BT, U <
B S
V
Yp. U2 +-
B @ S
④ Vo Y W + (B SO) X
3: repeat
4: V4+1 + Apply Eq. (78)
5: Sk+1 Apply Eq. (88)
Y Eq. (91)
6:
7:
Y Eq. (92)
U + U + B @ vh+1 Y k++
8:
9:
U + U +
B vk+ o S Y
Wh+1 + W + + s x
10:
11:
until Eq. (99) satisfied
Output: S. Y,, Y. U, Ug W
J.6. Full prior case
Prob. simplifies in case of full prior knowledge of B. Indeed, it is not needed to learn S since B. Accordingly, we get the following.
29

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Problem 4. Given E' E ELS En 8 4++ and B E {0, 1exhh, the linear constructive CA is given by the transpose of the product B V, where
V arg min
f(V);
VERÉxA
(100)
subject to B V
and
S(V) J(V)= = ((OVVBBBBBBB)) - >² + log log (BOV) E'
(101)
The solution can be obtained in similar manner for the partial prior knowledge case. Below, report the mathematical derivation for completeness without further comments.
Corollary J.4. The function f(V) in Eq. (101) is smooth. Additionally, define A (B V) and A (ATE'A) The gradient is
Vvf 2B @ ((E'A)) (1- (I--AA))
(102)
Proof. Smoothness directly follows from Proposition 5.1 by defining A = (B @ V), which is constrained to St(f, h) as given in Eq. (100). The gradient in Eq. (102) follows from the application of Eq. (34), together with the chain rule for derivatives.
Starting from Eq. (100), we get the following equivalent minimization problem
V Y
arg min VaRf
f(V):
YESu(E,))
(103)
subject to
Y - B V = 0zxh
Considering the scaled dual variable U E Rexh, the scaled augmented Lagrangian is
Lp (v, ((V) +
(104)
The ADMM recursion is
kk+ arg min Lp (V, Y U*)
VERrXA
y++ arg min hp (v++1 Y. U²) YESt(E,h)
(105)
U - + B -
J.6.1. 1. UPDATE FOR Vk+1
Starting from Eqs. (104) and (105), the subproblem we have to solve is
b++ arg min
S(V) ( X +
(106)
VeRtx
Eq. (106) is nonconvex due to the inherent nonconvexity of f(V). However, the latter function is smooth and differentiable w.r.t. V, as given in Corollary J.4. Hence, we apply the SCA framework. In detail, denote by g the SCA iteration and set V 4 for q 0. We derive strongly convex surrogate f(V; V around the point -'-ie., the solution at the iterate 4 exploiting Eq. (102),
(V - VO) + | V
(107)
Therefore, at each SCA iteration q, we solve a strongly convex problem in closed-form and then apply the usual smoothing operation by using a diminishing stepsize 7 E R4 following Eq. (82) and satisfying Eq. (81). Specifically,
V arg min VeRexi
J(V; V.) B y U*
(Strongly convex problem)
(108)
V = V + (Vo+1_ V9)
(Smoothing)
30

---

Causal Abstraction Learning based the Semantic Embedding Principle
The solution of the strongly-convex problem is given element-wise in Lemma J.5. Lemma J.5. The update for V can be computed element-wise as
1 T + bis
pbuzti - pbisuty + Tulj [Vr/iv]]))
g+l
(109)
Proof. The proof follows by imposing the stationarity condition
Opxh Vv Vr + +
V Y +
(110)
and solving for V.
We establish convergence for the update when
|| VOIlp STa, =0
(111)
and set Vk+1 V
J.6.2. UPDATE FOR Yk+1
Starting from Eqs. (104) and (105), the subproblem solve
Y yk+1_
arg min YESU(EA) A
- U*
(112)
proXsu(e,) (Y), with Y = Bo vk+1 U
Denoting by P,P,p y the polar decomposition of the matrix Y, the update
y = Up
(113)
J.6.3. STOPPING CRITERIA
The empirical convergence of proposed method is established according to primal and dual feasibility optimality conditions (Boyd al., 2011). The primal residual, associated with the equality constraint Eq. (103),
R == y - B V++1
(114)
The dual residual, which can be obtained from the stationarity condition, is
R B (Y -y)+1_y()-.
(115)
As k + oo, the norm of the primal and dual residuals should vanish. Hence, the stopping criterion can be set in terms of the norms
( d |
lpp and (ii) d 1R*+1p
(116)
Specifically, given absolute and relative tolerance, namely T" and 7" in R4, respectively, convergence in practice established following Boyd al. 1), when
(i) d < / r" max Y ||||+I|| T BOV and (ii) di Thh + + U (117)
The full prior version of CLinSEPAL. summarized in Algorithm
Algorithm 4 CLinSEPAL (full prior case)
Input: Σ. Σd B, p. T, rf, 7 n²
2: Initialize: Vo Réxh y h), U B B B _
3: repeat
4: Vi+1 + Apply Eq. (108)
5: Yk+1 +- Eq. (113)
6: Uk+1 +- U Bo vh+1 yk+1
7: until Eq. (117) is satisfied
8: Oufput: V, Y, U
31

---

Causal Abstraction Learning based the Semantic Embedding Principle
K. Metrics and Hyper-parameterss
This section provides the definition metrics monitored our empirical assessment Sees. 6 and 7. Additionally, we report the hyper-parameene configuration for Algorithms used in the experiments.
Metrics. Denote by V* and V the ground-truth and the learned (transpose of the) linear CA, both being matrices in RéxA,
The metries are defined as follows.
Fraction of learned constructive morphisms: We define constructiveness as
constr (number of rows with one nonzero entry)/ + (number of columns with at least one nonzero entry)/h.
(118)
Then, indicating by S the number of experiments, the metric is given by the number of V leading to constr divided
by S.
KL divergence: Eq. (3) evaluated at V;
Frobenious absolute distance:
V
(119)
F1 score: Given
True positive rate tpr: (true positive, tp: number of predicted nonzero entries in V existing in V*Mmmme of nonzero entries in V*),
False discovery rate fdr: (false positive, fp: number of predicted nonzero entries in V that do not exist in V*)/(tp + fp);
the F1 results in the harmonic mean of tpr and (1 fdr).
Hy perrpaaraeeeers
CLinSEPAL: = T 10 30 = 0.1 for the full prior case and æ 0.01 for the partial prior case, T" = 10-3 T" = 10 7- = 10-4, The same hyper-parameters were used in the experiments in Sec. 7;
LIMMEPAL-ADMM: p 1, 1, T" 10-4, -- = -0-4;
LinSEPAL-PG: À 1 (2 -1 0.5, kK 10 K 1000
32

---

Causal Abstraction Learning based the Semantic Embedding Principle
L. Additional Results on Synthetic Data
Constroctivweee
divergnnce
3.0m
10
38-
1077
37-
8
36-
10-1
10-15
Froberis aborivte divance
Fl scowe
CUiSERAL LHSEPAL-ADMM
1.0m
0.0-
1iSFFAI-PP
3.5
0.5
3.6
341
8
0.7
3.0-
0.6-
0.25 Fraction unavailable prior knowlodge
0.75
0.25
0.5
0.75
Fraction of unarailable prior knowledgge
Figure 7: The figure shows the results in the pp setting where the 90% threshold on constructiveness was removed. Differently from Fig. 4, top left boxplots are for constructiveness values. The remaining plots are in Fig. From the figure (abstracted) low- and high-level probability measures (top right), the quality of learned CAs is poor compared to that of learned CAs from CLinSEPAL (bottom plots).
that although LinSEPAL-ADMM and LinSEPAL-PG minimize the misalignment between the
M. Additional Material for the Causal Abstraction of Brain Networks
This section provides additional material about the full and partial prior applications of CLinSEPAL to brain data, given in See. Specifically, Fig. depicts the ground truth linear CA and the learned linear CA by CLinSEPAL for the full prior setting: whereas Fig. 10 the results for the partial prior setting. Regarding the partial prior setting, we also report in Fig. 9 the partial prior received understand the performance of CLinSEPAL with varying degree of uncertainty (low, medium, high), as discussed in Sec. 7. The color coding for the partial prior setting refers to the following classification, reported unaltered from (D Acunto et al., 2024):
input by CLinSEPAL for all the settings, and in Fig. 11 the monitored metrics better
Red for ROIs corresponding to cognitive functions, attention, emotion, and decision makingg
Orange for those related auditory processing, speech and language processing, and memory;
Blue for those concerning memory formation and memory retrieval;
Pink for those associated with sensory integration and somatosensory;
Purple for the ROIs within the visual network and related to the visual memory;
Green for those within the motor network;
Yellow for those regarding the motor control and the posture.
33

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Ground truth causal abstraction morphism
Central
Cerebellum
Frontal
Lateral
Frontal
Medial
Frontal
Orbital
Macro ROIs
Insula
Limbic
Occipital
Parietal
Lateral
Parietal Medial
Subcortical Gray Nuclei
Temporal
Lateral
Temporal Pole
Vermis
Learned causal abstraction morphism
Central
Cerebellum
Frontal Lateral
Frontal
Medial
Frontal
Orbital
Macro ROIs
Insula
Limbic
Occipital
Parietal -
Lateral
Parictal Medial
Subcortical Gray Nuclei
Temporal Lateral
Temporal Pole
Vermis
PrecGy
RolandOperc
Postcentral
Cereb II-
VI-
Cereb VII X
FrantMid
Sup
FrontSupMed Med
TrontinfTn
FrontinfOperc
SMA
FrontSupO
Paracentral Lob
Olfactory
FrontMed
FrantMiddrbs
FrontinfOrb
Insula
CingAnt
CingMid-
CingPost-
Hippocampus
ParaHippoc
Lingual
Occipital
Cuncus
Calcarine
Fusiform
Parieta Sup
Parietal Inf
SupraMarginal
Ang
Precuneus
Amygdala
Pa Pallidum-
Putamen
Caudate
Thalamus
Hesch
Temp Sup
TempMid.
Temp
TempPole-
Vermis
Cereb III
ROIs
0.0
0.2
0.4
0.6
0.8
1.0
Figure 8: The figure shows (top) the ground truth linear CA and (bottom) the learned linear CA for the simulated full prior setting in Sec. 7.
34

---

Causal Abstraction Learning based the Semantic Embedding Principle
Ground truth causal abstraction structure
Labels func. activity
black
blue
green
magenta
orange
purple
red-
yellow
Partial causal abstraction prior, low uncertainty case
Labels func. activity
black
blue
green-
magenta-
orange
purple-
red-
yellow
Partial causal abstraction prior, medium uncertainty
Labels func. activity
black
green
magenta-
orange
purple-
red-
yellow
Partial causal abstraction prior, high uncertainty case
Labels func. activity
black
blue
green
magenta
orange
purple
yellow-
Olfactory
Pippo-ippoc
Hippocampus pus
PeracentralLobe Lob
SMA
PrecGy
Thalamus
ParietalSup
Postcentral
FrontinfTri
Heschl-
RolandOperc
TempSup
SempPole-
SupraMargan Marginal
Temp
FrontInfOpers
Occipital
Fusiform
TempInf
Cuneus
Lingual-
Calcarine
FrontMedOrb
FrontMid
FrontMiddrbb
FrontinfOrt
Front SupMed
FrontS SpontSupOrbs rb
Precuneus
SuntSup
Insula
Parietallnf
CingAnt
CingMid
CingPost
Amygdala
Angular
Cereb III- VI
Cereb Il
Careb VII X
Putamen
Caudate
Pallidum
Vermis
ROIs
Figure 9: Starting from the top, the figure shows (i) the ground truth structure for linear CA, and the partial prior provided as input to CLinSEPALeerreed linear CA for the simulated partial prior the (ii) low, (iii) medium, and (iv) high uncertainty settings discussed in Sec. 7.
35

---

Causal Abstraction Learning based on the Semantic Embedding Principle
Ground truth causal abstraction morphism
Labels func. activity
black
blue
green
magenta
orange
purple
yellow
Learned causal abstraction morphism, low uncertainty case
Labels func, activity
black
blue
green
magenta
orange
purple
red
yellow
Learned causal abstraction morphism, medium uncertainty case
Labels func. activity
black
blue
green
magenta
orange
purple
red
yellow
Learned causal abstraction morphism, high uncertainty case
Labels func. activity
black
blue
green
magenta
orange
purple
yellow-
Olfactory
ParaH-ippoc
Hippocampus
ParacentralLod
SMA
Prec Gy
Thalamus
ParictalSup Sup
Postcentral
FrontInfTri Tri
Heschl-
Roland pprc
TempSup
TempPole
Supra Marginal
Templid
FrontInfOper perc
Occipital
Fusiform
Templnf
Cuneus
Lingual
Calcarine
FrontMedO
FrontMid
FrontMidOri
FrontinfOrb
FrontSupMes
FrontSupOrb
Precuneus
FontSup
Insula
Parietalinf
CingAnt
CingMVdd
Post
Amygdala
Angular
Cereb VI-
Cereb II-
Putamen-
Caudate
Pallidum
Vermis
Cereb ViI
ROIs
0.2
Figure 10: Starting from the top, the figure shows (i) the ground truth linear CA, and the learned linear CA for the simulated partial prior setting with (ii) low, (iii) medium, and (iv) high uncertainty in Sec. 7.
36

---

Causal Abstraction Learning based on the Semantic Embedding Principle
KL divergence
Frobenius absolute distance
True positive rate
False discovery rate
10:72
0.4-
0.11-
10-1
0.94
0.2-
0.92
0.02
111
0.00
modmm sca
0.00
mediums
fow
medium
low
medium
high
medium Uncertainty
high
Uncortainty
Uncertaintyy
Uncertainty
Figure 11: Starting from the left, the figure provides the (i) the KI divergence evaluated at the learned V, (ii the Frobenious absolute distance, (iii) the true positive rate, and (iv) the false discovery rate for the simulated partial prior setting with low, medium, and high uncertainty in Sec. 7.
37


Reviews:
[{"review_id": "ne6C0tjNAp", "reviewer": "Reviewer_232e", "summary": "This paper formalises the problem of causal abstraction in category theory language, then it introduces the Semantic Embedding Principle (SEP). Intuitively, SEP states that if we go from a high level model to a low level and then abstracting back, one should get the initial high level model back, the way I understand it is that high level models should contain all information of the low level model that produced it. The authors make some general assumptions in causal abstraction into non-assumptions and using SEP they and certain assumptions they establish the problem as a Riemannian Optimisation problem which then they go on to test on synthetic data and then real data from brain networks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their effort, valuable comments, and appreciation of our work. We address below the Reviewer’s concerns. We are happy to further discuss to dispel any additional concerns.\n\n__Claims And Evidence__\n\n__[Our claims]__ Our claims are highlighted in the “Contributions” paragraph of Sec.1. We support them both theoretically and empirically. We are open to reducing any eventual excessive claims as per Reviewer suggestion. At the moment, it is not clear to us which claim is problematic. We kindly ask the Reviewer to specify not supported or justified claims.\n\n__[Assumptions]__ We believe the Reviewer’s concern is centered on the assumptions behind our work. We clarify this point below.\n \nRegarding the SOTA, we do not believe – and do not state in the manuscript – that the assumptions of existing works are _unacceptable_. We say they are _restrictive_ to tackle CA learning in real-world applications (cf. line 60 1st col, lines 398-399 2nd col). \nAccordingly, we make only one assumption, viz. (A1), supported by empirical evidence (cf. Sec.7). \n\nAlso leveraging (A1), we pose the general SEP-based CA learning problem (cf. Prob1). This problem does not assume any specific (i) functional form for CA, (ii) probability measures for the involved SCMs, and (iii) distance function for quantifying the misalignment (cf. lines 167 1st col - 170 2nd col). Prob.1 should be read as a learning paradigm for CA rather than a single learning problem.\n\nAs an application, we decline Prob.1 to the case of (i) linear CA, (ii) Gaussian measures for the endogenous, and (iii) KL divergence, arriving at Prob.s 2 and 3. We remark that (i)-(iii) are not assumptions within our work, rather a particular case of Prob 1. \n\nConcerning the Reviewer’s statement, it’s unclear to us what they mean by “Gaussianity of the errors”. If error stands for exogenous variables, then we remark that we do not make any such assumption in our work. If they mean Gaussianity of the endogenous probability measures, then we provide below a simple counterexample showing that (nonlinear) functional forms other than linear for causal modules are compatible with the (i)-(iii). We will consider lognormal distributed variables as those are relevant in application domains such as quantitative finance: stock prices are considered to be log-normally distributed (Black and Scholes, 1973; Fama, 1965).\n\n__Counterexample.__ Denote by $U_i$ and $L_i$ the exogenous and endogenous variables of the low-level SCM. Denote by $V_j$ and $H_j$ the exogenous and endogenous of the  high-level SCM. Let $T:[0,1] \\rightarrow [0,1]$ be a measure preserving map (e.g., $T(x)=1- |2x-1|$), $f$ and $g$ be the quantile function and CDF of two Gaussians,  $N(0, c_1^2+c_2^2+c_3^2)$ and $N(0, c_4^2)$, respectively. \n\n_Causal Abstraction complying with SEP_: $H_1=1/\\sqrt{2} (L_1 + L_2)$, $H_2=L_3$.\n\n* Low-level SCM: \n  * Exogenous: $U_1, U_2, U_3$; each following $\\log{N(0,1)}$\n  * Endogenous: $L_1=\\log{U_1}; L_2=\\log{U_2}; L_3= (f \\circ T \\circ g) (c_1 L_1 + c_2 L_2 + c_3 \\log{U_3})$\n  * Observational distributions for the endogenous: $L_1 \\sim N(0,1), L_2 \\sim N(0,1), L_3 \\sim N(0,c_4^2)$.\n \n* High-level observational distributions entailed by CA: $H_1 \\sim N(0,1)$, $H_2 \\sim N(0, c_4^2)$\n* High-level SCM:\n  * Exogenous: $V_1, V_2$; each following $\\log{N(0,1)}$ \n  * Endogenous: $H_1 = \\log{V_1}, H_2 = (f \\circ T \\circ g)(d_1 H_1 + d_2 \\log{V_2})$; with $d_1=\\sqrt{ c_1^2 + c_2^2 }$ and $d_2=c_3$.\n  * Observational distributions for the endogenous: $H_1 \\sim N(0,1)$, $H_2 \\sim N(0, c_4^2)$\n\nFinally, the KL divergence is used as an objective function in Prob. 2 and 3, and not limited to any specific probability measure and functional form for the SCM. \n\n__[Relations to existing methods]__ We deliberately selected KL divergence for its relevance in ML applications. Furthermore we consider the application of KL to probability measures of different dimensionality, differently from previous work (Kekic et al., 2023; Dyer et al., 2024). Additionally, Zennaro et al. (2023) use a regularized Jensen-Shannon divergence, Felekis et al. (2024) an $\\omega$-informed cost of transport with entropic and do-intervention regularization terms; Massida et al. (2024) perform OLS estimation between the low- and high-level SCMs data. None of the works above is similar to the Riemannian optimization problems, viz. Prob.s 2 and 3, posed in our linear CA application.\n\n_Additional Ref.s To Those in The Paper_\n\n[1] Black, F., & Scholes, M. (1973). The pricing of options and corporate liabilities. Journal of Political Economy, 81(3), 637-654.\n\n[2] Fama, E. F. (1965). The behavior of stock-market prices. The Journal of Business, 38(1), 34-105.\n\n__Weaknesses__\n\nSee “Claims” above.\n\n__Comments or Suggestions__\n\nDef. 4.1 and Prob. 1 are general and suitable for any CAs. SEP requires the CA having a right-inverse (cf. lines 175-176 1st col), this is the condition to be enforced during the optimization process.", "claims_and_evidence": "From what I understood of the paper (and I have to admit I don’t think I fully understood it), I would say the claims are only partly supported. Let me elaborate. The authors claim that most of the previous work makes assumptions that are unacceptable in the causal abstraction setting and thus are not so applicable whereas their assumptions are based on information of the structure of CA. For example, the assumptions they use for the experiments are linear causal abstractions, constructability of the abstraction and Gaussianity of the errors. I don’t know how the authors feel about this, but I would say that these assumptions are as strong as assuming the functional form of the SCM.\n\nIn the end the optimisation ends up being similar to other research, namely the KL divergence between the pushforward of the low level distribution against the high level model.", "methods_and_evaluation_criteria": "Yes, their methods seem reasonable to me. Although as mentioned above, they seem very similar to what we already see in the abstraction literature with the backing of the Categorical formalisation and the exception of the need for interventional data, which they don’t assume have access to.", "theoretical_claims": "Not in detail.", "experimental_designs_or_analyses": "I checked what is on the main document. They seem reasonable. Furthermore, I appreciate the application to brain networks as a real world scenario.", "relation_to_broader_scientific_literature": "The contribution seems relevant to the literature. It gives the categorical perspective from Rischel (2020) applicability.", "essential_references_not_discussed": "The authors do a good job at referencing related literature. There is nothing very obvious that I believe they missed.", "other_strengths_and_weaknesses": "Strengths:\n- I think the use of the categorical language to go beyond interventional consistency is interesting and has the potential to allow the application of causal abstraction in other areas like they show with the brain networks.\n- The optimisation procedure and its description looks very good as well although admittedly the details are beyond my optimisation knowledge in relation also to the time I can invest into reviewing the paper.\n\nWeaknesses:\n- Unless I’m misunderstanding something in the paper the excessive claims, as discussed above seem to be the greatest weakness.", "other_comments_or_suggestions": "Please see the weaknesses for some potential discussion. Additionally, I would like to know what do the authors think is needed to change the assumption of having a linear abstraction to something more complex. That is, how easy is it to find a space that satisfies SEP that we can also optimise on?", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LVleIFgmtc", "reviewer": "Reviewer_Mguo", "summary": "This paper addresses the challenge of learning causal abstractions (CAs) between structural causal models (SCMs) at different resolutions, a critical task for bridging causal evidence across scales (e.g., molecular vs. organism-level processes). The authors propose the Semantic Embedding Principle (SEP), which enforces that embedding a high-level SCM into a low-level one and abstracting back should reconstruct the original high-level model. They formalize SEP using a category-theoretic framework, decoupling structural and functional components of CAs. For learning, they focus on linear CAs under partial prior knowledge (assumption A1), framing the problem as Riemannian optimization over the Stiefel manifold. They develop optimization methods (LinSEPAL-ADMM, LinSEPAL-PG, CLinSEPAL) tailored to smooth/nonsmooth objectives in Gaussian settings, validated on synthetic and neuroimaging data.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their effort and valuable  comments. We address below the Reviewer’s major concerns on the relationship between CA learning and CRL, and we are happy to further discuss to clarify any additional points.\n\n__Claims And Evidence__\nCausal abstraction is intuitively introduced at the beginning of Sec.1, from line 46 1st col to line 49 2nd col; and formally in Def. 2.3. However, we are happy to improve our manuscript as per Reviewer suggestion by adding the proposed text in italic reported below, within our reply to “Weaknesses”.\n\n__Theorethical Claims__\nWe do not make any claim about the _identifiability_ of the causal abstraction for the reasons reported below in [W3.2].\n\n__Experimental Designs Or Analyses__\nPlease refer to [W1-W2] below. CA and CRL tackle different learning tasks, although a comparison between them is valuable to further improve the paper. \n\n__Relation To Broader Scientific Literature__\nPlease refer to [W1-W2] below. We believe that our work is correctly placed within the literature, although considering the comparison with CRL is valuable and planning to add it in the revised version of the manuscript.\n\n__Essential References Not Discussed__\nPlease refer to [W1-W2] below.\n\n__Weaknesses__\n\n__[W1-W2]__ We agree with the Reviewer that an additional comparison between CA learning and CRL within Sec.1 would improve the paper, specifically aiding the reader in setting apart CA learning from CRL. We propose to add the following discussion in the revised version:\n\n_Causal abstraction (CA) learning aims at learning a mapping between two different SCMs, for instance, the architecture of a neural network and a human-interpretable causal model [2], or an SCM of brain regions of interest (ROIs) and one of brain functional activity (see Sec.7)._ \n\n_Within CA literature, it is usual to distinguish between low- and high-level variables (or SCMs). Although the same adjectives are usually employed also in causal representation learning (CRL, [3]) literature, they convey different meanings in the two research fields._      \n\n_In CA, both low- and high-level refer to endogenous variables being causal and observed. Specifically, the latter are said to be causal since they relate to each other within the SCM, and are the relevant variables for interventions and counterfactual reasoning.\nInstead, in CRL, the low-level variables are observed but not causal, that is, mere mathematical functions of high-level causal but unobserved variables, where causal has the same meaning as above. As an example, the low-level variables could be the pixels of an image, whereas the high-level ones are concepts related by an SCM [4, 5]. Additionally, the high-level variables could also not be labeled [3]._\n\n_Consequently, also the goal of CRL is deeply different from that of CA: Given the low-level variables, CRL algorithms aim at learning (i) the high-level variables and (ii) the causal structure underlying these variables. In brief, while CRL extracts a meaningful causal representation from non-causal data to improve model performance and interpretability [4, 5]; CA learns mappings between already meaningful representations to enable causal knowledge transfer and communication between SCMs working at different levels of abstractions._\n\n__[W3.1]__ Identifiability of CA is not key as identifiability of causal structures in CRL and causal discovery. In CA learning, we mainly care about (structural and distributional) interventional consistency. It is well known that there might exist multiple causal abstractions between low- and high-level SCMs (see Example 5.2 in [1] where symmetry allows for multiple interventionally consistent causal abstractions).  \n\n__[W3.2]__ As we drop (NA1), (NA2), and (NA4), our work shows that it is possible to learn causal abstractions without assuming any knowledge of the underlying low- and high-level SCMs. \n\n_References_\n\n[1] Zennaro, F. M., Bishop, N., Dyer, J., Felekis, Y., Calinescu, A., Wooldridge, M., & Damoulas, T. (2024, September). Causally Abstracted Multi-armed Bandits. In Uncertainty in Artificial Intelligence (pp. 4109-4139). PMLR.\n\n[2] Geiger, A., Lu, H., Icard, T., & Potts, C. (2021). Causal abstractions of neural networks. Advances in Neural Information Processing Systems, 34, 9574-9586.\n\n[3] Schölkopf, B., Locatello, F., Bauer, S., Ke, N. R., Kalchbrenner, N., Goyal, A., & Bengio, Y. (2021). Toward causal representation learning. Proceedings of the IEEE, 109(5), 612-634.\n\n[4] Yang, M., Liu, F., Chen, Z., Shen, X., Hao, J., & Wang, J. (2021). Causalvae: Disentangled representation learning via neural structural causal models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 9593-9602).\n\n[5] Komanduri, A., Wu, Y., Huang, W., Chen, F., & Wu, X. (2022, December). Scm-vae: Learning identifiable causal representations via structural knowledge. In 2022 IEEE International Conference on Big Data (Big Data) (pp. 1014-1023). IEEE.", "claims_and_evidence": "The claims are generally supported by providing some definition. However, the paper still lacks clarity when introducing those new terminologies, e.g., the introduction of causal abstraction.", "methods_and_evaluation_criteria": "Yes, the (LinSEPAL-ADMM, LinSEPAL-PG, CLinSEPAL) are well-suited for the problem.", "theoretical_claims": "The theoretical claims seems sound. However, the paper does not provide detailed proofs for the identifiability of causal abstractions.", "experimental_designs_or_analyses": "The experimental design is generally sound but the experiments lacks discussion with the related causal representation methods.", "relation_to_broader_scientific_literature": "The key contribution is somewhat limited in the broader literature on causal reasoning and representation learning. While the introduction of SEP and its formalization using category theory builds on prior work in causal abstraction (Rubenstein et al., 2017; Beckers & Halpern, 2019) and extends it to a learning framework, the paper could better situate itself relative to causal representation learning methods, which address similar challenges but from a different perspective.", "essential_references_not_discussed": "Yes, the paper would benefit from a more thorough discussion of causal representation learning methods, such as CausalVAE [1] and SCM-VAE [2], which also aim to learn causal structures from data but focus on disentangled representations rather than multi-resolution abstractions.", "other_strengths_and_weaknesses": "Strengths\n- A causal abstraction learning method is proposed with the integration of Riemannian optimization.\n\n- Experiments on synthetic and real-world brain data showcase the feasibility of the approach under varying levels of prior knowledge.\n\nWeaknesses:\n\n- As for clarity, I think in the start of paper, it should first clarify what is causal abstraction and how is it related and difference to the similar concept of causal representation learning.\n- Moreover, this paper fails to properly motivate why we need the causal abstraction and how it can be used in terms of application and how other methods (e.g., the causal representation learning methods [1-2]) fails on those tasks. \n- As for the causal abstraction, can you show the identifiability of the causal abstraction? Moreover, is the the structure of the SCM need to be priorly specified for causal abstraction?\n\n[1] Yang, Mengyue, et al. \"Causalvae: Disentangled representation learning via neural structural causal models.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n[2] Komanduri, Aneesh, et al. \"Scm-vae: Learning identifiable causal representations via structural knowledge.\" 2022 IEEE International Conference on Big Data (Big Data). IEEE, 2022.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the weankesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q4HHIAuq3n", "reviewer": "Reviewer_rKnp", "summary": "This paper introduces a framework for learning causal abstractions (CA) when structural causal models (SCMs) are unknown, interventional data is unavailable, and observational data is misaligned. The authors proposed the Semantic Embedding Principle (SEP), which helps to reconstruct the relationship between the low-level and high-level causal variables. In particular, this paper focuses on linear CA problem. The linear CA problem can be formulated as a Riemannian optimization problem on the Stiefel. Various optimization algorithms are proposed to solve this problem.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the Reviewer for their effort, valuable comments, and appreciation of our work. We address below the Reviewer’s concerns in a concise manner due to text limit. We are happy to further discuss any additional concerns.\n\n__Weaknesses__\n\n__[W1]__ \nThe Reviewer is right, in real-world applications, systems often display nonlinear interactions. However, the weakness they point out does not apply to our work since _we do not assume any linearity of the low- and high-level SCMs_. \n\nIndeed, from both the theoretical and learning perspectives, (i) the category-theoretic treatment of SCM, (ii) SEP in Def. 4.1, and (iii) the SEP-based CA learning problem in Prob.1 are general and do not make any assumptions on the functional forms of the involved SCMs. \n\nIt’s only from the application perspective that we decline Prob.1 to the case of linear CA. However, the linearity of the CA does not imply the linearity of the causal modules of the SCMs (cf. “Counterexample” in reply to Reviewer 232e). \n\n__[W2]__ We agree with the Reviewer on the importance of the comparison with baselines. Unfortunately, we were not able to find any baseline suitable for a fair comparison as we dropped many assumptions of existing methods, viz. (NA1)-(NA5), as the Reviewer acknowledges. Requirements: \n* Zennaro et al. (2023): (NA1), (NA2)\n* Felekis et al. (2024), $(\\tau,\\omega)$ CA setting: (NA2),(NA3)\n* Massida et al. (2024): (NA4),(NA5)\n* Kekic et al. (2024) perform _targeted reduction of an SCM_ in the $(\\tau,\\omega)$ abstraction setting, a different task than ours. Require (NA3) (NA4) \n* Dyer et al. (2024) consider a different problem within the setting of $(\\tau,\\omega)$-abstraction. Require (NA3)\n* Geiger et al. (2021) and (2024) leverage CA formalism to rigorously analyze explainability of neural networks, a different task from ours. \\\nGiven a neural network as a low-level SCM, the assumption is that there exists a high-level interpretable SCM. The goal is to evaluate the alignment between the two. Here, “low-level” means “black-box”; “high-level” refers to “human-understandable” SCMs built from theoretical and empirical modeling work. Differently, in our work “low-” and “high-level” mean “micro”(fine-grained) and “macro”(coarse-grained). It’s not a mere difference of terminology. Consider the SCMs $\\mathcal{B}$ and $\\mathcal{N}$ in Sec.3 of Geiger et al. (2024): $\\mathcal{B}$ is a high-level model for $\\mathcal{N}$ although both have the same structure. In our work, since there is no difference in interpretability between SCMs, the previous setting would lead to a contradiction, since it would mean considering as a (macro) abstraction of an SCM a rotation of it.  \\\nFurther, Geiger et al. (2021) and (2024) use _interchange intervention training_ (IIT) objectives, developed for neural networks and requiring the possibility to perform interventions over both the black-box and human-interpretable models. Citing from Geiger et al. (2024): _“interchange intervention (also known as activation patching), in which a neural network is provided a ‘base’ input, and sets of neurons are forced to take on the values they would have if different ‘source’ inputs were processed”_. Due to (NA1)-(NA3), it is not possible to adapt IIT to our setting.\n\n__Comments__\n\n__[S1]__ We devoted App.B to “Category theory essentials”. If the paper will be accepted, we will add a concise background to the main using the extra page.\n\n__[S2]__ Thanks, there is a typo in the text. The corrected version is: \n_“... the columns of the support $(B \\odot S)^\\top$ must sum up to one, [...] the rows of the support  $(B \\odot S)^\\top$ must contain at least a one.”_\n\n__[S3]__ Thanks, we will fix the link. A non-assumption is an assumption made by existing methods that we do not make. We will specify it better in the paper.\n\n__Questions__\n\n__[Q1]__ We work purely at the semantic (distributional) level dropping (NA1)-(NA5). Category theory (CT) is applied to isolate the distributional layer of the SCMs. This has an impact on SEP in Def. 4.1 since CT requires the involved mappings to be measurable. Finally, CT is used to generalize the existing CA $\\alpha$-framework – which is posed in category-theoretic terms – into our setting. \n\n__[Q2]__ Please refer to [W2]. We can add the reported discussion in our manuscript to motivate the absence of a comparison with baselines. Additionally, among future works we will add the investigation of SEP in the setting of Geiger (2021, 2024). It is an intriguing research question that could lead to jointly aligning and compressing human-understandable models to AI ones in a principled manner. \n\n__[Q3]__ As stated in lines 194-195, _“Only if we identify the true constructive abstraction, we are guaranteed interventional consistency.“_. We plan to investigate in which cases interventional consistency can be guaranteed without additional assumptions (cf. Sec. 8).\n\n_References_: Already in the paper. Geiger (2021) and (2024) are [1,2] of the review.", "claims_and_evidence": "Claims are supported by evidence.", "methods_and_evaluation_criteria": "The proposed method and evaluation criteria make sense.", "theoretical_claims": "I did not check the soundness of all the proof.", "experimental_designs_or_analyses": "I checked the design of the experiments.", "relation_to_broader_scientific_literature": "This paper studies causal abstraction without interventional data and prior knowledge about the SCM. This is an important problem because many current causal abstract algorithm requires some kind of conjecture about the underlying SCM [1,2]. Without prior knowledge, these methods fail.", "essential_references_not_discussed": "I think the author should at least discuss the following two papers [1,2]. These papers study one application of causal abstraction in the literature. \n\n\n[1]Geiger, Atticus, et al. \"Causal abstractions of neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 9574-9586.\n[2]Geiger, Atticus, et al. \"Finding alignments between interpretable causal variables and distributed neural representations.\" Causal Learning and Reasoning. PMLR, 2024.", "other_strengths_and_weaknesses": "Strength: The paper introduces the Semantic Embedding Principle (SEP), a novel and well-motivated approach to ensuring that high-level causal knowledge is faithfully embedded in low-level models. Unlike prior works that rely on full knowledge of SCMs, known DAG structures, or interventional data, this method operates when only partial prior knowledge is available. The methods are empirically validated on synthetic data as well as resting-state fMRI data, showcasing their effectiveness in neuroscience applications.\n\nWeakness: This paper focuses on the linear CA problems. Since many real-world causal systems are inherently nonlinear, this limits the immediate applicability of the approach. In the experiment part, there is no direct comparison with base line methods.", "other_comments_or_suggestions": "1. It would be helpful if the author could provide some background knowledge on category theory, which can help the readers understand this paper better. Most readers may not be familiar with category theory. \n\n\n2. Line 257: the paper states that rows of the support $(B\\odot S)^T$ must sum up to one, but later in (iii) \"the columns of the support $B\\odot S$\" must contain at least a one. Wouldn't the columns of $B\\odot S$ be the rows of $(B\\odot S)^T$ ? It seems to me (ii) already implies (iii). \n\n\n3. The hyperlink (NA1)-(NA5) and (A1) do not work. I am kind of confused what is non-assumption.", "questions_for_authors": "I appreciate it if the author could clarify the following problems. \n\n1.  It seems to me that I do not need category theory to understand the SEP. Could the author explain how the category theory formulation lead to this principle? Or what is beneficial using the category theory language? \n\n2. In the experiment part, the authors compare the performance of three proposed methods. Can the authors compare their methods with existing methods [1,2]? I understand the setting may be a bit different. It is also beneficial to see without prior knowledge how good can the proposed method perform. \n\n3. If I understand this paper correctly, this paper is doing CA based on observational data. In practice, what we care about most is what would happen if we do interventions, so one key question here is after learning the abstraction map using the proposed method, can the abstraction map stay consistent under interventions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "43MRdAj8zi", "reviewer": "Reviewer_p62B", "summary": "The authors use a category theoretical formalization SCM and causal abstraction to derive and optimize similarity measures over the measurable spaces of observed distribution of the corresponding low- and high-level representations of data.\n\nTowards this existing notions of $\\alpha$-abstractions and constructive causal abstractions are leveraged to formulate a semantic embedding principle (SEP) that implies a right-inversive (=existence of a consistent high- to low-level map) abstraction which allows for the formalization of a distance measure between low- and high-level data. They formulate a \"SEP-based CA Learning\"-Problem and employ several optimizations to minimize distances between the probability measures of the high- and low-level distributions. To obtain unique solutions from observational data, prior knowledge about the functional form of the abstraction is induced to enforce the constructiveness of the abstraction. The authors reformulate the learning problem under the prior knowledge constraint as an Riemannian optimization problem within a Stiefel manifold.\n\nThe authors propose and derive several variants of a \"Linear Semantic Embedding Principle Abstraction Learner\" (LinSEPAL-ADMM, LinSEPAL-PG, CLinSEPAL) over smooth and non-smooth setups utilizing different optimization methods. Ablation experiments on synthetic data under the presence of full and partial prior knowledge are presented, showing robust performance of the CLinSEPAL method. The best performing LinSEPAL method is furthermore applied the real-world fMRI brain data, where the authors simulate the coarsening of brain regions of interest (ROI) between a raw data model and combination several ROI into macro ROI. In a second experiment, uncertainty about the grouping of some ROI is simulated such that the method has to decide the assignment of ROI to several macro ROI, resulting in moderate errors.\n\n**Update after rebuttal.** The rebuttal and following answers were able to fully address my concerns regarding the role of $\\mathbf{B}$, partial prior knowledge. The subsequent explanation and presented proof on bounded eigenvalues and constructive CAs were able to further strengthen the contributions of the paper. Considering this and the other reviewers discussion I have raised my score to an accept.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you to the Reviewer for their effort, valuable comments, and appreciation of our work. We address below all the Reviewer’s concerns, also providing an additional theoretical contribution in [Q2]. We are happy to further discuss any additional concerns.\n\n_Claims_\n\nSee [W1-W2].\n\n_Experimental Designs_\n\n[Uncertainty] See [W2].\n\n[Brain application] See [W3].\n\n[Fig.4] Agreed, we will report the results for our proposed methods that do not guarantee constructiveness in the Appendix.  \n\n_Weaknesses_\n\n[W1] We do not assume the availability of aligned (i.e., jointly sampled) data from the SCMs. Hence, we cannot pose any regression problem.\n\nAssuming jointly sampled data are available (but again, that’s a different work  from ours), denoting by $h$ the number of nodes in the high-level model and considering full-prior knowledge, then one could solve separately $h$ linear regression problems subject to unitary $\\ell_2$-norm constraints for the vectors of coefficients for complying with SEP. Thus, the individual problems are not mere regressions, rather nonconvex problems due to the constraints $\\beta_i^\\top \\beta_i=1$, $i \\in [h]$ and $\\beta_i$ being the $i$-th _vector_ of coefficients.   \n\nThanks, we will add these details as a _Remark_.\n\n[W2] We believe there is a misunderstanding, probably due to lines 369-370 where we say _“by forgetting the mapping for 25%, 50%, and 75% of the variables”_. This passage in the text may be ambiguous to the reader and we’ll modify it. We clarify below this point that raised a major concern.\n\nIn lines 433-435 we say: _“We then express this partial information via uncertainty over B, meaning that some rows of B have more than one entry equal to one”_. This is exactly what we do in the partial prior setting of the synthetic experiments in Sec.6.\n\nIndeed, by _“forgetting the mapping for 25%, 50%, and 75% of the variables”_ we mean that  25%, 50%, and 75% of the rows in $B$ have all entries equal to one. Thus, for each case, a specific fraction of nodes in the low-level considers all nodes in the high-level as plausible abstractions and our algorithms have to identify the correct high-level node for each low-level one.\n\nHence, Fig.4 already shows the results for partial prior knowledge on synthetic data.\n\n[W3] Agreed, we will add the $B$ used in the experiments to App. L.\n\n_Comments_\n\n[C1] Thank you, we will correct the typo in Fig.5.\n\n[C2] Thanks, we will introduce $V^\\star$ before.\n\n_Questions_\n\n[Q1] In Eq.(3) we look at KL as a function of $V$. The constant term is exactly $-\\log{ \\det{\\Sigma^h}} - h$. We will specify this in line 202 2nd col.\n\n[Q2] Below we translate Remark 1 into a rigorous additional result we will add to the paper. It establishes a spectral characterization for linear CA and Gaussian measures, valid for any information-theoretic metric and $\\phi$-divergence. \n\n_Theorem._ Let $\\chi^\\ell \\sim N(0_\\ell, \\Sigma^\\ell)$, $\\chi^h \\sim N(0_h, \\Sigma^h)$, where $\\Sigma^\\ell$ and $\\Sigma^h $ are positive definite and $\\ell>h$. Denote by $0<\\lambda\\_1\\leq…\\leq \\lambda\\_\\ell$ the eigenvalues of $\\Sigma^\\ell$, and by $0<k\\_1\\leq…\\leq k\\_h$ those of $\\Sigma^h$. If a linear CA $V \\in \\text{St}(\\ell,h)$ complying with SEP from $\\chi^\\ell$ to $\\chi^h$ exists, then \n$$\n\\lambda\\_i \\leq k\\_i \\leq \\lambda\\_{i+\\ell-h}, \\forall  i \\in [h]. \\tag{1}\n$$\n_Proof._ If a linear CA $\\mathbf{V} \\in \\mathrm{St}(\\ell,h)$ exists, then $\\chi^h=\\varphi\\_\\text{push}^V(\\chi^\\ell)$. Thus the kernel of any information-theoretic metric or $\\phi$-divergence is nonempty, and $V^\\top \\Sigma V=\\Sigma^h$ implying the eigenvalues of $V^\\top \\Sigma V$ are those of $\\Sigma^h$. By the Ostrowski’s theorem for rectangular $V$ (cf. Th. 3.2 in Higham & Cheng, 1998) we have \n$$\nk_i = \\theta_i \\mu_i, \\quad i \\in [h]; \\tag{2}\n$$ \nwhere \n$$\n\\lambda\\_i \\leq \\mu\\_i \\leq \\lambda\\_{i+\\ell-h}, \\tag{3}\n$$ \nand \n$$\n\\text{eigvls}(V^\\top V)\\_1 \\leq \\theta_i \\leq \\text{eigvls}(V^\\top V)\\_h . \\tag{4}\n$$\nSince $V \\in \\text{St}(\\ell,h)$, by (4) $\\theta_i=1$ for each $i \\in [h]$. Substituting the latter into (2), we get $k_i=\\mu_i$, thus obtaining (1) by (3).\n\n_Ref.:_ Higham, N. J., & Cheng, S. H. (1998). Modifying the inertia of matrices arising in optimization. Linear Algebra and its Applications, 275, 261-279.\n\n[Q3] The matrix $S$ guarantees a constructive CA. Consider the partial prior knowledge case where $B$ has more than a single 1 per row. By disentangling the support $S$ and the coefficients $V$ we can learn and enforce the support of CA to be constructive through the second and third constraints in Eq. (5). \n\nIn the nonsmooth case we propose a simpler unconstrained formulation at the price of losing constructiveness guarantees for CA. In fact, we simply penalize entries in $V$ corresponding to zeros in $B$. Clearly, this does not guarantee the constructiveness of $V$, especially in the case of partial prior knowledge. \n\n[Minor] Thanks, on the LHS should be $\\chi^h$.", "claims_and_evidence": "The authors argue that previous assumptions about availability of interventional data, structural knowledge of the SCM or functional form seem to be unrealistic or are infeasible to obtain. To overcome this problem, the authors work under the assumption of partial prior knowledge on the structure of factor assignments between the causal abstractions being available. To my understanding, the assumed prior knowledge is embedded in the form of a assignment matrix which indicates the relation between low- and high-level features, such that the CA learning problem transforms into a parameter regression problem under the presence of full prior knowledge. Even though the authors present a convincing case where full prior knowledge might available, learning the exact parameter assignment is arguably the core problematic of CA learning. In that regard, the authors present a final experiment on brain data where uncertainty about factor assignments is induced to the prior information. However, the property to recover correct factor assignments is only analyzed marginally. (See 'Methods And Evaluation Criteria' below).\n\nUnder the assumption of given prior information however, the presented algorithms seem yield convincing results, with LinSEPAL-ADMM and LinSEPAL-PG declining under partial knowledge (with missing factor assignments) and CLinSEPAL even obtaining robust performance in the latter case.", "methods_and_evaluation_criteria": "The authors report results for distribution distance in terms of KL-divergence, F1 score and Frobenious absolute distance of the learned map and analyze the correctness of the learned factor assignments ('learned morphisms'). The reported metrics are suited to asses the performance of the respective algorithms.", "theoretical_claims": "The authors present semantic embedding principle (SEP) that implies the existence of a right-inversive causal abstraction (CA) between the high- and low-level data measures. Furthermore, the problem of SEP-based CA learning is formalized, which states as a goal, the learning of a CA which minimizes the distance between data representations and complies with the SEP. Both, the definition of SEP and the SEP-based CA learning, follow naturally from the category theoretic and measure theoretic formalizations. The problem formulation within a Stiefel manifold and the consequent formalization as a Riemannian optimization problem is laid out clearly and seems to be correct. While I am not an expert in the proposed optimization algorithm I followed the derivations of the CLinSEPAL (Sec. 5.2) method in Appendix J, which, to the best of my knowledge, seems to be without immediate errors.", "experimental_designs_or_analyses": "A first experiment is conducted over synthetically generated data. The data generation and chosen dimensions of the experimental setups are reasonable to demonstrate the correct workings of the algorithms. Algorithms are evaluated under full and partial evidence, meaning the omission of variable assignments in the prior knowledge. As discussed in 'Claims And Evidence', the algorithm should be tested to recover CAs in the light of uncertainty/multiple possible factor assignments / morphisms.\n\nThe experiment on coarsened brain regions of interest with full prior knowledge seems to follow domain specific knowledge and is sound and well conducted. With regard to the evaluation of partial knowledge the authors might indicate the specific prior knowledge matrix $\\mathbf{B}$ that is used for the respective low, medium and high setups. Particularly, to indicate the number of assignments the algorithm can choose from in each setting.\n\n\n\nFinally, I would like to recommend to still report results of all methods for all metrics in Fig. 4. Worse performing methods might still yield reasonable results --even if, for the wrong reasons-- and might give insights on whether those metrics can be used to make validate the validity of the prior knowledge.", "relation_to_broader_scientific_literature": "Being able to relating findings across different levels of causal abstractions is an important goal as it allows allows for the transformation of results between different approaches and yields models that allow to communicate low-level findings on a higher-level. Within the past few years, the automated learning of causal representation, therefore, has attracted increasing interest. The authors cite and discuss relevant approaches of the field which come --do to the inherent unidentifiability of factors from observational data-- with different particular assumptions. The proposed method(s) require prior knowledge on the function structure of abstraction, motivated on a measure theoretic and category theory theoretic formalization of SCM. To the best of my knowledge, the presented approach mark a novel contribution in terms of motivation and optimization.", "essential_references_not_discussed": "To the best of my knowledge, the authors cite and discuss relevant literature of causal abstraction learning and related category theoretical perspectives. While the authors utilize on a measure-theoretic view on acyclic SCM which is induced via recursive applications of push-forward measures, a more general view (that, e.g., allows for cyclic causal relations) might be taken with the use of transition probability kernels as formalized by Park et al. [1].\n\n\n\n[1] Park, J., Buchholz, S., Schölkopf, B., & Muandet, K. (2023). A measure-theoretic axiomatisation of causality. *Advances in Neural Information Processing Systems*, *36*, 28510-28540.", "other_strengths_and_weaknesses": "**Strengths**\n\nThe problem setup is well setup and derived. Intermediate steps over category theoretical are laid out clearly and follow naturally. Albeit the strong use of prior concepts and formalisms, all concepts are well described and the derivations in the paper are mostly self-contained.\n\nThe embedding of the problem into Stiefel Manifolds is an interesting insight and allows for the application of well-known Riemannian optimization methods.\n\nThe presented synthetic and real-world experiments confirm the robust application of the derived methods. Interestingly, the particular CLinSEPAL approach is demonstrated to perform well on real-world brain data, even under the presence of only partial knowledge.\n\n\n\n**Weaknesses**\n\nThe main weaknesses have mainly been discussed in the prior sections and mostly regard the identification of morphisms in the case of partial prior knowledge and specifically concern the following points:\n\n1) The presence of full prior knowledge might be an even stronger constraint than the knowledge about underlying DAG and possibly transforms the task into a mere parameter regression problem(?). The authors might elaborate (or compare) to the specific differences to a naive regression approach for the full prior information setting that simply regresses the non-zero entries of $\\mathbf{B}$.\n2) The arguably interesting case of identifying the right morphisms in the presence of only uncertain partial prior knowledge is not demonstrated over synthetic data. The paper might be improved by adding analysis on the effects of uncertain partial prior knowledge over synthetic data.\n3) In regard to the former point and evaluation is presented on real-world brain data. However, details on the extend of uncertainty are not specified. The authors might add the utilized  $\\mathbf{B}$ matrices or specify the number of non-zero entries per row per setting.", "other_comments_or_suggestions": "* The caption of figure 5 (\"from Ind (right) to Prob (left)\") does not match the labels in the figure.\n* $\\mathbf{V}^{\\star}$ is referred to in line 203 before its actual definition in eq. 4.", "questions_for_authors": "I would like to ask the authors to reply to the points listed in the weaknesses. Furthermore, I have the following questions on (possibly minor details of) the paper:\n\n1) Could the authors elaborate on the derivation of Eq. 3? Specifically, the log determinant term seems to only involve a single determinant, while KL divergence it is usually formulated as a quotient involving both of the arguments.\n2) Could the authors, elaborate on the condition of constructive CAs being bound by the range of eigenvalues? The mentioned relation was not immediately obvious and was not explained further in the paper.\n3) What is the reason for inducing $\\mathbf{S}$ for the smooth problem setup? How is it different from learning $V$ directly (and why is this distinction not necessary in the nonsmooth approach)?\n\n\n\nMinor:\n\n* Possible misunderstanding or typo [l165 (right column); 'zeroing distance function']: Wouldn't $\\chi_{\\#}^{\\alpha^*_{\\mathcal{X}}}$ map $\\chi^{\\mathcal{l}}$ onto $\\chi^{\\mathcal{h}}$. Why can $\\chi^{\\mathcal{l}}$ be on both sides of the equation?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper develops the semantic embedding principle to learn causal abstractions without interventional data. The semantic embedding principle is formalized through category theory, and formulate a tractable learning problem for the linear causal abstraction based on the Steifel manifold. Experiments are conducted on synthetic data and fMRI. The paper develops heavy machinery, which was checked by reviewers (see Reviewer p62B). The main concern was the role of the partial prior knowledge which was clarified in the rebuttal.
Decision: Accept (poster)