Learning Discrete Concepts in Latent Hierarchical Models
Authors: Lingjing Kong
Guangyi Chen
Biwei Huang
Eric P. Xing
Yuejie Chi
Kun Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: representation learning
causal representation learning
generative models
causal discovery
hierarchical models
Primary area: causal_inference
TL;DR: We develop identification theory for discrete latent hierarchical model from potentially high-dimensional, continuous data distributions.

Abstract: Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models.
    Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking.
    In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images).
    We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible.
    Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images.
    We substantiate our theoretical claims with synthetic data experiments.
    Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights.

Learning Discrete Concepts in Latent Hierarchical Models
Lingjing Kong', Guangyi Chen 12, BBwei Huang', Eric P. Xing Yuejie Chi', and Kun Zhang
CCarnegie Mellon University
Mohamed bin Zayed University of Artificial Intelligence University of California San Diego
Abstract
Learning concepts from natural high-dimensional data (e.g.. images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via hicrarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e-g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyelic graphs prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, discuss our theory's 's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights.
Introduction
Learning semantic discrete concepts from unstructured high- nigh-dimensional data, such images and text, is crucial building machine learning models with interpretability. transferability, and compositionality, empirically demonstrated by extensive existing work [[АI Despite these empirical successes, limited work is devoted the theoretical front: the notions of concepts and their relations are often heuristically defined. For example, concept bottleneck models [22 K use human-specified annotations and recent methods [14 employ pretrained multimodal models like
CLIP [17 to explain features with neural language. This lack of rigorous characterization impedes deeper understanding of this task and the development of principled learning algorithms.
In natural images, the degree/extent of certain attributes (e.g., position, lighting) is often presented in continuous form and main concepts of practical concern are often discrete in nature (e.g., object classes and shapes). Moreover, these concepts are often statistically dependent, with the dependence potentially resulting from some higher-level concepts. For example, the correlation between specific dog's 's eye features and fur features may arise from a high-level concept for breeds (Figure [ Similarly, even higher-level concepts may exist and induce dependence between high-level concepts, giving rise to hierarchical model that characterizes all discrete concepts at different abstraction levels underlying dinh-siimnsional data distributions. In this work, we focus on concepts that can be defined discrete latent variables and related via hierarchical model. Under this formalization, the query on the recoverability concepts and their relations from unstructured high-dimensional distribution (e.g., images) amounts to the following causal identification problem:
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Under what conditions is the discrete latent hierarchical causal model identifiable from high- dighndimensional continuous data distributions?
Identification theory for latent hierarchical causal models has been a topic of sustained interest. Recent work [8 20) investigates identification conditions of latent hierarchical structures under the assumption that the latent variables are continuous and influence each other through linear functions. The linearity assumption fails to handle the general nonlinear influences among discrete variables. Another line of work focuses on discrete latent models. Pearl 221. Choi et al. [22] study latent trees with discrete observed variables. The tree structure can be over-simplified to capture the complex interactions among concepts from distinct abstract levels (e.g., multiple high-level concepts can jointly influence a lower-level one). Gu and Dunson 23 assume that binary latent variables can be exactly grouped into levels and causal edges often appear between adjacent levels, which can also be restrictive. Morcover, these papers assume observed variables are discrete, falling short of modeling the continuous distribution like images the observed variables. Similar to goal, Kivva et al. [24] show the discrete latent variables adjacent to the potentially continuous observed variables can be identified. However, their theory assumes the absence of higher-level latent variables and thus cannot handle latent hierarchical structures.
In this work, we show identification guarantees for the discrete hierarchical model under mild conditions on the generating function and causal structures. Specifically, we first show that when continuous observed variables (i.c., the leaves of the hierarchy) preserve the information of their adjacent discrete latent variables (i.c., direct parents in the graph), we can extract the discrete information from the continuous observations and further identify each discrete variable up to permutation indeterminacy. Given these "low-level" discrete latent variables, we establish graphical conditions to identify the discrete hierarchical model that fully explains the statistical dependence among the identified "low-level" discrete latent variables. Our conditions permit multiple paths within latent variable pairs and flexible locations of latent variables, encompassing a large family of graph structures including as special cases non-hierarchical structures [24), trees 21
22 25 26
and multi-level directed acyclic graphs (DAGs) 23 27
(see example graphs in Figure 2 Taken
together, our work establishes theoretical results for identifying the discrete latent hierarchical model
governing nigh-dimensional continuous observed variables, which to the best of our knowledge is the
first effort in this direction. We corroborate our theoretical results with synthetic data experiments.
As implication of our theorems, discuss novel interpretation of the state-of-the-ar latent diffusion (LD) models (28) through the lens of a hierarchical concept model. We interpret the denoising objective at different noise levels as estimating latent concept embeddings corresponding hierarchical levels in the causal model, where higher noise level corresponds to high-level concepts. This perspective explains and unifies these seemingly orthogonal threads of empirical insights and gives rise to insights for potential empirical improvements. We deduce several insights from our theoretical results and verify them empirically. In summary, our main contributions
follows.
We formalize the framework of learning concepts from high-dimensional data as a latent-variable identification problem, capturing concepts at differrntt abstraction levels and their interactions.
We present identification theories for the discrete latent hierarchical model. To the best of knowledge, our result is the first to address discrete latent hierarchical model beyond trees [21 26
and multi-level DAGs 23 while capable of handling -ighesimeniioaal observed variables.
We provide interpretation of latent diffusion models as hierarchical concept learners. We supply empirical results to illustrate our interpretation and showcase its potential benefits in practice.
2 Related Work
Concept learning, In recent years, significant strand of research has focused on employing labeled data to learn concepts in generative models' latent space for image editing and manipulation [ Concurrently, another independent research trajectory has been exploring unsupervised concept discovery and its potential learn more compositional and transferable models 7I Concurrently, a plethora of work has been dedicated to extracting interpretable concepts from nigh-dimensional data such as images. Concept-bottlenee
[12] first predicts a set human-annotated concepts as an intermediate stage and then predicts the task labels from these intermediate concepts. This paradigm has attracted a large amount of follow-up work [13 29 33 A recent surge of pre-trained multimodal
models (c.g., CLIP [7] can explain the image concepts through text directly [4]16].

---

Latent variable identification. Complex real-world data distributions often possess hierarchical structure among their underlying latent variables. The identification conditions of latent hierarchical structures are investigated under the assumption that the latent variables are continuous and influence each other through linear functions 18|20) and nonlinear functions 34 addition, prior work [21 studies fully discrete cases and thus falls short of modeling the continuous observed variables like images. To identify latent variables under nonlinear transformations, a line of work [35]- 38 assumes the availability of auxiliary information (e.g. domain/class labels) and that the latent variables' probability density functions have sufficiently different derivatives over domains/classes. Another line of studies [39 44) refrains from the auxiliary information by assuming sparsity and mechanistic independence, disregarding causal structures among the latent variables.
26 22 23
Please refer to Section A1 for more extensive related work and discussion.
3 Discrete Hierarchical Models
Data-generating process. We formulate the data-generating process as the following latent-variable
model. Let x denote the continuous observed variables x == [x1 En E X C IRd which represents the high high-dimension data we work with in practice (c.g., images). Let d := ....... dral
be discrete latent variables that are direct parents to x (as shown in Figure (b) and take on values
Co. We denote the joint domain as from finite sets, i.e., d, E 22 for all i E [d.] and 2 < 12 (d)
2(4) : =2((0) x ...... x Sid These discrete variables are potentially related to each other causally (e.g., d4 and d3 in Figure Ic) or via higher-level latent variables (e.g., d1 and d2 in Figure I ()). Let c := c,,,..,, Cn.l E C Id be continuous latent variables that represent the continuous information conveyed in observed variables x. The generating process is defined in Equation | and
illustrated in Figure a).
x : g(d,c) c)2
where we denote the generating function with g: cd, - x. We denote the resultant bipartite graph T. In this context of image generation, the discrete subspace d gives description
from (d. c| of concepts present in the image x (e.g., dog's 's appearance, background objects), and the continuous subspace e controls extents/degrees of specific attributes (e.g., sizes, lighting, and angles).
Discrete hierarchical models. As discussed above, discrete vari- ables d1..... da, represent dis- tinct concepts that may be de- pendent either causally or purely statistically via higher-level con- cepts, as visualized in Figure [c). For instance, the dog's eye fea- tures and nose features are de- pendent, which higher-level concept "breeds" could explain. We denote such higher-level la- tent discrete variables as [- ,,,.). where 2, E ()
d
X2
X3
X5
X6
X7
Xs
(a) Subspaces
(b) The "bottom" level
High-level concepts: dog breeds, outdoor,
Low-level concepts: eyes, fur, grass,
d1
d2
da
d4
ds
X1
X2
X3
X11
X5
Xg
X7
X8
for all i [2.] and 2 < [2(+1
(c) The discrete hierarchical model
Figure 1: Latent hierarchical graphs. The dashed circle in (a) in- dicates that the continuous variable can be viewed as exogenous variable. Dashed edges in (b) indicate potential statistical dependence.
oo and (2(=) :- 22(=) x x 2
Graphically, these variables z are
not directly adjacent to observed variables x (Figure [c)). High-level discrete variables z may constitute a hierarchical structure until the dependence in the system is fully explained. Since the discrete variables encode major semantic concepts in the data, this work primarily concerns discrete variables d and its underlying causal structure. The continuous subspace c can be viewed as exogenous variables and is often omitted in the causal graph (c.g., ( We leave identifying continuous attributes in e as future work.
'e use the unbolded symbol x; to distinguish each observed variable x; from the collection x. Our theory allows 2; to be multi- dileniidieesion....
3

---

Given this, we define the discrete hierarchical model as follows. The discrete hierarchical model (Figure
I c)) G == (E, V) a DAG that comprises discrete latent variables d dp We denote that directed edge set with E and the collection of all variables with V == (D. Z]. where D and are vectors and z in a set form and all leaf variables in 9 belong to D. We assume the distribution over all variables V respects the Markov property with respect to the graph G. We denote all parents and children of a variable with Ch(-) and Pa(-) respectively and define the neighbors as ne(.) Ch(-) U Pa(.). We say variable set A are pure children of B, iff (A)(A) = UA,eA Pag (A.) = B and A B = 0. As shown in Figure [c), # is a pure child of dj.
2n.- ... 21....
Objectives. Formally, given only the observed distribution p(x), we aim to:
1. identify discrete variables d and the bipartite graph I:;
2. identify the hierarchical causal structure G.
4 Identification of Discrete Latent Hierarchical Models
We present theoretical results the identifiability of discrete latent variables d and the bipartite graph I in Section 4.2 (i.e., Objective T
and the hierarchical model G in Section 3.ctioo[4.3 (i.e., Objective
Additional notations. We denote the set containing components of x with X, the set of all variables with V == (V", E*). As the true generating process involves c, C, g. P, and G (defined in Section 33) we
V. the entire edge set with E* == E U r, and the entire causal model with
G = define their statistical estimates with d, é, å. and I through maximum likelihood estimation over the full population p(x) while respecting conditions on the true generating process. We use [Supp(L)) for the cardinality of a discrete variable set L's support (all joint states) and P A. B for the joint probability table whose two dimensions are the states of discrete variable sets A and B respectively.
4.1 General Conditions for Discrete Latent Models
It is well known that causal structures cannot be identified without proper assumptions. For instance, one may merge two adjacent discrete variables d1 E 01(4) and d2 E (2 into a single variable d e 22(0) x 22/59 while preserving the observed distribution p(x). We introduce the following basic conditions on the discrete latent model to eliminate such ill-posed situations. Condition 4.1 (General Latent Model Conditions).
i Noonddeenneuucy]: TP = kn 2 1z) > 0, for all k21, E 22d) x 22 for all variable o E V", P (v)(a(())= kn) * P (u)Pa(()) = k2) if kn & k2.
i [No-twins]: Distinct latent variables have distinct neighbors ne(v:) & ne( if U1 # 02 V.
ii [Maximality:: J: There is DAG ğ" = (V+ EE) resulting from splitting latent variable in g (i.e., turning 2 into 2G,1 and ε12 with identical neighbors and cardinality [[2]] | + |
J. such that IP
(++ is Markov w.rt. G and 9 satisfies ii
Discussion. Condition 4.1 necessary set of conditions for identifying latent discrete models, which is employed and discussed extensively [24] Intuitively, Condition (ii discrete states and graph edges that exert no influence on the observed variables x. Condition 4.1 ii m
excludes dummy
constrain the latent model to be the most informative graph without introducing redundant latent variables, thus forbidding arbitrary merging and splitting over latent variables.
4.2 Discrete Component Identification
We show with access to only the observed data x, we can identify each discrete component d: up to permutation indeterminacy (Definition 4.2) and a corresponding bipartite graph equivalent T.
Definition 4.2 (Component-wiis Identifiability). Variables E N and à N are identified component-win wise there exists a permutation म, such that d, (d(((()) with invertible function h;-
That is, our estimation de captures full information of dz(1) and information from d, such that j f (i). The permutation is fundamental indeterminacy for disentanglement (37
36 24
We use "components" to refer to individual discrete variables d; in the vector d.

---

Remarks on the problem. A large body of prior work [37 35 4 requires continuous or even differentiable density function over all latent variables and domain/class labels or counterfactual counterparts to generate variation. Thus, their techniques do not transfer naturally to our latent space with both continuous and discrete parts [c, d] and no supervision of any form. With a similar goal, Kivva et al. (24) assumes access to an oracle (Definition 12.1 to the mixture distribution over p(x), which is not directly available in the general case here. Kivva et al. [41] assumes a specific parametric generating process, whereas we focus
generic non-parametric generative model (Equation D
High-level description of our proposed approach. We decompose the problem into two tractable subproblems: 1) extracting the global discrete state d from the mixing with the continuous variable c; 2) further identifying each discrete component d, from the mixing with other discrete components dj (i * j) and the causal graph r. For 1), we show that, perhaps surprisingly minimal conditions on the generating function suffice to remove the information of c and thus identify the global state of For 2), we observe that the identification results in 1) can be viewed as a mixture oracle over p(x), which enables us to employ techniques from Kivva et [4 to solve the problem.
We introduce key conditions and formal theoretical statements as follows.
Condition 4.3 (Discrete Components Identification).
i [Connected Spaces] The continuous support C C TR= is closed and connected.
i UInveriibility & Continuity]: The generating function g in equation ii invertible, and for any fixed d, g(d.-) and its inverse are continuous.
ii (Non-Subset Observed Children]: For any pair d, and dj. one's 's observed children are not the subset of the other's, ss, (d,) & Chr(d,).
Discussion on the conditions. Condition 4.3 requires the continuous support be regular in contrast with the discrete variable's s support. Intuitively, the continuous variable often controls the extents/degrees of specific attributes (e.g., sizes, lighting, and angles) and takes values from connected spaces. For instance, "lightning" ranges from the lowest to the highest intensity continuously. Condition[4.3 fia.oommn4.-.jeenuuuuuee einuuress the generating process preserves latent variables' information 37 Thanks to the high dimensionality. images often have adequate capacity to meet this condition. For instance, the image of a dog contains a detailed description of the dog's 's breed, shape, color, lighting intensity, and angles, all of which are decodable from the image. Condition 4.3 iii ensures that each latent component should exhibit sufficiently distinguishable influences on the observed variable x. Practically, this condition indicates that the lowest-level concepts influence diverse parts of the image. These concepts are often atomic, such as a dog's ' ear, eyes, or even finer, which often don't 't overlap. This condition is adopted in prior work [24] 41 and related to the notation of sparsity. Along this line, prior work 46 assumes pure observed children for each discrete variable, which is strictly stronger. Recent work [47] assumes each latent variable is connected to a unique set of observed variables. This condition implies Condition 4.3 because if z0 's children form a subset of #1 's children, then one cannot find subset of observed variables whose parent is 2g alone.
35 42
Theorem 4.4 (Discrete Component Identification), Under the generating process in Equation 7 and
Condition 4.3 the estimated discrete variable d and the true discrete variable d are equivalent up to an invertible function, i.e., d = h(d) with h(.) invertible. Moreover, if Condition 4.1 and Condition further hold, we attain component-wiss identifiability (Definition 4.2) and the bipartite graph up to permutation of component indices.
Proof sketch. Intuitively, each state of the discrete subspace d indexes manifold g(d, : c that maps the continuous subspace c to the observed variable x. These manifolds do not intersect in the observed variable space X' regardless of however close they may be each other, thanks to the invertibility of the generating function (Condition 4.3 This leaves a sufficient footprint in for us to uniquely identify the manifold it resides in, giving rise to the identifiability of d. This reveals the discrete state of each realization of and equivalently the joint distribution p(d, x) where we merge all components in into a discrete variable d. Identifying this joint distribution enables the application of tensor decomposition techniques 24 to disentangle the global state d into individual discrete components d, and the causal graph I', under Condition 4. and Condition 4.3
5

---

4.3 Hierarchical Model Identification
We show that we can identify the underlying hierarchical causal structure G that explains the depen- dence among low-level discrete components de that we identify in Theorem
Remarks on the prob- lem. Benefiting from the identified discrete com- ponents in Theorem 44.4 we employ as observed variables to identify the discrete latent hierarchi- cal model G. Although discrete latent hierarchi- cal models have been un- der investigation for an ex- tensive period, existing re- sults mostly assume rela-
de 28
ds
du d2 de d4 da de d7 ds
d1 d2 d3 da ds de d7 ds
da ds de d7 ds dg
(a) Trees.
(b) Multi-level DAGS.
(c) Ours.
Figure 2: Graphical comparison. Tree Structures permit undirected path
between any two variables. Multi-level DAGs require partitioning variables into levels with edges only between adjacent levels. Our conditions allow multiple paths between variables across levels and include non-leaf observed variables.
tively strong graphical conditions the causal structures are either trees 266 21 22 or multi-level
DAGs [23 48). which can be restrictive in capturing the complex interactions among latent variables among different hierarchical levels. Separately, recent work [19 20 has exhibited more flexible graphical conditions for linear, continuous latent hierarchical models. For instance, prior work [20) allows for multiple directed paths disparate edge numbers within a variable pair and potential non-leaf observed variables. Unfortunately, their techniques hinge on lincarity and cannot directly apply to discrete models of high nonlincarity.
High-level description of our approach. The central machinery in prior work 19 20 is The- orem A2 49,, which builds a connection between easily computable statistical quantities (i.e.. cub-oovariance matrix ranks) and local latent graph information. Dong et al. [20) utilize graph search algorithm to piece together these local latent graph structures to identify the entire hierarchical model. Ideally, if we can access these local latent structures in the discrete model, we can apply the same graph search procedure and theorems to identify the discrete model. Nevertheless, Theorem A2 relies on linearity (i.e., each causal edge represents linear function). which doesn't hold in the discrete case, We show that interestingly, Theorem [A2 can find a counterpart in the discrete case (Theorem 4.8) despite the absence of linearity. Since given the graphical information from Theo- rem A2 the theory in Dong et al. [20] is independent of statistical properties, we can utilize flexible conditions and algorithm therein by obtaining the same graphical information with Theorem 4.8
To present Theorem 4.8 we introduce non-negen non-negative rank rank+ (-) 50 (Definition 4.5) and
separation 511 (Definition as follows.
Definition 4.5 -Nonnnegttive Rank). The non-negative rank of non-negative A E R,*= is equal to the smallest p for which there exist E RTT and C E IR such that A BC.
Definition 4.6 (Treks). A trek Ti,, in a DAG from vertex i j consists of a directed path Pks from to and a direct path Pej from & to j, where we refer to Pki as the i side and Pky as the side.
Definition 4.7 (T-separation). Let A, B, CA. and Cp be subsets (not necessarily disjoint) of vertices in DAG. Then (Ca, CB) t-separates A and B if every trek from A to B passes through either vertex in CA on the side of the trek or a vertex CB on the B side of the trek.
Intuitively, a trek is a path containing at most one fork structure and no collider structures. It is known that can formulate d-separation special form of t-separation (see Theorem ATD. Thus, t-separation is at least as informative as d-separation. As detailed in Dong et al. (20). t-separation can provide more information when latent variables are involved, benefiting from Theorem A2
49
Theorem 4.8 (Implication of Rank Information on Latent Discrete Graphs). Given two sets of variables A and B from non-degenerate, faithful (Condition 4.7 / Condition 1 discrete model G, it follows that rank, + (P A.B) = min [L) partition (L,, L2) I-separates A and B G}.
Example. Suppose every variable in Figure a) is binary, then for A = {di, d2, da), B {dg. ..., ds), rank (P (PA.B) a) 4 since A and B are t-separated by {ds, 2 with 4 states.

---

49 for linear models, Theorem 4.8 acts as an oracle to reveal the minimal t-separation set's cardinality between any two variable sets discrete models beyond linearity. This enables us to infer the latent graph structure from only the observed variables' statistical information. To the best of our knowledge, Theorem 4.8 is the first to establish this connection and can be of independent interest for learning latent discrete models in future work. Although the computation of non-negative ranks can be expensive [50], existing work [52] 53 demonstrates that
Discussion. Parallel to Theorem A2
regular rank tests are decent substitutes, we observe in our synthetic data experiments (Section 5
We present the identification conditions for discrete models as follows (Condition 4.10).
Definition 4.9 (Atomic Covers). Let A C V be set of variables in G with Supp A)] k. where t of the k states belong to observed variables d,, and the remaining
are from latent variables 2z-
A is an atomic cover if A contains a single observed variable, or if the following conditions hold: of atomic covers C, with [Supp(C)) 2 A 1 L1 such that UcEcC C PChg (A)
(i) There exists
and VC C2 E C, C1 n C2 = 0.
(ii) There exists set of covers N,, with [Sup 2 +-1 1 I, such that every element in UNENN =
is a neighbour of V and (UN n (Ucec
(iii) There does not exist a partition of A = A1 U A2 such that both A1 and A2 are atomic covers.
Example. In Figure 2 (c), {22} is an atomic cover if its pure child (dz) and its neighbors {21, As, 24 possess more than Supp(22) 22) + states separately. Otherwise, {22, d1} } can be an atomic cover (some of) pure children {23, 24} and neighbors {2,, dg. d3} } possess Supp(zg) + 1 states separately. Condition 4.10 (Discrete Hierarchical Model Conditions).
i [aithfulress]] All the conditional independence relutions are entailed by the DAG.
ii [Basic Graphical Conditions] Each latent variable = E Z corresponds to a unique atomic cover in G and no is involved in any triangle structure (i.e., three mutually adjacent variables).
in |Graphical Condition on Colliders] In latent graph G. (i) there exists set of variables C such that every variable in C is a collider of two atomic covers L1, Lz, and denote by A the minimal set of variables that d-separates L from L2g. (ii) there is a latent variable in L L2, C
or A, then we must have |Supp( C) + SSupp (A)l Z [Supp(L1)] + [Sup)(L22]
Discussion on the conditions. Condition 4.10 is known the faithfulness condition widely adopted for causal discovery [51] 24, 54 18). which attributes statistical independence to graph structures rather than unlikely coincidence 55 [T. In linear models, Dong et al. (20] introduce atomic covers (Definition 4.9 to represent group of indistinguishable variables. In the discrete case, an atomic cover consists of indistinguishable latent states, which we merge into single latent discrete variable (Condition ii). Intuitively, we treat each state as a separate variable and merge those belonging to the same atomic cover the end of the identification procedure. This handles discrete variables of arbitrary state numbers, in contrast with the binary or identical support assumptions [22] 23 which
we use an alternative condition in Theorem AT2 Condition 4. 10 ii requires each atomic cover to possess sufficiently many children and neighbors to preserve its influence while avoiding problematic triangle structures to ensure the uniqueness of its influence. In contrast, existing work (23) assumes at least three pure children for each latent variable, amounting to six times more states. Condition 10 i ensures adequate side information (large | to discover latent colliders C, admitting graphs more general than tree structures 21 26 22 (i.e., no colliders). Overall, our model encompasses a rich
class of latent structures more complex than tree structures and multi-level DAGS 23 (Figure
Following Dong et al. 20), we introduce the minimal graph operator O Omin (Definition 4.1 and Figure ATp. D.. which merges certain redundancy structures that rank information cannot distinguish.
19 20)). We can merge atomic covers L into P in g if (i) L is pure child of P. (ii) all elements of and P are latent and [L)p = [Supp(P)), and (iii) the pure children of L form a single atomic cover, or the siblings of L form a single atomic cover. We denote such an operator as the minimal-graph operator Omin (9).
Definition 4.11 (Minimal-grapp) Operator
Theorem 4.12 (Discrete Hierarchical Identification). Suppose the causal model G satisfies Condi- tion 4.7 and Condition 4.10 We can identify up the Markov equivalence class of Omin
Proof sketch. As discussed above, Theorem 4.8 gives a graph structure oracle equivalent to The- orem A2 which we leverage to prove Theorem 4.12 Besides the rank test, the major distinction
7

---

Table 1: F1 scores for our method and the baseline Kívva et al. [24]
Figure A2
exhibits the graphs.
Graph
Graph 2
Graph
Graph 4
Graph
Graph
Graph
Graph
Graph
Baseline
0.67 0.0
0.69 0.1
0.67 1 0.0
0.67
0.2
0.63 + 0.0
0.65 4 0.0
0.67 + 0.0
0.65 I 0.0
0.63 1 0.0
Ours
0.94 0.1
0.98 0.1
0.94
= 0.0
0.98
0.2
0.94
= 0.1
0.93 - 0.0
0.93 = 0.1
0.96 = 0.0
0.93 = 0.1
Table 2: F1 scores for our method and the baseline Dong et al. [20]
Figure A3
exhibits the graphs.
Graph
Graph2
Graph 3
Graph 4
Graph 5
Graph 6
Graph
Baseline
0.24 0.3
0.48 +
0.33
+ 0.1
0.0
0.55 0.1
0.0 0.0
Ours
1.0
1.0
0.73 0.0
0.73 0.0
0.75 0.0
0,95 + 0.0
1.0
between Theorem A2 and Theorem 4.8 is that the former returns the number of variables in the minimal t-separation set whereas the latter returns the number of states. Applying the search algorithm from Dong et al. [20] alongside our rank test from Theorem 4.8 to discrete model g results in graph G. In G. each latent variable z in G is split into set of variables ,0)),..,, =({uup(==]] as an .....
atomic cover, with the set size equal to the state number of z. We can then reconstruct the original graph G from 9 by merging these atomic covers into discrete variables, We present our algorithm in Algorithm [and highlight the differences from that in Dong al. 20
Our techniques can also utilize the identical support condition (c.g., binary latent variables) 233 22 for identification under slightly different conditions. We present the results in Theorem A12
Synthetic Data Experiments
Experimental setup. We generate the hierarchical model G with randomly sampled parameters. and follow (24) to build the generating process from to the observed variables x (i.c., graph I') by a Gaussian mixture model. The graphs are exhibited in Figure A2 and Figure A3 in Appendix A4 We
follow Dong et al. (0 to use FI score for evaluation. More details can be found in Appendix A4
Results and discussion. We choose Kivva et al. (24 as our baseline because it is the only method we know designed to learn non- non-aaramttric, discrete latent model from continuous observations. We evaluate both methods graphs in Figure A2 As shown in Table and Table our method consistently achieves near-perfect scores, while the baseline, despite correctly identifying and directing edges among d components, cannot handle higher-level latent variables.
To verify Theorem 4.8
we evaluate Algorithm and a baseline 20 on graphs satisfying the
conditions on G (i.c., purely discrete models in Figure A3). Our method performs well on graphs that meet conditions of Theorem [A2 and achieves decent scores on graphs that do not (Figure A3 and
(e)). The significant margins over the baseline validate Theorem 4.8 and Theorem A12
Interpretations of Latent Diffusion
In this section, we present a novel interpretation of latent diffusion (LD) 28 from the perspective of our hierarchical concept learning framework. Concretely. the diffusion training objective can be viewed as performing denoising autoencoding at different noise levels [56 57). Denoising autoencoders 588 59
and variants (60 61 have shown the capability of extracting high-level,
semantic representations as their encoder output. In the following, we adopt this perspective to interpret the diffusion model's representation (i.e., the UNet encoder output) through our hierarchical model, which connects the noise level and the hierarchical level of the latent representation in our causal model. For brevity, we refer to the diffusion model encoder's output as diffusion representation.
Discrete variables and representation embeddings. In practice, discrete variables are often modeled as embedding vectors from a finite dictionary (e.g., wording embeddings). Therefore. although diffusion representation is not discrete, we can interpret it as an ensemble of embeddings of involved discrete variables. Park et al. the diffusion representation into a finite set of hasis vectors that carry distinct semantic information, which can be viewed as the concept embedding vectors.
62 empirically demonstrates that one can indeed decompose
Given an image x, LD first discretizes it with a vector-quantization generative adversarial network (VQ-GAN) 633 dd = fvo(x). Through the lens of our framework, VQ-GAN represents the image with rich but finite set of embeddings of bottom-level concepts d and discards nuances in the continuous representation c, inverting the generation process in Equation]] I
Vector-quantizatio
8

---

suppllles concopts.
Noken leand
Denoising objectives. As discussed, diffusion
training can be viewed as denoising the cor-
rupted embedding d to restore noiseless d [57]
1
59
64
for a designated denoising model fa at
noise level d: arg max (dd,.Q.((. d) [ (d)d (2)
ft
where y denotes the text prompt. Under this objective, the model is supposed to compress the noisy view d, extract a clean, high-level representation, together with additional informa- tion from the text y. to reconstruct the original embedding d. Formally, the denoising model ft = =fdectt fene,t performs auto-encoding (s(e)) = /eme.((d)) and à = (2s(s))))))))))))) y), where we use (()) to indicate the dependence on the noise level t. We can view the compressed representation Z)(() set of high-level latent variables in the hierarchical model: the encoder Senc. maps the noisy view d, to high-level latent variables 2s, and the decoder Sáec,t assimilates the text information and reconstructs the original view d. In practice, ft is implemented as a single model (c.g.. UNets) paired with time embeddings. We visualize this process in Figure
Noise drowns low-loool information
Figure 3: Diffusion models estimate the latent hier- archical model. Different noise levels correspond to different concept levels. To avoid cluttering, we leave out vector quantization.
Noise levels and hierarchical levels. Intuitively, the noise level controls the amount of semantic information remaining in dz For instance, high noise level drowns the bulk of the low-level concepts d, leaving only sparse high-level concepts in d, In this case, the diffusion representation As()) estimates high concept level in the hierarchical model. In Figure 33, high noise level may destroy low-level concepts, such as the sand texture and the waveforms, while preserving high-level concepts, such as the beach and the sunrise. In Section 7.1 we follow Park et al. [62] to demonstrate diffusion representation's semantic levels under different noise levels.
Theory and practice. We connect LD training and estimating latent variables in the hierarchical model in an intuitive sense. Our theory focuses on the fundamental conditions of the data generating process and does not directly translate to guarantees for LD. That said, our conditions naturally have implications on the algorithm design. For instance, a sparsity constraint on the decoding model may facilitate the identification condition that variables influence each other sparsely (e.g., pure children in Condition 10p. In Section 7.3 we show such constraints are beneficial for concept extraction. We hope that our new perspective can provide more novel insights into advancing practical algorithms.
7 Real-world Experiments
7.1 Discovering Hierarchical Concept Structures from Diffusion Models
In Figure 4 we extract concepts and their relationships from LD through our hierarchical model interpretation. Our recovery involves two stages: determining the concept level and identifying causal links. We add textual concept, like "dog", into the prompt and identify the latest diffusion step that would render this concept properly. If "dog" appears in the image only when added at step O and "eye" appears when added from step 5, indicates that "dog" high-level concept and observe
higher-level concept than
"eyes". After determining the levels of concepts, we intervene changes in low-level ones. No significant changes indicate no direct causal relationship. We explore the relationships among the concepts "dog". "tree", "eyes", "ears", "branch", and "leaf". Figure 4 presents the final recovered graph and intermediate results. See Section A5.3 AS.. for more investigation.
7.2 Diffusion Representation as Concept Embeddings
We support our interpretations in Section 6 that diffusion representation can be viewed as concept embeddings, and corresponds to high-level concepts for high noise levels. Following Park et al. [62]. we modify the diffusion representation along certain directions found unsupervisedly. We can observe that this manipulation gives rise to semantic concept changes rather than entangled corruption Figure 5 Editing the latent representation at early steps corresponds to shifting global concepts.

---

Dog
Tree
Reference
Intervene: dog Intervene: tree
Brunch
Ear
Leaf
(a) The recovered concept graph.
(b) Identify the causal link by "intervention".
Step 0
Step 5
Step 15
Reference
Step 0
Step
Step Reference
Inject
Inject
dog
Tree
Inject
Inject
dog and
tree and
ear
leaves
Inject
Inject
pointy
brown
ear
leaves
Inject wide
Inject
eye
branch
(c) Identify the concept level by the last effective diffusion step dotted lines).
Figure 4: Recovering concepts and their relationships from L.D. (a) The final recovered concept graph among concepts "dog", "tree", "eyee", "cars", "branch", and "leaf". (b) Identifying causal links through "interventions" example, we compare two prompts that vary "dog": dog with wide eyes and wilting tree with short branches, in a cartoon style" and "a big dog with wide eyes and a wilting tree with short branches, in a cartoon style". We observe significant changes in "eyes" but not in "branch". indicating causal link between "dog" and "eyes" but not between "dog" and "branch". (e) Identifying concept levels by the last effective diffusion step. For example, use the base prompt "a with long branches, in cartoon style" and prepend "dog" steps and 15. Only injecting "dog" step works. Similarly, injecting "wide eyes" works at both steps 0 and 5, indicating that "dog" is a higher-level concept than "eyes".
Original
0.6T
In Figure 5] the latent representation in earlier steps (step T) determines breeds (the top row), species (the middle row), and gender (the bottom row). In contrast, the latent representation later steps (step 0.67)) correlates with the dog collar, cat eyes, and shirt patterns. Implementation details and additional results are provided in Appendix A5
7.3 Causal Sparsity for Concept Extraction
Recent work 65 shows that concepts can be extracted as low- rank parameter subspaces of LD models via LoRA [66]. This low-rankness limits the complexity of text-induced changes, resembling sparse influences from latent concepts to their de- scendants. Our theory suggests that different levels of concepts may require varying sparsity levels to capture. We present em- pirical evidence in Section A5.4] Motivated by this, design an adaptive sparsity selection mechanism for capturing con- cepts at different levels. Inspired by Ding al. [67]. we imple- ment a sparsity constraint on the LoRA dimensionality for the model select the LoRA rank each module automatically, benefiting concept extraction (see Appendix A3.4).
le
Figure 5: Semantic latent space. We
modify the diffusion model's representa- tion (UNet encoder's 's output) along prin- cipal directions steps T and 0.6T. Structure changes indicate the semanties of representation and manipulation at the early time ''' induces global shifts. See more examples in Figure A9
8 Conclusion
In this work, we cast the task learning concepts as the identification problem of a discrete latent hierarchical model. Our theory provides conditions to guarantee the recoverability of discrete concepts. Limitations: Although our theoretical framework provides lens for interpretation, our conditions do not directly guarantee diffusion's success, which would require nontrivial assumptions. Also, Algorithm can be expensive for large graphs due to the dependency on the state count. We leave giving guarantees to diffusion models and efficient graph learning algorithms future work.
10

---

Acknowledgments. We thank the anonymous reviewers for their valuable insights and recommenda- tions, which have greatly improved our work. The work of L. Kong and Y. Chi supported in part by NSF DMS-213400 This material is based upon work supported by NSF Award No. 2229881, 1, Al Institute for Societal Decision Making (AI-SDM), the National Institutes of Health (NTH) under Contract 1980,, and grants from Salesforce, Apple Inc., Quris AI, and Florin Court Capital. This research has been graciously funded by the National Science Foundation (NSF) CN$2414087, NSF BCS2040381. 1, NSF IIS2 (2S2123952, NSF ISS 1955532, NSF IIS2123955 NSF IIS231 1990; the Na- tional Institutes of Health (NIH) RO1GM140467; the National Geospatial Intelligence Agency (NGA) HM04762010002; the Semiconductor Research Corporation (SRC) AIHW award 2024AH3210; the National Institute of General Medical Sciences (NIGMS) ROIGMI 40467; and the Defense Advanced Research Projects Agency (DARPA) ECOLE HR001 12390063. Any opinions, findings, and conclusions or recommendations expressed in this publication are those of the author(s) and do necessarily reflect the views of the National Science Foundation, the National Institutes of Health, the National Geospatial Intelligence Agency, the Semiconductor Research Corporation, the National Institute of General Medical Sciences, and the Defense Advanced Research Projects Agency.
References
[1] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Eleventh International Conference on Learning Representations, 2022.
[2] Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerability" of generative adversarial networks. In International Conference Learning Representations, 2019.
[3] Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. Advances in neural information processing systems, 33:9841-995 2020.
[4] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent space of gans for semantic face editing. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 9243 9252, 2020.
[5] Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2021.
[6] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 22500-223 22510, 10, 2023.
[7] Christopher P Burgess, Loic Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representa- tion. arXiv preprint 1r11h:1901.11390 2019.
[8] Francesco Locatello, Dirk Weissenborn, Thomas Unterthiner, Aravindh Mahendran, Georg Heigold, Jakob Uszkoreit, Alexey Dosovitskiy, and Thomas Kipf. Object-centric learning with slot attention. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
[9] Yilun Du, Shuang Li, Yash Sharma, Joshua B. Tenenbaum, and Igor Mordatch. Unsupervised learning of compositional energy concepts. In Proceedings of the International Conference on Learning Representations (ICLR 2022), 2022.
[10] Yilun Du, Kevin Smith, Tomer Ulman, Joshua Tenenbaum, and Jiajun Wu. Unsupervised discovery of 3d physical objects from video. In Conference on Computer Vision and Puttern Recognition (CVPR 2022), 2022.
[11] Nan Liu, Yilun Du, Shuang Li, Joshua B. Tenenbaum, and Antonio Torralba. Unsupervised compositional concepts discovery with text-to-image generative models. Proceedings of the 2023 Conference on Neural Information Processing Systems (NeurIPS 2023), 2023.
11

---

[12] Pang Wei Koh, Thao Nguyen, Yew Siang Tang, Stephen Mussmann, Emma Pierson, Been Kim, and Percy Liang. Concept bottleneck models. In International conference on machine learning. pages 1533853443 PMLR, 2020.
[13] Mateo Espinosa Zarlenga, Pietro Barbiero, Gabriele Ciravegna, Giuseppe Marra, Francesco Giannini, Michelangelo Diligenti, Frederic Precioso, Stefano Melacci, Adrian Weller, Pietro Lio, et al. Concept embedding models. In NeurlPS 2022-36th Conference Neural Information Processing Systems, 2022.
[14] Tuomas Oikarinen and Tei Weng. Clip-dissect: Automatic description of neuron represen- tations in deep vision networks. In ICLR 2022 Workshop PAIR {"eettacciccrrumm 2Struct: Privacy, Accountability, Interpretability, Robustness, Reasoning on Structured Data, 2022.
[15] Mazda Moayeri, Keivan Rezaci, Maziar Sanjabi, and Soheil Feizi. Text-to-concept (and back) via cross-model alignment. In International Conference Machine Learning, pages 25037-25060. PMLR, 2023.
[16] Mazda Moayeri, Keivan Rezaci, Maziar Sanjabi, and Soheil Feizi. Text2concept: Concept activation vectors directly from text. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3743-3748, 2023.
[17] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748 8763..73..06 PMLR, 2021.
[18] Feng Xie, Biwei Huang, Zhengming Chen, Yangbo He, Zhi Geng, and Kun Zhang, Identification of linear non-gaussian latent hierarchical structure. In International Conference on Machine Learning, pages 24370-24387. PMLR, 2022.
[19] Biwei Huang, Charles Jia Han Low, Feng Xie, Clark Glymour, and Kun Zhang. Latent hierarchical causal structure discovery with rank constraints. Advances in Neural Information Processing Systems, 35:5549-5561, 2022.
[20] Xinshuai Dong, Biwei Huang, Ignavier Ng, Xiangchen Song, Yujia Zheng, Songyao Jin, Roberto Legaspi, Peter Spirtes, and Kun Zhang. A versatile causal discovery framework to allow causally-related hidden variables. In The Twelfth International Conference on Learning Representations, 2023.
[21] J. Pearl. Probabilistic Reasoning in Intelligent Systems: Networks of Plausible Inference. Morgan Kaufmann, 1988.
[22] Myung Jin Choi, Vincent YF Tan, Animashree Anandkumar, and Alan Willsky. Learning latent tree graphical models. Journal of Machine Learning Research, 12:1771-1812, 2011.
[23] Yuqi Gu and David B. Dunson. Bayesian pyramids: Identifiable multilayer discrete latent structure models for discrete data, In Journal of the Royal Statistical Society Series Statistical Methodology, 2023.
[24] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Learning latent causal graphs via mixture oracles. Advances Neural Information Processing Systems, 34: 18087-18101, 2021.
[25] Mathias Drton, Shaowei Lin, Luca Weihs, and Piotr Zwiernik. Marginal likelihood and model selection for gaussian latent tree and forest models. Bernoulli, pages 1202-1232 2017.
[26] Nevin L Zhang. Hierarchical latent class models for cluster analysis. The Journal of Machine Learning Research, 5:697-723, 2004.
[27] Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In Sanjoy Dasgupta and David McAllester, editors, Proceedings of the 30th International Conference Machine Learning, volume 28 of Proceed- ings of Machine Learning Research, pages 249-257, Atlanta, Georgia, USA, 17-19 Jun 2013. mlr press/ v28/ anandkumar html
PMLR. URL http:tttt///occeeeee
12

---

[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the 2022 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR 2022), 2022.
[29] Mert Yuksekgonul, Maggie Wang, and James Zou. Post-hoc concept bottleneck models. In The Eleventh International Conference on Learning Representations, 2023.
[30] Eunji Kim, Dahuin Jung, Sangha Park, Siwon Kim, and Sungroh Yoon. Probabilistic concept bottleneck models. In International Conference on Machine Learning, pages 16521-16540. PMLR, 2023.
[31] Marton Havasi, Sonali Parbhoo, and Finale Doshi-Velez. Addressing leakage in concept bottleneck models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:// fn
[32] Chenming Shang, Shiji Zhou, Hengyuan Zhang, Xinzhe Ni, Yujuu Yang, and Yuwang Wang. Incremental residual concept bottleneck models. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11030-11040, 2024.
[33] Kushal Chauhan, Rishabh Tiwari, Jan Freyberg, Pradeep Shenoy, and Krishnamurthy Dvijotham. Interactive concept bottleneck models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 5948-5955, 2023.
[34] Lingjing Kong, Biwei Huang, Feng Xie, Eric Xing, Yuejic Chi, and Kun Zhang. Identification of nonlinear latent hierarchical models. Advances in Neural Information Processing Systems, 36, 2023.
[35] Ilyes Khemakhem, Diederik Kingma, Ricardo Monti, and Aapo Hyvarinen. Variational au- toencoders and nonlinear ica: A unifying framework. In International Conference on Artificial Intelligence and Statistics, pages 2207 20772217.11 PMLR, 2020.
[36] Ilyes Khemakhem, Ricardo Monti, Diederik Kingma, and Aapo Hyvarinen. Ice-beem: Iden- tifiable conditional energy-based deep models based on nonlinear ica. Advances in Neural Information Processing Systems, 11112268-12777 2020.
[37] Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in neural information processing systems, 29, 2016.
[38] Aapo Hyvarinen, Hiroaki Sasaki, and Richard Turner. Nonlinear ica using auxiliary variables generalized contrastive learning. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 859-868. PMLR, 2019.
[39] Jack Brady, Roland Zimmermann, Yash Sharma, Bernhard Schölkopf, Julius von Kügelgen. and Wieland Brendel. Provably learning object-centric representations. In International Conference on Learning Representations (ICLR 2024), 2024.
[40] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-product extrapolation. Advances in Neural Information Processing Systems, 36, 2024.
[41] Bohdan Kivva, Goutham Rajendran, Pradeep Ravikumar, and Bryon Aragam. Identifiability of deep generative models without auxiliary information. Advances in Neural Information Processing Systems, 35: 35:15687-15711 2022.
[42] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel, Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. Advances in neural information processing systems, 34: 16451-16467, 2021.
[43] Lingjing Kong, Shaoan Xie, Weiran Yao, Yujia Zheng, Guangyi Chen, Petar Stojanov, Victor Akinwande, and Kun Zhang. Partial disentanglement for domain adaptation. In International Conference Machine Learning, pages 11455-11472 PMLR, 2022.
13

---

[44] Sanjeev Arora, Rong Ge, and Ankur Moitra. Learning topic models going beyond svd. In 2012 IEEE 53rd annual symposium on foundations of computer science, pages 1-10. IEEE, 2012.
[45] Sanjeev Arora, Rong Ge, Yonatan Halpem, David Mimno, Ankur Moitra, David Sontag, Yichen Wu, and Michael Zhu. A practical algorithm for topie modeling with provable guarantees. In International conference machine learning, pages 280-288. PMLR, 2013.
[46] Gemma E Moran, Dhanya Sridhar, Yixin Wang. and David M Blei. Identifiable variational autoencoders via sparse decoding. arXiv preprint arXiv:2110.10800 2021.
[47] Yujia Zheng, Ignavier Ng, and Kun Zhang. On the identifiability of nonlinear ica: Sparsity and beyond. arXiv preprint arXiv:2206.0775/ 2022.
[48] Yuqi Gu. Blessing of dependence: Identifiability and geometry of discrete models with multiple binary latent variables. arXiv preprint arXiv:2203.04400 2022.
[49] Seth Sullivant, Kelli Talaska, and Jan Draisma. Trek separation for gaussian graphical models. The Annals of Statistics, 38(3). June 2010. ISSN 0090-5364. doi: 10.1214/09-aos766 URL
http://dx doi www.oo..o....o.ooooooooo... 1214/09-11999 00904609
[50] Joel Cohen and Uriel G Rothblum. Nonnegative ranks, decompositions, and factorizations of nonnegative matrices. Linear Algebra and its Applications, 190:149-166 1494-68,111 1993.
[51] Peter Spirtes, Clark Glymour, and Richard Scheines. Causation, prediction, and search. MIT press, 2001.
[52] A. Anandkumar, D. Hsu, F. Huang, and S. M. Kakade. Learning high-dimensional mixtures of graphical models, 2012.
[53] Bijan Mazaheri, Spencer Gordon, Yuval Rabani, and Leonard Schulman. Causal discovery under latent class confounding. arXiv preprint (tti::2111.0745.. 2023.
[54] Feng Xie, Ruichu Cai, Biwei Huang, Clark Glymour, Zhifeng Hao, and Kun Zhang. Generalized independent noise condition for estimating latent variable causal graphs. Advances in Neural Information Processing Systems, 13:14491-14902, 2020.
[55] Jan Lemeire and Dominik Janzing. Replacing causal faithfulness with algorithmic independence of conditionals. Minds and Machines, 23:227-249, 2013.
[56] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computation, 23(7):1661-1674 2011.
[57] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[58] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning. pages 1096- 1103, 2008
[59] Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, Pntore-Antoine Manzagol, and Léon Bottou. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of machine learning research, 11(12). 2010.
[60] Deepak Pathak, Philipp Krahenbuhl. Jeff Donahue, Trevor Darrell, and Alexei A. Efros. Context encoders: Feature learning by inpainting, 2016.
[61] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners, 2021.
[62] Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh. Understanding the latent space of diffusion models through the lens of riemannian geometry. Advances in Neural Information Processing Systems, 36:24129-244114 2023.
14

---

[63] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2021), 2021.
[64] Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for -xtate--mmae synthesis, 2022.
[65] Rohit Gandikota, Joanna Materzyúska, Tingrui Zhou, Antonio Torralba, and David Bau. Concept sliders: Lora adaptors for precise control in diffusion models. arXiv preprint arXiv:2311.12022 2023.
[66] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proceedings of the 2021 International Conference on Learning Representations (ICLR 2021). 2021.
[67] Ning Ding, Xingtai Lx, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In Proceedingy of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), 2023.
[68] Animashree Anandkumar, Daniel Hsu, Adel Javanmard, and Sham Kakade. Learning linear bayesian networks with latent variables. In International Conference on Machine Learning, pages 249-257. PMLR, 2013.
[69] Casper Kaae Sønderby, Tapani Raiko, Lars Maaløe, Søren Kaae Sønderby, and Ole Winther. Ladder variational autoencoders. Advances in neural information processing systems, 29, 2016.
[70] Shengjia Zhao, Jiaming Song, and Stefano Ermon. Learning hierarchical features from deep generative models. In International Conference on Machine Learning, pages 4091-4099. PMLR, 2017.
[71] Xiaopeng Li, Zhourong Chen, Leonard KM Poon, and Nevin L Zhang. Learning latent superstructures variational autoencoders for deep multidimensional clustering. arXiv preprint arXiv:1803.05206. 2018.
[72] Felix Leeb, Giulia Lanzillotta, Yashas Annadani, Michel Besserve, Stefan Bauer, and Bernhard Schölkopf. Structure by architecture: Structured representations without regularization. The Eleventh International Conference on Learning Representations, 2022.
[73] Andrew Ross and Finale Velei-Velez. Benchmarks, algorithms, and metrics for hierarchical disentanglement. International Conference on Machine Learning, pages 9084-9094 PMLR, 2021.
[74] Jascha Dohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics.. In Proceedings of the International Conference on Machine Learning (ICMI. 2015), 2015.
[75] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020.
[76] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations (ICLR 2022), 2022.
[77] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis. In Advances in Neural Information Processing Systems 34 (NeurIPS 2021), 2021.
[78] Alex Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In Proceedings of the International Conference on Machine Learning (ICML 2021), 2021.
[79] Mingi Kwon, Jaescok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. In The Eleventh International Conference Learning Representations, 2022.
[80] Jooyoung Choi, Jungbeom Lee, Chaehun Shin, Sungwon Kim, Hyunwoo Kim, and Sungroh Yoon. Perception prioritized training of diffusion models. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11472-11484 2022.
15

---

[81] Giannis Daras and Alexandros G Dimakis. Multiresolution textual inversion, arXiv preprint arXiv:2211.77115. 2022.
[82] Qiucheng Wu, Yujian Liu, Handong Zhao, Ajinkya Kale, Trung Bui, Tong Yu, Zhe Lin, Yang Zhang. and Shiyu Chang. Uncovering the disentanglement capability in text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1900-1910, 2023.
[83] Antonio Sclocchi, Alessandro Favero, and Matthieu Wyart. A phase transition in diffusion models reveals the hierarchical nature of data. arXiv preprint urXiv:2402.16991 2024.
[84] Judea Pearl. Causality. Cambridge university press, 2009.
[85] Yanming Di. t-separation and d-separation for directed acyclic graphs. preprint, 2009.
16

---

Appendix for
"Learning Discrete Concepts in Latent Hierarchical Causal Models"
Table of Contents
A1 Related Work
18
A2 Proof for Theorem 4.4
19
A3 Proof for Theorem 4.12
20
A4 Synthetic Data Experiments
24
A5 Real-world Experiments
25
A5.1 Implementation Details
25
A5.2 Sparsity in the Hierarchical Model
26
................................
A5.3 Discovering Hierarchical Orders from Diffusion Models
................................
27
A5,4 Causal Sparsity for Concept Extraction
29
A5.5 More Examples
29
17

---

A1 Related Work
Concept learning. In recent years, a sigmificant strand of research has focused on employing labeled data to learn concepts in generative models' latent space for image editing and manipulation [1-6). Concurrently, another independent research trajectory has been exploring unsupervised concept discovery and its potential to learn more compositional and transferable models, as shown in Burgess et al. Locatello et al. [8]. Du et al. 9 10]. Liu et al. [I]]. These prior works focus on the empirical methodological development of concept learning by proposing novel neural network architectures and training objectives, with limited discussion on the theoretical aspect. In contrast, our work investigates the theoretical foundation of concept learning. Specifically, we formulate concept learning as an identification problem for discrete latent hierarchical model and provide conditions under which extracting concepts is possible. Thus, the existing work and our work can be viewed as two complementary lines of research for concept learning. Concurrently, a plethora of work has been dedicated to extracting interpretable concepts from high-dimensional data such as images. cancept-bottlenees
[12] first predicts a set of Human-annotated concepts an intermediate stage and then predicts the task labels from these intermediate concepts. This paradigm has attracted A recent surge of pre-trained multimodal models (e.g., CLIP (77 can explain the image concepts through text directly [14416]. In contrast with these successes, our work focuses on the formulation of concept learning and theoretical guarantees.
a large amount of follow-up work 13 29 33
Latent hierarchical models. Complex real-world data distributions often possess a hierarchical structure among their underlying latent variables. On the theoretical front, Xie et al. (18), Huang et al. [9, Dong et al. (20) investigate identification conditions of latent hicrarchical structures under the assumption that the latent variables are continuous and influence each other through linear functions. Kong et al. 34 extends the functional class to the nonlinear case over continuous variables. Pearl [2]] Zhang [26]. Choi et al. [22] Gu and Dunson [23] study fully discrete cases and thus fall short of modeling the continuous observed variables like images. Specifically, Pearl (21, Zhang (26), Choi et al. [22] focus on the latent trees in which every pair of variables is connected through exactly one undirected path. Gu and Dunson (3 assume a multi-level DAG [68] in which variables can be partitioned into disjoint groups (i.e., levels), such that all edges are between adjacent levels, with the observed variables as the bottom level (i.c., leaf nodes). In contrast, we show that we can not only extract discrete components from continuous observed variables but also uncover higher-level concepts and their interactions. Our graphical conditions admit multiple paths within each pair of latent variables, flexible hierarchical structures that are not necessarily multi-level, and flat structures in which all latent variables are adjacent to observed variables [24]. On the empirical side, prior work (69) improves the inference model of vanilla VAEs by combining bottom-up eata-dependent likelihood terms with prior generative distribution parameters. Zhao et al. (70 assign more expressive (deeper) neural modules to higher-level variables learn more disentangled generative model. Li et al. 71 present a VAE/clustering approach to empirically estimating latent tree structures. Leeb et al. 722 propose to feed latent variable partitions into different decoder neural network layers and remove the prior regularization term to enable high-quality generation. Like our work, Ross and Doshi- Velez (73) consider discrete latent variables. However, their focus is on empirical evaluation benchmarks and metrics, without touching on the theoretical formulation of this task. Unlike these efforts, our work concentrates on the formalization of the data- generatinn process and the theoretical understanding. Thus, these two lines complement each other.
Latent variable identification. Identifying latent variables under nonlinear transformations is central to representation learning on complex unstructured data. Khemakhem et al. K35 36). Hyvarinen and Morioka 37 Hyvarinen et al. 3 assume the availability of auxiliary information (e.g., domain/class labels) and that the latent variables' probability density functions have sufficiently different derivatives over domains/classes. However, many important concepts (c.g., object classes) are inherently discrete. Since latent variables are not equipped with differentiable density functions, identifying these concepts necessitates novel techniques. Our theory requires neither domain/class labels nor differentiable density functions and can accommodate discrete variables readily. Another line of studies [39 40) refrains from the auxiliary information by making sparsity and mechanistic independence assumptions over latent variables, disregarding causal structures among the latent variables. Moreover, images may comprise abstract concepts and convey sophisticated interplay among concepts at various levels of abstraction. In this work, we address these limitations by formulating the concept space as a discrete hierarchical causal model, capturing concepts at distinct levels and their causal relations.
18

---

Latent diffusion understanding, Diffusion probabilistic models 75, 75 28 become the workhorse for state-of-the-ar image generation. Diffusion models' empirical success sparked plethora of efforts to probe into their empirical properties. Kwon et al. 77.. Park et al. [62] discover that the UNet bottleneck representation exhibits highly structured semantic properties, traversing over which manipulates the generated image in meaningful manner. Choi et al. [80], Daras and Dimakis [81], Wu et al. [82). Sclocchi et al. [83] realize that early/late diffusion steps at the inference correlate with coarse/fine features in the output. Recently, Gandikota et al. [65] showcase that concepts are encoded by low-rank influences in latent diffusion models. The theoretical insights in our work consolidate these apparently separate strands of empirical observations and also lead to new understandings that could enhance empirical methodologies.
76.78 have recently
A2 Proof for Theorem 4.4
Condition 4.3 (Discrete Components Identification).
i /Connected Spaces] The continuous support C C R is closed and connected.
[Iverrtiillty & Continuity]: The generating function g in equation iis invertible, and for any fixed d, g(d,-) and its inverse are continuous.
ii (Non-Subset Observed Children]: For any pair d, and dj. one's 's observed children are not the subset of the other's, (dr) & Chr(d,).
Theorem 4.4 (Discrete Component Identification). Under the generating process in Equation 7 and
Condition 4.3 the estimated discrete variable d and the true discrete variable d are equivalent up to an invertible function, i.e., d = h(d) with h(.) invertible. Moreover. if Condition 4.1 and Condition
further hold, we attain component-wise identifiability (Definition 4.2) and the bipartite graph up to permutation of component indices.
Proof of Theorem 4.4 Part 1. The estimate d and the true variable are related through the map [å, é] = و o g(d, c). In the following, we show that the induced relation between and à is invertible under Condition 4.3 The estimated generating process respects the conditions on the true generating process.
We denote that support of the estimate d as (dd)) First, we show by contradiction that for each state E (2dd), corresponds to at most one state & E Q(d) of the estimate d.
two distinct states in and kz. That is, there exist C1 C2 E and ,,,0 E Ć. such that g o y(k ei) = [k,, é1] and g- g (k, C2) [hy, 62). On one hand, As g(k, :) is continuous function and C is connected, the image I(k) = g(k,C) is connected set. On the other hand, f(k,) g(,,, C) and ((x2) == 0k2g are two separate sets due to the invertibility and continuity of 9 and the closed-ness of C. To see this, invertibility implies that Σ(ks) and (kk) are disjoint. The fact that Ò is continuous over è and has continuous inverse over è implies that I(ki) and (kk)) preserve the closed-ness of C. The space formed by two disjoint closed subspaces is disconnected. Since l: corresponds to i and k2 it follows that I(k) Σ U it where I, and iz are nonempty subsets of In, and L.z respectively and inherit their separability. As I(k) is a union of two nonempty separated sets, disconnected. This contradicts the connectedness of I(k). Therefore, for each state k E (2(d), corresponds to at most one state & E (2)
Suppose that corresponds
Having established that each state of d corresponds to at most one state of d, we now show that states ky, b2 of å corresponding to distinct states k1 * 12 of d must also be distinct, i.e., hy + ky if h * l2- Suppose that 3ke * kz, such that the corresponding states h = h2. We denote k = kn h2 and two arbitrary points x1 == g(ki, c1) and x2 := g(kg, 62) from modes kn and ko respectively. As the two estimated discrete states collapse at k, it follows that
x = g(k c1) = g(k, E1)
(3)
gk2 02) g(k, 22). Since ((.,)) is continuous and ê is connected set, the image (ks, C) is path-connected. Thus, we can find path f [0, 1] + X' such that f0) x, and f(1) X2. Also, each point on the path
(4)
19

---

has a positive probability density due to positive p()) and IP (a= However, the two images
g(k,, C) and g(k2,C) disconnected due the invertibility of g and ln + k2. On any path from X1 to X2, there exists points Xy such that the density is strictly zero due to the discrete structure of d. Thus, we have arrived at contradiction. We have shown that if 1 o k2, the corresponding estimated states are distinet L # k2.
Since for for each (2(4), corresponds most state i and distinct states k1, lt2 give rise to distinct states ki, k2, we have proven that for each (2()) corresponds exactly one estimated state & E Q(d))
Definition A2.1 (Mixture Oracles). Let x be a set of observed variables and E [2(4) be a discrete latent variable. The mixture model is defined as IP (x) Ekerra P (d ks) P (x k)) A mixture
oracle MixOracle(x) takes (x) as input and returns the number of components [2(4) | the weights (dd k) and the component P (x|d = k) for k E ((4))
on can reconstruct the bipartite graph I between d and x, and the joint distribution P (d1 Kiy..r tna kna) from IP (x) and MixOracle(x).
Theorem A2 (Kivva et al. (24)). Under Condition 4.1 and Condition 4.3 it1
Part 2. Step 1: Given the first result in Theorem 4.4 we can identify the Proof of Theorem Part
discrete state index k for each realization of x (up to permutations). Since we can do this to all realizations of x and we are given P (x), we can compute the cardinality of the discrete subspace [. the marginal distribution of each latent state PP (d = k). and the conditional distribution P (xx|d = kb) for ((d).
Step 2: Step shows the availability of the mixture oracle MixOracle (i.e., [2] IP (d A)), and P ( kb) as defined in Definition A2.1 Now, all conditions employed Theorem A2 are ready, namely Condition 1. Condition 4.3 in and MixOracle (the consequence of step 1). The derivation
where in Kivva et al. [24] entails identifying a map from the discrete subspace state index d
dna] = [Ki,.... , where k; E s2(") is the state index of the i-th component d,. Thus, we can utilize this map to identify the state index for each individual discrete variable d: from the global index k.
k E (2()) to all discrete components' state indices [d1....,de
Step 3: As stated in Step 2, all conditions in Theorem A2 hold in our problem. Since Theorem Theo- rem A2 additionally identifies the bipartite graph I between {x1, 22, 23....) and(di, d2, da.... the same follows in case.
A3 Proof for Theorem 4.12
In this section, we present a proof for Theorem 4.12
Since all variables are discrete for this proof, for set of variables A, we adopt the notation A to indicate the joint state of all variables in A.
As outlined in Section 4 will derive Theorem 4.8 which serves the bridge between the distribu- tional information and the graphical information, equivalent to the role of Theorem A2 Sullivant et al.
49 in Dong et al. 20 Huang al. [9
To familiarize the reader with the context, we introduce Theorem A2 and the involved graphical definitions treks|4.6) t-separation 4.7, and its connection between d d-separation 184).
Definition 4.6 (Treks). A trek T in DAG from vertex i consists of directed path Pss from k to and a direct path Paj from to j. where we refer to Pki as the i side and Pæy as the j side.
Intuitively, trek is path containing most one fork structure and collider structures. Given this definition, a notion of t-separation is introduced [5]] reminiscent of the classic d-separation.
'e abuse the notation (+) to denote probability density functions for continuous variables and mass functions for discrete variables.
20

---

Definition 4.7 (T-separation). Let A, B, CA, and CB be subsets (not necessarily disjoint) of vertices in DAG. Then (CA. CB) t-separates A and B if every trek from to B passes through either vertex in CA on the A side of the trek or a vertex CB on the B side of the trek.
Theorem A1 (Equivalence between d d-separation and t-separation (85)). Suppose we have disjoint vertex sets A, B, and C in a DAG. Set C d-separates set A and set if and only if there exists partition C: == CA U Cв such that (CA. CB) t-separates A U C and B U.C.
Theorem shows that one can reformulate d-separation with a special form of t-separation. Thus, t-separation is at least as informative as d-separation. Further, as detailed in Dong et al. [20]. I-separation can provide more information when latent variables are involved, benefiting from Theorem A2 [49].
Theorem A2 (Covariance Matrices and Graph Structures 49 ). Given two sets of variables
A and B from a linear model with graph G, it follows that rank(EA.B) = min L E-separates A from B in G}. where ΣA,B denotes the generic covariance matrix between A and B.
Theorem A2 reveals that one access local latent graph structures, ie.. the cardinality of the minimal separation set between two subsets of observed variables, through computable statistical quantities, e.g., covariance matrix ranks. Dong al. (20) utilize these local latent graph structures, together with graphical conditions, to develop their identification theory for linear hierarchical models. Ideally, if we can access such local latent structures the discrete hicrarchical model, we can apply the same graph search procedure and theorems in Dong et al. [20] to identify the discrete model. Nevertheless, Theorem [2 relies on the linearity of the causal model (i.e., each causal edge represents a linear function), which doesn't 't hold in the discrete case. This motivates us to derive a counterpart of Theorem A2 for discrete causal models.
To this end, we introduce a classic theorem (Theorem A3 that connects the non-negative rank of a joint probability table with latent variable states.
Definition 4.5 (Non Nog-negttive Rank). The non-negative rank of non-negative A E R is equal to the smallest p for which there exist E TR= and E TR such that A BC.
Theorem A3 (Non-negative Rank and Probability Matrix Decomposition (50). Let P E RmXnn be a bi-variate probability matrix. Then its non-negative rank rank (P) is the smallest non-negative integer p such that P can be expressed as a convex combination of p rank-one bi-variate probability matrices.
Given this machinery, we now derive Theorem 4.8 which provides equivalent information in discrete models as Theorem A2 linear models.
Theorem 4.8 (Implication of Rank Information on Latent Discrete Graphs). Given two sets of variables A and B from a non-degenerate, faithful ( (Condition 4.7 Condition 4 discrete model G, it follows that rank, (P A.B) min (L) partition (L,, L2) r-separates A and B in G}.
Proof. We express the joint distribution table PA.B as
i Σ P(A = i P =r), r),
(5)
rE(R]
where R E N+ is the smallest possible value. This is always possible since we can assign L as either A or B and obtain a trivial expression.
We note that A \ L, B \ L, and L are disjoint because if A n B is nonempty, it must be a subset of L. Since the graph G is degenneeee (Condition implies the graphical condition that A L and B L are d-separate given L.
4.1 [ and faithful (Condition 4.10 Equation 5
The equivalence relation in Theorem A implies that a partition of L t-separates A and B. Thus, the minimal cardinality R is equal to the smallest number of discrete states of L that t-separates and B. Moreover, Theorem A3 implies that the minimal number of states is equal to the non-negative rank of P PAB. i.c., R rank (PA.B), which concludes our proof.
With Theorem hand, we leverage existing structural identification results on linear hierarchical models (Theorem A10 to obtain the identification results of desire (Theorem 4.12
21

---

We introduce formal definitions of linear models, pure children, and the minimal graph operator, which we refer in the main text. Definition A3.4 (Linear Causal Models 20 19
A linear causal model is DAG with variable set V and an edge set E, where each causal variables e is generated by its parents Pa(v) through a linear function:
U2 = Σ
V + te
(6)
1y (P(())))
where aijj is the causal strength and e; is the exogenous variable associated with V,- Definition A3.5 (Pure Children). A variable set Y are pure children of variables X in graph G. iff Pag (Y) Uy,ey X and X knY Y 0. We denote the pure children of X in G by PChg(X) (X).
Basically, the definition dictates that variable Y has no other parents than X.
Definition 4.11 (Minimal-graph) Operator [99 20 ) We can merge atomic covers into in G if (i) L is pure child of P. (ii) all elements of and P are latent and [Supp(L)] = [Supp(P)), and (iii) the pure children of L form a single atomic cover, or the siblings of L form a single atomic cover. We denote such an operator as the minimal-grapp operator mii (9).
This operator merges certain structural redundancies not detectable from rank information (Lemma A9). Please refer to Figure AT an example.
19
20
Definition A3.6 (Atomic Covers (Linear Models)). Let be set of variables in 9 with | k, where of the k variables are observed variables, and the remaining k t are latent variables. A is an atomic cover if A contains a single observed variable, or if the following conditions hold:
(i) There exists a set of atomic covers C, with |C| 2 k + 1 - t, such that UceeC C S PChg (V) and VC C2 E C, C, n C2
(ii) There exists a set of covers NC, with IM 2 k + - t, such that every element in UNEN is neighbour of V and (UNevNN n (UGECC) = 0.
(iii) There does not exist a partition of A = A₁ U A2 such that both A₁ and A2 are atomic covers.
Theorem A7 (Linear Hierarchical Model Conditions).
i [Rank Faithfulness]: All the rank constraints on the covariance matrices are entailed by the DAG.
i [Basic Graphical Conditions] For any L E V. L belongs to at least one atomic cover (Defi- nition A3.60 the linear model G (Definition A3.4) and no latent variable involved any triangle structure (i.e., three mutually adjacent variables).
ii [Graphical Condition on Colliders] In a latent graph G. (i) there exists set of variables C such that every variable C is a collider of two atomic covers L1, L2, and denote by A the minimal set of variables that d-separates L, from L2, (ii) there is a latent variable in L1 L2, C
or A, then we must have |C| + A Z [L + [2]]-
Definition A3.8 (Skeleton Operator 19 20 ). Given atomic cover A in graph G, for all E A, a is latent, and all e (A)(A), such that a and e are not adjacent, we can draw an edge from a to c.
We denote such an operator as skeleton operator (G).
The skeleton operator introduces additional edges to fully connect atomic clusters 19 20 which are indistinguishable from the rank information (Lemma A9). Please refer to Figure Al for an example.
Lemma A9 (Rank Invariance Huang et al. 19). The rank constraints are invariant with the minimal- graph operator and the skeleton operator; that is, G and (,(Omin())) are rank equivalent.
Theorem A10 (Linear Hierarchical Model Identification (0)). Suppose the G is a linear latent causal model (Definition A3.4) that satisfies Condition A7 Then the hierarchical causal model G is
identifiable up to the Markov equivalent class of O, min (9)).
We note that linear model conditions (Condition A7) and discrete model conditions (Condition 4.10 differ mainly in the substitutes of variables in the linear models with states the discrete models. This originates from the local graph structures we can access, i.e., states in Theorem and and variables in Theorem A2 The skeleton operator Omin (Definition) A3.8 is not necessary under Condition 4.10 since each cover represents a discrete variable whose states must all be connected to its neighbors.
22

---

We now present Theorem 4.12 and its proof.
Condition 4.10 (Discrete Hierarchical Model Conditions).
i [Faithfulnes]] All the conditional independence relations are entailed by the DAG.
i (Basic Graphical Conditions] Each latent variable z ZZ corresponds to a unique atomic cover in G and no is involved in any triangle structure (i.e., three mutually adjacent variables).
ii [Graphical Condition on Colliders] In a latent graph G. (i) there exists set of variables C such that every variable C is a collider of two atomic covers L1, L2, and denote by A the minimal set of variables that d-separates L1 from L2, (ii) there is a latent variable in L1, L2, C or A, then we must have |Supp(C) + [Supp(A)] 2 {Supp(L1)] + |Supp(L22)
Theorem 4.12 (Discrete Hierarchical Identification). Suppose the causal model 9 satisfies Condi- tion 4.7 and Condition 4. 10 We can identify 9 up to the Markov equivalence class of O m
Proof. We observe that the linearity condition (Definition A3.4 in Theorem A10 is only utilized to
invoke Theorem A2 to access the cardinality of the smallest I-separation set between any two sets of observed variables in the linear model. Through this, the graph identification results in Theorem AAI derived based on graph search algorithm repeatedly querying partial graph structures under Condition A7
supplies partial graph structures equivalent to Theorem A2 The difference is that Theorem returns the number of variables in the smallest t-separation set while Theorem 4.8 returns the number of states in the smallest t-separation set. Thus, running Algorithm up to Step (i.e., the original search algorithm Dong et al. 20) with a different rank oracle in Theorem 4.8 highlighted in blue) will return a graph with latent nodes representing discrete states. Algorithm| T is guaranteed to correctly discover all the atomic covers (Theorem TO) and each atomic cover corresponds to latent discrete variable (Condition 4. 1. Thus, we can obtain each true latent variable by merging all the latent nodes A, in each atomic cover A into a discrete latent variable whose support cardinality (2)) equals to the number of latent nodes |A We highlight this procedure (Step 9 in Algorithm [.. Moreover, as all latent nodes (i.c., latent states) in an atomic cover belong to one discrete variable, these latent nodes in adjacent atomic covers must be fully connected. Thus, we do not need the skeleton operator O as for linear models (Theorem A10). This concludes proof for Theorem 4.12
For discrete models (Condition 4. Theorem 4.8
Theorem follows the same reasoning in Theorem 4.12 with the main difference in organizing latent nodes/states into latent discrete variables. Condition A3.11 (Discrete Hierarchical Model Conditions for Identical Supports).
i [Faithfulness]: All the conditional independence relations entailed by the DAG.
ii [Basic Graphical Conditions]: Each latent variable L belongs at least one atomic cover in G and no 2 is involved in any triangle structure (i.e., three mutually adjacent variables).
ii [Graphical Condition on Colliders]: In a latent graph G, if there exists a set of variables C such that every variable in C is a collider of two atomic covers L,, L2, and denote by A the minimal set of variables that d-separates L from L2, (ū) there is a latent variable in L1, L2, C A, then we must have [Supp( C)| + [Supp(A) 2 [Supp(Ln)) + [Supp(L22]--
We introduce the skeleton operator O,
19 20 (Definition A3.8) that include edges between adjacent
covers indistinguishable rank information.
Theorem A12 (Discrete Hierarchical Identification on Identical Supports). Suppose the causal model satisfies Condition 4. i Condition A3. and [Supp(>)] = K 2 for all z E Z. We can identify up the Markov equivalence class of O,( O min (9))-
Proof. The bulk of the proof overlaps with the proof of Theorem 4.12 Following the same reasoning
of the proof of Theorem 4.12 we can obtain a graph with latent nodes representing discrete states before Step 9 and Step 10 in Algorithm I Under the identical support condition in Theorem A12 we
23

---

ds
dg
d10
25
23
dis
dg
d1o
23
ds
dg
d10
di du
da
da do da
d7
d1 d2 da
d4
de du d7
d1 d2
da
de
di do
da
(a) g
(b) mina (9)
(c)
(9))
Figure Al: The discrete graph 9 satisfies conditions in Theorem A12 (i.e., identical supports).
After applying the minimal-graph operator to the graph G. 24 is merged its parent 25. and the rank constraints do not change. After applying the skeleton operator to the graph in (b), 21 has an edge to d7 and 23 has an edge d1. We adopt this example from Huang al. 19
can directly group K states in an atomic cover into latent variable as in Algorithm I Step I0 Since the true latent variable cardinality is known to be identical, we don't t need Condition 4.1 i11 to ensure the structure is well defined. Under Condition A3.1I each atomic cover may contain multiple discrete latent variables, depending on the cover size. It could be possible that one latent variable is not connected to all latent variables in an adjacent atomic cover, as the linear model case. However, this difference cannot be detected from the rank information (Lemma A9). Thus, we need to retain the skeleton operator O, inherited from Theorem A10 This concludes our proof for Theorem AT2
Algorithm 1: The overall procedure for Rank-based Discrete Latent Causal Model Discov- ery. [20] We denote the latent nodes in an atomic cover as A and all observed nodes in the model G Xg. We use blue color to highlight our modifications needed for Theorem 4.12 and
Theorem A1 respectively.
Input :Samples from all observed variables X.g
Output Markov equivalence class g'
def Lateer/aaraaieeeauoo/aisssooo Xg)):
Phase 1: g' FindCISkeleton(Xg)) (Algorithm)) P
for Each Q group of overlapping maximal cliques, in G' do
Set an empty graph G X Uqe Q, No = (N
E Xo s.l. N, X are adjacent g'); X U UNe) (Algorithm 3
Phase 2: g" FindCausalClustersse
Phase 3: G" = RefineCausalClnsterssse
X U N e) (Algorithm[5)
Transfer the estimated DAG g" to the Markov equivalence class and update g' by g": Orient remaining causal directions that can be inferred from structures;
Theorem 4.12
replace each cover A to discrete variable with [Supp(>)] |Az| & update
10
Theorem A12
convert each cover A to log (|Ar.) discrete variables of cardinality K & update g':
11
return
A4 Synthetic Data Experiments
Data-generating processes. For the hierarchical model G, we randomly sample the parameters for each causal module, i.e., conditional distributions p(a, Parents(2;)), according to a Dirichlet distribu- tion over the states of 2; with coefficient 1. For simplicity, set the support size of latent variables to 2. Like Kivva et al. [24), we build the generating process from d to the observed variables (i.e., graph T') by Gaussian mixture model where each state of the discrete subspace corresponds to one component/mode in the mixture model. We truncate the support of each component improve the invertibility (Condition 4.3 f
follow conditions in Theorem A12 and
The graphs are exhibited in
Figure A2 and Figure A3
We adopt F1 score (i.e., Precision Recall to assess the graph learning results 20 We compute recall and precision by checking whether the estimated model correctly retrieves edges in the true causal graph. Ranging between to 1, high scores indicate the search algorithm can recover ground-truth causal graphs. We repeat each experiment over at least 5 random seeds.
Metrics.
2Precision Recall
24

---

Algorithm 2: Phasel: FiskelS [20] (Stage of PC (SI). We denote the joint probability table between two sets A and B as PA P.BB. the adjacent nodes as Adj. non-negative rank with rank and the collection of d-separation sets as Sepset. We use blue color to highlight our modifications.
Input :Samples from observed variables X
Output skeleton g"
def Stage/P((Xc)t
Initialize complete undirected graph G' on Xg: repeat
repeat
Select an ordered pair X, Y that are adjacent in G', s.t., (Adig: 2
Select a subset S S Adjg (X)
) s.t., [S] =
If rank (P
[S], delete the edge between X and Y from G' and record S in Sepset(X, Y) and Sepset(Y X).:
until all X, s.t., (Adig (X)) (Y) 2 and all S E Adig. (X)YY) [S] tested;;
n==+1;;
10 until no adjacent X, Y s.t., [Adig (X) KYY m return g'
Implementation details. Our method comprises two stages: 1) learning the bottom-level discrete variable dl and the bipartite graph I' from the observed variable 2) learning the latent hierarchical model 9 given the bottom-level discrete variable cl discovered in 1). For stage 1), we follow the clustering implementation in Kivva et al. [24] under the same hyper-pane arper-parameter setup in the original implementation. For stage 2), we apply Algorithm T to learn the hierarchical model G. We opt for Step 10 in Algorithm because we evaluate graphs with binary latent variables that meet the conditions of Theorem A12 Following Anandkumar et al. [52] Mazaheri et al. 53 we perform conventional rank computation rather than non-negative rank computation and find this replacement satisfactory. We conduct our experiments on a cluster of 64 CPUs. All experiments can be finished within half an hour. The search algorithm implementation is adapted from Dong al. (20].
Graphical structures. Table | and Table correspond to Figure A2 and Figure A3 respectively. As mentioned above, the graphs meet the conditions of Theorem A12 with the latent variable cardinality equal to two (binary variables).
A5
Real-world Experiments
A5.1
Implementation Details
We employ the pre-trained latent diffusion model 28 SD v1.4 across all our experiments. The inference process consists of 50 steps.
For experiments in Section 7.1 we inject concepts by appending keywords to the original prompt. For instance, we inject the concept pair ("sketch"" "wide eyes") in Figure A5 as follows. For the inference steps 10, we feed the text prompt "A picture of person", for steps 10 20, "a photo of person, in a sketch style", and for steps 20 - 50, "a photo of person, in sketch style, with wide eyes". For the reverse injection order (injecting "wide eyes" before "sketch"), we inject the following prompts at the three-step stages: "A picture of a person", "a photo of a person, with wide eyes", and "a photo of a person, with wide eyes, in sketch style".
For experiments understanding the UNet's latent presentation (Figure 5) we adopt the open-sourced code of Park al. [62].
we randomly generate images with the pre-train latent diffusion model and record their attention score across layers. To compute the relative sparsity, we select the threshold as 1/4096 and compute the proportion of the attention scores over this threshold. For the attention visualization, randomly select a head from the last attention module in the UNet architecture.
For the attention sparsity experiment (Figure A4),
We follow the implementation of Gandikota et al. 665 to train concept sliders of various ranks. We adopt their evaluation protocol to obtain CLIP and LPIPS scores over 20 randomly sampled images for each rank, concept, and scale combination. We evaluate ranks in {2, 4, 8} and scales {1, 2, 4, 5).
25

---

Algorithm 3: Phase2: FindCausalChusters (20 We use ||. to denote the number of all elements in a set of sets. use the term "nodes" to refer to dummy variables that represent states rather than causal variables in the intermediate graph. We use blue color to highlight our modifications.
Input :Samples from observed variables Xg
Output:Graph G
FindCausalCusstersgs Xg):
Active set S + Ng = (1x1)
1/ S is set of covers Only when nothing can found // udner current à we by
repeat
G' found Search(g', S, Xg, k); Iound then else & & +
until & is sufficiently large;
return g':
def Search(G', Xg. kj: Rank deficiency set D 0 for T & PowerSer(S)) (from to 0) do
// To store rank deficient combinations
10
11
S' (S(T) (UTerPChor (T)): for t 1000000 do
// Unfold S to get st
repeat
14
Draw a set of t observed covers X' C S' Ag:
repeat
Draw set of covers C S s.t., |[|]| ++ and get + ('((XC))))) if rank (Peux, = k and NoCollider(C, X. N) then Cdd
until exhausted; D * then
19
for D. E D do if IPag: UX.)
then P UD)
Create new latent nodes L, s.t., P L Pag (D:) X and
L = k |Pag (D,) X Update G' by taking elements of D, the pure children of P: P atomic then Update S (S) P;
return G True:
// Return to search with h =
26
until X exhausted:
return G', S, False;
// Return to search with l k
Algorithm 4: NoCollider (20). We use blue color to highlight our modifications.
Input :C, X
Output Output:Whether Whether there exists O E C s.t., a collider of C)(O) and def NoCollider(C, X, NN:
for c |C| do
Draw C' C Cst. (C'.ttC''|= =c repeat
if rank mankkt(Pexxxe (Pouxxux) < C U then return False;
until C' exhausted; return True
The rank selection technique, inspired by Ding al. 677, involves multiplying each LoRA's inner dimension with a scalar parameter and imposing to penalty these scalar parameters. The weight 3, - 4, le 55.. We repeat each run at
on the lo penalty selected from {le - 1,, Ie least three random seeds. The code can be found here
We conduct all our experiments on 2 Nvidia L40 GPUs. Each image inference takes the same time as in standard SD v1.4 (i.c., within two minutes). Each concept slider in Figure takes around half an hour train.
A5.2 Sparsity in the Hierarchical Model
To verify the sparse structure condition in Theorem 4.12 we view the attention sparsity in the LD model as an indicator of the connectivity between specific hierarchical level and the bottom concept level. Figure A4 visualizes the attention sparsity of an LD model over diffusion steps and specific
26

---

Algorithm 5: Phase3: RefincCausalClusters
Input :Graph G'
Dutput:Refine graph
def RefineCaasalCCussteess Xa):
repeat
Draw atomic V from G':
Delete V, neighbours of V that latent, and all relating edges from G' get ;;
G' FindCausalClusters(@ Xg);
until No more V found and all V exhausted;
return
Graph
Graph
Graph
Graph
Graph
Graph
Graph
(b) Graph 8
Graph
Figure A2: Causal graphs evaluated in Table 1 We denote the observed variables with x, the bottom-level latent discrete variables with d, and the high-level latent discrete variables with 2.
attention patterns in the model. We observe that the sparsity increases the generative process progresses, which reflects that the connectivity between the hierarchical level (zg,s, ) and the bottom level variable (d) becomes sparse and more local march down the hierarchical structure, which indicates a gradual localization of the concept.
A5.3 Discovering Hierarchical Orders from Diffiusion Models
We provide further evidence that latent representations different diffusion steps correspond to different levels of the hierarchical causal model. We select concept pairs, each with higher-level and lower-level concepts. For example, in ("sketch," "wide eyes"). "sketch" is more global, while "wide eyes" is more local. We alter the text prompt during diffusion generation for concept injection, appending a sketch style" inject "sketch" (see Appendix [5 for prompts). In Figure global concepts are successfully injected at early diffusion steps and local ones at late steps (top row). Reversing this order fails, as shown the bottom row. For example, injecting "sketch" early and "wide eyes" late renders both correctly, but the global concept "sketch" is absent under the reverse
27

---

Graph
Graph
(c) Gruph
(d) Graph 4
de
(c) Graph
(f) Graph
(g) Gruph
Figure A3: Causal graphs evaluated in Table We denote the bottom-level latent discrete variables with d, and high-level latent discrete variables with z. Since baselines cannot extract discrete subspaces, directly feed the algorithms the bottom-level discrete variables and test their structure learning performances.
(a) Attention sparsity.
(b) Early step (index 1).
(c) Middle step (index 481). (d) Late step (index 981).
Figure A4: Sparsity patterns in latent diffusion models' attention. We compute the proportions of the attention scores lower than fixed threshold over the entire model. We can observe that the sparsity increases greatly towards small timesteps, i.c., the lower levels of the hierarchical model, which verifies our theory.
28

---

(sketch, wide eyes)
(pink bricks, flowers)
(anime, cat)
(messy, flowers)
(female, disinterested)
Figure A5: Hierarchical Concept Ordering. We inject concepts of distinct abstraction levels into the generating process at different time steps. In the top row, the concept injection follows the hierarchical order, which renders injected concepts faithfully. The bottom reverses the hierarchical order and cannot incorporate concepts properly. More examples in Figure A8
injection order. This supports theory that concepts are hierarchically organized, with higher-level concepts related to earlier diffusion steps.
A5.4 Causal Sparsity for Concept Extraction
Figure A6lure/Asstn shows that indeed concepts different abstraction levels have desirable representations different ranks. For instance, the concept of bright weather is appropriately conveyed by a rank-2 LoRA and higher-rank LoRAs alter the background. The same observation occurs other concepts, where inadequate ranks fail to capture the concept faithfully and unnecessary ranks entangle the target concept with other attributes.
Figure A7 presents the CLIP and LPIPS evaluation for the baseline and our approach, where the CLIP score evaluates the alignment between the image and the target description and the LPIPS score measures the structure change between the edited image and the original image. We can observe that under the sparsity constraint, approach attains the highest CLIP score and the lowest LPIPS score when compared with the baselines of several ranks, indicating a higher level of alignment and a lower level of undesirable entanglement.
A5.5 More Examples
29

---

Original
Rank
Rank 4
Rank 8
Bright weather
Snow
Castle
Dogs
Figure A6: Concepts have varying levels of sparsity. We show that concepts of various abstraction levels correspond to different sparsity levels. For instance, bright weather appropriately conveyed by rank-2 LoRA and higher-rank LoRAs alter the background. Inadequate ranks fail to capture the concept faithfully and unnecessary ranks entangle the target concept with other attributes.
30

---

35
cases savol acaho
sratt useed srate
drsfucc rurkl
30
20
15
bright seaatior
foedining fond
abstrock ove
Blog
Concepts
(a) CLIP scores.
rerkd suuret
custicop:: opo sanel scels
api raww acalet
0.4
canit.scav shonnecipp mikk mikk. srives
0.0
cuttle
bright westher
Now
foedinigg food
abotractt arr
dogs
Concepts
(b) LPIPS scores.
Figure A7: CLIP/LPIPS evaluation. We evaluate approach and baselines individual rank constraints. A high CLIP score is favorable as it indicates semantic alignment. A low LPIPS score more favorable it indicates minimal excessive changes. We compare the optimal fixed rank setting on each concept. For instance, "castle opt: aan4_sscaled indicates that the optimal setting for the concept "castle" is the LoRA of rank 4 and scale 3. With a adaptive rank selection, our approach outperforms or keeps with the optimal fixed setting across different concepts. We repeat each training over three random seeds.
method "sparse" with
31

---

Cyberpunk High
Church style With tree
Young White hair
Crowd people Excavator
Kitchen With flowers
Figure A8: More examples for Figure A5
Original
0.6T
Original
T
0.6T
Figure A9: More examples for Figure
5
32

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Section 4 Section 6 and Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
33

---

Justification:
All proof' are in Appendix A2 and Appendix
A3
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See Appendix A4 and Appendix
AS
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
34

---

Answer: [Yes]
Justification:
We provide an URL in the appendix.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
wwt.Suubmisioooooiiooooooooo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
See Appendix A4 and Appendix A5
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Experiments in Table and Table 2 are averaged over five seeds. Experi- ments in Figure A7 are computed over at least twenty images and over three seeds for the computation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
35

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A4 for synthetic experiments and Appendix A5 for diffusion experi- ments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This paper is theoretical nature and does not have directly social impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
36

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We mainly use existing codebases, as acknowledged in Appendix A4 and Appendix A5
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cow/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
37

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
38


Reviews:
[{"review_id": "9zH44Nq2RY", "reviewer": "Reviewer_ZL2x", "summary": "This paper presents a theoretical framework for learning discrete concepts from high-dimensional data using latent hierarchical models. \n\nThe authors propose formalizing concepts as discrete latent causal variables within a hierarchical causal model, and discuss under which condition the identification of these concepts and their relationships is possible. The theoretical contributions identifying those conditions and providing both theoretical insights and empirical validation using synthetic data, along with an interpretation of latent diffusion models through the proposed framework.", "questions": "- Given my limited expertise, I am curious about the applicability of this framework in a supervised setting, particularly in relation to classification tasks. Could this framework be adapted, to show for example that supervised model learn only a part of the hierarchical graph relevant to classification problems?", "limitations": "Yes, the limitations identified by the authors are accurate and well-documented. Regarding the weakness I mentioned, I reserve the right to increase the score if the authors adequately address my major concerns.", "rating": 7, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "First, I apologize to the authors as I am not at all an expert in causal inference, and highly unsure about my remark (whether there be positive or negative), I also would like to mention to the author that I ensure myself the AC is aware of this. \n\nNevertheless, concerning the strength that I identified:\n\n1. Up to my knowledge, the paper introduces an interesting formalization of what diffusion model are doing: learning concepts over hierarchical models, which is an interesting viewpoint and could help us better understand those models.\n2. The identification conditions and theorems seems well-formulated (again, not an expert).\n3. The authors try to validate their theoretical claims with real data experiments, I especially like the 6.1 which seems to partially verify their claim", "weaknesses": "Nevertheless, according to me (again, not an expert) this paper has problems, some more important than others.\n\nSo I will separate them into major problems (**M**) and minor problems (_m_). I want to make it clear that for me, all these problems are solvable and do not detract from the quality of the paper.\n\nLet's start with what I think are the Major problems (**M**):\n\n**M1**. Practicality of Recovering Hierarchical Graphs:\n- I am left wanting more; the theoretical framework seems solid, but I would like to see concrete results. For example, can you recover the concept tree for the dog class in Stable Diffusion (or smaller diffusion model) ? Demonstrating this would undeniably highlight the paper's value and lead to a clear acceptance from my side. As it stands, I wonder if this framework could eventually teach us anything about diffusion models.\n\n\n**M2**. Empirical Evidence for Real-world Data:\n- The paper partially validates its claims using synthetic data. Even Section 6.1 is great, but clearly not enough to validate your claim. I'd say Figure A.4 in the appendix is another proof, and I expected maybe a comparison of this sparsity level with a real hierarchical model. Could we recover any information from the tree using this sparsity curve?\n\n**M3**. Realism of Condition 3.3:\n- There are doubts about the practicality and realism of Condition 3.3, especially 3.3-ii. It would be beneficial to discuss how realistic these conditions are in practical scenarios and provide more context or examples to substantiate them.\n\nNow for the minor problems:\n\n_m1_. Partial Literature Review:\n- The related work section doesn't discuss concept extraction, which is a significant field with many papers every year at this conference. To me, you should at least cite or mention this literature.\n\n_m2_. Clarity of Theoretical Explanations:\n- Some parts of the theoretical explanations, particularly in Sections 3.2 and 3.3, are dense and may be difficult for readers to follow. Additional clarifications and examples would improve the accessibility of these sections.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your careful assessment and thoughtful comments on our work! We address your concerns point to point as follows.\n\n>M1. \"Practicality of Recovering Hierarchical Graphs\".\n\nThank you for the great question! In light of your suggestion, we’ve included in our revision the following experiment, where we extract concepts and their relationships from Stable Diffusion through our hierarchical model interpretation. Please find the results in the submitted PDF file in the global response.\n\nOur recovery involves two stages: determining the concept level and identifying causal links. We add a textual concept, like \"dog\", into the prompt and identify the latest diffusion step that would render this concept properly. If \"dog\" appears in the image only when added at step 0 and \"eyes\" appears when added from step 5, it indicates that \"dog\" is a higher-level concept than \"eyes\". After determining the levels of concepts, we intervene in a high-level concept and observe changes in low-level ones. No significant changes indicate no direct causal relationship.\n\nAs shown in the submitted PDF, we explore the relationships among the concepts \"dog\", \"tree\", \"eyes\", \"ears\", \"branch\", and \"leaf\". We provide the final recovered graph and some intermediate results of both stages. Although the current experiment scale (number of involved concepts) is relatively small, we hope these results demonstrate how to leverage the hierarchical causal model in our work to investigate black-box diffusion models.\n\n>M2: \"Empirical Evidence for Real-world Data\".\n\nThank you for the interesting question. Given your question, we spent quite some time attempting to develop a quantitative relation between the attention sparsity and the graphical structure. Unfortunately, the gap seems nontrivial given that these two quantities belong to quite different objects (causal graphs, deep-learning layers) -- the same attention sparsity may correspond to a large family of causal graphs. For now, we think this metric may better serve as a qualitative indicator of the graphical connectivity, as in the manuscript. \n\n>M3: \"Realism of Condition 3.3\".\n\nThank you for pointing this out. Given your suggestions, we’ve included the following discussion in our revised manuscript (original line 184).\n\nCondition 3.3 i: “In practice, the continuous variable $\\mathbf{c}$ often controls extents/degrees of specific attributes (e.g., sizes, lighting, and angles) and takes values from intervals (which are connected spaces). For instance, the variable related to “lightning” ranges from the lowest to the highest intensity continuously.”\n\nCondition 3.3 ii: “For images, the invertibility condition assumes that images preserve the semantic information from the latent variables. Thanks to their high dimensionality, images often have adequate capacity to contain rich information to meet this condition. For instance, the image of a dog contains a detailed description of the dog’s breed, shape, color, lighting intensity, and angles, all of which are decodable from the image.”\n\nCondition 3.3 iii: “Practically, this condition indicates that lowest-level concepts influence diverse parts of the image. Lowest-level concepts are often atomic such as a dog’s ear, eyes, or even finder. As we can see, these atomic features often don’t overlap in the image (e.g., ears and eyes are separate).”\n\n>m1: \"Partial Literature Review\".\n\nThank you for the helpful suggestion! Thanks to your pointer, we have included the following paragraph in our revision. Please let us know if we have overlooked important works and we would be happy to include them. \n\n“A plethora of work has been dedicated to extracting interpretable concepts from high-dimensional data such as images. Concept-bottleneck [12] first predicts a set of human-annotated concepts as an intermediate stage and then predicts the task labels from these intermediate concepts. This paradigm has attracted a large amount of follow-up work [13][a-e]. A recent surge of pre-trained multimodal models (e.g., CLIP [17]) can explain the image concepts through text directly [14-16]. In contrast with these successes, our work focuses on the formulation of concept learning and theoretical guarantees.”\n\n[a] Post-hoc Concept Bottleneck Models. Yuksekgonul et al. ICLR 2023.\n\n[b] Probabilistic Concept Bottleneck Models. Kim et al. ICML 2023.\n\n[c] Addressing Leakage in Concept Bottleneck Models. Havasi et al. NeurIPS 2022.\n\n[d] Incremental Residual Concept Bottleneck Models. Shang et al. CVPR 2024.\n\n[e] Interactive concept bottleneck models. Chauhan et al. AAAI 2023.\n\n>m2: \"Clarity of Theoretical Explanations\".\n\nThank you for your suggestion! Besides the examples for Condition 3.3 as quoted above, we have included the following for Condition 3.7.\n\n\"Intuitively, Condition 3.7-ii requires that each discrete variable has sufficiently many children and neighbors to preserve its influence while avoiding problematic triangle structures to ensure the uniqueness of its influence. Condition 3.7-iii requires sufficient side information (large $|\\mathbf{A}|$) to identify collider $\\mathbf{C}$.\"\n\n>Q: \"applicability of this framework in a supervised setting\".\n\nThank you for the intriguing question! We believe this is possible. In the supervised setting, the prediction target is often a high-level concept. To make correct predictions, the model may need to leverage low-level concepts, like ears and fur for classifying cats. Thus, one may impose sparsity constraints on the representation to view which features are engaged for predicting a specific class and infer certain concept structure therein. For example, if predicting cats and dogs calls for latent variables $(z_{1}, z_{2})$ and  $(z_{2}, z_{3})$ respectively, one may infer that \"cat\", as a high-level variable, is a parent to $ (z_{1}, z_{2}) $, and \"dog\" is a parent to $ (z_{2}, z_{3}) $.\n\n---\n\nPlease let us know if it is unclear and we would be happy to discuss further!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wtADHZqnHx", "reviewer": "Reviewer_7auw", "summary": "This paper introduces a theoretical framework for learning discrete concepts from high-dimensional data using latent hierarchical causal models. The key contributions are:\n\n1) Formalizing concept learning as identifying discrete latent variables and their hierarchical causal structure from continuous observed data.\n2) Providing identifiability conditions and proofs for recovering discrete latent variables and their hierarchical relationships.\n3) Interpreting latent diffusion models through this hierarchical concept learning lens, with supporting empirical evidence.\n\nThe work bridges theoretical causal discovery with practical deep generative models, offering new perspectives on how concepts might be learned and represented.", "questions": "1. How robust is the identification process to small violations of the invertibility condition (Condition 3.3-ii)? Are there relaxations of this condition that could make the method more applicable to real-world data while maintaining identifiability?\n2. Your interpretation of latent diffusion models suggests a correspondence between diffusion steps and concept hierarchy levels. How might this insight be used to design a diffusion process that explicitly learns and respects a given hierarchical concept structure?\n3. The theory assumes discrete latent variables, but the latent space in diffusion models is continuous. How do you reconcile this discrepancy, and could your framework be extended to handle continuous latent variables with discrete-like behavior?", "limitations": "The authors adequately discuss the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Novel formalization of concept learning as a causal discovery problem, providing theoretical grounding for an important area of machine learning\n2. Rigorous proofs for identifiability of discrete latent variables and hierarchical structures under relatively mild conditions\n3. Flexible graphical conditions that allow for more complex hierarchical structures than previous work\n4. Interesting connection drawn between the theoretical framework and latent diffusion models, with empirical support\n5. Clear potential impact on understanding and improving deep generative models for concept learning", "weaknesses": "1. The identification conditions (Condition 3.3 and 3.7) may be too restrictive for real-world scenarios. For instance, the invertibility requirement on the generating function g (Condition 3.3-ii) could be difficult to guarantee in practice, especially for complex high-dimensional data.\n2. The method relies heavily on rank tests of probability tables (Theorem 3.5), which can be computationally expensive and numerically unstable for large state spaces or when probabilities are close to zero.\n3. The approach assumes a clear hierarchical structure among concepts, but real-world concepts often have complex, overlapping relationships that may not fit neatly into a DAG structure.\n4. The theory doesn't address how to handle noise or uncertainty in the observed data, which could significantly impact the identification of discrete states and the overall graph structure.\n5. While the connection to latent diffusion models is interesting, the paper doesn't provide a concrete mechanism to leverage the theoretical insights for improving diffusion model architectures or training procedures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your encouraging words and thoughtful comments! We address your concerns as follows.\n\n>W1: “The identification conditions (Condition 3.3 and 3.7) may be too restrictive… for complex high-dimensional data.”\n\nThank you for your comments. Maybe counterintuitively, the high dimensionality of image data actually makes Conditions 3.3 and 3.7 more likely to hold.\n\nSpecifically, for the invertibility condition, note that since $\\mathbf{x}$ is generated by $\\mathbf{z}$ ($\\mathbf{x} := g(\\mathbf{z})$), $\\mathbf{x}$ cannot contain more information than $\\mathbf{z}$. Thus, the non-invertibility problem arises only when $\\mathbf{x}$ fails to preserve the information of $\\mathbf{z}$. High dimensionality gives $\\mathbf{x}$ sufficient capacity to contain such rich information and facilitates invertibility. This assumption is also well-accepted in the community on image data [32-34]. \nThis also applies to Condition 3.3-iii and Condition 3.7-ii: with a large dimension of the observed variable $\\mathbf{x}$, the children of each latent variable $d$ are less likely to overlap (Condition 3.3-iii), and latent variables are more likely to have unique, observed descendants (Condition 3.7-ii).\n\nAt the same time,  we acknowledge that there do exist situations where some assumptions are violated. However, since science is constructed step by step, we hope our results can shed light on further development of this field.\n\n>W2: “...rank tests (Theorem 3.5), which can be computationally expensive and numerically unstable…”\n\nThank you for raising this concern. We agree that rank tests are not easy (as noted in lines 415-417). However, we hope and believe that as these tests become increasingly important [19,20,41,42], more stable and efficient methods will be developed shortly. Therefore, we hope the current limitations of rank tests do not diminish the value of this contribution.\n\n>W3: “... overlapping relationships that may not fit neatly into a DAG structure”.\n\nBy \"overlapping relationships,\" we were wondering if you referred to cyclic relations among latent variables. (Please let us know if we've misunderstood this.) We agree that this paper focuses on the DAG structure. If non-DAG is necessary to address specific issues, we believe the ideas articulated in this paper can still provide valuable insights, although many practical issues should be considered.\n\n>W4: “... how to handle noise or uncertainty in the observed data..”\n\nThank you for raising the issue of noise or uncertainty in the observed data, which is a significant and challenging problem in causal discovery and causal representation learning, as noted in recent contributions [a]. Addressing these problems is highly nontrivial, so we believe it should and will be tackled after solutions to the basic settings are clear, which is what we aim to achieve in this paper.\n\n[a] Causal Discovery with Linear Non-Gaussian Models under Measurement Error: Structural Identifiability Results. Zhang et al. UAI 2018.\n\n>W5: Concrete mechanism for improving diffusion models.\n\nThank you for the comment. In our initial attempts, we applied basic ideas like sparsity to improve concept extraction techniques from the diffusion model, as shown in our experiments (Section 6.2 & Figure A6). We are actively working on building more principled generative models using our theoretical insights. Additionally, we hope the connection to hierarchical causal models presented in this work can inspire the community to develop more controllable and interpretable generative models.\n\n>Q1: robustness to invertibility violation and possible relaxations.\n\nThis is a great question. We have been working on this for a long time. Recently, it seems hopeful to solve this problem by greatly extending [b]. While they consider a single latent variable in the probabilistic setting, extending to multiple latent variables seems highly possible but requires substantial investigation.\n\n[b] Instrumental variable treatment of nonclassical measurement error models. Hu et al. Econometrica, 2008\n\n>Q2: “... to design a diffusion process that explicitly learns and respects a given hierarchical concept structure?”\n\nGreat question! We believe this correspondence can benefit the controllability of current diffusion models, which remains challenging for existing methods [c,d]. Unlike standard diffusion models, our framework suggests injecting different concepts at various diffusion steps, ensuring that concepts at different levels are properly rendered in the image [d]. We see more structured language and image interaction as a promising direction where our work could provide valuable insights.\n\n[c] Compositional Text-to-Image Generation with Dense Blob Representations. Nie et al. ICML 2024.\n\n[d] ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment. Hu et al. ArXiv 2024.\n\nQ3: The discrepancy between discrete variables and continuous diffusion latent space; possible extensions to continuous latent variables.\n\nThank you for the question. We view the continuous latent space of diffusion models as an ensemble of embedding vectors for discrete concepts, similar to word embeddings in NLP (see lines 330-335). Park et al. [52] and our experiments (lines 388-399) support this interpretation, showing that the continuous space can be decomposed into a finite set of basis vectors, each carrying distinct semantic information.\n\nRegarding potential extensions, our theoretical analysis is connected to identification results on continuous latent variable models [19,20] (see lines 223-233) and can be adapted accordingly. We should note that for continuous models, strong assumptions like linearity are still required [19,20]. We hope to develop a completely non-parametric framework to handle such models in the near future.\n\n---\n\nPlease let us know if you have further questions -- thank you so much!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kWYnxgGE7i", "reviewer": "Reviewer_ae89", "summary": "This work introduces a novel identifiability analysis for a hierarchical latent models where latent variables are discrete and observations are continuous. The novelty of the result resides in the fact that previous results consider mainly continuous latent variables or make stronger assumptions on the form of the latent graph. An algorithm based on the theory is proposed and tested on synthetic data. Analogies between the approach and diffusion models are drawn.\n\n**Review summary**\nOverall I believe the theoretical contribution is interesting, important and novel, but the presentation requires some non-trivial restructuring since at the moment, a significant portion of the content of the paper is relayed to the appendix which makes it very hard to follow. I also thought the section on Diffusion Models was unconvincing and a bit disconnected from the rest of the contributions. I provided some suggestions, including submitting to a venue that allows for more space, like JMLR for instance. Given this, I can only recommend borderline acceptance.", "questions": "Condition 3.1\n- The full support condition seems a bit strong, can the author discuss what it would mean for the running example with the dog?\n- The function ne(v) was not defined, this is neighbors of v, right? It’s the union of Parents and children of v, correct?\n\nThe sparsity condition of Condition 3.3(iii) seems to be crucial for disentanglement in Theorem 3.4. How does this assumption compare to other works using sparsity of the decoder for disentanglement, such as [b,c,d]? I really believe there should be a discussion comparing the graphical assumptions with those of [d].\n\nDefinition 3.6: At line 256, what is the support of a set of atomic covers? (Supp(C) ?)\n\n**References**\n\n[65] Sébastien Lachapelle, Divyat Mahajan, Ioannis Mitliagkas, and Simon Lacoste-Julien. Additive decoders for latent variables identification and cartesian-product extrapolation. Advances in Neural Information Processing Systems, 36, 2024.\n\n[a] S. Lachapelle, P. R. Lopez, Y. Sharma, K. Everett, R. L. Priol, A. Lacoste, and S. Lacoste-Julien. Nonparametric partial disentanglement via mechanism sparsity: Sparse actions, interventions and sparse temporal dependencies, 2024.\n\n[b] J. Brady, R. S. Zimmermann, Y. Sharma, B. Scholkopf, J. von Kugelgen, and W. Brendel. Provably ¨ learning object-centric representations. In Proceedings of the 40th International Conference on Machine Learning, 2023.\n\n[c] G. Elyse Moran, D. Sridhar, Y. Wang, and D. Blei. Identifiable deep generative models via sparse decoding. Transactions on Machine Learning Research, 2022.\n\n[d] Y. Zheng, I. Ng, and K. Zhang. On the identifiability of nonlinear ICA: Sparsity and beyond. In Advances in Neural Information Processing Systems, 2022", "limitations": "Limitations were discussed properly throughout the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- I believe the problem of identifiability in hierarchical latent variable models is interesting and important.\n- The theory presented seems non-trivial and valuable (I did not read the appendix)\n- Most identifiability results assumes continuous latent variables, so I was pleased to see further progress made in the case of discrete latents, which is much less common in the literature.\n- I appreciated the high-level explanation of the proof technique between lines 223-233 which makes the connection to prior work transparent.\n- The work is transparent about its limitations.\n- Many examples are presented, which is helpful to understand the complex notions.", "weaknesses": "**Writing**\n\nI thought the writing was quite good and easily understandable up until Section 3.3, where quality started degrading in my opinion. It really looks like the authors were running out of space and decided to relay *a very large* portion of the content to the appendix. Here's a (probably non-exhaustive) list of important concept and contributions which were relayed to the appendix:\n- t-separation\n- non-negative rank \n- the minimal-graph operator \n- the skeleton operator \n- Condition A3.15\n- Algorithm 1 (this is the main practical contribution!)\n- adaptive sparsity selection mechanism for capturing concepts at different levels (another practical contribution)\n- The literature review.\n\nI can understand when a proof or even when a few very technical assumptions are kept in the appendix, as long as it does not interfere with understanding what is said in the main paper. But here, all these notions are referred to in definitions and assumptions and this really makes some sections unreadable. Also, some of these notions are not standard at all, like t-separation (I'm familiar with d-seperation) or non-negative rank (I'm familiar with the standard notion of rank) and would benefit from explanations in the main text. \n\nIn addition, Algorithm 1, which is the main practical contribution, is described only in the appendix. Same thing for the adaptive sparsity selection mechanism for capturing concepts at different levels in diffusion models from Section 6.2. The literature review is in the appendix.\n\n**Diffusion models experiments**\n\nI appreciate the effort to include more realistic experiments in a theoretical paper, but here I felt like Sections 5 & 6 on diffusion models were disconnected from the rest of the paper… My understanding is that the authors do not apply Algorithm 1 developed so far to the diffusion model. It seems the point of these sections is to draw what I believe to be very vague connections between the assumptions of their hierarchical model and the hierarchical nature of diffusion models. Section 6 only shows that different noise level of the latent space of a diffusion model correspond to our intuitive sense of “abstract levels”. But AFAIK this is a well known observation, no? Section 6.2 introduces another algorithm with only very high-level explanations with details in appendix. \n\n**Suggestions for improvements**\n\nI believe this manuscript would be more suited for a journal like JMLR than for a conference. The additional space would allow the authors to present all definitions in the main text and give intuitions for their meaning (for instance, the definition of atomic cover is very dense and could benefit from more explanations and intuitions. The recursive nature of the definition makes it quite challenging to grasp IMO). This also avoid the endless back and forth between main text and appendix. \n\nAnother possibility would be to remove the section on latent diffusion models, but even then this might not be enough. \n\n**Relatively minor points:**\n\n- Line 130-132: I believe the estimators d_hat, c_hat, g_hat and \\Gamma_hat should be defined more explicitly, given how crucial they are to the results. In this phrasing, it is not clear whether these are estimated on a finite dataset or the full population.\n- Table 1 and 2 are not referred to in the main text.\n- Condition 3.1: The notion of splitting a latent variable is not properly explained.\n- Condition 3.3: By definition, the support of a random variable is closed. See for example: https://en.wikipedia.org/wiki/Support_(mathematics)#In_probability_and_measure_theory . The only subsets of Rn that are both open and closed are the empty set and Rn itself. I’m guessing the authors were hoping to include more sets in their theory. It might be possible by assuming the set is “regular closed”, which means it is equal to the closure of its interior. This was done in a similar setting in [65].\nInteresting to see that (iii) resembles the notion of G-preservation from [a] (see Definitions 11-12 and Proposition 3)\n- Be careful with phrasing like line 237 “We define t-separation in Definition A3.2” as it sounds a bit like the authors are introducing this notion, but it’s not the case (source is cited properly in appendix).\n- Confusion around t-separation: In Theorem 3.5, it is written “L t-separates A and B in G”, but the definition of t-sep refers to a tuple, i.e. “(L_1, L_2) t-separates A and B in G”. Not sure what the statement means. \n- Text is too small in Figure 3\n- Line 119: The definition of pure child was a bit confusing. In particular I thought B could contain more nodes than just the parents of A. Why not just repeat Definition A3.8 in the main text? (ne need to have a definition environment)\n- Typo on line 141, V_1 or v_1 ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing our theoretical contribution as “interesting, important, and novel” and we highly appreciate your detailed, constructive suggestions on the writing.\nIn light of your suggestion, we have put a lot of effort into revising the paper to explain all the involved definitions clearly, as we will detail below.\nWe hope that this contribution should be visible to the community sooner. Therefore, we’d like to follow the tradition of publishing the paper at NeurIPS.\nWe’d appreciate your feedback – thank you in advance!\n\n>W1: Writing suggestions.\n\nWe are grateful for your thoughtful suggestions. In light of your feedback, we have made the following modifications to improve the readability of Sec. 3.3:\n1. Trek-separation and non-negative ranks: we have moved definitions in the appendix to Sec. 3.3 (original lines 234) and explained their distinctions with d-separation and ranks.\n2. Minimal-graph operator: we have added references to Figure A1 (a)(b) in the main text and moved its definition to the original line 288. \n3. Condition A3.15 and the skeleton operator: we have moved Theorem 3.9 to the appendix and keep the reference in lines 300-30. Consequently, we keep Condition A3.15 and the skeleton operator at the current location in the appendix.\n4. Algorithm 1: as our theory only involves simple changes of the original algorithms in [20], we tentatively leave the algorithm at its original location. Instead, we have added the following line in line 299 to highlight the modifications “our modifications consist of changing the original rank to the non-negative rank over probability tables and setting discovered covers as discrete variables as mentioned above”.\n5. Adaptive sparsity: we have moved its description (original lines 1013-1015) to Sec. 6.2.\n6. Literature review: we feel that the introduction has covered key literature, so for now we leave it in the appendix and will move it to the main text if given one additional page for the final version.\n\nTo compensate for the space, we have moved large chunks of discussion in Sec. 5 to the appendix, along with Theorem 3.9 and its discussion. We’d like to note that all these edits are limited to Sec. 3.3 and Sec. 5 locally and the global structure of the paper remains.\n\nWe’d love to hear your feedback!\n\n>W2: “Diffusion model experiments.”\n\nThank you for the comment. In light of your comments, a portion of Sec. 5 has been moved to the appendix, while the main paper highlights its connection to our main results and findings. In our humble opinion, the hierarchical model is a valuable framework for reasoning about diffusion generation processes. We believe this could provide inspiration for the community to advance towards more controllable and interpretable generative models. \n\n>W3.1: Estimator definitions.\n\nMany thanks! We have added the following to line 132: “where we assume access to the full population $p(\\mathbf{x})$”.\n\n>W3.2: Table 1, 2 references,\n\nThanks for the reminder! We’ve added references in lines 314, and 318 respectively.\n\n>W3.3: notion of splitting variables.\n\nThanks a lot! We’ve replaced “splitting” as “(i.e., turning a latent variable $z_ {i}$ into $\\tilde{z} _{i,1}$ and $\\tilde{z} _{i,2}$ with identical neighbors and matched cardinalities $ |\\Omega^{z} _{i} | =  | \\tilde{\\Omega}^{z} _{i,1} | + | \\tilde{\\Omega}^{z} _{i,2} | $ ).”\n\n>W3.4: Random variable support.\n\nGreat point! We have modified “open” to “closed”, as this doesn’t affect our main proof.\n\n>W3.5: definition phrasing.\n\nThanks! We’ve edited it to “We introduce … [43].”.\n\n>W3.5: t-separation notations.\n\nThanks! We’ve edited it as “ a partition $(\\mathbf{L} _{1}, \\mathbf{L} _{2})$ t-separates…” in Theorem 3.5.\n\n>W3.6: small text.\n\nThanks for pointing it out! We have updated the fonts.\n\n>W3.7: pure children clarification.\n\nYou’re totally right! We’ve moved Definition A3.8 to line 119.\n\n>W3.8: typos.\n\nThanks! We have corrected it to $v_{1}$.\n\n>Q1: Condition 3.1.\n\nGreat question. We acknowledge that this condition may be strong and we wished to avoid it. Unfortunately, it seems technically necessary without additional assumptions. For instance, its necessity is discussed in [24] for one-layer mixture models and we believe this is also the case for hierarchical models. Since science is established step by step, we hope our results could serve as the basis for more relaxed conditions in the community.\n\nIn the distribution of dog images, suppose “head” has only two children “eyes” and “nose”. This condition means for each type of “head”, all combinations of shapes of “eyes” and “nose” should have non-zero probabilities. In reality, some combinations may be rare but can still appear at extremely small probabilities. That said, we agree that there are certainly cases where this can be violated, e.g., deterministic relations.\n\nYes, you’re totally right about $\\text{ne}(v)$. We’ve included “$\\text{ne}(v):= \\text{Pa}(v) \\cup \\text{Ch}(v)$” in line 119 - thanks!\n\n>Q2: Discussion on sparsity conditions.\n \nGreat suggestion! We’ve added the following discussion to the original line 184:\n“Condition 3.3-iii is related to the notation of sparsity in disentanglement literature. Brady et al. [b] divide the latent representation into blocks and assume no shared children among blocks, which can be stringent if one aims to identify fine blocks. Moran [c] assumes pure observed children for each discrete variable, which is strictly stronger than Condition 3.3-iii. The structural sparsity in Zheng et al. [d] implies Condition 3.3-iii. By contrapositive, if latent variable $z_{0}$’s children form a subset of a distinct variable $z_{1}$’s children, then we cannot find a subset of observed variables whose parent is $z_{0}$ alone.”\n\n>Q3: Definition 3.6 clarification.\n\nIt’s the collection of all states of variables in the cover. We’ve defined this notation in line 236 – thanks!\n\n---\n\nPlease let us know if you have further concerns and we are more than happy to discuss more!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q98nQFeaUM", "reviewer": "Reviewer_BR1s", "summary": "This paper studies the framework that identifies the discrete hierarchical latent variables for learning concepts from observed data examples. The proposed theory can be used to interpret the generating process of latent diffusion probabilistic models from the perspective of constructing object concepts.", "questions": "Please see the weakness.", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper is, in general, well-written and well-motivated and focuses on the difficult problem of capturing concepts in vision problems.\n\n 2. This work includes thorough theoretical derivation and details.\n\n3. The illustrations of synthetic data demonstrate the applicability of the proposed method, and the results look interesting.", "weaknesses": "This paper is nice to read, while I have only limited experience in such a causal hierarchical modelling area. My questions can be found below.\n\n1. From Sec. 2, given the description that the continuous latent variables c seem to control a lower level of features of data, while in Figure 1. a, it seems to be independent to concept factor d at the same level of the hierarchical structure. Could you elaborate on the relation and difference between c and d?\n\n2. For LD experiments, at the early steps of the diffusion process (i.e., T), Figure 5 presents controlling of bread and species (high-level) features, while the low-level (e.g., object angle, background) can remain the same. But, in Figure A8, such capability does not hold, especially for the shoes. Can the author explain the reason behind this?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and dedicating your valuable time to our work! We address your concerns as follows.\n\n\n>W1: “From Sec. 2, given the description that the continuous latent variables c seem to control a lower level of features of data, while in Figure 1. a, it seems to be independent to concept factor d at the same level of the hierarchical structure. Could you elaborate on the relation and difference between c and d?”\n\nSure, we are happy to elaborate! As you correctly observed in Figure 1.a, the continuous variable $\\mathbf{c}$ directly influences the observed variable (e.g., images) $\\mathbf{x}$ and is independent of the discrete variables $\\mathbf{d}$ (please also see Equation 1).\n\nIntuitively, $\\mathbf{d}$ contains all the discrete concepts/information in the image distribution, for instance, object classes or categories of shapes, whereas $\\mathbf{c}$ captures the information, such as illumination, angles. These two sources of information are complementary and together fully describe the image information.\n\nPlease note that the independence is not restrictive because even if $ \\mathbf{c} $ depends on $ \\mathbf{d} $ we can reduce it to the independent case by replacing $ \\mathbf{c} $ with its the exogenous variable, which is independent of $\\mathbf{d}$. \n\nPlease let us know if we have cleared your question – thank you!\n\n\n>W2: “For LD experiments, at the early steps of the diffusion process (i.e., T), Figure 5 presents controlling of bread and species (high-level) features, while the low-level (e.g., object angle, background) can remain the same. But, in Figure A8, such capability does not hold, especially for the shoes. Can the author explain the reason behind this?”\n\nThank you for the interesting question! We consider the object angle as a continuous variable (i.e., part of $\\mathbf{c}$ in Figure 1.a) due to its continuous nature, so it does not belong to the discrete hierarchical structure. We speculate the consistent camera angles in Figure 5 result from the high probability of this specific angle in the data distribution, since most of the close-up face photos are captured from this angle in the training dataset.\n\n---\nPlease let us know if you have remaining issues -- we are more than happy to engage!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their efforts and helpful comments regarding our paper. We are encouraged that all reviewers appreciate our theoretical contribution to the identifiability of latent discrete hierarchical graphs and our novel formalization of concept learning as a latent causal discovery problem. These contributions provide an intriguing perspective on latent diffusion models and have the potential to enhance the understanding and improvement of these models. Reviewer BR1s also highlights that the paper is well-written and enjoyable to read, while Reviewer ae89 values the provided examples that aid in comprehending the complex concepts.
 
Below, we give a summary of the responses:
* To Reviewer ZL2x: we’ve included additional experiments to extract concept structures from Stable Diffusion and attached the PDF file here. We’ve also included an additional literature review as suggested by the reviewer and clarification/interpretation of our theoretical conditions.
* To Reviewer 7auw: we’ve responded to our theoretical conditions, rank tests, graphical structures, and robustness. We’ve shared the potential usage of our work to advance the field of diffusion models.
* To Reviewer ae89: we’ve re-organized Sec. 3.3 and Sec. 5 to improve the readability of our work. We’ve corrected typos, added the references of Tables 1 and 2, and included a discussion of sparsity conditions.
* To Reviewer BR1s: we’ve elaborated on the formulation and experimental results.

Please see our detailed responses to each reviewer. We hope our revisions and explanations satisfactorily address the concerns raised. Once again, thank you for generously contributing your time and expertise to the community.

Meta-review: Scores 5 5 6 7

Great paper.  Well written and includes meaningful theoretical derivations and good experiments on synthetic data.

The reviewers were quite positive about the paper, some even enthusiastic.  One criticism was (reviewer ae89) that too much content is contained in the appendix.  However, in the rebuttal the authors pointed out how they restructured the paper, they added further experiments and added some discussion about how this relates to diffusion models.

So, this paper is a clear accept for me.
Decision: Accept (poster)