Can Diffusion Models Disentangle? A Theoretical Perspective
Authors: Liming Wang
Muhammad Jehanzeb Mirza
Yishu Gong
Yuan Gong
Jiaqi Zhang
Brian H. Tracey
Katerina Placek
Marco Vilela
James R. Glass
Venue: NeurIPS 2025 poster ()
Keywords: diffusion models
disentanglement
learning theory
Primary area: theory

Abstract: This paper presents a novel theoretical framework for understanding how diffusion models can learn disentangled representations with commonly used weak supervision such as partial labels and multiple views. Within this framework, we establish identifiability conditions for diffusion models to disentangle latent variable models with \emph{stochastic}, \emph{non-invertible} mixing processes. We also prove \emph{finite-sample global convergence} for diffusion models to disentangle independent subspace models. To validate our theory, we conduct extensive disentanglement experiments on subspace recovery in latent subspace Gaussian mixture models, image colorization, denoising, and voice conversion for speech classification. Our experiments show that training strategies inspired by our theory, such as style guidance regularization, consistently enhance disentanglement performance.

Can Diffusion Models Disentangle? A Theoretical Perspective
Liming Wang'
Muhammad Jehanzeb Mirza
Yishu Gong?
Yuan Gong
Jiaqi Zhang!
Brian H. Tracey?
Katerina Placek2
Marco Vilela?
James R. Glass
'aassahuueetts Institute Technology, "Takeda aeaiing@csail.mit.o.
Abstract
This paper presents a novel theoretical framework for understanding how diffusion models can learn disentangled representations with commonly used weak supervision such partial labels and multiple views. Within this framework, we establish identifiability conditions for diffusion models to disentangle latent variable models with stochastic, non-invertible mixing processes. We also prove fmite-sample global convergence for diffusion models to disentangle independent subspace models. To validate our theory, conduct extensive disentanglement experiments on subspace recovery latent subspace Gaussian mixture models, image colorization, denoising, and voice conversion for speech classification. Our experiments show that training strategies inspired by our theory, such as style guidance regularization, consistently enhance disentanglement performance.
Introduction
Extracting hidden structure from raw sensory data fundamental to progress multimodal perception [1-5], scientific discovery [6-15], Al-assisted content creation [16-20], and many more. Autonomous vehicles must localize objects and auditory events while suppressing background noise to navigate safely in open- world conditions, and data-driven drug-discovery systems need to group and recombine functionally related chemical components to propose therapies for emerging diseases. Deep learning-based creative tools likewise hinge isolating user-specified factors (e.g., speaker style or lighting) while leaving other aspects untouched.
Many latent factors are disentangled in the sense that they vary independently. For example, speech content persists regardless of the speaker, and object shape remains consistent under different lighting conditions. This intuition inspires breakthroughs in linear [21] and non-linear [22] independent component analysis (ICA), modern deep learning- based disentanglement [23, 24] and causal representation learning [25]. However, impossibility result shows that fully unsupervised disentanglement unattainable in general [26]. Recent work on disentanglement therefore relies on weak labels [27, 28] multi-view supervision [29, 30].
Applications such video editing and drug discovery often require both latent factor extraction and controlled synthesis of novel samples from the latent factors. Diffusion models (DMs) [31-33] based on learning score functions [34] of probability distributions, excel at generation and power state-of-the-art
Work done at MIT. now with XAI
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

editors and simulators [16, 17, 19]. However, standard DMs learn only the data marginal, encoding latent factors implicitly. To control data generation using the latent structure, conditional DM (CDM) inject side information into the score function [35-37] and achieve impressive empirical success in disentanglement tasks such as voice conversion and image editing [38, 39, 20, 35, 36]. Yet principled understanding of when these models learn identifiably disentangled representations is still missing.
In our work, we broaden the disentanglement theory diffusion models and provide the first tearning-theoretic framework for DM-based disentanglement, which poses unique challenges. First, a sample from DM is generated by integrating stochastic differential equation (SDE), so the generated sample becomes a stochastic. non-in ien-ineertiblee mapping of the latent variables. The lack of an analytic inverse precludes the saaunne-of-varabble calculus and ICA-style arguments that underpin classical disentanglement theory [26, 40]. Further, the extra uncertainty injected by the stochastic drift can make it more difficult leverage the commonly used weak supervision. We tackle these challenges by (i) proposing notions of approximate disentanglement applicable to stochastic, non-invertible settings, (ii) marrying cnormatiion-eeulariise score matching with recent finite-sample analyses of score-based models [41]. The resulting framework lays the groundwork for the following contributions:
1. We show that, under mild Lipschitz assumptions, DMS recover approximately disentangled representations of two latent factors (e.g., content and style), when given either partial supervision or multi-view inputs. For independent subspace models, we further prove global convergence in the finite-sample regime using gradient descent.
2. Building on the theory, we introduce a novel style-guided score matching loss that attains a global optimum for the ondependent-subspace case and improves disentanglement in practice.
3. Extensive experiments on Gaussian mixture models, image editing, and voice conversion for speech classification demonstrate that our disentanglement quality and downstream classification accuracy.
training strategies consistently enhance
The rest of this paper is organized as follows: In Section 2, we provide the background on diffusion mod- els. Section formalizes the problem of content-style disentanglement, and later in Section we present the main theoretical results and Section 5 details empirical evaluations on synthetic, image and speech data, demonstrating and supporting the theoretical findings. Finally, Section 7 concludes the paper.
2 Background: diffusion models
Diffusion models (DMs) [31-33] approximate the pdf px =: Po of an r.v. N~ Po via a two-stage process: noising and denoising. In the noising stage, data progressively corrupted using an SDE:
dX, (X,
Xo ~po
(1)
where B₁ is a Brownian motion. We adopt the choices ((X,,t):== == - X and E(t)= = 22, leading to the Ornstein -hleebbcck (OU) process:
dX, dX,=-X,++V2dB, Xo~Po-
(2)
which converges to a standard Gaussian [42, 41]. Let denote the marginal of X, and Pts. the conditional pdf of X, given X,. In the denoising stage, the goal is to recover X from noisy versions X,,12 to by simulating the time-reversed process X;= X
dXx== = [X _ + 2V 2 d[t (dL+ 220B ~P-
which converges back to Po [43]. Since the score function Va Volog (X.) is unknown, DM leams score estimator 8y: R x [0,7] - dx by minimizing the score matching objective:
= :(0)==E,,pllla (X V (X.))
This objective is equivalent to conditional score matching objective involving Pejo:
N₂ (t)
Lo() (x), -
V=llopP:j po
so (x,tt)++
(4)
where N, is standard Gaussian and (tt) := 11-exp(-222 During inference, new samples generated by simulating an estimated SDE, with 85 ~pr and discretized time steps E [ky) (k+)))))
dX;"" = [X;; + 22g T V2d,.
(5)

---

3 Content-style disentanglement
The task of content-style disentanglement can be formalized through latent variable model (LVM). For clarity, we refer to all random entities scalars, vectors, or matrices as random variables (r.v.'s) and focus on the continuous case. For any r.v. X. let pxx denote its probability density function (pdf). We assume two latent factors: the content Z pz taking values in R and the style G GC~GG PG taking values in Rdd, which jointly generate an observable sample X Rddx through the noisy, non-invertible and nonlinear transformation with an invertible mixing function f: f:R4z x R de + Rdx:
(6)
where ō is the noise level and N is independent, standard Gaussian noise. We assume that Z and G are statistically independent, as is common in disentanglement [24, 44, 26] and ICA literature [21, 40J.
Although exact independence between Z and G' is assumed for clarity, our framework naturally extends to settings where independence holds only approximately, as will be discussed subsequently.
This partition of latent factors arises in several settings. In controllable generation, Z encodes persistent attributes (c.g., object identity) while G governs editable factors (c.g., pose or lighting). In seeervuueeei learning (SSL), Z captures invariant content across views modalities, and G captures modality- or augmentation-speeifee variations. In the low-noise regime 5 -> O, our goal is to recover the latent variables Z and G from observations of X. A common notion of recovery is block identifiability [45-48, 29, 30], which ensures that subgroups of scalar latent variables can be recovered up an invertible transformation. However, exact block identifiability is not achievable in the presence of noise 8 0 due to the non-invertibility of the mixing process, motivating a need for approximate disentanglement To this end, propose two complementary criteria: (1) approximate information- -theoretic disentanglement and (2) editability.
(c,v) disenttnnnan. To quantify how well the learned content and style representations are separated and informative, we define an information theoretic notion of approximate disentanglement.
Definition 3.1 disenntneinnn,,lllmentt) Let (2,G) be content and style encodings inferred from an obeeree X. They are (is))ddisettnngler if. for some c,v20, 0, (i) I I(2;G) S (ii) I (2) I[Z,G:)) where (A:B) denotes the mutual information (MI) between r.v's and B.
These conditions ensure that the learned latent factors are (i) nearly independent (as the true and are), and (ii) retain most of the information about the observed data X. The definition remains meaningful even as 8 5 and I ((Z,G:X) -> oc, since it based on a bounded difference in mutual information.
e-editability. many applications, it desirable to modify style while preserving content. This motivates the following notion of editability based conditional sample generation.
Definition 3.2. (c-editability) Let (2, G) be encodings inferred from X. and let &' ~ ~PG be an independent style encoding. (2, G) are e-editable if there exists a generative model q(: 2, G") such that the generated sample X q(- 2,G') satisfies Ezzpzzdy (Px12 Px12) S e, where == + eN is smoothed version of X, and dry is the total variation distance.
This definition captures the ability to recombine content and style encodings to generate new samples that are consistent with the original content. For example, a facial image editing task, the encoding 7 may represent identity while G captures facial expression. By swapping & with a new expression encoding G' we can generate a new image that preserves identity but alters the expression. Deterministic decoders are allowed as a special case with g(- =,6)=5f(2.c; for some function f.
The notions of (c,v) diiiedddiintnnlemmen and e'-editability are complimentary but not equivalent. In particular, disentanglemen geetmantneeeentttne in the mutual information sense does not guarantee editability. The following example illustrates this by constructing encodings that are perfectly disentangled (i.e., (0, 0) diseenntaagl) yet fail the editability criterion due to a hidden ambiguity introduced during recombination. The proof is provided in Appendix A.
3

---

Example 3.1. Suppose content Z~ (~,(0011) and G (0,1) are independent standard Gaussian rvv's and consider the noiseless setting with ö = 0. Further, suppose Pf(z,G] # Pf(- (f() Then we can choose the content/style encodings to be Z Zsgn(G), G = G and the decoder where sgn(æ) denotes the sign of z. Then (2,G) are (O.0)-disentangled but not c-editable for some c>0.
((2,())=((
Intuitively, the encoder flips the sign of the content variable depending on the style. This transformation preserves both independence and informativeness, but causes ambiguity when recombining 2 with a new style G¹, since the decoder cannot distinguish whether the original sign should be restored. As a result, samples generated from new combinations do not match the original content distribution.
Weakly supervised disentanglement. When neither Z nor G is observed, disentanglement is in general impossible for nonlinear mixing functions: the observable pdf px alone does not identify whether the data is generated from disentangled entangled LVM [26]. We consider two practical settings where side information is available to help resolve this ambiguity.
Definition 3.3 (Content disentanglement). Assume Eq. 6, and suppose known style function g: Rdxx + Rdc is given such that (z(()) = for all z. The goal of content disentanglement is to learn encodings (2.G) from X that are ( -disentangled and c'eddiable.
This setting appears in applications such as image editing and voice conversion [49, 38, 50]. For example, in voice conversion, ( is a speaker embedding extracted from pre-trained speaker recognition model (e.g. speaker embeddings from ECAPA-TINN, which achieves almost perfect speaker verification error rate [5 This assumption mirrors common practice in the voice conversion literature and is not unique to framework. In image editing, g(-) could represent learned text embeddings of editing instructions.
Definition 3.4 (Multi-view disentanglement). Assume there are multiple views X X²,.,, "".".." X"v, where each X" is generated as X = (1,,'") Sisi,,iisi Sisnv, with i.i.d view-specific styles G 's. ii.d standard Gaussian noise 's view-specific noise levely 8, 's and invertible view-specific mixing functions fi:Idd,, x RRdz, + RRxx, Then the task of multi-view disentanglement is to learn encodings (2, (2,G") for all such that (Ž,G) ()) are (e.V)-disentagller and é'eddiabbl.
This setting is prevalent in (e.g. [52-58].. When J.=fi f1 (e.g. multiple camera views), each X may correspond to a different augmentation. When f. differ (c.g., across sensory modalities), the views may represent distinct but semantically aligned representations. For clarity, we focus the unimodal two-view case (ny = 2), which readily generalizes to multimodal scenarios with ny 22 To facilitate our theoretical analysis of DM- 3M--dsed disentanglement, which are common in the analysis of DMs [59, 42, 41].
adopt the following mild assumptions,
Assumption 3.5. The sample X is sub-gaussian with second moment xx Assumption 3.6. The score function of the sample pdf px is X
Assumption 3.5 ensures that sample values do not exhibit heavy tails, which could destabilize the diffusion process. Assumption 3.6 ensures that the score function does not change too abruptly, preventing discontinuities that could hinder accurate recovery of content and style during denoising.
4
Theory: diffusion model-based disentanglement
In this section, first present our theoretical results for DM-based disentanglement, and then discuss multi-view disentanglement, and finally provide results for disentanglement of independent subspaces.
4.1 Content disentanglement with diffusion models
This section analyzes the ability of DMs to achieve approximate disentanglement in the content disentanglement task defined in Definition 3.3. First, we introduce a conditional DM trained with regularized score matching objective tailored for this task. Second, we show that this model can learn content and style encodings (2, G)) that are (e,v) ddisentangee with arbitrarily small e,v as the noise level 8 + Finally, we discuss why such encodings may still fail to be é' editable for some e' >0 0. Formal proofs are provided in Appendix B.
Overview.

---

G
9
G
X
9
G
X
XT
80
g+
$0
X L Kn
Z
X
80
L
S0
$1 KT
G
i
G²
X²
2 2
2
(a) Content disentanglement
(b) Multi-view disentanglement (ny myy==2 2)
Figure 1: Graphical model of diffusion- disentanglement under different types of weak supervision. Shaded nodes denote latent variables; clear nodes denote observed variables. Learnable components are marked in red and orange, while the frozen component is shown in gray. Leftt: Content disentanglement. The style encoder g(-) is known and fixed. The model learns content encoder 2 and a score estimator Sp to estimate the score of Px 2, c. thereby disentangle content Z from style.
Right: Multi-view disentanglement. Given paired views xx²x sharing content Z, the model learns to recover Z and style G' of view 1 by estimating the score of Px|22, G using
to): The model architecture is illustrated in Figure 1a. Style is represented by fixed encoder where 9(-) is assumed known. Content is learned through trainable encoder yielding 2 = (Xto). To enable controllable generation, both 2 and G are fed as inputs conditional score estimator so IR x TZ x Rd x [0,7] Rx which estimates the score of the conditional pdf | Ž, We train this model using the following regularized score matching objective: hyperparameters 7,P 0:
No (tt)
L. (0,6)) = PoP Po
so X,, (x,2,6,1)+
(X))
Content information regularizer
Conditional score matching loss
where max (r.0). The score matching loss encourages the model to match the conditional score, while the regularizer limits the information that the content encoder can extract from the input, preventing overfitting via direct copying. In practice, mutual information is often approximated using tractable variational bounds [60-62], which serve as surrogates for the content information regularizer. Using the above steup, we prove the following theorem.
=
Theorem 4.1. Suppose Assumption 3.5-3.6 hold, and (0" .6²) be minimizer of L. defined in Eq." 7 with p= ((Z + 0, C2/T T for some C1,C > 0. Set tu = -lg (0) and 2 == zΦ- (X). vdx the encodings (2.G) = (2g- (Xi,) (()))(((,,))) are dissendisett with c- c o
for any 5 < min 1
2 for some constants C3.C,>0 >.
Intuition. Theorem 4.1 shows that under sub- gabbsaausiinn tail assumptions and Lipschitz- scores, DM can achieve (aaz)-disentanglemeed with (e.r) + as the noise level - 0. The rate at which disentanglement improves depends inversely on the Lipschitz constant - higher sensitivity in the score function slows disentanglement by amplifying noise- induced variations. The regularized objective mitigates this trading off between predictive power and the amount of content information retained in the content encoding 2.
Theorem 4.1 extends to cases where content and style are only approximately independent, i.e., perturbation (Appendix B. Assump-
I(Z;G) < for some 6 0, by treating the dependency tion 3.5 can also be relaxed to bounded variances, as the proof relies on moment control. However, it is important note that approximate disentanglement the MI sense does imply editability. As illus- trated in Example 1, the model may still leak style information into the content encoding. This limita- tion, which we refer to as content distortion, prevents achieving vanishing e' eeditability even 0.
4.2 Multi-view disentanglement with diffusion models
This section analyzes the ability of DMs to achieve editability in the multi-view disentanglement setting defined Definition 3.4. We introduce DM trained with a modified score matching objective, analogous to the content disentanglement case. We then show that the learned
Overview.
5

---

encodings (2.G) are e-editable with c - as 6 6 Finally, we discuss how the result generalizes to more than two views and non i.idd styles, and what it reveals about the role of different types of weak supervision. Full proofs are in Appendix C.
The model architecture is illustrated in Figure 1b. We define the encoders Z := Zóz (Xi,) and Ioa (Xix) each view The score estimator sy (Xe,22, 2,, .t) is trained with the following loss:
N2 (tt)
|
(0,0)) Es pop% oo1io so (x)
+1++
(8)
Style information regularizer
Conditional score matching loss
where o (0 The regularizer encourages G'1 to encode only the style of X At inference time, we sample from the estimated reverse process Eq. 5 but use the conditional score estimator to guide generation. Under this setup, We prove the following theorem.
Theorem 4.2. Under Assumption 3.5-3.6 and additional regularity conditions (Assumption C.2-C.4). Consider the two-view, unimodal setting with 81 =: 8 and 62 0. Let (0" (00000/)00000 +z) be minimizer of G 9 (X') for each view i. Set to log(11 82)1/2 and the diffusion
Lm and Z' = 202 6² step size nj qs=0s xg. for some constant Cş >> Then for some constants C,,00>00 (0) ((2
xe (ii) the content encoding 22 and the style encoding &² are Crv x
Intuition. Theorem 4.2 establishes that in the multi-view setting, DMs can achieve (c. v)- disentanglement and e-editability with vanishingly small e, even though v camnot be made arbitrarily small. In other words, perfect separation of content and style is theoretically achievable in terms of independence and editability, even some information loss is unavoidable. The editability error depends on the Lipschitz constant of the decoder, the intrinsic data variance, and the latent dimension- ality-factors that all amplify sensitivity to noise. To mitigate this, we introduce a style information regularizer, which suppresses content leakage into the style encoder. This prevents entanglement and enables reliable "mix-and-match" generation. Without this regularization, residual content in the style embedding can undermine both disentanglement and editing performance.
While Theorem 4.2 is stated for i.d. styles across views, readily extends nearly independent styles, i... those satisfying I (G' ;G2 Z) e, for some small E, (see Appendix C.4). Our framework also handles more than two views and heterogeneous view-specific mixing functions. A key challenge arises when content and style are highly correlated or when the sample size is small. In these settings, noise can break delicate statistical dependencies, leading to spurious entanglement, especially under limited data, which can mask the true latent structure further due to overfitting. These issues are not covered by prior identifiability results [48, 29], which assume noiseless generation and invertible mixing functions.
4.3 Disentanglement of independent subspaces with diffusion models
Overview. This section considers special case of content disentanglement- when the underlying distribution follows an independent subspace model (ISM) [45-47, 63]. ISMs have served foundation in classical disentanglement literature and are also sup- ported empirically in recent self- self-supervieed learning studies [64]. with applications in both discriminative and generative disentanglement tasks [65, 66, 39, 50, 67]. Under the ISM setting, we prove that a carefully constructed DM trained with gradient descent can simultancously achieve (e=))ddieettanglmmen and e'- editability even in the finite-sample setting - offering stronger theoretical guarantees than those in the general content disentanglement case (Section 4. We also outline how these results can be extended to the multi-view case in the discussion. Full proofs provided in Appendix D. To begin, we formally define an ISM below.
X
U
PP(())
58
G.
V
Figure 2: Dual-encoder score network for content-style disentanglement of ISM
Definition 4.3. An independent subspace model (ISM) is an LVM defined by
Z pz, G G~Pa
Pa: X Az Z AcG.
(9)

---

where Az E &Rdxxdz Ag E Rdxxdaa xdc are orthogonal matrices such that their column spaces are orthogonal and span Ráx, That is, R(Az)' =R(Aa) with dy dz+da=dyy where R(A) is the column spuce of matrix A. Let X, denote the noisy version of X at time L in the diffusion process. Then we define Z4 : A
G. (
This model generalizes the setting in [41], which assumes Gaussian noise for G,. A key property of ISMs is that the score function of the marginal pt (X)) decomposable as shown below.
Lemma 4.4. For any t 220, 0, the score of the pdf p. (X,) under the ISM satisfies
5" (' = V loogp (X,) = Az loogp, (=(X.)) Ag yylogro, (g(X)))-
Lemma 4.4 shows that for ISM, the score function is a linear combination of the score functions of content and style. Motivated by Lemma 4.4, we propose a dual encoder network for learning 8 (xx(), as shown in Figure 2:
so (=zz,t)==Us33 82 (z,t)+ (x,t)+ + o ( (WN()))))))))((
(10)
where N-(-) denotes two-layer ReLU neural net and PE(() is a time position encoding. The first branch of the dual network computes the content score 8z X2,() of the noisy sample X,, while the second branch computes the style score sG (G,,)) from the noisy style G,. The two scores are combined to produce the final score function sp( (X,,)). Unlike earlier sections, s8 here is unconditional. To train this network, we use a regularized score matching loss:
LA (0) -21E, pp (at) (z,t) (1))sss0 (2, BA, Be (c( (2,0)
(0) (II) 2
Lo score matching loss
Loont style guidance regulariier
Balancing loss
where A, > 0 is the style guidance weight and oi is the empirical pdf. The style guidance regularizer encourages style separation by forcing the score network to explain part of the data using only style, thereby implicitly discouraging it from encoding style information redundantly in the content branch; The balancing loss, defined in Appendix D, helps prevent poor local minima. For general latent variable models, this is a heuristic regularization rather than formal MI minimization, and we will add a clearer explanation and acknowledge. The following theorem analyzes training dynamics by studying the gradient flow:
(12) Theorem 4.5. Under Assumption 3.5-3.6.forr some to dependent n. and let min (dr cc, and let the positional encoding PE(.) be bounded and linearly independent over E [lo. Define PM to be the projection matrix onto R R(M), and o; (s()) be the i-th largest singular value of the operator Then for some Xy, Eq. 12 converges to critical point 0 = 10z A,. .V] sush that with probability at
dx 24 20g2/
least 1 - ( 1) Encodings 7 := Pū X and ===g(X) are 0
,0 -disentangled; 2)
Odz (65 (0b))
Let 2= = (o)) Then (2,,0) are O Tmin (o
2 2/10
editablle.
Jat/is
Intuition. Theorem 4.5 shows that for ISMs, a DM trained with gradient descent can recover the content and style subspaces as the number of samples n- co. co. This enables both approximate (..")-disentanglemenn and e'-editaillity. A key component is the style guidance regularizer, which encourages the model to separate content and style by encouraging the model to use the style information during score matching. Importantly, the speed and quality of subspace recovery depend on the strength of the signal in the score function. In particular, recovery is faster when the content and style score functions have larger minimum singular value. This highlights practical consideration: well-separated or high-contrast latent factors lead to more reliable disentanglement, while near- degenerate cases may require additional regularization or supervision.
Theorem 4.5 extends to the multi-view setting using the dual encoder network 89.0 (X) f) (xz (x((X,)) where 8z extracts content from the second view. In this case, we use a content guidance loss L applied to (U,sz) instead of (V,sg), to encourage reliance on content information from the second view and suppress residual content in the style encoder sg- Our bound
depends on the data dimension dx, but extensions to lower-dimensional latent representations are
7

---

7
2
O 6
7
2
/
O
4
Y
5
7
0
0
9
O
60
9
0
7
7
6
9
7
8
7
9
7
3
4
7
9
0
0
7
4
0
3
7
3
7
2
7
2
7
7
4
5
6
4
7
(a) Input
(b) Baseline (A,
Colorized (A,
(d) Reference
(e) Input
(f) Baseline (A,-
De-noised =3) (h) Reference
Figure 3: Disentanglement results MNIST and CIFAR-10. Top: Disentanglement results on
MNIST. The content is the gray-scale digit image, and the style is the background color. Bottom:
Disentanglement results on CIFARIO. The content is the clean image and the style is the corruption on the image. Both 3b-3c and 3f-3g suggest disentanglement is achieved with the style guidance loss.
Table 1: Quantitative results for image denoising and face editing on CelebA test sets for various models. Conditional (cond.) InfoGAN and cond. 3- 8-VAE stand for InfoGAN [69] and 8 8-VAE [70] adapted to the content disentanglement setting by conditioning on the observed style variable. DDPM stands for denoising diffusion probabilistic model (DDPM) [33].
Denoising
Face Editing
LPIPS(
SSIM(1)
CLIP Score(t) LPIPS(L)
cond. InfoGAN
0.83
0.16
0.24
0.39
cond. 8-VAE
0.57
0.17
0.25
0.37
DDPM (ours), A,
0.53
0.18
0,25
0.35
DDPM (ours), A, 0.03
0.47
0.19
0.26
0.30
DDPM (ours), A, 0.3
0.35
0.23
0.26
0.32
DDPM (ours), A, 3
0.33
0.24
0.26
0.38
possible via residual connections in the score network in Figure 2. While our focus is on unconditional score matching, our results extend conditional settings with appropriate corrections for cross terms. Lastly, although our analysis assumes infinite-wide width two-layer ReLU networks, similar convergence behavior may hold for deeper or finite-widis networks [68].
5
Experiments
This section presents empirical evaluation of our theoretical framework by testing whether DMs can achieve approximate disentanglement under settings in Section 4.1.4.3. We begin with synthetic datasets generated from Gaussian mixture models (GMM), which instantiate the ISM framework from Section 4.3 validate guarantees in Theorem 4.5. We then move more realistic settings using standard image datasets. Specifically, we apply our DM-based disentanglement method to two tasks: image colorization
MNIST [71] and image denoising CIFAR10 [72]. Finally, validate Theorem 4. 1-4.2 on real-world speech task, voice conversion (VC) adaptation, by adopting the DM-based VC framework from [38].
Implementation details. For the GMM dataset, we use a two-layer ReLU network consistent with Theorem 4.5. For MNIST and CIFAR, design [74]. We optimize regularized score-matching objective, inspired by Eq. 11:
U-Net [73] architecture, following common DM
N₂ ((t)
N₂
LA (0,0) E. Et.
so 2.
X,
so(X G.t)
(13)
(tt)
where X, controls the style guidance weight. Note that we omit the balancing term for empirical simplicity, and style guidance applied by zeroing out content, equivalent to the style regularizer in Eq. 11 when the input data has ISM structure.

---

GMM disentanglement results. First, we conduct subspace recovery experi- ments on latent subspace GMMs (L.S-
0.25
Scheduler
Subspace Recovery Erd
0.35
Suespace Recovery
0.30
=B- ve
vp
GMM), class of L.SMs where each sub- space follows GMM. Figure 4 shows the subspace recovery error as a function of the style guidance weight X, and the sample size n. Consistent with the pre- dictions of Theorem 4.5, the LSGMM achieves the smallest subspace recon- struction error when the style guidance weight is sufficiently large, and the re- sult is consistent across different noise schedulers. Moreover, since all score net- works wide, two-layer MLPs trained using gradient-based methods, these re- sults provide further support for Theo- rem 4.5. Figure also reveals sublinear decay rate of the subspace recovery error as the sample size increases, aligning with Theorem 4.5.
subvp cosine
025
subvp
0.15
mm+
cosine
0.10
0.15
0.10
0.05
0.05
10-1
10" 10-2
10a
30
120
480
1920
Sample Size
Figure GMM disentanglement results with the score estimator in Eq. 10. The subspace recovery eπrоr (de- fined in Appendix E. normalized between [0,1]. In four random trials, DM consistently recovers (error < 0.1) the correct content subspace and achieves disentanglement with sufficiently large style guidance weight X, and sample size.
Image disentanglement results, Next, we validate our theoretical findings on image data from MNIST and CIFAR-10. For MNIST, we perform colorization, treating digit shape as content and background color as style. The results are visualized in Figure 3. Without regularization, the model fails to disentangle these factors, simply copying the input (Figure 3b). With the proposed regularizer, it successfully disentangles color from shape (Figure 3c). On CIFAR10, 10, we test image denoising where the content is the clean image and the style independent noise. In this setup, we introduce random color shift as the noise, though our approach can, in principle, be extended to other types of independent noise. The results show similar improvements with regularization, consistent with Theorem 4.5, We have further verified our theory on two more disentanglement tasks on the CelebA image dataset. First, we perform denoising with the same setting as CIFAR 10 and compared our approach with two standard baselines, infoGAN [69] and beta- VAE [70] adapted to the content disentanglement setting by conditioning on the observed style variable. As shown in the first two columns of Table we observe the same trend as our previous experiments. Further, we perform face editing with gender as the style variable. As shown in the last two columns of Table 1, our method is superior to the conditional GAN baseline in terms of style transfer (CLIP score) and content preservation (LPIPS [75] score). Additional examples and quantitative results using different regularization weights for multiple metrics provided in Appendix E.2. Speech disentanglement results. Lastly, we apply DM-based disentanglement to a real- world application: voice conversion adapta- tion (VCA). In this setup, the style corresponds to speaker identity, while the content captures attributes like emotional state or health con- dition. Our goal is to learn representations that disentangle these factors, enabling robust classification under speaker shifts. We use the IEMOCAP dataset [76] as a testbed, where generaliza- tion across unseen speakers serves a proxy for successful disentanglement. This task in- stantiates multi-view disentanglement (Defi- nition 3.4): different speakers provide style views, and the shared emotion across record- ings is the content. We are also actively de- veloping additional vision-domain multi-view experiments.
KNNVC No VC
TRAANVC
DiffVC
Pitch Shift
0.9
Accuracy
0.8
speech emotion recognition
0.7
0.6
4
8
10
Number of Speakers
Figure Emotion recognition results on IEMO- CAP a probing task for speech disentanglement. DM-based disentanglement between emotion (con- tent) and speaker (style) outperforms other methods. Data augmentation using multiple speakers further improves disentanglement.
As shown in Figure 5, our DM-based approach achieves higher classification accuracy than

---

several baselines, including no conversion, pitch shifting, and recent VCA models [65, 77]. Perfor- mance improves as more target speakers are used during training, consistent with our theoretical prediction of Theorem 4.1 that DMs can indeed achieve approximate disentanglement for speech data. also validates Theorem 4.2 by demonstrating that multi-style data enhances content-style disentan- glement. Further, more speech metrics and additional experiments beyond emotion classification (e.g.. Alzheimer detection [78], Amyotrophic Lateral Sclerosis severity [79]) are detailed in Appendix E.3. Further, we also note that our current experiments focus on controlled settings aligned with the theory. Extensions to large-scale experimentation are left as an exciting future avenue of research.
Related works
Diffusion model theory. Early theoretical works on DMs analyzed their ability to learn data distribu- tions, under different assumptions like log-Sobelev inequality [59], bounded moments [80, 42] and score approximation in - [81] and 1,² [59, 42] norms. Later works compared DMs with likelihood- based models [82], and studied their capability to recover Gaussian mixtures [83-86], Ising models [87], low-dimensional subspaces [41, 66, 88, 89] and manifold structures [90]. Recently, [91, 92] analyzed the convergence of conditional DMs and the role of classifier-fre guidance. Others have analyzed the training [93-95] and sampling [95-97] dynamics of DMs.
Disentangled representation learning. Disentanglement is defined via factorized representa- tion [98, 21-23], group equivariance [99] or approximate independence [100]. It underpins many advances in self-supervised learning [52, 54, 101, 101,56], multimodal learning [102-104] and controllable generation [49, 38, 35, 36, 105]. Theoretically, disentanglement traces back independent component analysis (ICA) [106]. later extended to correlated factors [48, 29] and studied through the lens of mod- emn SSL techniques such as data augmentation, contrastive loss [107] and self-distillation [108]. A key result [26] shows that unsupervised disentanglement is impossible without additional inductive biases, leading to weakly supervised methods using multiple views [29, 30], auxiliary labels [27, 109, 28], temporal cues [110, 111] and isometric constraints [112]. Other works analyze the disentangling capacity of (variational) auto-encoders [49, 113-116, 100, 117, 118].
7 Conclusion
We presented the first seemiing-theoretii framework for DM-based disentanglement, addressing un- solved challenges of prior works that focused primarily on deterministic, invertible mixing processes. Our framework introduces two notions of approximate disentanglement that generalize classical for- mulations stochastic, non-invertible settings and shows how DMs can achieve them with partial labels or multi-view inputs. Moreover, in the special case of ISMs, we derive stronger guarantees, including finite-sample global convergence for gradient-based training. Our experiments across several domains, spanning Gaussian mixture recovery, image colorization, denoising, and voice conversion, show that -ooided training methods, such soyle-guidane regularization, lead to improved dis- entanglement and better downstream performance. Our framework lays the foundation for principled disentanglement with DMs and future works more complex latent structures and modalities.
Acknowledgments and Disclosure of Funding
This research was supported by Takeda Development Center Americas, INC. (successor in interest to Millennium Pharmaceuticals, INC.) This research could not be possible without the immense contributions from people living with ALS who have participated for months and years in the ALS Research Collaborative Study. We thank the ALS patient and caregiver community, in particular Augie's Quest for Cure, for financially supporting ALS TDI's data collection work.
References
[1] Y. A. Farha, A. Richard, and J. Gall, "When will you do what? anticipating temporal occurrences of aaativvities" in CVPR, pp. 5343-5352, 2018.
[2] Harwath, A. Recasens, D. Surís, G. Chuang, A. Torralba, and J. Glass, "Jointly discovering visual objects and spoken words from raw sensory input," in ECCV, pp. 649-665, 2018.
10

---

[3] H. Zhao, C. Gan, A. Rouditchenko, C. Vondrick, J. McDermott, and A. Torralba, "The sound of pixels," " ECCV, September 2018.
[4] M. Hamilton, A. Zisserman, J. R. Hershey, and W. T. Freeman, "Separating the "chirp" from the "chat": Supe-supe visual grounding of sound and language," in CVPR, pp. 13117-13127, 2024.
[5] E. Araujo, A. Rouditchenko, Y. Gong. S. Bhati, S. Thomas, B. Kingsbury, Karlinsky, R. Feris, J. R. Glass, and H. Kuehne, "CAV-MAE Sync: Improving Contrastive Audio-Visua Mask Autoencoders via Fine-Grained Alignment," in CVPR, (Nashville, 2025.
[6] C. Wunsch, The Ocean Circulation Inverse Problem. Cambridge University Press, 1996.
[7] R. Vigário, Jousmääii, M. Hämäläinen, R. Hari, and E. Oja, "Independent component analysis for identification of artifacts in maggettoeneephhalorapiie recordings," in Neural Information Processing System, vol. 10, 1997.
[8] Brown, S. Yamada, and J. Sejnowski, "Independent component analysis at the neural cocktail party," Trends in Neurosciences, vol. 24, pp 63, 2001.
[9] M. Zhdanov, Geophysical Inverse Theory and Regularization Problems. Elsevier, 2002.
[10] W. Choi, C. Fang- Yen, K. Badizadegan, S. Oh, N. Lue, R. R. Dasari, and M. S. Feld, "Tomo- graphic phase microscopy," Nature Methods, vol. 4, pp. 717-719, Sept. 2007.
[11] M. Lustig, D. Donoho, and J. M. Pauly, "Sparse mri: The application of compressed sensing for rapid mr imaging," Magnetic Resonance Medicine, vol. 58, pp. 1182-1195, Dec. 2007.
[12] J. Virieux and S. Operto, "An overview of fale-wav inversion in exploration geophysics," Geophysics, vol. 74, no. 6, pp. WCCI-WCC26, 2009.
[13] K. Chadan and P. C. Sabatier, Inverse Problems in Quantum Scattering Theory. Springer Science & Business Media, ed., 2012.
[14] M. A. Iglesias, K. J. Law, and A. M. Stuart, "Ensemble kalman methods for inverse problems," " Inverse Problems, vol. 29, no. 045001, 2013.
[15] Chael, K. Bouman, Johnson, M. Wielgus, Blackburn, C.-K. Chan, J. R. Farah, D. Palumbo, and D. Pesce, "eht-imaging: v1.1.0: Imaging interferometric data with regularized maximum likelihood." https::pp// /tt.i//iiioo/// oooiddooooooooooooooooooooooooeo 10. 5201/221///// 2281//enodo... 2562874, 2019. Software.
[16] A. Ramesh, P. Dhariwal, A. Nichol, C. Chu, and M. Chen, "Hierarchical text-conditional image generation with clip latents," arXiv preprint arXiv:2204.06112 2022.
[17] J. Betker, G. Goh, L. Jing, T. Brooks, J. Wang. L. Li, L. Ouyang, J. Zhuang, J. Lee, Y. Guo, and W. Manassra, "Improving image generation with better captions." https:///coot openai. com/paper haa//dallleeee/eeeeeeeeee e- 3. pdf, 2023. Accessed: 2025-05-09.
[18] A. Vyas, B. Shi, M. Le, Tjandra. Y.-. Wu, Guo, Zhang, X. Zhang, R. Adkins, W. Ngan, J. Wang, I. Cruz, B. Akula, A. Akinyemi, B. Ellis, R. Moritz, Y. Yungster, A. Rakotoarison, L. Tan, C. Summers, C. Wood, Lane, M. Williamson, and W.-. Hsu, "Audiobox: Unified audio generation with natural language prompts," 2023.
[19] OpenAI, "Sora: Creating video from text," 2024. Accessed: 2025-05-09.
[20] B. Seed Team, "Seed-TTS: A family of high-quality versatile speech generation models," arXiv 2e9p::mtarxxi::24000222333 240..024300000... 2024.
[21] P. Comon, "Independent component analysis, a new concept?," Signal processing, vol. 36, no. 3, pp. 287-314, 1994.
[22] Hyvärinen and Pajunen, "Nonlinear independent component analysis: Existence and uniqueness results." Neural Networks, vol. 12, pp. 429-439, April 1999.

---

[23] Y. Bengio and Y. LeCun, "Scaling learning algorithms towards AI," in Large Scale Kernel Machines, MIT Press, 2007.
[24] I. Higgins, L. Matthey, A. Pal, C. Burgess, X. Glorot, M. Botvinick, S. Mohamed, and A. Lerch- ner, VAEE: Learning basic visual concepts with constrained variational framework," in International Conference on Learning Reprexentations, 2017.
[25] B. Schölkopf, F. Locatello, S. Bauer, N. R. Ke, N. Kalchbrenner, A. Goyal, and Y. Bengio, "Toward causal representation learning," Proceedings of the IEEE, vol. 109, no. 5, pp. 612-634, 2021.
[26] F. Locatello, S. Bauer, M. Lucic, Raetsch, S. Gelly, B. Schölkopf, and O. Bachem, "Chal- lenging common assumptions in the unsupervised learning of disentangled representations, in Proceedings of the 36th International Conference on Machine Learning (K. Chaudhuri and R. Salakhutdinov. eds.), vol. 97 of Proceedings of Machine Learning Research, pp. 4114-4124 PMLR, Jun 2019.
[27] J. Chen and K. Batmanghelich, "Weakly supervised disentanglement by pairwise similarities," in AAAI, vol. 34, pp. 3495-3502, 2020.
[28] R. Shu, Y. Chen, A. Kumar, Ermon, and B. Poole, "Weakly supervised disentanglement with guarantees," in International Conference on Learning Representations, 2020.
[29] I. Daunhawer, A. Bizeul, E. Palumbo, A. Marx, and J. E. Vogt, "Identifiability results for multimodal contrastive learning," in International Conference on Learning Representations, 2023.
[30] D. Yao, D. Xu, S. Lachapelle, S. Magliacane, P. Taslakian, G. Martius, J. von Kügelgen, and F. Locatello, "Multi-view causal representation learning with partial observability," in International Conference on Learning Representations, 2024.
[31] J. Sohl-Dickstein E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using nonequilibrium thermodynamics,, " Proceedings of the 32nd International Conference Machine Learning (F. Bach and D. Blei, eds.), vol. 37 of Proceedings of Machine Learning Research, (Lille, France), pp. 2256-2265, PMLR, 2015.
[32] Y. Song and S. Ermon, "Generative modeling by estimating gradients of the data distribution," in Neural Information Processing System, 2019.
[33] J. Ho, A. Jain, and Abbeel, "Denoising diffiusion probabilistic models," in Neural Information Processing System, vol. 33, pp. 6840-666 0001-881111 2020.
[34] A. Fisher, "The detection of linkage with 'dominant' abnormalities," Annals of Eugenics, vol. 6, no. 2, pp. 187-201, 1935.
[35] Q. Wu, Y. Liu, H. Zhao, A. Kale, Bui, T. Yu, 7. Lin, Y. Zhang, and S. Chang, "Uncovering the diffusion models," in CVPR, pp. 1900-1910, 2023.
in text-to-image catpiiiiilyyaaiiiiiiiiiiee disentanglement capability
[36] T. Yang, Y. Wang, Y. Lu, and N. Zheng, "Disdiff: Unsupervised disentanglement of diffusion probabilistic models," " in Neural Information Processing System, 2023.
[37] D. A. Hudson, D. Zoran, M. Malinowski, A. K. Lampinen, A. Jaegle, J. L. McClelland, L. Matthey, F. Hill, and Lerchner, "Soda: Bottleneck diffusion models for representa- tion Iearning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). p. to appear, June 2024.
[38] V. Popov, I. Vovk, V. Gogoryan, T. Sadekova, M. S. Kudinov, and J. Wei, "Diffusion- based voice conversion with fast maximum likelihood sampling scheme," in International Conference on Learning Representations, 2022.
[39] H.-Y. Choi, S.-H. Lee, and S.-W. Lee, "Diff-HierVC: Diffusion-based Hierarchical Voice Conversion with Robust Pitch Generation and Masked Prior for Zero-shot Speaker Adaptation," in Proc. INTERSPEECH 2023, pp. 2283-2287, 2023.
12

---

[40] A. Hyvärinen and P. Pajunen, "Nonlinear independent component analysis: Existence and uniqueness results," Neural Networks, vol. 12. pp. 429-4399999 099-439,444 439, Apr. 1999.
[41] M. Chen, K. Huang, T. Zhao, and M. Wang, "Score approximation estimation and distribution recovery of diffusion models on low-dimensional data," in International Conference on Machine Learning, 2023.
[42] S. Chen, S. Chewi, Li, Y. Li, A. Salim, and A. Zhang, "Sampling is as easy as learning the score: theory for diffusion models with minimal data assumptions," in International Conference Learning Representations, 2023.
[43] Anderson, "Reverse-time time diffusion equation models," Stochastic Processes and their Applications, vol. 12, no. 3, pp. 313-326, 1982.
[44] C. Eastwood and C. I. Williams, "A framework for the quantitative evaluation of disentangled representations," in International Conference on Learning Representations, 2018.
[45] M. A. Casey and A. Westner, "Separation of mixed audio sources by independent subspace analysis," in Proceedings of the International Computer Music Conference (ICMC). 154-161, 2000.
[46] A. Hyvärinen and P. Hoyer, "Emergence of phase- and shift-invariant features by decomposition of natural images into independent feature subspaces," Neural Computation, vol. 12, no. 7, pp. 1705-1720, 2000.
[47] F. Theis, "Towards a general independent subspace analysis," in Neural Information Processing System, vol. 19, pp. 1361-1368 2006.
[48] J. V. Kügelgenn Y. Sharma, Gresele, W. Brendel, Schölkopf. M. Besserve, and F. Locatello, "Self-ssuperised learning with data augmentations provably isolates content from style," in Neural Information Processing System (A. Beygelzimer, Y. Dauphin, P. Liang, Jnd W. Vaughan, eds.), 2021.
[49] K. Qian, Y. Zhang, S. Chang, X. Yang, and M. Hasegawa-Johnson, "AutoVC: Zero-shot voice style transfer with only autoencoder loss," in International Conference on Machine Learning, pp. 5210-5219, 2019. 2019.
[50] F. Huang, K. Zeng, and W. Zhu, "DiffVC+: Improving diffusion- based voice conversion for speaker anonymization," in Interspeech, 2024.
[51] Desplanques. J. Thienpondt, and Demuynck, "Ecapa-tdnn: Emphasized channel attention, propagation and aggregation in tdnn based speaker verification," in Interspeech, 2020.
[52] T. Chen, S. Korbblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations, in International Conference on Machine Learning (H. D. III and A. Singh, eds.), vol. 119 of Proceedings of Machine Learning Research, pp. 1597-1607, PMLR, 13-18 Jul 2020.
[53] J.-B. Grill, F. Strub, F. Altché, C. Tallee, P. H. Richemond, F. Buchatskaya, C. Doersch, B. Á. Pires, Z. Guo, M. G. Azar, B. Piot, K. Kavukcuoglu, R. Munos, and M. Valko, "Bootstrap your own latent: A new approach to self- learning," in Advances in Neural Information Processing Systems (NeurIPS), vol. 33, pp. 21271-21284, 2020.
[54] He. H. Fan, Y. Wu, S. Xie, and R. B. Girshick, "Momentum contrast for unsupervised visual representation learning," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9726 9735, 2020.
[55] X. Chen and K. He, "Exploring simple siamese representation learning." "nniing in CVPR, (Nashville, TN, USA), pp. 15745-15753, 2021.
[56] M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin, "Emerg- ing properties in self-supervised vision transformers," in Proceedings of the International Conference on Computer Vision (ICCV), 2021.
13

---

[57] L. Logeswaran and H. Lee, "An efficient framework for learning sentence representationss" in Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
[58] S. Schneider, A. Baevski, R. Collobert, and M. Auli, "wav2vec: Unsupervised pretraining for speech recognition," in Proceedings of Interspeech 2019, 20th Annual Conference of the International Speech Communication Association, pp. 3465-346655 2019.
[59] H. Lee, J. Lu, and Y. Tan, "Convergence for score-based generative modeling with polynomial complexity," in Neural Information Processing System (A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, eds.), 2022.
[60] M. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and D. Hjelm, "Mutual information neural estimation," in Proceedings of the 35th International Conference on Machine Learning (ICML), vol. 80 of Proceedings of Machine Learning Research, pp. 531-540, PMLR, 2018.
[61] A. van den Oord, Y. Li, and O. Vinyals, "Representation learning with contrastive predictive coding," " arXiv preprint arXiv: 1807.03748, 2018.
[62] P. Cheng, W. Hao, S. Dai, Liu, Z. Gan, and L. Carin, "Club: A contrastive log-ratio upper bound mutual information," " in International Conference on Muchine Learning, pp. 1779-1788, PMLR, 2020.
[63] N. Dehak, J. Kenny, R. Dehak, P. Dumouchel, and Ouellet, "Front-end factor analysis for speaker verification," " in IEEE Transactions on Audio, Speech, and Language Processing. vol. 19. pp. 788-798, IEEE, 2011.
[64] O. D. Liu, H. Tang, and S. Goldwater, "Self-supervised predictive and phonetic information in orthogonal subspaces," in Interspeech, 2023.
prediici coding models encode speaker
[65] M. Baas, B. van Niekerk, and Kamper, "Voice conversion with just nearest neighbors," in Interspeech, 2023.
[66] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. Dimakis, and S. Shakkottai, "Solving linear inverse problems provably via posterior sampling with latent diffusion models," in Neural Information Processing System, 2023.
[67] H. Kamper, B. van Nickerk, of self-supervised features through the lens of voice conversion," in Interspeech, 2025.
J. Zaïdi, and M.-A. Carbonneau, "Linearve: Linear transformations
[68] S. Du, J. Lee, H. L. Wang, and X. Zhai, "Gradient descent finds global minima of deep neural networks," in International Conference on Machine Learning (K. Chaudhuri and R. Salakhut- dinov, eds.), vol. 97 of Proceedings of Machine Learning Research, pp. 1675-1685, PMLR, 09-15 Jun 2019.
[69] X. Chen, Y. Duan, R. Houthooft, Schhllmnn, I. Sutskever, and P. Abheel, "Infogan: Interpretable representation learning by information maximizing generative adversarial nets," in Neural Information Processing System, NIPS' 16, (Red Hook, NY, USA), pp. 2180-2188, Curran Associates Inc., 2016.
[70] H. Kim and A. Mnih, "Disentangling by factorising," in International Conference on Machine Learning (J. Dy and A. Krause, eds.), vol. 80 of Proceedings of Machine Learning Research, pp. 2649-26588 PMLR, 10-15 Jul 2018.
[71] L. Deng, "The mnist database of handwritten digit images for machine learning research." IEEE Signal Processing Magazine, vol. 29, no. 6, pp. 141-142, 2012.
[72] A. Krizhevsky, V. Nair, and G. Hinton, "Learning multiple layers of features from tiny images." http://www. cs. toronto. Corootoo.du//kkrttt.... adu/"kriz/iifar. html, 2009. Accessed: 2025-01-22 -25.
[73] O. Ronneberger, P. Fischer, and T. Brox, "U-net: Convolutional networks for biomedical image segmentation," in Medical Image Computing and Computer-Assisted Intervention (MICCAI), 2015.
14

---

[74] Y. S.ngg J. Sohl- Dick-eDccstein, D. P. Kingma, A. Kumar, S. Ermon, and B. Poole, "Score-based generative modeling through stochastic differential equations," " in International Conference on Learning Representations, 2021.
[75] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang, "The unreasonable effectiveness of deep features as a perceptual metric," in CVPR, 2018.
[76] C. Busso, M. Bulut, C.-C. Lee, A. Kazemzadeh, E. Mower, S. Kim, J.-N. Chang, S. Lee, and S. Narayanan, "lemocap: Interactive emotional dyadic motion capture database," in Language Resources and Evaluation (LREC), pp. 335-33335 339,,,,,000 2008.
[77] J. Park, S. W. Yang, J. S. Kim, W. Shin, and S. W. Han, "Triaan-ve: Triple adaptive attention normalization for any-to-any voice conversion," in ICASSP, 2023.
[78] S. Luz, F. Haider, S. de la Fuente, and B. M. Davida Fromm, "Alzheimer's dementia recognition through spontaneous speech: The adress challenge," in Proceedings of Interspeech 2020, pp. 217-2216,, 2020.
[79] F. Vieira, S. Venugopalan, A. Premasiri, et al., "A machine- based objective measure for ALS disease severity," NPJ Digital Medicine, vol. 5, p. 45, 2022.
[80] A. Block, Y. Mrouch, and A. Rakhlin, "Generative modeling with denoising auto-encoders and langevin sampling." arXiv preprint arXiv:2002.001000 2020.
[81] V. D. Bortoli, J. Thornton, Heng, and A. Doucet, "Diffusion schrödinger bridge with applica- tions to score-based based generative modeling," in Neural Information Processing System (M. Ran- zato, A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, eds.), vol. 34, pp. 17709- Curran Associates, Inc., 2021.
[82] C. Pabbaraju, D. Rohatgi, A. Sevekari, H. Lee, A. Moitra, and A. Risteski, "Provable benefits of score matching," in Neural Information Processing System, 2023.
[83] K. Shah, S. Chen, and A. Klivans, "Learning mixtures of gaussians using the ddpm objective," in Neural Information Processing System, 2023.
[84] Y. Wu, M. Chen, Li, M. Wang, and Y. Wei, "Theoretical insights for diffusion guidance: case study for gaussian mixture models," arXiv preprint 2403022403.01639, 2024.
[85] S. Chen, V. Kontonis, and K. Shah, "Learning general gaussian mixtures with efficient score matching," arXiv preprint arXiv:2404.11893,22222
[86] G. Li, C. Cai, and Y. Wei, "Dimension-frre converggee of diffusion models for approximate gaussian mixtures." arXiv preprint arXiv:2500.0050000 2025.
[87] S. Mei and Y. Wu, "Deep networks as denoising algorithms: Sample-efficient learning of diffusion models in high-dimensional graphical models," arXiv preprini 1:X1v:2309.114400 2023.
[88] J. Y.-C. Hu, W. Wu, Z. Li, S. Pi, Z. Song, and H. Liu, "On statistical rates and provably efficient criteria of latent diffusion transformers (dits)," in The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[89] P. Wang, H. Zhang, Z. Zhang, S. Chen, Y. Ma, and Q. Qu, "Diffusion models learn low- dimensional distributions via subspace clustering," arXiv preprint arXiv:2409.0244 02426, 2024.
[90] V. D. Bortoli, A. Thornton, H. Hutchinson, J. A. Mathieu, Y. Teh, and A. Doucet, "Conver- gence of demoising diffusion models under the manifold hypothesis." Transactions on Machine Learning Research (TMLR), 2022.
[91] M. Chidambaram, K. Gatmiry, S. Chen, H. Lee, and J. Lu, "What does guidance do? fine- grained analysis simple setting," in Neural Information Processing System, 2024.
[92] Fu, Z.. Yang, M. Wang, and M. Chen, "Unveil conditional diffusion models with classifier-free guidance: A sharp statistical theory." ArXiv, vol. abs/2403.1 1968, 2024.
15

---

[93] P. Li, Li, H. Zhang, and J. Bian. "On the generalization properties of diffusion models," in Neural Information Processing System, vol. 36, pp. 2097-2127, 2023.
[94] A. Han, W. Huang, Y. Cao, and D. Zou, "On the feature learning in diffusion models," in International Conference on Learning Representations, 2025.
[95] Y. Wang, Y. He, and M. Tao, "Evaluating the design space of diffusion-based generative models," arXiv preprint arXiv:2406. 12839, 2024.
[96] G. Li, Y. Wei, Y. Chi, and Y. Chen, "A sharp convergence theory for the probability flow odes of diffusion models," arXiv preprini arXiv:2408.023320 2024.
[97] M. Li and S. Chen, "Critical windows: Non-asymptotic theory for feature emergence in diffusion models," in International Conference Machine Learning, pp. 177474-274999 2024.
[98] J. Schmidhuber, "Learning factorial codes by predictability minimization." Neural Computation, vol. 4, no. 6, pp. 363-879, 1992.
[99] I. Higgins, D. Amos, D. Pfau, Racaniere, L. Matthey, D. Rezende, and A. Lerchner, "Towards a definition of disentangled representations," arXiv preprint
2018.
[100] Z. Pan, L. Niu, J. Zhang, and L. Zhang, "Disentangled information bottleneck,"" in AAAI, 2021.
[101] J. Zbontar. L. Jing, I. Misra, Y. LeCun, and S. Deny, "Barlow twins: Self-supervised learning via redundancy reduction," " in International Conference on Machine Learning (M. Meila and T. Zhang, eds.), vol. 139, pp. 123 10-1220, PMLR, 2021.
[102] Y.-H. H. Tsai, S. Bai, P. P. Liang, J. Z. Kolter, L.-P. Morency, and R. Salakhutdinov, "Multi- modal transformer for unaligned multimodal language sequences," in Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (A. Korhonen, D. Traum, and L. Märquez, eds.), (Florence, Italy). pp. 6558-6569 Association for Computational Linguistics, July 2019.
[103] P. Poklukar, M. Vasco, H. Yin, F. S. Melo, A. Paiva, and D. Kragic, "Geometric multimodal contrastive representation learning," in Proceedings of the 39th International Conference on Machine Learning (K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu. and S. Sabato, eds.), vol. 162 of Proceedings of Machine Learning Research, pp. 17782- 17800. PMLR, 17-23 Jul 2022.
[104] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir, "Multimae: Multi-modal multi-task masked autoencoders," in Computer Vision ECCV 2022. pp. 348-367, Springer Nature Switzerland, 2022.
[105] J. Hahm, J. Lee, S. Kim, and J. Lee, "Isometric representation learning for disentangled latent space of diffusion models," in International Conference on Machine Learning, vol. 235 of Proceedings of Machine Learning Research, pp. 17224-17245, PMLR, 2024.
[106] A. Hyvärinen and F. Oja, "Independent coolepenndettcommmeen analysis: Algorithms and applications," Neural Networks, vol. 13, no. 4-5, pp. 411-430, 2000.
[107] R. S. Zimmermann, Y. Sharma, S. Schneider, M. Bethge, and W. Brendel, "Contrastive learning inverts the data generating process," in Proceedings of the 38th International Conference on Machine Learning, vol. 139 of Proceedings of Machine Learning Research, pp. 12999,11299,, PMLR, 2021.
[108] Y. Tian, X. Chen, and S. Ganguli, "Understanding self-supervised learning dynamics without contrastive pairs," in International Conference Machine Learning (M. Meila and T. Zhang, eds.). vol. 139. pp. 102788 PMLR, 2021.
[109] F. Locatello, B. Poole, G. Rätsch, B. Schölkopf, O. Bachem, and M. Tschannen, "Weakly- supervised disentanglement without compromises," in International Conference on Machine Learning, vol. 119, pp. 6348-6359, PMLR, 2020.
[110] A. Hyvärinen and H. Morioka, "Unsupervised feature extraction by time-cotraastive learning and nonlinear ICA." in Neural Information Processing System, pp. pp. 3765-3773, 2016.
16

---

[111] I. Khemakhem, R. P. Monti. D. P. Kingma, and A. Hyvärinen, "ICE-BeeM: Identifiable condi- tional energy-based deep models based on nonlinear ICA," in Advances Neural Information Processing Systems (NeurIPS), vol. 33, pp. 10235-10246, 2020.
[112] D. Horan, E. Richardson, and Y. Weiss, "When is unsupervised disentanglement possible?," in Advances in Neural Information Processing Systems (A. Beygelzimer, Y. Dauphin, Liang, and J. W. Vaughan, eds.), 2021.
[113] M. Rolinek, D. Zietlow, and G. Martius, "Variational autoencoders pursue pea directions (by accident)," " in CVPR, pp. 12406-12415, 2019.
[114] E. Mathieu, Rainforth, Siddharth, and Y. W. Teh, "Disentangling disentanglement in varia- tional autoencoders," in International Conference on Machine Learning, vol. 97 of Proceedings of Machine Learning Research, pp. 4402 4412, PMLR, 2019.
[115] I. Khemakhhem D. P. Kingma, R. Monti, and Hyvärinen, "Variational autoencoders and nonlinear ICA: A unifying framework," in Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), vol. 108, pp- 2207-2217, PMLR, 2020.
[116] Bao, J. Lucas, S. Sachdeva, and R. Grosse, "Regularized linear autoencoders recover the principal components, eventually," in Neural Information Processing System, pp. 6971-6981, 2020.
[117] P. Bhowal, A. Soni, and S. Rambhatla, "Why do variational autoencoders really promote disentanglementt?" in Forty-first International Conference Machine Learning, 2024.
[118] T. Uscidda, I.. Eyring, K. Roth, F. Theis, 7. Akata, and marco cuturi, "Disentangled rep- resentation learning with the gromov-monge gap." International Conference on Learning Representations: 2025.
[119] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida, "Spectral normalization for generative adversarial networks," International Conference on Learning Representations, 2018.
[120] Anil, J. Lucas, and R. Grosse, "Sorting out lipschitz function approximation," in International Conference on Machine Learning, pp. 291-301, PMLR,, 2019.
[121] H. Gouk, E. Frank, B. Pfahringer, and M. Cree, "Regularisation of neural networks by enforcing lipschitz continuity," Machine Learning, vol. 110, no. 2. pp. 393-416, 2021.
[122] J.-F, Le Gall, Brownian Motion, Martingales, and Stochastic Calculus, vol. 274 of Graduate Texts in Mathematics. Springer, 2016.
[123] A. Jacot, F. Gabriel, and C. Hongler, "Neural tangent kemnl:: Convergence and generalization in neural networks," Neural Information Processing System, 2018.
[124] Barron, "Universal approximation bounds for superpositions of a sigmoidal function," " IEEE Transactions on Information Theory, vol. 39, no. 3, pp. 930-945, 1993.
[125] R. Ge, C. Jin, and Y. Zheng. "No spurious local minima in nonconvex low rank problems: A unified geometric analysis," " in Proceedings of the 34th International Conference on Machine Learning (D. Precup and Y. W. Teh, eds.), vol. 70 Proceedings of Machine Learning Research, pp. 1233-1242, PMLR, 06-11 Aug 2017.
[126] J. Lee, M. Simchowitz, M. I. Jordan, and B. Recht, "Gradient descent only converges to minimizers," in Conference on Learning Theory, 2016.
[127] B. Hajek and M. Raginsky, "ECE 543; Statistical Learning Theory." http://maxin.eca.
illinois. edu/teaching/SLTT 2021. Accessed: 2025-01-25.
[128] Vershynin, High-DDmensional A Introduction with Applications in Data Science.
Cambridge University Press, Sept. 2018
[129] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever, "Robust speech recognition via large-scale weak supervision," International Conference on Machine Learning, 2023.
17

---

[130] K. Baba, W. Nakata, Y. Saito, and H. Saruwatari, "The 105 system for the VoiceMOS Challenge 2024: Transfer learning from deep image classifier to naturalness MOS prediction of high-quality synthetic speech," in IEEF Spoken I Language Technology Workshop (SI.T), 2024.
[131] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in International Conference on Learning Representations, 2015.
[132] Nichol and P. Dhariwal, "Improved denossnnn diffusion probabilistic in International Conference Machine Learning, 2021.
[133] Y. Song and S. Ermon, "Improved techniques for training score-based generative models," in Neural Information Processing System, 2020.
[134] Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, T. Gregory, Z. Lin, N. Gimelshein, L. Antiga, Desmaison, A. Kopf, F. Yang. Z. De Vito, M. Raison, Tejani, Chilamkurthy, B. Steiner, I.. Fang, J. Bai, and S. Chintala, "Pytorch: An imperative style, high-performance deep learning library," in Advances Neural Information Processing Systems 32, pp. 2024-8035, Curran Associates, Inc., 2019.
[135] M. Ravanelli, T. Parcollet, A. Moumen, S. de Langen, C. Subakan, P. Plantinga, Y. Wang, P. Mousavi, L. D. Libera, A. Ploujnikov, F. Paissan, D. Borra, S. Zaiem, Z. Zhao, S. Zhang, G. Karakasidis, S.-L. Yeh, P. Champion, A. Rouhe, R. Braun, F. Mai, J. Zaluaga-Gooeer S. M. Mousavi, A. Nautsch, X. Liu, S. Sagar, J. Duret, S. Mdhaffar, G. Laperriere, M. Rouvier, R. D. Mori, and Y. Esteve, "Open-source conversational ai with SpeechBrain 1.0," 2024.
[136] A. Baevski, H. Zhou, A. Mohamed, and M. Auli, "wav2vee 2.0: A framework for sulf-super learning of speech representations," in Neural Information Processing System, 2020.
[137] Y. Wang, A. Boumadane, and A. Heba, "A fine-tuned wav2vec 2.0/hubert benchmark for speech emotion recognition, speaker verification and spoken language understanding." " arXiv preprint arXiv:2111.027355 2021.
[138] Z. Ma, M. Chen, H. Zhang, Z. Zheng, W. Chen, X. J. Ye, X. Chen, and T. Hain, "Emobox: Multilingual multi qulti--orpuss speech emotion recognition toolkit and benchmark," in Interspeech, 2024.
[139] L. Wang, Y. Gong, N. Dawalatabad, M. Vilela, K. Placek, B. Tracey, Gong, A. Premasiri, F. Vieira, and J. Glass, "Automatic prediction of amyotrophic lateral sclerosis progression using longitudinal speech transformer," in Interspeech, 2024.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the main claims in the abstract and introduction are an accurate reflection of the paper's contributions and scope. The stated goals and innovations are clearly substantiated in the theoretical sections (Sec. 4. 1-.3) and experimental sections (Sec. 5), and there is no overstatement or misalignment between the framing and the actual content. The abstract provides a faithful summary, and the introduction sets realistic expectations for what the paper achieves.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper diseuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in the discussion of each section throughout the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to teeech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable. the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
19

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Full set of assumptions and a complete and correct proof for each theorem are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yss]
Justification: All information needed to produce the main experimental results are provided in the paper and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. case of closed-souree models, it may be that access to the model is limited in some way (c.g.. to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
the
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Data and code will be released upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
public/guides Sewiidde/CodeSumissiiooo sooiciooiiiiooiic for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips. wttpcccppppcccccet.cc...cc.cccct //ceigiiiee guides /wt////oooeeesseeeeeeeee wwm.iisssooooiiioo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines, If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All the training and test details provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance the experiments?
Question:
Answer:
[Yes]
Justification: We average results over multiple runs and report error bars other statistical significance metrics for the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
21

---

The method for calculating the error bars should be explained (closed form formula, call library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors). should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma enror bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (e.g. negative emror rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
The details about computation resources are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research condueted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: htt////nurrpppp
Answer:
[Yes]
Justification:
The research is ethical according to the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special considera- tion due laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The paper presents foundational research not tied to particular applications.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
22

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets are fully licensed.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use in the pack- age should be provided. For popular datasets, paperswi thcode. cww.coo////ooooo com/datasets datasette has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
23

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All assets are well-documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodll part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contri- bution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper has no such risk.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration not required.
Answer: [NA]
Justification: LLMs are not used for non-standard purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (https: httpp:://nuurppp cc//oonfeeeencce/220///111 for what should or should not be described.
25

---

Contents
Proof of Example 3.1
27
B Proof of Theorem 4.1
27
B.1 Main proof
27
B.2 Proof of Lemma B.2
29
B.3 Proof of Lemma B.3
31
B.4 Proof of Lemma B.4
32
B.5 Proof Lemma B.5
32
B.6 Proof of Lemma B.6
32
B.7 Extension to correlated content and style
33
Proof of Theorem 4.2
33
C.1 Main proof
33
C.2 Proof of Lemma C.5
................................
34
C33 Proof of Lemma C.6
................................
36
C.4 Extension to correlated view-specific styles
36
D Proof Theorem 4.5
36
D.1 Main proof
37
D.2 Proof of Lemma 4.4
41
D.3 Proof of Theorem D.2
41
D.4 Proof of Theorem D.3
42
D.5 Proof of Lemma D.5.
44
D.6 Proof of Lemma D.6
46
D.7 Proof of Lemma D.7
47
D.8 Proof Lemma D.8
47
D.9 Proof of Lemma D.4
48
D.10 Proof of Lemma D.9.
52
D.11 Proof of Lemma D.10
53
E Experiment details
53
E.1 Latent subspace GMM disentanglement
53
E.2 Image disentanglement
54
E.3 Speech disentanglement
56
26

---

A Proof of Example 3.1
Example A.1. (Example 3.1, restated) Suppose content Z (0.1) and G G~N((.11 are independent standard Gaussian r.v.'s and consider the noiseless setting with 6 -. Further, suppose (2 #
22)- Then we can choose the conten//stye encodings to be 2 = Zsgn(G), G = G and the decoder f(2.G) = f (G) G), where sgn(z) denotes the sign of Then (2, G) are (0.0)- disentangled but not e-editable for some e 0.
Proof. First of all, notice that 2 (0,1) regardless of the value of & owing to the even symmetry
of the standard Gaussian distribution, which implies 2 L. G, or equivalently 1 ((2;) = O. Second, by construction, f(2,G) = =
= X, and thus G(2,G:X) T(Z,G;X) 0. Therefore by definition, (2,G) are (0.0) (0.0)-disentangled, However, for an iid sample Č Pc. the conditional distribution Pf(ž,örjzz
(PPf(-2,')) + 22)((z,')]z # (z)z)) due to the fact that sgn( sgnn (') is a symmetric Bernoulli variable and (.( # Pf(- Z. Therefore, (2,0) is not edditable for c= drw(p Px
B Proof of Theorem 4.1
For clarity, we restate the theorem below.
Theorem B.1. Suppose Assumption 3.5-3.6 hold, and (0", ,Ø") be minimizer of L? defined in Eq. with p = (Z C,8, C2/T for some C₁,C₂ > 0. Set to Then for any min 1 vdx the encodings (2,G) = (zg- (Xx.).g(X1.)) are (e,v) disenttnsdt with c C x 8 for some constants C3,C2>00 0.
and 2:= 2g- (X).
B.1 Main proof
Our result relies crucially on the following lemma. Lemma B.2. There exists (01,01) such fhatfor
{ 1 1 log(1 -lg -5)1/2
8 8< min
2 Vdx
the followings hold:
1. The followings hold for the MIs (X.)) (x,
(1ax)))k))))1)//2233-000xxxxxx)) ((=xx))))))=//((:))+(0ooooo I(zø) (
2. The conditional score matching loss satisfies
Le (f(,0)) - -1) =0 Jdx
-
First, we assume Lemma B.2 to be true and defer its proof to Section B.2. Therefore, by the property of (01,01) and the optimality of (0", ,0),
(6" .6') ) (0 .6" ( (Xx, < (1)0,,1)++/0 (1
(1+03 22x6xx
+ 2(T
27

---

for some C1 = =1000xxxx and :- -2:=1+030 Since both Le and (Z; are nonnegative, this implies
(oxx +C178+>(((2:X)-
Le(0" ,0)5
2(T -1)
I(zg- (X))))))))) (1+03 2x(T--t1) +C18.
1+ 2(--h) and p= (())))((0) then we have
Choose 7 --2(--1)
where C₁ =1=0({,oxxx).
Let ((X): - =()(((/)llppp denote the differential entropy of continuous random variable Then by definition, we have
I(Z,G:X) (.xG;:))N(Z..)))) M
As a result,
(1)
(16)
- -
(I +(Xr:X12,
(14)
(50)
where terms (ii))) can be upper bounded by item as 3C18. To bound the term (ic), the definition I(2,G;X):
12GG:))-MXX- -og(())e-
and apply the maximum entropy inequality on ((X7,2(X.)))C;;x))
(ee -e -e-F)s0-- (Xr. 2 G,G,T)++e Xx x||
(X,,,
log2
2
24(X)
22 lim L .t)) 1og2
2h(X)
logC2.
The last inequality uses the optimality of (8", .6") and therefore (x.,2.g.))) 's needs to achieve minimal loss at any E [t, which is upper- bounded by item 2 of Lemma B.2 as
lim C2 2
C2
lim (0 +
b 2(T - 2T -1) (1-e1--2T))
To bound term (v) of the RHS Eq. 14, notice that (2,G) and Xt, are invertible, and thus
I [[)x,x())))=
Combining the bounds and choose T' == (((ogt))))) conclude that
concll
(X., (acp::/c00gggaaccc...oo0333a----
28

---

B.2 Proof of Lemma B.2
To prove the lemma, we need the following technical lemmas, whose proofs are deferred to B.3, B.4 and B.5 respectively.
Lemma B.3. Suppose random variable =ox++N upy, for independent random variables X px and N N (0,,o2la) Ia), where a:= v2. Then, for any distribution g(zl22) with an L-Lipschitz score score function in x: for any 2 E X. then the following inequality holds:
( log ((vlz)
(x<22)
for some constand C independent of || and d.
Lemma B.4. For 8 < 1/2 a:d a: 82, the KL divergence between two Gaussian distributions N(ap,8Faa) and N' L. &2 La) is upper-bounded,
DKL. (((a,,6 N az (
82
|
Lemma B.5. Suppose random variable Y = =ax Npy. for and N ~ NN(0,a where o== 02 Further, suppose px (x)> 0.ror any E Rd and the score function of px I--Lipschitz. then the following holds
for independent random variables X X~pp ~x px
max (xx.xxxxxxx)))..xxxxxxxxxx))
for some constant C.
Assuming Lemma B.3-B.5, define == f= f(Z,G) and choose to: 10022181--00 Then by the property of the OU process.
=a,=aX
where a := 11-82 Next, since the mixing function f is invertible, we can write its inverse f where we call 2: R: + R2 the true content encoder
Rdx + Rd Rd f ( and 9: Rdx + Rdo the true style encoder such that = =(X)= Z X) G. By Definition 3.3, we have access the true style encoder g(-). Set o1 such that 2oz (x) = (zr). Further, define
<(>[x): : :=((x, 1Xx (=|2z)
: (2),
r == ( (=|xr);
P());=
then by definition,
((x)X..):x) DKL (p( (x((x)](((2)
-Epx
-E,x [DLL ([(((()))))lp(Z Dg (
Sppx [Dxx(((2]x))]lpl (xx
<Epx DKL (x
where the first inequality uses the non-negativity of DKL and the second inequality uses the triangle inequality of DKL: (KL(plog) < (r)(p](r) DEL (rllq) for any pdfs p.g.r. Further, by the data processing inequality,
Epx DKL (xL((((((()))lp((x)x)
=Epx DKL (x (P=(()) x)
SEpx ( |1x
Nyylar 1,2 Laxx
-Epxxe, (x.2) log
Px (r) ] (x())
(I)
=Epxxe, (F.1) log
(x
Px (z)
DDL(pxxlle + (z,n)
log px (v)
+O( +(]xxx)
29

---

where the second-6 soad-to-lat to-tast equality uses Lemma B.4:
Myyaax,82 Idxx
E Bpxx,, (2.z) log
M
Ny Tax)
=pxx (2.))) log
log
+d (1g///(1/a)
NN R. 2 lax)
(15)
To proceed, notice further that
Px Px(x))
(xx (lpx) Bxxx, (2.00) log
px px(1)
=0(),xaxx);
(16)
where the first equality uses Lemma B.3 with 6 < 11//dx, and the second inequality uses Lemma B.5. Combining this with Eq. 15 yields
where G C==0(A,axdx). Using a similar strategy, we can prove that
For the last MI MI (2) define
and notice that
-(2;G1))
- -Epx haap://hhhcciiiiiacccc/////////n
-Epx Dann44/ceh)itt/t//0////22////111
Epx D4L1///111//7)))-P///CC///111111
< Exx Daa.01/2200//00///00000021221110
< Epp DxL((1(2.G(3)]lp((226x))) <ppx (RL proppppp..////ppp===000oo0
where we again apply non-negate non-negativity of DKL on the first inequality and triangle inequality of DKL on the second inequality. For the last equality, we combine Eq. 15 and Eq. 16.
To prove item
0 so that the score function
se, (X,, exp( exp(-t)f(2,C)) -X
1-exp(-21
Then by setting ti := 1 the loss Le becomes
Lc(02,01)= = T /' (() X112
(1-exp(-2t)) dt
(01,0))))) p=" dr
T-t Je'l 2(T-1)2
(0(0),01)
1
< (ET -226)
2(T-11)
2211-1 21 -1
2(T (t) -1)
Further, notice that
X10 21
5C20dxx
30

---

where C2 == 1 + 8 The second equality uses the definition of inverse functions so that
(c(x)),()))=zz(dd Rdx The last inequality uses the fact that
E X - = -
x12
-B|[a-11Xx+MM
= =1_ - 5 dx
As a result, item follows from
Lo(0n,01) C2 -2 S 2 0))1/2) T -1) -1)
C2ôdx
=0(0g)
ōdx
with the choice 4 and ô and the fact that 0<0 <<0<min miin,.)...
B.3 Proof of Lemma B.3
To begin, we make use the following lemma proved Section B.6.
Lemma B.6. Suppose the score function sy (x]2) := V vllog((>))) of the probability density q(x|2) is L-Lipschitz function of x, then the following inequality holds:
((vl2) log (x(22)
- [sll:ll+++ss,s00)llllllllll--- L
x
2
Set (<|z)) V (z)ggg)))2) then by Lemma B.6,
E (x log (v]+) q(x]2))
E Bexr
For the first of the RHS, notice that
E Pmar
= //BCC//XHHH|000000////////111110 < X/22811X11++-GG011--000X111+7772
where 2y:=2sup.Els s, 02|z)]1² and C. large enough. To bound the second term of the RHS, notice
that
L
L (1-))====)))
E Bpxy(r.n)lly-pp
2
2
Lo² 2
< 2
(0'E)
(o
Combining the two terms yields
E Epxyz(e.g.)) log (vy22)
(<)22)
for some C> 0 large enough.
31

---

B.4 Proof of Lemma B.4
Use the formula for the KL divergence between Gaussians:
(KI(N
8"
1ay laul1/all +dlog
d/2+
(6/0)2
2(6/a)2
|011
1
2 82
+dlog
2
(114|12- -oll==ddj0-
d)o2
d
8²
log
1+
2
2
1-62
(11al1 [|pl|2-a 1 d)62
2d0²
|1112
d
8².
2
2
B.5 Proof of Lemma B.5
By Jensen's 's inequality,
DRL(PY|IPx
-k -E log
px(v)
P.x (x(o)((gla
E PxteeWoooxx log
px(x)
-E Bxtooooooooo log
+dloga
px py(
px (I) 02d
S [px(ttWW)llooo log
px (1) 2
SCL(0=E|x] II)),
for some C' 0, where the last inequality uses Lemma B.3. Similarly, apply apply Jensen's 'ss inequality and Lemma B.3,
(x = Exp [
ExxcaNWsssoo//aattnoog px (a
px Ia
Px (x)
=EpxtaWWil/aa.//apuu log
dloga
px
B.6 Proof of Lemma B.6
by the Lipschitz property of the score function of sq. for any (z.y) X²2
((y|=) (x]22)
+)|+ T.yly - 1112
log
2
L L 11
< [,,( [1=)|==]]|
2
Apply the Lipschitz property of s again,
log (1)l=)
T. g x
< (1||=||+ |s//://)+++
(17)
2
32

---

B.7 Extension to correlated content and style
To extend Theorem 4.1 correlated content and style, we can add additional term the right-hand side of Eq. 14 Theorem 4.2 as
T(2;O) = (i)+ (fi) (10))000))0000000000000000000000
(18)
where (i)-(v) are defined and bounded as in Appendix B.1.
C Proof of Theorem 4.2
For clarity, we restate the theorem below.
Theorem C.1. Under Assumption 3.5-3.6 and additional regularity conditions (Assumption C.2-C.4).
Consider the two-view, unimodal setting with ò1 =: ò and ô2 0. Let (0" (6) be minimizer of Lm and ži 242 (X'), G = 962 (X') for each view i. Set to = - log - 8²) and the diffusion step size n:= Cs ( SCd (ii) the content encoding 222 and the style encoding G are C7 x editable.
8 XgT for some constant C3 > Then for some constants Co Cf > 0: (i)
C.1 Main proof
Let X = G¹), where the last equality assumes the unimodal setting. By the invertibility of the mixing function, there exist functions = Rddx - Rd such that =(X*) Z and g: g:Rddx + do such that g (x*) = iE (1,2). To prove this theorem, need additional assumptions below.
Assumption C.2. For any 0 e 4 the estimated score function so(x,2.9.t) is o(1)
Ae Lipschitz in
all arguments.
Assumption C.3. The style encoder 9ea : :Rox - Rdc is Xy-lippshhtz in x E X and the content encoder Zøz Rdx + RR2z is ---Lipschitz. Further, there exists Φ1,G E @G such that 2 (.) =(())
and 901 = --(-)-
Assumption C.4. The mixing function f is --iisshhhitz
Assumption C.2 ensures the function class for the bottleneck and the score function are Lipschitz, analogous to the setting in [41]. Assumption C.3 assumes that the content and style encoders are both Lipschitz and the true content and style functions are realizable by the function classes & and &o respectively. This is a mild assumption since the style and content encoders are assumed to be NNs and common choices of neural architectures are either already Lipschitz or constrained to be so through regularizations [1 19-121]. For Assumption C.4, note that the invertibility of so is not strictly necessary and can be relaxed to injectivity by partitioning the domain.
Again, we start by introducing a couple helpful lemmas and postponing their proofs to Section C.2 and C.3.
Lemma C.S. There exists (01,0 (0,,02,,1H6.1) such that for
1
) }.
8 5<min
t0 100(111112)/ 4 =-=log(1 10)
2 /dx
the followings hold:
1. The followings for MIs aGd Ind
and
[((2:))=({xoxxxx3)
1(0=2)=((A,oxxxx0
2. The regularized score matching loss for the multi-view disentanglement in Eq. 8 satisfies
-(21)
= \ x
Lm (0, (1),
21) - -T) -0
T
where 61 = (6,1,,
33

---

Lemma C.6. (Novikov's 's condition) The following bound holds:
T- [X, -oooo-----lllooraaaall (X,''
[x exp 0
|x (x)) <00
2: + 2 and g: Fx Rd be the true content and style encoders defined in Section B. and define X = 1(Z', G*), i E (1,2,3) By Definition 3.4, we have X X Further, define
x21 = f21, to be sample from the following estimated backward process:
dx.. [X?" 2sg (82), 22,0 T knn)T--n)d v2dB .88 ~pr, (kn,
(19)
We begin by proving item (i) of the theorem. To this end, we apply the first part of Lemma C.5 and data processing inequality:
(20)
Next, we prove item (ii). Then apply triangle incquality:
Epp drv ( 12 12 S Drp(drv((pp (2 drw tra PPppppe22e (21)
(b)
To bound term (a), we check that Novikov's condition holds by Lemma C.6. and apply Girsanov's
theorem on the estimated backward process Eq. 19 with i == - llo((1 8) 1/2 followed by data processing inequality:
En drv x pxa
-0 (vdxnto Lo(0- .z 02)
=
where the last equality applies Lemma C.5. Set T we obtain
E 1
To bound the second term (b), applying data processing inequality, Pinsker's inequality, triangle inequality and data processing inequality again in that order,
drv drv(Pxu
EdTv(p226 P2 2 S hatp:///oooooepeeeeoeeeeeeeeeeee
=drv (02v(P20011
Po ( (owp://ggo0ttoppp2222222eppe
P2
1(22;G
1(G)Z))
I(X2:X'IZ)
+0(V = ((xo)xxx)))))xxx) (22)
2
where the last inequality uses Lemma C.5 and the last equality using the conditional independence x² 2
Combining the bounds on (it), we obtain
Bpp dr (2 o 03dx00 (oxxxx) - =
C.2 Proof of Lemma C.5
We start by proving item By Definition 3.4, the styles of different views are independent, i.e.
G Il. X2 and thus x() =0. Choose parameters 1,,,,, such that
for (x) = f(x), z0n,1 (x) 2(x), Doa, = (r)=)())))
34

---

Further, define
((gl=>) : PGi xx =
((glz')==por = x (g)x²), (fc)
(z²),
and by a similar argument as the proof of Lemma B.2 in Section B.2,
Further, by data processing inequality,
pxy DKL. ( lpg x²²) s xx PKL. xx
=Ex² DKL (p(z "llao="")lla"""""""aa"""""""""""
=Epx1 DKL (a x ()
(23)
where the first equality uses the fact that -'X and thus
x= Again by a similar argument as in Section B.2,
-=))=(()(x²), (=0=)
(o _²²) =()==()
Ep
= = =[C')x))
As a result, by the chain rule of MI.
=1(C)=/133++()ooxxxxx)
=0()_oxdx0),
(24)
where the first inequality uses the conditional independence &' 11 X² Z and the last equality uses the conditional independence 6 11 -| = X2. This concludes the proof of item 1.
To prove item 2, consider the score function of the form
se, (2,) exp(-t)f G exp(-t)f(Z,C') 1-exp(-2t) 1-exp(-20
and set t1 = glog 8. the loss La becomes
(0,00111)=== T-
(1-exp(-2) -2t))²
Lm (01,01)
1
1
(0 -2211
S
2(T -h)
21
2TT -1
2(T -17-41)17 (1) - 1)( -1)
By Assumption C.3,
= = (100000-11111000--X----222221111
where the inequality uses the fact that -
E5 E X
+aN + -1
+x
((0x) (1o%-11)+22)844.
Plugging this into Eq. 24 yields
+22 -x -
Lm (0
2(T -17-11) ((T-1)) 11) 1)
A1dg
This proves item 2 of the lemma.
35

---

C.3
Proof of Lemma C.6
First, by the equivalence the forward and reverse process.
1 T-h
Ex (x: exp
2
=
80 (X
1x | 11
2
=Epxi (1
|s0 1,0 .t V lo; (x)
(25)
2 2
20( ( (x) (t)
-Epxi po (xx))xxp
(26)
where the second-to lossadtto-last equality uses the closed- form formula of the conditional score function of an OU process and (t):= = v1 e- Define A: (1 + + >g), and by Assumption C.2 again, we have
2, 6 (t) || dt
Eexp
(())) sp
X
(t(
x 2
x
Eexp
dt
oft)*
|| /"
/² di
| 11
A +²((()
|,7111
( x2
SBexp
+2X²
dt
((1)a
a(1)a
(t)
/sup:
1273
Seepp
<XC,
82
282
where the last inequality uses the fact that X is sub-gaussiae while stst, <i dx is sub-gaussian [122, 41] and independent of X, and the second-to-lass inequality uses the bounds
dt
T
T
T
< (1)a
41
(11)²
(1-e-2h1)
82
T
T
at)2 < ot,)²
1-e
1-2-211
8"
C.4 Extension to correlated view-specific styles
To extend Theorem 4.2 correlated view-specific styles with I(G' ()) we can replace every step in the proof of Theorem 4.2 that involves the conditional independence X IL X2 Z and thus 1(X) x 0 Eq. 22 and Eq. 24) with
by data processing inequality. The rest of the proofs follow as before.
D Proof of Theorem 4.5
For clarity, we restate the theorem below.
Theorem D.1. Under Assumption 3.5-3.6, for some to dependent on n and let min min(dr.dut -> co. and let the positional encoding PE(.) be bounded and linearly independent over E [to,T] Define Par be the projection matrix onto R(M), and (s(s) to be the i-th largest singular value of the operator s.
Then for some > Eq. 12 converges to critical point 0 = 10z Ú, [.V such that with probability at
least 1 (H)): 1) Encodings 2 = P and ==Xg(X)) are (
d 2/110///
-isentangged;; 2)
Tdz (82)nare
Let Z,, 2 Then (Zr,, are O
1og 1/2 Tmin(o
editable.
az (1))) (
36

---

D.1
Main proof
To prove the theorem, we need Lemma 4.4 proved in Section D.2 and the following theorems, whose proofs postponed to Section D.3-D.4. Note that in the proofs, drop the subscript and use the population versions of the losses when v. + co. In subsequent analysis, we use the following balancing loss:
nn = R2 B Rop (x) + -agcoraalll)))))lllo.)))
where Ü = [- and s(x.t):=(s2(z..)) scc(ttt))) (
Theorem D.2. For the linear subspace model in Definition 4.3 and the objective in Eq. 11 with n oo. then any minimizer (U*, "V,)"") of Eq. 11 satsify -(U*)=R((( R(Az) and ((V*))-(AA))
and the neural network weights are initialized by standard Gaussians. Further, choose PE(.) such that PE(1) 's are bounded and linearly indeperdlent for all L E T. Then for X2 =3, the system of gradient flow equations in Eq. 12 converges to critical point (U, V. such that -(U)= -R(A))
Theorem D.3. Suppose nin,n,dr,de
R(Ag).
Recall that the two branches of the dual encoder network 8% and & are defined as
dæ
22
1
(15T 2)
[x .PE(t)] +++
(27)
vdH
z
dy
& (g.1))
=oo)=))))))))
1
(2)
(0
[g PP(())])++
(28)
dd
j=1
Further, define the neural tangent kernels (NTKs) [123] for the score functions sz and sg as
Kz(x,t,a ,t') = Jvee(02))2( (voc((2)sz(z's f'),
Kalx,t, : (ece_ ) 2/ [toi]],
where vec(f) denotes the flattened version of the parameter
For i. samples ja',, [.'',.,,,= let the Hilbert space spanned by the NTKs Kz( ss (K( be HK2 and HKg respectively. Further, for any fxxtt) -f Sxx (x,,t,a t')c(a' t')dx'dt' MK,c(x,)) Red Vx,,, define the NTK norm as
IV / EE, EL (ere. K(r,,,d (
(29)
Further, define the subspace score matching losses
Lz((isz): ([s_::)))=== bcC
LG Begc==/ehaeelladddlaad--deee-ee-
and their empirical versions
n Lz(U,sz):=
l (z( U,8))
71
12
Lo(V := Σ (o(F'
We also need the following lemma proved in Section D.9.
Lemma D.4. (generalization error bound) Let min(dr,dd) + co, and for i.i.d samples ("r,a))) let the reproducing kernel Hilbert space (RKHS) spanned by the NTKs K z (x..,y)) 's and K (xnd.))) ) 's be H.yy Kz and H. H.k respectively, and denote K (z,(2,)) == Kz (z,t,t Further, define the function classes
37

---

Sz as
01
Sg =
f
G
2,2. 2,2.
WNNN.Iff
Amin (Kz)
1=1
U:||U|F S Cz (
u
ai(o))do 1//2 Mo] [x] (x(x
Sg
/ Σ ca aKaa
VNEN.11 SCo/
)mn)(Kz)
V:IlV (
where o, (A) is the i-th largest singular value the operator A. Then with probabiiyy at least 1 1-0 (
d% log'' 2 n
Lz (U,
min Lz(U,sz)+0 (U.
d Lggog" n
La(V
sa)5
min (Vaa)EVxSS
Lc(Visc)+O
Now ready prove the Theorem 4.5. To this end, first prove the following statements:
The population score matching loss satisfies: Lo(0)=0)
& log'n
2. The subspace recovery are bounded by:
=0(1
dš dg (82)m
=0
dolog"n o (s6)m
|Pb 1Po
IP AcA
First, by the universal approximation theorem of neural networks (e.g. [124]) and the Lipschitzness of the true score functions, for Cz,Ca large enough, 8 Ez and sq E and therefore
L2(U,s2) = L2 =0, (U.x)
min
min (V Lo(V,8a) = La La (Aa
As a result, we have with probability at least (2))
Lz ((())) Lo(V,sc) scnn.
where e(n)=
(V
&llozzn
de
log2n
Therefore, we prove item by noticing
2 ((
1 log
We proceed to prove item Let Qz E xd2 QQ E RR. xda be orthogonal matrices such that
PRIÚ = Q R(V) = QGQT.
then we have Ü3z:=QzQTH and QoQ sa.
Then applying Lemma of [41] yields with probability at least 1- 48,
c(n)
|| PP - 2
Odz (s
(n)
| PR(V) 2
od (8g)
38

---

To prove disettaiiannn,,lemenn, define =1 = and Z1 == Pu.:: Then by definition,
- + + -1 (-
Since 2| is function of Z and C= G. we have the independence relations Zi -l G and 2 11 Z and therefore by data processing inequality,
((::)<1((;211)
Pžzı (2.9)
Erzz, (z.9) log
P2(2)pz_ (y)
P2 (2-y)
-Epaa (ow))og log
(2(2)
pz (z)
P2 (3-1)
=Epar (o. log
+2p22 (og
P2 (z)
p2j (z)
p2
121 -2) 4)
=epg(s)
+ +R -Bpzz (x,0) log
(2(2)
p2d (2)
P2 (2- pzj (z)
=-DRI(P2l1pz + p2 (2,3- log
p7j (=-y)
pz (a y)
Epp2 (5.5-v) log
Bpp, (v)e (= log p7 (z)
p7 (z)
-0 -Bpppt(t--)pp,, (v) || ||y|| +E (v) w - Acll?)
(x-9))z
where the last inequality uses the nonnegativity of KI divergence and the second-to-las equality combines Lemma 6.6 with the fact that Z = Po A2 Z is 1-Lipschitz function of Z and therefore its score function along the column space of Pt: Az is A,-Lipschitz. To upper bound || Acll we utilize item by noticing that
1 1
c(n)
rann(()) -2)) - 20 (82)²
where the last inequality uses the fact that rank(U)) S dz. Asaresult,
v((n) =0 Odz (5%)
dx 3/ Odz (5%)
10g3
T(2;6)=0
Further, would like to bound the mutual information gap (2,G;X) - I(ZZG;X) by finding estimator of the sample X from (Ž, To this end, show that [Pu, is invertible with high probability. This is the case since by item 2,
log 1/4
log 3/4
- 1P--
PP Poo+Py - Id
with high probability. Therefore, by matrix perturbation inequality,
log 3/4
σax ((PooPb))2100
>0 0
n2/4
for sufficiently large n. As result, we conclude there exists an X and therefore preserve its information perfectly. Therefore, (2,G) are (ol "log2"/ disentangled.
that reconstruct
.0)
ad (2
To prove e-editability, choose :- boo and 4 llog for some > 0 and let q(- G2 be the pdf of the following estimated reverse diffiusion process time 7-1:: diffiussoooooooo
dx; = [&t +2sg( G V2dB ,8, ~PT LE [km
39

---

where
sg(2,2.g.1) E =x - x-p22 Pvg Pp2
(30)
ott)
For generated samples X x- (1 G) and X'~ a( 2,, G'), where & ~p is an i.d sample of G we have
12 + dPx 2Pp212)
(16)
To bound (i), notice that the score estimation error 8g-
N₂
L.:-B E, Beg
2
G.X.
s0l 8 (x(Xx,22
{1))
Te 2
310 +6+
1
dt
T T' J1
a())
Pt Pri PP 2
(e22 1) -1) E.
2(T
dx 5T
Pv 200 + Pv Pv 1
TR.
1ogg 9/18 7/
dx Tn1
Tmin (8 02 (5 ( 17716
where the last equality uses 8: 1oga/"n 1//
and the fact that
Po Po +PvG-
22 Po Az A - Az Az +2
(PVAAAT
8 1og 3/2
1og
min
(s ) ss 1/2
) do
n1/8
Further, using an argument similar to Section C.3, we can check that the Novikov's condition holds for
$g. and using an argument similar to Section C.2, we can apply Girsanov's theorem show that
dx 10g
dx
(1) 0
(min ( oda (5)) I T/2
Further, notice that
1 G12)
P2 G.. (2..1.2222gg. (2(g,2)
= -Ep,, log
P2., 1z (((22)
PGN, (2 -11(2)
-Bpgi 0iz
log PizliioNN (8- (8))(=))
PsN. -==
= = --DKL |P + log
PoN. (8- (8)f((=)
1
o 11 [1
62
-0
1
|PPoAcll) -o
10g
5/4
=0
x
5z
8laz (sy)n!/3
Ozz (8 1n18
40

---

where the inequality uses Lemma B.6. By Pinsker's 's inequality and data processing inequality similar the proof of Theorem 4.2, we have
12-21)
Odz ($ 1
Combining the bounds on (i)( we have 2eo and G are o
8x 100
editable.
Tmin(o
1/2 az (0))
de (0)))))
D.2
Proof of Lemma 4.4
Let J, E IRd xd2 denote the Jacobian matrix of vector function R + Rd with respect to vector x E Rd Use the change of variables formula for pdf on X; = A Z AGG, we have
Pa (x( pz, / 777zccccccccccb))eee((4dddd11111
= =pz,G, [/foccc((ccllleddddddd/////////d
where the last equality uses the fact that the matrix A= is orthogonal and thus [det(A)] =1. Further, by Definition 4.3, Ze G,, 2 since Z and 4 N, No, due to the orthogonality of R(Az) and R(A.). Therefore,
pa(x) = a (z))) = (=(x) (g(x)).
Plugging this into the score of (x( and applying the chain rule yields
V (z)
= A by
D.3 Proof of Theorem D.2
Let s (z): V Vooopz (z(x)) and (g) V V-l--pgooppp. (9) and PA be the projection matrix onto R(A),
then for any U,,V),
Lo(S2,sc.U,V)
R. p(() PP
PA (Bc0:3ahHHHc(000-ee0200033333333
(31)
To analyze the equality condition, notice by the fact that minimizing over larger set leads to smaller
loss and the independence between 7, and G.,
B, Be. Voc
E (x) / hwwpal/hhhhgggzzaaaao//////eohea
= -Ep #ba.://leeek44r22c/00000//----a3 (32)
with equality if and only if
= - *P, V (o( Vz,,
E Bp. PA
by the orthogonality principle. As a result, the equality of Eq. 32 is achieved if and only if
(x) PA Po)
(33)
(34)
PAd (x.. = A (2),
(35)
41

---

Now, we turn our attention to the regularizer L, and notice that for any Us2 that satisfies Eq. 33-35,
Lr(s,, Ex (V 2220p) (x)]1 (e.p:| (ss)(()
2 PA -
where both equalities are achieved if and only if
Finally, we shall show that for any (, there exists some optimal solution (U,V,82,8c) (8)) such that
V===A
(36)
V (VV==sclg))sc((t)) ),()
(37)
To this end, consider the SVD of the operator A (z,) Aase(((z)))) as
Σg
0 Odzxdo
[
fz(z,t)
Yx,t, (2s)
2al
Odoxdz
(c(g(2).))
s.t.
-z=diag(o1 . Jdz
(5t)),
[z(r,t)
%
( 2. 2al =E fc(g(z).t) .t [f2(x,t) fa(g(x)))) 11 -I
Set U = 22 1/2 , (z. t) = E2/222(2 fz(2, t), V == D6El (z( = End fal t), we have (U,V,8,,8)) satisfies Eq. 36. Further, notice that
-T- 2 -1/² = -x =f
/ =sglt))))))))
-ーV=E//00000 =0= -0- 2/2² "/"== =x
VVV= - Rccl))) T - -ss((())))ss(
-
Therefore, (U,,,z,8c) also satisfies 37.
D.4
Proof of Theorem D.3
Define matrices
U
Ô T(
W(x,()== =
$2 (2( sa(g(x),t)
W(x.))
(xx()===
0dxx+)xddz1 Wo(,t)
Az
AG
A
("(x())= =
8 (x,1)) (1((),))
8 (x,1))
('
(x,x)))==
0dxx+1ixd2 W
N(z,t): W (z,t))((a.t) W (
((,,):= "."
Further, define the direction improveennt of W of (z,t)'s and W (z,t)'s respectively
A(,,):=((z:))) - (z.t))
(c.:):
R:= argmin
E
N:RT n=nR =lax
[ > [Wc(st)-
argmin
Emp(()
R:R R=RR =lax
42

---

where for any matrices M, define Mg to be its first d2 columns and Mc to be its (d2+1)-th through dx-th columns.
For any set of matrices {C(vv())lExxxeetor)) and probability measures d (v)'s, define random matrix C w.p. p(t)qe(y);
C=C
where p(t) is some fixed distribution of the diffusion time t. Further, define blockwise representation of C as
C
Co
C2
C1
c11
where Co is C deleting the last row and column.
For a pair random matrices (C1, C2), define defie
C₁ C2 C==CC(y,))Cill w.p. (1))))
Next, let [A.B] denote the bilinear form between random matrices (A.B) weighted by the operator K., and (A.B) [|. be inner product between random matrices A and B if IC is positive definite. Then we define the following bilinear forms between random matrices (A,B) with weight operators I, G and Hi respectively
(A.B)z = E, i (v) (A(,.t).((ytt)
[A. g : E, (w) (Ao( | (vn())))
[ Ho : (1)
It can be verified that (z z is indeed an inner product satisfying properties such as conjugate symmetry, linearity in the first argument and positive definiteness, and therefore we can define the norm | I == (A, In addition, it can be checked that l. and [, are conjugate symmetric and linear in the first argument. An important relation we use repeatedly later is the fact that
[CC.0l,.+ (C, |C
(38)
since
[C, (CC2)o+g==2(le (10
|1c, - 11222
Under these definitions, prove the following Lemma in Section D.5.
Lemma D.5. Let 7 ==:==A+r (0) with X, =3, and W be approximate critical point of i(W) so that (VL(W) A > cl A A for any random matrix A' Then the following holds
A Alel
- +|2++ell Alz
(39)
where v2f(W) denotes the Hessian operator of the functional f as W.
To proceed, we use the following lemma for random matrices analogous to Lemma 40 and 41 in [125] and defer proof Section D.6.
Lemma D.6. If x (X) TY (X)) is positive semi-definite (PSD) matrix, then for independent, identically distributed random variables X.X',
(XX -Y()))()))
max
-XX))UXX))
Further, we prove in Section D.7 the following lemma showing that gradient descent converges to local optimum of the objective İ.
Lemma D.7. The gradient flow equation Eq. 12 converges in probability to solution (U,V. 22.0).))
such that for some e > and random matrix A A((1):
( (W), A A'
A' A AA
43

---

To prove the theorem, first consider the singular value decomposition po) (a,t) (a,t))W((a)) ((z,t) f and by the definition of the direction of improvement,
R:=
argmin
IW- W N--RRR 2
RRT - R=Táx
((.t)) (W.W z= argmax (Beni(e) (a,tt) ((r.),
argmax RR =R R=lax
RR = R=lax
( -4)
argmax R R=lax
where the last equality holds since :² R. is orthogonal, |R $1
and
dx
E, 1
Exe
1=1
with equality iff R' = Tax- asaresult
(x W (z.t) = =.p, W (x.t)W((a)))
PSD. Applying Lemma D.6 (X) R ((X) W yields
Á T 1 - S2 - -Ñ' N - N - N 1 -
Combining this with Lemma D.5 yields
AA -0.8| Á
Therefore, we have the LHS to be positive only if / |z O. Set = 0 and applying Theorem D.2 yields -(U)) R(A2) R(() y-= = ((G)).
D.5 Proof of Lemma D.5
Define Lo NN then it can be verified that
[o)0)= - 1 N I N To
(0)= [N, A LWN
Define (W) (W), then we have
i^ (W) (W)+ 1 W
Ho + NNN
Then, consider the Fréchet derivative (,(W) along A (t), can be shown that
( Ar W A = N "WWT + +W INH, + (N + WA WA (V),
N AN-N AN WA 12 N AW + +WA I (WW))
= -N-N' +WA |2H,+g+ (W
(40)
where the last equality uses the fact that NN', W 3 No WW. 0.
=0 To this notice that
I N. la - (AAT ÚA + Eepp)))pttt (I,1)) 8 (1' ())((1())))))))))))))))))))))))) s
('' (c))))))))))55((((
E, (2)s (2,t) AT (z.t) (x) (I,t) A As(z,t)
=T<AA(AT A- Es (z (x,t)))()+
E E (x,t) (E (4' ttss (x',t)T [] A A A)s(z,t))=0 =0,
44

---

where the last inequality uses the fact that Ls w-) - 0 and therefore
U- = E, (z) +
V+ (
Similarly, we can show that [N',, www lg-
Now, consider the Hessian of L along A by taking the Fréchet derivative of Eq. 40,
|A = A WA
AW
WA
TEM G + CONNNNAA AA T +9
N WA +AW AAAAA.
= =W + WAT 7 + AA _ J2Ho+o+22)
WA
+AW WAT+AW
where we use Eq. 38 in the last equality. For the first term of the right-hand side, by the choice of A,
A
W
WA
- - - +A
AA
2 N" AW A +WA I- IN-NN 1
= AA
1 (W) A -
For the second term,
N--N
WN-N.AC N' AA
12 N WA AW 2 IN N N 12
12
=(Vi.~- (W) A) 2NN' - -z- (41)
Combined with the fact that
N' WA -W
we obtain
| -1 -A 1 12 (W
N, 19 -X, -
AAA
1 N N' 1FA-::111H..22......-A-AAAAAAAAA
T - N - No +
A -V AA (42)
where the first inequality uses the fact that
[ (AAT)+++++ (em (x.1) ((,,t))(( x<x', .t) 2E,
(
= =IU A px) (2, (z.t) 11220,
and the second inequality uses the condition that W is an approximate critical point of AM (W). It remains to bound the I.p related terms. Define N = W2, = W and similarly 2 and NG+ Notice that
N + 1 +NINGGG [ING
L, (W) N NNNNN-GNNN
= - NG -N 2 No
NG.,NGN No
Ho Z
"Ho Ho
1
N
No N
=,NG NC* N
N Ixo W
where the second-to-lase equality uses the fact that
N " Ha = 21E, -
= 34M2
45

---

where the second-to- equality uses the independence of the content variable Z and the style variable and the last equality uses the property of the score function:
Epz (=) Raz V. (oggz((2)
v. /
V=pz(=)d==
Therefore, we can apply Eq. 42 with A, - to obtain
A W)
A G G) (w)
(43)
S A A 1 -
(44)
Plugging Eq. 43 into Eq. 42 yields
A <
AA
1 -
+ellA
ققا 1 - -
12
A 7
where the last equality uses the definition of N,N A and A For example, for N, we have
WWT
WoWd A
A
IN =
N 2 + IN
112
|N
11
W
WoWd 3
9
3
N
where in the last equality we set X, =3.
D.6
Proof of Lemma D.6
First, let A(X) (YX)))))))XXX))))) definition,
and
Expanding the square norm,
E UIA
A(X))
2E(A(X)U(X)) +U(X)A(X)) A(X)A())) "
- -IEEA(X) A(X) | (XXX)//RAAXXX)XXX))XIIIIA33))))
+2IAA\) U(X))
((EA(X) [XX))
} |IAA(X) | A(X)) "
1
21 A(X)
V2
2
where the second equality uses the symmetry of (X) T A(X) the last inequality uses the PSD of (X) (X))
Similarly,
TEA U(X))
((X)) (((X)))AAXXX())) 4(X)+
A(X)))XX))))))) A(X)]
2 /TWp://AAAATTT---------/-/E/----
where the last inequality uses the PSD of EX) (X)..
46

---

D.7 Proof of Lemma D.7
The proof relies on the following lemma.
Lemma D.8. Suppose the score network parameters are initialized randomly as 0z 00)00 (0) ~ (O, Then du the NTKs K and Ka converge some kernels KG and KG fixed during training. Further, define the minimal eigenvalues of operator K (to, X x To + R. as
Amin (K)==
inf (1,[ B1 p(x)pppe) K (xx,t,, (traa,t)))((( (') (t)))) ,t),
then the minimal eigenvalues of K22 and KG satisfy min (Amin (22 ) (K)) 0.
Define the random gradient of loss L with respect to random matrix Y with probability density
VyL(Y)= Vy(a))))))))(()) w.p. p(t)
By the definition of the gradient flow equations in Eq. 12, we have
i(w) (V))<(Ww,.W)z
= --IIcd//))---VVV2V------
V II L L(W]??
where the first two terms of the RHS by the property of the gradient flow, vanishes if and only if the gradients L(W and Vv (LW) become O. For the third term of the RHS, notice that
[[ Jop sz V L(W)I2
-Err' (x)) V (2,) L(W) S2,,t()))) Jo 82(2 VVVldlttVVVVzieeelVVe oz
=Bi pa())p V V L() (2lll))(2((22app (2
2dmia (Kz)
(=x.t) (W)
By Lemma D.8, Amin (Kz) du d= Amin (K x3 0 and thus the term vanishes if and only if
(z)(t)p())) (e V (W)
Similarly, show that the gradient flow converges only if
(x(()) Ee V o)
For the second order condition, we use the well- result that gradient descent (with small noise) is able to escape saddle points almost surely [126].
D.8 Proof of Lemma D.8
Due to the symmetry in their forms, it suffices to prove the statement for 8z and Kz and omit the subseript when the context is clear. Large of large number and the standard theory on neural tangent
kernel [1 123 yields
K2 du Evec Exoteey (000(02)2((2 f'),
(45)
which stays fixed during training.
Define i:= [x: .PE(t)] and a()): (B(1)T Later, we will slightly abuse the notation to represent (x.t) when F appears in the true score functions. By definition,
K (x, ,1) - vec V (00)
EEvio
J
0 8z((')
-EN1O. (0 0 (())TT J (0 8
0l) a(')
-Exio.)(a(t) &l Tol) (a(F)) Iax ) = ( (
a(8))
47

---

where @ the Kronecker product. Notice that by definition K 1Z) = 0. Similarly,
(x,t,x' f) EN (er ) ,
=EN1o.nJa(a)s22(1) (o [
=: "ltTd()Eaa d')ENTO,1 (
=( 2'
where the independence between ,00,09' / to cancel out 02 and
[000))] "2 of
S (ā,,")
0, otherwise.
By assumption, we have (() are zero-mean Gaussians and thus
TT
arcsin
2m
|
where we apply the formula for bivariate Gaussian variable (N), Pr[N1 2 0,N2 0 arcsin(p) where p:= Cov( N₁ No) VVar(N, (ar(
As result,
K (",i")) T
2
5 ||ا|||
TTa
arcsin
Ldz
π
(1) Notice that K z a positive definite operator since for any finite of distinct samples X x= -=(āi....,.n with 4, Ht, for all i.j, the matrix
K ( (+t,,,))) > XT X Iaz 00
where we use |*||| > and PE(t,) is are linearly independent from the condition of the lemma, and thus their Gram matrix X X 0. Consequently, Kz K/ K 0 and therefore its minimal eigenvalue is positive.
D.9 Proof of Lemma D.4
As dHrdT + 00, we have =z= KG Kg. First, we prove the following lemmas in Section D.10 and D.11 respectively.
Lemma D.9. Suppose > max de), and min du} - oo, the following holds for the empirical risk minimizer (82,80,U, with probability at least - 2exp(-n))
Lz LV -8v)=0,
a (s )d//2
|| | SCzV
Amin (K2)
σ ) )dl
| SCas
Amnn (K))
for some constant Cz >
Lemma D.10. For the NTK of the estimated content and style score functions K2 and KG defined in Lemma D.7 and for any (æ.t) E X [t,,T), the following holds:
|K 2
[Kott,,
(1]]|=[==+||PE|)||}}}
48

---

Then we make use of the following properties of sub- gaussiann random variables from Lemma 16 of [41].
Lemma D.11. Consider a probability density function p(x) S exxp(-Cl<=l//2) for E R and
constant C>0. 0. Let R a fixed radius. Then the following holds
2dx
pla)dr
R
2
|0|
CT(d/2+1)
/ |
2d
p(x)dxs
R
CT(d/2+1)
Define pz := Czy
Ammm (K2) and without loss of generality, assume suPre[to.2]] |PE((|) S T. To
Amm
prove lemma D.4, we first bound the Rademacher average of Sg as
(n E.n
sup sg (
s (x*))
dz Ci (gl K K (, g)xxx Ben
sup E,n
sup
$2.j
Cij K Z
eECz (i)
oECz (atmm) z=1
i=1
dz
dz
P7
E.
Eit K 2 w
PZ
E
K
2 J
i=1
Kz
P2
)p:) Pr: le')Ee, c, K2 (6.
S
Pz
Σ (zt))
Averaging over 3nhh and applying Cauchy-Schward inqquality we can bound the Rademacher complexity of the data-dependent function class Sz as
Eg Rr (Sz) = Egg0 Pz
P2
E (e (K(2(()))))))
P2CK2 vn
Vn
where CKN / (aLpp(()))))()) Tr(K (T(K2
To proceed, let the content subspace matrix and score function class learned by solving Eq. 12 given training data [i' ....,,' be (( and Sz(il 1=n) respectively, then by Lemma D.6, with probability least 1 2exp( ((())))))))) the event
E S
(46)
will occur, which implies with the same probability bound, the empirical risk minimizer over Sz (x satisfies
L.z (Ú,8))= (U,sz)eU(() xx (atm))
min
Σ
Eez (z' = min "E2((' U.aeES2 n i=1
Therefore, let an empirical risk minimizer of Lz over Sz be ú' ,,/2, then for any > 0, we can bound the generalization error probability as
Pr Lz (Z((082) (U,zz)elxSz
min
Lz (2(Us22)+
<Pr(E)+Pr (z((",,8) .82))
min (U,z))uXXSz
L Lz(U,zz)+e
(47)
<Pr Lz((" (()))))))
min Lz(U,s2) +E + +2exp(-(((n))
EUx
49

---

Next, since the squared loss Eg is not Lipschitz with respect to z, we apply a truncation argument on Lz. To this end, define the truncated version of L as
L (U.sz):= (z, ||
for some radius R> 0. Similarly we can define its empirical version as L Then Ly admits the following decomposition:
L2 (32)) (Z(
= =L mune (0', trunc
L7 Hune (Ū" (0,,) LZ Linn (A2,87) ( L (0',82) m2 ( (U. (U, L2 (2)
Lz
(()
To bound (1), notice that for 8z E Sz, with balanced weight U U'u
((I,,tUUU,))) 1 | ((x,;; |
sup sup TEX. [ |=
< sup sup (U..))
sn 2 (z. [| +T2)
sup sup meX.:c((o..[(Usss383x..
=(3p2Amin (K)) 223) TT)
where the second-to-last inequality uses the Lipschitz property of 8
| Maa +T
(48)
where the last inequality uses Lemma D. as follows:
|Usz(ı,,t] > <
dz Spzv Amin (Kz),
ssz. K ( k2
(49)
S pz v Amin Kz)
82 K2 15 K2
3
22 Amin (xz)
K2 (2(2,,4 5
Amin (2 [
Using Eq. 48-49, can show that
[Usz(E') A2s2( | (
||a A| (lll
< < 3 (K2)/2+ R
Let Agrnne ==2(p2 3dmin (0/mminKK2/222 (K2)/2+
R22 +7 and use the inequality for max fooox||ss/o [|s/,/[]} SA,
we deduce that
U' {z)))((; ((,(("UUss)))))) [ Amrnel || A2s (i')ll,
and thus {Zt is Amnne Linsci-iipschit of U [ (z))
Denote
US : UUs==
50

---

and apply the standard generalization bound for Lipschitz function on L (see, e.g., Theorem 9.1 of [127]) conclude that with probability least 1,
(1/g(1/6)
(1) 2Bn Rm, (noUSSS
V
Avv R2 + T
B
8log(1/6 (16)
< 2mmmm
(US2)
Vn
2
130Kz +A V R72
8log(1/6)
< 2Ammm
Vn -0 Asp -o(>ap2 (R2
2
log1 /6
-0
AadzTeR2 vn
log1/
1/8 n1
+ t.op7 RR² +
+h,
R²
vn
n
where the second inequality uses triangle inequality within the definition of Rademacher complexity followed by the bound Eq. 48, and the third inequality uses the contraction principle of Rademacher complexity [127].
To bound (ii), notice that
(ii) B Pe (2) |
$ 2E, m))
(10 |
< 2E px())
(1,", |>||>
3
2 2++)) 1 11112
<
2 2P2
= (3p2 Amin
|=)||=1 | (x))
111
(3 (K22) dxx
exp( exp /1
7(dx/2+1)
3
-t())
-xx/++1
5 P2 Amin (Kz)dz |P((||
11xx -2 expl RR 1121200x)
3(o'
T(dx/2 [(dx/2+1)
=0 2 dxx/241
Rdd
expp-RR2/200 /20
T(dx ((dx/2+1)
where the second-to-lase inequality uses Eq. 48 and the last inequality applies Eq. 49.
Take R:=0(v dx logdx + logn) such that (fi) < T2 and then combining (i) and (ii) yields with probability least 1-5.
Lz L 1+ log1/8
(d +logn)
(50)
Vn
Setting 8:= 1 and combining Eq. 47 and Eq. 50 yields with probability at least 1 (2))
1 (+glog
(dx/2 /g/+++og))) + -o
dā
Lz(0,Sz)-Lz
Vn
Combining this bound with Fq. 47 yields the desired bound. And similarly, we can prove the bound for
åG.
51

---

D.10
Proof of Lemma D.9
As dæ dr -2000 we have K2 Kg== By the optimality of $2 and $6 and Theorem D.3, we have
(82(x',)) A
2 (=)
(51)
( +
The first two equalities immediately yield
L2 Lo Lo =v)
To prove the last two inequalities in the lemma, let E,,, [fxx = Ei Then Eq. 51 implies that
Ep. (z.85)
B.. (r( Tr (B
-Tr(Ep, (2, S2 = (tt (E..l//))))((ll. (Ep.1182(z,t)11))
(52)
Let
s 1)) Es)
and (M) be the i-th eigenvalue of the matrix M, then we have
Ep. ||.1 $2 (z, Ep, || (2,
Ep. || (x,
(53)
Ae Ae (52)) S | (82 (8%))
Further, suppose 8z (z) = Kz (ā, (z,5") "0,,""')c(("" (ī*) and define
Kz ..1)
Kz ( "")
Kz:-
1. :
Kz Kz(i",t")
Kz(3",t") [m)
then we have
C K23C2
|| 16
- E2KZzz -
K (x' _e (F))cz(f (+-))
nd, (Kz)
ndn(K2)
==1
1=1
v || ll dz
An (k2) Bp. 82 (z)1
E
Ppn || 1
Amin (Kz)
Amin (Kz)
remains to prove the concentration of the operator norm |X(s))llap; To this end, use the assumptions that sz (x,() is A and (z,() is sub- -abbsaa with sub-gan gaussaas norm most Øzdz + T2 to conclude that s3, t) is uab-gaussian with sub- gaussian norm at most X, (0 T2). Then Theorem 4.7.1 of [128] yields with probability at least 1 2xx,
(s,())) around || |Eo (+2)
(v
(vrtu rru r+u n
-
Ex (s E(8)) SC
| [1($)))
(54)
n
where r:= Tr(E(s))) the stable rank of (s>)) Asaresult with probability at least 1 (-2xpe(xn))
Cz|1E(s 11////////// 2//4
1/4
(1)5))
82 1x 5 Amin 1/2
Czy
Auni SminkKy) (k2)
52

---

where C2:= C2= suffices.
Finally, using similar argument show that
01 Amiu (KG)
|| SCo/
with probability at least 1 1-2exp(-99)
D.11 Proof of Lemma D.10
As Asd dr -> 0o, we have =z= K2 K K. By definition of the NTKs,
|K (2,t,x,t)
EN(o.,) EN(or))la((,t))1==
||
Ex(o,,))a(z,t)]] ||:1 d2
du
lim
E Exion (2 (0 ( +))+
1 2 dz
du
dH
dir
lim du +00
Exion
(1)T 8)+ T
|0|1
d2 =
||
||
3
|
dH
2
2
Similarly, can prove the bound for Kg-
Table 2: Quantitative results for image disentanglement on MNIST and CIFAR10 test sets. MSE, PSNR and SSIM stands for mean squared error, peak signal-to-nov toise ratio and structural similarity respectively between the generated and target samples. LPIPS [75] a perceptual metric based on features from deep image classifiers. The results are averaged over two random trials.
MNIST
CIFARIO
MSE(L)
PSNR(1)
SSIM(t)
LPIPS(J)
MSE([)
PSNR(1)
SSIM(t)
LPIPS(
A, 0
0.1020.01
5.210.2
0.42
0.30
0.44
3.5_0.2
0.05.2.00
0.62
A, 10-00.0 0.03
0.07
9.642.
0.53
0. 0.1820.04
0.35
4.4-0.1
0,05-2..0
0.61
A, = 0.3
0.01 =0.00
16.0+a.3
0.66
0. -0.00
0.1820.06
7.3-p.4
0.06
0.53+g.oz
A =3
0.01 10.00
17.3_0.
0.66
0. 1..00
0.11
9.3
0.07
0.49
E Experiment details
E.1 Latent subspace GMM disentanglement
For the synthetic disentanglement experiment, we choose the subspace dimension be dz dG =5 and sample the content variable via ~~ NN(( Lā ) ++ [N( ,oālaz) and the style variable via
G 1. (Nf o La [ug L where = 0.1. In this way, we generate i.i.d 4000 samples for training.
We follow the network architecture shown in Figure 2 with dи 512. For the time embedding PE(-), we opt for Gaussian Fourier projection layer to encode temporal in- formation between (0,1] defined as:
Table 10: The default noise sched- ule hyperparameters for the synthetic data experiments. Continuous time (1 [10-5,1] is used the expression.
sin (2x(2t) cos (tT
PE(()
Name
aft)
(
where Ω ~ (0s12 9000 We train the models for 10,000 steps with an Adam [131] optimizer with learning rate 10-5 and batch size equal to the entire training set.
00.00----00000 1000-.075
1 - 8.
sub-VP
(1 9.00 y
(cosine)
1-c --
53

---

7
O
7
7
4
0
7
0
4
3
O
7
O
5
7
2
9
1
2
7
2
9
7
2
8
2
9
7
3
2
7
0
/
6
7
2
7
6
4
7 3
7
7
8
4
7
1
6
7
8
4
3
8
6
6
3
6
3
6
3
6
6
O
G
9
6
0
5
4
Y
7
9
6
0
5
4
7
G
9
6 9
O 4
4
K
6
5
5
4
G
9
2
/
9
4
8
7
G
2
/
4
8
9
2
7
4
7
Z
3
7
4 9
9 O
3
4
9
2
8
O
5
4
L
6
7
5
5
4
6
9
O
5
8
4
2
5
(a) Input
(b) Baseline (A, 0)
(c) Colorized (A, 3)
(d) Reference
8
6
6
7
8
8
5
7 7
8
S
S
8 S
R
0
6
4
7
0
6
4
6
3
D
/
6
3
0
6
7
8
2
0
a
9
9
7
8
2
0
9
9
7
8
2
0
A
9
9
8
2
0
6
5
5
0
3
4
5
5
6
3
5
5
5
4
0
4
6
5
5
4
L
6
5
4
6
5
4
6
5
6
5
4 2
4
6
S
5
4
4
4
7
2
3
1
7
2
3
2
7
2
3
7
4
4
7
Z
3
8
5
O
8
5
8
O
8
5
8
4
2
5
/
3
2
5
0
2
0
2
5
(c) Input
(f) Baseline (A. 0)
g Colorized (A+ 3)
(h) Reference
0
9
D
3
6
4
2
0 9
D
3
4
2
9 D
3
1
6
2
O 9 D
3
6 4 2
6
/
{
3
9
3
6
3
9
3
6
5
6
/
3
2
4
5
9
3
9
0
2
S
3
9
0
2
9
5
9
3
0
2
4
9
3
0
6
5
5
7
2
2
7
3
6
5
5
3
2
7
3
6
5
7
3
2
7
6
5
5
7
3
2
7
8
7
/
4
7
3
2
/
3
%
7
8
7
2
2 3
4 0
8
9
2
2
4
8
9
2
2
4
8
7
2
2
5
9
P
7
S
P
7
3
O
P
S
9
P
4
9
5 7
4
7
2
2
(i) Input
Baseline (A,- ()
(k) Colorized (A,
(I) Reference
Figure 6: More image colorization results on MNIST
To ensure convergence. we pretrained the speaker score network 3 We experiment with various noise schedulers, including the variance exploding (VE), vanilla variance preserving (VP) [3 sub-VP [74] and cosine VP [132]. The detailed schedule hyperparameters are listed Table 10 and are chosen based on rules of thumbs in [133, 74]. To evaluate the subspace recovery, we use the subspace recovery error defined as:
A(U,V,AZ,AG)= 1 (1)Pu - 1a0-PAzl 1 PAo 11:),
(55)
2dx
whose range is [0,1].
E.2 Image disentanglement
Forall experiments, use Gaussian Fourier projection layer to encode temporal information between (0.1). Further, the U-Net architecture used as the score function for MNIST and CIFARIO are shown in Table 3 and Table 4 respectively. To capture the content information of the image, we use the 16 x 16 feature map from a pretrained vit mat.-aaaccacch-l---------
variant of the DINO model [56]. The feature map is then resampled to the same size as the image.
For both datasets, we train the DM using an Adam optimizer [131] with batch size of 128 and learning rate 10-4 for 50 epochs. A VE schedule is used during conditional score matching. During inference, we use probability flow [74] with 500 steps to perform sampling. For the CIFAR denoising experiment, we feed all-zero matrix as the noise map. All models are implemented in Pytorch [134] on two A5000 GPUs. The training time approximately an hour for both datasets and the inference approximately 10 seconds for 64 samples.
54

---

(a) Input
(b) Baseline (A, 0)
(c) De-noised (A, = 3)
(d) Reference
(e) Input
(f) Baseline (Ar =0)
(g) De-noised (A, 33
(h) Reference
(i Input
(i) Baseline (Ar h-=00
(k) De-noised (A,- =3)
(l) Reference
Figure 7: More image denoising results on CIFARIO
Full quantitative results on MNIST and CIFAR10 are shown in Table 2. And more examples are shown in Figure for MNIST and Figure 7 for CIFARIO.
55

---

Table 3: U-Net score network used in the MNIST colorization experiment. The input is x 28 x 28 created by stacking the image, the projected and resized DINO feature map to 3 x 28 x 28 and the background color vector broadcasted
x 28 x 28. a- denotes that component a accepts hidden embedding from the previous layer and component b accepts the time embedding, and the outputs of and are added with proper broadcasting. "Conv2d" refers to 2-D convolutional layer, "GroupNorm" stands for group normalization layer, and "ConvTrans24 Trans2d" stands for 2-D transposed convolutional layer.
MNIST U-Net
x Conv2d with stride (DINO projection layer)
9x32x3x3 Conv2d 3onv2d with stride 2 1024 x Linear
GroupNorm with 4 groups Swish activation
32x x Conv2d with stride 2 + 1024 64 Linear
GroupNorm with 32 groups Swish activation
64x Conv2d with stride 2 1024 128 Linear
GroupNorm with 32 groups Swish activation
128 256 x33 Conv2d with stride 2 1024 256 Linear
GroupNorm with 32 groups Swish activation
x56 128x 4 ConvTrans2d with stride 2 1024 x 128 Linear
GroupNorm with 32 groups
Swish activation
256 x 64 ConvTrans2d with stride 2 1024 64 Linear
GroupNorm with 32 groups Swish activation
x2 x ConvTrans2d with stride 2 x Linear
GroupNorm with 32 groups
Swish activation
64 x 3 x x Trans22d with stride 1
E,3
Speech disentanglement
The dataset statistics are shown in Table 5. The overall results for realistic datasets are shown in Table
For the IEMOCAP dataset, we use a system available on SpeechBrain [135] that finetunes on the wav2vec 2.0 backbone [136] with a multi-layer perceptron classifier (MLP) [137]. The classifier is trained using Adam optimizer for 30 epochs with a batch size of 4 and a learning rate of 10-1 for the MLP and the 10 - learning rate for wav2vec 2.0 weights. The system is then evaluated using the standard classification accuracy metric and 5-fold cross validation [76, 138]. For each fold, we use all 8 speakers from the training set as target speakers.
On the ALS and ADReSS, we use whisper- medium [129] features, as they have shown to be the most effective for speech impairment classification [139]. To avoid unfair comparison, We concatenate hidden representations over all layers of the whisper-medium encoder rather than selecting a particular layer and perform mean pooling over the frame-level features. For both datasets, we follow the standard splits used in previous works [79] to have no overlaps between speaker in the training and test sets. And for both datasets, we use the most frequent speakers from the training set as target speakers for the VC to achieve maximize conversion quality via better speaker representation.
56

---

Table 4: U-Net score network used in the CIFARIO denoising experiment. The input x 32 created by stacking the image, the projected and resized DINO feature map to 3 x 32 x 32 and the noise map broadcasted to 3 32 32. dual encoder with separate U-Nets for the content and the style variables is used and a + denotes that component accepts the content embedding from the previous layer, component accepts the previous style embedding and accepts the time embedding. Further, the outputs of and the outputs of b,c are added separately with proper broadcasting, a-b here denotes that a accepts the previous content embedding, b accepts the previous style embedding and the two outputs are added.
CIFARIO U-Net
384 6 x x Conv2d with stride (DINO projection layer)
Conv2d with stride 2 + 1024 x 32 Linear
GroupNorm with 4 groups Swish activation
32x64x3x3 32 64 Conv2d with stride 2 + 1024 Linear
GroupNorm with 32 groups Swish activation
64x 122222xxxx2222 64 128 3 3 Conv2d with stride 2 + 1024 x 128 Linear
GroupNorm with 32 groups Swish activation
228x256x3x22 128 x x 3 Conv2d with stride 2 + 1024 256 Linear
GroupNorm with 32 groups Swish activation
256 x128xx44255 128x4x 4 Trans22 with stride 2 1024 x 128 Linear
GroupNorm with 32 groups Swish activation
64 256x64x4x4+256 64 4 Cony Trans2d with stride + 1024 64 Linear
GroupNorm with 32 groups Swish activation
128x32 4x++ 128 x 32 x 4 Cony Trans2d with stride 2 + 1024 x 32 Linear
GroupNorm with 32 groups Swish activation
64x3x3x3+64x Trans22d with stride
Table 5: Datasets and VC-adapted classifiers used during realistic data experiments
Feature
Classifier
#Classifiers
Reference
VC
DMM-hsed
Reference
JEMOCAP
wav2vec 2.0 base
MLP
[76]
THAAN-VC
No
[77]
ADReSS
whisper-medium
SVM
[78]
KNN-VC
No
ALS-TDI
5
whisper--meiumm
SVM
15
[79]
Dif--V
Yes
[38]
We apply the VCs in mostly a zero-shot, plug-and-play fashion, and leave finetuning specific datasets for future works. For the DCC we use the publicly available score network and vocoder checkpoints trained on LibriTTS and adopt the original inference hyperparameter settings for all experiments. Similarly, we use the pretrained models and for other VC models. Further, we use a maximum of 120 second speech from the target speaker to compute the target speaker embedding for all models except KNN-VC, where we use all the target speech as the pool for nearest neighbor search. We also compare VC adaptation with common data augmentation technique such as pitch shifting, where we shift the pitch of all the speech utterances to equally spaced pitch levels over the F0 range of the training speech
57

---

Table Overall results on realistic datasets. All metrics are between 0-100. single (average); B:
single (best); MV: majority vote; SMV: soft majority vote.
Impairment
Emotion
VC Vyte
ALS TDI, FI
ADReSS, FIT
JEMOCAP, Acc. (5 f
B
MV
SMV
B
MV
SMV
A
B
MV
SMV
No VC
54.9
54.9
54.9
54.9
70.6
70.6
70.6
70.6
71.5
71.5
71.5
71.5
Pitch shifting
55.8
60.3
57.6
61.5
71.2
77.1
77.1
68.8
60.6
55.1
61.1
61.1
KNN-VC
55.8
61.7
64.8
49.9
71.5
79.2
79.2
83.3
70.4
69.3
71.4
71.5
TrAANN-C
55.7
60.7
61.7
53.3
72.4
75.0
83.3
63.1
64.1
67.2
DCC
47.0
51.2
50.3
49.2
65.6
69.4
66.7
70.8
87.0
94.3
96.5
97.2
Table Emotion recognition results on IEMOCAP. speakers in the training set of each fold are used target speakers. WER stands for word error rate computed using the whisper-large- model [129] and lower is better. MOS stands for mean opinion score computed using UTMOS [130] and higher is better.
Accuracy
VC type
Voting type
WER (L)
MOS (t)
2
3
4
5
Avg.
No VC
72.6
76.6
68.9
68.9
70.3
71.5
9.0
2.3
single (best)
64.0
65.3
57.4
57.7
58.6
60.6
single (avg)
61.3
62.1
50.2
48.9
53.0
55.1
Pitch shifting
36.4
1.3
majority
61.2
65.3
58.5
57.8
62.5
61.1
soft majority
60.8
65.4
57.8
57.5
61.5
61.1
single (best)
71.2
75.4
68.3
71.9
69.1
70.4
single (avg)
69.6
72.6
67.0
69.9
67.4
69.3
KNN-VC
9.8
1.6
majority
70.3
75.6
68.5
72.8
70.0
71.4
soft majority
70.3
76.1
68.5
72.8
69.9
71.5
single (best)
65.5
66.9
63.0
67.5
62.6
65.1
single (avg)
64.6
66.3
61.1
65.6
63.1
64.1
TH
16.6
1.9
majority
66.9
69.0
63.9
67.9
66.5
66.8
soft majority
66.6
69.9
63.6
68.8
67.0
67.2
single (avg)
87.0
88.3
86.2
87.6
86.1
87.0
single (best)
94.4
94.8
94.2
95.2
92.9
94.3
Diff-VC
22.9
2.5
majority
97.5
96.7
95.2
98.1
94.9
96.5
soft majority
97.7
97.6
96.3
98.7
95.6
97.2
data with levels equal to the number of target speakers and train separate classifiers for each level as in the case of using VC adaptation.
For ALS severity classification as shown in Table 6, KNN- VC achieves the best performance among the VCs, reaching 65% macro-F1 with 15 target speakers and hard majority voting, compared to 54.9% when training without VC adaptation and 61.7% with pitch shifting. For cognitive impairment detection as shown in Table TriA VVC performed the best followed by the KNN-VC method, both achieved 83.3% macro-F1 with soft majority voting, which is 12.7% better than methods without VC. adaptation and 14.5% and 6.29 2. better than the pitch shifting adaptation using hard and soft majority voting respectively. On JEMOCAP, we found that Diff-VC performs the best, reaching an average of 97.2% accuracy, which 25.7% better than the VC-VC classifier and 36.1 1% than the pitch shifting adaptation. Though a phenomenon out of the scope of predictions by our theory, we hypothesized that such "specialization" of the VC methods due to the different level generalization ability of different VCs to latent variables other than the speaker identity, such as recording conditions and health conditions of speaker. For instance, Diff-VC does not perform well on ALS compared to KNN-VC, probably due to the domain mismatch between the health conditions of its training set, which contains little pathological speech, compared KNN-VC which uses the WavL. representation trained on much larger speech dataset with diverse speech.
58

---

Table 8: Alzheimer detection results on ADReSS
type
Voting type
Precision
Recall
F1
Accuracy
No VC
71.4
70.8
70.6
70.8
single (avg)
71.8
71.4
71.4
71.2
single (best)
77.1
77.1
77.1
77.1
Pitch shifting
majority
77.1
77.1
77.1
77.1
soft majority
68.8
68.8
68.7
68.8
single (avg)
71.8
71.5
71.4
71.5
single (best)
80.0
79.2
79.1
79.2
KNN-VC
majority
79.4
79.2
79.1
79.2
soft majority
83.6
83.3
83.3
83.3
single (avg)
72.5
72.4
72.4
72.4
single (best)
75.2
75.0
75.0
75.0
TH
majority
77.5
77.1
77.0
77.1
soft majority
83.3
83.3
83.3
83.3
single (avg)
65.7
65.4
65.4
65.6
single (best)
69.4
69.4
69.4
69.4
Diff-VC
majority
66.7
66.7
66.7
66.7
soft majority
72.2
70.8
70.4
70.8
Table 9; ALS severity classification results on ALS-TDI with a whisper-medium-SSVM classifier
VCtype
Voting type
Precisiont
Recall
Fif
No VC
59.8
53.7
54.9
single (avg.)
60.5
54.1
55.8
single (best)
67.4
57.7
60.3
Pitch shifting
majority
73.0
54.9
57.6
soft majority
68.4
59.0
61.5
single (avg.)
58.5
54.6
55.8
single (best)
65.7
59.5
61.7
KNN-VC
majority
67.9
62.9
64.8
soft majority
51.1
49.6
49.9
single (avg.)
60.2
54.5
55.7
single (best)
68.0
58.2
60.7
TrAAN--C
majority
69.9
59.1
61.7
soft majority
54.1
52.8
53.3
single (avg.)
48.2
47.0
47.0
single (best)
53.0
51.0
51.2
Diff-VC
majority
49.8
50.9
50.3
soft majority
50.1
48.8
49.2
As to the advantage of hard vs. soft voting, we observe different trends across different datasets and VC methods, On ALS- hard voting works better than soft voting by 8.4% and 16% for the best methods DC and KNN-VC, though worse by 3.9% and 1.3% for pitch shifting and Diff-VC. On IEMOCAP, the gap between soft and hard voting is negligible, with soft majority voting shows 1.1% 7% edge over hard majority voting across VC methods. On ADReSS, we found soft voting methods to be better than hard voting for all the VC methods by 4.1% 6.2%, while worse for the pitch shifting method by 8.3% (68.8% vs. 77.1%). Since soft voting uses random classifier for voting, it tends to perform well when the model is "confidently" correct and "hesitantly" wrong, as it puts more weights on confident classifiers than hesitant ones. This suggests that the average confidence score estimated in terms of the classifier posteriors incorrect examples will be high for classifier
59

---

ensembles that perform well with hard voting than soft voting. Table 7 8 9 show the complete results for the realistic dataset experiments.
60


Reviews:
[{"review_id": "CUreTCjMh2", "reviewer": "Reviewer_KBNb", "summary": "This paper establishes a theoretical framework for diffusion model based disentanglement under weak supervision. The authors address key challenges caused by the stochastic and non-invertible nature of DMs and propose two formal criteria for approximate disentanglement. They provide theoretical guarantees for both content-style disentanglement and multi-view disentanglement.  Experiments across synthetic, image, and speech data.", "questions": "Dose the method works on CelebA? may qualiative results and quantitative results compare with some recent methods [1,2] will make this work more solid.\n\n[1] Ren, Xuanchi, et al. \"Rethinking content and style: exploring bias for unsupervised disentanglement.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n[2] Wang, Zhizhong, Lei Zhao, and Wei Xing. \"Stylediffusion: Controllable disentangled style transfer via diffusion models.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.", "limitations": "See weakness.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 1, "originality": 1, "strengths_and_weaknesses": "Strengths:\n- Well-Defined Metrics: Introduces new formal definitions for approximate disentanglement ((ε,ν)-disentanglement and ε-editability) suited to diffusion models.\n- Well-structured theory​​: Clear progression from general identifiability (Sec 4.1–4.2) to specialized ISM convergence (Sec 4.3).\n\nWeaknesses:\n- The ISM assumption may not hold for complex data. Experiments are mostly small-scale and controlled; there’s a lack of large-scale, diverse benchmarks that would test generalization in more realistic scenarios.\n- Limited novelty: The idea of employing mutual information-based regularization to promote disentanglement in diffusion models has been previously explored, notably in DisDiff [1].\n- The work lacks empirical comparisons against existing content-style disentanglement approaches, making it difficult to assess the relative effectiveness of the proposed method.\n- The title may misslead the readers, since the paper mostly focuses on content-style disentanglement, which is different from the general disentanglement [1,2,3]\n\n[1]  Yang, Tao, et al. \"DisDiff: Unsupervised Disentanglement of Diffusion Probabilistic Models.\" Advances in Neural Information Processing Systems 36 (2023).\n\n[2] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International conference on machine learning. PMLR, 2018.\n\n[3] Chen, Xi, et al. \"Infogan: Interpretable representation learning by information maximizing generative adversarial nets.\" Advances in neural information processing systems 29 (2016)", "was_revised": true, "final_justification": "Most of the problems have been solved except for the following two points. First, I don't know whether this method can really work on datasets like Celeba. For example, the authors should present some qualitative results of separating the angle and ID of the face. From the experimental results provided, it still doesn't compare with some more recent competitive methods. Second, I haven't seen the final version of the paper yet, so I don't know whether overclaiming and giving credit to other work have been properly resolved. I think I'll keep the rating for now.", "rebuttal": "Dear reviewer KBNb,\n\nThank you for the thoughtful feedback and for acknowledging the strengths of our paper. Below, we address the main concerns raised:\n\n**1. On the ISM assumption and generalization to complex data**\n\nWe agree that ISM represents a simplified model; however, we emphasize that ISM is only one part of our broader theoretical framework. Sections 4.1 and 4.2 provide results under general **nonlinear, non-invertible** mappings with only subgaussian and Lipschitz assumptions (see Theorems 4.1 and 4.2). ISM is used in Section 4.3 specifically to establish stronger guarantees such as **finite-sample global convergence**, which are rare in existing literature. The theoretical contributions are general.\n\nFurther, even though the ISM assumption does not always hold for raw data, such structures often emerge in the latent space of self-supervised representations. For instance, previous study [1] has found that self-supervised representation models have learned orthogonal subspaces for content (phonetic) and style (speaker) information, as mentioned in Section 4.3 of our paper. Further, practical algorithms making the ISM assumption have recently achieved competitive results in disentanglement tasks such as voice conversion [2] and image editing [3], demonstrating the applicability of the ISM assumption to complex data.\n\n[1] O. D. Liu, H. Tang, and S. Goldwater, “Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces,” in Interspeech, 2023.\n\n[2] Kamper, Herman, Benjamin van Niekerk, Julian Zaïdi, and Marc-André Carbonneau. “LinearVC: Linear Transformations of Self-Supervised Features through the Lens of Voice Conversion,” 2025.\n \n[3] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. Dimakis, and S. Shakkottai, “Solving linear inverse problems provably via posterior sampling with latent diffusion models,” in NeurIPS, 2023.\n\n**2. On additional experiments (e.g. CelebA)**\n\nThe main contribution of the paper is to develop a theoretical framework for general diffusion model-based content-style disentanglement, and our experiments are intentionally structured to validate different settings (content, multi-view, and ISM) in a controlled yet representative manner. As noted in the paper (Section 5 and Appendix E.3), we also include real-world evaluations on speech tasks (e.g., IEMOCAP) using voice conversion and emotion recognition to demonstrate applicability beyond synthetic data.  \n\nWe have further verified our theory on two more disentanglement tasks on the CelebA image dataset. First, we perform **denoising** with the same setting as CIFAR 10 and compared our approach with two standard baselines, infoGAN [4] and beta-VAE [5] adapted to the content disentanglement setting by conditioning on the observed style variable, which are representative of the approach taken in the papers suggested by the reviewer. As shown in the Table below, we observe the same trend as our previous experiments. \n| **Method**                         | **Style Guidance Weight** | **LPIPS (↓)** | **SSIM (↑)** |\n|-----------------------------------|----------------------------|--------------------|---------------|\n| conditional InfoGAN\t\t| -\t\t| 0.83 \t| 0.16 \t|\n| conditional beta VAE\t| -\t\t| 0.57\t| 0.17\t|\n| DDPM \t\t\t| 0\t\t| 0.53\t| 0.18\t|\n| DDPM (ours)\t\t\t| 0.3\t\t| 0.47\t| 0.19\t|\n| DDPM (ours)\t\t\t| 0.03\t\t| 0.35\t| 0.23\t|\n| DDPM (ours)\t\t\t| 3\t\t| **0.33**\t| **0.24**\t|\n\nFurther, we perform **face editing** with gender as the style variable. As shown in the table below, our method is superior to the conditional GAN baseline in terms of style transfer (CLIP score) and content preservation (LPIPS score).\n| **Method** | **Style Guidance Weight** | **CLIP Score (↑)** | **LPIPS (↓)** |\n|----------------|----------------------------|--------------------|---------------|\n| conditional InfoGAN|-\t\t| 0.241\t  | 0.39\t\t|\n| conditional betaVAE | -  | 0.254 |\t0.37 |\n| DDPM | 0\t | 0.250 \t\t| 0.35          |\n| DDPM (ours) | 3 \t| 0.259              | **0.30**          |\n| DDPM (ours) | 6 \t| 0.259              | 0.32          |\n| DDPM (ours) | 9\t| **0.261**              | 0.38          |\n\nMoreover, we plan to perform larger-scale experiments on more diverse datasets in the future.\n\n[4] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International conference on machine learning. PMLR, 2018.\n\n[5] Chen, Xi, et al. \"Infogan: Interpretable representation learning by information maximizing generative adversarial nets.\" Advances in neural information processing systems 29 (2016)\n\n**3. On empirical comparisons with content-style disentanglement methods**\n\nWe appreciate this suggestion and agree that comparisons with existing baselines would strengthen the empirical section. However, as our primary focus is theoretical, our experiments are designed to validate the **theoretical predictions** (e.g., convergence trends, editability, MI bounds) in a controlled manner.\n\nThat said, we **do include comparisons** with several baselines in the speech domain (Fig. 5), such as pitch shifting, TriAAN-VC, and nearest-neighbor VC methods. In our response to 2, we also compare our method with two standard baselines for disentanglement, conditional InfoGAN and beta-VAE. For image data, direct comparisons with methods such as StyleDiffusion [Wang et al, ICCV 2023] are non-trivial due to differences in task setup and model supervision, but we plan to incorporate these in future versions.\n\n**4. On novelty compared to DisDiff and mutual information-based methods**\n\nWe respectfully disagree that our contribution lacks novelty. While DisDiff [Yang et al., 2023] introduces MI-based regularization for DMs, it is a **fully empirical** work. Our work makes the **first theoretical contributions** toward **identifiability and convergence guarantees** of DM disentanglement under stochastic, non-invertible mappings – challenges not addressed in DisDiff. Our formulation of **$(\\epsilon,\\nu)$-disentanglement** and **$\\epsilon$-editability** (Defs. 3.1-3.2), and the theoretical results (Theorems 4.1-4.5) including finite-sample convergence, are novel and unique to our paper. \n\nMoreover, our use of MI regularization is not merely adopted from prior work but is grounded in our theoretical derivation to achieve approximate identifiability. These elements distinguish our work both in terms of scope and depth.\n\n**5. On title scope**\n\nWe will change the title to “Can diffusion models disentangle content from style? A theoretical perspective” to limit the scope of our theory to content-style disentanglement. \n\n---------------------------------------------------------------\n\nWe thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions."}, {"review_id": "guqU3CcRNa", "reviewer": "Reviewer_qfQD", "summary": "This paper proposes a theoretical framework for learning disentangled representations using conditional diffusion models under weak supervision conditions, such as partial labels and multiple views. While most prior work on identifiable representation learning has focused on deterministic, invertible mixing processes, diffusion models inherently involve stochastic, non-invertible mixing processes. To address this, the authors propose approximate disentanglement within conditional diffusion models under partial supervision or multi-view scenarios, supported by theoretical guarantees derived from Lipschitz continuity assumptions. The theoretical results are further validated through empirical experiments involving Gaussian mixture models, MNIST colorization, and CIFAR-10 image denoising, demonstrating effective disentanglement capabilities of the diffusion models.", "questions": "- In Equation (10), what is $U$ and $V$? They are not explicitly defined in the manuscript. As Equation (11) only optimizes for $\\theta$ and not $U$ or $V$, it seems they are not learnable components. Are they some kind of pre-defined matrix?\n\n- Could the authors clarify the meaning of $n$ in Equation (11)?\n\n- In Line 235, the paper mentions that “The style guidance regularizer encourages style separation by reducing the mutual information between the content encoder and $X$.” However, it is unclear why score prediction using only style information (i.e., $Vs_G(g(x),t)-s^*(x,t)$) necessarily reduces mutual information between the content encoder and $X$. Could the authors clarify if it is possible for the content encoder to still capture excessive information about the style?\n\n- Could the authors elaborate more on the role of balancing loss in Equation (11)? Although the full derivation and proof are presented in Appendix, it was personally hard to follow the full derivation. Could the authors provide intuition on how this term avoids poor local minima? \n\n- In the experimental section, it is mentioned that Equation (13) is used for training on MNIST and CIFAR-10 as well. However, it is not clear how Equation (13) is derived from Equation (11). Firstly, it lacks balancing loss. Does it imply that balancing loss is a redundant term from the formulation? Additionally, since Equation (11) relies on the assumption of the independent subspace model (ISM), could the authors discuss the validity of applying Equation (13) to datasets like MNIST and CIFAR-10, where this assumption may not hold?\n\n- Although multi-view disentanglement is highlighted as a core contribution, relevant experimental validation is missing. Could the authors provide empirical evidence demonstrating multi-view disentanglement?\n\n- In image disentanglement experiment, could the authors provide comparison to related work (e.g., [1,2,3])? It would improve overall understanding if the authors provide comparison to previous work and provide analysis. \n\n- A wrong image is inserted in Figure 6(d) in the Appendix. \n\n[1] Chen et al., Weakly supervised disentanglement with guarantees, in AAAI 2020. \n\n[2] Shu et al., Weakly supervised disentanglement with guarantees, in ICLR 2020. \n\n[3] Locatello et al., Weakly supervised disentanglement without compromises, in ICML 2020.", "limitations": "yes, (although the authors do not allocate a section for indicating the limitation, it was written in the discussions throughout the paper)", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\nThe paper is well-written and easy to follow. Formulating a definition of formal disentanglement for stochastic and non-invertible mappings of latent variables is a valuable contribution to the field. This paper introduces a clearly defined formulation of approximate disentanglement through the concepts of $(\\epsilon,\\nu)$-disentanglement and $\\epsilon$-editability, addressing an important theoretical gap.\n\n\n**Weaknesses**\n\nA major concern is in the empirical justification of the proposed framework. Although the proposed theoretical foundation presented is concrete, the connection to empirical evidence is somewhat weak. For example, how does Equation (13) derived from Equation (11), why does it lack balancing loss, how can Equation (13) apply to datasets where it does not follows independent subspace model (ISM), lack of experiments on multi-view disentanglement, etc. Please refer to the Question Section below. Weak connection to the empirical study makes it challenging to assess the empirical validity and overall contribution of the proposed theoretical framework.", "was_revised": true, "final_justification": "While this work requires strong assumptions such as the ISM structure or style information for content-style disentanglement, which may limit its applicability to more general scenarios, the theoretical framework built on these assumptions is novel and insightful. Therefore, I am leaning towards weak accept.", "rebuttal": "Dear reviewer qfQD,\n\nWe sincerely thank you for careful reading and for highlighting both the clarity of our theoretical contributions and the originality of our disentanglement framework. Below, we respond to the key concerns raised regarding the empirical connection, equation derivation, and experimental scope. \n\n**1. Empirical validation of theory and derivation of Eq. (13) from Eq. (11)**\n\nWe appreciate the reviewer’s careful inspection. Eq. (13) is a practical implementation of Eq. (11) tailored to image domains (MNIST, CIFAR-10) where the true ISM structure is unknown or approximate. We drop the balancing loss in Eq. (13) for simplicity since empirically the model does not rely on the loss to escape bad local minima.\n\nEq. (13) is equivalent to Eq. (11) without the balancing loss and when the input data is an ISM. The form in Eq. (13) looks different from Eq. (11) primarily because the (13) is using conditional score matching while (11) is using unconditional score matching. The two are equivalent as discussed in Section 2, and our theory can be easily adapted to conditional score matching as well, as discussed in line 250-251. \n\nFurther, the style-guided regularization losses in (13) (denoted as $L_r’$) and (11) (denoted as $L_r$) are related. Suppose we assume ISM and restrict the function class to be a variant of the dual encoder score network in Fig. 2, i.e. $$s_{\\theta}(X_t,\\hat{Z},G,t)=Us_Z(X_t,\\hat{Z},t)\\mathbf{1}[\\hat{Z}=\\mathbf{0}_{d_Z}]+Vs_G(X_t,G,t),$$\nand suppose the all-zero vector is outside the support of the content estimator $\\hat{Z}$, then we have \n$$L_r’=\\mathbb{E}||Vs_G(X_t,G,t)+N_t/\\sigma(t)||^2.$$\nThis takes the same form as $L_r$ in (11), except in two aspects: i) this is conditional score matching; ii) $s_G$ is a score function conditioned on $G$ instead of an unconditional one in (11). We have already discussed i), and ii) poses no issue either since the score function conditioned on $G$ still satisfies the ISM assumption and can still be handled by our theory.\n\n**2. Role of balancing loss and intuition**\n\nIntuitively, the balancing loss tries to balance between the subspace matrices $(U,V)$ and the subspace score networks $(s_Z^{\\theta_Z},s_G^{\\theta_G})$ to have similar operator norms (by viewing the subspace score networks as operators). Otherwise, suppose $(s_Z^{\\theta_Z},s_G^{\\theta_G})$ have operator norms much larger than $(U,V)$, then $U \\approx 0_{d_X\\times d_Z}, V\\approx 0_{d_X\\times d_G},$ since their combined operator, the full score network $s_{\\theta}$, has finite norm. Further, since the gradients $\\nabla_{\\theta_Z} L^{\\lambda_r}$ and $\\nabla_{\\theta_G} L^{\\lambda_r}$ scale with $U,V$, this implies $\\nabla_{\\theta_Z} L^{\\lambda_r}\\approx 0$ and $\\nabla_{\\theta_G} L^{\\lambda_r}\\approx 0,$ even though the learned content and style spaces $R(U)$ and $R(V)$ are nowhere near the true spaces $R(A_Z)$ and $R(A_G)$. This results in a bad local minimum.     \n\nThe balancing loss also does not affect the output of the optimal score network, since for any “imbalanced” optimal parameters $(U,V,\\theta_Z,\\theta_G)$, there always exist another $(U,V,\\theta_Z’,\\theta_G’)$ that achieves zero balancing loss. The argument based on singular value decomposition is provided in 1306-1309 of Appendix D.3. \n\nTherefore, the balancing loss helps the model to avoid bad local minima but does not change the location of at least one global minima for the regularized score matching loss. On the other hand, we empirically observe the model to converge near a global optima without the balancing loss, so we discard it for our experiments. The balancing loss is thus largely a proof artifact, and we leave analysis without such a loss for future works.\n\n**3. Multi-view disentanglement experiments**\n\nWe appreciate the reviewer highlighting this. Our speech experiments (Sec. 5, Fig. 5) do instantiate multi-view disentanglement: different speakers provide style views, and the shared emotion across recordings is the content. This matches the Definition 3.4 setup. We acknowledge that this connection could be more explicitly stated and will revise the text to clarify this mapping between theory and practice. We are also actively developing additional vision-domain multi-view experiments.\n\n**4. Role of $U$ and $V$ in Eq. (10), and learnability**\n\nThank you for pointing this out. U and V are learnable linear projections that align the outputs of the content and style encoders with the overall score function, motivated by Lemma 4.4. We adopt this architecture to reflect the ISM-based decomposability of scores into subspace components. Their gradients are computed and optimized during training (see gradient flow Eq. 12). We will update the paper to explicitly define U and V as learnable parameters.\n\n**5. Meaning of $n$ in Eq. (11)**\n\nSure. $n$ is the number of samples in the training set. It is mentioned in the “Intuition” section in Line 243-244.\n\n**6. Intuition behind style guidance regularizer and mutual information**\n\nWe agree this point deserves further clarification. The key idea is that **when the score network is forced to explain part of the data using only style**, the model is implicitly **discouraged from encoding style information redundantly in the content branch**. For general latent variable models, this is a heuristic regularization rather than a formal MI minimization, and we will add a clearer explanation and acknowledge.\n\n**For ISM, however, this regularization does lead to MI minimization.** To see this, consider the extreme case where no style guidance regularizer is used, then the content encoder can simply copy from the input X and maximize the score matching loss $L_0$ by letting $U$ covering the entire data space, and in this case the mutual information between the content encoder and X, $I(\\hat{Z};X)=I(X_{t_0};X)$ is maximized; As style guidance weights $\\lambda_r$ increase, the style encoder starts to play an increasingly important role in score prediction until it can predict the style component of the score perfectly. In turn, the content encoder becomes less and less predictive of the style component. Further, by Theorem 4.5, the subspaces $R(U)$ and $R(V)$ recover the content and style subspaces up to vanishingly small error for sufficiently large $\\lambda_r$, and thus $I(\\hat{Z};X)\\approx I(Z_{t_0};X)<I(X_{t_0};X)$ by data processing inequality and that the true content encoder is injective. For a more precise relation between the style regularizer weight and $I(\\hat{Z};X)$, one can upper bound $I(\\hat{Z};X)$ in terms of subspace recovery errors using an argument similar to Appendix D.1.   \n\n**7. Comparison to prior work in image experiments**\n\nWe have made a comparison with related works in two additional disentanglement experiments on the CelebA image dataset. First, we perform **denoising** with the same setting as CIFAR 10 and compared our approach with two standard baselines, conditional GAN [1] and beta-VAE [2] adapted to the content disentanglement setting by conditioning on the observed style variable, which are representative of the approach taken in the papers suggested by the reviewer. As shown in the Table below, our method is superior to the baselines and follow the same trend with the style regularization weight as in our experiments on MNIST and CIFAR10. \n\n| **Method**                         | **Style Guidance Weight** | **LPIPS (↓)** | **SSIM (↑)** |\n|-----------------------------------|----------------------------|--------------------|---------------|\n| conditional GAN\t\t| -\t\t| 0.83 \t| 0.16 \t|\n| conditional betaVAE\t| -\t\t| 0.57\t| 0.17\t|\n| DDPM \t\t\t| 0\t\t| 0.53\t| 0.18\t|\n| DDPM (ours)\t\t\t| 0.3\t\t| 0.47\t| 0.19\t|\n| DDPM (ours)\t\t\t| 0.03\t\t| 0.35\t| 0.23\t|\n| DDPM (ours)\t\t\t| 3\t\t| **0.33**\t| **0.24**\t|\n \nFurther, we perform **face editing** with gender as the style variable. As shown in the table below, our method is superior to the conditional InfoGAN [1] and betaVAE [2] baseline in terms of style transfer (CLIP score) and content preservation (LPIPS score).\n| **Method** | **Style Guidance Weight** | **CLIP Score (↑)** | **LPIPS (↓)** |\n|----------------|----------------------------|--------------------|---------------|\n| conditional InfoGAN | -\t\t| 0.241\t  | 0.39\t\t|\n| conditional betaVAE | -  | 0.254 |\t0.37 |\n| DDPM | 0\t | 0.250 \t\t| 0.35          |\n| DDPM (ours) | 3 \t| 0.259              | **0.30**          |\n| DDPM (ours) | 6 \t| 0.259              | 0.32          |\n| DDPM (ours) | 9\t| **0.261**              | 0.38          |\n\n[1] Kim, Hyunjik, and Andriy Mnih. \"Disentangling by factorising.\" International conference on machine learning. PMLR, 2018.\n\n[2] Chen, Xi, et al. \"Infogan: Interpretable representation learning by information maximizing generative adversarial nets.\" Advances in neural information processing systems 29 (2016)\nMoreover, we plan to perform larger-scale experiments on more diverse datasets in the future.\n\n**8. Miscellaneous: wrong figure and limitations**\n\nThank you for pointing it out. We acknowledge the formatting error in Fig. 6(d) and will fix it in the camera-ready version. We will also combine our scattered discussions of limitations into a dedicated section in the final version for clarity.\n\n--------------------------------------------------------------\n\nWe thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions."}, {"review_id": "0AI764IW4k", "reviewer": "Reviewer_cYrQ", "summary": "This paper investigates the disentanglement ability of diffusion models. The authors provide key theoretical insights, showing that under certain conditions with partial supervision or multi-view inputs, diffusion models can disentangle content and style latent variables. Additionally, in the special case of independent subspaces, they introduce a novel score loss and prove global convergence results with finite sample . These theoretical contributions are supported by numerical experiments, validating the convergence rate under the independent subspace assumption. The authors further validate the general disentanglement theorems on two real-world image datasets and in a voice conversion (VC) adaptation task.", "questions": "1. Does Definition 4.3 imply that the observation X is a linear combination of latent variables Z and G? If so, wouldn’t that be a rather strong assumption for real-world datasets?\n\n2. In the multi-view disentanglement setting, is it implicitly assumed that X and the concatenated latent variables (Z,G) have the same dimensionality? If so, could this constrain the applicability of the model?\n\n3. I’m having trouble understanding how the example: \"a speaker embedding extracted from a pre-trained speaker recognition model\", which is used to illustrate a case where the style value is known. Could you explain more about how this is connected to \"knowing the style function/value\"? Is the speaker embedding considered the exact value of the style variable G? Or does using a pre-trained model imply that we fully know the style function, or are we just estimating it?", "limitations": "Although the paper is theoretically solid and presents three key theorems, I find it challenging to build a clear logical connection or progression among them, particularly between the final theorem and the first two. It would be helpful if the authors could provide more explanation on how these results build upon one another or fit into a cohesive framework. Additionally, some of the assumptions made in the theoretical analysis are quite strong. I list them all in both the weakness and question parts and would appreciate further discussion from the authors on how these assumptions might be relaxed or addressed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1.\tTheoretical contributions are solid and well-formulated.\n\n2.\tThe paper investigates an interesting problem, offering a novel perspective on disentanglement with diffusion models under certain supervision and assumptions.\n\n3.\tEven though the whole paper is notation-heavy, the writing style is accessible and visually engaging. The use of colored highlights and intuitive text boxes improves readability, and the examples in the main paper help understand key definitions.\n\n\n### Weaknesses\n\n1.\tThe connection between the multi-view setup and the independent subspace assumption is not clearly established for me. In my understanding, these are somewhat orthogonal concepts: independent subspaces do not necessarily imply a separation into content and style, and vice versa.\n\n2.\tSome of the assumptions made in this paper are quite strong and may not hold in real-world scenarios, potentially limiting the practical applicability of the theoretical findings. For instance, in the context of content disentanglement, the assumption that the style function is known is hard to justify for two main reasons:\n\n(1) Precisely identifying which part of the data corresponds to the style or changing factors is overly idealized and rarely feasible in practice.\n\n(2) Knowing the exact style value for all training samples is extremely challenging and unrealistic in most settings.\n\nAdditional assumptions that I find restrictive are listed in the question section.", "was_revised": true, "final_justification": "I am happy about understanding the cohesion among all the different settings in this paper.", "rebuttal": "Dear reviewer cYrQ,\n\nThank you for the thoughtful and constructive feedback. We address the concerns below: \n\n**1. Clarifying the connection between multi-view setup and ISM assumption**\n\nThank you for this important observation. We agree that multi-view disentanglement (Theorem 4.2) and the independent subspace model (ISM) (Theorem 4.5) address distinct settings. The former analyzes general nonlinear, stochastic mappings under weak supervision, while the latter provides stronger guarantees in a special case with linear structure.\n\nOur intent is not to claim that ISM is a general case of multi-view disentanglement. Instead, we show that when the ISM structure holds, we can derive finite-sample convergence guarantees using gradient dynamics (Thm 4.5). The general framework (Defs. 3.1-3.4) encompasses both settings. We will make this relationship clearer in the revision by explicitly distinguishing the scope and assumptions of each theorem.\n\n**2. Assumption of known style function**\n\nThis is a valid concern. The assumption of a known style function appears only in the “content disentanglement” setting (Def. 3.3, Thm. 4.1), where the goal is to **disentangle content given a fixed style** – a setting aligned with applications like **voice conversion** and **image editing**.\n\nIn practice, we **approximate the style function using a pre-trained model** (e.g. speaker embeddings from ECAPA-TDNN, which achieves almost perfect speaker verification error rate [1]). While this is not a ground-truth oracle, it provides a **fixed and consistent representation of style** across training and inference, which makes the theoretical setup meaningful. This assumption mirrors common practice in the voice conversion literature and is not unique to our framework.\n\nWe agree it is an idealization, but as a **weaker alternative to requiring full latent access**, it enables principled analysis under partial supervision. In real-world scenarios where even pre-trained models are unavailable, we propose the **multi-view formulation (Def. 3.4, Thm. 4.2)**,  which removes the need for known style and instead leverages paired observations. We will highlight the distinction more clearly. \n\n[1] Desplanques, B., Thienpondt, J., & Demuynck, K. “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification”. Interspeech 2020. \n\n**3. Clarifying Definition 4.3 (independent subspace model)**\n\nYes, Def. 4.3 assumes a linear mixing model to latent factors, which is a simplified case used to derive **finite-sample convergence guarantees**. As mentioned in the paper (Sec. 4.3), the ISM assumption is not required in general – our earlier results (Theorems 4.1 and 4.2) apply to **nonlinear, stochastic mappings** under mild Lipschitz assumptions. The ISM-based result (Thm 4.5) serves to complement these general results with **stronger guarantees** and clearer intuition about training dynamics under structure.\n\nEven though the ISM assumption does not always hold for raw data, such structures often emerge in the latent space of self-supervised representations. For instance, previous study [2] has found that self-supervised representation models have learned orthogonal subspaces for content (phonetic) and style (speaker) information, as mentioned in Section 4.3 of our paper. Further, practical algorithms making the ISM assumption have recently achieved competitive results in disentanglement tasks such as voice conversion [3] and image editing [4], demonstrating the applicability of the ISM assumption to complex data.\n\n[2] O. D. Liu, H. Tang, and S. Goldwater, “Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces,” in Interspeech, 2023.\n\n[3] Kamper, Herman, Benjamin van Niekerk, Julian Zaïdi, and Marc-André Carbonneau. “LinearVC: Linear Transformations of Self-Supervised Features through the Lens of Voice Conversion,” 2025. \n \n[4] L. Rout, N. Raoof, G. Daras, C. Caramanis, A. Dimakis, and S. Shakkottai, “Solving linear inverse problems provably via posterior sampling with latent diffusion models,” in NeurIPS, 2023.\n\n**4. Dimensionality assumptions in multi-view setting**\n\nWe appreciate the opportunity for clarifying. No, we do **not require** that the observation $X\\in \\mathbb{R}^{d_X}$ has the same dimensionality as $(Z,G).$ Our theory allows for arbitrary encoder functions mapping inputs to latent spaces of lower dimension. For instance, the encoder can output $\\hat{Z}\\in \\mathbb{R}^{d_Z},$ even if $d_Z<d_X.$ The score function operates in observation space, but the latent representations can reside in lower-dimensional subspaces. We will clarify this in the definitions and theorem statements.\n\n**5. Speaker embedding as known style function**\n\nWe appreciate the opportunity to clarify. When we refer to using speaker embeddings (e.g., d-vectors), we treat the **pretrained embedding extractor** $g(\\cdot)$ as a **proxy for the true style function**. This is standard in voice conversion pipelines and aligns with how weak supervision is commonly defined in practice. We never claim that this represents the true style variable $G$, but rather a fixed, known surrogate used during both training and inference, consistent with Def. 3.3. We will revise the text to make this distinction clearer and emphasize that our theory supports both fixed-style and fully-learned (multi-view) scenarios.\n\n**6. Logical progression across theorems**\n\nThank you for pointing this out. The three theorems build progressively:\nTheorem 4.1 (Content disentanglement): proves approximate disentanglement under known style supervision.\nTheorem 4.2 (Multi-view disentanglement): proves approximate disentanglement under multi-view supervision with learned content and style encoders.\nTheorem 4.5 (ISM content disentanglement): focuses on linear latent structure and shows **finite-sample global convergence** with gradient descent.\nThe first two theorems highlight **approximate disentanglement under weak supervision**. Theorem 4.5 is a special case of Theorem 4.1 when the data distribution has ISM structure, which allows us to study the training dynamic of disentanglement and prove stronger disentanglement guarantees. The special case of Theorem 4.2 with ISM is omitted since its proof is analogous to Theorem 4.5, though we provide a sketch of the proof strategy in Line 245-248. \n\n**7. Assumption discussion and relaxation**\n\nWe agree and thank the reviewer for the suggestion. We already mention in Appendix B.7 and C.4 that some assumptions (e.g., exact independence between $Z$ and $G$, or i.i.d styles) can be **relaxed** to approximate independence settings. \n\nFurther, under the content disentanglement setting, the assumption of access to the true style variable can be relaxed by assuming instead access to a **sufficiently good style estimator** $G’$, such as a reliable speaker embedding model in voice conversion. Such an estimator is essentially a perturbed version of the true style variable, and since the score function is assumed to be Lipschitz in our theory, it is robust against such perturbations and all theorems proved in our paper still hold, albeit with additional error terms due to the style estimation error.\n\nWe will consolidate these discussions into the main text and include a new limitations and future directions paragraph addressing how assumptions might be relaxed in practice.\n\n------------------------------\nWe thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions."}, {"review_id": "kOls704kvJ", "reviewer": "Reviewer_34h9", "summary": "The paper presents a theoretical framework for disentangling content and style in diffusion models. The authors first introduce two information-theoretic concepts called ($\\epsilon, \\nu$)-disentanglement and $\\epsilon$-editability. The former states that the restored content and style are approximately statistically independent, while the letter is satisfied if a change of style leads to a data distribution with unchanged content. Subsequently, two settings (access to the ground truth style and multiple views with shared content) and technical conditions are presented under which disentangling and editability can theoretically be achieved.", "questions": "* In Equation 7, what forces $I(\\hat{Z},\\hat{G};X)=I(Z,G;X)$ as $\\delta \\to 0$? Why can $\\hat{Z}$ not just be constant? \n* Why is the ground-truth decoder/generator $f$ in the score function in line 1122 and line 1204? Where is this equation for the score function coming from?\n* In line 1077 ,where is $C_2$ used?\n* How is the estimated content, $\\hat{Z}$, used in equation 13 for the experiments?\n* Why is the background in the CIFAR10 colorization task in the input always black? Shouldn't this also work with other background colors in the input?\n* How do ($\\epsilon, \\nu$)-disentanglement and $\\epsilon$-editability relate to deterministic formalizations of disentanglement? That is, there exist functions $h_1$, $h_2$ such that $\\hat{Z}=h_1(Z)$ and $\\hat{G}=h_2(G)$.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n* The paper adresses the problem of disentanglement in diffusion models, which is an important topic in generative modelling.\n* The paper provides a novel view on disentanglement from the lense of information theory\n\n**Weaknesses:**\nAt first glance, the theoretical results seem plausible, so I tried to follow the theoretical derivations in the appendix, but due to time constraints I could only look at a small part, as the derivations are quite extensive. Some equations seem to come out of nowhere (see questions below). So I have some concerns about the validity, but it is possible I didn't understand some parts.\n* The experimental part is difficult to follow. It is not clear to me from the short description how the datasets look like and how the results were obtained. The paper would benefit greatly from additional clarifications of the experimental setups.\n* There seems to be a gap between the theoretical contribution of the paper and its empirical validation. The presented theorems are based on the losses in Equation 7, 8, and 11, but the experiments are based on the loss in equation 13 without sufficient explanation how it relates.\n* It would improve readability of the appendix if theorems and lemmas were restated before the proofs. Otherwise one needs to switch back and forth quite a lot.", "was_revised": true, "final_justification": "I now have a clearer understanding of the connection between theory and experiments, and my main concern regarding clarity has been largely addressed by the proposed revisions. While the practical applicability remains limited, I believe the theoretical contribution provides a novel and valuable perspective on disentanglement. I am therefore also willing to raise my score to 4.", "rebuttal": "Dear reviewer 34h9,\n\nWe thank you for the thoughtful comments and for recognizing the originality and importance of our work. We appreciate your engagement with both the theoretical and empirical parts of the paper, and we address your concerns below. \n\n**1. On experimental details in Section 5**\n\nWe defer the experimental details to the Appendix E to focus on the exposition of our theoretical results. But we will consider adding the dataset description back to the main text in the extra space of the camera-ready version of the paper.\n \n**2. On theoretical-empirical gaps and the role of Eq. (13)**\n\nWe appreciate this concern and agree it deserves clarification. Eq. (13) is a **practical implementation** of Eq. (11), tailored for high-dimensional image/speech data. Specifically:\n\n* We omit the balancing term for empirical simplicity;\n\n* Style guidance is applied by zeroing out content, equivalent to the style regularizer in Eq. (11) when the input data has ISM structure;\n\n* For MNIST colorization/CIFAR-10 denoising/voice conversion, the ISM assumption is not explicitly enforced, but the structure is approximately satisfied in these tasks either directly or after the input is transformed into self-supervised representation [1,2]. \n\nThus, Eq. (13) is not theoretically disjoint from Eq. (11)—rather, it's an **approximated empirical proxy** aligned with our theoretical principles. We will make this relationship clearer in Sec. 5 and Appendix E.\n\n[1] O. D. Liu, H. Tang, and S. Goldwater, “Self-supervised predictive coding models encode speaker and phonetic information in orthogonal subspaces,” in Interspeech, 2023.\n\n[2] Kamper, Herman, Benjamin van Niekerk, Julian Zaïdi, and Marc-André Carbonneau. “LinearVC: Linear Transformations of Self-Supervised Features through the Lens of Voice Conversion,” 2025. \n\n**3. On Eq. (7): Why not make $z_{\\phi}(X)$ constant**\n\nWe appreciate the opportunity for clarifying. While the regularizer in Eq. (7) enforces an upper bound on mutual information between content and input, the **score matching loss term** still requires $z_{\\phi}(X)$ to be predictive of the data’s conditional score. If $z_{\\phi}(X)$ were constant, the model could not satisfy the conditional score matching objective. Thus, **the interplay between the loss and regularizer** ensures that $z_{\\phi}(X)$ retains just enough content information for prediction without overfitting. This tension promotes disentanglement. \n \n**4. On score function using ground-truth generator (lines 1122 and 1204)**\n\nIn Appendix A and B, we use the true generative function $f(Z,G)$ only for **theoretical analysis**, such as constructing counterexamples (e.g., for demonstrating non-editability in Example 1) or establishing upper bounds of the optimal conditional score matching loss in the proof of Theorem 4.1 and Theorem 4.2. In practice, the model learns a score estimator $s_{\\theta},$ and the generator is not known. We will clarify in the text that these instances refer to **analytical derivations** only, not model implementation.  \n\n**5. On use of $C_2$ in line 1077**\n\nIt is used in line 1086, but we agree it appears too early, and will consider moving it to the later part of the proof.\n\n**6. On use of $z_{\\phi}(X)$ in Eq. (13)**\n\nIn Eq. (13), $z_{\\phi}(X)$ is passed to the score estimator $s_{\\theta}(X_t, z_{\\phi}(X), G, t).$ The **second term** in the loss zeroes out $z_{\\phi}(X)$ to encourage style-specific signal separation (style guidance). The loss structure is consistent with Eq. (11) but omits the balancing term since our model is able to work without it empirically.  \n\n**7. On CIFAR10 background being black**\n\nIn our CIFAR-10 experiments (Sec. 5, Fig. 3), we perform **denoising**, not colorization and the input is not always black. The colorization task is conducted on MNIST, which is gray-scale. In principle, our method works for any background color.\n\n**8. On relation to deterministic disentanglement formalizations**\n\nThank you for asking this important question. Classical deterministic disentanglement assumes that **invertible functions** exist to recover latent variables (e.g., $Z=z(X),\\,G=g(X)$). However, diffusion models are stochastic and non-invertible, making such assumptions inapplicable.  Our proposed $(\\epsilon,\\nu)$-disentanglement and $\\epsilon$-editability generalize these notions to stochastic settings by relying on **mutual information** and **distributional distances** rather than pointwise mappings. \n\nThese criteria reduce to classical notions in the **low-noise limit**, where DMs become nearly deterministic and when the content and style variables are discrete. We will add a paragraph in Section 3 connecting our framework to deterministic disentanglement theories.\n\n**9. On theoretical derivation in the appendix**\n\nThank you for this candid feedback. We understand that Appendix B–D contains dense derivations. We will improve the clarity by:\n\n* Restating the theorems and definitions before proofs.\n\n* Adding brief and intuitive summaries before technical steps\n\n* Clarifying notation and referencing which result each equation builds on\n \n------------------\nWe thank you again for the insightful comments and hope our response helps to address your concern and makes our paper more suited for the NeurIPS conference. Happy to discuss further and consider further revisions."}]

Author rebuttal:

Meta-review: This paper presents a novel theoretical framework for understanding disentanglement in diffusion models under weak supervision. The authors introduce two key criteria: (1) approximate information-theoretic disentanglement and (2) editability. They demonstrate identifiability results in three scenarios: known-style supervision, multi-view data, and independent subspace models. In these cases, they establish finite-sample global convergence guarantees. Beyond the theoretical contributions, the authors validate the framework through controlled experiments using synthetic, image, and speech data. The results show that training strategies inspired by their analysis can enhance disentanglement performance in practice.

Most reviewers had a positive view of the paper, and the discussion period was lively. The authors adequately addressed most of the concerns raised by the reviewers, including additional comparisons with Celeb A and a clarification regarding the writing relationship to prior work (DisDiff). The paper's focus is on the theoretical framework, and while its performance on Celeb A may not fully match the leading methods for this task, I still consider this work valuable. Overall, the paper presents interesting new theories and empirical validation, and I recommend its acceptance. The comments provided in the rebuttal should be integrated to enhance the manuscript's readability.
Decision: Accept (poster)