DiffSDA: Unsupervised Diffusion Sequential Disentanglement Across Modalities
Authors: Hedi Zisling
Ilan Naiman
Nimrod Berman
Supasorn Suwajanakorn
Omri Azencot
Venue: ICLR 2026 Poster (2026)
Keywords: Sequential Disentanglement
Diffusion Models
Unsupervised Learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: DiffSDA is a new diffusion-based, modality-agnostic framework that outperforms prior methods in unsupervised sequential disentanglement across real-world data like time series, video, and audio.

Abstract: Unsupervised representation learning, particularly sequential disentanglement, aims to separate static and dynamic factors of variation in data without relying on labels. This remains a challenging problem, as existing approaches based on variational autoencoders and generative adversarial networks often rely on multiple loss terms, complicating the optimization process. Furthermore, sequential disentanglement methods face challenges when applied to real-world data, and there is currently no established evaluation protocol for assessing their performance in such settings. Recently, diffusion models have emerged as state-of-the-art generative models, but no theoretical formalization exists for their application to sequential disentanglement. In this work, we introduce the Diffusion Sequential Disentanglement Autoencoder (DiffSDA), a novel, modal-agnostic framework effective across diverse real-world data modalities, including time series, video, and audio. DiffSDA leverages a new probabilistic modeling, latent diffusion, and efficient samplers, while incorporating a challenging evaluation protocol for rigorous testing. Our experiments on diverse real-world benchmarks demonstrate that DiffSDA outperforms recent state-of-the-art methods in sequential disentanglement.

DIFFSDA: UNSUPERVISED DIFFUSION SEQUENTIAL DISENTANGLEMENT ACROSS MODALITIES
Hedi Zisling'
Ilan Naiman"
Nimrod Berman Ben-Gurion University
Ben-Gurion University
Ben-Gurion University
(hedi z, n imani, bermann @post bgu ac.
Supasorn Suwajanakorn
Omri Azencot
VISTEC
Ben-Gurion University
aopasorn.s@visteee..
azencor hawccco@bbguuc. aciil
ABSTRACT
Unsupervised representation learning, particularly sequential disentanglement, aims to separate static and dynamic factors of variation in data without relying on labels. This remains challenging problem, as existing approaches based on variational autoencoders and generative adversarial networks often rely multiple loss terms, complicating the optimization process. Furthermore, sequential disen- tanglement methods face challenges when applied to real-world data, and there is currently no established evaluation protocol for assessing their performance in such settings. Recently, diffusion models have emerged state-of- oatt-of-the-rrr generative models, but no theoretical formalization exists for their application to sequential disentanglement. In this work, introduce the Diffusion Sequential Disentan- glement Autoencoder (DiffSDA), a novel, moodll-agnsttic framework effective across diverse real-world data modalities, including time series, video, and audio. DifISDA leverages a new probabilistic modeling, latent diffusion, and efficient samplers, while incorporating a challenging evaluation protocol for rigorous test- ing. Our experiments diverse real-world benchmarks demonstrate that DifISDA outperforms recent state-of-the-art methods in sequential disentanglement. Code is available at https: /ttgt/gttth///t/// con/ het///zencot..o group/DiEfSDA
INTRODUCTION
Unconditional generation (Ho et al., 2020; Dhariwal & Nichol, 2021; Rombach al., 2022), and more broadly, unsupervised learning (Bengio et al., 2012), play central role in todays machine learning research, as enables leveraging large-scale data without requiring expensive annotations. Within unsupervised learning, disentangled representation learning has become particularly significant (Ben- gio al., 2013). This approach seeks to decompose latent representations into distinct factors, where each factor captures a specific variation in the data. Such representations improve interpretability (Liu et al., 2020), mitigate biases (Creager et al., 2019), and improve generalization (Zhang et al., 2022). A prominent challenge is develop modal-agnostic approach for sequential data such video, audio, and time series. In particular, the goal is to decompose the sequential signal into separate static and dynamic latent components in unsupervised manner. For example, in video of a person speaking. the static factors could represent the person's 's facial appearance, while the dynamic factors encode facial movements. audio recordings, static factors may correspond the speaker's identity, while dynamic factors capture content of the speech.
Despite recent advancements, most sequential disentanglement methods (Tulyakov et al., 2018; Yingzhen & Mandt, 2018; Bai et al., 2021; Han et al., 2021; Naiman et al., 2023; Berman et al., 2024) rely on VAEs and GANs, which often require complex optimization with extensive hyperparameter tuning. For instance, C-DSVAE (Bai et al., 2021) requires five hyperparameters solely to balance its various loss terms. Moreover, these models are often evaluated on toy datasets and struggle to produce high-quality samples in real-world scenarios. The reliance on VAEs and GANs is directly related to
"Equal contribution

---

the absence of modeling framework for sequential disentanglement within diffusion- -based modeling. Further, existing diffusion architectures do not produce disentangled representations (Preechakul et al., 2022; Wang et al., 2023). We hypothesize that diffusion- based framework can reduce hyperparameter tuning and improve sample quality, paving the way for more robust and scalable approaches to unsupervised sequential disentanglement.
In this work, we introduce Diffusion Sequential Disentanglement Autoencoder (DiffSDA), a novel probabilistic framework for sequential disentanglement. Unlike prior tools (Bai et al., 2021; Naiman et al., 2023), our method models static and dynamic factors as interdependent, enhancing the expressivity of their marginal distributions. Notably, our approach is based on single standard diffusion loss term, while producing high-quality results. Furthermore, DiffSDA is modal-agnostic, allowing to disentangle data across diverse modalities, such as video, audio, and time series, with only minor adjustments to the network. This stands in contrast to modal medal-dependent methods, such as animation- based approaches for video, which rely on temporal and spatial consistency properties inherent to visual data (Siarohin et al., 2019), or methods designed specifically for audio that depend on spectral or temporal cues (Xu et al., 2024a).
Practically, we implement sequential semantic encoder and adopt the efficient sampling framework EDM (Karras et al., 2022). Moreover, we incorporate a latent diffusion module (LDM) (Rombach al., 2022) into our architecture, which enables robust handling of high- dimensionnl weal-world data, outperforming prior sequential disentanglement methods. Finally, using our method, we demonstrate that applying principal component analysis (PCA) to the latent static and dynamic representations reveals a further disentanglement into multiple interpretable factors, showcasing the richness of the learned representations.
We perform a comprehensive evaluation of our model on standard benchmarks for sequential disen- tanglement (Naiman et al., 2023) across three diverse data domains: audio. time series, and video. To further advance the field, we introduce a novel evaluation protocol for high-quality visual sequential disentanglement, incorporating three high-resolution video datasets and multiple quantitative metrics. A Additionally, we propose new post-training approach for disentangling representations into multiple factors. For the first time, our work presents a zero-shot task to demonstrate the generalizability of the factorization framework. Through these extensive evaluations, we show that DifTSDA not only effectively disentangles real-world data but also outperforms recent state-of-the-ae methods. Our key contributions are summarized as follows:
1. We propose novel modal-agnostic probabilistic framework for sequential disentanglement grounded in diffusion processes. Unlike most existing approaches, our formulation accom- modates dependent static and dynamic factors of variation. The model is optimized using single, unified score estimation loss.
2. Our design enables the effective disentanglement of high-dimensional, real-world data and supports zero-shot disentanglement tasks. Moreover, demonstrate DifiSDA's capability to disentangle static and dynamic information into multiple interpretable factors.
3. We provide a comprehensive evaluation demonstrating our model's superiority in both qualitative and quantitative tasks, outperforming state-of-the-art methods. Additionally, we introduce a novel evaluation protocol specifically designed for iideo-based disentanglement.
2 RELATED WORK
Generative modeling is a fundamental methodology for effectively sampling from numerical approximations of data distributions. Prominent approaches include variational autoencoders (VAEs) and generative adversarial networks (GANs) (Kingma, 2013; Goodfellow et al., 2014). More recently, diffusion models (Sohl-Dickstein et al., 2015) and score matching (Hyvärinen & Dayan, 2005; Vincent, 2011) have emerged as powerful alternatives, outperforming VAËS and GANs in generating high-quality samples through iterative denoising of latent variables (Ho al., 2020; Dhariwal & Nichol, 2021). These methods are unified under a score-based modeling framework (Song et al., 2021). A critical challenge in generative modeling lies in representation learning, where semantic encodings of inputs are derived in an unsupervised manner. A related topic, center to this work, the study of modal-agnostic disentangled representations, aiming to decompose data of various modalities into distinct factors of variation (Bengio et al., 2013).
2

---

Disentangled Representation Learning. Most existing works disentangled learning leverage VAEs and GANs to decompose non non-sequential (Higgins et al., 2017; Chen et al., 2018; Kim & Mnih, 2018; Tran et all, 2017; Karras et al., 2020; Ren et al., 2021) and sequential (Hsu et al., 2017; Yingzhen & Mandt, 2018; Zhu et al., 2020; Bai et al., 2021; Han et al., 2021; Naiman et al., 2023; Berman al., 2024; Simon al., 2025; Villegas al., 2017; Tulyakov et al., 2018) data. A key limitation of these approaches lies in their reliance on complex loss formulations, which typically involve multiple regularizers alongside the standard VAEs and GANs losses. While significant progress has been made in enhancing the generative capabilities of VAEs and GANs (Vahdat & Kautz, 2020; Karras et al., 2020), state-of-the-e methods for sequential disentanglement largely focus on simple datasets, far from real-world scenarios, with few exceptions like SPYL's preliminary results (Naiman et al., 2023). In contrast, works in animation (Siarohin al., 2019; Hu, 2024; Xu al., 2024b) have shown strong results on real-world data by leveraging video priors for disentangling objects and motion. However, these modal-dependent approaches can exploit relaxed assumptions and specialized tools, whereas our modal-agnostic method can adapt to diverse modalities, including video, audio, and time series.
Table I: A comparison between animation, diffusion, and sequential disentanglement methods.
Method
Modal Agnostic
Efficient
Real-World
Latent Factorization
Latents Prior
Loss Terms
ank millon
ROM Siarohin et al. (2019)
X
X
NIA
2
(2024) MA Xu et al. (2024b)
X
x
/
/
X
NIA
2
non
DifIAE Proechakul et (2022)
x
x
/
X
NIA NIA
InfoDiff Wang et (2021)
x
x
/
X
2
soquene tall
SPYL Naiman et (2023)
X
independent
DBSE Berman al. (2024)
/
X
independent
2
Ours
dependent
Diffusion-Basse Disentanglement. The emergence of diffussin models has recently enabled novel approaches for non-sequential disentanglement (Kwon et al., 2022; Yang et al., 2023; Wang et al., 2023; Yang al., 2024; Zhu et al., 2024; Baumann et al., 2024), achieving high-resolution image generation with disentangled factors. Moreover, other efforts have concentrated on structuring their latent representations. For instance, DiffAE (Preechakul et al., 2022) introduces an autoencoder to facilitate the manipulation of visual features, while InfoDiffusion (Wang et al., 2023) adds a loss regularizer enhance disentanglement. Despite these advances, to the best of our knowledge, no theoretical formalization, and specifically, probabilistic modeling, has yet been proposed for diffusion-basee disentanglement in sequential settings. Furthermore, practical approaches for this domain remain unexplored.
To contextualize our work within the landscape of existing tools, we present a comparative summary in Tab. 1, highlighting how our approach either advances or maintains all key aspects of representation learning. Specifically, while animation methods (FOM, AA, MA) and non-sequential diffusion tools (DiffAE, InfoDiff) handle real-world data, they are modal-dependent and do not provide latent factorization. Within sequential disentanglement approaches (SPYL, DBSE), only our work supports real-world data via single loss optimization.
METHOD
In this section, we introduce a novel probabilistic framework for unsupervised sequential disentangle- ment based on diffusion models. Currently, none of the existing approaches leverage diffusion models for unsupervised sequential disentanglemennt leaving a significant gap in the field. Our framework addresses this gap by establishing probabilistic modeling formalization and providing an efficient implementation for disentangling static and dynamic factors in sequential data. Background on diffusion models, diffusion autoencoders, and additional details about the method can be found in App. A and App. B. Throughout this section, and the subsequent ones, the subscripts represent time in the diffusion process, and superscripts indicate time in the sequence, e.g., a sequence state of the diffusion process is denoted by xf, E [0, T] and E {1...., V). T and V represent the maximum diffusion and sequence times, respectively. We consider discrete time sequences of continuous time diffusion processes; however, our modeling can be extended to additional settings.
3

---

80
h
X
UIET
Linear
LSTM
1-V
LSTM
d
UNET
x
Gaussian Sampler
Figure I: DiffSDA processes sequences XbV via semantic and stochastic encoders (top and bottom). Their outputs (8o, d.v. xt v are fed to a stochastic decoder yielding a denoised x (right).
3.1 PROBABILISTIC MODELING
Existing frameworks for sequential disentanglement lack a probabilistic modeling foundation for diffusion-basel modeling. To address this gap. propose novel probabilistic approach based on two diffusion models. The first model details the latent-independent distribution density of the static (time-invariant) and dynamic (time-variant) factors, &o and d.... respectively. The second model specifies the observed distribution and its dependence on the disentangled factors. Formally, the joint distribution is given by
I.V I.V p(x XT 80, 8T, d. .d. pro(so, db. 8T, v II PTO X 80, di)
(1)
where (sp db. 8T, d.v is a standard diffusion process with pro(.) being the transition distribution from time T to time 0. The state distribution of (xo x7 So, dī) is conditioned on the latent x and the factors so and dj.
Importantly, our probabilistic approach differs from existing work (Bai et al., 2021; Naiman et al., 2023) in that static and dynamic factors are interdependent. We motivate model by three main reasons: i) expressiveness the overall dependence facilitates learning of different state tra- jectories, leading to higher expressivity in the marginals Pso(-); and ii) efficiency- our sampler is not autoregressive, allowing for fast and parallelized sampling; and iii) causality-our model has the ability learn intricate relationships between the static and dynamic factors, if needed. We evaluate both the dependent and independent approaches on our model to highlight the effectiveness of our approach. summary, adopting dependent modeling improves generation quality by 13%. Further details can be found in App. G.1.
Given sequence x) po(x):V) the posterior distribution of the latent variables x!V and latent factors and dl-V is composed of three independent distributions. Further, unlike the non-autoregressive prior Eq. 1, here, we explicitly assume temporal dependence. The posterior distribution reads
p(x!
IVV So, d.
(::) por (x
I:V
1:V
v:V) x
r(h r=1
dh
ST) Xx
where x and 8o conditioned the entire input x and the dynamic factors only depend on previous dynamic factors and current and previous data elements. We employ score matching (Hyväri- nen & Dayan, 2005; Song et al., 2021), to optimize for the denoising parametric map Dg. The map Dp takes the noisy latent x" time t, and disentangled factors zíj := (su, d5), and it returns an estimate of the score function V log Poe Poc xx). Overall, the optimization objective reads
0* arg min, E, AE Da
Vx log Poe
where d, E R a positive weight, 4 T is uniformly sampled over [0, T). the variables xT, X sampled from their respective distributions, por(-), (.)), and zi via the densities in Eq. 2.
4

---

The inner expectation is taken over x z and xj. Importantly, Pro of su. di is not used in Eq. 3, and thus its optimization can be separated.
Notably, make assumptions about the given data x! ensuring that framework remains modal-frree and independent of specific properties of video, audio, or time series data. This theoretical compatibility with any type of sequence makes it highly adaptable to diverse applications.
3.2 DIFFUSION SEQUENTIAL DISENTANGLEMENT AUTOENCODER
Our architecture, shown in Fig. 1, comprises three main components: (1) sequential semantic encoder, (2) stochastic encoder, and (3) stochastic decoder. At high level, the sequential semantic encoder factorizes data into separate static and dynamic components, while the stochastic decoder denoises the noisy latent representation produced by the stochastic encoder, conditioned on disentangled factors. Notably, unlike prior works, our implementation achieves disentanglement with single, simple loss term.
Encoders. Inspired by prior work in sequential disentanglement (Yingzhen & Mandt, 2018), we design a novel sequential semantic encoder to extract 80 and db.V. Particularly, it consists of U-Net (Ronneberger et al., 2015) for video data and an MLP for other modalities, coupled with linear layers that independently process each sequence element. Then, an LSTM module summarizes the sequence into a latent representation h The last hidden, h is passed to linear layer to produce Su, whereas I.V are processed with another LSTM and a linear layer to produce d) d!:V Our stochastic encoder follows the EDM framework (Karras al., 2022), adding noise N/0,, of to each element Xy. yielding x = X5 + c. These encoders realize practice the posterior in Eq. 2.
real video .-> Made
real vadeos
real videos
Female +..
zero-shox
Figure 2: We present swap (left), zero-shot (middle), and multifactor disentanglement (right) results on multiple real-world and high-resolution visual datasets. See Sec. 4 for further details.
Decoder. efficiently handle real-world sequential information, we follow the decoding in EDM (Karras et al., 2022), featuring only 63 neural function evaluations (NFEs) during inference. Our decoder Dg takes as inputs the noisy input x and disentangled factors 2 = (Sus df), and it returns denoised version of x,, denoted by X Given any [0, T] and 7 E {1..... VJ. the decoder parameterized independently from other times t', 7' as follows
x = D2(x] 2 = Ce
skip x + out Fe (cxx 2J, poise
where ot modulates the skip connection, in put scale the input/output magnitudes, and c moise maps noise at time f into conditioning input for the neural network Fg. conditioned on z through AdaGN.
While prior sequential disentanglement works depend on intricate prior modeling, regular- ization terms, and mutual information losses, leading to many hyper-parameter and challenging training, we opt for simpler objective containing single loss term that is based on Eq. 3,
Loss.
E (axT.mT
1 cqui (xg
skip ct
x 11
(5)
5

---

real videos
Ours
SPYL
Figure 3: We present dynamic swap results of our approach (third row) and SPYL (fourth row) on CelebV-HQ (left), VoxCeleb (middle), and TaiChi-HD (right).
where Fg takes inputs x 26, and eooie, While our loss in Eq. does not include auxiliary terms, it promotes disentanglement due to two main reasons: i) the static factor so is shared across and thus will not hold dynamic information, and ii) the dynamic factors dz IR are low-dimensional (i.e., k is small), making it difficult for dī to store static features. We empirically validate these assumptions through experiments presented in App. G.2. Finally, we briefly mention that support high-resolution sequences, we incorporate latent diffusion models (LDM) (Rombach et al., 2022), using a pre-trained VQ-VAE autoencoder to reduce the high-dimensionnalty of input frames. Instead of factorizing all the equations above with new symbols for the features VQ-VAE produces, we denote by x..VV the input sequence, and we abuse the notation x) to denote the latent features, ie.. xDiV [xx]iV) and xhi D(x)-V), where & and D are the VQ-VAE encoder and decoder, respectively.
4 RESULTS
Below, we empirically evaluate the modeling capabilities of DifISDA in comparison to recent modal- agnostic state-of-the-art methods (see Tab. 1), SPYL (Naiman et al., 2023) and DBSE (Berman et al., 2024). In general, we consider quantitative and qualitative experiments. For video, we include three high-resolution, real-world visual datasets that have not been previously used for sequential disentanglement: VoxCeleb (Nagrani et al., 2017), CelebV-HQ (Zhu et al., 2022), and Tai (Siarohin et al., 2019), along with the popular MUG dataset (Aifanti et al., 2010). For audio, we consider TIMIT (Garofolo, 1993) and a new dataset, Libri Speech (Panayotov et al., 2015). The time series datasets are PhysioNet, ETThI, and Air Quality (Tonekaboni et al., 2022). Detailed descriptions of the datasets and their pre-processing can be found in App. D, while extended baseline comparisons are provided in App. H.1. For brevity, we omit below the subscript indicating the diffusion step for clean samples (corresponding time step 0).
4.1 CONDITIONAL SWAP IN VIDEOS
We begin our tests with the conditional swap task (Yingzhen Mandt, 2018). Given two sample videos x, & Po. the goal in this experiment is to create new sample x, conditioned on the static factor of x and dynamic features of x. This is done by extracting the latent factors = (s, d!:V) and = (8, .l:V) for x and &, respectively. The new sample x is defined to be the reconstruction of z (s, d du:V through sampling, see Alg. 1. In an ideal swap, X preserves the static characteristics of x while presenting the dynamics of x, thus demonstrating strong disentanglement capabilities of the swapping method. We show in Fig. 2 (left) swap example of DifISDA, where the top two rows are real videos, and the third row shows the new sample obtained by preserving the static features of the first row and using the dynamics of the second row. Remarkably, while the people in these sequences are very different, many fine details are transferred, including head angle and orientation, well mouth and eyes orientation and openness. In Fig. 3. present additional swap results CelebV-HQ (left), VoxCeleb (middle), and TaiChi-HD (right), comparing DifISDA (third row) to SPYL. (fourth row). Our approach produces high-quality samples, while swapping the dynamics of

---

real videos
zero- zero-shol shot
Figure 4: Zero-shot swap results, training on VoxCeleb and tested on CelebV-HQ or MUG.
the second row into the first row, whereas SPYL struggles both with the reconstruction and swap. Additional conditional and unconditional swap results appear in App. H.3 and App. H.4, respectively.
In addition to the above qualitative evaluation, we also want to quantitatively assess DiffSDA's effectiveness. We report App. F results from the traditional quantitative benchmark. where pre-trained judge (classifier) is used to determine if swapped content is correct (Bai et al., 2021). However, there are two main issues with the benchmark: i) depends on labeled data, making it relevant to only small number of datasets; and ii) results are sensitive to the expressivity and generalizability of the judge. For instance, swapping a smiling expression from person to person B, may result in person B having smile, different from the one in the data. In these cases, the judge may wrongly classify a different expression to the smiling person B, see App. F for further discussion.
Towards addressing these issues, propose new unsupervised swapping metrics to quantitatively measure the model's 's disentanglement abilities. We adopt estimators commonly used in animation for assessing if objects and motions are preserved (Siarohin al., 2019). Specifically, we utilize the average Euclidean distance (AHD) that is based on the distances between the latent representations of images. Further, we also employ the average keypoint distance (AKD) which computes the distances between selected keypoints in images. Intuitively, AED and AKD have been designed to identify the preservation of objects and motions in images, respectively. See App. E for definitions.
Equipped with these new metrics, we perform conditional swapping over pre-defined random list of sample pairs, x, &. Particularly, we reconstruct new samples of the form z := (s, dd:V) and zd : (s, d encoding dynamic and static swaps, respectively. We compute the AED of with respect to (arising from x), expecting their static features to be similar. Following the same logic, we compute the AKD of x (reconstructed from zd) and x, as they share the dynamic factors. Our findings are presented in Tab. 2, where DiffSDA outperforms SOTA previous (SPYL. DBSE) approaches across all datasets, except for AED on TaiChi- -DD, where we attain the second best error. Notably, our AKD errors are significantly lower than SPYL and DBSE. Further, apply these metrics to assess reconstruction performance, as well as the mean squared error (MSE), with the results shown in Tab. 3. Again, DiffSDA is superior current SOTA methods. Additionally, include generative evaluation in App. G.4, comparing our approach to previous methods.
Table Preservation of objects (AED) and motions (AKD) is estimated across several datasets and methods. The labels 'static frozen' and 'dynamics frozen' correspond to samples z and z
AEDI (static frozen)
AKD| SPYL
(dynamics frozen)
SPYL
DBSE
Ours
DBSE
Ours
MUG (64 x 64)
0.766
0.773
0.751
1.132
1.118
0.802
VoxCeleb (256 x 256)
1.058
1.026
0.846
4.705
10.96
2.793
CelehV-HQ (256 x 256)
0.631
0.751
0.540
39.16
28.69
6.932
TaiChi-HD (64 x 64)
0.443
0.325
0.326
7.681
6.312
2.143
7

---

sample large batch of static vectors & E R², with the static latent size, j = 1,...,b b = 215 Then, we compute PCA the matrix formed by arranging {&,] in its columns, yielding the principal components (v.) given that b > h. We utilize the latter pool of static variability by exploring the latent 4.2 ZERO- SHOT VIDEO DISENTANGLEMENT
In the previous sub-section, the conditional swap was performed on the held-out test set of each dataset on which we trained on. In contrast to previous work, for the first time, we perform the same task on dataset unseen during training. We show example in Fig. (middle) of zero-shot swap, where our model was trained on the VoxCeleb dataset (Ist row) and the inferred sequence was taken from MUG (2nd row). Particularly, we froze the static features of the MUG sample and swapped the dynamic factors with those of VoxCeleb (3rd row). Remarkably, in addition to changing the facial expression of the person, DifISDA also adds the necessary details to mimic the body pose. We emphasize that the MUG dataset does not include sequences similar to the third row in Fig. 2, but rather zoomed-in facial videos as shown the second row, thus, our zero-shot results present a significant adaptation to the new data. Additionally, we include in Fig. 4 zero-shot examples where DiffSDA trained on VoxCeleb and evaluated on CelebV-HQ or MUG. These results further highlight the effectivity of our approach in transferring dynamic features across different datasets. Finally, we provide more zero-shot examples in App. H.5.
4.3 TOWARD MULTIFACTOR VIDEO DISENTANGLEMENT
Multifactor sequential disentanglement is challenging problem, where the objective is to produce several static factors and several dynamic factors per frame (Berman et al., 2023; Barami al., 2025). Here, we show that our model has the potential to further disentangle the static and dynamic features into additional factors of variation. Inspired by DiffAE (Preechakul et al., 2022), we explore the learned latent space in an unsupervised linear fashion, particularly, using principal component analysis (PCA). Namely, to obtain fine-grate thaiind semantic static factors of variation,
space from a static code 8 of a real example x in the test set, i.e.,
j4a + an VT
8
os +14+
(6)
0
where μı and o are the mean and variance of the sampled static features, (- and a E [-K, k). notice that 0 recovers the original sequence. The new sample is obtained by reconstructing the new static features ø with the original dynamic factors d' dl:V of x.
We demonstrate static PCA exploration in Fig. 2 (right) VoxCeleb. The middle row is the real video, whereas the top and bottom rows use positive and negative a values, respectively. Our results show that traversing in the positive direction yields more maseuline appearances, and in contrast, going in the negative direction produces more feminine characters. Importantly, we highlight that other static features and the dynamics are fully preserved across the sequence. In App. H.6, present further results on full sequences using multiple values to demonstrate the gradual transition in the latent space. Notably, we find in our exploration principal components that control other features such as skin tone, image blurriness, and more.
Table 3: Reconstruction errors are measured in terms of AED, AKD, and MSE across several datasets and models. We find DifISDA to be ordens-of-magnitude better than other methods.
AED DBSE
AKD| DBSE
MSE DBSE
SPYL
Ours
SPYL
Ours
SPYL
Ours
MUG
0.49
0.49
0.11
0.47
0.48
0.06
0.001
0.001
3e-7
VoxCeleb
0.99
1.03
0.37
2.27
2.43
1.09
0.005
0.003
5e 4
CelebV-HQ
0.70
0.78
0.29
15.0
13.8
1.26
0.012
0.006
6e 4
TaiChi-HD
0.32
0.29
0.001
4.31
3.83
0.10
0.018
0.007
2e
8

---

4.4 SPEAKER IDENTIFICATION AUDIO
Our approach is inherently modal-agnostic and extends beyond the video domain. Unlike methods tailored specifically for video or audio, which often require extensive modifications when applied to new modalities, our method is versatile and can adapt to different modalities with minimal adjustments to the backbone architecture. For example, to process audio data, we simply replace the U-Net with an MLP. In Tab. we demonstrate the adaptability of our model by successfully disentangling audio data from the TIMIT dataset and Libri Speech, where TIMIT is a widely used benchmark for speech-related tasks and Libri Speech is an additional dataset add for this benchmark. Following the speaker identification benchmarks (Yingzhen & Mandt, 2018), we evaluate disentanglement quality using the Equal Error Rate (EER), standard metric in speech tasks. Specifically, the Static EER measures how effectively the static latent representations capture speaker identity, and similarly, the Dynamic EER assesses the dynamic latent representations. Notably, deeetdiiet model should yield a low Static EER (capturing speaker identity in static representations) and high Dynamic EER (capturing content-related dynamics without speaker identity). The overall goal is to maximize the gap between these two metries (Dis. Gap). Our model, achieves in TIMIT disentanglement gap improvement of over 11% with a 42.29% compared to 31.1 11% achieved by DBSE, thereby surpassing current ofte-offtheeart methods. Similar strong performance is achieved on Libri Speech as well. These results highlight the efficacy of our approach in the audio domain. Additional details regarding the dataset, evaluation metrics, and implementation are provided in the appendix. Furthermore, we report speech quality and reconstruction results in App. G.3, further validating our model's effectiveness in the audio domain.
Table 4: Disentanglement metrics on TIMIT and LibriSpeech
Method
TIMIT
LibriSpeech Dynamic EER
Static FER
Dynamic EER
Dis. Gapt
Static EER
Dis. Gapt
DSVAE
5.64%
19.20%
13.56%
15.06%
28, 28.94%
13.87%
SPYL
3.41%
33.22%
29.81%
24.87%
49.76%
24.89%
DBSE
3.50%
34.62%
31.11%
16.75%
22.61%
5.58%
Ours
4.43%
46.72%
42.29%
11.02%
45.94%
34.93%
4.5 DOWNSTREAM PREDICTION AND CLASSIFICATION TASKS ON TIME SERIES INFORMATION
Generative modeling of time series data has advanced rapidly in recent years (Naiman et al., 2024; Fadlon et al., 2025; Gonen et al., 2025). In the context of disentanglement, we evaluate our approach on time series data, following the evaluation protocol in (Berman et al., 2024). The evaluation is car- ried out in two main independent setups: 1) We assess the quality of the learned latent representations using predictive task. The model is trained dataset, and at test time, the static and dynamic factors are extracted and used as input features for predictive model. Two tasks are considered: (i) predicting mortality risk with the PhysioNet dataset (Goldberger et al., 2000), and (ii) predicting oil temperature using the ETThl dataset (Zhang et al., 2017). Performance is evaluated using AUPRC and AUROC for PhysioNet, and Mean Absolute Error (MAE) for ETTh1. 2) We investigate the model's ability to capture global patterns within its disentangled static latent representations, which have been shown to enhance performance (Trivedi et al., 2015). Following similar procedure, the model is trained, and now only the static representations are extracted. These representations are then used as input features for a classifier. For the PhysioNet dataset, Intensive Care Unit (ICU) unit types are used as global labels, while for the Air Quality dataset, the month of the year serves as the target variable. Further details regarding datasets, metrics, and implementation can be found in App. D and App. E. We compare our method vs. state-of-the -hr baselines, including DBSE, SPYL, and GLR (Tonekaboni et al., 2022). Results for predictive and classification tasks are given in Tab. 5. Notably, our model outperforms across all tasks.

---

Table Time series prediction and classification benchmarks.
Task
GLR
SPYL
DBSE
Supervised
Ours
AUPRCT
0.37 + 0.09
0.37
0.02
0.47 + 0.02
0.44 + 0.02
0.50 + 0.006
pred.
AUROCT
0.75
0.76
0.86
0.80 + 0.04
MAEJ (ETThI)
+
12.2 0.00
11.2+
10.19 0.20
9.89 + 0.280
PhysioNert
2.49
47.0
5639
2..00
64.6 + 0.35
Air Qualityt
3
$7.9
65.9
62.43
69.2 + 1.50
CONCLUSIONS
The analysis and results of this study underscore the potential of the proposed DiffSDA model to address key limitations in sequential disentanglement, specifically the context of complex real- world visual data, speech audio, and time series. By leveraging novel probabilistic framework, diffusion autoencoders, efficient samplers, and latent diffusion models, DifISDA provides a robust solution for disentangling both static and dynamic factors in sequences, outperforming existing state-of-the -ort-----tt-tttt methods. Moreover, the introduction of a new real-world visual evaluation protocol marks a significant step towards standardizing the assessment of sequential disentanglement models. Nevertheless, while DiftSDA shows promise in handling high-resolution videos and varied datasets, future research should focus on optimizing computational efficiency and extending applicability to more diverse sequence modalities, such as sensor data. Such modalities present unique challenges, as varying temporal characteristics and distinct data patterns, which may require adapting the model architecture and training strategies. In addition, given that our current video generation process potentially limiting spatio-temporal coherence, an interesting direction
operates frame-by-framee for future work is integrate DifISDA with latent video diffusion models, e.g., LVDM (He et al., 2022), or related architectures to further strengthen its generative fidelity. Finally, a key challenge ahead lies in fully extending our multifactor exploration procedure to effectively disentangle and represent multiple interacting factors (Berman et al., 2023). We leave these considerations and further explorations for future work.
ACKNOWLEDGEMENTS
This research was partially supported by the Lynn and William Frankel Center of The Stein Faculty of Computer and Information Science, Ben-Gure Ben-Gurion University of the Negev, ISF grants 668/21 1 and 1299/25, an ISF equipment grant, and by the Israeli Council for Higher Education (CHE) via the Data Science Research Center, Ben-Gurion University of the Negev, Israel.
10

---

REFERENCES
Niki Aifanti, Christos Papachristou, and Anastasios Delopoulos. The MUG facial expression database. In I 11th International Workshop on Image Analysis for Multimedia Interactive Services WIAMIS 10, pp. 1-4, 2010. 6, 18, 22
Samuel Albanie, Arsha Nagrani, Andrea Vedaldi, and Andrew Zisserman. Emotion recognition in speech using cross-modal transfer in the wild. In Proceedings of the 26th ACM international conference on Multimedia, pp. 292-301, 2018. 20
Brian DO Anderson. Reverse-time diffusion equation models. Stochustic Processes and their Applications, 12(3):31-326, 1982. 16
Junwen Bai, Weiran Wang, and Carla Gomes. Contrastively disentangled sequential variational autoencoder. Advances in Neural Information Processing Systems, 34:10105-10115 10118, 2021. 1, 2,2, 3, 4, 7, 18
Tal Barami, Nimrod Berman, Ian Naiman, Amos Haviv Hason, Rotem Ezra, and Omri Azencot. Disentanglement beyond static vs. dynamic: A benchmark and evaluation framework for multi- factor sequential representations. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, Datasets and Benchmarks Track, 2025. 8
Stefan Andreas Baumann, Felix Krause, Michael Neumayr, Nick Stracke, Vincent Tao Hu, and Björ Ommer. Continuous, subject-specifie attribute control in T2I models by identifying semantic directions. arXiv preprint arXiv:2403. 17064, 2024. 3
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Unsupervised feature learning and deep learning: A review and new perspectives. CoRR, abs/1206.5538, 1(2665):2012, 2012.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learningg A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8):1798-182) 2013. 1,2
Nimrod Berman, Ilan Naiman, and Omri Azencot. Multifactor sequential disentanglement via structured Koopman autoencoders. In The Eleventh International Conference on Learning Repre- sentations, 2023. 8, 10
Nimrod Berman, Ilan Naiman, Idan Arbiv, Gal Fadlon, and Omri Azencot. Sequential disentanglement by extracting static information from single sequence element. In Forty-first International Conference on Machine Learning. 2024. 1, 3, 6,
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
22563 22665,22575,222 2023. 24
Adrian Bulat and Georgios Tzimiropoulos. How far we from solving the 2d & 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017. 21
Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2d pose estimation using part affinity fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017. 21
Ricky TQ Chen, Xuechen Li, Roger Grosse, and David Duvenaad. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
Elliot Creager, David Madras, Jöm--Hnrik Jacobsen, Marissa Weis, Kevin Swersky, Toniann Pitassi, and Richard Zemel. Flexibly fair representation learning by disentanglement. In International conference on machine learning, pp. 1436-1445. PMLR, 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat GANs on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021. 1, 2, 16

---

Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similarity metrics based on deep networks. In Lee, M. Sugiyama, U. Luxburg. Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 29, 2016. 17
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high- high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pultern recognition, pp. 12873-12883, 2021. 17
Gal Fadlon, Idan Arbiv, Nimrod Berman, and Omri Azencot. A diffusion model for regular time series generation from irregular data with completion and masking. In Advances in Neural Information Processing Systems (NeurIPS) 39, 2025.
John S Garofolo. TIMIT acoustic phonetic continuous speech corpus. Linguistic Data Consortium, 1993, 1993. 6, 20
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph Mietus, George Moody, Chung Chung-Kang Peng, and Eugene Stanley. PhysioBank, PhysioToolkii, and PhysioNet: components of new research resource for complex physiologic signals. circulation, 101(23):e11-=222) 2000. 20
Tal Gonen, Itai Pemper, Ilan Naiman, Nimrod Berman, and Omri Azencot. Time series generation under data scarcity: A unified generative modeling approach. In Advances in Neural Information Processing Systems (NeurIPS) 39, 2025.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
Jun Han, Martin Renqiang Min, Ligong Han, Li Erran Li, and Xuan Zhang. Disentangled recurrent wasserstein autoencoder. In 9th International Conference on Learning Representations, ICLR, 2021. 1,3
Yingqing He, Tianyu Yang. Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.1322. 2022. 10
Alexander Hermans, Lucas Beyer, and Bastian Leibe. defense of the triplet loss for person re-identification. arXiv preprint arXiv:1703.07733 2017. 21
Trina Higgins, Loic Matthey, Arka Pal, Christopher P Burgess, Xavier Glorot, Matthw M Botvinick,
Shakir Mohamed, and Alexander Lerchner. beta-VAE: Learning basic visual concepts with constrained variational framework. ICLR (Poster), 2017. 3
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020. 1,2
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. Advances in neural information processing systems, 30, 2017.
Li Hu. Animate anyone: Consistent and controllable image-to-video synthesis for character animation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp.
1153-8163, 2024. 3
Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching.
Journal of Machine Learning Research, 6(4), 2005. 2, 4
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1 1125-1133 2017. 17
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of styleGAN. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 81 10-8119, 2020.
12

---

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances in neural information processing systems, 35:26565-26577, 2022. 5, 17
Tero Karras, Miika Aittala, Jaakko Lehtinen, Janne Hellsten, Timo Aila, and Samuli Laine. Analyzing
and improving the training dynamics of diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24174-24184, 2024. 17
Hyunjik Kim and Andriy Mnih. Disentangling by factorising. In International conferrecee on machine learning, pp. 2649-2658. PMLR, 2018.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint
2013 2, 17
Mingi Kwon, Jaeseok Jeong, and Youngjung Uh. Diffusion models already have a semantic latent space. arXiv preprint arXiv:2210. 10960, 2022.
Wenqian Liu, Runze Li, Meng Zheng, Srikrishna Karanam, Ziyan Wu, Bir Bhanu, Richard Radke, and Octavia I Camps. Towards visually explaining variational autoencoders. in 2020 iece. In CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 13-19, 2020. 1
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. VoxCeleb: A large-scale speaker identifica- tion dataset. In 18th Annual Conference of the International Speech Communication Association, Interspeech, pp. 2616-2620. ISCA, 2017. 6, 20, 22
Ilan Naiman, Nimrod Berman, and Omri Azencot. Sample and predict your latent: Modality-free sequential disentanglement via contrastive estimation. In International Conference on Machine Learning, pp. 25694-25717. PMLR, 2023. 1, 2, 4, 21
Ilan Naiman, Nimrod Berman, Itai Pemper, Idan Arbiv, Gal Fadlon, and Omri Azencot. Utilizing image transforms and diffusion models for generative modeling of short and long time series. Advances in Neural Information Processing Systems, 37:121699-111330 2024.
Vassil Panayotov, Guogue Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an ast corpus based on public domain audio books. 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 5200-5210. IEEE, 2015. 6, 20
Konpat Preechakul, Nattanat Chatthee, Suttisak Wizadwongsa, and Supasorn Suwajanakorn. Dif- fusion autoencoders: Toward a meaningful and decodable representation. In Proceedings of the IEFE/CVE conference on computer vision and pattern recognition, pp. 10619-10629, 2022, 2, 8. 16, 17
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with VQ-VAE-2. Advances neural information processing systems, 32, 2019. 17
Chandan KA Reddy, Vishak Gopal, and Ross Cutler. Dnsmos: A non-intrusive perceptual objective
speech quality metrie. In ICASSP. 2021. 22, 25
Xuanchi Ren, Tao Yang, Yuwang Wang, and Wenjun Zeng. Learning disentangled representa- tion by exploiting pretrained generative models: A contrastive learning view. arXiv preprint arXiv:2 10543, 2021. 3
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine leurning, pp. 1278-1286. PMLR, 2014. 17
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022. 1. 6, 17, 18
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-asiisted Intervention-MCCC 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pp. 241. Springer, 2015. 5
13

---

Sefik Ilkin Serengil and Alper Ozpinar. Lightface: A hybrid deep face recognition framework. In 2020 lovations in Intelligent Systems and Applications Conference (ASYU), pp. 23-27. IEEE, 2020. 21, 24
Aliaksandr Siarohin, Stéphane Lathuilière, Sergey Tulyakov, Elisa Ricci, and Nicu Sebe. First order motion model for image animation. In Conference Neural Information Processing Systems (NeurIPS), December 2019. 2, 6, 19, 21,22
Mathieu Cyrille Simon, Pascal Frossard, and Christophe De Vleeschouwer. Sequential representation learning via static-dynamic conditional disentanglement. In European Conference on Computer Vision, pp. 110-126. Springer, 2025.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference machine learning, pp. 2256-2265. PMLR, 2015. 2, 16
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020. 17
Yang Song, Jascha Sohl-Dickstedd Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. 16
Sana Tonekaboni, Chun-Liang Li, Sercan O Arik, Anna Goldenberg, and Tomas Pfister. Decoupling local and global representations of time series. International Conference on Artificial Intelligence and Statistics, pp. 8700-871 14. PMLR, 2022. 6, 9, 20, 21
Luan Tran, Xi Yin, and Xiaoming Liu. Disentangled representation learning GAN for pose-invariant face recognition. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 1415-1424, 2017.
Shubhendu Trivedi, Zachary A Pardos, and Neil T Heffernan. The utility of clustering in prediction tasks. arXiv preprint arXiv: 10.../009.066. 2015.
Sergey Tulyakov, Ming- Yu Liu, Xiaodong Yang, and Jan Kautz. MoCoGAN: Decomposing motion and content for video generation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 1526-1535. 2018. 1,3
Arash Vahdat and Jan Kautz. NVAE: A deep hierarchical variational autoencoder. Advances in neural information processing systems, 33: 1967-99977,,, 2020. 3
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in
neural information processing systems, 30, 2017. 17
Ruben Villegas, Jimei Yang, Seunghoon Hong, Xunyu Lin, and Honglak Lee. Decomposing motion and content for natural video sequence prediction. arXiv preprint 17Xiv31706.08033, 2017. 3
Pascal Vincent. A connection between score matching and denoising autoencoders. Neural computa- tion, 23(7):1661-1674 2.11. 2
Jun Wang, Yinglu Liu, Yibo Hu, Hailin Shi, and Tao Mei. Facex-zoo: A pytorch toolbox for face recognition. In Proceedings of the 29th ACM international conference Multimedia, pp.
3779-3782, 2021. 3, 20
Yingheng Wang, Yair Schiff, Aaron Gokaslan, Weishen Pan, Fei Wang. Christopher De Sa, and Volodymyr Kuleshov. InfoDiffusion: Representation learning using information maximizing diffusion models. In International Conference Machine Learning, pp. 36336433335. PMLR, 2023. 2, 3, 16
Sicheng Xu, Guojun Chen, Yu-Xiao Guo, Jiaolong Yang. Chong Li. Zhenyu Zang, Yizhong Zhang, Xin Tong, and Baining Guo. Vasa-1: 1: Lifelike audio-driven talking faces generated real time. arXiv preprint 24 2024a.
14

---

Zhongcong Xu, Jianfeng Zhang, Jun Hao Liew, Hanshu Yan, Wei Liu, Chenxu Zhang, Jiashi Feng, and Mike Zheng Shou. Magicanimate: Temporally consistent human image animation using diffusion model. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1481-1490, 2024b.
Tao Yang, Yuwang Wang, Yan Lu, and Nanning Zheng. DisDiff:: Unsupervised disentanglement of diffusion probabilistic models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. 3
Tao Yang, Cuiling Lan, Yan Lu, et al. Diffusion model with cross attention as inductive bias for disentanglement, arXiv preprint arXiv:2402.09711 097122 2024. 3
Li Yingzhen and Stephan Mandt. Disentangled sequential autoencoder. In Jeifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5670-5679. PMLR, 10-15 Jul 2018. 1, 3, 6, 9, 17
Hanlin Zhang, Yi-Fan Zhang, Weiyang Liu, Adrian Weller, Bernhard Schölkopf, and Eric P Xing. Towards principled disentanglement for domain generalization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8024-8034, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unrcasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 586-595, 2018. 17
Shuyi Zhang, Bin Guo, Anlan Dong, Jing He, Ziping Xu, and Song Xi Chen. Cautionary tales on air-quality improvement in Beijing. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, 473(2205):2017043)
2017. 9, 20
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang, Li Zhang, Ziwei Liu, and Chen Change
Loy. CelebV-HQ: A large-scale vidco facial attributes dataset. In ECCV, 2022. 6, 20, 22
Ye Zhu, Yu Wu, Zhiwei Deng, Olga Russakovsky, and Yan Yan. Boundary guided learning- free semantic control with diffusion models. Advances in Neural Information Processing Systems, 36, 2024. 3
Yizhe Zhu, Martin Renqiang Min, Asim Kadav, and Hans Peter Graf. S3VAE: sule-supe sequential VAE for representation disentanglement and data generation. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 6538-6547, 2020.
15

---

A BACKGROUND
A.1 DIFFUSION MODELS
Diffusion models (Sohl-Dickstein et al., 2015) are family of SOTA generative models, that were recently described using stochastic differential equations (SDEs), diffusion processes, and score-based modeling (Song et al., 2021). We will use diffusion models and score-based models interchangeably. These models include two processes: the forward process and the reverse process. The forward process (often not learnable) is an iterative procedure that corrupts the data by progressively adding noise to it. Specifically, the change to the state x, can be formally described by
dx. = f(x,, t)dl + g(()dw:
where w is the standard Wiener process, f(-,() L) is vector-valued function called the drift coefficient, and y(-) a scalar function known as the diffusion coefficient. From a probabilistic viewpoint, Eq. 7 is associated with modeling the transition from the given data distribution, Xo Po- to Pt. the probability density of x,, E [0, 77. Typically, the prior distribution pr is a simple Gaussian distribution with fixed mean and variance that contains no information of po- The reverse process, which is learnable, de-noises the data iteratively. The reverse of a diffusion process is also a diffusion process, depending on the score function Vx log pr (x) and operating reverse time (Anderson, 1982). In our approach, we utilize the conditioned reverse process
dx; = [f(x,, t) g(t) V po ujjdř+ (()dw,
(8)
where w is standard Wiener process as time progresses backward from T to 0, dt is an negative timestep, and u is condition variable. Diffusion models are generative by sampling from pr and use V log Pi (x; u) to iteratively solve Eq. 8 until samples from Po are recovered.
A.2 DIFFUSION AUTOENCODERS
Although diffusion models are powerful generative tools, they are not inherently designed to learn meaningful representations of the data. To address this limitation, several works (Preechakul et al., 2022; Wang et al., 2023) have adapted diffusion models into autoencoders, resulting in diffusion autoencoders (DiffAEs). These models have demonstrated the ability to learn semantic representations of the data, allowing certain modifications of the resulting samples by altering their latent vectors. To this end, DiffAEs introduce semantic encoder, taking a data sample Jip and returning its semantic latent encoding žsem- Then, the latter vector conditions the reverse process, enhancing the model's ability to reconstruct and manipulate data samples. In practice, the denoiser also conditioned on feature map h and the time 6, combined using an adaptive group normalization (AdaGN) layer (Dhariwal & Nichol, 2021). The AdaGN block is defined as
AdaGN(h, t, 2sem) 2g (tg GroupNorm(h)) + to)
where 2g is the output of linear layer applied to 2sem t, and t, are the outputs of a multi-layer perceptron (MLP) applied to the time I, and multiplications are done element-wiss
B DIFFSDA MODELING
B. UNSUPERVISED SEQUENTIAL DISENTANGLEMENT
Unsupervised sequential disentanglement is challenging problem in representation learning, aiming to decompose given dataset to its static dime-independent and dynamic túiee-dependent factors of variation. Let {x) : N J==1 be a dataset with N sequences x == [x)..... x }, where x7 Rd. We omit the subscript j for brevity, unless noted otherwise. The goal of sequential disentanglement to extract an alternative representation of x via a single static factor $ and multiple dynamic
factors d V.V Note that s shared across the sequence.
We can formalize the sequential disentanglement problem as generative task, where every sequence xI:V from the data space X' is conditioned on some z!:V from latent space 2. We aim to maximize the probability of each sequence under the entire generative process
V(x = / p(x
z I.V VV
dz L.V
(10)
16

---

where z I.V := (s, d ). One of the main challenges with directly maximizing Eq. (10) is that the latent space is too large to practically integrate over. Instead, a separate distribution, denoted here as q( x x!:V),, ) used to narrow search to be only over z !:V associated with sequences from the dataset D. Importantly, the distributions p(xl:V x and q(zl.V [xl:V take the form of a decoder and encoder in practice, suggesting the development of autoencoder sequential disentanglement models (Yingzhen & Mandt, 2018). The above p(xi-V
|z!i) and q(z):V 1:V xl:V) are denoted by
x) respectively, in Eq. and Eq. 2. (ro X7, Su, d5) and p(x; I-V do L.V
B.2 NIGH-RESOLUUTIN DISENTANGLED SEQUENTIAL DIFFUSION AUTOENCODER
In addition to transitioning to real-world data, our goal manage high-resolution data for unsuper- vised sequential disentanglement, for the first time. Drawing inspiration from Rombach et al. (2022), we incorporate perceptual image compression, which combines an autoencoder with a perceptual loss (Zhang al., 2018) and patch-based adversarial objective (Dosovitskiy & Brox, 2016; Esser et al., 2021; Isola et al., 2017). Specifically, we explore two main variants of the autoencoder. The first variant applies small Kullback-Leible penalty encourage the learned latent space to approximate a standard normal distribution, similar to a VAE (Kingma, 2013; Rezende et al., 2014). The second variant integrates vector quantization layer (Van Den Oord et al., 2017; Razavi et al., 2019) within the decoder. Empirically, we find that the VQ-VAE-based model performs better when combined with our method. Given pre-trained encoder & and decoder D, we extract xj = E (xx)). which represents a low-dimensional latent space where figh-frequency.. imperceptible details are abstracted away. Finally, x can be reconstructed from the latent x by applying the decoder xō = (xx)). The EDM formulation in Eq. 4 makes relatively strong assumptions about the mean and standard deviation of the training data. To meet these assumptions, we opt to normalize the training data globally rather than adjusting the value of dataa which could significantly affect other hyperparameters (Karras et al., 2024). Therefore, we keep dana at its default value of 0.5 and ensure that the latents have a zero mean during dataset preprocessing. When generating sequence elements, we reverse this normalization before applying D.
B.3
PRIOR MODELING
We model the prior static and dynamic distribution with pro(so, db.v ST, d. To sample static and dynamic factors, we train a separate latent DDIM model (Song et al., 2020). Then, we can extract the factors by sampling noise, and reversing the trained model. Specifically, we learn (2 2 where 2o (So, d).V) are the outputs of sequential semantic encoder. The training is done by simply optimizing the Liatent with respect to DDIM's output (-(-):
T
Ltatent Σ
Rggi .e [Z
V.V 1) - Eell]
(11)
where E, E RV N 1). V dimensions respectively. Additionally, z.:V is the noise version of z; as described in Song et al. (2020). For designing the architecture of latent model, we follow Preechakul et al. (2022) and is based on 10 MLP layers. Our network architecture and hyperparamters are provided in Tab. 8.
the sequence length, s, d are the static and dynamic factors
B.4 REVERSE PROCESSES
The detailed reverse sampling algorithm is provided in Alg. 1. We follow Karras et al. (2022) sampling techniques, however, each step in our reverse process is conditioned on the latent static and dynamic factors extracted by our sequential semantic encoder. As in Preechakul et (2022), we observe that auto auto-encoding is improved significantly when using the stochastic encoding technique. Since we have a different reverse process, we provide the algorithm for stochastic encoding for our modeling in Alg. 2. Finally, when performing conditional swapping, we observe that performing stochastic encoding on the sample from which we borrow the dynamics and using it as an input to Alg. 1, improves the results empirically. That is, given two sample videos x, & Po. to create new sample x, conditioned on the static factor of x and dynamic features of x, we use the stochastic encoding of in Alg.
17

---

Algorithm 1 Conditioned Stochastic Sampler with o(t) = and s(t) 1.
1: procedure CONDIIINNDSTOCHHSSIISSSPPLLLLLLL
4E (e(00....N)) N11) 20 x..
Shaice)
2: if I.V # None then
3:
Algorithm output
4: else
sample X (O, ty 1)
min Sama N
vZ-1)
if 1:6 (Smiinnn
6: for (N. , 13 do
b 76
sample 6; ~ NO, Suise I)
otherwise
8:
f, t ttt,
P Sclect temporarily increased noise level f Add new noise to move from f, to t; Evaluate dx/d/
9:
& - +x + V
6,
10:
d. [x] Dx zb; (.)
11:
x t x + (4-1 - l:)d
> Take Euler step from & to 4:-1
12:
if f # 0 then
13:
d/ +- (x|_1 Do ( -1: /ta /ti-1
b Apply 2dd order correction
14:
x,- & (fi-1 ( dd)
15:
return Xu
Algorithm 2 Stochastic Encoding with (t) = and s(1) = 1.
1: procedure STOCHASTICENCODEERL.. 4e<o.....N); Vie(..... N-1,, xb sample 2o (0, 1)
20 2b.1)
min
Sqg vz-1)
if 1. (umm otherwise
3: for E (O....,
1] do
7
4: sample E; ~ (O, Suise I)
5:
f < t, 7it,
D. Select temporarily increased noise level f Add new noise to move from t, to t; Evaluate dx /df to
6:
à. + 2. + Vt,
6,
7:
d, - (x] -Do(x] t.)t
8:
x[+1 < x + +1++1 lid.
b Take Euler step from & to 4++1
9:
if 1.41 amx then
10:
d' + (x]=1 (x 2o;ts+1)) /ta41
b Apply 2dd order correction
11:
x
(f++1 4:) ( + d!)
12:
return x
1:1
C
The hyperparameters used in our autoencoder are listed Tab. and Tab, 7, detailing the configura- tions for each dataset: MUG, TaiChi-HD, VoxCeleb, CelebV-HQ, TIMIT, LibriSpeech, PhysioNet, Air Quality and ETTh We provide the values of essential parameters such as sequence lengths. batch sizes, learning rates, and the use of P Preaa and Puua to manage noise disturbance during training. In addition, the table specifies whether VQ-VAE was employed. Tab. 8 outlines the architecture of our latent DDIM model, including batch size, number of epochs, MLP layers, hidden sizes, and the 3 scheduler. These details are essential for understanding the model's 's structure and its training process. For the VQ-VAE model, we utilized the pre-trained model from (Rombach et al., 2022) with hyperparameters f = 8, Z = 256, and = 4. which encodes a frame of size 3 x 256 256 into a latent representation of size x 32 32.
D DATASETS
MUG. The MUG facial expression dataset, introduced by Aifanti et al. (2010), contains image sequences from 52 subjects, each displaying six distinct facial expressions: anger, fear, disgust, happiness, sadness, and surprise. Each video sequence in the dataset ranges from 50 to 160 frames. To create sequences of length 15, as done in prior work (Bai et al., 2021), we randomly select 15 frames from the original sequences. We then apply Haar Cascade face detection crop the faces and resize them to 64 x 64 pixels, resulting in sequences of x E IR 15x3x64x64 The final dataset comprises 3,429 samples. In the case of of the zero shot experiments resize the images to 256 x 256 pixels.
18

---

Table 6: Hyperparameters for Video datasets.
Dataset
MUG
TaiChi-HD
VoxCeleb
CelebV-HQ
P Punen
-.2
-1.2
-0.4
-0.4
Pad
1.2
1.2
1.0
1.0
NFE
71
63
63
63
Ir
le-4
1c-4
lc-4
1c-4
bsz
8
16
16
16
#Epoch
1600
40
100
450
Dataset repeats
150
1
1
s dim
256
512
512
1024
d dim
64
64
12
16
hidden dim
128
1024
1024
1024
Base channels
64
64
192
192
Channel multipliers
[1, 2,
Attention placement
[2]
Encoder base ch Encoder ch. mult.
64
192
192
[1, 2,
Enc. attn. placement Input size
[2]
64
64
x 64
64
3 x 256 256
3 x 256 256
Seq len
15
10
10
10
Optimizer
Adam (weight decay= le-5)
Backbone
Unet
GPU
RTX 4090
3 RTX 4090
Table 7: Hyperparameters for audio and TS.
Dataset
TIMIT
LibriSpeech
Physionet
Airg
ETTH
Pmmen
00.
-0.4
-0.4
-0.4
-0.4
Pad
1.0
1.0
1.0
1.0
1.0
NFF
63
63
63
63
63
Ir
Le-4
le-3
5e-5
1e-4
le-4
bsz
128
128
30
10
10
#Epoch
750
200
200
200
200
s dim
32
32
24
16
16
d dim
2
2
4
4
hidden dim
128
256
96
512
512
Base channels
256
64
256
256
128
Channel multipliers
[4. 4, 4, None
Attention placement
Encoder base ch Encoder ch. mult. Enc. attn. placement Input size Seq len
128
128
96
128
256
[4, 4, 4, None
80
80
10
10
6
68
68
80
672
672
Optimizer
Adam (weight decay= le-5)
Backbone
MLP
GPU
RTX 4090
TaiChi-HD. The TaiChi-HD dataset, introduced by Siarohin et al. (2019), contains videos of full human bodies performing Tai Chi actions. We follow the original preprocessing steps from FOMM (Siarohin et al., 2019) and use 64 64 version of the dataset. The dataset comprises 3,081 video chunks with varying lengths, ranging from 128 to 1,024 frames. We split the data into 90% for training and 10% for testing. To create sequences of length 10, similar the approach used for the MUG dataset, we randomly select 10 frames from the original sequences. The resulting sequences resized to 64 64 pixels, forming x 1g10xxx61x61.
19

---

Table 8: Network architecture of our latent DDIM.
Parameter
MUG
TaiChi-HD
VoxCeleb
Celebv-HQ
Batch size
128
128
128
128
#Epoch
500
500
200
1000
MLP layers (N)
MLP hidden size
1216
5008
2528
4736
B scheduler
Linear
Learning rate
le-4
Optimizer
AdamW (weight decay= le-5)
Train Diff
1000
Diffusion loss
L2 loss with noise prediction
GPU
RTX 4090
VoxCeleb. The VoxCeleb dataset (Nagrani al., 2017) is a collection of face videos extracted from YouTube. We used the preprocessing steps from Albanie et al. (2018), where faces are extracted, and the videos are processed 25/6 fps. The dataset comprises 22,496 videos and 153,516 video chunks. We used the verification split, which includes 1,211 speakers in the training set and 40 different speakers in the test set, resulting in 148,642 video chunks for training and 4,874 for testing. create sequences of length 10, we randomly select 10 frames from the original sequences. The videos are processed at a resolution of 256 256 resulting in sequences represented as x E I210x3x256
V-Q. The V- dataset (Zhu et al., 2022) is a large-scale collection of high-quality video clips featuring faces, extracted from various online sources. The dataset consists of 35,666 video clips involving 15,653 identities, with each clip manually labeled with 83 facial attributes, including 40 appearance attributes, 35 action attributes, and 8 emotion attributes. The videos were initially processed at a resolution of 512 x 512. We then used Wang et al. (2021) to crop the facial regions, resulting videos 256 256 resolution. To create sequences length 10, we randomly selected 10 frames from the original sequences, producing sequences represented as x E R 121x3x226x2266
TIMIT. The TIMIT dataset, introduced by Garofolo (1993), is collection of read speech designed for acoustic-phonedt research and other speech- tasks. It contains 6300 utterances, totaling approximately 5.4 hours of audio recordings, from 630 speakers (both men and women). Each speaker contributes sentences, providing diverse and comprehensive pool speech data. To pre-process the data we use mel-spectogram feature extraction with 8.5ms frame shift applied to the audio. Subsequently, segments of 580ms duration, equivalent 68 frames, are sampled from audio and treated as independent samples.
LibriSpeech. The LibriSpeech dataset Panayotov et al. (2015) is corpus of read English speech derived from audiobooks, containing 1,000 hours of speech sampled at kHz. For our training, we used the ra n-cl e6 subset, which consists of 363.6 hours of speech from 921 speakers. As validation and test sets, we use dev-cles ean and test-cle celtnclean, each containing 5.4 hours of speech from 40 unique speakers, where there processing, we extract mel-spectrogram features with an 8.5 ms frame shift applied to the audio. We then sample segments of 580 ms duration (equivalent to 68 frames) from the audio, treating them as independent samples.
identity overlap across all subsets. For pre-
PhysioNet. The PhysioNet ICU dataset (Goldberger et al., 2000) consists of medical time series data collected from 12,000 adult patients admitted to the Intensive Care Unit (ICU). This dataset includes time-dependent measurements such as physiological signals. laboratory results, and relevant patient demographics like age and reasons for ICU admission. Additionally, labels indicating in- hospital mortality events are included. Our preprocessing procedures follow the guidelines provided in (Tonekaboni et al., 2022).
Air Quality. The UCI Beijing Multi-site Air Quality dataset (Zhang et al., 2017) comprises hourly records of air pollution levels, collected over four-year period from March 1, 2013, February 28,
20

---

2017, across 12 monitoring sites in Beijing. Meteorological data from nearby weather stations of the China Meteorological Administration is also included. Our approach to data preprocessing, as described in (Tonekaboni et al., 2022), involves segmenting the data based different monitoring locations and months of the year.
ETTh1. The ETThl dataset is subset of the Electricity Transformer Temperature (ETT) dataset, containing hourly data over a two-year period from two counties in China. The dataset is focused on Long Sequence time series Forecasting (LSTF) of transformer oil temperatures. Each data point consists of the target value (oil temperature) and six power load features. The dataset is divided into training, validation, and test sets, with a 12/4/4-month split.
E METRICS
Average Keypoint Distance (AKD). To evaluate whether the motion in the reconstructed video is preserved, we utilize pre-trained third-party keypoint detectors on the TaiChi-HD, VoxCeleb, CelebV-HQ, and MUG datasets. For the VoxCeleb, CelebV-HQ and MUG datasets, we employ the facial landmark detector from Bulat & Tzimiropoulos (2017), whereas for the TaiChi-HD dataset, we use the human-pose estimator from Cao et al. (2017). Keypoints are computed independently for each frame. AKD is calculated by averaging the L1 distance between the detected keypoints in the ground truth and the generated video. The TaiChi-HD and MUG datasets are evaluated at a resolution of 64 x 64 pixels, and the VoxCeleb and Celeb datasets at 256 x 256 pixels. If the model output is at a lower resolution, it is interpolated to 256 x 256 pixels for evaluation.
Average Euclidean Distance (AED). To assess whether the identity in the reconstructed video is preserved, we use the Average Euclidean Distance (AED) metric. AED is calculated by measuring the Euclidean distance between the feature representations of the ground truth and the generated video frames. We selected the feature embedding following the example set in Siarohin et al. (2019). For the VoxCeleb, CelebV-HQ. and MUG datasets, use VGG-FACE for facial identification using the framework of Serengil & Ozpinar (2020), whereas for TaiChi-HD, we use a network trained for person re-identification (Hermans al., 2017). TaiChi-HD and MUG are evaluated resolution of 64 x 64 pixels, and VoxCeleb and CelebV-HQ at 256 x 256 pixels.
To ensure fairness when measuring AED and AKD, we created a predefined dataset of example pairs, ensuring that all models are evaluated on the exact same set of pairs. This is important because when measuring quantitative metrics, the results may vary depending on the dynamics swapped between two subjects, as e.g., the key points in AKD in the original video are influenced by the identity of the person. To address this issue, we establish a fixed set of pairs for a consistent comparison across all methods.
Accuracy (Acc). As in Naiman et al. (2023), we used this metric for the MUG dataset to evaluate a model's 's ability to preserve fixed features while generating others. For example, dynamic features are frozen while static features are sampled. Accuracy is computed using a pre-trained classifier, referred to as the "judge"" which is trained on the same training set as the model and tested on the same test set. For the MUG dataset, the classifier checks that the facial expression remains unchanged during the sampling of static features.
Inception Score (IS). The Inception Score is a metric used evaluate the performance of the model generation. First, we apply the judge, to all generated videos arg I.V obtaining the conditional predicted label distribution p (y)x+ Next, we compute p(y). the marginal predicted label distribution, and calculate the KL-divergence KL [p (ylld [lp(v)]- Finally, the Inception Score is computed as IS- exp
KL [p (v)l) (I(())). We use this metric evaluate our results on MUG dataset.
Inter-Entropy (Hllbb)))) )). This metric reflects the confidence of the judge in its label predic- tions, with lower inter-entropy indicating higher confidence. It is calculated by passing k generated sequences {z=! } jisk into the judge and computing the average entropy of the predicted label distribu- tions: $ E H H(py (xd We use this metric evaluate our results on MUG dataset.
21

---

Intra-Entropy (H).. This metric measures the diversity of the generated sequences, where higher intra-entropy score indicates greater diversity. is computed by sampling from the learned prior distribution p(y) and then applying the judge to the predicted labels y. We use this metric to evaluate our results on the MUG dataset.
Equal Error Rate (EER) metric is widely employed in speaker verification tasks. The EER represents the point at which the false positive rate equals the false negative rate, offering balanced measure of performance in speaker recognition. This metric, commonly applied to the TIMIT dataset, provides robust evaluation of the model's ability to disentangle features relevant speaker identity.
EER.
The Area Under the Precision-Recall Curve (AUPRC) is metric that evaluates the balance between precision and recall by measuring the area beneath their curve. A higher AUPRC reflects superior model performance, with values nearing 1 being optimal, indicating both high precision and recall.
AUPRC.
The Area Under the Receiver Operating Characteristic Curve (AUROC) measures the trade-off between true positive rate (TPR) and false positive rate (FPR), quantifying the under the curve of these rates. A higher AUROC signifies better performance, with values close to being desirable, representing model that distinguishes well between positive and negative classes.
AUROC.
MAE. Mean Absolute Error (MAE) calculates the average magnitude of errors between predicted and observed values, offering a simple and intuitive measure of model accuracy. As computes the average absolute difference between predicted and actual values, MAE is resistant to outliers and provides a clear indication of the model's prediction precision.
DNSMOS. Deep Noise Suppression Mean Opinion Score (DNSMOS (Reddy et al., 2021)) is a neural network-bascd metric introduced to estimate the perceptual quality of speech processed by noise suppression algorithms. Trained to predict human Mean Opinion Scores (MOS), DNSMOS provides no-reference quality assessment that correlates strongly with subjective human judgments. evaluates both the speech quality and the effectiveness noise reduction, offering a comprehensive measure of audio clarity and intelligibility. This metric is especially useful in evaluating real- world performance of speech enhancement systems without the need for costly and time-consuming human listening tests.
A
B
D
6
Figure 5: Rows A and B are two inputs from the test set. Row C shows a dynamic swap example, using the static of A and dynamics of B. In row D we extract the same person from A, but with the dynamics as labeled in B. Finally, in row E, we extract the same person from A with the dynamics that are predicted by the classifier.
F MUG AND JUDGE METRIC ANALYSIS
While our results show significant improvement previous methods VoxCeleb (Nagrani et al., 2017), CelebV-HQ (Zhu et al., 2022), and TaiChi-HD (Siarohin et al., 2019), both in terms of disentanglement and reconstruction, performance MUG (Aifanti al., 2010) is only on par
22

---

A
B
Figure 6: Rows A and are two inputs from the test set. Row C shows a dynamic swap example, using the static of A and dynamics of B. In row D we extract the same person from A, but with the dynamics as labeled in B. Finally, in row E, we extract the same person from with the dynamics that are predicted by the classifier.
Table 9: Judge benchmark disentanglement metries MUG.
MUG
Method
Acct
ISt
H (vla))
Hwy)
Reconstruction (MSE)
MoCoGAN
63.12%
4.332
0.183
1.721
DSVAE
54.29%
3.608
0.374
1.657
R-WAE
71.25%
5.149
0.131
1.771
S3VAE
70.51%
5.136
0.135
1.760
SKD
77.45%
5.569
0.052
1.769
C-DSVAE
81.16%
5.341
0.092
1.775
SPYL
85.71%
5.548
0.066
1.779
1.311e-3
DBSE
86.90%
5.598
0.041
1.782
1.286e-3
Ours
81.15%
5.382
0.090
1.773
2.669e-7
with the state-of-the-ee art methods. Since MUG labeled dataset, the traditional evaluation task involves the unconditional generation of static factors while freezing the dynamics, resulting in altering the appearance of the person. The generated samples are then evaluated using an off-the-shelff judge model (See App. E), which is neural network trained to classify both static and dynamic factors. If the disentanglement method disentangles these factors effectively, we expect the judge to correctly identify the dynamics while outputting different predictions for the static features, since the latter were randomly sampled and should differ from the original static factor.
Surprised by results MUG, investigated the failure cases to understand the limitations of our model. In particular, we examined scenarios where we freeze the dynamics and swap the static features between samples, and then we generate the corresponding output. In Fig. show an example where the static features of the second row are swapped with those of the first row, and the resulting generation is displayed in the third row. We observe that while the dynamics from the second row are well-preserved, the generated person retains the identity of the first row. However, the classifier incorrectly predicts the dynamics for the sequence. To further investigate this, we extracted ground-truth -ruth example of the person from the first row in the dataset expressing the expected emotion and the predicted one. the last two rows of Fig. 5, we show the same person with predicted dynamics (fourth row) and the same person with the dynamics that the classifier predicted (fifth row). We provide another example of the same phenomenon Fig.
We observe that while the judge predicts the wrong label for our generated samples in rows C, the facial expressions of the people there align better with the actual dynamics rows B. This suggests that the classifier is biased towards the identity when predicting dynamics, potentially forming a discrete latent space where generalization to nearby related expressions is not possible. Importantly, the judge attains 99% accuracy on the test set. We conclude that utilizing judge can be problematic for measuring new and unseen variations in the data. This analysis motivates us to present the AKD and AED, as detailed above in App. E.
23

---

G ADDITIONAL EXPERIMENTS
G.1 DEPENDENT VS. INDEPENDENT PRIOR MODELING
In Sec. describe our approach to prior modeling, highlighting our decision generate latent factors dependently rather than independently. as done in previous state-of-the-ar methods. Beyond being a parameter- and time-efficient choice, we empirically validate the advantages of our approach in the following experiment.
In this experiment, we compare two setups: (1) dependent generation of static and dynamic latent vectors, and (2) independent generation of these latent vectors using two latent DDIM models: one for the static vector and another for the dynamic vectors. To quantitatively assess the effectiveness of both approaches, we measure the Frä@chet Video Distance (FVD) Blattmann et al. (2023), metric derived from the well-established FID score for videos. This metric evaluates how well a generative model captures the observed data distribution, where lower scores indicate better performance.
We conduct our evaluation on the VoxCeleb dataset, training two latent models. The independent model achieves an FVD score of 75.03, whereas our dependent approach achieves significantly lower score of 65.23, representing a 13% improvement. This result underscores the expressive advantage of modeling latent factors dependently.
G.2 ADDITIONAL ANALYSIS OF DIFESDA DISENTANGLEMENT COMPONENTS
This section explores the impact of two key components of our method on disentanglement quality: i) the static latent factor so shared across all time steps T, and ii) the dimensionality of the dynamic latent factor dj-
To analyze these effects, we trained four models the VoxCeleb dataset for 100 epochs, maintaining a static latent dimension of 128 while varying the size the dynamic latent factor and whether the static latent factor was shared not. The models were evaluated using our conditional swapping protocol and a verification metric based on the VGG-FACE framework proposed in Serengil & Ozpinar (2020). Specifically, we assessed identity consistency by freezing the static factor and swapping the dynamic factor, with the verification score representing the percentage of cases where identity was correctly preserved across frames.
As shown in Tab. 10, our results indicate that the optimal performance (first row of the table) is achieved when dā has smaller dimensionality, and the static factor is shared. Other configurations reveal significant trade-offs: increasing dī dimensionality results in higher AED scores but reduced verification accuracy, indicating weaker disentanglement of the static factor. Similarly, when 80 not shared, the AKD score degrades significantly, suggesting ineffective disentanglement of the dynamic factor. These findings underscore the importance both (i) and (ii) in achieving robust sequential disentanglement.
Table 10: Disentanglement effect of VoxCeleb dataset
di size
8 shared?
Verification ACC f (Static Frozen)
AED J. (static frozen)
AKD J (dynamics frozen)
16
64.36%
0.925
2.882
128
18.03%
1.054
2.077
16
56.75%
0.898
12.64
128
48.41%
0.980
12.28
To strengthen these observations, we repeated the study on MUG, where ground-truth static (identity) and dynamic (action) labels allow a clean swap-task evaluation (Tab. 11). The trends mirror VoxCeleb: (i) sharing the static representation & is critical when & is not shared, the dynamic pathway collapses. In this case, identity remains stably preserved under static swap (as it should), but dynamic recognition deteriorates toward chance because temporal variation is absorbed into or suppressed by the static channel, indicating failed sequential disentanglement; and (ii) constraining the dimensionality of the dynamic latent djj provides a helpful bottleneck that limits identity leakage and sharpens the separation between static and dynamic factors. Although the absolute gaps MUG are more modest than in Tab. 10, the qualitative agreement across datasets reinforces that both sharing s and limiting the capacity of di key for robust sequential disentanglement.
24

---

Table 11: Disentanglement effect of MUG dataset
di size
8 shared?
Verification ACC 1 (Static Frozen)
Action ACC y (dynamics frozen)
64
95.59%
80.08%
256
/
92. 12%
81.28%
64
99.69%
16.71%
256
X
99.83%
18. 18.18%
G.3
SPEECH QUALITY AND RECONSTRUCTION COMPARISON
This section discusses the results of speech reconstruction and quality evaluation presented table 12 on the LibriSpeech dataset. We compare the reconstruction performance using the Mean Squared Error (MSE) on the spectrograms and assess speech quality using the Deep Noise Suppression Mean Opinion Score (DNSMOS) (Reddy et al., 2021). The DNSMOS metric has maximum score of but the original (reference) dataset achieves score of 3.9, as shown in the REF row of the table. As can be seen in the table, our model outperforms all comparable methods, achieving the lowest MSE and the highest DNSMOS among the evaluated approaches.
Table 12: Disentanglement and generation quality metrics on Libri Speech. For generation quality, report MSE the spectogram and Deep Noise Suppression Mean Opinion Score (DNSMOS).
Method
MSEL
DNSMOST
REF
3.9
Libri Speech
DSVAE
5.53e-2 2
3.13
SPYL
4.10c
2.21
DBSE
6.72c
2.88
Ours
1.83e 4
3.41
G.4 GENERATIVE QUALITY COMPRESSTON
This section discusses the generative quality results shown in Table 13, evaluated using the FrÅ&chet Video Distance (FVD) on the VoxCeleb dataset. We generated the same number of samples in the test set and computed the FVD score against the test set. This process was repeated five times for each model using different five diffrret seeds to obtain robust estimate. We report the mean FVD along with the standard deviation. The results demonstrate that our model outperforms existing state-of-te of-the-art sequential disentanglement models the video generation task.
Table 13: FrA@chet Video Distance (FVD) results on VoxCeleb dataset assess video generation quality. All experiments were conducted across five different random seeds to ensure robustness and account for variability in generation.
Model
FVDJ
SPYL DBSE
582.28 + 1.15
1076.44 + 2.22
Ours
65.23 + 0.81
G.5 EFFECT OF VQ- VAE ON ZERO-SHOP SWAPS
In this subsection, we provide further details regarding the experiment aimed at evaluating the role of the VQ-VAE in zero-shot cross-dataset transfer. The experiment designed to isolate the
25

---

contribution of the VQ-VAE by removing it entirely and training DiffSDA directly in pixel space downsampled VoxCeleb. This setup allows us to examine how well the model maintains disentan- glement when the unified latent space provided by VQ-VAE is absent. The observed deterioration in generalization, particularly in the stability and consistency of identity and expression transfer, indicates that the VQ-VAE plays crucial role in producing coherent cross-dataset representations that support strong zero-shot disentanglement as seen in the examples of Fig. 7.
26

---

Real videos
Swergeed videos
Real videos
Swappsd videos
Real videos
Swapped videos
Real videos
Swapped videos
Figure 7: Each panel shows, in the first and second rows, real video pairs from the VoxCeleb and MUG datasets, respectively. We perform conditional swapping using two models: one trained on VoxCeleb with VQ- VAE and another trained on VoxCeleb without VQ- VAE at resolution of 64 x 64. The resulting swaps are shown in the final two rows. In the first two examples, the dynamics are taken from the VoxCeleb video, while the last two examples, the dynamics come from the MUG video.
27

---

H ADDITIONAL RESULTS
H.1 EXTENDED BENCHMARK RESULTS
In this section, expand the comparisons from the main paper by adding results for additional baselines on the same tasks and datasets. Specifically, we include time-series prediction on PhysioNet and ETTh1 (Tab. 14), time-series classification on PhysioNet and Air Quality using only the static latents (Tab. 15), and disentanglement on TIMIT reported as Static EER, Dynamic EER, and Disen- tanglement Gap (Tab. 16). Across these additions, our method remains competitive or superior the added baselines. We also add CDSVAE results on MUG and provide a MUG-only summary (Tab. 17): under swaps, model better preserves identity (with z frozen) and motion (with zd frozen), and in reconstruction achieves uniformly lower SAEAAAKDDSEE-uuupporioe stronger disentanglement and higher-fidelity reconstructions.
Table 14: Time series prediction benchmark.
PhysioNet
ETTh1
Method
AUPRC
AUROC 1
MAE
VAE
0.157 = 0.05
0.564 0.04
13.66 + 0.20
GP-VAE
0.282 = 0.09
0.699
14.98 - 0.41
C-DSVAE
0.158 0.01
0.001
12.53 0.88
GLR
0.365 + 0.09
0.00
12.27 + 0.03
SPYL
0.367 0.02
0.06
12.22 - 0.03
DBSE
0.473 = 0.02
0.858 0.01
11.21 = 0.01
Ours
0.50 + 0.006
0.87 + 0.004
9.89 + 0.280
RF
0.446 0.04
0.802 0.04
10.19 + 0.20
Table 15: Time series classification benchmark.
Method
PhysioNet
Air Quality
VAE
34.71 + 0.23
27.17 + 0.03
GP-VAE
42.47 + 2.02
36.73 1.40
C-DSVAE
32.54 + 0.00
47.07 1.20
GLR
38.93 + 2.48
50.32 + 3.87
SPYL
46.98 + 3.04
57.93 3.53
DBSE
56.87 + 0.34
65.87 0.01
OUR
64.6 + 0.35
69.2 + 1.50
RF
62.00 +: 2.10
62.43 0.54
Table 16: Disentanglement metrics TIMIT
Method
Static EER,
Dynamic EER
Dis. Gapt
FHVAE
5.06%
22.77%
17.71%
DSVAE
5.64%
19.20%
13.56%
R-WAE
4.73%
23.41%
18.68%
S3VAE
5.02%
25.51%
20.49%
SKD
4.46%
26.78%
22.32%
C-DSVAE
4.03%
31.81%
27.78%
SPYL
3.41%
33.22%
29.81%
DBSE
3.50%
34.62%
31.11%
Ours
4.43%
46.72%
42.29%
28

---

Table 17: MUG results only, Preservation of objects (AED) and motions (AKD) under conditional swapping, and reconstruction errors (AED/AKD/MSE), Labels "static frozen' and 'dynamics frozen' correspond to samples z" and z².
CDSVAE
SPYL
DBSE
Ours
Swap
AED +. (static frozen)
0.774
0.766
0.773
0.751
AKD J. (dynamics frozen)
1.170
1.132
1.118
0.802
Reconstruction
AED +
0.56
0.49
0.49
0.11
AKD +
0.50
0.47
0.48
0.06
MSE L
0.001
0.001
0.001
3e-7
H.2 RECONSTRUCTION RESULTS
In Figs. 8 to 11, we present several qualitative reconstruction examples across all datasets.
Figure Reconstruction results of CelebV-HQ (256 256). The first row each pair is the original video and the second row is its reconstruction.
H.3 ADDITIONAL RESULTS: CONDITIONAL. SWAP
In what follows, we present more results for the conditional swapping experiment from the main text (Sec. 4.1). In each figure, the first two rows show the original sequences (real videos). The third and
29

---

OO
TO0
8
8
8
8
8
3
8
8
8
8
3
Figure 9: Reconstruction results of VoxCeleb (256 x 256). The first row for each pair is the original
video and the second row is its reconstruction.
fourth rows are the results of the conditional swap where we change the dynamic and static factors, respectively. We show our results for all datasets in Figs. 12 to 15.
H.4 ADDITIONAL RESULTS: UNCONDITIONAL SWAP
In addition to the conditional and zero-shot shot tasks considered above, we can also perform such tasks in unconditional manner. Specifically, given real sequence x v.V with its factors (s, d!'V", we can unconditionally sample new ($, d!:V) using our separate DDIM model (see Sec. 3). We then reconstruct the static swap ($, d and the dynamic swap (s, d':V) similarly as described above. In Fig. 16. present unconditional swap results on CelebV-HQ (left), VoxCeleb (middle), and TD (right). The middle rows represent the original sequences, whereas the top and bottom demonstrate dynamic and static swaps, respectively. Across all datasets and swap settings, our approach succeeds in modifying the swapped features while preserving the frozen factors, either in the static or the dynamic examples. In addition, also present more results where each figure is composed of separate panels. In each panel, the middle row represents the original sequence. In the top row, we sample new dynamic factors and freeze the static factor. In the bottom row below, we sample a new static factor and freeze the dynamics. We show our results on all datasets in Figs. 17 to 20.
H.5 ADDITIONAL RESULTS: ZERO-SHOT DISENTANGLEMENN
Here we extend the results from Sec. 4.2. We provide additional examples of conditional swapping when the model is trained dataset and evaluated another dataset, unseen during training.
30

---

Figure 10: Reconstruction results TaiChi-HD, The first row for each pair the original video and the second row is its reconstruction.
Specifically, in Fig. 21, we show examples where the model is trained on VoxCeleb and tested on MUG. Additionally, in Fig. 22, the model is trained on VoxCeleb and tested on CelebV-HQ. Finally, in Fig. 23, the model is trained on CelebV-HQ and tested on VoxCeleb.
H.6 ADDITIONAL RESULTS: MULTIFACTOR DISENTANGLEMENT
In this section, we present more examples for traversing the latent space, separately for the static and dynamic factors. For static factors, show in Figs. 25 to 36. There, we find different factors of variation such as Male to Female, younger to older, brighter and darker hair color, and more. Each row in the figure In addition, we present full examples of dynamic factor traversal in Figs. 37 to 48, demonstrating various factors of variation. Among the factors are facial expressions, camera angles, head rotations, eyes and mouth control, etc.
video, and the different columns represent the traversal in values (see Eq. 6).
31

---

Figure 11: Reconstruction results of MUG. The first row for each pair is the original video and the second row is its reconstruction.
32

---

Real videos
Swarmed videns
Real videgs
Shapped vadcos
Rod viddes
Swarnsd videos
Real videos
Satanned videos
Figure 12: Each panel contains pair of original videos from CelebV-HO (Real videos), and pair of conditional swapping of the dynamic and static factors (Swapped videos).
33

---

Real
Sward videos
Real videos
Swapnne videus
Real siders
Swarnedl videos
Real viders
Swapaed videos
Figure 13: Each panel contains a pair of original videos from VoxCeleb (Real videos), and pair of conditional swapping of the dynamic and static factors (Swapped videos).
34

---

Swarped
Swarped
Suapped
Swapped
Figure 14: Each panel contains pair of original videos from TaiChi-HD (Real videos), and pair of conditional swapping of the dynamic and static factors (Swapped videos).
35

---

Real videos
Swapped videos
Real videos
Swapped videos
Real videoe
Snapped videos
Real videos
Swapped videos
2
3
&
E
M
R
Figure 15: Each panel contains pair of original videos from MUG (Real videos), and a pair of
conditional swapping of the dynamic and static factors (Swapped videos).
real edmummic
static +--rca video-- ddvaamic
statie --0 vileo- dymamcc
Figure 16: Unconditional dynamic (top) and static (bottom) swap results on CelebV-HQ (left),
VoxCeleb (middle), and TaiChi-HD (right).
36

---

dynsmiis
static real
static < rell -> dynamies
statle real >> dynamies
>>> dynamies
real
static
Figure 17: CelebV-HQ unconditional swapping. The middle row represents the original video (real),
the row above shows a dynamic swap (dynamics), and the row below shows static swap (static).
37

---

< real >>> dynaniles
stotie
real -> dynamuss
statk <
real >>
Stati <
- dynamics
real
stakic
Figure 18: VoxCeleb unconditional swapping. The middle row represents the original video (real),
the row above shows a dynamic swap (dynamics), and the row below shows static swap (static).
38

---

dynamics
>>
< real
static
stalic < real >> dynamicss
dynunicss
>>
stalic < real
dynumss
< real
static
Figure 19: TaiChi-HD unconditional swapping. The middle row represents the original video (real),
the row above shows dynamic swap (dynamics), and the row below shows static swap (static).
39

---

S
Figure 20: MUG unconditional swapping. The middle row represents the original video (real), the
row above shows a dynamic swap (dynamics), and the row below shows a static swap (static).
40

---

Figure 21: Each panel contains in its first and second rows a pair of real videos from VoxCeleb and MUG, respectively. We perform conditional swapping using model that was trained on VoxCeleb, but we zero-shot swap the dynamic and static factors of MUG example (Swapped videos).
41

---

Real
vale Real
Sa-ppel
Figure 22: Each panel contains in its first and second rows pair of real videos from VoxCeleb and C-H. We perform conditional swapping using a model that was trained on VoxCeleb, but we zero-shot swap the dynamic and static factors of a CelebV-HQ example (Swapped videos).
42

---

Real videus
Swarned videos
Real videess
Swapped videus
Real valeos
Swampal videos
Real sideos
S videos
Figure 23: Each panel contains in its first and second rows pair of real videos from CelcbV-HQ and VoxCeleb. We perform conditional swapping using a model that was trained on CelebV-HQ, but we zero-shot swap the dynamic and static factors of a VoxCeleb example (Swapped videos).
43

---

facce
surprised <... real vided
fiiee night real video
<
Figure 24: Traversing the latent space of DifISDA via PCA reveals multiple dynamic variations on
CelebV-HQ, including surprised and serious expressions, and different head orientations.
<
Figure 25: Traversing between Male appearances and Female appearances.
Real video
Figure 26: Traversing over a darker hair factor.
44

---

Sharp
Real video
Blury
Figure 27: Traversing between sharper and blurry videos.
Real video
Brighter h
Figure 28: Traversing over a brighter hair factor.
45

---

>> Young
Real video
Old <
Figure 29: Traversing between younger and older appearances.
Brighter
Real
Darker
Figure 30: Traversing over skin color variations.
46

---

Female
Figure 31: Traversing between Male appearances and Female appearances.
Real video
Figure 32: Traversing over a darker hair factor.
47

---

Sharp
video
Blury
Figure 33: Traversing between sharper and blurry videos.
Real video
Brighter h
Figure 34: Traversing over a brighter hair factor.
48

---

Young
Old
Figure 35: Traversing between younger and older appearances.
Brighter
Real videdd
Darker
Figure 36: Traversing over skin color variations.
49

---

Righs
Figure 37: Traversing head rotation factor.
Right
Figure 38: Traversing over head angles.
50

---

Up
Figure 39: Traversing over up and down rotations.
Surpeise
Figure 40: Traversing over facial expressions.
51

---

Open
Figure 41: Traversing over mouth openness factor.
>
Open
Figure 42: Traversing over eyes openness factor.
52

---

Righs
Left
Figure 43; Traversing over a head rotation factor.
Right
Figure 44: Traversing over various head angles.
53

---

Up
Down
Figure 45: Traversing over up and down head rotations.
Calm
Surpeise <
Figure 46: Traversing over facial expressions.
54

---

Open
Figure 47: Traversing over mouth openness factor.
39
Close
videe
Open
Figure 48: Traversing over eyes openness factor.
55


Reviews:
[{"review_id": "1KhhNHWaRG", "reviewer": "Reviewer_DzJi", "summary": "This paper propose a unsupervised diffusion model that can disentangle factors in a time series data. The model is insprired on a principled probabilistic model, with the ability of both dynamic-static and conponent level disentanglement. The extensive experiments results verify the conclusion.", "questions": "1. The video is generated frame by frame, which may reduce the spatio-temporal consistency in the video generation. Is it possible to adapt this model to LVDM or similar models to strength its video generation ability?\n2. Can the unsupervisely learned concepts be combined with other supervised condition signals? For example, how this method can contribute to the current T2V generation framework?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. A probabilitic model is provided, which provide a theoritical guarantee for disentanglement in diffusion models.\n2. Extensivce experiments are provided, which makes the paper more convincing.\n3. The paper is clear and easy to follow.", "weaknesses": "### Major Concerns\n1. This paper assumes $d$ and $s$ are independent. However, in many real world senarios, they cannot be independent, as sometimes the content of a video can decide the type of actions to be taken in a video. This limits the generalization of thie model. \n2. The dataset is limited to simple scenarios like human face, where this assumption is reasonable. However, the performance in open-domain dataset is not verified.\n3.  This paper claims the ability of disentanglement, but some traditional disentanglement metrics are not compared, i.e., SAP, modularity, e.t.c..\n4. The baseline models in the paper are out-dated, especially for the video generation part. \n### Minor Concerns\n1. Typo: line 203 pr-evious", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Hogm9Vac0p", "reviewer": "Reviewer_d96r", "summary": "DiffSDA is a modal-agnostic diffusion sequential disentanglement autoencoder that factorizes static and dynamic factors using a single, unified score-estimation loss.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. It allows static and dynamic factors to be interdependent, which makes the model more general.\n\n2. It can be applied to video, audio and time-series data, and achieves sota performance.\n\n3. It present a new evaluation protocol for high-quality visual sequential disentanglement.", "weaknesses": "1. I am curious to what extent this model can reduce hyperparameter tuning. The paper claims that other methods always rely on multiple loss terms, making the optimization process more complex, whereas this model only requires a single standard loss term. However, in Appendix B.2, it seems that the VQ-VAE is pre-trained with a perceptual loss and a patch-based adversarial objective, and in Appendix B.3, you train a latent DDIM prior. Could you elaborate on how this method is actually simpler than GAN- or VAE-based approaches that can also disentangle dynamic and static components?\n\n2. In Figure 3, it seems that SPYL failed to reconstruct the image well. Since reconstruction is usually simpler than disentanglement, this might be due to bad hyperparameters. If such trivial failures like SPYL’s could be avoided, the comparison would be more meaningful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Dui00XlMz", "reviewer": "Reviewer_jQnL", "summary": "This paper proposes DiffSDA, a novel framework for unsupervised sequential disentanglement that leverages diffusion models to separate static and dynamic factors of variation across diverse modalities such as video, audio, and time-series data.\nThe key claimed contributions are:A novel, modal-agnostic probabilistic framework for sequential disentanglement based on diffusion processes, which models static and dynamic factors as interdependent.An efficient design that achieves disentanglement using a single, unified score estimation loss, simplifying optimization compared to methods requiring multiple loss terms.Demonstrated effectiveness on high-dimensional, real-world data, with capabilities for zero-shot disentanglement and further factorization into multiple interpretable factors via post-hoc PCA.The introduction of a new evaluation protocol for video-based disentanglement, including high-resolution datasets and unsupervised metrics (AED, AKD).", "questions": "1.What key property of the diffusion model or its learned representations enables such strong cross-dataset, zero-shot disentanglement?\n2.Are the semantic factors discovered via PCA (e.g., gender, age) consistent and stable across different random seeds and model initializations?\n3.From an optimization standpoint, please justify the efficiency of the single loss formulation over a multi-term objective, for instance, by analyzing its mitigation of gradient conflict.\n4.Please provide a quantitative comparison of training/inference time and GPU memory footprint against key baselines like SPYL and DBSE.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The work successfully bridges powerful diffusion models with the challenging problem of sequential disentanglement, a domain previously dominated by VAE/GAN-based approaches (e.g., SPYL, DBSE). The modal-agnostic claim is well-supported by experiments.\n2. The use of a single diffusion loss term is a significant advantage, effectively circumventing the complex hyperparameter tuning and multi-term loss balancing required in prior works (e.g., C-DSVAE uses five loss weights). This makes the method more accessible and robust.\n3. The paper provides extensive experiments. Qualitatively, the conditional/zero-shot swap and multifactor traversal results (e.g., Figs. 2, 3, 4, 23-47) are visually compelling and demonstrate clear disentanglement. \n4. Quantitatively, the model consistently outperforms strong baselines (SPYL, DBSE) across multiple metrics (AED, AKD, etc.) and datasets.", "weaknesses": "1. Why is the dependent modeling of static and dynamic factors theoretically justified or preferable for disentanglement? The empirical result (13% FVD improvement) is convincing, but an intuitive or formal explanation is lacking.\n2. A more detailed analysis of why the single diffusion loss naturally leads to disentanglement, beyond the empirical observations that the static factor is shared and the dynamic factors are low-dimensional, would be highly valuable.\n3. The paper highlights the efficiency of the EDM sampler (63 NFEs) but does not provide a comparative analysis of training or inference costs against the baselines. Given that diffusion models are generally more computationally intensive, a discussion of this trade-off would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GfyVmstew9", "reviewer": "Reviewer_nupF", "summary": "This work presents a new systematic approach to latent factor disentanglement of sequential data. It is based on a combination of several models 1) to extract static and dynamic components of latent factors, 2) to stochastically encode sequential data, 3) to conditionally decode sequential data based on latent factors. Additional components to deal with high-dimensional data (such as video) might be required as well, such as VQ-VAE.\n\nThe proposed method is described using the language of probabilities, akin to what we are used to when dealing with diffusion models.\n\nThe authors perform a thorough and extensive validation campaign, using an appropriate experimental protocol. They define both qualitative and quantitative metrics for the numerous tasks they address and show that the proposed method is superior to the state of the art.", "questions": "* From the introduction, the requirement for \"labeled\" data stems from, e.g., text-to-image generation. Original work from Ho et al., and from Song et al., are unconditional models which do not require labeled data.\n\n* Check your claims: when the authors say they propose a novel probabilistic approach, they are actually using standard practice in diffusion modeling. The main difference is that they combine two diffusion processes, one for learning (and generating) disentangled latents for static and dynamic components, and one for learning (and generating) the sequential data distribution, conditioned on the disentangled latents. Both models are learned concurrently using a single loss. While my remark does not want to diminish the technical contribution of the proposed method, I think that the claim for novelty is too strong.\n\n* As stated in lines 212-213, eq. (3) does not require training for the latents $(s_0, d_0^{1:V})$ generative model, thus this optimization is separate. Then, the loss in eq. (3) is not the only loss you consider in your overall training of DiffSDA, right? Then, I would like to ask the authors to clarify why in lines 221-222 they claim that only a single loss term is required.\n\n* In line 227, the last hidden representation $h^V$ is fed to a linear layer to produce $s_0$, but this is not reflected in Figure 1. This is just nitpicking, but for clarity, the figure could be easily amended to be coherent with the textual description of the architecture.\n\n* In lines 265-269, the authors mention that for high-dimensional data such as video, they abuse the notation (for ease of reading), and use symbols that denote ambient space, to indicate vector-quantized latent variables. This is clear. The question is: when considering other modalities, such as audio or general timeseries, there is no need for the VQ-VAE embeddings right?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The experimental protocol used to assess DiffSDA and compare it to alternatives is extensive and globally well done. The authors have produced convincing qualitative and quantitative results indicating that the proposed method outperforms current SOTA approaches.\n\n* Measuring quantitatively the effectiveness of disentanglement is a hard problem. The authors worked really hard on this issue and 1) tested an external judge (a classifier), and 2) proposed an unsupervised swapping metric to sidestep the limitations of 1).\n\n* This work is self contained: as a reader, I greatly appreciate the quality and depth of the material in the appendix, especially the details on the metrics, as well as the additional results and ablations.", "weaknesses": "* It's minor, but I would argue that the narrative used in the introduction about the requirement for \"labels\" for generative models is a bit misleading. Text-to-image generation indeed requires pairs of image and prompts, but unconditional models have flourished before those work, and they do not require labels. In general, I find the problem of learning disentangled representation sufficiently recognized as an important issue to address in the community, that does not need additional motivations.\n\n* I am a bit worried about the claim of \"simplicity\" of the proposed approach. I view the proposed method as a complex system, involving several models (some of which pretrained, others that require separate training) combined in a coherent design, which is far from being simple. Also, the lack of apparent hyper-parameters (a critic the authors advance for other methods on disentanglement from the state of the art), is in my humble opinion, another illustration of downplaying the complications of the proposed system. In practice, diffusion models are not so easy to train, as demonstrated by the choice of the authors to rely on an enormous amount of work that has been done to come up with \"recipies\" to elucidate the design space of diffusion models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)