A Study of Posterior Stability for Time-Series Latent Diffusion

## Abstract

Abstract Latent diffusion has demonstrated promising results in image generation and permits efficient sampling. However, this framework might suffer from the problem of posterior collapse when applied to time series. In this paper, we first show that posterior collapse will reduce latent diffusion to a variational autoencoder (VAE), making it less expressive . This highlights the importance of addressing this issue. We then introduce a principled method: dependency measure , that quantifies the sensitivity of a recurrent decoder to input variables. Using this tool, we confirm that posterior collapse significantly affects time-series latent diffusion on real datasets, and a phenomenon termed dependency illusion is also discovered in the case of shuffled time series. Finally, building on our theoretical and empirical studies, we introduce a new framework that extends latent diffusion and has a stable posterior. Extensive experiments on multiple real time-series datasets show that our new framework is free from posterior collapse and significantly outperforms previous baselines in time series synthesis.

### 1 Introduction

Latent diffusion (Rombach et al., ) has achieved promising performance in image generation and offers significantly higher sampling speeds than standard diffusion models (Ho et al., ). However, we find that, when applied to time series data, this framework might suffer from posterior collapse (Bowman et al., ), an important problem that has garnered significant attention in the literature on autoencoders (Baldi, ; Lucas et al., ), where the latent variable contains little information about the data and it tends to be ignored by the decoder during conditional generation. In this paper, we aim to provide a systematic analysis on the impact of posterior collapse on latent diffusion and improve this framework based on our analysis.

##### Impact analysis of posterior collapse.

We first show that a strictly collapsed posterior reduces the latent diffusion to a variational autoencoder (VAE) (Kingma & Welling, ), indicating that this problem makes the framework less expressive, even weaker than a vanilla diffusion model. We then introduce a principled method termed dependency measure, which quantifies the dependencies of an autoregressive decoder on the latent variable and the input partial time series. Through empirical estimation of these measures, we find that the latent variable has an almost exponentially vanishing impact on the recurrent decoder during the generation process. An example (i.e., the green bar chart) is shown in the upper left subfigure of Fig. 1. More interestingly, the upper right subfigure illustrates a phenomenon we call dependency illusion: Even when the time series is randomly shuffled and thus lacks structural dependencies, the decoder of latent diffusion still heavily relies on input observations (instead of the latent variable) for prediction.

##### New framework to solve the problem.

We first point out that the root cause of posterior collapse lies in the improper design of latent diffusion, which leads to avoidable KL-divergence regularization and a lack of mechanisms to address the insensitive decoder (Bowman et al., ). Building on these findings, we propose a novel framework that extends latent diffusion, allowing the diffusion model and autoencoder to interact more effectively. Specifically, by treating the diffusion process as a form of variational inference, we can eliminate the problematic KL-divergence regularization and permit an unlimited prior distribution for latent variables. To let the decoder be more sensitive to the latent variable, we also apply the diffusion process to simulate a collapsed posterior, imposing a significant penalty on the occurrence of dependency illusion. As demonstrated in the lower two subfigures of Fig. 1, our framework exhibits no signs of posterior collapse, such as the vanishing impact of latent variables over time.

In summary, our paper makes the following contributions:

- •
We are the first to systematically study posterior collapse in latent diffusion, introducing the technique of dependency measure (Sec. 3.2) for analysis. We show that the problem renders latent diffusion as inexpressive as a simple VAE (Sec. 3.1) and the latent variable also loses control over time series generation in this case (Sec. 3.3);
- •
We present a new framework (Sec. 4.2) that improves upon time-series latent diffusion, which eliminates the risky KL-divergence regularization, permits an expressive prior distribution, and features a decoder that is sensitive to the latent variable;
- •
We have conducted extensive experiments (Sec. 6 and Appendix F) on multiple real time-series datasets, showing that our framework exhibits no symptoms of posterior collapse (or dependency illusion) and significantly outperforms previous baselines.

We will publicly release our code upon paper acceptance.

### 2 Background: Latent Diffusion

Figure: Figure 1: The global and local dependency measures $m_{t,0},m_{t,t-1}$ (as defined in Sec. 3.2) respectively quantify the impacts of latent variable $\mathbf{z}$ and observation $\mathbf{x}_{t-1}$ on predicting the next one $\mathbf{x}_{t}$. We can see that the latent variable $\mathbf{z}$ of latent diffusion loses control over the condition generation $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$, with dependency illusion (as introduced in Sec. 3.3) in the case of shuffled time series. In contrast, our framework has no such symptoms of posterior collapse.
Refer to caption: extracted/5896890/images/study_3.png

The architecture of latent diffusion consists of two parts: 1) an autoencoder (Baldi, ) that maps high-dimensional or structured data into low-dimensional latent variables; 2) a diffusion model (Sohl-Dickstein et al., ) that learns the distribution of latent variables.

##### Autoencoder.

An implementation for the autoencoder is VAE (Kingma & Welling, ). Let $\mathbf{X}$ and $q^{\mathrm{raw}}(\mathbf{X})$ respectively denote the raw data of any form (e.g., pixel matrix) and its distribution. The encoder $\mathbf{f}^{\mathrm{enc}}$ is designed to cast the data $\mathbf{X}$ into a low-dimensional vector $\mathbf{v}=\mathbf{f}^{\mathrm{enc}}(\mathbf{X})$. To get latent variable $\mathbf{z}$, a reparameterization trick is performed as

$$ $\bm{\mu}=\mathbf{W}_{\mathrm{\mu}}\mathbf{v},\ \ \ \bm{\sigma}=\exp(\mathbf{W} _{\mathrm{\sigma}}\mathbf{v}),\ \ \ \mathbf{z}=\bm{\mu}+\mathrm{diag}(\bm{ \sigma})\cdot\bm{\epsilon},\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),$ (1) $$

where $\mathbf{W}_{\mathrm{\mu}},\mathbf{W}_{\mathrm{\sigma}}$ are learnable matrices, and $\mathrm{diag}(\cdot)$ is an operation that casts a vector into a diagonal matrix. The above procedure, which differentially samples a latent variable $\mathbf{z}$ from the posterior $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})=\mathcal{N}(\mathbf{z};\bm{\mu},
\mathrm{diag}(\bm{\sigma}^{2}))$, is called variational inference (Blei et al., ). The decoder $\mathbf{f}^{\mathrm{dec}}$ takes latent variable $\mathbf{z}$ as the input to recover the real sample $\mathbf{X}$. In VAE, the decoder output $\mathbf{f}^{\mathrm{dec}}(\mathbf{z})$ is used to parameterize a predefined generation distribution $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$.

For training, VAE is optimized in terms of the evidence lower bound (ELBO), an upper bound of the exact negative log-likelihood:

$$ $\mathcal{L}^{\mathrm{VAE}}=\mathbb{E}_{\mathbf{z}\sim q^{\mathrm{VI}}(\mathbf{ z}\mid\mathbf{X})}[-\ln p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})]+\mathrm{D} _{\mathrm{KL}}(q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})\mid\mid p^{\mathrm{ prior}}(\mathbf{z})),$ (2) $$

where the prior distribution $p^{\mathrm{prior}}(\mathbf{z})$ is commonly set as a standard Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I})$. The last term of KL divergence leads the prior $p^{\mathrm{prior}}(\mathbf{z})$ to be compatible with the decoder $\mathbf{f}^{\mathrm{dec}}$ for inference, but it is also one cause of posterior collapse (Bowman et al., ).

##### Diffusion model.

An implementation for the diffusion model is DDPM (Ho et al., ). The model consists of two Markov chains of $L\in\mathbb{N}^{+}$ steps. One of them is the diffusion process, which incrementally applies the forward transition kernel:

$$ $q^{\mathrm{forw}}(\mathbf{z}^{i}\mid\mathbf{z}^{i-1})=\mathcal{N}(\mathbf{z}^{ i};\sqrt{1-\beta^{i}}\mathbf{z}^{i},\beta^{i}\mathbf{I}),$ (3) $$

where $\beta^{i},i\in[1,L]$ is some predefined variance schedule, to the latent variable $\mathbf{z}^{0}\coloneqq\mathbf{z}\sim q^{\mathrm{latent}}(\mathbf{z})$. Here the distribution of latent variable $q^{\mathrm{latent}}(\mathbf{z})$ is defined as $\int q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})q^{\mathrm{raw}}(\mathbf{X})d
\mathbf{X}$. The outcomes of this process are a sequence of new latent variables $\{\mathbf{z}^{1},\mathbf{z}^{2},\cdots,\mathbf{z}^{L}\}$, with the last one $\mathbf{z}^{L}$ approximately following a standard Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I})$ for $L\gg 1$.

The other is the reverse process, which iteratively applies the backward transition kernel,

$$ $p^{\mathrm{back}}(\mathbf{z}^{i-1}\mid\mathbf{z}^{i})=\mathcal{N}(\mathbf{z}^{ i-1};\bm{\mu}^{\mathrm{back}}(\mathbf{z}^{i},i),\sigma^{i}\mathbf{I}),\ \ \ \bm{\mu}^{\mathrm{back}}(\mathbf{z}^{i},i)=\frac{1}{\sqrt{\alpha^{i}}}\Big{(} \mathbf{z}^{i}-\beta^{i}\frac{\bm{\epsilon}^{\mathrm{back}}(\mathbf{z}^{i},i)} {\sqrt{1-\bar{\alpha}^{i}}}\Big{)},$ (4) $$

where $\alpha^{i}=1-\beta^{i}$, $\bar{\alpha}^{i}=\prod_{k=1}^{i}\alpha^{k}$, $\bm{\epsilon}^{\mathrm{back}}(\cdot)$ is a neural network, $\mathbf{z}^{L}$ is an initial sample drawn from $\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, and $\sigma^{i}$ is some backward variance schedule. The outcome of this process is a reversed sequence of latent variables $\{\mathbf{z}^{L-1},\mathbf{z}^{L-2},\cdots,\mathbf{z}^{0}\}$, where the last variable $\mathbf{z}^{0}$ is expected to follow the density distribution of real samples: $q^{\mathrm{latent}}(\mathbf{z}^{0})$.

To optimize the diffusion model, common practices adopt a loss function as

$$ $\mathcal{L}^{\mathrm{DM}}=\mathbb{E}_{i,\mathbf{z}^{0},\bm{\epsilon}}[\|\bm{ \epsilon}-\bm{\epsilon}^{\mathrm{back}}(\sqrt{\bar{\alpha}^{i}}\mathbf{z}^{0}+ \sqrt{1-\bar{\alpha}^{i}}\bm{\epsilon},i)\|^{2}],$ (5) $$

where $\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$, $\mathbf{z}_{0}\sim q^{\mathrm{latent}}(\mathbf{z}^{0})$, and $i\sim\mathcal{U}\{1,L\}$.

### 3 Problem Analysis

In this section, we first formulate the problem of posterior collapse in the framework of time-series latent diffusion and show its significance. Then, we define proper measures that quantify the impact of posterior collapse on the models. Finally, we conduct empirical experiments to confirm that time-series diffusion indeed suffers from this problem.

#### 3.1 Formulation of Posterior Collapse and Its Impacts

Let us focus on time series $\mathbf{X}=[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{T}]$, where every observation $\mathbf{x}_{t},t\in[1,T]$ is a $D$-dimensional vector and $T$ denotes the number of observations. A potential risk of applying the latent diffusion to time series is the posterior collapse (Bowman et al., ), which occurs to some autoencoders (Bahdanau et al., ), especially VAE (Lucas et al., ). Its mathematical formulation in the framework of latent diffusion is as follows.

##### Problem formulation.

The posterior of VAE: $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})$, is collapsed if it reduces to the Gaussian prior $p^{\mathrm{prior}}(\mathbf{z})=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})$, irrespective of the time-series conditional $\mathbf{X}$:

$$ $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})=p^{\mathrm{prior}}(\mathbf{z}), \forall\mathbf{X}\in\mathbb{R}^{TD}.$ $$

In this case, the latent variable $\mathbf{z}$ contains no information about time series $\mathbf{X}$, otherwise the posterior distribution $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})$ would vary depending on different conditionals. Above is a strict definition. In practice, one is mostly faced with a situation where $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})\approx p^{\mathrm{prior}}(\mathbf{z})$ and it is still appropriate to say that the posterior collapses.

##### Implications of posterior collapse.

A typical symptom of this problem is that, since the latent variable $\mathbf{z}$ carries very limited information of time series $\mathbf{X}$, the trained decoder $\mathbf{f}^{\mathrm{dec}}$ tends to ignore this input variable $\mathbf{z}$, which is undesired for conditional generation $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$. Besides this empirical finding from previous works, we find that posterior collapse is also significant in its impact on the expressiveness of latent diffusion. Let us first see the below conclusion.

###### Proposition 3.1 (Gaussian Latent Variables) .

For standard latent diffusion, suppose its posterior $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})$ is collapsed, then the distribution $q^{\mathrm{latent}}(\mathbf{z})$ of latent variable $\mathbf{z}$ will shape as a standard Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I})$, which is trivial for the diffusion model to approximate.

###### Proof.

The proof is fully provided in Appendix A.
∎

In other words, latent variable $\mathbf{z}$ is just Gaussian in the case of posterior collapse. The diffusion model, which is known for approximating complex data distributions (Dhariwal & Nichol, ; Li et al., ), will in fact become a redundant module. Therefore, posterior collapse reduces latent diffusion to a simple VAE, which also samples latent variable $\mathbf{z}$ from a standard Gaussian $\mathcal{N}(\mathbf{0},\mathbf{I})$. We conclude that the problem makes latent diffusion less expressive.

[leftmargin=0pt, rightmargin=0pt, innerleftmargin=10pt, innerrightmargin=10pt, skipbelow=0pt]
*Takeaway*: The problem of posterior collapse not only lets the decode $\mathbf{f}^{\mathrm{dec}}$ tend to ignore the latent variable $\mathbf{z}$ for conditional generation $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$, but also reduces the framework of latent diffusion to VAE, making it less expressive.

###### Proposition 3.1 (Gaussian Latent Variables) .

###### Proof.

#### 3.2 Introduction of Dependency Measures

It is very intuitive from above that the problem of posterior collapse will make the latent variable $\mathbf{z}$ lose control of the decoder $\mathbf{f}^{\mathrm{dec}}$. To make our claim more solid and confirm that the problem happens to time-series diffusion, we introduce some proper measures that quantify the dependencies of decoder $\mathbf{f}^{\mathrm{dec}}$ on various inputs (e.g., variable $\mathbf{z}$).

##### Autoregressive decoder.

Consider that decoder $\mathbf{f}^{\mathrm{dec}}$ has an autoregressive structure, which conditions on latent variable $\mathbf{z}$ and prefix $\mathbf{X}_{1:t-1}=[\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{t-1}]$ to predict the next observation $\mathbf{x}_{t}$. With abuse of notation, we set $\mathbf{x}_{0}=\mathbf{z}$ and formulate the decoder as

$$ $\mathbf{h}_{t}=\mathbf{f}^{\mathrm{dec}}(\mathbf{X}_{0:t-1}),\ \ \ \mathbf{X}_ {0:t-1}=[\mathbf{x}_{0},\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{t-1}]$ (6) $$

where the representation $\mathbf{h}_{t},t\geq 1$ is linearly projected to multiple parameters (e.g., mean vector and covariance matrix) that determine the distribution $p^{\mathrm{gen}}(\mathbf{x}_{t}\mid\mathbf{z},\mathbf{X}_{1:t-1})$ of some family (e.g., Gaussian). Examples of such a decoder include recurrent neural networks (RNN) (Hochreiter & Schmidhuber, ) and Transformer (Vaswani et al., ). We put the formulation details of these example in Appendix B.

##### Dependency measure.

The symptom of posterior collapse is that the decoder $\mathbf{f}^{\mathrm{dec}}$ heavily relies on prefix $\mathbf{X}_{1:t-1}$ (especially the last observation $\mathbf{x}_{t-1}$) to compute the representation $\mathbf{h}_{t}$, ignoring the guidance of latent variable $\mathbf{x}_{0}=\mathbf{z}$. In other words, the variable $\mathbf{z}$ loses control of decoder $\mathbf{f}^{\mathrm{dec}}$ in that situation, which is undesired for conditional generation $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$.

Inspired by the technique of integrated gradients (Sundararajan et al., ), we present a new tool: dependency measure, which quantifies the impacts of latent variable $\mathbf{x}_{0}=\mathbf{z}$ and prefix $\mathbf{X}_{1:t-1}$ on decoder $\mathbf{f}^{\mathrm{dec}}$. Specifically, we first set a baseline input $\mathbf{O}_{0:t-1}$ as $[\mathbf{x}_{0}=\mathbf{0},\mathbf{x}_{1}=\mathbf{0},\cdots,\mathbf{x}_{t-1}=
\mathbf{0}]$ and denote the term $\mathbf{f}^{\mathrm{dec}}(\mathbf{O}_{0:t-1})$ as $\widetilde{\mathbf{h}}_{t}$. Then, we parameterize a straight line $\bm{\gamma}(s):[0,1]\rightarrow\mathbb{R}^{tD}$ between the actual input $\mathbf{X}_{1:t-1}$ and the input baseline $\mathbf{O}_{0:t-1}$ as

$$ $\bm{\gamma}(s)=s\mathbf{X}_{0:t-1}+(1-s)\mathbf{O}_{0:t-1}\coloneqq[s\mathbf{x }_{0},s\mathbf{x}_{1},\cdots,s\mathbf{x}_{t-1}].$ (7) $$

Applying the chain rule in differential calculus, we have

$$ $\frac{d\mathbf{f}^{\mathrm{dec}}(\bm{\gamma}(s))}{ds}=\sum_{j=0}^{t-1}\sum_{k= 1}^{k=D}\frac{d\mathbf{f}^{\mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)} \frac{d\gamma_{j,k}(s)}{ds}=\sum_{j=0}^{t-1}\sum_{k=1}^{k=D}x_{j,k}\frac{d \mathbf{f}^{\mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)},$ (8) $$

where $\gamma_{j,k}(s)$ denote the $k$-th dimension $s\cdot x_{j,k}$ of the $j$-th vector $s\mathbf{x}_{j}$ in point $\bm{\gamma}(s)$. With the above elements, we can define the below measures.

###### Definition 3.2 (Dependency Measures) .

For an autoregressive decoder $\mathbf{f}^{\mathrm{dec}}$ that conditions on both latent variable $\mathbf{x}_{0}=\mathbf{z}$ and the prefix $\mathbf{X}_{1:t-1}$ to compute representation $\mathbf{h}_{t}$, the dependency measure of every input variable $\mathbf{x}_{j},j\in[0,t-1]$ to the decoder is defined as

$$ $m_{t,j}=\frac{1}{\|\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}\|^{2}}\Big{<} \mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t},\sum_{k=1}^{D}\Big{(}x_{j,k}\int_{0} ^{1}\frac{d\mathbf{f}^{\mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)}ds\Big {)}\Big{>},$ (9) $$

where operation $<\cdot,\cdot>$ represents the inner product. In particular, we name $m_{t,0}$ as the global dependency and $m_{t,t-j},1\leq j<t$ as the j-th order local dependency.

We provide the derivation for dependency measure $m_{t,j}$ and detail its relations to integrated gradients in Appendix C. In practice, the integral term can be approximated as

$$ $\int_{0}^{1}\frac{d\mathbf{f}^{\mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s )}ds=\mathbb{E}_{s\in\mathcal{U}\{0,1\}}\Big{[}\frac{d\mathbf{f}^{\mathrm{dec} }(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)}\Big{]}\approx\frac{1}{|\mathcal{S}|}\sum _{s\in\mathcal{S}}\frac{d\mathbf{f}^{\mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{ j,k}(s)},$ (10) $$

where $\mathcal{S}$ is the set of independent samples drawn from uniform distribution $\mathcal{U}\{0,1\}$. According to the law of large numbers (Sedor, ), this approximation is unbiased and gets more accurate for a bigger sample set $|\mathcal{S}|$. Notably, the defined measures have the following properties.

###### Proposition 3.3 (Signed and Normalization Properties) .

The dependency measure $m_{t,j},\forall j\in[0,t-1]$ is a signed measure and always satisfies $\sum_{j=0}^{t-1}m_{t,j}=1$.

###### Proof.

The proof is fully provided in Appendix D.
∎

We can see that the measure $m_{t,j}$ can be either positive or negative, with a normalized sum over the subscript $j$ as $1$. If $m_{t,j}\geq 0$, then we say that vector $\mathbf{x}_{j}$ has a positive impact on the decoder $\mathbf{f}^{\mathrm{dec}}$ for computing representation $\mathbf{h}_{j}$: the bigger is $m_{t,j}$, the larger is such an impact; Similarly, if $m_{t,j}<0$, then the vector $\mathbf{x}_{j}$ has a negative impact on the decoder: the smaller is $m_{t,j}$, the greater is the negative influence. Besides, it is also not hard to understand that there exists a negative impact. For example, the latent variable $\mathbf{z}\sim q^{\mathrm{latent}}(\mathbf{z})$ might be an outlier for the decoder $\mathbf{f}^{\mathrm{dec}}$, which locates at a low-density region in the prior distribution $q^{\mathrm{prior}}(\mathbf{z})$.

###### Definition 3.2 (Dependency Measures) .

###### Proposition 3.3 (Signed and Normalization Properties) .

###### Proof.

##### Example Application.

Fig. 1 shows several examples of applying the dependency measures, where each subfigure contains a sample of time series (i.e., blue curve) generated by some model and two types of dependency measures (i.e., red and green bar charts) estimated by Eq. (9). Specifically, every point $\mathbf{x}_{t}$ in the time series corresponds to a green bar that indicates the global dependency $m_{t,0}$ and a red bar that represents the first-order local dependency $m_{t,t-1}$. In the upper left subfigure, we can see that the positive impact of latent variable $\mathbf{z}$ on the decoder (e.g., $m_{t,0}$) decreases over time and vanishes eventually. From the lower right subfigure, we can even see that some bars (i.e., local dependency $m_{t,t-1}$) are negative, indicating that the variable $\mathbf{x}_{t-1}$ has a negative impact on predicting the next observation $\mathbf{x}_{t}$.

[leftmargin=0pt, rightmargin=0pt, innerleftmargin=10pt, innerrightmargin=10pt, skipbelow=0pt]
*Takeaway*: Dependency measure $m_{t,j},0\leq j<t$ quantifies the impact of latent variable $\mathbf{x}_{0}=\mathbf{z}$ or observation $\mathbf{x}_{j},j\geq 1$ on the decoder $\mathbf{f}^{\mathrm{dec}}$. This type of impact can be either positive or negative, which is reflected in the value of measure $m_{t,j}$.

#### 3.3 Empirical Dependency Estimations

Figure: Figure 2: Dependency measures $m_{t,0},m_{t,t-1}$ averaged over $500$ multivariate time series, with $3$ standard deviations as the error bars. We can see that the latent variable $\mathbf{z}$ of latent diffusion has a vanishing impact on the decoder $\mathbf{f}^{\mathrm{dec}}$, a typical symptom of posterior collapse. We also observe a phenomenon of dependency illusion in the case of shuffled time series.
Refer to caption: extracted/5896890/images/_study_1.jpeg

We are mainly interested in two types of defined measures: One is the global dependency $m_{t,0}$, which estimates the impact of latent variable $\mathbf{x}_{0}=\mathbf{z}$ on the decoder $\mathbf{f}^{\mathrm{dec}}$; The other is the first-order local dependency $m_{t,t-1}$, which estimates the dependency of decoder $\mathbf{f}^{\mathrm{dec}}$ on the last observation $\mathbf{x}_{t,t-1}$ for computing representation $\mathbf{h}_{t}$. In this part, we empirically estimate these measures, with the aims to confirm that posterior collapse occurs and show its impacts.

##### Experiment setup.

Two time-series datasets: WARDS (Alaa et al., ) and MIMIC (Johnson et al., ) are adopted. For each dataset, we extract the observations of the first $12$ hours, with the top $1$ and $5$ features that have the highest variances to form univariate and multivariate time series. To study the case where time series have no structural dependencies, we also try randomly shuffling the time steps of ordered time series. With the prepared datasets, we respectively train latent diffusion models on them and sample time series from the models.

##### Insightful results.

The upper two subfigures of Fig. 1 illustrate the estimated dependencies $m_{t,0},m_{t,t-1}$ for a single time-series sample $\mathbf{X}$, while Fig. 2 shows the dependency measures averaged over $500$ samples. We can see that, for both ordered and shuffled time series, the global dependency $m_{t,0}$ exponentially converges to 0 0 with increasing time step $t$, indicating that latent variable $\mathbf{z}$ loses control of the generation process of decoder $\mathbf{f}^{\mathrm{dec}}$ and the posterior is collapsed. More interestingly, as shown in the right part of Fig. 1, while there is no dependency between adjacent observations $\mathbf{x}_{t-1},\mathbf{x}_{t}$ in shuffled time series, we still observe that the first-order measure $m_{t,t-1}$ is significantly different from 0 0 (e.g., around $0.1$ to $0.2$). This phenomenon might arise as neural networks overfit and we name it as dependency illusion.

[leftmargin=0pt, rightmargin=0pt, innerleftmargin=10pt, innerrightmargin=10pt, skipbelow=0pt]
*Takeaway*: Time-series latent diffusion exhibits a typical symptom of posterior collapse: latent variable $\mathbf{z}$ has an almost exponentially decreasing impact on generation process $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$. More seriously, we observe a phenomenon of dependency illusion.

### 4 Problem Rethinking and New Framework

In this section, we first analyze how the framework design of latent diffusion makes it tend to suffer from posterior collapse. Then, based on our analysis, we propose a new framework, which extends latent diffusion but addresses the problem.

#### 4.1 Risky Design of Latent Diffusion

Previous works (Semeniuta et al., ; Alemi et al., ) have identified two main causes of the problem: KL-divergence term and strong decoder. For time-series latent diffusion, we will explain as below that those causes indeed exist, but are in fact avoidable.

##### Unnecessary regularization.

The KL-divergence term $\mathrm{D}_{\mathrm{KL}}(q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})\mid\mid p^{
\mathrm{prior}}(\mathbf{z}))$ in Eq. (2) moves the posterior $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})$ towards prior $p^{\mathrm{prior}}(\mathbf{z})$, which has the side effect of posterior collapse by definition. In essence, this term is tailored for VAE, such that it is valid to sample latent variable $\mathbf{z}$ from the Gaussian prior $p^{\mathrm{prior}}(\mathbf{z})$ for inference. However, for latent diffusion, the variable $\mathbf{z}$ is sampled from the diffusion model, which can approximate a non-Gaussian prior distribution. Hence, the interaction between the VAE and diffusion model is not properly designed, which incurs a limited prior $p^{\mathrm{prior}}(\mathbf{z})$ and a risky KL-divergence term $\mathrm{D}_{\mathrm{KL}}(\cdot)$.

##### Unprepared for recurrence.

The strong decoder is also a cause of posterior collapse, which happens to sequence autoencoders (Bowman et al., ; Eikema & Aziz, ). Time series $\mathbf{X}\in\mathbb{R}^{TD}$ have a clear temporal structure, so the corresponding decoder $\mathbf{f}^{\mathrm{dec}}$ is typically a RNN, which explicitly models the dependency between different observations $\mathbf{x}_{i},\mathbf{x}_{j},i\neq j$. For predicting the observation $\mathbf{x}_{j}$, both latent variable $\mathbf{z}$ and previous observation $\mathbf{x}_{i},i<j$ are the inputs to the decoder $\mathbf{f}^{\mathrm{dec}}$, so variable $\mathbf{z}$ is possible to be ignored.

Latent diffusion is primarily designed for image generation, with U-net (Ronneberger et al., ) as the backbone, which consists of many layers of feedforward neural networks (FNN) (Svozil et al., ). For example, convolution neural networks (CNN) (Krizhevsky et al., ), self-attention (Vaswani et al., ), and MLP (Lu et al., ). These FNN layers are highly sensitive to the input variables, so the original design of latent diffusion lacks a mechanism to address the possible insensitivity, which is the case of time-series decoder.

[leftmargin=0pt, rightmargin=0pt, innerleftmargin=10pt, innerrightmargin=10pt, skipbelow=0pt]
*Takeaway*: The improper design of latent diffusion is the root cause of posterior collapse, which results in the avoidable KL-divergence regularization, limits the form of prior distribution $p^{\mathrm{prior}}(\mathbf{z})$, and lacks a mechanism to handle the insensitive decoder.

#### 4.2 New Framework

Figure: Figure 3: In this example, path $\mathbf{X}\rightarrow\mathbf{z}^{1}$ is the variational inference (which gets rid of KL-divergence regularization) and path $\mathbf{X}\rightarrow\mathbf{z}^{3}$ shows the collapse simulation (which is to increase the sensitivity of decoder $\mathbf{f}^{\mathrm{dec}}$ to latent variable $\mathbf{z}$). Compared with time-series latent diffusion, our framework is free from posterior collapse and has a unlimited prior $p^{\mathrm{prior}}(\mathbf{z})$.
Refer to caption: x1.png

In light of previous analyses, we propose a new framework that lets the autoencoder interact with the diffusion model more effectively than latent diffusion. With this better framework design, we can eliminate the KL-divergence term, permit a free from of prior distribution $p^{\mathrm{prior}}(\mathbf{z})$, and increase the sensitivity of decoder $\mathbf{f}^{\mathrm{dec}}$ to latent variable $\mathbf{z}$.

Importantly, we notice a conclusion (Ho et al., ) for the diffusion process (i.e., Eq. (3)):

$$ $q^{\mathrm{forw}}(\mathbf{z}^{i}\mid\mathbf{z}^{0})=\mathcal{N}(\mathbf{z}^{i} ;\sqrt{\bar{\alpha}^{i}}\mathbf{z}^{0},(1-\bar{\alpha}^{i})\mathbf{I}),$ (11) $$

where the coefficient $\bar{\alpha}^{i}$ monotonically decreases from $1$ to approximately 0 0 for $i\in[0,L]$. In this sense, suppose the initial variable $\mathbf{z}^{0}$ is set as $\mathbf{v}=\mathbf{f}^{\mathrm{enc}}(\mathbf{X})$, then we can infer that the random variable $\mathbf{z}^{i}\sim q^{\mathrm{forw}}(\mathbf{z}^{i}\mid\mathbf{z}^{0})$ contains $\bar{\alpha}^{i}\times 100\%$ information about the vector $\mathbf{v}$, with $(1-\bar{\alpha}^{i})\times 100\%$ pure noise. For $i\rightarrow 0$, the diffusion process is similar to the variational inference (i.e., Eq. (1)) of VAE, adding slight Gaussian noise to the encoder output $\mathbf{v}$. For $i\rightarrow T$, the variable $\mathbf{z}^{i}$ simulates the problem of posterior collapse since $q^{\mathrm{forw}}(\mathbf{z}^{i}\mid\mathbf{z}^{0})\approx\mathcal{N}(\mathbf{
z}^{i};\mathbf{0},\mathbf{I})$.

##### Diffusion process as variational inference.

Considering the above facts, we first treat the starting few iterations of the diffusion process as the variational inference. Specifically, with a fixed small integer $N\ll L$, we sample a number $i$ from uniform distribution $\mathcal{U}\{0,N\}$ and let the diffusion process convert the encoder output $\mathbf{v}=\mathbf{f}^{\mathrm{enc}}(\mathbf{X})$ into the latent variable:

$$ $\mathbf{z}=\mathbf{z}^{i}\sim q^{\mathrm{forw}}(\mathbf{z}^{i}\mid\mathbf{z}^{ 0}),\mathbf{z}^{0}=\mathbf{v}.$ (12) $$

In terms of the formerly defined generation distribution $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$ (parameterized by the decoder $\mathbf{f}^{\mathrm{dec}}$), a negative log-likelihood loss $\mathcal{L}^{\mathrm{VI}}$ is incurred as

$$ $\mathcal{L}^{\mathrm{VI}}=\mathbb{E}_{i\sim\mathcal{U}\{0,N\},\mathbf{z}^{0}}[ -\bar{\alpha}^{\gamma i}\ln p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z}=\mathbf{ z}^{i})],$ (13) $$

where $\gamma\in\mathbb{N}^{+},\gamma N\leq L$ is a hyper-parameter, with the aim to reduce the impact of a very noisy latent variable $\mathbf{z}$. As multiplier $\gamma$ increases, the weight $\bar{\alpha}^{\gamma i}$ decreases.

Similar to VAE, the variational inference in our framework also leads the latent variable $\mathbf{z}$ to be smooth (Bowman et al., ) in its effect on decoder $\mathbf{f}^{\mathrm{dec}}$. However, our framework is free from the KL-divergence term $\mathrm{D}_{\mathrm{KL}}(q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})\mid\mid p^{
\mathrm{prior}}(\mathbf{z}))$ of VAE (i.e., one cause of the posterior collapse), since we can facilitate $\mathbf{z}\sim q^{\mathrm{latent}}(\mathbf{z})$ at test time through applying the reverse process of the diffusion model (i.e., Eq. (4)) to sample variable $\mathbf{z}^{i},i\in[0,N]$.

Figure: Algorithm 1 Training

##### Diffusion process for collapse simulation.

Then, we apply the last few iterations of the diffusion process to simulate posterior collapse, with the purposes of increasing the impact of latent variable $\mathbf{z}$ on conditional generation $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$ and reducing dependency illusion.

Following our previous variational inference, we set $\mathbf{z}^{0}=\mathbf{f}^{\mathrm{enc}}(\mathbf{X})$ and apply the diffusion process to cast the initial variable $\mathbf{z}^{0}$ into a highly noisy variable $\mathbf{z}^{i},i\rightarrow L$. Considering that the variable $\mathbf{z}^{i}$ contains little information about the encoder output $\mathbf{f}^{\mathrm{enc}}(\mathbf{X})$, it is unlikely that the decoder $\mathbf{f}^{\mathrm{dec}}$ can recover time series $\mathbf{X}$ from variable $\mathbf{z}^{i}$, otherwise there is posterior collapse or dependency illusion. In this sense, we have the following regularization:

$$ $\mathcal{L}^{\mathrm{CS}}=\mathbb{E}_{i\sim\mathcal{U}\{M,L\},\mathbf{z}^{i}}[ (1-\bar{\alpha}^{\lceil\frac{i}{\eta}\rceil})\ln p^{\mathrm{gen}}(\mathbf{X} \mid\mathbf{z}=\mathbf{z}^{i})],$ (14) $$

which penalizes the model for having a high conditional density $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$ for non-informative latent variable $\mathbf{z}=\mathbf{z}^{i},i\in[M,L]$. Here $M\in\mathbb{N}^{+}$ is close to $L$, $\lceil\cdot\rceil$ is the ceiling function, and $\eta\geq 1$ is set to reduce the impact of informative variable $\mathbf{z}^{i}$.

For a strong decoder $\mathbf{f}^{\mathrm{dec}}$, such as long short-term memory (LSTM), the regularization $\mathcal{L}^{\mathrm{CS}}$ will impose a heavy penalty if the decoder solely relies on previous observations $\{\mathbf{x}_{k}\mid k<j\}$ to predict an observation $\mathbf{x}_{j}$. In that situation, a high prediction probability will be assigned to the observation $\mathbf{x}_{j}$ even if the latent variable $\mathbf{z}$ contains very limited information about the raw data $\mathbf{X}$.

##### Training, inference, and running times.

Our framework is very different from the latent diffusion in both training and inference. We depict the workflows of our framework in Fig. 3, with the training and sampling procedures respectively placed in Algorithm 1 and Algorithm 2. For training, the key points are to compute three loss functions: $\widehat{\mathcal{L}}^{\mathrm{VI}}$ for likelihood maximization, $\widehat{\mathcal{L}}^{\mathrm{DM}}$ for training diffusion models, and $\widehat{\mathcal{L}}^{\mathrm{CS}}$ for collapse regularization. For inference, the main difference is that the stopping time of the backward process is not 0 0, but a random variable. From these pseudo codes, we can see that our framework is almost as efficient as latent diffusion. We provide an in-depth analysis and empirical experiments about the running times of our framework in Appendix F.2.

### 5 Related Work

Besides latent diffusion, our paper is also related to previous works that aim to mitigate the problem of posterior collapse for VAE (Kingma & Welling, ). We collect three main types of such methods and apply them to improve the VAE of latent diffusion, with the corresponding experiment results shown in Table 1 of Sec. 6 and Table 4 of Appendix F.3. In the following, we briefly introduce those baselines and explain their limitations.

##### KL annealing.

This method (Bowman et al., ) assigns an adaptive weight to control the effect of KL-divergence term, so that VAE is unlikely to fall into the local optimum of posterior collapse at the initial optimization stage. While this method indeed mitigates the problem, it still cannot fully eliminate the negative impact of the risky KL-divergence regularization.

##### Decoder weakening.

A representative method in this class is Variable Masking (Semeniuta et al., ), which randomly masks input observations to the autoregressive decoder, such that the decoder is forced to rely more on the latent variable for predicting the next observation. However, this method will make the model less expressive since the decoder is weakened.

##### Skip connections.

With the aim to improve the impact of latent variables on the recurrent decoder, this approach (Dieng et al., ) directly feeds the latent variable into the decoder at every step, not only at the first step. However, the latent variable in that case acts as a constant input signal at every time step, so the decoder will still tend to ignore this redundant information.

Compared with the above baselines, our framework can address the problem of posterior collapse and is free from their side effects (e.g., less expressive decoder). The experiment results in Sec. 6 and Appendix F confirm that our framework indeed performs better in practice.

### 6 Experiments

Figure: Figure 4: The results of averaged dependency measures and error bars for our framework, which should be compared with those (e.g., Fig. 2) of latent diffusion, showing that our framework has a stable posterior and is without dependency illusion.
Refer to caption: extracted/5896890/images/_study_2.jpeg

We have conducted extensive experiments to verify that our framework is free from posterior collapse and outperforms latent diffusion (with or without previous baselines) in terms of time series generation. Due to the limited space, some other important empirical studies are put in the appendix, including more time-series datasets and another evaluation metric in Appendix F.3, diverse data modalities (e.g., text) in Appendix F.4, ablation studies in Appendix F.1, and the study of running times in Appendix F.2. The experiment setup is also placed in Appendix E.

#### 6.1 Stable Posterior of Our Framework

To show that our framework has a non-collapsed posterior $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})$, we follow the same experiment setup (e.g., datasets) as Sec. 3.3 and average the dependency measures $m_{t,0},m_{t,t-1}$ over $500$ sampled time series. The results are illustrated in Fig. 4. For ordered time series in the left two subfigures, we can see that, while the global dependency $m_{t,0}$ still decreases with increasing time step $t$, it converges into a value around $0.5$, which is also a bit higher than the converged first-order local dependency $m_{t,t-1}$. These results indicate that latent variable $\mathbf{z}$ in our framework maintains its control of decoder $\mathbf{f}^{\mathrm{dec}}$ during the whole conditional generation process $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$.

For shuffled time series in the right two subfigures, we can see that the global dependency $m_{t,0}$ is always around or above $1$, and the local dependency $m_{t,t-1}$ is negative most of the time. These results indicate that the decoder $\mathbf{f}^{\mathrm{dec}}$ only relies on latent variable $\mathbf{z}$ and the context $\mathbf{x}_{t-1}$ even has a negative impact on conditional generation $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$, suggesting our framework is without dependency illusion. Based on all our findings, we conclude that: compared with latent diffusion (Fig. 2), our framework is free from the effects of posterior collapse (e.g., strong decoder).

#### 6.2 Performances in Time Series Generation

**Table 1: Wasserstein distances of different models on three widely used time-series datasets. The lower the distance metric, the better the generation quality. More results from other time-series datasets, with another evaluation metric, are placed in Table 4 of Appendix F.3.**
| Model | Backbone | MIMIC | WARDS | Earthquakes |
| --- | --- | --- | --- | --- |
| Latent Diffusion | LSTM | 5.19 | 7.52 | 5.87 |
| \hdashlineLatent Diffusion w/ KL Annealing | LSTM | 4.28 | 5.74 | 3.88 |
| Latent Diffusion w/ Variable Masking | LSTM | 4.73 | 6.01 | 4.26 |
| Latent Diffusion w/ Skip Connections | LSTM | 3.91 | 4.95 | 3.74 |
| \hdashlineOur Framework | LSTM | $\mathbf{2.29}$ | $\mathbf{3.16}$ | $\mathbf{2.67}$ |
| Latent Diffusion | Transformer | 5.02 | 7.46 | 5.91 |
| \hdashlineLatent Diffusion w/ KL Annealing | Transformer | 4.31 | 5.54 | 3.51 |
| Latent Diffusion w/ Variable Masking | Transformer | 4.42 | 5.97 | 4.45 |
| Latent Diffusion w/ Skip Connections | Transformer | 3.75 | 4.67 | 3.69 |
| \hdashlineOur Framework | Transformer | $\mathbf{2.13}$ | $\mathbf{3.01}$ | $\mathbf{2.49}$ |

In this part, we aim to verify that our framework outperforms latent diffusion in terms of time series generation, which is intuitive since our framework is free from posterior collapse. We also include some other methods that are proposed by previous works to mitigate the problem, including KL annealing (Fu et al., ), variable masking (Bowman et al., ), and skip connections (Dieng et al., ). We adopt the Wasserstein distances (Bischoff et al., ) as the metric.

The experiment results on three commonly used time-series datasets are shown in Table 1. From the results, we can see that, regardless of the used dataset and the backbone of autoencoder, our framework significantly outperforms latent diffusion and the baselines, which strongly confirms our intuition. For example, with the backbone of Transformer, our framework achieves 2.53 points lower than latent diffusion w/ KL Annealing on the WARDS dataset.

### 7 Conclusion

In this paper, we provide a solid analysis of the negative impacts of posterior collapse on time-series latent diffusion and introduce a new framework that is free from this problem. For our analysis, we begin with a theoretical insight, showing that the problem will reduce latent diffusion to VAE, rendering it less expressive. Then, we introduce a useful tool: dependency measure, which quantifies the impacts of various inputs on an autoregressive decoder. Through empirical dependency estimation, we show that the latent variable has a vanishing impact on the decoder and find that latent diffusion exhibits a phenomenon of dependency illusion. Compared with standard latent diffusion, our framework gets rid of the risky KL-divergence regularization, permits an unlimited prior distribution, and lets the decoder be sensitive to the latent variable. Extensive experiments on multiple real-world time-series datasets show that our framework has no symptoms of posterior collapse and notably outperforms the baselines in terms of time series generation.

## Appendix

### Appendix A The Impact of Posterior Collapse

Under the assumption of posterior collapse, the below equality:

$$ $q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})=p^{\mathrm{prior}}(\mathbf{z})= \mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I}),$ (15) $$

holds for any latent variable $\mathbf{z}\in\mathbb{R}^{D}$ and any conditional $\mathbf{X}\in\mathbb{R}^{TD}$. Then, note that

$$ $\displaystyle q^{\mathrm{latent}}(\mathbf{z})$ $\displaystyle=\int q^{\mathrm{VI}}(\mathbf{z}\mid\mathbf{X})q^{\mathrm{raw}}( \mathbf{X})d\mathbf{X}=\int\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})q^{ \mathrm{raw}}(\mathbf{X})d\mathbf{X}$ (16) $\displaystyle=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I})\int q^{\mathrm{raw }}(\mathbf{X})d\mathbf{X}=\mathcal{N}(\mathbf{z};\mathbf{0},\mathbf{I}),$ $$

which is exactly our claim.

### Appendix B Recurrent Encoders

We mainly implement the backbone of decoder $\mathbf{f}^{\mathrm{dec}}$ as LSTM (Hochreiter & Schmidhuber, ) or Transformer (Vaswani et al., ). In the former case, we apply the latent variable $\mathbf{z}$ to initialize LSTM and condition it on prefix $\mathbf{X}_{1:t-1}$ to compute the representation $\mathbf{h}_{t}$. Formally, the LSTM-based decoder $\mathbf{f}^{\mathrm{dec}}$ is as

$$ $\left\{\begin{aligned} &\mathbf{s}_{t}=\mathrm{LSTM}(\mathbf{s}_{t-1},\mathbf{ x}_{t-1}),\forall t\geq 1\\ &\mathbf{h}_{t}=\mathbf{W}_{f}^{2}\tanh(\mathbf{W}_{f}^{1}\mathbf{s}_{t})\end{ aligned}\right.,$ (17) $$

where $\mathbf{s}_{t}$ is the state vector of LSTM and $\mathbf{W}_{f}^{2},\mathbf{W}_{f}^{1}$ are learnable matrices. In particular, for the corner case $t=1$, we fix $\mathbf{s}_{0},\mathbf{x}_{0}$ as zero vectors.

In the later case, we just treat latent variable $\mathbf{z}$ as $\mathbf{x}_{0}$. Therefore, we have

$$ $\left\{\begin{aligned} &[\mathbf{s}_{t-1},\mathbf{s}_{s-2},\cdots,\mathbf{s}_{ 0}]=\mathrm{Transformer}(\mathbf{x}_{t-1},\mathbf{x}_{t-2},\cdots,\mathbf{x}_{ 0})\\ &\mathbf{h}_{t}=\mathbf{W}_{f}^{2}\tanh(\mathbf{W}_{f}^{1}\mathbf{s}_{t-1}) \end{aligned}\right.,$ (18) $$

where the subscript alignment results from self-attention mechanism.

### Appendix C Derivation of Dependency Measures

Integrated gradient (Sundararajan et al., ) is a very effective method of feature attributions. Our proposed dependency measures can be regarded as its extension to the case of sequence data and vector-valued neural networks. In the following, we provide the derivation of dependency measures.

For the computation $\mathbf{h}_{t}=\mathbf{f}^{\mathrm{dec}}(\mathbf{X}_{0:t-1})$, suppose the output of decoder $\mathbf{f}^{\mathrm{dec}}$ at origin $\mathbf{O}_{0:t-1}$ is $\widetilde{\mathbf{h}}_{t}$, then we apply the fundamental theorem of calculus as

$$ $\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}=\int_{0}^{1}\frac{d\mathbf{f}^{ \mathrm{dec}}(\bm{\gamma}(s))}{ds}ds,$ (19) $$

where $\bm{\gamma}(s)$ is a straight line connecting the origin $\mathbf{O}_{0:t-1}$ and the input $\mathbf{X}_{0:t-1}$ as $\bm{\gamma}(s)=s\mathbf{X}_{0:t-1}+(1-s)\mathbf{O}_{0:t-1}$.
Based on the chain rule, the above equality can be expanded as

$$ $\displaystyle\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}$ $\displaystyle=\int_{0}^{1}\sum_{j=0}^{t-1}\sum_{k=1}^{k=D}\frac{d\mathbf{f}^{ \mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)}\frac{d\gamma_{j,k}(s)}{ds}ds$ (20) $\displaystyle=\sum_{j=0}^{t-1}\Big{(}\int_{0}^{1}\sum_{k=1}^{k=D}x_{j,k}\frac{ d\mathbf{f}^{\mathrm{dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)}ds\Big{)},$ $$

where $\gamma_{j,k}(s)$ denote the $k$-th dimension $s\cdot x_{j,k}$ of the $j$-th vector $s\mathbf{x}_{j}$ in point $\bm{\gamma}(s)$. Intuitively, every term inside the outer sum operation $\sum_{j=0}^{t-1}$ represents the additive contribution of variable $\mathbf{x}_{j}$ (to the output difference $\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}$) along the integral line $\bm{\gamma}(s)$.

To simplify the notation, we denote the mentioned term as

$$ $\mathbf{m}_{t,j}=\int_{0}^{1}\sum_{k=1}^{k=D}x_{j,k}\frac{d\mathbf{f}^{\mathrm {dec}}(\gamma_{j,k}(s))}{d\gamma_{j,k}(s)}ds.$ (21) $$

Since $\mathbf{m}_{t,j}$ is a vector, we map the new term to a scalar and re-scale it as

$$ $m_{t,j}=\frac{<\mathbf{m}_{t,j},\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}>}{< \mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t},\mathbf{h}_{t}-\widetilde{\mathbf{h} }_{t}>},$ (22) $$

which is exactly our definition of the dependency measure.

### Appendix D Properties of of Dependency Measures

Firstly, in terms of Eq. (22), it is obvious that the dependency measure $m_{t,j}$ is signed: the measure can be either positive or negative. Then, based on Eq. (20), we have

$$ $\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}=\sum_{j=0}^{t-1}\mathbf{m}_{t,j}.$ (23) $$

By taking an inner product with the vector $\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t}$ at both sides, we get

$$ $<\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t},\mathbf{h}_{t}-\widetilde{\mathbf{h }}_{t}>=<\sum_{j=0}^{t-1}\mathbf{m}_{t,j},\mathbf{h}_{t}-\widetilde{\mathbf{h} }_{t}>=\sum_{j=0}^{t-1}<\mathbf{m}_{t,j},\mathbf{h}_{t}-\widetilde{\mathbf{h}} _{t}>.$ (24) $$

By rearranging the term, we finally arrive at

$$ $1=\sum_{j=0}^{t-1}\frac{<\mathbf{m}_{t,j},\mathbf{h}_{t}-\widetilde{\mathbf{h} }_{t}>}{<\mathbf{h}_{t}-\widetilde{\mathbf{h}}_{t},\mathbf{h}_{t}-\widetilde{ \mathbf{h}}_{t}>}=\sum_{j=0}^{t-1}m_{t,j},$ (25) $$

which is exactly our claim.

### Appendix E Experiment Details

**Table 2: Ablation studies of the hyper-parameters $N,M$, which are respectively used in the estimations of likelihood loss $\mathcal{L}^{\mathrm{VI}}$ and collapse penalty $\mathcal{L}^{\mathrm{CS}}$. Here LD is short for latent diffusion and the symbol $-$ means “Not Applicable”.**
| Model | Backbone | $N$ for $\mathcal{L}^{\mathrm{VI}}$ | $M$ for $\mathcal{L}^{\mathrm{CS}}$ | Diffusion Iterations $L$ | MIMIC | WARDS |
| --- | --- | --- | --- | --- | --- | --- |
| Latent Diffusion | Transformer | $-$ | $-$ | 1000 | 5.02 | 7.46 |
| LD w/ Skip Connections | Transformer | $-$ | $-$ | 1000 | 3.75 | 4.67 |
| Our Framework | Transformer | $50$ | $100$ | $1000$ | $\mathbf{2.13}$ | $\mathbf{3.01}$ |
| Our Framework | Transformer | $50$ | $50$ | $1000$ | $2.59$ | $3.32$ |
| Our Framework | Transformer | $50$ | $150$ | $1000$ | $2.71$ | $3.46$ |
| Our Framework | Transformer | $50$ | $200$ | $1000$ | $2.83$ | $3.75$ |
| \hdashlineOur Framework | Transformer | $10$ | $100$ | $1000$ | $2.31$ | $3.16$ |
| Our Framework | Transformer | $100$ | $100$ | $1000$ | $2.38$ | $3.24$ |
| Our Framework | Transformer | $150$ | $100$ | $1000$ | $2.75$ | $3.41$ |

We have adopted three widely used time-series datasets for both analysis and model evaluation, including MIMIC (Johnson et al., ), WARDS (Alaa et al., ), and Earthquakes (U.S. Geological Survey, ). The setup of the first two datasets are introduced in Sec. 3.3. For MIMIC, we specially simplify it into a version of univariate time series for the illustration purpose, which is only used in the experiments shown in Fig. 1. All other experiments are about multivariate time series. For the Earthquakes dataset, it is about the location and time of all earthquakes in Japan from 1990 to 2020 with magnitude of at least 2.5 from U.S. Geological Survey (). We follow the same preprocessing procedure for this dataset as Li ().

We use almost the same model configurations for all experiments. The diffusion models are parameterized by a standard U-net (Ronneberger et al., ), with $L=1000$ diffusion iterations and hidden dimensions $\{128,64,32\}$. The hidden dimensions of autoencoders and latent variables are fixed as $128$. The conditional distribution $p^{\mathrm{gen}}(\mathbf{X}\mid\mathbf{z})$ is parameterized as a Gaussian, with learnable mean vector and diagonal covariance matrix functions. For our framework, $N,M$ are respectively selected as $50,100$, with $\gamma=2$ and $\eta=1$. We also apply dropout with a ratio of $0.1$ to most layers of neural networks. We adopt Adam algorithm (Kingma & Ba, ) with the default hyper-parameter setting to optimize our model. For Table 1 and Table 2, every number is averaged over $10$ different random seeds, with a standard deviation less than $0.05$. For the computing resources, all our models can be trained on 1 NVIDIA Tesla V100 GPU within 10 hours.

### Appendix F Additional Experiments

Due to the limited space of our main text, we put the results of some minor experiments here in the appendix. Notably, we will adopt more datasets and another evaluation metric.

#### F.1 Ablation Studies

We have conducted ablation studies to verify that our hyper-parameter selections $N=50,M=100$ are optimal. The experiment results are shown in Table 2. For both $N$ and $M$, either increasing or decreasing their values results in worse performance on the two datasets.

#### F.2 Study on Running Times

Our framework only incurs a minor increase in training time and enjoys the same inference speed as the latent diffusion. For training, while our framework will run the decoder a second time for collapse simulation $\mathcal{L}^{\mathrm{CS}}$, it can be made in parallel with the first run of decoder $f^{\mathrm{dec}}$ for likelihood computation $\mathcal{L}^{\mathrm{VI}}$. Therefore, the training is still efficient on GPU devices. Our framework also has a different way of variational inference to infer latent variable $\mathbf{z}$ from data $\mathbf{X}$. However, it admits a closed-form solution and is thus as efficient as the reparameterization trick of latent diffusion. For inference, our framework has no difference from the latent diffusion: sampling the latent variable $\mathbf{z}$ with the reverse diffusion process and running the decoder $f^{\mathrm{dec}}$ in one shot.

**Table 3: Comparison of Training and Inference Times on the MIMIC dataset.**
| Method | Training Time | Inference Time |
| --- | --- | --- |
| Latent Diffusion | 2hr 10min | 5min 12s |
| Our Framework | 2hr 50min | 5min 17s |

To show the running times in practice, we perform an experiment on the MIMIC dataset as shown in Table 3. We can see that our framework indeed only has a minor increase for training. Given that our framework is free from posterior collapse and delivers better generation performances, this slight time investment is well worth it.

#### F.3 More Datasets and Another Evaluation Metric

**Table 4: Comparison on two new time-series datasets, with another metric: MMD.**
| Method | Backbone | Retail | Energy |
| --- | --- | --- | --- |
| Latent Diffusion | Transformer | $0.037$ | $0.052$ |
| Latent Diffusion w/ Skip Connections | Transformer | $0.033$ | $0.043$ |
| Our Framework | Transformer | $\mathbf{0.025}$ | $\mathbf{0.031}$ |
| Latent Diffusion | LSTM | $0.041$ | $0.057$ |
| Latent Diffusion w/ Skip Connections | LSTM | $0.035$ | $0.047$ |
| Our Framework | LSTM | $\mathbf{0.027}$ | $\mathbf{0.033}$ |

We conduct additional experiments on 2 more public UCI time-series datasets (Bay et al., ): Retail and Energy, with another widely used evaluation metric: maximum mean discrepancy (MMD) (Dziugaite et al., ). Lower MMD scores indicate better generative models. From the results shown in Table 4. We can see that our framework still significantly outperforms the baselines in terms of all the new benchmarks. For example, with LSTM as the backbone, our framework achieves a score that is $29.79\%$ lower than Skip Connections on the Energy dataset.

#### F.4 More Data Modalities

**Table 5: Performance comparison on two text datasets, with BLEU as the metric.**
| Method | Backbone | ATIS | SNIPS |
| --- | --- | --- | --- |
| Latent Diffusion | Transformer | $37.12$ | $59.36$ |
| Latent Diffusion w/ Skip Connections | Transformer | $40.56$ | $65.41$ |
| Our Framework | Transformer | $\mathbf{51.73}$ | $\mathbf{78.12}$ |
| Latent Diffusion | LSTM | $35.38$ | $55.72$ |
| Latent Diffusion w/ Skip Connections | LSTM | $39.27$ | $60.31$ |
| Our Framework | LSTM | $\mathbf{48.46}$ | $\mathbf{71.45}$ |

**Table 6: Performance comparison on an image dataset, with FID as the metric.**
| Method | Backbone | CIFAR-10 |
| --- | --- | --- |
| Latent Diffusion | U-Net | 3.91 |
| Latent Diffusion w/ KL Annealing | U-Net | 3.87 |
| Our Framework | U-Net | $\mathbf{3.85}$ |

While our paper primarily focused on time series data, our framework is generally applicable to other types of data, including your mentioned text and images.

##### Experiment on text data.

For text data, considering that natural language sentences exhibits a sequential structure similar to time series, it is intuitive that the posterior of text latent diffusion might also collapse. This intuition is supported by many evidences from previous works (Bowman et al., ; Fu et al., ). To verify that our framework is capable of improving text latent diffusion, we have conducted an experiment using two publicly available text datasets: ATIS (Hemphill et al., ) and SNIPS (Coucke et al., ).

The numbers in this table represent BLEU scores (Papineni et al., ), a widely used metric for evaluating text generation models. Higher scores indicate better performance. As the results shown in Table 5, we can see that our framework has significantly improved the text latent diffusion and notably outperformed a strong baseline—Skip Connections—across all datasets and backbones. Therefore, our framework also applies to text data.

##### Experiment on image data.

For image data, we provide a detailed discussion in Sec. 4.1 of our paper: Image latent diffusion is rarely affected by posterior collapse due to its non-autoregressive decoder. To confirm this claim in practice, we conduct an experiment comparing latent diffusion with our framework on the widely used CIFAR-10 dataset (Krizhevsky et al., ).

The results are shown in Table 6. The numbers in this table represent FID scores (Naeem et al., ), a common metric for evaluating image generation models. Lower scores indicate better performance. Our results show that both the baseline model (i.e., KL Annealing) and our framework improve the image latent diffusion to some extent. However, the improvements are minor, suggesting that image models are almost free from posterior collapse.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 2
questions: 1.	Could the authors clarify how the dependency measure scales with longer time-series datasets? Does the framework handle large datasets efficiently?
2.	Have the authors considered extending this approach to other data types beyond time series? If so, how might the framework need to be adapted?
3.	Is there a specific reason for not including additional baselines, such as non-latent diffusion models, for comparison in the empirical section?
soundness: 2
strengths: 1.	The introduction of dependency measures to diagnose and address posterior collapse is both novel and insightful, providing a fresh perspective on an important issue within latent diffusion models.
2.	The paper offers a solid theoretical foundation for the analysis of posterior collapse, and the proposed framework is well-motivated by both theoretical insights and empirical observations.
3.	The proposed framework demonstrates significant improvements in the performance of time-series generation models, effectively addressing a key limitation in existing approaches.
summary: This paper addresses the problem of posterior collapse in latent diffusion models, specifically when applied to time series data. The authors provide a systematic analysis of this issue, showing that posterior collapse can reduce the expressiveness of latent diffusion to that of a variational autoencoder (VAE). They introduce a novel dependency measure to quantify the impact of latent variables on the generation process and identify a phenomenon called dependency illusion when time series data are shuffled. Building on these insights, the authors propose a new framework that eliminates the KL-divergence regularization, permits an expressive prior distribution, and ensures the decoder remains sensitive to the latent variable. Extensive experiments demonstrate that this framework avoids posterior collapse and significantly improves time series generation.
weaknesses: 1.	While the paper presents strong results for time-series data, it lacks a detailed discussion on the generalizability of the approach to other data modalities, such as images or text. Including a brief exploration or discussion of potential extensions could further enhance the contribution.
2.	The experimental details, including specific configurations for baselines and the selection of hyperparameters, are not fully elaborated in the main text. Providing more comprehensive explanations in these areas would improve the paper’s clarity and reproducibility.
3.	Although the results are promising, some of the visualizations could be made more intuitive, particularly for readers unfamiliar with latent diffusion models. Additionally, converting the figures to vector graphics would significantly improve their quality, as several of the current images appear blurry and lack sharpness, which makes interpretation more difficult. Enhancing the clarity of the figures would improve the overall presentation of the paper.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: - Could the authors include comparisons with recent state-of-the-art time series models, such as ARIMA, LSTMs, transformers, and TCNs, which are naturally robust against posterior collapse? This would contextualize the proposed method’s advantages relative to stable baselines.

- Could the authors provide clearer definitions or examples for terms like dependency illusion and posterior collapse in the context of latent diffusion models? A simplified explanation would improve accessibility.

- Are there specific real-world applications, such as anomaly detection or real-time forecasting, where this framework would be particularly useful? A discussion of practical use cases would strengthen the framework’s relevance.
soundness: 2
strengths: - The paper addresses a previously underexplored issue in time series diffusion models—posterior collapse—which has primarily been studied in variational autoencoders (VAEs) but not in the context of diffusion models for time series.
- The dependency measure provides an insightful tool for quantifying the decoder’s reliance on the latent variable. This measure enables detection of both posterior collapse and dependency illusion, offering valuable diagnostic capabilities for latent-variable models.
- The approach aligns with the paper’s theoretical objectives, yielding meaningful performance improvements.
summary: The paper investigates the issue of posterior collapse in latent diffusion models for time series data, where the latent variable becomes ineffective in influencing the model’s output. The authors propose a dependency measure to quantify how much the decoder relies on the latent variable, highlighting not only posterior collapse but also a related phenomenon termed dependency illusion. Then the paper introduces a new framework to address these issues by removing KL-divergence regularization and enhancing the decoder’s sensitivity to the latent variable, improving posterior stability. Experiments demonstrate that the proposed method achieves better performance than standard latent diffusion models with posterior collapse mitigation techniques across various time series datasets.
weaknesses: - The empirical evaluation lacks comparisons with stable time series models that naturally avoid posterior collapse, such as ARIMA, RNNs, LSTMs, transformers, and temporal convolutional networks. Including these baselines would provide context on whether the proposed framework offers advantages beyond mitigating posterior collapse. The author also did not compare with recent baselines for time series, which are diffusion-based. Please check papers published in NeurIPS/ICLR/ICML in the past two years. 

- The paper references Bowman et al. (2016) to support claims about posterior collapse in latent-variable models for time series, which may be outdated. This raises questions about whether latent diffusion models represent the current state of the art in time series modeling. Comparing the approach with recent state-of-the-art time series methods would strengthen the justification for the proposed framework.

- Although the datasets used are realistic, the paper does not discuss broader real-world applications or scenarios where posterior stability is crucial, such as in anomaly detection or real-time forecasting. Adding context on practical use cases would clarify the framework’s relevance.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: It’s unclear to me why negative local dependency is bad. The authors claimed that it’s because the previous timestamp’s data may be from a low density region and therefore an outlier. But in case that the actual next value to be decoded should indeed be an extreme value, why is that problematic?

Can you discuss the stability of the training? In cases where we 1) train the diffusion model and the decoder together 2) we require the decoder to decode the time series regardless of which timestamp’s noised version of the latent variable is selected.
soundness: 2
strengths: --The paper tries to focus on the specific issues of time-dependency collapse in the case of time series data and diffusion models.

--The shuffling experiments help illustrate how a latent variable is not being used strongly throughout all time steps
summary: This paper addresses the issue of posterior collapse in time-series latent diffusion models, a phenomenon that limits model expressivity by reducing it to a simpler VAE model. The authors introduce a dependency-measure regularization aimed at alleviating this collapse specifically within time-series data. Experiment results on three datasets (WARDS, MIMIC, and Earthquakes) demonstrates initial improvements in preventing posterior collapse over shorter timeframes.
weaknesses: --The problem is not sufficiently well motivated.  In particular, the two types of mode collapse which in time series (time-dependent and time-independent) are not discussed.  The reduction to a VAE is only about the elimination of the time-dependent influence.  The impact of this simplification is not sufficiently discussed.

--Moreover, the less expressivity is not shown explicitly to be a bad thing in the context of time series in general.  There are potentially time series which are driven by a static latent process

--Although the dependency measure is well-defined, there is little theoretical analysis exploring its properties and its relationship to the reduction to a VAE model

--There is no analysis of the results showing specifically how the introduced technique solved the problem of mode collapse. Results with good Wasserstein distance do not directly imply that the issue of mode collapse was resolved.

- This paper claims that they are the first to address the posterior collapse problem in latent diffusion for time series, but it really boils down to the old autoencoder problem. And the diffusion model became a redundant module when the input is a standard Gaussian distribution is a simple extension of the problem of autoencoder.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: N/A
soundness: 3
strengths: The study of the latent diffusion model when applied in the context of time series is a trended topic and super interesting. The authors approach this by defining a proper dependency measure to quantify the problem of posterior collapse of the latent variable, and propose a new framework inspired by re-thinking the design of VAE and autoencoders. The new framework is equipped with new loss functions and regularizations, free from posterior collapse. The discussion comes together with empirical support. Overall, the paper's content is clear and easy to follow.
summary: This paper proposes a new approach to establish a stable posterior for time series within the latent diffusion framework. The new approach circumvents the problematic KL-divergence regularization, prevents the posterior collapse, and maintains the influence of the latent variable in the decoder. The authors provide both theoretical and experimental support to their newly proposed framework.
weaknesses: 1. The experimental results mainly focus on real-world data to demonstrate the sampling benefits of the proposed method. Can the authors conduct synthetic data experiments to interpret and validate the effectiveness of the newly proposed component in the framework (e.g., the introduced loss function or regularization)?

2. The prediction ability of a time series model is critical. Can the authors evaluate the proposed framework in terms of other metrics, such as the predictive MAE, to demonstrate its prediction ability?

3. In addition to point 2, can the author compare with other advanced time-series models? Only compared with the latent diffusion family would not be convincing enough for advertising the model in the time series community.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1.You claim that “when applied to time series data, this framework might suffer from posterior collapse” in the first paragraph. Do you have any evidence to support this claim? Is this phenomenon due to the diminishing influence of the latent variable on the decoder over time steps? How do you justify the decreased dependency correspond to the posterior collapse of latent diffusion for time series data? 

2.In section 4.2, you mention that the the variational inference in your framework leads the latent variable to be smooth in its effect on decoder. Is this the reason why your framework can increase the sensitivity of decoder to latent variable?  Can your framework be applied to non-time series data? It seems the proposed method is not specific to time series data.
soundness: 2
strengths: The authors focus on an important issue of latent diffusion, that is posterior collapse, and propose a potential method to quantify the posterior collapse of latent diffusion for time series data.
summary: This paper aims to address the posterior collapse problem of latent diffusion for time series data. The authors propose a dependency measure method to quantify how posterior collapse happens. And they propose a KL-divergence regularization based method to improve the sensitivity of decoder to the latent variable for time-series latent diffusion.
weaknesses: 1.Regarding to the dependency illusion, you give an example (upper right subfigure of figure 1) to explain. But it’s unclear from figure 1 how you arrive at the conclusion that "Even when the time series is randomly shuffled and thus lacks structural dependencies, the decoder of latent diffusion still heavily relies on input observations (instead of the latent variable) for prediction." Could you clarify how you determine that the decoder "heavily" depends on input observations? Providing a more detailed explanation or additional quantitative evidence would help support this observation.

2.In section 3.1, the definition of posterior collapse seems a general term for all data, not only time series data. In section 3.2, you introduce a dependency measure to demonstrate the occurrence of posterior collapse in time series data. How does this measure specifically address time series data? Would this measure yield the same conclusion if applied to non-time series data?

3.As shown in Figure 4, the dependency of the decoder on the latent variable decreases across both datasets. Although this trend appears improved compared to Figure 2, it would strengthen your findings to compare your method against additional baseline models, rather than only basic latent diffusion.

4.There is a lack of experimental evidence supporting that the proposed dependency measure can accurately assess the impact of the latent variable on the decoder. You should compare your method with other measurement approaches and demonstrate how it outperforms them, providing a more comprehensive validation of its effectiveness.

5.The baselines are not sufficient enough. Only three baselines and all of them are before 2019. Please compare with more the state-of-art works.

6.Reference in this paper seems to be too old. And some of them are repeated. For example, papers in line 573 and line 576 are the same one.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: I agree on the importance of posterior collapse issue found by the authors. I am wondering how is the generation performance of time series diffusion model in which latent variables have the same dimension as the observations.
soundness: 3
strengths: Novelty: It is the first work to discuss the posterior collapse issue for time series latent diffusion. In particular, the paper introduces the novel dependency measure to quantify how the impacts of the latent variables on the generation of the predicting observations, decrease along the time steps. The authors develop a novel framework, which effectively avoid the dependency illusion issue, and outperforms the related time series latent diffusion models in terms of generation quality.

Clarity: This work clearly illustrates the posterior collapse and dependency illusion issues by plotting the dependency measures over time steps. The most parts of the analysis are clearly presented, and easy-to-follow.

Significance: The work demonstrate a significant issue for latent diffusion being applied to capturing time series data. The introduce dependency measure might be used in quantifying the posterior stability of the other related methods, and thus appears to be crucial.
summary: This work starts from an analysis on the posterior collapse issue of latent diffusion for capturing time series. In particular, the KL-divergence from the standard Gaussian prior to the latent variable distribution approximated using latent diffusion, may reduce the time series latent diffusion to a VAE model. The authors define a dependency measure, which shows that the influences of the latent variables on decoding the observations, will decrease to zero along the diffusion time steps. In particular, as analyzed in the paper, the decoder built upon recurrent neural nets may decode the current observations only using the past observations, and thus leads to the dependency illusion issues. To address the problems, the paper develops a novel framework, in which the authors remove the KL-divergence regularization that causes the posterior collapse, decode the predicting observations using the latent variable sampled at intermediate diffusion steps, and introduce a novel penalty to avoid dependency illusion issues. The final experiments demonstrate the new framework can effectively avoid posterior collapse, and thus achieves superior generation quality, in comparisions to some SOTA time series latent diffusion methods.
weaknesses: The final experiments only demonstrate the compared models on only three dataset, using Wasserstein distance to measure between the true and generated time series. Perhaps the experiments could be enhanced by considering more evaluations metrics?

Meta-review: This paper proposed a metric to measure the posterior collapse and introduced a notion of the posterior collapse based on it. They then define an enhanced framework which can alleviate the posterior collapse problem. I found that their posterior collapse is different from the model collapse and Reviewer h3BG is incorrect to some degree.

In conjunction with the reviewers' concerns regarding the comparison with more baselines, however, I, myself, also have some concerns.

1. They need to more carefully analyze what causes the posterior collapse. For instance, LSTMs and Transformers have oversmoothing and oversquashing problems and thereby, their latent vectors are limited in capturing all information in a long sequence. Isn't the posterior collapse by the low-capacity encoder, e.g., LSTMs and Transformers? 

2. Your new framework incurs more interactions and I agree that it can somehow address the problem. It would be nice if you can show that enhancing the encoder/decoder is not sufficient and your framework is needed. There are several papers on resolving the oversmoothing and oversquashing problems of RNNs and Transformers.

3. After solving the above two questions, I recommend that you put more baselines since there are many time series synthesis methods after TimeGAN. Some of them are diffusion-based methods for time series synthesis. 

All in all, I strongly encourage the authors improve the paper one more time. I think they can have a decent chance next time since their problem definition is new.

Decision: Reject