Disentangled interleaving variational encoding

Authors: Noelle Y. L. Wong, Eng Yeow Cheu, Zhonglin Chiam
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Representation Learning, Disentanglement, Probability Theory, Mathematical Optimization, Variational Autoencoder, Time Series Forecasting

Abstract:
Conflicting objectives present a considerable challenge in interleaving multi-task learning, necessitating the need for meticulous design and balance to ensure effective learning of a representative latent data space across all tasks without mutual negative impact. Drawing inspiration from the concept of marginal and conditional probability distributions in probability theory, we design a principled and well-founded approach to disentangle the original input into marginal and conditional probability distributions in the latent space of a variational autoencoder. Our proposed model, Deep Disentangled Interleaving Variational Encoding (DeepDIVE) learns disentangled features from the original input to form clusters in the embedding space and unifies these features via the cross-attention mechanism in the fusion stage. We theoretically prove that combining the objectives for reconstruction and forecasting fully captures the lower bound and mathematically derive a loss function for disentanglement using Naïve Bayes. Under the assumption that the prior is a mixture of log-concave distributions, we also establish that the Kullback-Leibler divergence between the prior and the posterior is upper bounded by the cross entropy loss, informing our adoption of radial basis functions (RBF) and cross entropy with interleaving training for DeepDIVE to provide a justified basis for convergence. Experiments on anonymous bidding data from the National Electricity Market of Singapore (NEMS) show that DeepDIVE disentangles the original input and yields more accurate forecasts, outperforming current state-of-the-art baselines. In the context of the power market, this study can enhance operational decisions and bidding strategies by offering insights into the embedded supply curve via the representation space.

Decision: reject
Reviewer scores: 3, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1729835087242

thread_type:
official_review

<thread_object>
creator: reviewer_n7k7

<content>
summary:
This paper proposes the new VAE architecture that has a special latent variable structure for time series forecasting.
This paper assumes that the target variable of VAE is a combination of future and past variables, 
and that the future variables do not depend on the latent variables, but only on the past variables.
The latent variables also consist of two vectors $a$ and $b$, and the posterior distribution of $a$ depends on $b$, but that of $b$ is independent of $a$.
Furthermore, it is assumed that $b$ consists of a mixture model of K classes. Finally, the likelihood function obtained under these assumptions is the proposed objective function.
In the experiment, the forecasting accuracy is compared using two time series datasets, and the visualization of the latent variable is discussed.

strengths:
- The proposed method is well explained using mathematical equations. 
The expansion of equations is easy to understand, and the assumptions necessary for the expansions are also clearly stated. 
However, as written in Weakness, the practical validity and motivation of the assumptions are unclear.
- In the experiments, the forecasting accuracy is higher than that of VAE and beta VAE on one dataset. 
However, if you only focus on the forecasting accuracy, it is difficult to say that VAE and beta VAE are appropriate baselines.
If the paper is a study of application, it should be compared with baselines for time series forecasting.
On the other dataset, baseline methods seem to be forecasting methods, but the proposed method does not necessarily outperform these baselines.

weaknesses:
- This paper does not succeed in positioning itself in the context of previous research. 
I strongly suggest that this paper should clarify whether the focus of the paper is a proposal of a method for a specific application or one of a fundamental method.
The abstract seems to state that the paper solves a problem in general multi-task learning, but the introduction seems to claim that it is a technique for solving the problem of time series (or ASC?) forecasting.
It is inconsistent and makes the motivation unclear. The paper needs to consistently state the research problem and clearly explain why the proposed method is effective for that problem.
If this paper solves a specific problem in time series forecasting, it should justify why that problem is important and common to broad domains.

- Related to the above, it is difficult to understand the reasons and principles behind the design of the proposed method.
Although the assumptions that lead to the theorem are shown, this paper does not mention the assumptions of the target tasks or the most suitable use cases of the proposed method.
As a result, it is difficult to discuss the generality and importance of the task being solved by the proposed method, and the impact of this paper is unclear.
The graphical models of a, b, x, and y might be useful for readers to understand the assumptions, 
and illustrations of how these variables relate to practical tasks will also help readers understand the effectiveness of the proposed method.

- The paper evaluates the visualization of latent variables but does not compare it with existing methods.
I think that disentanglement of latent variables in time-series data have already been discussed in previous research such as [a].
If the claim of this paper is the importance of disentangling latent variables, the proposed method should be compared with previous methods.

[a] Fraccaro, Marco, et al. "A disentangled recognition and nonlinear dynamics model for unsupervised learning." Advances in neural information processing systems 30 (2017).

- The baseline is not consistent in the evaluation on the two datasets.
Why are the baselines different in Table 1 and Table 2? Is it appropriate to compare VAE and beta VAE using time series data sets?

questions:
If I have misunderstood the paper, please point out it.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729892065067

thread_type:
official_review

<thread_object>
creator: reviewer_ks2u

<content>
summary:
This paper proposes Deep-DIVE, a framework to learn disentangled features from the original input to form clusters in the embedding space and unify the classified features via the cross-attention mechanism. Experimental results on time series forecasting showcase that the proposed framework could disentangle the features and provide better forecasts than existing methods.

strengths:
1. The paper is clearly written, easy to follow and understand.
2. Experiments compared with other baselines showcase that the proposed Deep-DIVE framework achieves better performance than existing baselines.
3. In terms of novelty, the DeepDIVE framework proposed in this work decomposes the latent space z into two distinct dimensions: marginal dimensions b and conditional dimensions a. The marginal dimensions b capture general trends and are independent of each other, while the conditional dimensions a are conditioned on b. This design enables b to capture shared patterns across conflicting tasks, while a learns task-specific features to avoid conflicts. Compared with existing methods, this approach better addresses the challenge of using a single variational encoding to model conflicting time series, resulting in improved performance and disentanglement.

weaknesses:
1. In the introduction section, the author motivates the proposed Deep-DIVE framework by criticizing existing deep learning approaches for time series forecasting as being black-box in nature and hard to optimize. However, time series forecasting (TSF) is a well-established problem. The author should provide further explanation to better justify why the proposed framework is helpful in TSF.

2. The assumption 2 that $q_{\phi}(b_i,b_j|x) = q_{\phi}(b_i|x)q_{\phi}(b_j|x)$ for any i and j is too strong. Although I understand the author's remark that this simplifying assumption often works well in practice, some explanation or intuition about why it often works would be helpful.

questions:
In the experiment results in section 5, Table 2, what does 'std' mean?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730495487069

thread_type:
official_review

<thread_object>
creator: reviewer_ubql

<content>
summary:
The authors propose to extend the use of Variational Autoencoders to generative forecasting and develop a novel objective for training.

strengths:
- The loss function seems to be novel and theoretically grounded.

weaknesses:
- Presentation: The paper is not well-written, it is difficult to follow and understand the idea.
    - The abstract is not specific. Multiple concepts such as multi-task learning, disentanglement, Naive Bayes, and other technical details are presented without a clear, coherent relationship among them. I suggest focusing on a central contribution, such as developing a non-conflicting objective for multi-task learning, and clarifying how elements like disentanglement, Naive Bayes, cross-entropy, and RBF specifically contribute to this goal.
    - Line 46 of the introduction states model explainability to be the key contribution. However, the abstract does not discuss about explainability. I suggest emphasizing a central contribution and ensuring that all concepts throughout the paper align with it.
    -  Line 50 of the introduction mentions conflicting objectives, but a more detailed discussion of how multiple objectives in multi-task learning may conflict and providing examples of these conflicting objectives in the context of the paper would make this concept clearer.
- Baselines: No discussion or reference of the comparison baselines in Table 2 are AUTOCTS(-KDF/KDP), DsaNet, and MtGnn provided. Could the authors provide a brief discussion on these baselines and why they were considered for comparison? Also, why were more recent baselines such as DeepGLO, TCN, and TLAE (as discussed in section 2.1) not included in the comparison?

questions:
- I couldn't find the paper referenced in section 2 (Wong et. al.). Is it published yet? Could the authors provide a link to the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730498047929

thread_type:
official_review

<thread_object>
creator: reviewer_etkb

<content>
summary:
This paper proposes a novel framework for enhancing the interpretability of representations learned from time-series prediction. The authors derive a variational evidence lower bound (ELBO) that extends the original VAE's ELBO to accommodate time-series forecasting scenarios, along with theoretical and implementation details for optimizing this learning objective. Specifically, the latent space is designed into separate marginal (denoted by "b") and conditional (denoted by "a") distributions that contribute to the calculation of the log-likelihood of both the look-back window and the forecasting window. Experiments are conducted on two time-series datasets, and both qualitative visualizations and quantitative metrics are evaluated for the proposed method.

strengths:
(1) The detailed theoretical derivations of the proposed ELBO for time-series forecasting are solid and make sense, and it could be promising and inspiring for future research on learning disentangled representations in time-series prediction tasks. 

(2) I appreciate the authors' efforts to report the variance (standard deviation) of the model's performance, which allows readers to better evaluate the results and comparisons.

weaknesses:
The main weakness of this paper lies in its writing and evaluations. 

(1) The writing lacks consistency between the text, figures, and equations. For example: (1a) While the inference and KL divergence for q(b∣x) is well-discussed, I had difficulties to find any description for inferring q(a∣x,b). In Fig. 1, it appears that a is directly computed from x, which contradicts the equation, and I couldn't find any other text to explain this. (1b) Although the "cross-attention mechanism" and "fusion stage" are mentioned in the abstract and introduction as part of this paper's contributions, they are never described or explained in the rest of this paper. It appears that the proposed framework is implemented just using an auto-encoder architecture, as suggested by Fig. 1 and line 412. Additionally, in line 518, the authors state, "Unlike attention maps and convolutional neural network (CNN) feature maps, DeepDIVE presents data representations in ...," which further increases the confusion about whether the proposed framework includes cross-attention. (1c) There are no descriptions or explanations in the main text for Fig. 3, and the figure caption is also limited.

(2) Although the combined effects of a and b in the latent space are evaluated in Fig. 2, separate evaluations and comparisons of a and b are missing, which I believe is critical for evaluating the proposed framework. Readers may be interested in understanding the differences between the representations learned in a and b, and how using separate a and b is beneficial compared to using just one latent variable. The authors are encouraged to conduct comprehensive experiments to demonstrate the representations learned in the marginal and conditional distributions in the latent space, separately, and how they impact the final prediction tasks. For example, the authors could consider using t-SNE plots for a and b individually and presenting traversal results for both of them to identify any expected differences.

questions:
(1) Total correlation is also an option for promoting the independence of the latent variables. Besides using Naive Bayes, do the authors have any experience or insights in deriving and optimizing total correlation terms within the KL divergence? Additionally, how would this affect the model performance compared to using Naive Bayes? For instance, one approach for optimizing total correlation can be found in [1].

[1] Chen, Ricky TQ, et al. "Isolating sources of disentanglement in variational autoencoders." Advances in neural information processing systems 31 (2018).
</content>
</thread_object>

</thread>
<thread>
creation date:
1732553634033

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment by Authors

comment:
Dear Esteemed Reviewers,

We thank all reviewers for their thorough evaluation and valuable input to help improve the manuscript! We are gratified that most reviewers find our derivations novel (ETkb, ubqL, kS2U), theoretically grounded (ETkb, ubqL, N7k7) and inspiring (ETkb), although we also acknowledge that the presentation of our work needs improvement (ETkb, ubqL, N7k7). In light of the reviews received, to better help readers understand the impact of the work and its position with regards to other related works, we have decided to strive for more focused discussion regarding the extension of the original VAE architecture to include the use of labelled data for a forecasting use case. We motivate the use of semi-supervised learning considering the fact that while it is common to employ a sequence as input to a time series forecasting model, other available labels that classify the entire sequence are often overlooked. For this task, the benefits of our variational encoding architecture are two-fold. Building upon the statistical foundations provided by the VAE for learning deep latent-variable models and corresponding inference models for reconstruction, DeepDIVE also provides a principled and well-founded semi-supervised framework for learning deep latent-variable models and corresponding inference models which is useful for reconstruction, forecasting, and classification. We also believe that probabilistic latent variable models such as VAEs hold great potential for facilitating autonomous planning and resource allocation in situations characterized by uncertainty, mirroring real-world scenarios. We will address all remaining concerns and clarifications in an individual response. We thank the reviewers again for their time.

We are also currently modifying the manuscript and will publish the updated manuscript within approximately 15 hours.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper applies the KL term decomposition techniques from disentangled VAE literature to improve time-series forecasting. The idea is to disentangle the latent features, and in decoding part processing them individually before final combination via the cross-attention.

The major issues of this paper from reviewer point of view are (1) lack of ablation study and certain baselines, and (2) confusing presentation regarding the explanation of the design choices. Most reviewers agreed that the technical derivations are solid though.

While the overall techniques and the key equations are clear to understand, I think the paper would benefit from a rewrite and reordering the presentation. Personally I would present Section 4 first with more space to explain the latent variable structure, and then go back to section 3 for the training details. Also I personally recommend streamlining the theorem presentations and proofs (e.g., put them to appendix), and just focusing on explaining the core intuition of the key loss function. The proof technique for many of the other theorems/propositions -- manipulating the KL term -- is well known in disentangled VAE & variational inference literature.

PS: the revised version of this paper exceeds the page limit (10 pages) for the main text.