Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose a learning criterion and numerous bounds for learning representations of dynamic systems that are at once maximally compressed (minimal mutual information with their original representation, i.e., minimal rate) while being maximally informative a future time point (or, due to symmetry, a previous timepoint). This leads to learned encodings which are, up to the SNR and trade-off parameter $\beta$,  reflective of the dynamical system.
strength_and_weaknesses: Strengths:
* The method is grounded in established literature (information bottleneck and variational information bottleneck, and more generally Rate-Distortion theory), yet contributes a novel criterion. Secondly, the recognition that the stationarity assumption reduces the learning criterion to a two term loss function, which is, in structure, very similar to many other information trade-offs in the literature.  
* Multiple bounds are explored for the mutual information minimization/maximization.  
* Experimental results are generally well done, modulo exact generation details.  

Weaknesses:
* Allowing time-windows T, for periodic or pseudoperiodic phenomena with period T' we might observe aliasing artifacts. While this remains stationary in a global sense, will such artifacts impede dynamics. Moreover, will the interpretation or downstream use of those dynamics be impeded by this induced false beat frequency at (T-T')/2?  
* The variational lower bound (VLB) of Theorem 3 does not appear to require a decoder, contrary to the comment in the sentence immediately following the statement of the theorem; it seems as though it instead requires a good estimate of the conditional likelihood (a prediction from either T to T' or vice versa).  

clarity,_quality,_novelty_and_reproducibility: It might be more clear to note the differing time-blocks at $T_0$ and $T_1$, since there is no reliance on any symmetry pattern (i.e., there is nothing special about -T versus T as far as I understand.

It might be also helpful to be more clear about the lifting process/simulation for the attractor datasets. While the general idea is conveyed, the exact details it seems could be easily shared (or code given to reproduce the test cases). Overall however this paper seems reproducible.

Overall the bounds are restatements of other results; this is somewhat clear in the paper, but it should still be noted re:Novelty for the purposes of review. There are several other differentiable mutual information estimators, including some that avoid the Gaussian encoder function. Though the paper is already quite thorough, it may be helpful to also test these functions, e.g. the correction of MINE in Song et al 2019 (called SMILE), and the Echo noise encoders in Brekelmans et al 2018.
summary_of_the_review: The authors present an elegant embedding method for dynamical systems, and the estimation machinery to fit the embedding efficiently. The method is novel, but also fits well into existing literature, adding new and interesting directions to Rate-Distortion based encodings. I think the ICLR community at large would be interested in such a manuscript.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose a method for self-supervised time-series prediction called Compressed Predictive Information Coding (CPIC). The core idea is that a sequence of data points are mapped to a Gaussian latent space via some encoder function. Then, these latent representations can be used for downstream tasks, e.g. regression. The authors propose minimizing the difference between two computationally intractable mutual information terms to train the model. To make their approach practically applicable, the authors propose to minimize a variational upper bound to the original objective. The authors test their approach on synthetic and some real-world regression problems.
strength_and_weaknesses: ## Strengths
The proposed approach is well-motivated and an interesting idea. It is quite simple; hence, I wonder if it has already been proposed elsewhere. However, I am not familiar enough with the sequential prediction literature to be able to comment on the originality of the approach though. 

Some of the empirical results also seem promising.

## Weaknesses
In addition to the list below, the paper suffers from severe stylistic and clarity issues; see the section below.
 - In Sections 4.1 & 4.2, the authors state four "theorems" that give certain variational bounds on the intractable terms in their proposed loss function in Eq 4. However, it is not entirely clear what the authors are claiming here as their original contribution. The statements in Thm 1 and 3 follow trivially using elementary information-theoretic arguments in two lines (Eqs 9 and 11 in the appendix). The statements of Thm 2 and 4, and 5 are essentially taken from other works with trivial substitutions. Could the authors please comment on what exactly they are claiming as their contribution here?
 - The authors never formally state the exact models used for their experiments, making their results impossible to interpret.
clarity,_quality,_novelty_and_reproducibility: The writing generally needs improvement. For example, there are countless typos, imprecise technical language and strange grammatical structures. However, there are also more major stylistic issues:
 - Much of the Introduction is dedicated to discussing related works in a way that doesn't directly pertain to the authors' proposed method. These parts should be moved into the related works section. The clearest example is the second paragraph, which should be (perhaps even verbatim) moved to Section 2.
 - Similarly, most of the first paragraph of Section 3 should also be moved to the Related Works section.
 - The authors' work seems to be very closely related to variational recurrent auto-encoders and variational state-space models, yet they are not discussed in the related works. Could the authors please comment on this?
 - In Section 3, the authors write: "A nonlinear CPIC refers to a stochastic nonlinear encoder including a nonlinear mean encoder and a linear variance encoder, while a linear CPIC refers to a stochastic linear encoder in which it replaces the linear mean encoder by a nonlinear mean encoder." - I highlight this particular sentence because it is supposed to describe the variants of CPIC that are later presented, but is very challenging to parse and probably contains errors.
 - In Thm 4, the definitions of "critic function" and "baseline function" are missing.
 - The authors only present results for the multi-sample upper bounds in the main text. Hence, I don't think presenting the univariate bounds is useful, and their discussion should be moved to the appendix.
 - Probably every section title in Section 5 should be reworded. In particular, "Numerical demonstration of the superiority of CPIC" should be renamed to "Results" or "Experiments".
 - "The motivation for using linear forecasting models is that good representations contribute to disentangling complex data in a linearly accessible way" - what does "linearly accessible way" mean?
 - Just below Eq 3: "$U \in \mathcal{R}^{N \times D}$" - what is $U$?
 - What is $f_1$ and $\gamma$ in Eq 8? In the sentence below, what is $\beta$?
 - Label font sizes in Figure 2 and the legend font sizes in Figure 3 should be increased as they are currently hard to read.
 - Instead of showing mean and best performance, Figure 3 should show median performance with the inter-quartile range as the error bars to give a better idea of performance.
 - "Finally, as the SNR gets lower (SNR = 0.001, bottom-right) all methods perform poorly, but we note that, numerically, considering predictive information in latent space is much better than that between latent and observation space." - why is stochasticity better numerically? There is no justification given for this claim.

summary_of_the_review: The authors present a nice, well-motivated idea. However, it is unclear what theoretical contributions they claim, the description of their empirical methodology is incomplete, and the presentation quality is generally low.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a way to optimize time series forecasting model through adapted information bottleneck loss. In particular, it is proposed to learn low dimensional representations of the input time series (using variational encoders) via maximizing the mutual information between representations of the input and target and minimizing mutual information between input and representation. The approach is termed compressed predictive information coding (CPIC) and the authors propose two different computable bounds, based on the existing estimators of mutual information, that can be used to train the models. Further, authors explore the empirical performance of the proposed bounds, first on the artificially generated data with clear structure, to check the ability of the representations learned with CPIC to reconstruct data. The second experimental evaluation is done on the real data where forecasting is made using linear regression on the learned representations.
strength_and_weaknesses: The paper is proposing an information bottleneck type of the loss for a different from usual setup. The time series forecasting is a complicated task that is an active research area and novel approaches are always interesting.

The paper uses existing estimators and bounds on mutual information, thus the main novelty of the paper is the formulation of the information bottleneck loss and finding methods that allow to compute it. The evaluation shows that the proposed method performs better than two existing approaches, though still with not very high performance.
clarity,_quality,_novelty_and_reproducibility: The writing of the paper has some drawbacks that make it harder to follow up the ideas. Some examples are in the following:
1) use \citep when citations are inside of the sentence
2) sequential data is not always dynamic and not always timeseries (using i.e. in this case is not justified)
3) some terms are used without introduction, for example "low-level information" and "compression complexity" in section3
4) it would be nice to specify what activation functions are used in the encoders
5) notation on page4, in the end, is not very nice: usually p(X(T), Y(T)) will denote a probability of having this tuple, not the probability distribution, which is meant there
6) typo in paragraph after Theorem3: q(y(T)|y(-T)), not q(y|x)
7) in section4.3 it is an upper bound on CPIC, not lower bound
8) in section4.3 it was unclear why there is a one-sample bound and multiple-samples bound. It should be explained more precisely.
9) in section5.2 it is mentioned that "predictions are conducted by linear regression to emphasize the structure learned by the unsupervised methods". It is a confusing formulation and it should be described in more details what is the reason to use linear regression.

The novelty of the paper is about formulating information bottleneck loss for the time series forecasting task.

The code for the experiments is not provided, reproducibility might be hard.
summary_of_the_review: The paper formulates an information bottleneck loss for timeseries forecasting problem, using representations learned with variational encoders. The existing estimators combined to produce a computable bound for the proposed loss and further this bound evaluated for the task of reconstruction and forecasting.
Overall, my main concern is about the reported evaluation results, which seem to be not very high. Since the core of the paper is formulation of the loss, the empirical evaluation should be very convincing.
Moreover, why the reported results for existing methods (CPC and DCA) are so much worse, nearly twice on average or even more? Are these methods so bad for the problems selected for experiments? Or only with the SNR that is very low? The reported results are also very low for the real world data. I wonder what is the state-of-the-art scores on the problems considered.

----
I thank the authors for the clarifications. I will stick with my score after the internal discussion.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims at extracting low-dimensional structure for dynamic data, especially for time series data. The proposed method, Compressed Predictive Information Coding (CPIC), both minimizes compression complexity and maximizes the predictive information
in the latent space. This work extends the prior works dynamical components analysis and deep autoencoding predictive components. This work includes the analysis for both linear and non-linear encoding. The core idea is to estimate the latent predictive information without heavy matrix computation, but with cheap approximations. The resulting model achieves good performance on synthetic lorenz attractor, two neuroscience datasets and other real-world scenarios.
strength_and_weaknesses: Pros:
1. The variational bound alleviates the cost of predictive information. It can potentially be used in more scenarios as a plug-in module.
2. Operating in the latent space makes more sense than input space and employs the power of non-linear layers.
3. Various experients and real-world datasets are shown.

Cons:
1. It seems that DAPC has a very similar framework, also with a deterministic and probabilistic encoders (also a VAE?). Could you further elaborate on the differences?
2. Could you further justify the stationarity assumption? Is it commonly seens in time series analysis?
3. When you demonstrate the qualitative results (e.g. synthetic data), maybe you can also show some results from other methods.
4. You can also show some time cost numbers to demonstrate the advantage of your method speed-wise.
clarity,_quality,_novelty_and_reproducibility: In general, this method extends the prior works and propose a variational bound to reduce the computation cost. The paper has both theoretical and empirical results. Most related methods are included for comparison. The code is not provided. The method is not straightforward to re-implement.
summary_of_the_review: This paper improves over previous method, and potentially reduces the time cost. Operating in the latent space also takes advantage of the large capacity brought by deep models. It would be better if the method is also tested on larger and more mainstream time series datasets.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: In this paper, the authors propose a new information bottleneck criteria to find a latent representation for time series. They provide a novel self-supervise learning criterion trained using a variational lower bound (See the summaries by the reviewers for more information).  

The strongest results in the paper are in Section 3, where the authors present the algorithm. All the reviewers liked this section, especially reviewer U5Xb is enthusiastic about this result. The algorithm is compelling and belongs at a major ML conference. 

The weakest aspects of the paper are the misleading Section 4 and, especially, the experimental results in Section 5. The reviewers point out that the theorems in Section 4 are the application of existing results to the proposed problem. Section 4 should be an easy fix. 

The main issue with the paper is the experimental results that are not reproducible. The authors have tried to respond to these criticisms, but the needed work is more fundamental. For example, in the Lorenz attractor results, the authors simulate the 3D time series and linearly map it to a 30-dimensional space (random matrix). They use their procedure to map it back to a 3D space. From Figure 2, top row, it seems that they recover exactly the Lorenz attractor trajectory. How they can avoid rotations or permutations is not explained. If this is not an issue, it would be fundamental to know why.

The experiments with real data are also not well-detailed and, it is impossible to know if the comparisons are fair to the other methods. To validate the results, they run a prediction task in which the authors mention that they use a linear classifier for interpretability, but if the encoder is nonlinear, the features found by the proposed procedure are not interpretable anyway. Also, we do not know the values for Q in these experiments or the structure of the neural nets used for each time series. 

Finally, the authors mention that T is either 1, 2, or 3. But they do not say which one it is. Even for T=3, the encoder does not look into a significant portion of the time series. T = 3 is too short for that. But for T=1 the encoder only looks at independent samples. It would be very important to understand the role of T in the algorithm and how it does process the data. If T=1 provides the best solution, the role of the encoder would be very different for a T>1 or even for a T that is in the tens of samples. This aspect needs to be studied before the paper can be published.  

I apologize to the authors for bringing new criticism to the experimental section of the paper. These issues arouse in a zoom meeting with the reviewers to decide if we should accept the paper for ICLR. We encourage the reviewers to improve the paper and submit it next year. 
justification_for_why_not_higher_score: The experimental section is not reproducible and the comparison to other methods might not be fair.
Decision: Reject