Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Through both rigorous theoretical analysis and empirical findings, this paper extends results on Learning using Privileged Information to general time series governed by linear latent dynamics and potentially nonlinear observation models. Under this general class of possible models, the authors demonstrate through learning theoretic analysis that learning with privileged (data that is available at training time but not at inference time) time series always leads to lower or equivalent risk, given that the nonlinear map from observations to latent variables is known up to a linear mapping. They then extend their results to the unknown mappings via an argument based on the use of random feature maps. Finally, the authors propose to approximate unknown mappings with deep learning architectures. In their empirical study, they demonstrate that the predicted learners with privileged information generally outperform baselines in terms of sample efficiency.
strengths_and_weaknesses: The paper benefits both from strong theoretical analysis and rigorous empirical set-up. The results presented are relevant, and significantly expand understanding of learning using privileged time series (LuPTS) information. To the best of my knowledge, the paper proposes both novel theoretical findings and explores a set of new approaches for LuPTS. Finally, the paper is well-written, with clear and concise use of language.

My only concern about the paper is motivation. It is not clear to me, except some cases alluded to in the introduction to the paper, in which applications one would be interested in using privileged information. Unfortunately, the empirical study also does not motivate a single application but rather transforms existing time series applications such that the problem definition fits. I believe the readers would benefit greatly from a discussion of motivation and potential impact.
questions: - Sample efficiency is qualitative and subjective. In Fig 4(b) OLS seems to be outperforming all baselines in terms of "sample-efficiency" in contradiction to the text. So what precisely is meant by sample efficiency here? Could that be made more rigorous?
- Figure 7(a) is not readable, and the conclusion is mostly not accessible. Could you please revise and explain?
- Will the experiments be made available and reproducible? There is no mention of this in the paper.
limitations: - I would recommend to spend much more time on the motivation, and amplify this in the experiments. For example, how does this setting connect to images (which is presented in the experiments)? 
- I would recommend to make the findings around L151 more precise instead of deferring to the appendix as I believe this is one of the key findings in the paper.

Minor point:
- L66 typo in "respect"
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper deals with the data generated by the following model.

A vector $z_1$ comes from some distribution. Then it is subjected to a chain of linear transformations. The vector $z_2$ is obtained as a linear transformation of $z_1$ (plus some noise), $z_3$ is a linear transformation of $z_2$ etc. Finally we get to $z_T$. Then $z_T$ generates a label $y$ by a linear transformation of $a$ (finite dimensional) feature mapping.

The vectors $z_t$ are never visible directly. There are two modes of access to them. In the first, "classical" scenario we get to see $x_1 = \Psi (x_1)$ where $\Psi$ is a linear transformation. In the second, "privileged" scenario we get access to $x_2 = \Psi(z_2), x_3 = \Psi(z_3)$ etc.

We can have either classical or privileged information at the training stage, but only get classical information at the test stage. The paper compares two learning approaches.

In the first, classical approach we use Least Squares to regress y on a feature mapping of $x_1$. In the second, privileged approach we try reconstruct the whole model using least squares on the training set to find all transition linear transformations etc and get $y$. The paper shows that the variance of the later approach is lower than the variance of the former.

The advantage disappears if the feature mapping is infinite-dimensional and we get non-singular Gram matrices on the data.

The experimental results include reconstruction of the feature mapping using neural networks.
strengths_and_weaknesses: I think this is an interesting result but its significance is somewhat limited. The authors very honestly show the key limitation for universal kernels. The result is certainly important for the dynamic systems community but I have doubts about wider ML audience. The very model appears limited (although it has important applications in neurophysiology).

Typos/suggestions

Page 3, line 93 on the form -> of the form ?

I also find the title somewhat misleading. This is not what is usually meant by a time series. "Dynamic systems" would be better.
questions: None. The paper is clearly written.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes new insights into the analysis of Learning using privileged information  (LuPI) learner with access to intermediate time series data and  generalizes it to nonlinear prediction tasks in latent dynamical systems. The authors extend theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform. In addition, the paper proposes algorithms based on random features, and representation learning for the case when the map is unknown. 
strengths_and_weaknesses: The paper is well written and theoretically sounds. The paper is clear easy to follow and bring to the fore an interesting contribution.

Particularly, the fact that the authors extend the LuPTS framework to nonlinear models and prediction tasks in latent dynamical systems is an important step. 

Furthermore, the authors prove that learning with privileged information leads to lower risk when the nonlinear map connecting latent variables and observations is known up to a linear transform. This is also an important contribution.

The bias/variance analysis is also timely and important. Also, the experimentation with real data namely the prediction of traffic volume and Alzheimer progression is critical to validating the approach.

However, the fact that the study is limited to predictions for a fixed time horizon given present observations limit the extend to which the approach could be validated and applied.
questions: What is the benefit of the approach for prediction if it is not experimented for long term prediction?

How would the approach fare for prediction using different prediction horizons?

Beyond bias/variance analysis, what is the significance of this approach?

Which kind of non-linearity makes the proposed approach non-longer valid?

What is the algorithmic complexity for its implementation and use in practical settings?
limitations: It would be good if the authors could discuss the impact of their bias/variance analysis on societal applications?
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The manuscript proposes to predict nonlinear outputs using time series privileged information during training. The privileged information is not used for inference. The authors prove in Theorem 1 that under some conditions, "the generalized LuPTS is never worse in expectation than the classical learner". Experiments on both synthetic and real-world data sets show that using privileged information increases the sample efficiency and helps latent variable recovery. 
strengths_and_weaknesses: * Originality

From the title, it is not clear how much this manuscript differs from [1] Karlsson et al. (2021) (https://proceedings.mlr.press/v151/k-a-karlsson22a/k-a-karlsson22a.pdf)
While in the main text, the manuscript tackles the task by modeling with latent variables for each observation, which is a major difference.
If I understand correctly, Theorem 1 in the manuscript is similar to Theorem 1 in [1] with some changes of conditions.
The manuscript should better formulate the difference from [1] 

* Quality

The manuscript is technically sound. 

* Clarity

The manuscript is overall clearly written. Some (minor) suggestions are as follows.

1. "**Unobserved**" applies to Figure 1 (b) but not Figure 1 (a).
2. It is not clear what random variables the "distribution $p$" models in line 62.
3. "the matrix inverse $(\cdot)^{-1}$" in line 104 can not be found in the text. 
4. Figure 3 (b) is not informative enough. 
5. It is not clear what the metric in experimental evaluation is used. What is $R^2$ in Section 4?


* Significance

The algorithm proposed in the manuscript would benefit the time series modeling community

---

[1] Karlsson, Rickard KA, et al. "Using time-series privileged information for provably efficient learning of prediction models." International Conference on Artificial Intelligence and Statistics. PMLR, 2022.
questions: * Stochastic process

If I understand correctly, the latent states $Z_t$ are assumed to be from a stochastic process, is it?
It is yet not clear from the main paper whether $Z_t$ is stationary or not. 
I see some conditions of matrix $A$ in Appendix (line 659), but I did not find the constraints of generating $A$. That is, it is not clear why $A$ is designed to be like that. 

* OLS Linear estimator 

What is the difference between OLS (line 100) and the linear regression?
$\hat{\theta}_C$ in eq(3) seems to be exactly the same as linear regression.

* $R^2$

What is $R^2$ in Section 4?

* Efficiency

The manuscript claims in line 237 that the main goal is to improve learning efficiency. 
How "learning efficiency" is defined and estimated in the manuscript?




limitations: The authors addressed the limitations.
review_title: null

Meta-review: metareview: This paper considers a particular setting of time series prediction with privileged information. A special case can be described as predicting x(t+k) from x(t). At training time one is also given x(t+1), x(t+2), ..., x(t+k-1) and a latent dynamics is assumed. The paper presents a learning algorithm that leverages privileged info at train time, provides rigorous theoretical analysis of this algorithm and  convincing numerical experiments. This paper is definitely of interest to ML community and would serve as an interesting contribution to the conference.
confidence: Certain
Decision: Accept