Efficient learning of nonlinear prediction models with time-series privileged information
Authors: Bastian Jung
Fredrik Daniel Johansson
Venue: NeurIPS 2022 Accept (2022)
Keywords: Privileged information
sample efficiency
time series
latent dynamics
representation learning

Abstract: In domains where sample sizes are limited, efficient learning algorithms are critical. Learning using privileged information (LuPI) offers increased sample efficiency by allowing prediction models access to auxiliary information at training time which is unavailable when the models are used. In recent work, it was shown that for prediction in linear-Gaussian dynamical systems, a LuPI learner with access to intermediate time series data is never worse and often better in expectation than any unbiased classical learner. We provide new insights into this analysis and generalize it to nonlinear prediction tasks in latent dynamical systems, extending theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform. In addition, we propose algorithms based on random features and representation learning for the case when this map is unknown. A suite of empirical results confirm theoretical findings and show the potential of using privileged time-series information in nonlinear prediction.

Efficient learning of nonlinear prediction models with time-series privileged information
Bastian Jung
Fredrik D. Johansson Chalmers University of Technology fredrik. f jww.anssonnchalmmrssss....
Chalmers University of Technology mai mail0bastianjug com com
Abstract
In domains where sample sizes are limited, efficient learning algorithms are critical. Learning using privileged information (LuPI) offers increased sample efficiency by allowing prediction models access auxiliary information training time which is unavailable when the models are used. In recent work, it was shown that for prediction linear-Gaussian dynamical systems, LuPI learner with access intermediate time series data is never worse and often better in expectation than any unbiased classical learner. We provide new insights into this analysis and generalize it to nonlinear prediction tasks in latent dynamical systems, extending theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform. In addition, we propose algorithms based on random features and representation learning for the case when this map is unknown. A suite of empirical results confirm theoretical findings and show the potential using privileged time-series information in nonlinear prediction.
Introduction
In data-poor domains, making the best use of available information central efficient and effective machine learning. Despite this, supervised learning often applied in such way that informative data ignored. A good example of this learning to predict the condition of a patient at set follow-up time based on information of the first medical examination. Classical supervised learning makes use only of the initial data to predict the disease status at the follow-up, although in many cases data about medications, laboratory tests vital signs routinely collected about patients also at intermediate time points. This information is privileged (Vapnik and Vashist 2009). it unavailable at the time of prediction, but be used for training model.
Learning using privileged information (LuPI) (Vapnik and Vashist 2009), generalized distilla- tion (Lopez-Paz et al. 2016) and multi-view learning (Rosenberg and Bartlett 2007) have been proposed to increase the sample efficiency by leveraging privileged information in learning. Theoreti- cal results guarantee improved learning rates (Pechyony and Vapnik 2010 Vapnik al. 2015) or tighter generalization bounds (Wang 2019) for large sample sizes under appropriate assumptions. However, privileged information is not al always beneficial; it must be related to the task hand (Jon. schkowski et al. 2015). Previous works do not identify such settings. Moreover, existing analyses do not state when learning with privileged information is preferable classical learning for problems with small sample sizes-iwiiee which is where efficiency is needed the most.
Karlsson et al. (2022) studied LuPI in the context of predicting outcome observed at the end of time series based on variables collected at the first time step. They showed that making use of data from intermediate time steps in particular settings always leads to lower equal prediction risk-for any sample cizp-ce to the best unbiased model which does not make use of this privileged information. However, their method called learning using privileged time series (LuPTS) was limited to settings where the outcome function, and estimators of it, are linear functions of baseline features.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

A1
A2
A
Aa
4
2
Z
Zp
X
V
V
Observed
Privileged
Unohserved
(a) Linear dynamical system, fully observed
(b) Linear latent dynamics, nonlinear observations
Figure I: Comparison between the linear data generating process and the latent nonlinear generaliza- tion in this work. v indicates the observation function of the latent system, with & its left inverse.
Moreover, their analysis did not study how variance reduction behaves as a function of increased input dimension. Hayashi et al. (2019) also learned from privileged intermediate time points but their study was limited to empirical results for classification using generalized distillation.
There an abundance of real-world prediction tasks with fixed follow-up times which can be framed as having access to privileged time-series information. Examples include predicting 90-day patient mortality (Karhade et al.) 2019) or patient readmission in healthcare (Mortazavi et al.) 2016). the churn of users of online service over fixed period (Huang al. 2012) yearly erop yields from satellite imagery of farms Yoou et al. 2017). In these cases, privileged time-series information comprises daily patient vitals, intermediate user interactions and daily satellite imagery respectively. We are motivated by finding sample-efficient learning algorithms that utilize privileged time-series information to improve upon classical learning alternatives in such settings.
Contributions. We extend the LuPTS framework to nonlinear models and prediction tasks in latent dynamical systems (Section 3 In this setting, we prove that learning with privileged information leads to lower risk when the nonlinear map connecting latent variables and observations is known up to linear transform (Section 3.T). In doing so, also find that when the representation dimension grows larger than the number of samples, the benefit of privileged information vanishes. We show that privileged learner using random feature maps can learn optimal models consistently, even when the relationship between latent and observed variables is unknown, and give a practical algorithm based on this idea (Section B.2). However, random feature methods may suffer from bias in small samples. As a remedy, we propose several representation learning algorithms aimed at trading off bias and variance (Section 3.3). In experiments, we find that privileged time-series learners with either random features or representation learning reduce variance and improve latent state recovery in small-sample settings (Section on both synthetic and real-world regression tasks.
Prediction and privileged information in nonlinear time series
We aim to predict outcomes Y in y C RR9 based on covariates Xi in X C R k X1 are observed at baseline time point = 1, starting a discrete time series on the form X1, X2.... X2, Y. Outcomes assumed to be compositions of representation 4, linear map A and Gaussian noise e,
Y = h(X,) + where h(z1) = 0 (z1), and E E~ (0
In addition to observations of X, and Y, privilegeed information (PI) is available in the form of samples of random variables, X2,..,, Xr, from intermediate time points between X₁ and Y. all taking values in X. Two data generating processes (DGPs) with this structure are illustrated Figure D Unlike baseline variables X1, privileged information is observed only training time, not test time. Therefore, it can only benefit learning, and not inference. Data sets of m E N. training samples, D = (( 1, 24,2; Ti 9:) J are drawn independently and identically distributed from a fixed, unknown distribution p over all random variables in our system. We let X, [1 ... Xm,t denote the data matrix of features observed time 1 = 1,..., T and Y [y1, ymm] the vector of all outcomes observed in D. A learning algorithm o D + H. maps data sets D E D to hypotheses h E H. An efficient algorithm sf has small expected risk (od) with respect to a loss L y x y + R over training sets of fixed size m drawn from p.
R,(c/)= = EDIRR where h (D) and (h)h) = EEpLL (,),()()),)))))
(2)
We study the regression setting with I. the squared error, T(, = lly v' - In analysis, we focus on univariate outcomes, q 1. but all results extend to multivariate outcomes, g 1.

---

Algorithm 1: Generalized LuPTS
0.9
Input: Data D ((X,), Y); Repr. & or kernel A
0.8
if using fixed representation & then
0.7
OLS
LuPTS
Z. (1) 2 (xm, for = 1,...?
0.6
T I
0.5
0.
2 2. 2 Z
2 ZT) 2.7
0.4
A
8
0.3
50
100
150
(.) 0g (.)
Features,
else if using kernel then
Figure 2: Two regimes of generalized LuPTS with varying feature dimension d, and sam- ple size m = 100. When m > d, gener- alized LuPTS is provably never worse than OLS. When m I d, they are equivalent. See Appendix F for details.
K. = xr X,, X for I,...
ki Il K.K] Y
hp()) = Ei=1 (.
return hp
Our main goal is compare privileged learners ddp which make use of privileged information, to classical learners Ac which learn without PI. We seek to identify conditions and algorithms for which privileged learning is more efficient, i.e., it leads to lower risk expectation, R(,4)) & R( for the same number of training samples. We describe such setting next.
2.1 Privileged information in latent dynamical systems
We study tasks where data are produced by latent dynamical process. Observations X, are generated from unobserved latent states Z,, through nonlinear transformation V see Figure lb This means that the target of learning, (xX1) = E Y X1). is nonlinear X₁. Latent dynamical systems like these have proven successful at modelling a variety of phenomena as for example fluid dynamics in
physics (Lee and Carlberg 2019) and human brain activity in neuroscience (Kao et al. 2013).
Assumption (Latent linear- Gaussian system). Let Zi.... ZT be latent states related as linear- Gaussian dynamical system in a space C TRdd with Z1 of arbitrary distribution. Further, let the observation function be an injective map T Z + X with reft-inverse (representation) tb X - Z. With A,, .... AT-1 E Rdxd B E R.d assume that 7,,..., ZT, X1....
11....Xyy XT, Y are generated as
Z = A + for t 222, and Vt X, (Z)) and == Zq cy
where E N (0, I and ey ~ ~N(0,03).
It is easy to verify that Assumption T consistent with 1b. but stronger: there are more systems that map X1 to as 0 (Xx) than systems where additionally X, = (Zz).. Nevertheless, is much more general than the results of Karlsson et al. (2022). limited to the linear setting, i.c. 4(x1) 21
Next, we present a generalized version of the LuPTS algorithm of Karlsson et al. 2022 that is provably preferable to classical learning when data is generated according to Assumption I and 4 is known up to linear transform. Further, we discuss how privileged learner can be made universally consistent for unknown representations &b when combined with random features.
3 Efficient learning with time-series privileged information
We analyze and compare learners from the privileged and classical paradigms which produce estimates of the form (((x)) 0T ((x1). motivated by ( We let each use representations & X' - 2 C &d from a family F shared by both paradigms, so that the hypothesis class H e h is shared as well.
3.1 Learning with true representations known up a linear transform
In this section, we assume that privileged and classical learners have access to common, fixed representation function & which is related to the true representation function through a linear transform, meaning there exists matrix B such that ④(-) (.(-).

---

Classical learning. As a classical learner to serve as a strong comparison point for our privileged learners, we use the ordinary least-squares (OLS) linear estimator applied the latent variables at the first time point inferred by Φ. With Z, = (1),1), (2 ) E Rmxdd
<c(D) = he(.) = og (.) where d. = 1
(3)
When m 2 d and & is known up to linear transform, he is the minimum-risk unbiased estimator which does not use privileged information. To accommodate the underdetermined case where m < the matrix inverse is replaced by the Moore-Penrose pseudo inverse (-)1 Penrose 1956).
Generalized LuPTS. For privileged learning, first compute 2, ((((21.) ((xm.)) for t= 1,... T. Then, we independently fit parameter estimates AI.... Ar, 3 of the dynamical system shown in Figure Ib by minimizing the squared error in single-step predictions. This equates to series of OLS estimates in 2. At test time, baseline variables I1 are embedded with & and the latent dynamical system is simulated for T time steps to form a prediction hx()1) = (Â.... ....... Ar.3) T ((x1). Putting this together, we arrive Algorithm | which we call generalized LuPTS. We may apply a simple matrix identity and replace terms Z, Algorithm by the Gram matrices K, = ,,,,, X,) of a reproducing kernel к with corresponding (implicit) feature map $ x(x, = (x)), ((z') Smola and Schölkopi 1998 This variant allows for learning with unknown 4.
We now state a result which says that under Assumption | for an appropriate fixed representation + or kernel K. generalized LuPTS is never worse in expectation than the classical learner in
Theorem 1. Let D be data set of size m drawn from p. consistent with Assumption ! Assume that the left inverse T X -> Z of the observation function or through kernel k( x²) (f(x), ((x'); ie., there exists a matrix B with linearly independent columns such that ((x)) (D(()) with & : + for all x E X. Then, holds for the privileged learner d = hp (Algorithm 1 and the classical learner (D() = he B
known up to a linear transform, explicitly
R(44) = R(de)) Ei, [Var
hp)]
(4)
Proof sketch. First, we show that the predictions made by generalized LuPTS are invariant to a linear transform B applied to (-)) during training, see Appendix A Then we consider two cases: (i) When
< m we may re-purpose the proof of Theorem in Karlsson et al. (022)) (ii) When m < Proposition below directly implies x,,,, (her)())(Xx,) he)] 0 and R( R(dc). O
Theorem implies that the privileged learner is at least as sample efficient as the classical one since Var(-) 2 0 and thus R(c4)) < R((e)) for the same number of training samples m. The result is direct generalization of the main result in Karlsson et al. (022)) We also observe that generalized LuPTS and the classical learner coincide under certain conditions when m S d.
Proposition 1. Let + X + 2 C TRd be any map with corresponding kernel K. Let K, be the Gram matrix of k applied to X, and he, hy be classical B and privileged (Algorithm 7 estimates. Then,
K, is invertible for all = h hh==1 hc-
K2 is noninvertible whenever m > assuming linearly independent features.
Proof sketch. When the Gram matrices K, are invertible, the pseudo- inverse coincides with the inverse, and factors K,K, in the LuPTS estimators cancel, making LuPTS and CL equal.
Li
Remarks. Theorem extends the applicability of LuPTS to a) nonlinear prediction through fixed feature map or b) kernel estimation and e) the underdetermined case of m < d. While previous work is restricted to observed linear systems our result considers the case of latent linear dynamics which only need be identified up to a linear transform (see Figure T. Proposition that no preferable privileged learner exists; it is statement only about generalized LuPTS. We may relate the result to the double descent characteristic previously observed for other linear estimators for fixed m and varying d (Loog al. 2020). After a phase transition around å m LuPTS's variance reduces for second time when it becomes equivalent to the classical learner (see Figure 2
M [does not claim

---

3.2 Random feature maps for unknown representations
When the true & is entirely unknown, as is often the case in practice, using a poor representation & may yield biased results for both classical and privileged learners. common solution in non- linear prediction is to use universal kernel, such as the Gaussian-RBF kernel. These have dense reproducing-ken kernel Hilbert spaces which allow approximation of any continuous function. However, universal kernels also have positive-definite and thus invertible Gram matrices (Hofmann et al., 2008). which according to Proposition |
eliiminates any gain in sample efficiency of generalized LuPTS.
Instead, we combine our algorithm with an approximation of universal kernels-randon feature maps. These methods project inputs x onto d features by random linear map W Rx and nonlinear element-wiss activation function. By choosing d < m,, we can benefit from the function approximation properties of universal kernels (see discussion below) and the variance reduction of generalized LuPTS. Popular random features include random Fourier features (RFF) (Rahimi and Recht 2007) and random ReLU features (RRF) (Sun et al. 2018
FFF ()) = 22/d cos( v J + b)] and & (RF(()) = (x;;
where (z) = m0 2) is the rectifier (ReLU) function and > 0 is a bandwidth hyperparameter.
For large enough numbers of random features and training samples, any continuous function h can be approximated up to arbitrary precision by a linear map w applied to the random features, e.g.. (xx) = w T (z), see Sun et al. 2019:; y; Rudi and Rosasco 2017). Applying the same argument to the step-wise estimators of LuPTS we can justity using random features in Algorithm I by the following observation: Under appropriate assumptions, we can construct a privileged learner using random feature maps which is universally consistent estimator of hz11) E|Y [x1]. The precise construction deviates somewhat from Algorithm [ but follows the same structure. We refer to Appendix B for a precise statement. As universal consistency describes the asymptotic behaviour in the limit of infinite samples and random features, this offers only limited insight into the benefits of privileged information in small sample settings, where performance will be a bias variance trade-off.
Variance reduction & bias amplification. Generalized LuPTS is only guaranteed lower variance compared to the classical estimator under Assumption | although our empirical results (Section 4 suggest this applies more widely. When & is bad approximation of t generalized LuPTS may amplify bias, increasing with the number of privileged time points, compared to classical learning. We show this theoretically in Appendix C [and also empirically in Appendix F Whether generalized LuPTS is still preferable to classical learning in terms of prediction risk appears to depend on the amount of bias that gets amplified. Our experiments imply the variance reduction mostly dominates when using random features, whereas this is not always the case for linear LuPTS. The phenomenon of bias amplification is familiar from e.g., model-based and model-free reinforcement learning (Kober et al. ))13). As the bias with random features may still be high for small sample settings, we next present privileged representation learning algorithms to trade off bias and variance more efficiently.
3.3 Privileged time-series representation learning
Up until now the representation & was considered fixed, either because & was known up linear transform (cxplicitly or implicitly) or because of the use of random feature methods. Generalized LuPTS (Algorithm
pp produces minimizers {A.). 3 of the following objective for fixed d
LSRL (Φ, (At))
A ((x,)) _
Σ
Σ
((2. )12 1 $1 (( (1
vel12] (5)
NT NT
Objective 5 and the systems described by Assumption lend themselves to methods which also learn the representation & in addition to the latent dynamics (A,, 3}. Next, we present three algorithms which combine the ideas of generalized LuPTS with the expressiveness of deep representation learning. All learners use equivalent encoders to represent &(-) and linear layers to model the relations between the latent variables { and the outcome Y. The classical learner predicts the outcome linearly from Z,. All architectures under consideration are visualized jointly in Figure 3a
For RFF, [Ww]a No 1). and for RRF [Wu]ss U((1,, 1) and by (4(022m)
5

---

8
B
Privileged
Rap, Leamers
Classic Rep. Learner
6
GRL
CRI
(a) Classical (left) and privileged (right) representation learners. & shared across all time steps. GRL models the direct maps 0, to the outcome; SRL models the single steps A, and B. CRL combines the two.
(b) Five example sequences
from Clocks-LGS image data.
Figure 3: Representation learning architectures (left) and samples from Clocks-LGS (right).
The first privileged representation learner directly optimizes objective j.sst like generalized LuPTS, but now also fitting the representation 4 parameterized by neural network. We refer to this model as stepwise representation learner (SRL). As we will see in experiments, drawback of this approach is that representations may favor predicting transitions &,, - 21,++1 with small error, while losing information relevant for the target outcome in the process. test time, for new input ø1, SRI. composes the stepwise dynamics to output hp(x1) = 3T A7 AJ ((11).
SRL.
CRL and GRI. To make sure that the learned representation & retains information about Y, we add linear outcome supervision to the representation ((x)) at each time step t. Recall that, by Assumption [ the expected outcome is linear in the latent state any time step. We introduce hyperparameter À E [0, 1] to trade off the two types of losses and arrive at the combined representation learner (CRL). With (Ø) {A,) {0.}) the entire parameter vector, CRL minimizes the objective
0, ((r,)) -vll2 LeRi (a) = NTq
A
1-X
Σ
A ((a.)) (( (6)
N(T 1)d 8,
8,8
We make test-time predictions using (x1) 01 (x1). In experiments, we highlight the case = I where only outcome supervision is used as greedy representation learner (GRL). For a precise definition of the GRL objective, see Appendix D GRL related multi-view learning, in which prediction of the same quantity is made from multiple "views" (cf. time points) (Zhao et al. 2017)
4 Experiments
We compare classical learning to variants of generalized LuPTS (Algorithm D and the privileged representation learners of Section 5 on two synthetic and two real-world time-series data sets. We (i) verify our theoretical findings by analyzing the sample efficiency and bias-variance characteristics of the given algorithms; (ii) demonstrate that generalized LuPTS with random features succeeds in settings where linear LuPTS suffers from large bias; (iii) point out that privileged representation learners offer even greater sample efficiency in practice and (iv) study how well these algorithms recover the true latent variables [Zr} and how this relates to predictive accuracy.
Experimental setup. We report the mean coefficient of determination (R²), proportional to the squared-error risk R2 for varying sample sizes, sequence lengths and prediction horizons. Experiments are repeated and averaged over different random seeds. In each repetition, a given model performs hyperparameter tuning on the training data using random search and five-fold being re- re-trained on all training data. The test set size is 1000 samples for synthetic data and 20% of all data available for real-world data sets. We consider six privileged learners of two groups. The first group comprises generalized LuPTS with the linear kernel (LuPTS) and the two random feature maps shown in Section 3.2 Random Fourier features (Fourier RF) and random ReLU features (ReLU RF). The classical learners for this group are OLS estimators used with the same kernel or feature map. The second group consists of the representation learners SRL, CRL and GRL. For tabular data, their encoder is a multi-layer perceptron with three hidden layers of 25 neurons each. For the image data they use LeNet-5 (LeCun et al. 1989). The classical learner (Classic Rep.) uses the same encoder with linear output layer. The results presented were found to be robust to small changes in training
validatiin before

---

LaPTS (Liner))
RaLURR
LaPTS
Fourier
LaPTS RF
0.8-
0.7
0.6
0.6
0.5
0.5
0.4
0.6
0.3
0.4
0.5-
0.2
0.3
0.1
0.4-
0.2
0.0
0.3 0
0.1
-0.1
500
1000
1500
500
1000
1500
100
200
300
400
Sample Size,
Sample Size,
Sample Size,
(a) Traffic, T= 3.
(b) Square-Sign, T 7= 5, = 10.
(c) ADNI, = 4.
Figure 4: Predictive accuracy of generalized LuPTS on three data sets, over 60 repetitions. The shaded area represents one standard deviation above and below the mean over repetitions.
parameters such as learning rate. For details the training process we refer to Appendix All experiments required less than 3000 GPU-h to complete using NVIDIA Tesla T4 GPUs
Data sets. We briefly describe evaluation data sets and refer to Appendix for further details. First, create two synthetic data sets in which latent states and outcomes are generated from linear- Gaussian systems as in Assumption [ To produce the observations {Xe} we use deterministic nonlinear function V/ Z X. In the first synthetic data set, which call Square-Sign. the nonlinear transformation Vr _ + 2d maps each latent feature Z(e,k) to a two dimensional vector such that
X, = x (Z = 12 (gn(Z Li.,,y sgn (Z().)
The second synthetic data set uses the same latent linear system with Z. R² and produces square images x2822 pixels) as observations. As the images are reminiscent of clocks we refer to this data Clocks-LGS. Example sequences of these observations angle, size and fill of the two clock hands encode the value of the corresponding latent variable. The outcome Y is linear function of Z1 with 1. For Square-Sign,
presented in Figure 3b The
3.
The Metro Interstate traffic volume data set (Traffic) (Hogue 2012) contains hourly records of traffic volume on the interstate 94 between Minneapolis and St. Paul, MN. In addition, the data contains weather features and a holiday indication. We predict the traffic volume for fixed time horizon given the present observations. Privileged information is observed every four hours.
We also predict the progression of Alzheimer's disease (AD) as measured by the outcome of the Mini Mental State Examination (MMSE) (Galea and Woodward 2005). The anonymized data were obtained through the Alzheimer's Disease Neuroimaging Initiative (ADNI) (ADNI, 2004). The outcome of interest is the MMSE score 48 months after the first examination. Privileged information are the measurements 12, 24 and 36 months. In addition we tested our algorithms on the PM25 data set (Lingg et al. 2016), where we predict the air quality in five Chinese cities (see Appendix
4.1 Sample efficiency, bias and variance
The main goal of our work is to improve learning efficiency by incorporating privileged time-series information. Across almost all prediction tasks and sample sizes, nonlinear variants of generalized LuPTS outperform their classical counterpart in terms of sample efficiency as be On the Traffic prediction task, LuPTSS ReLU RF outperforms linear LuPTS as the former appears to exhibit less bias. On the synthetic data of Square-Sign, linear models reach their best accuracy quickly, while they are limited by their lack of expressiveness. Generalized LuPTS amplifies this bias, making linear LuPTS worse than OLS. Random feature methods attain higher accuracy here but are generally less sample efficient. Generalized LuPTS combined with random features manages to decrease this gap significantly. On ADNI, we don't see benefit of using nonlinear models in general. We make similar observations on the PM25 air quality data set, see Appendix
Figure 4
"Cod reproduce all results is available https://gitubb.oo///ellttttt/
7

---

Clossic Kep.
SRL
GRL
CRL
Generalized Distiliation
LaPTS ReLU RF (best resuit)
X
0.9-
0.7
0.8
0.6
0.7
2 0.7-
2 0.5
0.6-
0.4
0.6
0.3-
0.5
500 Sample Size,
1500
1000
500
1500 2500
Sumple Size,
Sample Size, m
(a) Traffic, T
(b) Square-Sign. 5, 10.
(c) Clocks-LGS, T 6,
Figure 5: Predictive accuracy of the representation learners over 25 repetitions. Generalized distilla- tion is used as proposed by Hayashi et al. (2019). For details, see Appendix
30
30
Blas?
20
Bins'
20
LaFTS (Lincur)
Tourier
10
10
LaPTS Fouriar
OLS ReLU RP
15-
10
LaPTS ReLU RF
Variance
Variance
Classic Rep.
10
SRI.
GRL
CRL
0
200
400
600
800
1000
1200
1400
200
400
600
800
1000
1200 1400
Sample Size,
Sample Size,
Figure 6: Estimated squared bias and variance of models trained on Square- (T = 5, d = 10, q
over random training sets (60 for left group, 25 for the right), evaluate on 1000 test points.
The representation learners proposed in Section 3.3 are evaluated on the same data sets and on the image prediction task Clocks-LGS. Example results displayed in Figure 5 These demonstrate that directly transferring the LuPTS objective to neural networks in the form of SRL results in subpar performance. SRL does not seem to have strong enough incentive learn representations which accurately predict the outcome. Generalized distillation for privileged time series as suggested by Hayashi al. (2019), does not improve upon classical learning in our tasks. On all experiments displayed in Figure 5. CRL and GRL outperform the classical learner. The predictive accuracy of these models is similar on most tasks, as may be explained by the fact that CRL may reduce to GRL when choosing A = 1 in objective 6 Noticeably. the general observation of GRL and CRL being more sample efficient than the classic model, neither appears depend on the neural architecture used for the encoder, nor does the modality of the data play an important role, as the image prediction task Clocks-LGS (Figure Bch demonstrates. For additional empirical results, we refer to Appendix
To analyze the bias and variance characteristics of our algorithms, we can estimate the expected by computing Ey [X]] in synthetic
squared prediction bias, Ex, (X)) E[Y| DGPs in addition to the variance of the different estimators. Figure 6 depicts bias and variance for all models on the Square-Sign data. On the left panel, all variants of generalized LuPTS exhibit lower variance than classical learning, despite being biased. This holds generally: Across all experiments, we never encountered an example where the use of privileged information has not resulted in lower variance compared to classical learning. On the contrary, the privileged learners suffer higher bias than the comparable classical learners because of the bias compounding over the individual prediction steps as shown in Appendix C For the random feature variants however, the bias decreases with the number of samples. The representation learners display similar characteristics on the right panel of Figure 6 Learning the transitions between latent variables Z, appears to be associated with low variance and high bias as demonstrated by the results of SRL. GRL however, which does not model these transitions, exhibits the lowest bias and the largest variance of all privileged learners. As CRL able to trade off between these two objectives, the estimates for its variance and bias lie in between the corresponding values of the other two privileged learners.
8

---

0.80
Cande Reps
2500
Classic Rep.
CRL
SRL
SVCCA Caefficont
0.75-
4
0.70
Latent Compument
2-
2
0.65
0.60
1000
0.55
750
500
0.50
#700
200
300
500
500
-
0.45
0.60
0.65
0.70
0.75
0.80
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
0.0
2.5
5.0
R²
Latent Component
I Latent Component
(a) Mcan H and SVVCA coefficient over
(b) Visualizing the representations learned by Classic Rep. and CRL after applying SVCCA. The recovery target is shown in the top left corner. Both estimators are best in class and trained on 2500 samples.
25 training runs. Markers represent average
results over repeated experiments with size
indicating the sample size,
Figure 7: Analyzing the representations learned by the models of Section 3.3 using SVVCA on the Clocks-LGS image regression data set (same setup as in Figure 5ch.
4.2 Latent variable recovery
Under Assumption D is sufficient to identify the representation function up to a linear transform B to have provable gains from privileged information over a classical learner. In synthetic data, we can assess to what extent a representation & with this property has been found. As a proxy for the
existence of such transform, we can compute a measure correlation between ((X) and (XX), such as the Canonical Correlation Analysis (CCA) (Hotelling, 1936). Raghu et al. (217) introduced a modified version called Singular Vector Canonical Correlation Analysis (SVCCA) to compute correlations when dealing with noisy dimensions in neural network representations.
The mean SVCCA coefficients f and predictive accuracy of the representation learners are visualized One notices that GRL and CRL produce higher correlation coefficients than the classical learner while also predicting the outcome on the Clocks-LGS task more accurately. Further comparing the representations learned by privileged and classical models, Figure 7b shows a visual example the improved latent recovery of CRL on the same task. Concluding, the use of privileged time- series information does not only increase the sample efficiency of existing algorithms but can also aid the recovery of latent variables in latent dynamical systems.
in Figure 7a
Related work
Existing analyses for learning with privileged information guarantee improvements in asymptotic sample efficiency under strong assumptions (Vapnik and Vashist 2009) but are insufficient establish a clear preference for LuPI learners for tixed sample size. For example, |Pechyony and Vapnik (010) showed that for a specialized problem construction, empirical risk minimization (ERM) using privileged information can achieve fast learning rates, (11/n), while classical (non- (non-privileged ERM can only achieve slow rates, O ((1/vn). We are not aware of any generalization theory tight enough to establish a lower bound on the risk of classical learner larger than upper bound for PI learner. Our problem also related to multi-task (representation) learning (Maurer et al. 2016), see especially ( 6o. However, our goal is different in that only single task is of interest after learning.
In estimation of causal effects, learning from surrogate outcomes (Prentice 1989) has been proposed as a way to increase sample efficiency. Surrogates are variables related to the outcome which may be available even when the outcome not (Athey et al. 2019). We can view these as privileged information. While the problem shares structure with ours, the goal is to compensate for missing outcomes and analytical results give no guarantees for improved efficiency when both surrogates outcomes are always observed (Chen et al., 2008 Kallus and Mao 2020). /uoo and Perkovic (2022) showed that, the context of Sinear-Gaussee least-squares estimator is the asymptotically most efficient estimator of causal effects using only the sample covariance. In the linear case on path graph, their estimator coincides with ours. However, no analysis is provided for the nonlinear case or for fixed sample sizes.
system on a directed acyclic graph, a recursive

---

Discussion
We have presented learning algorithms for predicting nonlinear outcomes by utilizing time-series privileged information during training. We prove that our estimator preferable classical learning when data is generated from a latent-variable dynamical system with partially known components. The proof holds for the case when the latent dynamical system is recovered by the representation function up to a linear transform, assuming that a left inverse of the true observation generating function exists. However, this assumption does not appear to be necessary, as our empirical results demonstrate that privileged learning is preferable to classical learning even when these conditions cannot be guaranteed. Consequently, a more general theoretical result where less is known about the latent system seems attainable. For example, one might consider case in which only a few independent components of the latent variables are recovered by the representation used by the learning algorithm, while other components are treated as noise.
When the latent dynamical system is entirely unknown, we create practical estimators using random feature embeddings which outperform the corresponding classical learner across experiments. We show that a universally consistent learner can be constructed based on this idea, with slightly different form. As further alternative, we propose representation learning methods of related form using neural networks and demonstrate the empirical benefits also of this estimator over classical learning. In experiments, we analyze how the gap in risk between privileged and classical learning changes for different prediction horizons, as displayed in Figure [6 in Appendix I The results suggest that the risk advantage of privileged learners grows with the sequence length of the prediction task, despite the fact that the task becomes more difficult at the same time.
Our work focuses on the setting where data is observed as time series. This setting is chosen for its simple causal structure given by (latent) from many different applications. However, the ideas presented are not specifically tied to time and also apply the case when all variables are observed simultaneously as long as the causal structure remains sequential. Moreover, we believe that the theory presented here is not limited to sequential settings and generalizes to other causal structures, in particular directed acyelic graphs that connect the baseline covariates to the outcome. In either case, one might only have access to the baseline variable at test time. For example, the timely collection data for all covariates at test time might be very expensive or even impossible.
systems and because can be motivated
As pointed out before, Theorem | requires the recovery of the latent variables up linear transfor- mation. Nonlinear independent component analysis (ICA) (Hyvarinen and Morioka 2016) aims to solve precisely this problem and has been applied to time series via time-contrastive learning. This makes for an interesting connection between learning using privileged information and nonlinear ICA, as the experiments of Section 4.2 suggest that privileged time-series information aids the recovery of latent variables. Other remaining challenges include providing risk guarantees for learning with biased representations (including deep neural networks), with regularized estimators, and for more general data generating processes with weaker structural assumptions. We are hopeful that the utility demonstrated in this work will inspire future research to overcome these limitations.
Acknowledgments and Disclosure of Funding
We would like to thank Anton Matsson and Rickard Karlsson for insightful feedback and the Alzheimer's 's. Neuroimaging Initiative (ADNI) for collecting and providing the data on Alzheimer's s disease used in this project. The present work was funded part by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
References
ADNI. the Alzheimer's Disease Neuroimaging Initiative (ADNI), 2004. URL http:://dnii.... i 1oni edul
Susan Athey, Raj Chetty, Guido W Imbens, and Hyunseung Kang. The surrogate index: Combining short-term proxies to estimate long-term treatment effects more rapidly and precisely. Technical report, National Bureau of Economic Research, 2019.
10

---

Song Xi Chen, Denis HY Leung, and Jing Qin. Improving semiparametric estimation by using surrogate data, Journal of the Royal Statistical Society: Series B (Statistical Methodology). 70(4): 803-823, 2008.
Dheeru Dua and Casey Graff. UCI machine learning repository, 2017. URL http://archive.ics uuiiiduuu/// edu/ml
Mary Galea and Michael Woodward. Mini-mental state examination (mmse). Australian Journal of Physiotherapy, 51(3):198, 2005, ISSN 0004-9514, doi: http:///ci.ogg//0000////00404540 70034-9.
F Richard Guo and Emilija Perković. Efficient least squares for estimating total effects under linearity and causal sufficiency. Journal of Machine Learning Research, (3(14)):1-41, 2022.
Shogo Hayashi, Akira Tanimoto, and Hisashi Kashima. Long-term prediction of small time-series data using generalized distillation. In 2019 International Joint Conference Neural Networks (IJCNN), pages 1-8. IEEE, 2019.
Thomas Hofmann, Bernhard Schölkopf. and Alexander Smola. Kernel methods in machine learning. The Annals of Statisticss 36(3), jun 2008.
John Hogue Metro interstate traffic volume data set, 2012. URL https://archive.ios uci / http://aatttteeettooooeeteteteee
Harold Hotelling. Relations between Sets of Variates* Biometrika, 28(3-):321-377, 12 1936.
ISSN 0006-3444. doi: 10. (0.1093/biomet28.33333333
Bingquan Huang, Mohand Tahar Kechadi, and Brian Buckley. Customer churn prediction in telecom- munications. Expert Systems with Applications, 39(1):1414-1425 2012.
Aapo Hyvarinen and Hiroshi Morioka. Unsupervised feature extraction by time-contrastive learning and nonlinear ica. Advances in Neural Information Processing Systems, 29, 2016.
Rico Jonschkowski, Sebastian Höfer, and Oliver Brock. Patterns for learning with side information. arXiv preprint arXiv:1511.06422 2015.
Nathan Kallus and Xiaojie Mao. On the role of surrogates in the efficient estimation of treatment effects with limited outcome data. arXiv preprint arXiv:2003. 2020.
Jonathan C. Kao, Paul Nuyujukian, Stephen I. Ryu, Mark M. Churchland, John P. Cunningham, and Krishna V. Shenoy. Single-trial dynamics of motor cortex and their applications to brain-machine interfaces. Nature Communications, 6(1), July 2015. doi: 10. 10.10//commm88759)
Aditya V Karhade, Quirina CBS Thio, Paul T Ogink, Christopher M Bono, Marco L Ferrone, Kevin S Oh, Philip Saylor, Andrew J Schoenfeld, John H Shin, Mitchel Harris, et al. Predicting 90-day and 1-year mortality in spinal metastatic disease: development and internal validation. Neurosurgery, (5(4):B671 -1681, 2019.
Rickard Karlsson, Martin Willbo, Zeshan Hussain, Rahul G Krishnan, David Sontag, and Fredrik D Johansson. Using time-series privileged information for provably efficient learning of prediction models. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics., 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Jens Kober, J. Andrew Bagnell, and Jan Peters. Reinforcement learning in robotics: A survey. Int. J. Rob. Res., 3)(11)::22 12388111112 sep 2013. ISSN 0278-3649. doi: 10. 10.1177/02783664914497777
Y. LeCun, B. Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation Applied to Handwritten Zip Code Recognition. Neural Computation, 1(4): 541-551, 12 1989. ISSN 0899-7667. doi: 10.
Kookjin Lee and Kevin Carlberg. Deep conservation: latent-dynamics model for exact satisfaction of physical conservation laws, 2019.

---

Xuan Liang, Shuo Li, Shuyi Zhang, Hui Huang, and Song Xi Chen. Pm2.5 data reliability, consistency, and air quality assessment five chinese cities. Journal of Geophysical Research: Amossperes. 121(17):00220-002 September 2016. ISSN 2169-897X. doi: 10. 00.1002/2011JD02467777
Marco Loog, Tom Viering, Alexander Mey, Jesse H. Krijthe, and David M. J. Tax. A brief prehistory of double descent. Proceedings of the National Academy of Sciences, 117(20):10625 100000000100---6666..000 2020. doi: 10. 10..077/pnaa.201185
D. Lopez-Paz, B. Schölkopf, L. Bottou, and V. Vapnik. Unitying distillation and privileged informa- tion. In International Conference on Learning Representations (ICLR), November 2016.
Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes, The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1-32, 2016.
Bobak J Mortazavi, Nicholas S Downing, Emily M Bucholz, Kumar Dharmarajan, Ajay Manhapra, Shu-Xia Li, Sahand N Negahban, and Harlan M Krumholz. Analysis of machine learning techniques for heart failure readmissions. Circulation: Carliovascular Quality and Oulcomes, (6):629-640, 2016.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035.
Curran Associates, Inc., 2019.
Dmitry Pechyony and Vladimir Vapnik. On the theory of learnining with privileged information. Advances in neural information processing systems, 23, 2010.
F. Pedregosa. G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Pretten- hofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Roger Penrose. On best approximate solutions of linear matrix equations. In Mathematical Pro- ceedings of the Cambridge Philosophical Society, volume 52, pages 17-19. Cambridge University Press, 1956.
Ross L Prentice. Surrogate endpoints in clinical trials: definition and operational criteria. Statistics in medicine, 844):431-440, 1989.
Maithra Raghu, Justin Gilmer, Jason Yosinski, and Jascha Sohl-Dickstein. Sveca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability, 2017.
Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20, Curran Associates, Inc., 2007.
David S Rosenberg and Peter L Bartlett.
The rademacher complexity of co co-regularized kernel classes. In Artificial Intelligence and Statisties, pages 496-403. PMLR. 2007.
Alessandro Rudi and Lorenzo Rosasco. Generalization properties of learning with random features. Advances in neural information processing systems, 30, 2017.
Alex J Smola and Bernhard Schölkopf. Learning with kernels, volume 4, Citeseer, 1998.
Sun, A Gilbert, and A Tewari. On the approximation capabilities of relu neural networks and random relu features. arXiv preprint 11Xiv:1810.04374.22 2019.
Yitong Sun, Anna Gilbert, and Ambuj Tewari. On the approximation properties of random relu features, 2018
Vladimir Vapnik and Akshay Vashist. A new learning paradigm: Learning using privileged informa- tion. Neural networks. 22(5-6):544-5577 2009.
12

---

Vladimir Vapnik, Rauf Izmailov, et al. Learning using privileged information: similarity control and knowledge transfer. J. Mach. Learn. Res., 16(1):2023- 2049, 2015.
Weiran Wang. Everything old is new again: A multi-view learning approach to learning using privileged information and distillation. arXiv preprint W9Xiv::903.03694 2019.
Jiaxuan You, Xiaocheng Li, Melvin Low, David Lobell, and Stefano Ermon. Deep gaussian process for crop yield prediction based on remote sensing data. In Thirty- AAAI conference on artificial intelligence, 2017.
Jing Zhao, Xijiong Xie, Xin Xu, and Shiliang Sun. Multi-view learning overview: Recent progress and new challenges. Information Fusion, 38:43-54 2017. ISSN 1566-2535. doi: https://doi.org/ 10.10 10.1016/j. wttp://iifffus/// 2017.ff0...07..200000..
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] The introduction refers to the section where each contribution is made.
(b) Did you describe the limitations of your work? [Yes] See c.g., the discussion in Section
g and on the applicability of Theorem iin Section 3.] and 3.2]
(c) Did you discuss any potential negative societal impacts of your work? [N/A] We have not identified any potential negative societal impact directly related to our work.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Assump- tion and Theorem [1
(b) Did you include complete proofs of all theoretical results? [Yes] Yes, see the Appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [Yes] A URL to code repository is included in the paper.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section D and Appendix F
(c) Did you report error bars (e.g., with respect the random seed after running experi-
ments multiple times)? [Yes] Each result figure includes error regions.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section +
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] See Appendix E.2
(b) Did you mention the license of the assets? [Yes] See Appendix 32
(c) Did you include any assets either in the supplemental material URL? [Yes]
Only original code.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(c) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] ADNI data is anonymized.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "zqBrSbkUnWy", "reviewer": "Reviewer_gvbk", "summary": "The paper deals with the data generated by the following model.\n\nA vector $z_1$ comes from some distribution. Then it is subjected to a chain of linear transformations. The vector $z_2$ is obtained as a linear transformation of $z_1$ (plus some noise), $z_3$ is a linear transformation of $z_2$ etc. Finally we get to $z_T$. Then $z_T$ generates a label $y$ by a linear transformation of $a$ (finite dimensional) feature mapping.\n\nThe vectors $z_t$ are never visible directly. There are two modes of access to them. In the first, \"classical\" scenario we get to see $x_1 = \\Psi (x_1)$ where $\\Psi$ is a linear transformation. In the second, \"privileged\" scenario we get access to $x_2 = \\Psi(z_2), x_3 = \\Psi(z_3)$ etc.\n\nWe can have either classical or privileged information at the training stage, but only get classical information at the test stage. The paper compares two learning approaches.\n\nIn the first, classical approach we use Least Squares to regress y on a feature mapping of $x_1$. In the second, privileged approach we try reconstruct the whole model using least squares on the training set to find all transition linear transformations etc and get $y$. The paper shows that the variance of the later approach is lower than the variance of the former.\n\nThe advantage disappears if the feature mapping is infinite-dimensional and we get non-singular Gram matrices on the data.\n\nThe experimental results include reconstruction of the feature mapping using neural networks.", "questions": "None. The paper is clearly written.", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I think this is an interesting result but its significance is somewhat limited. The authors very honestly show the key limitation for universal kernels. The result is certainly important for the dynamic systems community but I have doubts about wider ML audience. The very model appears limited (although it has important applications in neurophysiology).\n\nTypos/suggestions\n\nPage 3, line 93 on the form -> of the form ?\n\nI also find the title somewhat misleading. This is not what is usually meant by a time series. \"Dynamic systems\" would be better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fXqVGJtGmYM", "reviewer": "Reviewer_s32b", "summary": "This paper proposes new insights into the analysis of Learning using privileged information  (LuPI) learner with access to intermediate time series data and  generalizes it to nonlinear prediction tasks in latent dynamical systems. The authors extend theoretical guarantees to the case where the map connecting latent variables and observations is known up to a linear transform. In addition, the paper proposes algorithms based on random features, and representation learning for the case when the map is unknown. ", "questions": "What is the benefit of the approach for prediction if it is not experimented for long term prediction?\n\nHow would the approach fare for prediction using different prediction horizons?\n\nBeyond bias/variance analysis, what is the significance of this approach?\n\nWhich kind of non-linearity makes the proposed approach non-longer valid?\n\nWhat is the algorithmic complexity for its implementation and use in practical settings?", "limitations": "It would be good if the authors could discuss the impact of their bias/variance analysis on societal applications?", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written and theoretically sounds. The paper is clear easy to follow and bring to the fore an interesting contribution.\n\nParticularly, the fact that the authors extend the LuPTS framework to nonlinear models and prediction tasks in latent dynamical systems is an important step. \n\nFurthermore, the authors prove that learning with privileged information leads to lower risk when the nonlinear map connecting latent variables and observations is known up to a linear transform. This is also an important contribution.\n\nThe bias/variance analysis is also timely and important. Also, the experimentation with real data namely the prediction of traffic volume and Alzheimer progression is critical to validating the approach.\n\nHowever, the fact that the study is limited to predictions for a fixed time horizon given present observations limit the extend to which the approach could be validated and applied.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "VOuBf7pIWU8", "reviewer": "Reviewer_LQNq", "summary": "The manuscript proposes to predict nonlinear outputs using time series privileged information during training. The privileged information is not used for inference. The authors prove in Theorem 1 that under some conditions, \"the generalized LuPTS is never worse in expectation than the classical learner\". Experiments on both synthetic and real-world data sets show that using privileged information increases the sample efficiency and helps latent variable recovery. ", "questions": "* Stochastic process\n\nIf I understand correctly, the latent states $Z_t$ are assumed to be from a stochastic process, is it?\nIt is yet not clear from the main paper whether $Z_t$ is stationary or not. \nI see some conditions of matrix $A$ in Appendix (line 659), but I did not find the constraints of generating $A$. That is, it is not clear why $A$ is designed to be like that. \n\n* OLS Linear estimator \n\nWhat is the difference between OLS (line 100) and the linear regression?\n$\\hat{\\theta}_C$ in eq(3) seems to be exactly the same as linear regression.\n\n* $R^2$\n\nWhat is $R^2$ in Section 4?\n\n* Efficiency\n\nThe manuscript claims in line 237 that the main goal is to improve learning efficiency. \nHow \"learning efficiency\" is defined and estimated in the manuscript?\n\n\n\n", "limitations": "The authors addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Originality\n\nFrom the title, it is not clear how much this manuscript differs from [1] Karlsson et al. (2021) (https://proceedings.mlr.press/v151/k-a-karlsson22a/k-a-karlsson22a.pdf)\nWhile in the main text, the manuscript tackles the task by modeling with latent variables for each observation, which is a major difference.\nIf I understand correctly, Theorem 1 in the manuscript is similar to Theorem 1 in [1] with some changes of conditions.\nThe manuscript should better formulate the difference from [1] \n\n* Quality\n\nThe manuscript is technically sound. \n\n* Clarity\n\nThe manuscript is overall clearly written. Some (minor) suggestions are as follows.\n\n1. \"**Unobserved**\" applies to Figure 1 (b) but not Figure 1 (a).\n2. It is not clear what random variables the \"distribution $p$\" models in line 62.\n3. \"the matrix inverse $(\\cdot)^{-1}$\" in line 104 can not be found in the text. \n4. Figure 3 (b) is not informative enough. \n5. It is not clear what the metric in experimental evaluation is used. What is $R^2$ in Section 4?\n\n\n* Significance\n\nThe algorithm proposed in the manuscript would benefit the time series modeling community\n\n---\n\n[1] Karlsson, Rickard KA, et al. \"Using time-series privileged information for provably efficient learning of prediction models.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3orR65qT7DJ", "reviewer": "Reviewer_oWQK", "summary": "Through both rigorous theoretical analysis and empirical findings, this paper extends results on Learning using Privileged Information to general time series governed by linear latent dynamics and potentially nonlinear observation models. Under this general class of possible models, the authors demonstrate through learning theoretic analysis that learning with privileged (data that is available at training time but not at inference time) time series always leads to lower or equivalent risk, given that the nonlinear map from observations to latent variables is known up to a linear mapping. They then extend their results to the unknown mappings via an argument based on the use of random feature maps. Finally, the authors propose to approximate unknown mappings with deep learning architectures. In their empirical study, they demonstrate that the predicted learners with privileged information generally outperform baselines in terms of sample efficiency.", "questions": "- Sample efficiency is qualitative and subjective. In Fig 4(b) OLS seems to be outperforming all baselines in terms of \"sample-efficiency\" in contradiction to the text. So what precisely is meant by sample efficiency here? Could that be made more rigorous?\n- Figure 7(a) is not readable, and the conclusion is mostly not accessible. Could you please revise and explain?\n- Will the experiments be made available and reproducible? There is no mention of this in the paper.", "limitations": "- I would recommend to spend much more time on the motivation, and amplify this in the experiments. For example, how does this setting connect to images (which is presented in the experiments)? \n- I would recommend to make the findings around L151 more precise instead of deferring to the appendix as I believe this is one of the key findings in the paper.\n\nMinor point:\n- L66 typo in \"respect\"", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper benefits both from strong theoretical analysis and rigorous empirical set-up. The results presented are relevant, and significantly expand understanding of learning using privileged time series (LuPTS) information. To the best of my knowledge, the paper proposes both novel theoretical findings and explores a set of new approaches for LuPTS. Finally, the paper is well-written, with clear and concise use of language.\n\nMy only concern about the paper is motivation. It is not clear to me, except some cases alluded to in the introduction to the paper, in which applications one would be interested in using privileged information. Unfortunately, the empirical study also does not motivate a single application but rather transforms existing time series applications such that the problem definition fits. I believe the readers would benefit greatly from a discussion of motivation and potential impact.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept