Compressed Predictive Information Coding
Authors: Rui Meng
Tianyi Luo
Kristofer Bouchard
Venue: Submitted to ICLR 2023 (2023)
Keywords: Predictive information
time series
variational inference

Abstract: Unsupervised learning plays an important role in many fields, such as machine learning, data compression, and neuroscience. Compared to static data, methods for extracting low-dimensional structure for dynamic data are lagging. We developed a novel information-theoretic framework, Compressed Predictive Information Coding (CPIC), to extract predictive latent representations from dynamic data. Predictive information quantifies the ability to predict the future of a time series from its past. CPIC selectively projects the past (input) into a low dimensional space that is predictive about the compressed data projected from the future (output). The key insight of our framework is to learn representations by balancing the minimization of compression complexity with maximization of the predictive information in the latent space. We derive tractable variational bounds of the CPIC loss by leveraging bounds on mutual information. The CPIC loss induces the latent space to capture information that is maximally predictive of the future of the data from the past. We demonstrate that introducing stochasticity in the encoder and maximizing the predictive information in latent space contributes to learning more robust latent representations. Furthermore, our variational approaches perform better in mutual information estimation compared with estimates under the Gaussian assumption commonly used. We show numerically in synthetic data that CPIC can recover dynamical systems embedded in noisy observation data with low signal-to-noise ratio. Finally, we demonstrate that CPIC extracts features more predictive of forecasting exogenous variables as well as auto-forecasting in various real datasets compared with other state-of-the-art representation learning models. Together, these results indicate that CPIC will be broadly useful for extracting low-dimensional dynamic structure from high-dimensional, noisy time-series data.

Under review as a conference paper at ICLR 2023
COMPRESSED PREDICTIVE INFORMATION CODING
Anonymous authors
Paper under double-blind review
ABSTRACT
Unsupervised learning plays an important role in many fields, such as machine learning, data compression, and neuroscience. Compared to static data, methods for extracting low-dimensional structure for dynamic data are lagging. We de- veloped a novel information-thhoreee framework, Compressed Predictive Infor- mation Coding (CPIC), to extract predictive latent representations from dynamic data. Predictive information quantifies the ability predict the future of time se- ries from its past. CPIC selectively projects the past (input) into low dimensional space that is predictive about the compressed data projected from the future (out- put). The key insight of our framework is to learn representations by balancing the minimization compression complexity with maximization of the predictive in- formation in the latent space. We derive tractable variational bounds of the CPIC loss by leveraging bounds on mutual information. The CPIC loss induces the la- tent space to capture information that is maximally predictive of the future of the data from the past. We demonstrate that introducing stochasticity in the encoder and maximizing the predictive information in latent space contributes to learn- ing more robust latent representations. Furthermore, our variational approaches perform better in mutual information estimation compared with estimates under the Gaussian assumption commonly used. We show numerically in synthetic data that CPIC can recover dynamical systems embedded in noisy observation data with low signal-to-noise ratio. Finally, we demonstrate that CPIC extracts features more predictive forecasting exogenous variables as well as auto- forecastnn in various real datasets compared with other state-of-the-arr representation learning models. Together, these results indicate that CPIC will be broadly useful for ex- tracting -ww-eemeniional dynamic structure from nigh-dimensional, noisy time- series data.
INTRODUCTION
Unsupervised methods play an important role learning representations that provide insight into data and exploit unlabeled data to improve performance in downstream tasks in diverse applica- tion areas Bengio et al. (2013); Chen et al. (2020); Grill et al. (2020): Devlin et al. (2018): Brown let al. (2020); Baevski al. (2020): Wang al. (2020). Prior work unsupervised representation learning can be broadly categorized into generative models such as variational autoencoders VAEs)
(Kingma & Welling 2013) and generative adversarial networks (GAN) (Goodfellow et al.) 2014).
discriminative models such as dynamical components analysis (DCA) (Clark et al. 2019 con- trastive predictive coding (CPC) (Oord al. 2018), and deep autoencoding predictive components (DAPC) (Bai et al., 2020). Generative models focus on capturing the joint distribution between representations and imputs, but are usually computationally expensive. On the other hand, discrim- inative models emphasize capturing the dependence of data structure in the low-dimensional latent space, and are therefore easier to scale to large datasets.
In the case of time series, some representation learning models take advantage of an estimate of mu- tual information between encoded past (input) and the future (output) (Creutzig & Sprekeler 2008 Creutzig et al. 2009 Oord et al. 2018). Although previous models utilizing mutual information ex- tract low-dimensional representations, they tend to be sensitive to noise in the observational space. DCA directly makes use of the mutual information between the past and the future (i.e., the predic- tive information (Bialek et al. 200T)) in a latent representational space that a linear embedding of the observation data. However, DCA operates under Gaussian assumptions for mutual information

---

Under review as a conference paper at ICLR 2023
estimation. We propose novel representation learning framework which is not only robust noise in the observation space but also alleviates the Gaussian assumption and is thus more flexible.
We formalize our problem in terms of data generated from stationary dynamical system and pro- pose an information-theone theorettih objective function for Compressed Predictive Information Coding (CPIC). Instead of leveraging the information bottleneck (IB) objective directly in Creutzig & Sprekeler (2008) and Creutzig et al. (2009), where the past latent representation is directly used to predict future observations, we predict the compressed future observations filtered by the encoder. It is because that in the time series setting, future observations are noisy, and treating them as labels not insightful. Specifically, our target to extract latent representation which can better predict future underlying dynamics. Since the compressed future observations are assumed to only retain the underlying dynamics, better compression thus contributes to extracting better dynamical repre- sentation. In addition, inspired by Clark et al. (2019) and |Bai et al, (2020), we extend the prediction from single input window of inputs handle high order predictive information.
Moreover, instead of directly estimating the objective information with Gaussian assumption |009|
(Creutzig Sprekeler 2008 Creutzig et al. 2009) [Clar et al. 2019 Bai et al., 2020), we developed variational bounds and tractable end-to-end training framework based on the neural estimator of mutual information studied in Poole et al. (2019). Note that our inference first leverages the varia- tional boundary technique for self--upervised learning the time series data. Since it alleviates the Gaussian assumption, it is applicable to a much larger class of dynamical systems.
In CPIC, also demonstrate that introducing stochasticity into either linear nonlinear coder robustly contributes to numerically better representations in different tasks. In particular, illustrate that CPIC can recover trajectories of chaotic dynamical system embedded in high- dimensional noisy observations with low signal signal-to-noise ratios in synthetic data. Furthermore, we conduct numerical experiments on four real-world datasets with different goals. In two neuroscience datasets, monkey motor cortex (MI) and rat dorsal hippocampus (HC), compared with the state-of- the-art methods, we show that the latent representations extracted from CPIC have better forecast- ing accuracy for the exogenous variables of the monkey's future hand position for M1, and for the rat's future position for HC. In two other real datasets, historical hourly weather temperature data (TEMP) and motion sensor data (MS), we show that latent representations extracted by CPIC have better forecasting accuracy of the future of those time series than other methods. summary, the primary contributions of our paper as follows:
We developed a novel information- theoretic self-supervised learning framework, Com- pressed Predictive Information Coding (CPIC), which extracts low-dimensional latent rep- resentation from time series. CPIC maximizes the predictive information in the latent space while minimizing the compression complexity.
We introduced the stochastic encoder structure where encode inputs into stochastic representations to handle uncertainty and contribute to better representations.
Based on prior works, we derived the variational bounds of the CPIC's objective function and tractable, end-to-end training procedure. Since our inference alleviates the Gaussian assumption common to other methods, it is applicable to much larger class of dynamical systems. Moreover, to the best of our knowledge, our inference the first to leverage the variational boundary technique for self-supervised learning time series data.
We demonstrated that, compared with the other unsupervised based methods, CPIC more robustly recovers latent dynamics in dynamical system with low signal-to-noise ratio in synthetic experiments, and extracts more predictive features for downstream tasks in vari- ous real datasets.
RELATED WORK
Mutual information (MI) plays an important role in estimating the relationship between pairs of variables. It is a reparancteriaaionnivvaaat measure of dependency:
p(zly))
T(X, Y) Ep(a.w)
log
(1)
p(x)
2

---

Under review as a conference paper at ICLR 2023
It used in computational neuroscience (Dimitrov et al. 2011 visual representation learning (Chen et al. 2020), natural language processing (Oord et al.) 2018 and bioinformatics (Lachmann et al. 2016 In representation learning, the mutual information between imputs and representations is used to quantify the quality of the representation and is also closely related to reconstruction error in gen- erative models (Kingma & Welling 2013] Makkzaan et al. 2015). Estimating mutual information is computationally and statistically challenging except in two cases: discrete data, as in Tishby et al. (2000) and Gaussian data, as in Chechik et al. (2005). However, these assumptions both severely constrain the class of learnable models (Alemi et al. (016). Recent works leverage deep learning models to obtain both differentiable and scalable MI estimation (Belghazi et al. 2018 Nguyen al.,
2010 Oord et al. 2018 Alemi et al., 2016 Poole et al. 2019 Cheng et al. 2020).
In terms of representation learning in time series, Wiskott & Sejnowski (2002); Turner Sahani 2007) targeted slowly varying features, Creutzig & Sprekeler] utilized the information bot- tleneck (IB) method (Tishby et al. 2000) and developed an nnformation-theorrtiee theoretic objective func- tion. Creutzig et al. (2009) proposed an alternative objective function based on specific state-space model. Recently, Oord et al. (2018) proposed CPC to extract dynamic information based on an autoregressive model on representations and contrastive loss on predictions. Clark et al. (2019;; Bai et al.] (020) proposed unsupervised learning approach to extract www-dmensional representa- tion with maximal predictive information(P1) (PI). All of the above unsupervised representation learning models, except for CPC, assume the data to be Gaussian, which may be not realistic, especially when applied to neuroscience datasets (O Doherty et al.) 2017; Glaser al., 2020), given the non- Gaussianity of neuronal activity. Here, we leverage recently introduced neural estimation of mutual information to construct upper bounds of the CPIC objective and develop an end-to-end training procedure. CPIC enables generalization beyond the Gaussian case and autoregressive models.
Recently, deep encoder networks are leveraged to model nonlinear relations between latent repre- sentations and observed data in time series (Chen et al., 2020 Bai et al.] 2020 He et al. 2020). However, use of complicated nonlinear encoders induced hinders computational efficiency (Wang
et al. 2019). CPIC proposes an efficient representation learning framework for time series that en- codes data with maximal predictive information. We also note that there exists several works on the time series modeling from generative modeling perspective. Initially, Fabius & Van Amersfoort (2014) leveraged the recurrent neural network with variational autoencoder to model time series Gaussian-process process state-space model. [eng et al. (2021) proposed variational structured Gaussian Gaussian-process process regression network which can efficiently handle more complicated relationships in time series. Most generative modeling inference would depend on the length of time series, while the inference of CPIC depends on the window size T, which is more scalable for long time series.
data. Frigola et al.] (2014) proposed variational Gaussian
3 COMPRESSED PREDICTIVE INFORMATION CODING
The main intuition behind Compressed Predictive Information Coding (CPIC) is to extract low di- mensional representations with minimal compression complexity and maximal dynamical structure. Specifically, CPIC first discards low-level information that is not relevant for dynamic prediction and noise that is more local by minimizing compression complexity (i.e., mutual information) be- tween inputs and representations to improve model generalization. Second, CPIC maximizes the predictive information in the latent space of compressed representations.
Compared with Clark et al. 2019 Bai al. (2020), CPIC first utilizes stochastic encoder to handle uncertainty of representations, which contributes to more robust representations, and also relieves the Gaussian assumption by constructing bounds of mutual information based on neural estimations. In more detail, instead of employing a deterministic linear mapping function as the encoder to com- press data as in Clark et al. (2019), CPIC takes advantage of a stochastic linear or nonlinear mapping function. Given inputs, the stochastic representation follows Gaussian distributions, with means and variances encoded from any neural network structure. A nonlinear CPIC utilizes a stochastic non- linear encoder which is composed of nonlinear mean encoder and linear variance encoder, while a linear CPIC utilizes a stochastic linear encoder which is composed of a linear mean encoder and linear variance encoder. Note that stochastic representations conditioned ized as a conditional Gaussian distribution, but the marginal distribution of the representation is a mixture of Gaussian distribution, which is widely recognized universal approximator of densities.
inputs are parameter-
3

---

Under review as a conference paper at ICLR 2023
On the other hand, avoiding the Gaussian assumption mutual information (Creutzig & Sprekeler 2008 Creutzig et al. [2009] Clark et al., 2019, Bai et al. 2020), CPIC leverages neural estimations of mutual information. Specifically,
propose differentiable and scalable bounds of the CPIC objective via variational inference, which enables end-to-end training.
Formally, let X [ E RN be stationary, discrete time series, and let X pust (I-T+11.... Fo) and Xfture = (Z1,...,, .T) denote consecutive past and future windows of length T. Then both past and future data are compressed into past and future representations denoted as Ypast (V-T+1,... 3/6) and Yiuure- (y),.- yT) with embedding dimension size Q. Similar to the information bottleneck (IB) (Tishby al. 2000), the CPIC objective contains trade-off between two factors. The first seeks to minimize the compression complexity and the second to maximize the predictive information in the latent (representation) space. Note that when the encoder is determin- istic the compression complexity is deprecated and when the encoder is stochastic the complexity measured by the mutual information between representations and inputs. In the CPIC objective, the trade-off weight 3 > 0 dictates the balance between the compression and predictive information terms:
min C, where = 3{I yass) + (Xtuuee: Yiunure) I(Ypast: Yfuurr))
(2)
where & refer to the model parameters which encode inputs to latent variables Y. Larger / promotes a more compact mapping and thus benefits model generalization, while smaller 3 leads to more predictive information in the latent space on training data. This objective function visualized in Figure I where inputs X are encoded into latent space Y via tractable encoders and the dynamics of Y are learned in model-free manner.
The encoder xX)] could be implemented by fitting deep neural networks (Alemi et al.) 2016) encode data X. Instead, CPIC takes an ap- proach similar to VAEs (Kingma & Welling 2013), in that it encodes data into stochastic representations. In particular, CPIC employs a stochastic encoder (gene in Figure to com- press input x; into Y/2
predictions
Y Ypast
future
Ye Ir N(p, diag(o?)).
(3)
for each time stamp f. The mean of ye is given by He = 9j. (aessoderrrz)))))). whereas the variance arises from o, g
Jenc
Jenc
Encoders Encoder and g Encoder can be any non- linear mapping and is usually modeled using neural network architectures. We use a two- layer perceptron with ReLU activation function (Agarap) 2018) for a nonlinear mapping. In terms of linear CPIC, we specify the mean of representation as f = u" xt. In both linear and nonlinear CPIC setting, if a, O. the stochas- tic encoder reduces to a deterministic encoder.
X Xpast
X future
Figure 1: The overall framework of compressed predictive information coding. The encoder com- press information of the input X into Y such that the predictive information between Yous and Yiuure is maximized while minimizing the mutual information between X and Y.
We extend single input to multiple inputs in the CPIC framework in terms of a specified win- dow size T. The selection of window size is discussed in Appendix A Due to the stationary assumption, the relation between past/future
blocks of input data X(-T),((T) E R-XTT and encoded data Y Y(-T), RΩxI are equivalent, ((--)) = PX(T),Y(T)- Note that -T and T indexes to past and future data. Without loss of generality, the compression relation can expressed as Y(T) 9f
Fincdder ( + E(T), where ((T) E AO blockdiagddiag())). diag((o))))) and noise standard g Enssddr (xz).
deviation
4

---

Under review as a conference paper at ICLR 2023
4 VARIATIONAL BOUNDS OF COMPRESSED PREDICTIVE INFORMATION CODING
In CPIC, since data X are stationary, the mutual information between the input data and the com- pressed data for the past is equivalent to that for the future ((x(=));)((7))= ((())) Therefore, the objective of CPIC can be rewritten as
min L = (T); ((T)) -
(4)
We developed the variational upper bounds on mutual information for the compression com- plexity ((T)) and lower bounds mutual information for the predictive information (()())))(()).
4.1 UPPER BOUNDS OF COMPRESSION COMPLEXITY
In the section, we derived tractable variational upper bound (VUB) depending on a single sample and leave-one-out upper bound (L10ut) (Poole et al.) 2019) depending on multiple samples.
Theorem By introducing variational approximation r(y(T)) to the marginal distribution ())(T)) a tractable variational upper bound of mutual information ((X (T)) is derived as IvuB (X (T):Y(T) (x))) (L(p(y(T))=(TT) (T)())))
Theorem By utilizing a Monte Carlo approximation for variational distribution r(y())), the LIOut upper bound of mutual information [(X YT));YYY)) is derived as I1JO (X(T;; Y (T)) p(u(T))e((T))
log 8-1 Plst)aaa ]] where is the sample size.
E
The derivation details are in Appendix B and In practice, the L1Out bound depends on the sample size S and may suffer from numerical instability. Thus, we would like to choose the sample size as large as possible. In general scenarios where p(y(T)) (xT)) is intractable, Cheng et al. (2020) proposed variational version of VUB and L1Out by using neural network to approximate the condition distribution ((v(T))x<TT)) Since the conditional distribution (=)()))(T)) is parameter- ized as known stochastic/deterministee encoder in CPIC, those variational versions not taken into consideration.
4.2 LOWER BOUNDS OF PREDICTIVE INFORMATION
For the predictive information (PI), we derived lower bounds ( using results in Agakov (2004); Alemi et al. (2016); Poole et al. (2019). In particular, we derived tractable unnor- malized Barber and Agakov (TUBA) (Barber Agakov 2003) lower bounds depending on a single
sample and an infoNCE lower bound (Oord ct al. 2018) depending multi samples. All derivation
details are discussed in Appendix
and
Theorem We derived a lower bound on predictive information (PI) I(((T)); Y(T)) as Iv (Y(T) is a variational conditional distribution.
((v(T))u(-T)) (T)())) + (U where ((y(T))y(-T))
However, this lower bound requires a tractable decoder for the conditional distribution ( (-T) (Alemi al. 2016). Alternatively we derived TUBA lower bound (Barber & Agakov 2003) which is free of the parametrization of decoder.
Theorem 4 By introducing a differentiable critic function f(x,y) v) and baseline function a(y(T)) defined Appendix E the TUBA lower bound of predictive information is derived as TTUBA (Y(-T),Y(TT) = - (())(()))))())))--))-) - where f(y(-T),
(Eru((TT)otrTT)l (efftttttttl
fvv(-T), v(T)) - log(a()(TT)).
Different forms of the baseline function lead to different neural estimators in the literature such as MINE (Belghazi et al. 2018) and NWJ (Nguyen et al. 2010). On the other hand, all TUBA based
5

---

Under review as a conference paper at ICLR 2023
estimators have high variance due the high variance of fz, v). Oord et al. 2018 proposed low-variance MI estimator based on conse-contrastive estimation called InfoNCE. Moreover, there exists other differentiable mutual information estimator including SMILE (Song & Ermon 2019)
and Echo noise estimator (Brekelmans et al. 2019).
Theorem 5 In the CPIC seiting, the InfoNCE lower bound of predictive information is derived as
InfoNCE (YRANCE(((AA)) (-T): = E
log 1
(5)
S
9 S e ThT)(T)s)
The expectation is over S independent samples from the joint distribution: p(y(-T), (()) following Markov Chain rule in Figure I [p(x(-T), (TT)-TT))()))((((T))))))
such plu((-2),b(I))
4.3 VARIATIONAL BOUNDS OF CPIC
We propose two classes of upper bounds CPIC based on whether the bounds depend on a single sample multiple samples. According the uni- uni-sample and multi-sample bounds derived in Sec- tion 1.1 and Section 4.2 we name the first class as uni-sample upper bounds, which take the VUB upper bound of mutual information for the complexity of data compression (T),),, (T)) and the TUBA as the lower bound of predictive information in equation [4 Thus we have
LUNI = (KL(p(((T)) r(v(T)) - -TBBA (X-T),
(6)
Notice that by choosing different baseline functions, the TUBA lower bound would be equivalent to different mutual information estimator such as MINE and NWJ. The second class is named as multi-sample upper bound, which take advantage of the noise-contrastive estimation approach. The multi-sample upper bound is expressed as
LMUL = L (X()) -
( (Y(T)
(7)
Two main differences exist between these classes of upper bounds. First, the performance of multi- sample upper bound depend on batch size while uni-sample upper bounds do not, so when compu- tational budgets do not allow large batch size in training, uni-sample upper bounds may be preferred in training. Secondly, multi-sample upper bound has lower variance than uni-sample upper bounds. Thus, they have different strengths and weaknesses depending on the context. We evaluated the performance of those variational bounds of CPIC in terms of the reconstruction performance in syn- thetic experiments in Appendix G and find that with sufficiently large batch size, the multi-sample upper bound would outperform most of the uni-sample upper bounds. Thus, without further spec- ification, we choose the multi-sample upper bound as the variational bounds of CPIC objective in this work. Furthermore, by whether we employ a deterministic or stochastic encoder. Notice that when choosing the de- terministic encoder, the compression complexity term (first term) equation and equation are constant.
classify the upper bounds into stochastic and deterministic versions
5 NUMERICAL EXPERIMENTE
In this section, we demonstrate the superior performance of CPIC in both synthetic and real data experiments. We first examine the reconstruction performance of CPIC in noisy observations of dynamical system (the Lorenz Attractor). The results show CPIC better recovers the latent trajec- tories from noisy high dimensional observations. Moreover, we demonstrate that maximizing the predictive information(PI) in the compressed latent space is more effective than maximizing PI be- tween latent and observation space as in Creutzig Sprekeler (2008): Creutzig et al. (2009), and also demonstrate the benefits of the stochastic representation over the deterministic representation. Sec- ondly, we demonstrate better predictive performance of the representation evaluated by linear fore- casting. The motivation for using linear forecasting models is that good representations contribute to disentangling complex data a linearly accessible way (Clark et al.) 2019). Specifically, we extract latent representations and then conduct forecasting tasks given the inferred representations

---

Under review as a conference paper at ICLR 2023
two neuroscience datasets and two other real datasets. The two neuroscience datasets multi- neuronal recordings from the hippocampus (HC) while rats navigate a maze (Glaser et al. 2020) and multi-neuronal recordings from primary motor cortex (MI) during reaching task for monkeys D'Doreryy et al. 2017). The two other real datasets are multi-city temperature data (TEMP) from 30 cities over several years (Gene, 2017) and 12 variables from accelerater, gyroscope, and grav- ity motion sensor (MS) recording human kinematics (Malekzadeh et al. 2018). The forecasting tasks for the neuroscience data sets to predict the future of the relevant exogenous variables from the past neural data, while the forecasting task for the other datasets is to predict the future of those time-series from their past. The results illustrate that CPIC has better predictive performance on these forecasting tasks compared with existing methods.
Determistic (PIC Deterministic CPIC (L) Stochastic CPIC
Stochastic CPIC (L)
SNR 10-1
R²=0.928
R²=0.943
R² =0-96.63
SNR 10
SNR 10-3 87-0.527
R²
---0.869
R² -0.32
R2=00050
RR=00464
RF=0.036
RF=0.413
Figure 2: Left panel. Top: 3D trajectories of lorenz attractor's ground-truth. Middle: 30D pro- jected trajectory. Bottom: Corrupted 30D trajectory with SNR=0.001.. Right Panel. 3D trajectories obtained by deterministic CPIC and stochastic CPIC with PI in latent space or between latent and observation space in terms of different SNRs (0.1, 0.008, 0.001). We refer (L) to the case with PI in latent space and (O) to the case with between latent and observation space. We encode the point-wise Euclidean distance between the aligned inferred latent dynamies and the true dynamics into color on trajectories. Color from blue to red corresponds to the distance from short to long respectively. Separate colorbars are used for their corresponding SNRs.
5.1 SYNTHETIC EXPERIMENT WITH NOISY LORENZ ATTRACTOR
The Lorenz attractor is a 3D time series that are realizations of the Lorenz dynamical system (Pche- lintsev 2014). It describes three dimensional flow generated as:
dr
dy Si(p -
dz xy
di ol - x),
dt
dt
exhibit chaotic behavior, done in recent works (She & Wu 2020 Clark et al. 2019 Zhao & Park 2017] Linderman et al.) 2017). We simulated the trajectories from the Lorenz dynamical system and show them the left-top panel in Figure 2 We then mapped the 3D latent signals to 30D lifted observations with random linear embedding in the left-middle panel and add spatially anisotropic Gaussian noise on the 30D lifted observations in the left-bottom panel. The noises are generated according to different signal-to-noise ratios (SNRs), where SNR is defined by the ratio of the variance of the first principle components of dynamics and noise in Clark et al. 2019). Specifically,
Lorenz sets the values
10, p- 8/3 and
utilized 10 different SNR levels spaced evenly on a log (base 10) scale between [-3, -1] and corrupt the 30D lifted observations with noise corresponding different SNR levels. Details of the simulation available in Appendix Finellllll
7

---

Under review as a conference paper at ICLR 2023
deploy different variants of CPICs to recover the true 3D dynamics from different corrupted 30D lifted observations with different SNR levels, and compare the accuracy of recovering the underlying Lorenz attractor time-series.
We aligned the inferred latent trajectory with the true 3D dynamics with optimal linear mapping due to the reparameterizationinnnaa performance based on the R² regression score of the extracted vs. true trajectories. We first compare the reconstruction performance on different variational bounds of CPIC with the latent dimension size Q = 3 and the time window size T = 4, and find that multi-sample upper bound outperforms uni-sample upper bounds for almost all of the 10 SNR levels. Thus, we recommend the multi samli upper bound for CPIC in practice and use that for further results. We also find that, compared to DCA (Clark et al., 2019) and CPC (Oord et al. 2018) CPIC is more robust to noise and thus better extracts the true latent trajectory from the noisy high dimensional observations. The detailed results are reported in Appendix H
measure of latent trajectories. We validated the reconstruction
In order demonstrate the benefits of introduc- ing stochasticity in the encoder and maximiz- ing the predictive information in latent space,
N = 10
N 100
1.2
3.0
we considered four variants of CPICs: with
Mean performance
stochastic or deterministic encoder, and with
predictive information in latent space or be-
tween latent and observation space. All four variants of CPIC models utilize the latent di- mension size
veg
8.4
GPG00
CPiclla
Crictt
CRKCO
3 and the time window size T= For each model and each SNR level, we run 100 replicates with random initializations. We show the aligned latent trajectories inferred from corrupted lifted observation for high, in- termediate and low SNR (0.001, 0.01, 0.1) lev-
10
10
Signallto-rttiee ratio (500)
Signal-to-ntisee ratis (SNR)
Best performance
els of noise with the median R² scores across 100 replicates in Figure 2 The point-wise dis- tances between the recovered dynamics and the ground-truth dynamics are encoded in the col- ors from blue to red, corresponding to short to long distance. For high SNR (SNR 0.1, top- right), all models did a good job of recover- ing the Lorenz dynamics though the stochas- tic CPIC with predictive information on latent space had larger R² than others. For interme- diate SNR (SNR 0.008, middle-right), we see that stochastic CPICs performs much bet- ter than the deterministic CPICs. Finally, as the SNR gets lower (SNR 0.001, bottom-right) all methods perform poorly, but we note that, numerically, considering predictive information in latent space is much better than that between latent and observation space.
CPOI EMC Setict
Rt
CFC den0))
16-7
10
10
16-7
10
Signal-to-miiee rat ISNR)
Signel-to-raiee ratio (SNR)
Figure 3: Comparison on R2 scores of latent dy- namics regression on 10 SNR levels for four vari- ants of CPIC. The first row shows the mean per- formance of It2 scores over running N=10/100 different random initializations and the second row shows the best performance over running N=10/100 different random initializations.
To more thoroughly characterize the benefits of stochastic encoding and PI in the latent space, we examined the mean of R² scores for the four variants each level of SNR across N = 10 and N 100 replicates in the top row of Figure 3 It shows that the CPIC with stochastic representations and PI in latent space robustly outperforms other variants on average. We also report the best R² scores for the four variants in the sense that we report the 122 score for the model with the smallest training loss across N runs. The bottom row of Figure 3 shows that CPIC with stochastic representation and PI in latent space achieves better reconstruction and robustness to noise than other variants, especially when the number of runs N is small. Even when N is large, stochastic CPIC with PI in latent space greatly outperforms others when the noise level is high. We note that in the case of high-dimensional noisy observations with large numbers of samples common in many modern real-world time series datasets, CPICs robustness to noise and capacity to achieve good results in small number of runs is a clear advantage. Moveover, we displayed the quantile anaylsis of the R² scores in Appendix I with consistent result.
8

---

Under review as a conference paper at ICLR 2023
5.2 REAL EXPERIMENTS WITH DIVERSE FORECASTING TASKS
In this section, we show that latent representations extracted by stochastic CPIC perform better in the downstream forecasting tasks on four real datasets. We compared stochastic CPIC with con- trastive predictive coding (CPC) (Oord et al. 2018), PCA, SFA (Wiskott & Sejnowski 2002). DCA (Clark et al. 2019) and deterministic CPIC. As for CPC. we use linear encoder for fair compar- ison. In addition, we compared the result from CPCs and CPICs with nonlinear encoder in which the linear mean encoder is replaced by multi-layer perceptron. For each model, we extract the latent representations (conditional mean) and conduct prediction tasks on the relevant exogenous variable at a future time step for the neural datasets. For example, for the MI dataset, we extract a consecutive 3-length window representation of multi-neuronal spiking activity to predict the mon- key's arm position in a future time step which is lag time stamps away. The details of experiments are available in Appendix J Neuroscientists often want to interpret latent representations of data to gain insight into the processes that generate the observed data. Thus, we used linear regression to predict exogenous variables, with the intuition that simple (i.e., linear) prediction model will only be sensitive to the structure in the data that is easiest to interpret as in YYu et al., 2008 Pandarinath al., 2018 Clark et al. 2019). Furthermore, the neuroscience data sets (MI and HC) present ex- tremely challenging settings for prediction of the exogenous variables due to severe experimental undersampling of neurons due to technical limitations, as well as sizeable noise magnitudes. For these tasks, R regression score is used as the evaluation metric to measure the forecasting perfor- mance. Four datasets are split into 4:1 train and test data and the forecasting task considered three different lag values (5, 10, and 15). For DCA and determinissiccstochaatte CPICs, we took three different window sizes T = 1, 2, and report the best R² scores. Table reports all N² scores and demonstrates that stochastic CPIC outperforms all other models except for the case for Temp data with forecasting at lag 15.
Table 1: Comparison between CPC-L (linear encoder), CPC-NL (non-linear encoder), PCA, SFA, DCA, D-CPIC-L (deterministic CPIC with linear encoder). S-CPIC-L (stochastic CPIC with linear encoder), D-CPIC-NL (deterministic CPIC with non-linear encoder), and S-CPIC-NL (stochastic CPIC with non-linear encoder) on H² regression scores on MI, Hippocampus, Temperature, and Motion sensor datasets with the optimal window size among T E [1, 2, 3] for three different lag values (5, 10, and 15). R²² regression scores are averaged across five folds.
Dataset
Lag
CPC-L
CPC-NL
PCA
SFA
DCA
D-CPIC-L 0.222
S-CPIC-L
D-CPIC-NL
S-CPIC-NL
0.041
0.168
0.135
0.203
0.215
0.223
0.232
0.264
0.180
0.157
0.235
0.249
0.291
15
0.068
0.152
0.145
0.199
0.200
0.202
0,203
0.226
0.252
0.025
0.018
0.007
0.112
0.113
0.120
0.127
0,145
0.150
HC
10
0.012
0.012
0.001
0.101
0.101
0.107
0.113
0.121
0.333
-0.002
0.002
-0.005
0.085
0.085
0.091
0.095
0.094
0.114
0.666
0.639
0.651
0.669
0.668
0.672
0.673
0.673
0.673
Temp
10
0.630
0.584
0.61.5
0.630
0.632
0.629
0.633
0.630
0.634
0.624
0.529
0.581
0.623
0.622
0.620
0.621
0.621
0.621
0.281
0.184
0.107
40.031
0.443
0.247
0.437
0.290
0.483
0.154
0.068
-0.107
0.385
0.243
0.425
15
0.182
0.136
0.044
-0.131
0.342
0.161
0.358
0.216
0.379
6 CONCLUDING REMARKS
We developed a novel information-theoredi framework, Compressed Predictive Information Cod- ing, extract representations in sequential data. CPIC balances the maximization of the predictive information in latent space with the minimization of the compression complexity of the latent repre- sentation. We leveraged stochastic representations by employing stochastic encoder and developed variational bounds of the CPIC objective function. We demonstrated that CPIC extracts more accu- rate -ineninnniional latent dynamics and more useful representations that have better forecasting performance in diverse downstream tasks in four real-world datasets. Together, these results indicate that CPIC will yield similar improvements in other real-world scenarios. Moreover, we note that in most real datasets, using nonlinear CPIC would lead to better representation in terms of prediction performance than linear CPIC.
htdel.html

---

Under review as a conference paper at ICLR 2023
REFERENCES
David Barber Felix Agakov. The im algorithm: a variational approach to information maximization. Advances in neural information processing systems, 16(320):201, 2004.
Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint
1.03::180330835,. 2018.
Alexander A Alemi, lan Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.00410, 2016.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0: frame- work for self-supprvised learning of speech representations. arXiv preprint arXiv:2006. 1477, 2020.
Junwen Bai, Weiran Wang, Yingbo Zhou, and Caiming Xiong. Representation learning for sequence data with deep autoencoding predictive components. arXiv preprint arXiv:2010.03115, 2020.
David Barber and Felix Agakov. Information maximization in noisy channels: A variational ap- proach. Advances in Neural Information Processing Systems, 16, 2003.
Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 2018.
Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning; A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 7788)1798-81888 2013.
William Bialek, Ilya Nemenman, and Naftali Tishby, Predictability, complexity, and learning, Neu- ral computation, 13(11) (11::))2-224666 2001.
Rob Brekelmans, Daniel Moyer, Aram Galstyan, and Greg Ver Steeg. Exact rate-distortion in au- toencoders via echo noise. Advances in neural information processing systems, 32, 2019.
Tom В Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14166 2020.
Gal Chechik, Amir Globerson, Naftali Tishby, Yair Weiss, and Peter Dayan. Information bottleneck for gaussian variables. Journal of machine learning research, 6(1), 2005.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020.
Pengyu Cheng, Weituo Hao, Shuyang Dai, Jiachang Liu, The Gan, and Lawrence Carin. Club: A contrastive log-ratio upper bound of mutual information. In International Conference on Machine Learning, pp. 1779-1788. PMLR, 2020.
David G Clark, Jesse A Livezey, and Kristofer E Bouchard. Unsupervised discovery of temporal structure in noisy data with dynamical components analysis. arXiv preprint urXiv:/905. (ate::905000990 2019.
Felix Creutzig and Henning Sprekeler. Predictive coding and the slowness principle: An information-theorrete approach. Neural Computation, 1064)1122-1001,, 2008
Felix Creutzig, Amir Globerson, and Naftali Tishby. Past-future information bottleneck dynami- cal systems. Physical Review E, 79(4) 2009.
Jacob Devlin, Ming- Heig-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 18.004805 2018.
Alexander G Dimitrov, Aurel A Lazar, and Jonathan D Victor. Information theory in neuroscience. Journal of computational neuroscience, 30(1) 3:-5) 2011.
10

---

Under review as a conference paper at ICLR 2023
Monroe D Donsker and SR Srinivasa Varadhan. Asymptotic evaluation of certain markov process expectations for large time, i. Communications on Pure and Applied Mathematics, 28(1):1-47, 1975.
Otto Fabius and Joost R Van Amersfoort. Variational recurrent auto-encoders. arXiv preprint arXiv:1412.6581, 2014.
Roger Frigola, Yutian Chen, and Carl Edward Rasmussen. Variational gaussian process state-space models. Advances in neural information processing systems, 27, 2014.
Selfish Gene. Historical hourly weather data.
https:/ /ttpp:://ww... www www.w.aggl....mm com/ datasets/
2017.
Joshua Glaser, Ari S Benjamin, Raced H Chowdhury, Matthew G Perich, Lee E Miller, and Kon- rad P Kording. Machine learning for neural decoding. Eneuro, 7(4), 2020.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Grill, Florian Strub, Florent Altché, Corentin Tallec Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint Sttv:20006.07333 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 9729-9738, 2020.
Diederik P Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Alexander Lachmann, Federico M Giorgi, Gonzalo Lopez, and Andrea Califano. Aracne-ap: gene network reverse engineering through adaptive partitioning inference of mutual information. Bioin- formatics, 32(14):2233-223 2016.
Scott Linderman, Matthew Johnson, Andrew Miller, Ryan Adams, David Blei, and Liam Paninski. Bayesian learning and inference in recurrent switching linear dynamical systems. In Artificial Intelligence and Statistics, pp. 914-922. PMLR, 2017.
Alireza Makhzani, Jonathon Shlens, Navdeep Jaitly, Ian Goodfellow, and Brendan Frey. Adversarial autoencoders. arXiv preprint arXiv: 1.056444 2015.
Mohammad Malekzadeh, Richard G Clegg, Andrea Cavallaro, and Hamed Haddadi. Protecting sensory data against sensitive inferences. In Proceedings of the 1st Workshop on Privacy by Design in Distributed Systems, pp. 1-6, 2018.
Rui Meng, Herbie Lee, and Kristofer Bouchard. Stochastic Collapsed Variational Inference for Structured Gaussian Process Regression Network. arXiv e-prints, art, arXiv:2106.00719, June
2021.
XuanLong Nguyen, Martin J Wainwright, and Michael Jordan. Estimating divergence functionals and the likelihood ratio by convex risk minimization. IEEE Transactions on Information Theory, 56(1 (6(11):5847-586 2010.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 271-279, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint / 2018.
Joseph E O' O'Doherty, MMB Cardoso, JG Makin, and PN Sabes. Nonhuman primate reaching with multichannel sensorimotor contex electrophysiology. Zenodo http://doii. org/10.5281/zenodo, 583331, 2017.
11

---

Under review as a conference paper at ICLR 2023
Chethan Pandarinath, Daniel J O' O'Shea, Jasmine Collins, Rafal Jozefowicz, Sergey D Stavisky, Jonathan C Kao, Eric Trautmann, Matthew Kaufman, Stephen Ryu, Leigh R Hochberg, et al. Inferring single-trial neural population dynamics using sequential auto-encoders. Nature methods, 15110)805-881, 2018.
AN Pehelintsev. Numerical and physical modeling of the dynamics of the lorenz system. Numerical analysis and Applications, 7(2):159-161) 2014.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180. PMLR, 2019.
Qi She and Anqi Wu. Neural dynamics discovery via gaussian process recurrent neural networks.
In Uncertainty in Artificial Intelligence, pp. 454-464, PMLR, 2020.
Bernard W Silverman. Density estimation for statistics and data analysis. Routledge, 2018.
Jiaming Song and Stefano Ermon. Understanding the limitations of variational mutual information estimators. arXiv preprint arXiv:1910.06222, 2019.
Naftali Tishby, Fernando C Pereira, and William Bialek. The information bottleneck method. arXiv preprint physics/0004055, 2000.
Richard Tumnr and Maneesh Sahani. A maximum-likelihon interpretation for slow feature analy- sis. Neural computation, 19(4); 102--1102,- 2007.
Weiran Wang. Qingming Tang, and Karen Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. In ICASSP 2020-2020 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pp. 6889-6893. IEEE, 2020.
Yihang Wang, João Marcelo Lamim Ribeiro, and Pratyush Tiwary. Past-future information bottle- neck for sampling molecular reaction coordinate simultaneously with thermodynamics and kinet- ics. Nature communications, 10(1):1-8, 2019.
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invari- ances. Neural computation, 14(4):715-770, 2002.
Byron M Yu, John P Cunningham, Gopal Santhanam, Stephen Ryu, Krishna V Shenoy, and Ma- neesh Sahani. Gaussian proocesss factor analysis for low-dimensional single-trial analysis of neural population activity. Advances in neural information processing systems, 21, 2008
Yuan Zhao and Il Memming Park. Variational latent gaussian process for recovering single-trial dynamics from population spike trains. Neural computation, 2993):1233-116, 2017.
APPENDIX
A SELECTION OF WINDOW SIZE
Selecting optimal window size 7 is important for the downstream use of the dynamics. Poor selec- tion of T' may cause aliasing artifacts. In general, we nee to select it by cross validation. Further- more, we can make plots of the predictive information as function of both window size T and the embedding dimension Q as diagnostic tools.
B DERIVATION OF IVUB
Directly estimating the compression complexity is intractable, because ((X ((T) =
Ex(ry (L(ppy(TT)TTT))))(((//uul P(y(T))) in which the population distribution p()(TT) is unknown. Thus introduce variational approximation to the marginal distribution of encoded inputs
12

---

Under review as a conference paper at ICLR 2023
p(y(T)), denoted (T)( Due to the non-negativity of the Kullback-Leibler (KL) divergence, the variational upper bound (VUB) is derived as
I(X(T); = Ex(r) (LLPW(C)))()))))(((() - -LL (9)
I Ex (KLPp)((C))a(())))(())))
)
C DERIVATION OF L1out
Generally, learning (T)) was recognised as the distribution density estimation problem (Silver- man 2018), which challenging. In this setting, the variational distribution r(y(T)) is assumed to be learnable, and thus estimating the variational upper bound is tractable. In particular, AAlemi al. 2016) fixed r(y(T) as a standard normal distribution, leading to high-bias in MI estimation. Recently, Poole et al. (2019) utilized Monte Carlo approximation for variational distribution. In our case, with S sample pairs ( (I) ri((()) - - E)s)(((())
p(y(T))
and the 1.1Out is derived below:
(xIo((((T)) =
log
(10)
5
Σ P ((u((T)
8-1
D DERIVATION OF IVLB
Similar to Agakov 2004). we replace the intractable conditional distribution p(y(T")ly( with
tractable optimization problem over variational conditional distribution ( (T))(( -T). )) It yields a lower bound on PI due to the non-negativity of the KL divergence:
X((()))) HT) Bxtu( (T)(())))(())) [log (v((T) w(-T))
(11)
where H(Y) is the differential entropy of variable Y and this bound is tight if and only if ((y(T))y(-1)) = p(y(T) (y( ((T)), suggesting that the second term in equation I1 equals the neg- ative conditional entropy - HH (Y (TT(
However the variational lower bound requires a tractable decoder for the conditional g(y|z). Alter- mneegy-based variational family for conditional distribution
natively, by considering
The conditional expectation in equation II can be estimated using Monte Carlo sampling based on the encoded data distribution ((y(-T), y (()). And encoded data are sampled by introducing the augmented data x--T) and (T) and marginalizing them out as
p(u((-T).u(T)) =
(12)
according to the Markov chain proposed Figure]
E DERIVATION OF ITUBA
According to Poole et al. (2019). by considering an energy-based variational family to express and conditional distribution (
c(u(T)lu(-T)) = (pp(((()))))))) (lu(T)))(((()))-))
(13)
Z(y(-T))
where f(r 3) is a differentiable critic function, 2(y(-T)) = Ep(u(T) (e (e(((()),((11)) is partition function, and introducing a baseline function a(y(T)), we derived a tractable TUBA lower bound (Barber & Agakov| 2003) of the predictive information as:
(Y(-T))YY(T)) 2 Eptst- Euot- v())) log (
= ITUBA IYUAA (Y(TT),
(14)
where ((y(-T) be)(-TTT)T))>)))))) = f(y(f((()) (T)) - log(a()(T))) is treated as an updated critic function. Notice that different choices of baseline functions lead to different mutual information estimators.
13

---

Under review as a conference paper at ICLR 2023
When (y(()) = 1, it leads to mutual information neural estimator (MINE) (Belghazi al.
201
when (()(T)) = (T)( it leads to the lower bound proposed in Donsker & Varadhan 1975
al. (2010) (NWJ) also known as f-GAN (Nowozin et al. 2016) and MINE-f (Belghazi et al. 2018). In general, the critic function
(DV) and when a(y(T))
c, recovers the lower bound in Innuggyyn
f(xr, and the log base ine function a(y) usually parameterized by neural networks (Oord
et al. 2018 Belghazi et al. 2018): Oord et al. (2018) used separable critic function f(x,3)
(x go ( while Belghazi al. (2018) used a joint critic function f(x, y) fa( y). and Poole
et al. (2019) claimed that joint critic function generally performs better than separable critic function
scale poorly with batch size.
F DERIVATION OF InffOCC
The derivation of infoNCE in our CPIC setting is trivial by treating Y-T)) and Y(T) the input and output in the infoNCE formula from the CPC setting (Oord et al. 2018).
G DETAILS OF SIMULATION
In this section, we first generated the 3D latent signals according the Lorenz dynamic system denoted as E R3xf We calculated the largest eigenvalue of the covariance matrix X as dynamic variance denoted as o dynamiics and the noise variance is hoise = o dynammics /SN R where SNR it is signal-to-noise ratio. Then we randomly generate a semi orthogonal matrix E 30333 Then we generated the true 30D signal VX embedded with additive spatially structured white noise, where the noise subspace Vnoise generated with median principle angles with respect to dynamics subspaces V. The noise covariance is generated via Enoise with the largest eigenvalue o noise: and then we generate the noisy signal at the nith dimension by [Ynoisy] N( X, Enoise), n 30. 1..
H MODEL COMPARISON IN TERMS OF Rh REGRESSION SCORE IN THE NOISY LORENZ ATTRACTOR EXPERIMENT
In this section, the R regression scores for CPC, DCA, deterministic stochastic CPICs (three uni- sample upper bounds in terms of NWJ, MINE, TUBA, and one multi-sampl upper bound) for all ten different SNRs are reported in Table 2 shows that stochastic CPIC with multi- sample upper bound outperforms other approaches in majority of SNRs. It also shows that that CPIC is most robust to the noisy data and thus detect best latent trajectories from noisy observation compared with CPC and DCA.
We also show the aligned latent trajectories inferred from corrupted lifted observation for high, intermediate and low SNR (0.001, 0.01, 0.1) levels of noise with the median Ri² scores across 100 replicates for PCA and DCA (as the extension of Figure 2 in Figure 4 The point-wise distances between the recovered dynamics and the ground-truth dynamics are encoded the colors from blue to red, corresponding to short to long distance. It show that stochastic CPIC outperforms both PCA and DCA.
I COMPARISON ON R² SCORES OF LATENT DYNAMICS REGRESSION FOR NOISY LORENZ ATTRACTOR IN TERMS OF QUANTILE ANALYSIS
We displayed the medium performance (with the inter-quantile range as the error bars) R2 scores of latent dynamics regression for noisy Lorenz attractor in Figure 5
DETAILS OF REAL-WORLD EXPERIMENTS
The four real data are Monkey motor cortical dataset (M1), Rat hippocampal data (HC), Temperature dataset. (Temp) and Accelerate dataset (MS).
14

---

Under review as a conference paper at ICLR 2023
PCA
DCA
SNR 10
Ro=0.014
R0-0.9
SNR 10"2
R20000
R²=0.480
SNR 10"3
R²=00.0008
R²=0.458
Figure 4: Left panel. Top: 3D trajectories of lorenz attractor's ground-truth. Middle: 30D projected trajectory. Bottom: Corrupted 30D trajectory with SNR=0.001. Right Panel. 3D trajectories ob- tained by PCA and DCA in terms of different SNRs 10.1, 0.008, 0.001). We encode the point-wise Euclidean distance between the aligned inferred latent dynamics and the true dynamics into color on trajectories. Color from blue to red corresponds to the distance from short long respectively. Separate colorbars are used for their corresponding SNRs.
Table R² regression scores for CPC, DCA, deterministic stochastic CIPCs including three uni- sample upper bounds (UNI): NWJ, MINE, TUBA, and one multi-sample upper bound (MUL) for all different SNRs
CPIC
Deterministic
Stochastic
SNR
CPC
DCA
UNI
MUL
UNI
MUL
NWJ
MINE
TUBA
NWJ
MINE
TUBA
0.001
0.132
0.458
0.554
0.543
0.547
0.482
0.539
0.550
0.553
0.459
0.00167
0.195
0.466
0.539
0.538
0.574
0.430
0.573
0.569
0.571
0,576
0.00278
0.265
0.473
0.573
0.573
0.573
0.413
0.587
0.583
0.590
0.588
0.00464
0.344
0.478
0.579
0.562
0.584
0.438
0.598
0.583
0.556
0,593
0.00774
0.421
0.480
0.597
0.559
0.515
0.912
0.582
0.579
0.589
0.598
0.01292
0.491
0.484
0.587
0.596
0.597
0.468
0.580
0.563
0.592
0.923
0.02154
0.547
0.486
0.590
0.596
0.592
0.688
0.568
0.599
0.864
0.930
0.03594
0.592
0.491
0.587
0.912
0.594
0.923
0.937
0.632
0.907
0.951
0.05995
0.635
0.952
0.933
0.837
0.936
0.474
0.970
0,939
0.896
0.970
0.1
0.671
0.953
0.920
0.893
0.889
0.922
0.926
0.910
0.854
0.989
J.1 MONKEY MOTOR CORTICAL DATASET
O" O'Doherty et al. 2017 7
released multi-electrode spiking data for both MI and S1 for two monkeys during a continuous grid baddd reaching task. We used MI data from the subject "Indy" (specifically, used the file "indy 20 160627__01 1o27_00.mmtt),, We discarded single units with fewer than 5,000 spikes, leaving 109 units. We binned the spikes into non-overlapping bins square-root transformed the data and mean-centered the data using sliding window 30 in width.
15

---

Under review as a conference paper at ICLR 2023
N 10
N 100
1.0
1.0
R regression score
0.7
0.7
Median performance of CPICs (O)
R regression
0.4
0.4
CPIC
CPIC
det(()
CPIC_ det(O)
10
10a
10
10-2
10-1
Signal-to-noise ratio (SNR)
Signal-to to-noosoeo ratio (SNR)
1.0
1.0
regression score
Median performance of CPICs (L)
0.7
0.7
regression
CPICIL)
CPICILI
CPIC det(l.)
CPIC det(L)
10-3
10-2
10-1
10->
10-2
10-1
Signal-to-noise ratio (SNR)
Signal-to-noise ratio (SNR)
Figure 5: Comparison on R² scores of latent dynamics regression on 10 SNR levels for four vari- ants of CPIC. The first row shows the medium performance (with the inter-quantile range as the error bars) of R² scores for CPICs (O) with PI between latent and obersvation space over running N=10/100 different random initializations, and the second row shows the performance for CPICs (L) with PI in latent space.
J.2 RAT HIPPOCAMPAL DATA
Glaser et al. 2020 released the original data. The data consist of minutes extracellular record- ings from layer CAI of dorsal hippocampus while a rat chased rewards square platform. We discarded single units with fewer than 10 spikes, leaving 55 units. We binned the spikes into non- overlapping 50 ms bins, then square-root transformed the data.
J.3 TEMPERATURE DATASET
The temperature dataset consists hourly temperature data for 30 U.S. cities over a period of 7 years from OpenWeatherMappoorg We downsampled the data by factor of 24 obtain daily temperatures.
J.4 ACCELEROMETER DATASET
Malekzadeh et (2018) released accelerometer data which records roll, pitch, yaw, gravity x, y. z, rotation and acceleration x, y, for total of 12 kinematic variables. The sampling rate is 50 Hz. We used the file "sub 19.csv" from "A DeviceMotion data.zip".
J.5 FORECASTING TASK
The forecasting task is the same in Clark al. (2019). We use the extracted consecutive 3-length window representation of endogenous data to forecast the future relevant exogenous variables at log In MI and HC, the endogenous variables are processed spiking data, and the exogenous variables are location data. Temp and MS, we assume endogenous variables and exogenous variables are the same, 30 U.S. cities" hourly temperature for Temp data and 12 kinematic variables for MS data.
16


Reviews:
[{"review_id": "p_FsPy5vgT", "reviewer": "Reviewer_jyD8", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims at extracting low-dimensional structure for dynamic data, especially for time series data. The proposed method, Compressed Predictive Information Coding (CPIC), both minimizes compression complexity and maximizes the predictive information\nin the latent space. This work extends the prior works dynamical components analysis and deep autoencoding predictive components. This work includes the analysis for both linear and non-linear encoding. The core idea is to estimate the latent predictive information without heavy matrix computation, but with cheap approximations. The resulting model achieves good performance on synthetic lorenz attractor, two neuroscience datasets and other real-world scenarios.", "summary_of_the_review": "This paper improves over previous method, and potentially reduces the time cost. Operating in the latent space also takes advantage of the large capacity brought by deep models. It would be better if the method is also tested on larger and more mainstream time series datasets.", "strength_and_weaknesses": "Pros:\n1. The variational bound alleviates the cost of predictive information. It can potentially be used in more scenarios as a plug-in module.\n2. Operating in the latent space makes more sense than input space and employs the power of non-linear layers.\n3. Various experients and real-world datasets are shown.\n\nCons:\n1. It seems that DAPC has a very similar framework, also with a deterministic and probabilistic encoders (also a VAE?). Could you further elaborate on the differences?\n2. Could you further justify the stationarity assumption? Is it commonly seens in time series analysis?\n3. When you demonstrate the qualitative results (e.g. synthetic data), maybe you can also show some results from other methods.\n4. You can also show some time cost numbers to demonstrate the advantage of your method speed-wise.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dcKoq14WuPP", "reviewer": "Reviewer_aQPA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a way to optimize time series forecasting model through adapted information bottleneck loss. In particular, it is proposed to learn low dimensional representations of the input time series (using variational encoders) via maximizing the mutual information between representations of the input and target and minimizing mutual information between input and representation. The approach is termed compressed predictive information coding (CPIC) and the authors propose two different computable bounds, based on the existing estimators of mutual information, that can be used to train the models. Further, authors explore the empirical performance of the proposed bounds, first on the artificially generated data with clear structure, to check the ability of the representations learned with CPIC to reconstruct data. The second experimental evaluation is done on the real data where forecasting is made using linear regression on the learned representations.", "summary_of_the_review": "The paper formulates an information bottleneck loss for timeseries forecasting problem, using representations learned with variational encoders. The existing estimators combined to produce a computable bound for the proposed loss and further this bound evaluated for the task of reconstruction and forecasting.\nOverall, my main concern is about the reported evaluation results, which seem to be not very high. Since the core of the paper is formulation of the loss, the empirical evaluation should be very convincing.\nMoreover, why the reported results for existing methods (CPC and DCA) are so much worse, nearly twice on average or even more? Are these methods so bad for the problems selected for experiments? Or only with the SNR that is very low? The reported results are also very low for the real world data. I wonder what is the state-of-the-art scores on the problems considered.\n\n----\nI thank the authors for the clarifications. I will stick with my score after the internal discussion.\n", "strength_and_weaknesses": "The paper is proposing an information bottleneck type of the loss for a different from usual setup. The time series forecasting is a complicated task that is an active research area and novel approaches are always interesting.\n\nThe paper uses existing estimators and bounds on mutual information, thus the main novelty of the paper is the formulation of the information bottleneck loss and finding methods that allow to compute it. The evaluation shows that the proposed method performs better than two existing approaches, though still with not very high performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "Omt83sE__lT", "reviewer": "Reviewer_U5Xb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a learning criterion and numerous bounds for learning representations of dynamic systems that are at once maximally compressed (minimal mutual information with their original representation, i.e., minimal rate) while being maximally informative a future time point (or, due to symmetry, a previous timepoint). This leads to learned encodings which are, up to the SNR and trade-off parameter $\\beta$,  reflective of the dynamical system.", "summary_of_the_review": "The authors present an elegant embedding method for dynamical systems, and the estimation machinery to fit the embedding efficiently. The method is novel, but also fits well into existing literature, adding new and interesting directions to Rate-Distortion based encodings. I think the ICLR community at large would be interested in such a manuscript.", "strength_and_weaknesses": "Strengths:\n* The method is grounded in established literature (information bottleneck and variational information bottleneck, and more generally Rate-Distortion theory), yet contributes a novel criterion. Secondly, the recognition that the stationarity assumption reduces the learning criterion to a two term loss function, which is, in structure, very similar to many other information trade-offs in the literature.  \n* Multiple bounds are explored for the mutual information minimization/maximization.  \n* Experimental results are generally well done, modulo exact generation details.  \n\nWeaknesses:\n* Allowing time-windows T, for periodic or pseudoperiodic phenomena with period T' we might observe aliasing artifacts. While this remains stationary in a global sense, will such artifacts impede dynamics. Moreover, will the interpretation or downstream use of those dynamics be impeded by this induced false beat frequency at (T-T')/2?  \n* The variational lower bound (VLB) of Theorem 3 does not appear to require a decoder, contrary to the comment in the sentence immediately following the statement of the theorem; it seems as though it instead requires a good estimate of the conditional likelihood (a prediction from either T to T' or vice versa).  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3uuFwUol-A", "reviewer": "Reviewer_3inW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a method for self-supervised time-series prediction called Compressed Predictive Information Coding (CPIC). The core idea is that a sequence of data points are mapped to a Gaussian latent space via some encoder function. Then, these latent representations can be used for downstream tasks, e.g. regression. The authors propose minimizing the difference between two computationally intractable mutual information terms to train the model. To make their approach practically applicable, the authors propose to minimize a variational upper bound to the original objective. The authors test their approach on synthetic and some real-world regression problems.", "summary_of_the_review": "The authors present a nice, well-motivated idea. However, it is unclear what theoretical contributions they claim, the description of their empirical methodology is incomplete, and the presentation quality is generally low.", "strength_and_weaknesses": "## Strengths\nThe proposed approach is well-motivated and an interesting idea. It is quite simple; hence, I wonder if it has already been proposed elsewhere. However, I am not familiar enough with the sequential prediction literature to be able to comment on the originality of the approach though. \n\nSome of the empirical results also seem promising.\n\n## Weaknesses\nIn addition to the list below, the paper suffers from severe stylistic and clarity issues; see the section below.\n - In Sections 4.1 & 4.2, the authors state four \"theorems\" that give certain variational bounds on the intractable terms in their proposed loss function in Eq 4. However, it is not entirely clear what the authors are claiming here as their original contribution. The statements in Thm 1 and 3 follow trivially using elementary information-theoretic arguments in two lines (Eqs 9 and 11 in the appendix). The statements of Thm 2 and 4, and 5 are essentially taken from other works with trivial substitutions. Could the authors please comment on what exactly they are claiming as their contribution here?\n - The authors never formally state the exact models used for their experiments, making their results impossible to interpret.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject