Representation Learning for Sequence Data with Deep Autoencoding Predictive Components
Authors: Junwen Bai
Weiran Wang
Yingbo Zhou
Caiming Xiong
Venue: ICLR 2021 Poster (2021)
Keywords: Mutual Information
Unsupervised Learning
Sequence Data
Masked Reconstruction

Abstract: We propose Deep Autoencoding Predictive Components (DAPC) -- a self-supervised representation learning method for sequence data, based on the intuition that useful representations of sequence data should exhibit a simple structure in the latent space. We encourage this latent structure by maximizing an estimate of \emph{predictive information} of latent feature sequences, which is the mutual information between the past and future windows at each time step. In contrast to the mutual information lower bound commonly used by contrastive learning, the estimate of predictive information we adopt is exact under a Gaussian assumption. Additionally, it can be computed without negative sampling. To reduce the degeneracy of the latent space extracted by powerful encoders and keep useful information from the inputs, we regularize predictive information learning with a challenging masked reconstruction loss. We demonstrate that our method recovers the latent space of noisy dynamical systems, extracts predictive features for forecasting tasks, and improves automatic speech recognition when used to pretrain the encoder on large amounts of unlabeled data.

Published as a conference paper at ICLR 2021
REPRESENTATION LEARNING FOR SEQUENCE DATA WITH DEEP AUTOENCODING PREDICTIVE COMPO- NENTS
Junwen Bai Corelll University junwenêcs. www.eoooo..oooooooo..
Weiran Wang"
Google welraavangggooggeeccom
Yingbo Zhou & Caiming Xiong Salesforce Research fyingbo. zhou, cxiong]esalesforceeco
ABSTRACT
We propose Deep Autoencoding Predictive Components (DAPC) - self- supervised representation learning method for sequence data, based the intu- ition that useful representations sequence data should exhibit a simple structure in the latent space. We encourage this latent structure by maximizing of predictive information of latent feature sequences, which is the mutual infor- mation between the past and future windows each time step. In contrast the mutual information lower bound commonly used by contrastive learning, the estimate of predictive information we adopt is exact under Gaussian assump- tion. Additionally, it can be computed without negative sampling. To reduce the degeneracy of the latent space extracted by powerful encoders and keep useful information from the inputs, regularize predictive information learning with a challenging masked reconstruction loss. We demonstrate that our method recovers the latent space of noisy dynamical systems, extracts predictive features for fore- casting tasks, and improves automatic speech recognition when used to pretrain the encoder large amounts of unlabeled data.
estimate
INTRODUCTION
Self-supervised representation learning methods aim at learning useful and general representations from large amounts of unlabeled data, which can reduce sample complexity for downstream su- pervised learning. These methods have been widely applied various domains such as computer vision (Oord et al., 2018: Hjelm et al. 2018: Chen et al. 2020 Grill et al., 2020), natural lan- guage processing (Peters
20.8] Devlin al. 2019 Brown al. 2020). and speech process- ing (Schneider et al.) 2019 Pascual et al. 2019b; Chung & Glass 2020 Wang et al. 2020 Bacvski al.] In the case of sequence data, representation learning may force the model recover the underlying dynamics from the raw data, so that the learnt representations remove irrelevant vari- ability in the inputs, embed rich context information and become predictive of future states. The effectiveness of the representations depends on the self-supervised task which injects inductive bias into learning. The design of self-supervision has become an active research area.
One notable approach for self-supervised learning is based maximizing mutual information be- tween the Iearnt representations and inputs. The most commonly used estimate of mutual informa- tion is based contrastive lcarning. A prominant example of this approach is CPC (Oord et al.) 2018), where the representation of each time step is trained to distinguish between positive samples which are inputs from the near future, and negative samples which are inputs from distant future or other sequences. The performance of contrastive learning heavily relies on the nontrivial selection
"Wor done during internship at Salesforce Research.
I Work done while Weiran Wang was with Salesforce Research.
Code is available http://gthhhh.oom///wweniiiii//

---

Published as a conference paper at ICLR 2021
of positive and negative samples, which lacks universal principle across different scenarios (He)
jet al. 2020; Chen et al. 2020] Misra & Maaten 2020). Recent works suspected that the mutual information lower bound estimate used by contrastive learning might be loose and may not be the sole reason for its success (Ozair et al. 2019 T'schannen et al. 2019).
In this paper, we leverage estimate of information specific to sequence data, known predictive information (PI, Bialek et al. 2001), which measures the mutual information between the past and future windows in the latent space. The estimate is exact if the past and future windows have joint Gaussian distribution, and is shown by prior work to be a good proxy for the true predictive information practice (Clark al. 2019). We can thus compute the estimate with sample windows of the latent sequence (without sampling negative examples), and obtain a well-defined objective for learning the encoder for latent representations. However, simply using the mutual information as the learning objective may lead to degenerate representations, as Pl emphasizes simple structures in the latent space and a powerful encoder could achieve this the cost of ignoring information between latent representations and input features. To this end, we adopt masked reconstruction task to enforce the latent representations to be informative of the observations as well. Similar to Wang et al. (000)), we mask input dimensions as well as time segments of the inputs, and use decoder to reconstruct the masked portion from the learnt representations, we also propose variants this approach achieve superior performance.
Our method, Deep Autoencoding Predictive Components (DAPC), is designed to capture the above intuitions. From variational inference perspective, DAPC also has natural probabilistic interpre- tation. We demonstrate DAPC on both synthetic and real datasets of different sizes from various domains. Experimental results show that DAPC can recover meaningful low dimensional dynamics from high dimensional noisy and nonlinear systems, extract predictive features for forecasting tasks, and obtain state-of- of-t--------rrr accuracies for Automatic Speech Recognition (ASR) with much lower cost, by pretraining encoders that are later finetuned with limited amount of labeled data.
2 METHOD
The main intuition behind Deep Autoen- coding Predictive Components is to max- imize the predictive information of la- tent representation sequence. To en- sure the learning process tractable and non-degenerate, we make a Gaussian as- sumption and regularize the learning with masked reconstruction. In the following subsections, we elaborate on how we es- timate the predictive information and how we design the masked reconstruction task. A probabilistic interpretation of DAPC is also provided to show the connection to deep generative models.
decoder
PAST
Pl
FUTURE
2/41
2
encoder
Figure 1: The overall framework of DAPC.
2.1 PREDICTIVE INFORMATION
Given sequence of observations X (21, where 3; R", extract the corresponding latent sequence 7 = {21, 22..... where z; E IRd
with an encoder function el (X), e.g., recurrent neural nets transformers (Vaswani al. 2017 Let T O be fixed window size, and de-
note Zpast 1 2e). Z atare (2++1; Z+TT} for any time step The predictive information (PI) defined as the mutual information (MI) between 2,6
Zhast and 7 future,
MI L.fturee= (2.aast) + HH future)
H( past 2f5 future
(I)
II this work, the latent sequence has the same length as the input sequence, but this is not an restriction; one can have a different time resolution for the latent sequence, using sub-sampling strategies such as that of Chan al. 2016).
2

---

Published as a conference paper at ICLR 2021
where H is the entropy function. Intuitively, PI measures how much knowing Zinast reduces the uncertainty about Zi.uure (and vice versa). PI reaches its minimum value 0 if Zpaae and Z Z/ftuue are independent, and it maximized if Z Z.future is a deterministic function of Zg Different from the MI estimate used by contrastive learning, which measures the MI between representation at each single time step and its future inputs, predictive information measures the MI between two windows of T time steps collectively. The window size T used in Pl estimation reflects the time resolution for which the time series is more or less stationary.
Pl was designed general measure of the complexity of underlying dynamics which persists for relatively long period of time (Li & Vitányi (008). Furthermore, PI is aware of temporal structures: different dynamics could lead Pl to converge or diverge even if they look similar. These virtues of Pl contribute to the versatility of this measure. The use of PI beyond a static complexity measure (and as a learning objective) done only recently in machine learning by Clark al. (2019), which proposes to learn a linear dimensionality reduction method named Dynamical Component Analysis (DCA) to maximize the PI of the projected latent sequence.
One approach for estimating Pl is through estimating the joint density P(Z P(Z Z, Z.uture), which can be done by density estimation methods such as k-NN and binning (Dayan & Abbott 2001 Kraskov iet al. 2004]. However, such estimates heavily rely on hyperparameters, and it is more challenging to come up with differentiable objectives based frameworks. Our approach for estimating PI is the same as that of DCA. Assume that every 27 consecutive time steps {2-T++1:- ...y St..... Zt+T) } in the latent space form a stationary, multivariate Gaussian distribution. (2) is used to denote the covariance of the distribution, and similarly
them that are compatible with deep learning
(Z) the covariance of T consecutive latent steps. Under the stationarity assumption, H HZgaarr) remains the same for any so we can omit the subscript t. and H (ZPast) is equal to H [Z Future) as In(2ree [2r ()). PI for the time series 2 reduces to
well. Using the fact that H (2
1
I = MI (uture) In (Z)
In [Z)(
2
Detailed derivations can be found in Appendix A It is then straightforward to collect samples of the consecutive 27 -eelhh windows and compute the sample covariance matrix for estimating E2r An empirical estimate of (Z) corresponds to the upper left sub-matrin of Egr(2) (2) Recall that, under the Gaussian assumption, the conditional distribution P P(Z faturre ( is again Gaussian, whose mean is linear transformation of Zrast Maximizing Lr has the effect of minimizing the entropy of this conditional Gaussian, and thus reducing the uncertainty of future given past.
Though estimation formula for PI is exact only under the Gaussian assumption, it observed by Clark et al. (2019) that the Gaussian- boussian-bbase estimate is positively correlated with a computation- ally intensive estimate based estimate. We make the same weak assumption, so that optimizing the Gaussian- estimate im- proves the true PI. Our empirical results show that representations learnt with the Gaussian PI have strong predictive power of future (see Sec 4.2). Furthermore, we find that a probabilistic version of DAPC (described in See 2.3) which models (Zeast future) with a Gaussian distribution achieves similar performance as this deterministic version (with the Gaussian assumption).
par-mmmmttriic density estimate, and thus good proxy for the full
We now describe two additional useful techniques that we develop for Pl-based learning.
Multi-scale PI One convenient byproduct of this formulation and estimation for IT is to reuse Ear (Z) for estimating T//2, Tr/a and so on, as long as the window size is greater than 1. Since the upper left sub-matrix of Σer (Z) approximates Σn (Z), we can extract (T) from Mgr(2) without any extra computation, and similarly for Σr/2 We will show that multi-scale Pl, which linearly combines PI at different time scales, boosts the representation quality in ASR pretraining.
Orthogonality penalty Observe that the PI estimate in 2 is invariant to invertible linear trans- formations in the latent space. To remove this degree of freedom, we add the penalty to encour- age latent representations to have identity covariance, so that each of the d latent dimensions will have unit scale and different dimensions are linearly independent and thus individually useful. This penalty is similar to the constraint enforced by deep canonical correlation analysis (Andrew et al. 2013), which was found be useful in representation learning (Wang al. 2015).
3

---

Published as a conference paper at ICLR 2021
2.2 MASKED RECONSTRUCTION AND ITS SHIFTED VARIATION
The PI objective alone can potentially lead to degenerate latent space, when the mapping from input sequence latent sequence is very powerful, as the latent representations can be organized in a way that increases our Pl estimate at the cost of losing useful structure from the input. This also observed empirically in our experiments (see See 4.1. To regularize PI-based learning. one simple idea is to force the learnt latent representations to be informative of the corresponding input observations. For this purpose, we augment PI-based learning with a masked reconstruction task.
Masked reconstruction was first proposed in BERT (Devlin et al. 2019), where the input text is fed to model with a portion of tokens masked, and the task reconstruct the masked portion. [Wang et al. (2020) extended the idea to continuous vector sequence data (spectrograms). The authors found that randomly masking input dimensions throughout the sequence yields further performance gain, compared to masking only consecutive time steps. We adopt their formulation in DAPC to handle continuous time series data.
Given input sequence X of length L and dimensionality n, we randomly generate a binary mask M E RaxL, where Mi,j = 0 indicates Xi,j is masked with value 0 and Mr, = indicates Xs is kept the same. We feed the masked inputs to the encoder e(-) to extract representations (in Hd) for each time step, and use ferd-forward network g(-) to reconstruct the masked input observations. e(.) and g() are trained jointly. The masked reconstruction objective can be defined as
R [| M) (X g(e(X O M) fro
Figure [ gives an illustration for the masked spectrogram data. We randomly generate nr time masks each with width up wy, and similarly np frequency masks each with width up to wy- In our experiments, we observe that input dimension masking makes the reconstruction task more challenging and yields higher representation quality. Therefore, this strategy is useful for general time series data beyond audio.
We introduce more improvement to masked reconstruction. Standard masked reconstruction recovers the masked inputs for the same time step. Inspired by the success of Autoregressive Pre- propose shifted variation of masked reconstruction,
dictive Coding (Chung & Glass 2020), in which the latent state 2; is decoded to reconstruct a future frame Ti+s (than æt). Formally, the shifted masked reconstruction loss R, is defined as
Rg = [I(1 - M @ (x* +s g(e(X @ MI
(4)
where - s indicates right-shifting time frames while the input dimensions remain unchanged. When s O, R, reduces the standard masked reconstruction objective, and in the ASR experi- ments we find that nonzero 8 value helps. We ensure no information leakage by enforcing that the portion to be reconstructed never presented in the inputs. As indicated by Chung et al. (2019b), predicting future frame encourages more global structure and avoids the simple inference from local smoothness domains like speech, and therefore helps the representation learning.
To sum up, our overall loss function is defined as the combination of the losses described above:
min Tx,7(X) -TT + alT;2) BR, + + Rortho 6gg
where ax y are tradeoff weights and Rortho = ||Σ - lalli,.. is the orthonormality penalty dis- cussed in Sec. 2.1 with Σn E Rdxd corresponding to the top left sub-matrix of Σ2r estimated from the latent sequence Z e(X @ M). The whole framework of DAPC is illustrated Figure
2.3 PROBABILISTIC INTERPRETATION OF DAPC
We now discuss a probabilistic interpretation of DAPC in the Variational AutoEncoder (VAE) frame- work (Kingma & Welling 2014). Let = (Xp.Xy) and = (Zp: Zy). where the subscripts and f denote past and future respectively. Consider a generative model, where the prior distribution is p(Zp. Zz) ~ N (0, Σ). One can write down explicitly p(ZrlZp), which is a Gaussian with
Msp E.
=pp=E, E.ppppppp..
The linear dynamics in latent space are completely defined by the covariance matrix Σ. Large predictive information implies low conditional entropy H H 7zp)
4

---

Published as a conference paper at ICLR 2021
Let (zp.Zz generate (Xp.Xy) Xy) with stochastic decoder (x We only observe X and would like to infer the latent Z by maximizing the marginal likelihood of X. Taking a VAE approach, we parameterize stochastic encoder eX for the approximate posterior, and derive lower bound for the maximum likelihood objective. Different from standard VAE, here we would not want to parameterize the prior be simple Gaussian, in which case the Zp and Z are independent and have zero mutual information. Instead we encourage the additional structure of high predictive information for the prior. This gives us an overall objective follows:
min / ((X) E.e.9
{ - - g Z)dz - dr nII(EE
where p(X) is the empirical distribution over training data, the first term corresponds to the recon- struction loss, the second term measures the KL divergence between approximate posterior and the prior, and the last term is the PI defined in (2).
The challenge is how parameterize the covariance E. We find that simply parameterizing it positive definite matrix, e.g., D = AA"", does not work well in our experiments, presumably because there too much flexibility with such formulation. What we find to work better is the pseudo-input technique discussed in VampPrior (Tomezak & Welling 2018): given a set of pseudo-sequences X which are learnable parameters (initialized with real training sequences), we compute the sample covariance from e(Z as E.
This approach yields an overall objective very similar to (5), with the benefit of well-defined generative model (and the Gaussian assumption being perfectly satisfied), which allows us to borrow learnin/inffrrnce techniques developed in the VAE framework. For example, masking the input for the encoder can be seen amortized inference regularization (Shu et al.] 2018). We show experimental results on this probabilistic DAPC in Appendix B and C In general, probabilistic DAPC performs similarly to the deterministic counterpart, though the traming process is more time and memory intensive. On the other hand, these empirical results show that deviating from the for deterministic DAPC, does not cause significant issues for representation learning in practice if proper regularization is applied.
Gaussian assumption, as is the
Related to this interpretations are VAE-base sequential models (Chung et al. 2015 Hsu al. 2017 Li & Mandi, 2018)) that also use reconstruction and enforce different structures/dynamee latent space. Most of them are designed for the purpose of generating high quality sequence data, while the qualities of their latent representations are mostly not shown for downstream tasks.
in the
RELATED WORK
Mutual information (MI) maximization is principal approach for representation learning (Bell & Sejnowski 1995), where the objective is to maximize the MI estimate between learnt representations and inputs. The currently dominant approach for estimating MI is based contrastive learning. For sequence data, CPC (Oord et al.) (0188) uses representations at current time as a classifier to discrim- inate inputs of nearby frames (positive samples) from inputs of far-away steps inputs from other sequences (negative samples) with a cross-entropy loss; this leads to the noise-contrastive estimation (NCE, Gutmann & Hyvärinen| 2010). Deep InfoMax (DIM, (Hjelm et al., 2018)) generalizes the NCE estimator with few variants, and proposes to maximize MI between global summary features and local features from intermediate layers (rather than the inputs in CPC). SimCLR (Chen al., 2020) extends the contrastive loss to use nonlinear transformation of the representation (than the representation itself) as a classifier for measuring MI. Contrastive Multiview Coding (Tian et al. 2019) generalizes the contrastive learning frame to multiple views. Momentum Contrast (He et al. saves memory with a dynamic dictionary and momentum encoder.
2020
Meanwhile, there have been concerns about the contrastive learning framework. One concern is that postive and negative sample selection sometimes time and memory consuming. To address this issue, BYOL (Grill et al.) 2020) proposes to get rid of negative samples by learning target network in an online fashion and gradually bootstrapping the latent space. Another concern is regarding the MI estimation. Though contrastive learning has MI backbone, Tschannen et al.) (2019) suggests that the inductive bias of the feature extractor and parametrization of estimators might contribute more than the MI estimate itself. Ozair al. (2019) McAllester & Stratos (2020) raise the concern
5

---

Published as a conference paper at ICLR 2021
SNR
DCA
CPC
Pi only
MR
DAPC (MR+PI)
0.3
1.0
5.0
Figure 2: Left panel. Top: the ground-truie -tunt 3D Lorenz attractor. Middle: the 30D non-linearly lifted trajectory. Bottom: corrupted 30D trajectory by white noise with SNR=0.3. Right Panel. 3D trajectories extracted by different methods for three SNR levels: 0.3, 1.0, 5.0.
that the MI lower bound used by contrastive learning might be too loose, and propose use estimate based on Wasserstein distance.
Unlike prior work, our principle for sequence representation learning is to maximize the MI between past and future latent representations, rather than the MI between representations and inputs (or shal- low features of inputs). Partially motivated by the above concerns, our mutual information estimate requires no sampling and is exact for Gaussian random variables. To keep useful information from input, we use masked reconstruction loss which has been effective for sequence data (text and speech), with an intuition resembling that of denoising autoencoders (Vincent et al. 2010
Note that by the data processing inequality, methods that maximize mutual information between current representation and future inputs also implicitly maximizes an upper bound of mutual in- formation between high level representations, since M2 Z Zfuture) MI(Zpast X future). Our method explicitly maximizes the mutual information between high level representations itself, while having another regularization term (masked reconstruction) that maximizes information be- tween current input and current representations. Our results indicate that explicitly modeling the trade-off between the two can be advantageous.
In the audio domain where we will demonstrate the applicability of our method, there has been sig- nificant interest in representation learning for reducing the need for supervised data. Both contrastive learning based (Schneider et al. 2019 Baevski et al. 2019. Jiang et al., 2019) and reconstruction-
based (Chorowski et al. 2019 Chung et al. 201 Song et al. [2019] Wang et al. 2020 Chung & Glass 2020 Ling al. 2020 Liu et al. 2020) methods have been studied, well methods that
incorporate multiple tasks (Pascual et al. 2019a Ravanelli et al. 2020). Our work promotes the use
of different MI estimate and combines different intuitions synergistically.
EXPERIMENTS
4.1 NOISY LORENZ ATTRACTOR
Lorenz attractor ("butterfly effect", see Appendix B} is a 3D time series depicting a chaotic sys- tem (Pchelintsev 2014), as visualized in Figure 2 We design a challenging dimension reduction task for recovering the Lorenz attractor from high dimensional noisy measurements. We first lift the 3D clean signals to 30D with a neural network of 2 hidden layers, each with 128 elu units (Clevert) al. 2015). This lifting network weights and biases drawn randomly from A(O 0.2). In addi- tion, we corrupt the 30D lifted signals with white noise to obtain three different signal-to-nois ratio (SNR) levels, 0.3, 1.0, and 5.0, and use the noisy 30D measurements (see bottom left of Figure 2 input for representation learning methods to recover the true 3D dynamics.
We generate a long Lorenz attractor trajectory using the governing differential equations, and chunk the trajectory into segments of 500 time steps. We use 250, 25 and 25 segments for training, val- idation, and test splits respectively. After the model selection on the validation split based on the R² regression score which measures the similarity between recovered and ground truth trajectories,

---

Published as a conference paper at ICLR 2021
0.10
0.3
PCA
0.10
A/e
DCA
0.05
0.05
0.1
MR
0.00
0.00
5
10
10
Lag
Lag
Figure 3: Different models' R2 score improvements over PCA for three forecasting tasks, each with three different lag values. Left: temperature. Middle: dorsal hippocampus. Right: motor cortex.
Table I: The Rº scores for full reconstruction (FR), full reconstruction with PI (FR+PI), masked reconstruction (MR) and DAPC (MR+PI). We also demonstrate the improvements in percentage brought by adding PI. On average, PI improves full reconstruction by 4.09% and masked recon- struction by 17.22%. Additionally, in the last column, show PI objective alone is not powerful enough to learn predictive components (R² scores are low).
dataset
lag
FR
FR+PI
Improvement
MR
MR+PI
Improvement
PI only
5
0.721
0.725
0.58%
0.705
0.724
2.68%
0.698
Temp
10
0.672
0,685
1.99%
0.672
0.691
2.85%
0.645
15
0.675
0.686
1.64%
0.673
0.707
4.99%
0.632
5
0.252
0.260
3.24%
0.251
0.304
21.25%
0.169
HC
10
0.222
0.231
4.46%
0.222
0.271
21.72%
0.137
15
0.183
0.194
5.97%
0.205
0.232
13.28%
0.085
0.390
0.405
4.03%
0.352
0.519
47.45%
0.239
M1
10
0.372
0.394
5.83%
0.369
0.422
14.64%
0.156
15
0.268
0.293
9.10%
0.241
0.304
26.10%
0,076
our model is applied to the test split. The optimal model uses = 4, 0, 0, 0.1, and = 0.1 is set to balance the importance of PI and the reconstruction error.
We compare DAPC to representative unsupervised methods including DCA (Clark et al.] 2019 CPC (Oord al. 2018), pure PI learning which corresponds DAPC with 3
and masked reconstruction (MR, Wang et al. 2020) which corresponds to DAPC without the PI term. Except for DCA which corresponds maximizing Pl with linear feedforward network, the other methods use bidirectional GRUs (Chung et al.) 2014) for mapping the inputs into feature space (although uni- GRU performs similarly well). A feedforward DNN used for reconstruction in MR and DAPC.
We show the latent representations of different methods in Figure 2 (right panel). DCA fails com- pletely since its encoder network has limited capacity to invert the nonlinear lifting process. CPC is able to recover the lobes, but the recovered trajectory is chaotic. Maximizing PI alone largely ignores the global structure of the data. MR able to produce smoother dynamics for high SNRs. but its performance degrades quickly in the noisier scenarios. DAPC recovers latent representa- tion which has overall similar shapes to the ground truth 3D Lorenz attractor, and exhibits smooth dynamics enforced by the PI term. In Appendix B provide the R² scores for different methods. These results quantitatively demonstrate the advantage of DAPC across different noise levels.
4.2 FORECASTING WITH LINEAR REGRESSION
We then demonstrate the predictive power of learnt representations in downstream forecasting tasks on 3 real-world datasets used by Clark et al. (2019). involving multi-city temperature time series data (Temp, |Beniaguev (2017), dorsal hippocampus study (HC, Glaser et al. (2020)) and motor cortex O'1.orrt et al. (2018)) 8)). For each model, unsupervised representation learning is performed on the training set with uni-directional GRU, which prevents information leakage from the future. After that, we freeze the model and use it feature extractor. The representations each time step are used as inputs for predicting the target at a future time step. As an example, we can extract representation for today's 's weather based past weather only (as the encoder uni-directional), and use it to predict future temperature which is lag days away (a larger lag generally leads to more difficult forecasting task). Following Clark al. (2019), the predictor from the extracted

---

Published as a conference paper at ICLR 2021
feature space to the target is linear mapping, trained on samples of paired current feature and future target, using a least squares loss. We use the same feature dimensionality as in their work, for each dataset. These forecasting tasks are evaluated by the R regression score, which measures the linear predictability. More details can be found in Appendix C
Besides DCA, CPC and MR, we further include PCA and SFA (Wiskott & Sejnowski 2002) (similar to DCA with = for PI estimation), which are commonly used linear dimension reduction methods in these fields. PCA serves as the baseline and we report lt² score improvements from other methods. Figure gives the performances of different methods for the three datasets, with three different lags (the number of time steps between current and future for the forecasting task): 5, 10, and 15. DAPC consistently outperforms the other methods. In Table[]] we show how PI helps DAPC improve over either full reconstruction (e.g., classical auto-encoder) or masked reconstruction, and how reconstruction losses help DAPC improve over PI alone on this task. These results demonstrate that the two types of losses, or the two types of mutual information (MI between input and latent, and MI between past and future) can be complementary to each other.
4.3 PRETRAINING FOR AUTOMATIC SPEECH RECOGNITION (ASR)
A prominent usage of representation learning in speech processing is to pretrain the acoustic model with an unsupervised objective, so that the resulting network parameters serve as good initial- ization for the supervised training phase using labeled data (Hinton et al.] 2012). As supervised ASR techniques have been improved significantly over the years, recent works start focus on pre-training with large amounts of unlabeled audio data, followed by finetuning on much smaller amounts of supervised data, so as reduce the cost of human annotation.
We demonstrate different methods on two commonly used speech corpora for this setup: Wall Street Journal (Paul & Baker 1992) and LibriSpeech (Panayotov et al. 2015). For WSJ, we pretrain on si284 partition (81 hours), and finetune on si84 partition (15 hours) or the si284 partition itself. For Librispeech, we pretrain on the train 960 partition (960 hours) and finetune on the train clean 100 partition (100 hours). Standard dev and test splits for each corpus are used for validation and testing.
In the experiments, we largely adopt the transformers- based recipe from ESPnet (Watanabe et al., 2018), as detailed in Kariia et al. (2019), for supervised finetuning. We provide the details regarding model architecture and data augmentation in Appendix [D. Note that we have spent effort in building strong ASR systems, so that our baseline (without pretraining) already achieves low WERs and improving over is non-trivial, This can seen from the result table where our baseline is often stronger than the best performance from other works. In the pretraining stage, pretrain an encoder of 14 transformer layers, which will be used to initialize the first 14 layers of ASR model. For masked reconstruction, we use 2 frequency masks as in finetuning, but found more time masks can improve pretraining performance. We set the number of time masks to 4 for WSJ, and 8 for LibriSpeech which has longer utterances on average.
The hyperparameters we tune include T, 8, or, 3 and y from our learning objective. We select hy- perparameters which give the best dev set WER, and report the corresponding test WER. In the end, we use T = 4 for estimating the PI term, = 0.05, 3 = 0.005 and set = 2 for WSJ and s 1 for LibriSpeech if we use shifted reconstruction. Since the pretraining objective is a proxy for extracting the structure of data and not fully aligned with supervised learning, we also tune the number of pretraining epochs, which is set for WSJ and for LibriSpeech. Other parameters will be shown in the ablation studies presented in Appendix D
We perform an ablation study for the effect of different variants of DAPC (MR+PI) on the WSJ dataset, and give dev/test WERs in Table 2 We tune the hyperparameters for multi-scale PI and shifted reconstruction for the 15-hour finetuning setup, and observe that each technique can lead to further improvement over the basic DAPC, while combining them delivers the best performance. The same typerparameters are used for the 81-hour finetuning setup, and we find that with more supervised training data, the baseline without pretraining obtains much lower WERs, and pure masked reconstruction only slightly improves over the baseline, while the strongest DAPC variant still achieves 7.9% and 11.7% relative improvements on dev93 and eva192 respectively.
In Table we provide a more thorough comparison with other representation learning methods on the LibriSpeech dataset. We compare with CPC-type mutual information learning methods including
8

---

Published as a conference paper at ICLR 2021
Table Ablation study on different variants of DAPC. We give WERs (%) of ASR mod- els pretrained with different variants on WSJ. Models are pretrained on 81 hours, and fine- tuned on either hours or 81 hours. All re- sults are averaged over 3 random seeds.
Table 3: WERs (%) obtained by ASR models pre- trained with different representation learning meth- ods on the test clean partition of Librispeech. Mod- els are pretrained on 960h unlabeled data and fine- tuned on 100h labeled data. Our results are averaged over 3 random seeds.
Methods
dev93
eva/92
Methods
WER (%)
Finetune on 15 hours
wav2vec Schneider et al.
6.92
W.O. pretrain
12.91=0.36
8.98=0.44
2019)
PI only
12.54:10.32
9.02=0043
discrete vev wav2vec
4.5
MR
12.27:10.23
8.15=0.34
(Baevski et al. 2019)
DAPC
12.31=10.36
7.24-0.20
wav2vec 2.0 (Baevski et al. 2020)
2.3
DAPC +
12.15=#0.35
7.64=0.15
multi-scale PI
DeCoAR Ling et al. 2020
6.10
+ +
TERA-large (Luu et 2020
5.80
11.93::0116
7.68=0.05
shifted recon DAPC + both
MPE (Liu & Huang. 2020)
9.68
11.57=+0.22
7.14
Bidir CPC (Kawakami et al.
8.70
Finetune on 81 hours
2020)
w.o. pretrain
6.34-10.13
3.94
W.O. pretrain
5.11
MR
6.24110.14
3.04
MR
5.02-10.09
DAPC
5.90
3.08 0.08
DAPC
4.86:10.08
DAPC both
5.84
0.08
DAPC+multi-scale Pl+shifted
4.70=10.02
recon
wav2vec (Schneider al. 2019), vq- uq-wav2ver 2ve2vec which performs CPC-type learning with discrete tokens followed by BERT-style learning (Baevski et al. 2019). and wav2vec 2.0 which incorporates
masking into contrastive learning (Baevski al. 2020). We also compare with reconstruction-
type learning approaches DeCoAR (Ling et al. 2020) and TERA (Liu et al. 2020 comparisons with more methods are given in Appendix D Observe that DAPC and its variant achieve lower WER than MR: though our baseline is strong, DAPC still reduces WER by 8%, while MR only improves by 1.76%. This shows the benefit of PI-based learning in addition masked reconstruction. Our method does not yet outperform ug-wav2ved and wav2vec 2.0; we suspect it is partly because our models have much smaller sizes (around 30M weight parameters) than theirs (vq-wav2vee has 150M weight parameters, and wav2vec has 300M weight parameters for the acoustic model, along with very large neural language model) and it is future work to scale up our method. In Appendix [D] provide additional experimental results where the acoustic model targets are subwords. DAPC achieves 15% relative improvement over the baseline (without pretraining), showing that our method is generally effective for different types of ASR systems.
5 CONCLUSIONS
In this work, we have proposed a novel representation learning method, DAPC, for sequence data. Our learnt latent features capture the essential dynamies of the underlying data, contain rich infor- mation of both input observations and context states, and are shown to be useful in a variety of tasks. As future work, we may investigate other predictive information estimators that further alle- viate the Gaussian assumption. On the other hand, more advanced variational inference techniques may applied to the probabilistic version of DAPC to boost the performance. DAPC provides general alternative for mutual information-banl based learning of sequence data and may investigate its potential usage in other domains such as NLP, biology, physics, etc.
REFERENCES
Galen Andrew, Raman Arora, Jeff Bilmes, and Karen Livescu. Deep canonical correlation analysis. In ICML, 2013.
Alexei Baevski. Michael Auli, and AbdeIrahman Mohamed. Effectiveness self-supervised pre- training for speech recognition. arXiv preprint 191 2019.

---

Published as a conference paper at ICLR 2021
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-superv learning of speech representations. arXiv preprint MaXX::2006..14477 2020.
Anthony J Bell and Terrence J Sejnowski. An information-maximiiztiion approach to blind separa- tion and blind deconvolution. Neural computation, 1995.
David Beniaguev. Historical hourly weather data 2012-2017, 2017. URL www. www. www.kaggle... com/ se fishgeneehhhhhe storica aattricca-hourr/ --huurlly---
her-
William Bialek, Iya Nemenman, and Naftali Tishby, Predictability, complexity, and learning. Neu- ral computation, 2001.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. arXiv preprint arXiv:2005.14165 2020.
W. Chan, N. Jaitly, Q V. Le, and O. Vinyals. Listen, attend and spell: A neural network for large vocabulary conversational speech recognition. In ICASSP, 2016.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
J. Chorowski, D. Bahdanau, D. Serdyuk, K. Cho, and Y. Bengio. Attention-based models for speech recognition. In NIPS, 2015.
Chorowski, R. J. Weiss, Bengio, and A. V. D. Oord. Unsupervised speech representation learning using wavenet autoencoders. IEEE Trans. Audio, Speech and Language Process. 2019.
Junyoung Chung, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. In NIPS 2014 Workshop on Deep Learning, December 2014, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances neural information processing systems, pp. 2980-2988, 2015.
Y-A. Chung, W-N, Hsu, H. Tang, and J. Glass. An unsupervised autoregressive model for speech representation learning. In Interspeech, 2019a.
Yu-An Chung and James Glass. Generative pre-training for speech with autoregressive predictive coding. In ICASSP, 2020.
Yu-An Chung, Wei-Ning Hsu, Hao Tang, and James Glass. An unsupervised autoregressive model for speech representation learning. Interspeech, 2019b.
David Clark, Jesse Livezey, and Kristofer Bouchard. Unsupervised discovery of temporal structure in noisy data with dynamical components analysis. In NIPS, 2019.
Djórk-Amé Clevert, Thomas Unterthiner, and Sepp Hochreiter. Fast and accurate deep network learning by exponential linear units (elus). arXiv preprint 1rXiv.1511.07289. 2015.
Peter Dayan and Laurence F Abbott. Theoretical neuroscience: computational and mathematical modeling of neural systems. Computational Neuroscience Series, 2001.
Jacob Devlin, Ming- Wei Chang. Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019.
Joshua Glaser, Ari S Benjamin, Raeed H Chowdhury, Matthew G Perich, Lee E Miller, and Kon- rad Kording. Machine learning for neural decoding. eNeuro, 2020.
A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In ICMI., 2006.
10

---

Published as a conference paper at ICLR 2021
Jean-Basties Grill, Florian Strub, Florent Altché, Corentin Tallec. Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, al. Bootstrap your own latent: A new approach to self-supervised learning. arXiv preprint arXiv:2006.07733 2020.
Michael Gutmann and Aapo Hyvärinen. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In International Conference on Artificial Intelligence and Statistics, 2010.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020.
Geoffrey Hinton, Li Deng, Dong Yu, George Dahl, Abdel rahman Mohamed, Navdeep Jaitly, An- drew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N. Sainath, and Brian Kingsbury, Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal Processing Magazine, 2012.
Devon Hjelm, Alex Fedorov, Samuel Lavoie- Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In ICLR, 2018.
Takaaki Hori, Jaejin Cho, and Shinji Watanabe. End-to-end speech recognition with word-based RNN language models. In SLT, 2018.
Wei-Ning Hsu, Yu Zhang, and James Glass. Unsupervised learning of disentangled and interpretable representations from sequential data. In Advances in neural information processing systems, pp. 1878-1889, 2017.
Dongwei Jiang, Xiaoning Lei, Wubo Li, Ne Luo, Yuxuan Hu, Wei Zou, and Xiangang Li. Im- proving transformer-based speech recognition using unsupervised pre-training. 191::191 [cs.CL], 2019.
Shigeki Karita, Nanxin Chen, Tomoki Hayashi, Takaaki Hori, Hirofumi Inaguma, Ziyan Jiang, Masao Someki, Nelson Enrique Yalta Soplin, Ryuichi Yamamoto. Xiaofei Wang, Shinji Watan- abe, Takenori Yoshimura, and Wangyou Zhang. A comparative study on transformer vs rnn in speech applications. In ASRU, 2019.
Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, and Aaron van den Oord. Learning robust and multilingual speech representations. arXiv preprint 20Xi::2011.1122. 2020.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014.
Alexander Kraskov, Harald Stögbauer, and Peter Grassberger. Estimating mutual information. Phys- ical review E, 2004.
Ming and Paul Vitányi. An introduction to Kolmogorov complexity and its applications. Springer, 2008.
Yingzhen Li and Stephan Mandt. Disentangled sequential autoencoder. arXiv:1803.02991 2018
arXiv preprint
Shaoshi Ling, Yuzong Liu, Julian Salazar, and Katrin Kirchhoff. Deep contextualized acoustic representations for semi-supervised speech recognition. In ICASSP, 2020.
Andy T Liu, Shang- Wen I.,, and Hung-yi Lee. Tera: Self-supervised learning of transformer encoder representation for speech. arXiv preprint arXiv:2007.06028, 2020.
Lu Liu and Yiheng Huang, Masked pre-trainn ire-trained encoder base on joint cic-transformer. arXiv preprint 1171 2020.
David McAllester and Karl Stratos. Formal limitations on the measurement mutual information. In AISTATS, 2020.
11

---

Published as a conference paper at ICLR 2021
Ishan Misra and Laurens der Maaten. Self-supervised learning of pretext-invariant representa- tions. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
Joseph E. O'Doherty, Mariana M. B. Cardoso, Joseph G. Makin, and Philip N. Sabes. Nonhu- man primate reaching with multichannel sensorimotor cortex electrophysiology: Broadband for indy 20160630 01, 2018. /tLp:tt:
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. Xrxiv preprint arXiv: 2018.
Sherjil Ozair, Corey Lynch, Yoshua Bengio, Aaron Van den Oord, Sergey Levine, and Pierre Ser- manet. Wasserstein dependency measure for representation learning. In NIPS, 2019.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: an asr corpus based public domain audio books. In ICASSP, 2015.
Daniel S. Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph, Ekin D. Cubuk, and Quoc Le Specaugment: A simple data augmentation method for automatic speech recognition. In Interspeech, 2019.
Pascual, M. Ravanelli, Serrà, A. Bonafonte, and Bengio. Learning problem-agnostic speech representations from multiple self-supervised tasks. In Interspeech, 2019a.
Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Learn- ing problem-agnostic agnostic speech representations from multiple self-supervised tasks. arXiv preprint arXiv:1904.08 03416. 2019b.
Douglas B. Paul and Janet M. Baker. Proceedings of the workshop Speech and Natural Language, 1992.
The design for the wall street journal- based CSR corpus. In
AN Pehelintsev. Numerical and physical modeling of the dynamics of the lorenz system. Numerical analysis and Applications, 2014.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL-HLT, 2018.
Mirco Ravanelli, Jianyuan Zhong, Santiago Pascual, Pawel Swietojanski, Joao Monteiro, Jan Trmal, and Yoshua Bengio. Multi-task self-supervised learning for robust speech recognition. In ICASSP, 2020.
Steffen Schneider, Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vee: Unsupervised pre-training for speech recognition. Interspeech, 2019.
Rui Shu, Hung H. Bui, Shengjia Zhao, Mykel J. Kochenderfer, and Stefano Ermon. Amortized inference regularization. In NIPS, 2018.
Xingchen Song. Guangsen Wang, Zhiyong Wu, Yiheng Huang, Dan Su, Dong Yu, and Helen Meng. Speech-Xinet: Unsupervised acoustic model pretraining for self-attention networks. ArXiv, 2019.
Steven H Strogatz. Nonlinear dynamics and chaos with student solutions manual: With applications to physics, biology, chemistry, and engineering. CRC press, 2018.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.055: 05849, 2019,
Jakub Tomezak and Max Welling. Vae with a vampprior. In International Conference Artificial Intelligence and Statistics, 2018.
Michael Tschannen, Josip Djolonga, Paul K Rubenstein, Sylvain Gelly, and Mario Lucic. On mutual information maximization for representation learning. ICLR, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NIPS, 2017.
12

---

Published as a conference paper at ICLR 2021
P. Vincent, H. Larochelle, I. Lajoie, Y. Bengio, and P. A. Manzagol. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. Journal of Machine Learning Research, 2010.
Weiran Wang, Raman Arora, Karen Livescu, and Jeff Bilmes. On deep multi-view representation learning. In ICML, 2015.
Weiran Wang, Qingming Tang, and Karen Livescu. Unsupervised pre-training of bidirectional speech encoders via masked reconstruction. ICASSP, 2020.
Watanabe, T. Hori, S. Kim, J. Hershey, and T. Hayashi. Hybrid CTC/attention architecture for end-to-end speech recognition. IEEE Journal Selected Topics in Signal Processing, 2017.
Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Enrique Yalta Soplin, Jahn Heymann, Matthew Wiesner, Nanxin Chen, Adithya Renduchintala, and Tsubasa Ochiai. ESPnet: End-to-end speech processing toolkit, Interspeech, 2018.
Laurenz Wiskott and Terrence Sejnowski. Slow feature analysis: Unsupervised learning of invari-
ances. Neural computation, 2002.
13

---

Published as a conference paper at ICLR 2021
A DERIVATION OF PREDICTIVE INFORMATION
In this section, we give a self-contained derivation of the predictive information.
A multivariate Gaussian random variable X E INN has the following PDF:
1 (-2 (x (x-1)) -11 11)
p
exp
2T
where μ is the mean and Σ is the covariance matrix with Σ 1[[(X B[X]( B[X]]
From the definition of entropy
H(X) f Inp(ddd
can derive the entropy formula for multivariate Gaussian
/ p(z))n
1 (x --)) --))dr
H(X)
exp(
2
/ 1l
p(x)) (-2(1r (1r - 11) - 11)) de
dx
(V2T)N [
1 マタ In(((Te)>
(6)
x
N), = N (((1
((xx)
Exx
Exy Eyy
Consider the joint Gaussian distribution
Y
[Exx
where E RP, and Y RR9 We can plug in the entropy (6) and obtained
MI(X,Y) = H(X) + H(Y - H(X,Y)
1 In (2xe)p|Exxx) + In ((2Te) In((Tee)pt E|)
1
2
- [xxx Evrl
For latent sequence z juture = {[++1,--.. 2. Based on our stationarity assumption, all the length-27 windows of states within the range are drawn from the same Gaussian distribution with covariance Exr (Z), and similarly for all the length-I windows. As result and under the stationary assumption, H (2!ust) future.
{21, 22, where 2; Rdd we define Zpast f2-T+1, 21 and
H H(7 (2 = (2mm))ed(1r(( H(Z z zfature) In((2r)2Tdd(S. (Z))), and
Ir = MI(Ziaat 2.fuuure
H (Zfast) + 11 (2, (Z,uuure H H2Zaast Zufuure
1
1
1
2 In( in((2 Td 1T (2)l) + (n((2 (2)1 - In (n((2Te)222 27'd Exr (2)))
In (Z)) In (Z)
The predictive information In only depend on T but not specific time index t.
B LORENZ ATTRACTOR
The Lorenz attractor system (also called "butterfly effect") is generated by the following differential equations: (Strogatz) 2018 Clark et al. 2019)
da = a x)
dy (p z) 7
dz Ty 3z
14

---

Published as a conference paper at ICLR 2021
the 3D coordinates, and we 10, 8/3, 28. The integration step for solving the system of equations 5 x 10
where (x, y, y. 2)
We lift the 3D trajectory into 30D using nonlinear neural network with 2 hidden layers, each with 128 neurons and the Exponential Linear Unit (Clevert et al.) 2013). We further perturb the 30D trajectory with additive Gaussian noise obtain datasets of three different Signal-to-Noise Ratios (SNRs, ratio between the power signal and the power of noise): 0.3, 1.0, and 5.0. The smaller the SNR is, the more challenging it recover the clean 3D trajectory (see the left panel of Figure for the comparison between noisy 30D trajectory and the clean 30D trajectory).
Table The R2 scores of recovered 3D trajectory noisy Lorenz attractor by different methods.
SNR
DCA
CPC
PI
MR
DAPC-prob DAPC-det
0.3
0.084
0.676
0.585
0.574
0.865
0,816
1.0
0.153
0.738
0.597
0.885
0.937
0,943
5.0
0.252
0.815
0.692
0.929
0.949
0.949
SNR
DCA
DAPC (det)
DAPC (prob)
0.3
1.0
5.0
Figure 4: Recovery of 3D trajectory of noisy Lorenz attractor by different methods.
We compare both deterministic and probabilistic DAPC against representative unsupervised meth- ods including DCA (Clark et al. 2019), CPC (Oord et al. 2018). pure PI learning which corresponds to DAPC with = 0, and masked reconstruction (MR) (Wang et al. 2020). Except for DCA which corresponds to maximizing PI with linear orthogonal feedforward net, the other methods use bidi- rectional GRU (Chung et al.) (014)) for mapping the inputs into feature space (although uni-GRU performs similarly well). A feedforward DNN is used for reconstruction in MR and DAPC.
More specifically, CPC, MR, DAPC all use the bidirectional GRU where the learning rate is 0.001, and dropout rate is 0.7. Our GRU has 4 encoding layers with hidden size 256. The batch size (20, 500, 30). For CPC, the temporal lag k=4. For DAPC, = 0.1, T = 4, s O, a 0, = 0.1. For masked reconstruction, we use at most 2 masks on the frequency axis with width up to and at most masks on the time axis with width up to 40. The DNN decoder has 3 hidden layers, each with size 512. DCA's setup is completely adopted from Clark et al. (2019). The same architectures are used in the forecasting tasks in Appendix
Figure provides qualitatively results for the recovered 3D trajectories by different methods (Fig- ure g in the main text contains a subset of the results shown here). Observe that DCA fails in this scenario since feature extraction network has limited capacity to invert the nonlinear lifting process. CPC is able to recover the 2 lobes, but the recovered signals struction able to produce smoother dynamics for high SNRs, but its performance degrades quickly in the more noisy scenarios. Both deterministic and probabilistic DAPC recover latent representa- tion which has overall similar shapes to the ground truth 3D Lorenz attractor, and exhibits smooth dynamics enforced by the PI term.
chaotic. Masked recon-
We quantitatively measure the recovery performance with the R² score, which defined as co- efficient of determination. R² score normally ranges from 0 to where means the perfect fit.
15

---

Published as a conference paper at ICLR 2021
Negative scores indicate that the model fits the data worse than horizontal hyperplane. The R results are given in Table 4 Our results quantitatively demonstrate the clear advantage DAPC across different noise levels.
Table 5: The R2 scores for the ablation study of (deterministic) DAPC for Lorenz attractor.
SNR
Full Recon
uni-GRU
Regular
0.3
0,803
0.857
0,865
1.0
0.812
0.905
0.937
5.0
0.852
0.903
0.949
Table 6: The R² scores for full reconstruction only and full reconstruction with PI.
SNR
Full Recon only
Full Recon with
0.3
0.441
0.803
1.0
0.737
0.812
5.0
0.802
0.852
SNR
Pl only
Full recon only
Full recon Pl
MR
DAPC (MR PI)
0.3
1.0
5.0
Figure 5: Illustration of how PI can improve both full reconstruction and masked reconstruction (MR). We can observe that PI can greatly improve the recovery quality, especially when SNR is low (very noisy).
Table 7: The R² scores for CPC with different temporal lags (k).
SNR
k=2
k=4
k=6
k=8
k=10
0.3
0.477
0.676
0.663
0.658
0.608
1.0
0.556
0.738
0.771
0.721
0.642
5.0
0.652
0.815
0.795
0.775
0.717
We also give an ablation study on several components of DAPC in Table where we attempt full reconstruction without masking, masked reconstruction with unidirectional encoder uni-GRU, and the regular setup (masked reconstruction + bi-GRU). Using full reconstruction yields worse results than using masked reconstruction all noise levels, while uni-GRU degrades the performance less.
We show Table and Figure 5 how PI can improve both full reconstruction and masked recon- struction. In Table when SNR=0.3, PI can greatly boost the performance of full reconstruction. We also tuned the temporal lag parameter w.r.t both quantitative and qualitative results (Table and Figure 6). CPC performance starts deteriorate after k=8, while k=4, 6, 8 have similar results. Based on the R² scores, we select k=4 as our final temporal lag. Similar tuning is also performed for CPC the downstream forecasting experiments in Appendix C
16

---

Published as a conference paper at ICLR 2021
SNR
k=2
k=4
k=6
k=8
k=10
0.3
1.0
5.0
Figure 6: Qualitative recovery results by CPC w.r.t. different temporal lags (k).
C DOWNSTREAM FORECASTING WITH LINEAR REGRESSION
For each of the three datasets used in downstream forecasting tasks, we divide the original dataset into trainingfvaliddtiion/esssgg splits. Unsupervised representation learning is performed on training split, validated the validation split and learns encoder e(-).
Denote the test sequence X = { Z2...., IL). The learnt e(-) transforms X into a sequence Z == = 22 ..., F of the same length. Note that use uni-directional GRU for representation learning so that and no future information is leaked. For the forecasting tasks, 2; will be used to predict target i/4 which corresponds to event of interest. For the multi-city temperature dataset (Beniaguev) 2017). Y: represents future multi-city temperatures, i.e., yi = Ij with log > For the hippocampus study (Glaser al. (020),, 2i; the multi-neuronal spiking activity of 55 single units recorded rat hippocampal CAT and y; is a future location of the rat. In the motor cortex dataset (O' O''Dhherty 2018)) x, 's collected from multi-neuronal spiking activity of 109 single units recorded in monkey primary motor cortex (MI), and y. are future behavior variables such cursor kinematics. The problems tend be more challenging with larger lag.
0.10
0.3
PCA
0.2
DCA
CPC
0.05
0.05
0.1
(det)
DAPC (prob)
0.00
10 Lag
15
10
15
10
15
Lag
Lag
Figure 7: Different models' 12 score improvements over PCA: Left) temperature, Middle) dorsal hippocampus and Right) motor cortex datasets. Lag is the number of time steps the future event is ahead of the current latent state.
0.10
0.00
PCA
SF4
&
0.05
0.00
0.05
DCA
MR
DAPC (6e)
000
0.00
DAPC (pok)
10
15
Log
Lag
Lus
Las
5
10
15
20
Figure 8: On temperature dataset, analyze the performances of different latent dimensions (di- mension of 2,): 5, 10, 15, 20. R corresponds to R2 score improvement over PCA.
The performance for forecasting tasks is measured by the linear predictability from 2; to y:. Specif- ically, solve the linear regression problem with inputs being the (zi, ys) pairs, and measure the
17

---

Published as a conference paper at ICLR 2021
RR score between the prediction and ground-truth target. We use the R score from the PCA pro- jection as a baseline, and provide the improvements over PCA obtained by different representation leamning methods in Figure 7 (so PCA's A is 0). Both deterministic and probabilistic DAPC con- sistently outperform other methods across all three datasets, with the deterministic version slightly outperforming the probabilistic one. Additionally, we provide sensitivity study of the latent dimen- sionality for all methods in Figure [] and DAPC outperforms others consistently across different dimensionalities. Table 8 shows the temporal lag tuning for CPC on the temperature dataset.
Table The 1t2 scores for CPC with different temporal lags (k) on the temperature dataset.
k=2
k=4
k=6
k=8
k=10
5
0,685
0.701
0.690
0.687
0.681
10
0.661
0.663
0.675
0.631
0.622
15
0.633
0.645
0.637
0.634
0.617
D ADDITIONAL RESULTS FOR AUTOMATIC SPEECH RECOGNITION (ASR)
The acoustic model is trained with multi-task objective (Watanabe et al. 2017) which combines attention Chorowski et al. (2015): Chan et al. (2016) and CTC (Graves al. 2006) losses, for predicting the output character sequence. We extract S0D fbank features plus 3D pitch features from audio, with frame size of 25ms and hop size of 10ms. Every 3 consecutive frames stacked to obtain the input sequence for the acoustic model. During ASR finetuning, the encoder shared by both attention and CTC consists of 14 transformer layers for WSJ and 16 layers for LibriSpeech, while the decoder consists of 6 transformer layers. All attention operations use 4 heads of 64 dimensions each, and the output of multi-head attention goes through one- one-hideen-laye position -iie feed- forward network of 2048 ReLU units, before it is fed into the next layer. During finetuning, we apply SpecAugment (Park al. 2019) to reduce overfitting, with max time warp set to 5 (frames), two frequency masks of width up to 30 frequency bins, and two time masks of width up to 40 frames. We use the Adam optimizer with a warmup schedule for the learning rate. Weight parameters of the last 10 finetuning epochs is averaged to obtain the final model. For word-level decoding, we use word RNNLM trained on the language model training data of each corpus, with a vocabulary size of 65K for WSJ, and 200K for LibriSpeech. We use the lookahead scores derived from word RNNLM during beam search, for selecting promising character tokens at each step, as done by Hori et al. 20118. A beam size of 20 is used for decoding.
In Table g we provide a thorough comparison with other representation learning methods on the LibriSpeech dataset. We compare with CPC-type mutual information learning methods including wav2vec (Schneider et al. 2019)), vq-wav2vec which performs CPC-type learning with discrete to- kens followed by BERT-stype learning (Baevski al. 2019), and more recent extension wav2vec 2.0 (Baevski et al. 22020). We also compare with two reconstruction-type learning approaches De- CoAR (Ling et al. (202)) and TERA (Liu et al. 2020). Note that TERA quite similar to MR (Wang et al. 2020) in performing masked reconstruction, although it adds recurrent layers to transformer- learnt representations for finetuning, while our implementation of MR uses a pure transformer-basee architecture throughout. We believe the advantage of MR over TERA mainly comes from the acous- tic model (attention vs. CTC) and a stronger language model (RNNLM vs. n-gram). Observe that DAPC and its variant achieve lower WER than MR: though our baseline is strong, DAPC still re- duces WER by 8%, while MR only improves by 1.76%. This shows the benefit of Pl-based learning in addition to masked reconstruction. Compared to Table Table incluude more recent works that are related to our DAPC. Futhermore, to show our improvement is robust to details of ASR recipe, include the comparison between baseline (without pretraining), MR, and DAPC when the ASR recipe uses 5000 unigrams as token set and decodes with token-level RNNLM. These results are de- noted with "(sub-word)" in Table g The relative merits between methods are consistent with those observed for the character recipe. DAPC obtains a relative improvement of 15% over the bascline on test clean (6.81% + 5.79%).
Furthermore, we compare DAPC with other state-of- the- methods on WSJ in Table 10 Compar- isons among different methods based on their key features are shown in Table I1
18

---

Published as a conference paper at ICLR 2021
Table 9: WER results of different methods on LibriSpeech. All representation methods are pre- Irain_clean_100 (100h). For MR and DAPC, the
trained the full corpus (960h) and finetuned default ASR recipe uses characters as token set and decodes with word RNNLM. For results denoted with "(sub-word)), the ASR recipe uses 5000 unigrams as token set and decodes with token-level RNNLM. All the results are averaged over 3 seeds.
Methods
dev_clean test_clean
wav2vec Schneider et (Baevski et al.
2019
6.92
discrete BERT+VQ- +vq--=v2-eaeee
2019
4.0
4.5
wav2vec 2.0 Baevski et 2020)
2.1
2.3
DeCoAR (Ling et al.
2020)
6.10
TERA-large (Lu ct al. 2020
5,80
MPE (Liu & Huang 2020)
8.12
9.68
Bidir CPC Kawakami et al. 2020
8.86
8.70
MR (Wang et 2020
4.66
5.02
MR (sub-word)
5.57
6.18
w.o. pretrain
4.84
5.11
DAPC
4.52
4.86
DAPC+multi-scale
4.46
4.77
DAPC+shifted recon
4.50
4.80
DAPC+multi scale Pl+shifted recon
4.42
4.70
w.o. pretrain (sub-word)
6.16
6.81
DAPC (sub-word)
5.50
5.79
Table 10: WERs of models pretrained on 81h split si284 and finetuned on 81h split si284.
Methods
dev93 WER (%)
eva/92 WER (%)
DeCoAR (Ling al. 2020
8.34
4.64
MPE
Liu & Huang
2020)
6.79
4.26
MR
Wang et al. 2020)
6.24
3,84
w. pretrain (Attentio+CTC)
6.34
3.94
DAPC + both
5.84
3.48
Table 11: We compare different methods based on their key features: gencratie/ddierrminnatie. contresiveiionncootrasiiie, whether using past-future mutual information estimation (p-f MI), and masking.
generative contrastive
p-f MI MI
masking
SFA
y
DCA
n
n
y
n
wav2vec
y
vq-wav2vec
y
DeCoAR
y
TERA
Y
MPE
π
Y
bidir CPC
y
DAPC
n
y
19


Reviews:
[{"review_id": "wn9e44RKFDz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper builds on the prior work of Dynamical Components Analysis (DCA) which maximizes the mutual information between past and future temporal windows around the current time step, referred to as the the Predictive Information (PI) loss.  \nIn this paper, the PI loss is used to train a neural encoder that learns continuous latent representations of input sequences. The PI loss is regularized to have orthogonal latent space. It is further improved by summing the PI loss applied at multiple scales, and by adding a masked reconstruction (MR) loss. The paper presents results on three domains, with speech recognition as the main one. \nThe paper presents a nice extension of DCA, as well as a probabilistic interpretation motivated by the variational autoencoder (VAE) framework. It is clearly written with good citations of previous work. \n\nOn the other hand, The experimental section requires more work. Here are some directions for improvements: \n1) Adding the MR loss, combines another effective pre-training mechanize from previous work. I'm assuming that in tables 1 and 2, DAPC (written alone) refers to models optimizing the PI loss without MR (please correct me if this is not true). If my understanding is correct, the PI loss doesn't require MR to avoid degenerate solutions. Further analysis for the ASR models would help the readers understand different cases when the MR is required. \n2) Following on the previous point, table 1 is missing the MR only results in the upper section, and missing the DAPC only in the section section. This is important to understand the relative contribution of each pre-training loss.\n3) In table 2, the reported results are using a 30M parameter model compared to larger previously published models (~150M parameters). Given that the results of the proposed model is worse than those larger models, it is not clear why the authors didn't pre-train larger models of similar capacity for fairer comparison to their proposed approach. It is known that pre-training larger models yield better final representations for downstream tasks. Would it be the case for DAPC? \n4) The pre-training on librispeech only is done for a 1 epoch. This seems pretty short compared to prior pre-training work on the same dataset (correct me if this is not accurate). Do the learned representations using DPAC stop improving after 1 epoch of pre-training? If possible, please share results of pre-training for larger number of epochs for the DPAC only, MR only, and combined loss. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "vl4mag7rZyS", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes Deep Autoencoding Predictive Components (DAPC), a self-supervised representation learning approach for sequential data. In this approach, the model learns to maximize the predictive information, which is the mutual information between past and future time windows. In order to avoid degenerate solutions, the proposed approach relies on a second loss that optimizes masked reconstructions.\n\nStrengths:\n* The paper is written clearly and is easy to follow. \n* The proposed method seems to outperform the baselines across all considered tasks.\n* The usage of predictive information as an objective on the latent representations is well motivated and seems to improve performance over a non-regularized autoencoder.\n\nWeaknesses:\n* The paper focuses exclusively on one interpretation of their approach, in which the optimization of the predictive information is the main objective and the reconstruction loss merely pushes the model to avoid degenerate solutions. Explicitly considering the reverse interpretation, in which the predictive information is a regularizer applied to an autoencoder to enforce a better latent structure, could provide more depth to the paper: \n    1) Following this interpretation, it would make sense to include more references in the related works section describing autoencoding approaches for representation learning.\n    2) Based on this interpretation, it would also be interesting to investigate the effect that the predictive information regularizer has on the autoencoding model. This is partially done in section 4.3, but missing from the other two experiments. \n    3) Based on this interpretation, the probabilistic interpretation of DAPC (sec 2.3) can be reduced to describing a standard VAE with additional regularization on the latent space in the form of predictive information.\n\n* The experiments show strong results for the proposed DAPC. However, it remains unclear how much fine-tuning has been done for the DAPC, especially in comparison to the baselines. For example, I would imagine that the CPC model might perform better on the lorenz attractor (Fig. 2) when optimizing the parameter k (which influences the temporal lag between the positive samples). The experiments would provide a much stronger point for the superiority of DAPC if there was a clear outline of the considered hyperparameters for all models. Additionally, details, such as the employed model architectures, that would be needed for reproducing the results are missing.\n\nAdditional comment:\nIn the related work section, the paper states “Unlike prior work, our principle for sequence representation learning is to maximize the MI between the past and future latent representations, rather than the MI between representations and inputs.”. That characterization of previous work is not entirely correct: CPC, for example, optimizes the MI $I(x_{t+k}, c_t)$ between the future input $x_{t+k}$ and the current representation $c_t$. Through the data processing inequality, this will result in the model implicitly also optimizing the MI $I(c_{t+k}, c_t)$ between the future and current representation (Tschannen et al., 2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "TzHhD1-gade", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is an interesting paper with several proposed theories on how to improve self supervised learning objectives through a \"predictive information\" type objective, combined with a masked reconstruction loss. The hypothesis is interesting, and has some benefits, specifically that it does not require contrasting with negatives unlike many recent SSL methods. \n\nMost experiments performed on the speech recognition task. While the theory and discussion looks plausible, the experiments are somewhat on the weak side. Specifically:\n\n1) the differences in word error rates in the ablations are pretty small. The authors claim to run 3 seeds for each result and report the mean - can we see the standard deviation?\n2) why not compare to wsj results from cpc-style training such as wav2vec and vq-wav2vec?\n3) eval on Librispeech is only on the clean set. Why not show results on noisy, where differences between techniques should be somewhat more apparent?\n4) there are newer models to compare against, e.g. wav2vec 2.0\n5) the authors claim that they perform worse than e.g. BERT + vq-wav2vec because their model is smaller. But they use standard transformers as their backbone - it should be straight forward to scale this up and have apples to apples comparison.\n6) this is positioned as a general SSL technique - what about experiments in other modalities like nlp or vision?\n7) is \"DAPC\" just the PI loss without masked reconstruction? (i am not sure if \"shifted recon\" just adds the shifting, or it adds the entire reconstruction dimension). I think that is the case, but i am not 100% sure - maybe its worth to make this more clear.\n\nOverall I liked the premise of the paper but unfortunately the experiments left me unconvinced in the value of this approach and its various components\n\nUpdate:\nthanks for your reply. i remain not fully convinced of the improvements with the proposed method and i look forward to additional experiments in NLP. i do think this approach is valuable for additional study however and updated my rating to reflect this", "extra_scores": {}, "extra_text": {}}, {"review_id": "TtEng02JQ4", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper contributes to the growing body of work on self-supervised representation learning approaches. It appears to have a strong theoretical foundation based on the concept of predictive information. The results seem competitive and there are visible efforts to place the method in the context of existing work.\n\nPros:\n- Strong theoretical foundation, well explored.\n- Competitive results in multiple contexts.\n- Mostly clear with adequate detail for the method and experiments.\n- Originality: combining DCA estimation of PI with additional regularization terms + shifted masked reconstruction.\n\nCons:\n- Although there is plenty of discussion of related work and how the current method fits in with existing approaches, it would be helpful to tie these other approaches back more explicitly to the compared methods in Fig 3 / Tab 1 / Tab 2. Perhaps it would be possible even to add a table comparing the methods discussed across various dimensions (e.g. discriminative/generative, contrastive, mutual information entities [representation vs input, past vs future], model size, and other important characteristics in which they differ).\n- It is not clear whether the results for other published methods are taken from their respective publications or reproduced. If taken from publications, are the models used comparable?\n- I would like more discussion on the pros and cons of using masked reconstruction versus full reconstruction (as in a canonical auto-encoder) as a way to avoid learning degenerate representations.\n\nOverall, I think this paper would be of interest to the community.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper combines a few different ideas for representation learning on sequential data and is able to achieve competitive WER on the Librispeech ASR dataset. I appreciate the fact that the authors engaged with reviewers and tried to improve the paper. While I get a sense that the final system has many moving parts, I believe the paper meets the bar for acceptance at ICLR.
Decision: Accept (Poster)