Representation Learning for Sequence Data with Deep Autoencoding Predictive Components

1 INTRODUCTION . Self-supervised representation learning methods aim at learning useful and general representations from large amounts of unlabeled data , which can reduce sample complexity for downstream supervised learning . These methods have been widely applied to various domains such as computer vision ( Oord et al. , 2018 ; Hjelm et al. , 2018 ; Chen et al. , 2020 ; Grill et al. , 2020 ) , natural language processing ( Peters et al. , 2018 ; Devlin et al. , 2019 ; Brown et al. , 2020 ) , and speech processing ( Schneider et al. , 2019 ; Pascual et al. , 2019b ; Chung & Glass , 2020 ; Wang et al. , 2020 ; Baevski et al. , 2020 ) . In the case of sequence data , representation learning may force the model to recover the underlying dynamics from the raw data , so that the learnt representations remove irrelevant variability in the inputs , embed rich context information and become predictive of future states . The effectiveness of the representations depends on the self-supervised task which injects inductive bias into learning . The design of self-supervision has become an active research area . One notable approach for self-supervised learning is based on maximizing mutual information between the learnt representations and inputs . The most commonly used estimate of mutual information is based on contrastive learning . A prominant example of this approach is CPC ( Oord et al. , 2018 ) , where the representation of each time step is trained to distinguish between positive samples which are inputs from the near future , and negative samples which are inputs from distant future or other sequences . The performance of contrastive learning heavily relies on the nontrivial selection ∗Work done during an internship at Salesforce Research . †Work done while Weiran Wang was with Salesforce Research . 1Code is available at https : //github.com/JunwenBai/DAPC . of positive and negative samples , which lacks a universal principle across different scenarios ( He et al. , 2020 ; Chen et al. , 2020 ; Misra & Maaten , 2020 ) . Recent works suspected that the mutual information lower bound estimate used by contrastive learning might be loose and may not be the sole reason for its success ( Ozair et al. , 2019 ; Tschannen et al. , 2019 ) . In this paper , we leverage an estimate of information specific to sequence data , known as predictive information ( PI , Bialek et al. , 2001 ) , which measures the mutual information between the past and future windows in the latent space . The estimate is exact if the past and future windows have a joint Gaussian distribution , and is shown by prior work to be a good proxy for the true predictive information in practice ( Clark et al. , 2019 ) . We can thus compute the estimate with sample windows of the latent sequence ( without sampling negative examples ) , and obtain a well-defined objective for learning the encoder for latent representations . However , simply using the mutual information as the learning objective may lead to degenerate representations , as PI emphasizes simple structures in the latent space and a powerful encoder could achieve this at the cost of ignoring information between latent representations and input features . To this end , we adopt a masked reconstruction task to enforce the latent representations to be informative of the observations as well . Similar to Wang et al . ( 2020 ) , we mask input dimensions as well as time segments of the inputs , and use a decoder to reconstruct the masked portion from the learnt representations ; we also propose variants of this approach to achieve superior performance . Our method , Deep Autoencoding Predictive Components ( DAPC ) , is designed to capture the above intuitions . From a variational inference perspective , DAPC also has a natural probabilistic interpretation . We demonstrate DAPC on both synthetic and real datasets of different sizes from various domains . Experimental results show that DAPC can recover meaningful low dimensional dynamics from high dimensional noisy and nonlinear systems , extract predictive features for forecasting tasks , and obtain state-of-the-art accuracies for Automatic Speech Recognition ( ASR ) with a much lower cost , by pretraining encoders that are later finetuned with a limited amount of labeled data . 2 METHOD The main intuition behind Deep Autoencoding Predictive Components is to maximize the predictive information of latent representation sequence . To ensure the learning process is tractable and non-degenerate , we make a Gaussian assumption and regularize the learning with masked reconstruction . In the following subsections , we elaborate on how we estimate the predictive information and how we design the masked reconstruction task . A probabilistic interpretation of DAPC is also provided to show the connection to deep generative models . 2.1 PREDICTIVE INFORMATION . Given a sequence of observations X = { x1 , x2 , ... } where xi ∈ Rn , we extract the corresponding latent sequence Z = { z1 , z2 , ... } where zi ∈ Rd with an encoder function e ( X ) , e.g. , recurrent neural nets or transformers ( Vaswani et al. , 2017 ) .2 Let T > 0 be a fixed window size , and denote Zpastt = { zt−T+1 , ... , zt } , Z future t = { zt+1 , ... , zt+T } for any time step t. The predictive information ( PI ) is defined as the mutual information ( MI ) between Zpastt and Z future t : MI ( Zpastt , Z future t ) = H ( Z past t ) +H ( Z future t ) −H ( Z past t , Z future t ) ( 1 ) 2In this work , the latent sequence has the same length as the input sequence , but this is not an restriction ; one can have a different time resolution for the latent sequence , using sub-sampling strategies such as that of Chan et al . ( 2016 ) . where H is the entropy function . Intuitively , PI measures how much knowing Zpastt reduces the uncertainty about Zfuturet ( and vice versa ) . PI reaches its minimum value 0 if Z past t and Z future t are independent , and it is maximized if Zfuturet is a deterministic function of Z past t . Different from the MI estimate used by contrastive learning , which measures the MI between representation at each single time step and its future inputs , predictive information measures the MI between two windows of T time steps collectively . The window size T used in PI estimation reflects the time resolution for which the time series is more or less stationary . PI was designed as a general measure of the complexity of underlying dynamics which persists for a relatively long period of time ( Li & Vitányi , 2008 ) . Furthermore , PI is aware of temporal structures : different dynamics could lead PI to converge or diverge even if they look similar . These virtues of PI contribute to the versatility of this measure . The use of PI beyond a static complexity measure ( and as a learning objective ) is done only recently in machine learning by Clark et al . ( 2019 ) , which proposes to learn a linear dimensionality reduction method named Dynamical Component Analysis ( DCA ) to maximize the PI of the projected latent sequence . One approach for estimating PI is through estimating the joint density P ( Zpastt , Z future t ) , which can be done by density estimation methods such as k-NN and binning ( Dayan & Abbott , 2001 ; Kraskov et al. , 2004 ) . However , such estimates heavily rely on hyperparameters , and it is more challenging to come up with differentiable objectives based on them that are compatible with deep learning frameworks . Our approach for estimating PI is the same as that of DCA . Assume that every 2T consecutive time steps { zt−T+1 , ... , zt , ... , zt+T } in the latent space form a stationary , multivariate Gaussian distribution . Σ2T ( Z ) is used to denote the covariance of the distribution , and similarly ΣT ( Z ) the covariance of T consecutive latent steps . Under the stationarity assumption , H ( Z past t ) remains the same for any t so we can omit the subscript t , and H ( Zpast ) is equal to H ( Zfuture ) as well . Using the fact that H ( Zpast ) = 12 ln ( 2πe ) dT |ΣT ( Z ) | , PI for the time series z reduces to IT = MI ( Z past , Zfuture ) = ln |ΣT ( Z ) | − 1 2 ln |Σ2T ( Z ) | . ( 2 ) Detailed derivations can be found in Appendix A . It is then straightforward to collect samples of the consecutive 2T -length windows and compute the sample covariance matrix for estimating Σ2T ( Z ) . An empirical estimate of ΣT ( Z ) corresponds to the upper left sub-matrix of Σ2T ( Z ) . Recall that , under the Gaussian assumption , the conditional distribution P ( Zfuture|Zpast ) is again Gaussian , whose mean is a linear transformation of Zpast . Maximizing IT has the effect of minimizing the entropy of this conditional Gaussian , and thus reducing the uncertainty of future given past . Though our estimation formula for PI is exact only under the Gaussian assumption , it was observed by Clark et al . ( 2019 ) that the Gaussian-based estimate is positively correlated with a computationally intensive estimate based on non-parametric density estimate , and thus a good proxy for the full estimate . We make the same weak assumption , so that optimizing the Gaussian-based estimate improves the true PI . Our empirical results show that representations learnt with the Gaussian PI have strong predictive power of future ( see Sec 4.2 ) . Furthermore , we find that a probabilistic version of DAPC ( described in Sec 2.3 ) which models ( Zpast , Zfuture ) with a Gaussian distribution achieves similar performance as this deterministic version ( with the Gaussian assumption ) . We now describe two additional useful techniques that we develop for PI-based learning . Multi-scale PI One convenient byproduct of this formulation and estimation for IT is to reuse Σ2T ( Z ) for estimating IT/2 , IT/4 and so on , as long as the window size is greater than 1 . Since the upper left sub-matrix of Σ2T ( Z ) approximates ΣT ( Z ) , we can extract ΣT ( Z ) from Σ2T ( Z ) without any extra computation , and similarly for ΣT/2 ( Z ) . We will show that multi-scale PI , which linearly combines PI at different time scales , boosts the representation quality in ASR pretraining . Orthogonality penalty Observe that the PI estimate in ( 2 ) is invariant to invertible linear transformations in the latent space . To remove this degree of freedom , we add the penalty to encourage latent representations to have identity covariance , so that each of the d latent dimensions will have unit scale and different dimensions are linearly independent and thus individually useful . This penalty is similar to the constraint enforced by deep canonical correlation analysis ( Andrew et al. , 2013 ) , which was found to be useful in representation learning ( Wang et al. , 2015 ) . 2.2 MASKED RECONSTRUCTION AND ITS SHIFTED VARIATION . The PI objective alone can potentially lead to a degenerate latent space , when the mapping from input sequence to latent sequence is very powerful , as the latent representations can be organized in a way that increases our PI estimate at the cost of losing useful structure from the input . This is also observed empirically in our experiments ( see Sec 4.1 ) . To regularize PI-based learning , one simple idea is to force the learnt latent representations to be informative of the corresponding input observations . For this purpose , we augment PI-based learning with a masked reconstruction task . Masked reconstruction was first proposed in BERT ( Devlin et al. , 2019 ) , where the input text is fed to a model with a portion of tokens masked , and the task is to reconstruct the masked portion . Wang et al . ( 2020 ) extended the idea to continuous vector sequence data ( spectrograms ) . The authors found that randomly masking input dimensions throughout the sequence yields further performance gain , compared to masking only consecutive time steps . We adopt their formulation in DAPC to handle continuous time series data . Given an input sequence X of length L and dimensionality n , we randomly generate a binary mask M ∈ Rn×L , where Mi , j = 0 indicates Xi , j is masked with value 0 and Mi , j = 1 indicates Xi , j is kept the same . We feed the masked inputs to the encoder e ( · ) to extract representations ( in Rd ) for each time step , and use a feed-forward network g ( · ) to reconstruct the masked input observations . e ( · ) and g ( · ) are trained jointly . The masked reconstruction objective can be defined as R = || ( 1−M ) ( X − g ( e ( X M ) ) ) ||2fro . ( 3 ) Figure 1 gives an illustration for the masked spectrogram data . We randomly generate nT time masks each with width up to wT , and similarly nF frequency masks each with width up to wF . In our experiments , we observe that input dimension masking makes the reconstruction task more challenging and yields higher representation quality . Therefore , this strategy is useful for general time series data beyond audio . We introduce one more improvement to masked reconstruction . Standard masked reconstruction recovers the masked inputs for the same time step . Inspired by the success of Autoregressive Predictive Coding ( Chung & Glass , 2020 ) , we propose a shifted variation of masked reconstruction , in which the latent state zi is decoded to reconstruct a future frame xi+s ( than xi ) . Formally , the shifted masked reconstruction loss Rs is defined as Rs = || ( 1−M→s ) ( X→s − g ( e ( X M ) ) ) ||2fro ( 4 ) where → s indicates right-shifting s time frames while the input dimensions remain unchanged . When s = 0 , Rs reduces to the standard masked reconstruction objective , and in the ASR experiments we find that a nonzero s value helps . We ensure no information leakage by enforcing that the portion to be reconstructed is never presented in the inputs . As indicated by Chung et al . ( 2019b ) , predicting a future frame encourages more global structure and avoids the simple inference from local smoothness in domains like speech , and therefore helps the representation learning . To sum up , our overall loss function is defined as the combination of the losses described above : min e , g Ls , T ( X ) =− ( IT + αIT/2 ) + βRs + γRortho ( 5 ) where α , β , γ are tradeoff weights and Rortho = ||Σ1 − Id||2fro is the orthonormality penalty discussed in Sec . 2.1 , with Σ1 ∈ Rd×d corresponding to the top left sub-matrix of Σ2T estimated from the latent sequence Z = e ( X M ) . The whole framework of DAPC is illustrated in Figure 1 .

Summary: This paper proposes Deep Autoencoding Predictive Components (DAPC), a self-supervised representation learning approach for sequential data. In this approach, the model learns to maximize the predictive information, which is the mutual information between past and future time windows. In order to avoid degenerate solutions, the proposed approach relies on a second loss that optimizes masked reconstructions.