On Predictive Information Sub-optimality of RNNs

1 INTRODUCTION . Remembering past events is a critical component of predicting the future and acting in the world . An information-theoretic quantification of how much observing the past can help in predicting the future is given by the predictive information ( Bialek et al. , 2001 ) . The predictive information is the mutual information ( MI ) between a finite set of observations ( the past of a sequence ) and an infinite number of additional draws from the same process ( the future of a sequence ) . As a mutual information , the predictive information gives us a reparameterization independent , symmetric , interpretable measure of the co-dependence of two random variables . More colloquially , the mutual information tells us how many bits we can predict of the future given our observations of the past.1 Asymptotically , a vanishing fraction of the information in the past is relevant to the future ( Bialek et al. , 2001 ) , thus systems which excel at prediction need not memorize the entire past of a sequence . Intriguingly , certain biological neurons extract representations that efficiently capture the predictive information in sequential stimuli ( Palmer et al. , 2015 ; Tkačik & Bialek , 2016 ) . In Palmer et al . ( 2015 ) , spiking responses of neurons in salamander retina had near optimal mutual information with the future states of sequential stimuli they were exposed to , while compressing the past as much as possible . Do our artificial neural networks perform similarly ? In this work , we aim to answer this question by measuring artificial recurrent neural networks ’ ( RNNs ) capacity to compress the past while retaining relevant information about the future . Our contributions are as follows : • We demonstrate that RNNs are suboptimal at extracting predictive information on the tractable sequential stimuli used in Palmer et al . ( 2015 ) . • We thoroughly validate the accuracy of our mutual information estimates on RNNs and optimal models . • We repeat our analysis on the real-world Aaron Koblin Sheep sketch dataset ( Ha & Eck , 2017 ) , demonstrating that deterministic RNNs continue to perform suboptimally . • We demonstrate that on small datasets , models trained to compress the past have improved sample quality over naively-trained RNNs . 1 Because of its symmetry , this is equivalent to the number of bits we can reconstruct of the past given observations of the future . 2 BACKGROUND AND METHODS . We begin by providing additional background on predictive information , mutual information estimators , stochastic RNNs , and Gaussian Information Bottleneck . These tools are neccessary for accurately evaluating the question of whether RNNs are optimal in the information plane , as we require knowledge of the optimal frontier , and accurate estimates of mutual information for complex RNN models . 2.1 PREDICTIVE INFORMATION . Imagine an infinite sequence of data : Xt . The predictive information ( Bialek et al. , 2001 ) in the sequence is the mutual information between some finite number of observations of the past ( T ) and the infinite future of the sequence : Ipred ( T ) = I ( Xpast ; Xfuture ) = I ( { Xt−T+1 , . . . , Xt } ; { Xt+1 , . . . } ) . ( 1 ) For a process for which the dynamics are not varying in time , this will be independent of the particular time t chosen to be the present . More specifically , the predictive information is an expected log ratio between the likelihood of observing a future given the past , versus observing that future in expectation over all possible pasts : Ipred ≡ E [ log p ( xfuture|xpast ) p ( xfuture ) ] = E [ log p ( xfuture|xpast ) Ex′past [ p ( xfuture|x′past ) ] ] . ( 2 ) A sequential model such as an RNN provides a stochastic representation of the entire past of the sequence Z ∼ p ( z|xpast ) . For any such representation , we can measure how much information it retains about the past , a.k.a . the past information : Ipast = I ( Z ; Xpast ) , and how informative it is about the future , a.k.a . the future information : Ifuture = I ( Z ; Xfuture ) . Because the representation depends only on the past , our three random variables satisfy the Markov relations : Z ← Xpast ↔ Xfuture and the Data Processing Inequality ( Cover & Thomas , 2012 ) ensures that the information we have about the future is always less than or equal to both the true predictive information of the sequence ( Ifuture ≤ Ipred ) as well as the information we retain about the past ( Ifuture ≤ Ipast ) . For any particular sequence , there will be a frontier of solutions that optimally tradeoff between Ipast and Ifuture . A common method for tracing out this frontier is through the Information Bottleneck ( Tishby et al. , 2000 ) Lagrangian : min p ( z|xpast ) I ( Z ; Xpast ) − βI ( Z ; Xfuture ) , ( 3 ) where the parameter β controls the tradeoff . An efficient representation of the past is one that lies on this optimal frontier , or equivalently is a solution to Eqn . 3 for a particular choice of β . For simple problems , where the sequence is jointly Gaussian , we will see that the optimal frontier can be identified analytically . 2.2 MUTUAL INFORMATION ESTIMATORS . In order to measure whether a representation is efficient , we need a way to measure its past and future informations . While mutual information estimation is difficult in general ( McAllester & Stratos , 2019 ) , recent progress has been made on a wide range of variational bounds on mutual information ( Poole et al. , 2019 ) . While these provide bounds and not exact estimates of mutual information , they allow us to compare mutual information quantities in continuous spaces across models . There are two broad families of estimators : variational lower bounds powered by a tractable generative model , or contrastive lower bounds powered by an unnormalized critic . The former class of lower bounds , first presented in Barber & Agakov ( 2003 ) , are powered by a variational generative model : Ifuture = E [ log p ( xfuture|z ) p ( xfuture ) ] ≥ H ( xfuture ) + E [ log q ( xfuture|z ) ] . ( 4 ) A generative model provides a demonstration that there exists at least some information between the representation z and the future of the sequence . For our purposes , H ( xfuture ) , the entropy of the future of the sequence is a constant , determined by the dynamics of the sequence itself and outside our control . For tractable problems , such as the toy problem we investigate below , this value is known . For real datasets , this value is not known so that we can not produce reliable estimates of the mutual information . It does , however , still provide reliable gradients of a lower bound . One example of such a generative model is the loss used to train the RNN to begin with . Contrastive lower bounds can be used to estimate Ifuture for datasets where building a tractable generative model of the future is challenging . InfoNCE style lower bounds ( van den Oord et al. , 2018 ; Poole et al. , 2019 ) only require access to samples from both the joint distribution and the individual marginals : I ( X ; Z ) ≥ INCE ( X ; Z ) , EpK ( x , z ) [ 1 K K∑ i=1 log ef ( xi , zi ) 1 K ∑K j=1 e f ( xj , zi ) ] . ( 5 ) Here f ( xj , zi ) is a trained critic that plays a role similar to a discriminator in a Generative Adversarial Network ( Goodfellow et al. , 2014 ) . It scores pairs , attempting to determine if an ( x , z ) pair came from the joint ( p ( x , z ) ) or the factorized marginal distributions ( p ( x ) p ( z ) ) . When forming estimates of Ipast , we can leverage additional knowledge about the known encoding distribution from the stochastic RNN p ( z|xpast ) to form tractable upper and lower bounds without having to learn an additional critic ( Poole et al. , 2019 ) : E [ 1 K K∑ i=1 log p ( zi|xipast ) 1 K ∑ j p ( z i|xjpast ) ] ≤ I ( Z ; Xpast ) ≤ E [ 1 K K∑ i=1 log p ( zi|xipast ) 1 K−1 ∑ j 6=i p ( z i|xjpast ) ] . ( 6 ) We refer to these bounds as minibatch upper and lower bounds as they are computed using minibatches of size K from the dataset . As the minibatch size K increases , the upper and lower bounds can become tight . When logK I ( Z ; Xpast ) the lower bound saturates at logK and the upper bound can be loose , thus we require using large batch sizes to form accurate estimates of Ipast . 2.3 STOCHASTIC RNNS . Deterministic RNNs can theoretically encode infinite information about the past in their hidden states ( up to floating point precision ) . To limit past information , we devise a simple stochastic RNN . Given the deterministic hidden state ht , we output a stochastic variable zt by adding i.i.d . Gaussian noise to the hidden state before reading out the outputs : zt ∼ N ( ht , σ2 ) . These stochastic outputs are then used to predict the future state : x̂t+1 ∼ N ( gdecoder ( zt ) , σ2o ) , as illustrated in Figure 1 . With bounded activation functions on the hidden state ht , we can use σ2 to upper bound the information stored about the past in the stochastic latent zt . This choice of stochastic recurrent model yields a tractable conditional distribution p ( zt|x≤t ) ∼ N ( ht , σ2 ) , which we can use in Eq . 6 to form tractable upper and lower bounds on the past information . We will consider two different settings for our stochastic RNNs : ( 1 ) where the RNNs are trained deterministically and the noise on the hidden state is added only at evaluation time , and ( 2 ) where the RNNs are trained with noise , and evaluated with noise . 2.4 GAUSSIAN INFORMATION BOTTLENECK . To evaluate optimality of RNNs , we first focus on the tractable sequential stimuli of a simple Brownian Harmonic Oscillator as used in ( Palmer et al. , 2015 ) . A crucial property of this dataset is that we can analytically calculate the optimal trade-off between past and future information as it is an instance of the Gaussian Information Bottleneck ( Chechik et al. , 2005 ) . Consider jointly multivariate Gaussian random variables X ∈ RDX and Y ∈ RDY , with covariance ΣX and ΣY and cross-covariance ΣXY . The solution to the Information Bottleneck objective : min T I ( X ; T ) − βI ( Y ; T ) , ( 7 ) is given by a linear transformation T = AX + ε with ε ∼ N ( 0 , Σε ) . The projection matrix A projects along the lowest eigenvectors of ΣX|Y Σ −1 X , where the trade-off parameter β decides how many of the eigenvectors participate . Further details can be found in Appendix A.1 .

Summary: This manuscript shows that good ability to compress past information in RNNs help them to predict the future, and that improving upon this ability leads to more useful RNNs. The manuscript first adapts modern mutual-information estimators to mini-batch settings in order to measure the information that an RNN has on the past. It then considers stochastic training, adding Gaussian noise to the hidden states during the training of the RNNs to limit past information. A significant section is dedicated to an empirical study that shows that classically-train MLE RNNs lead to internal representations with a suboptimal mutual-information to the past and the future. For LSTM and GRU architecture, stochastic training actually significantly helps. Experiments on applications such as synthetizing hand-drawn sketches suggest that stochastic training leads to more useful RNNs.