On Predictive Information Sub-optimality of RNNs
Authors: Zhe Dong
Deniz Oktay
Ben Poole
Alexander A. Alemi
Venue:  (2020)

Abstract: Certain biological neurons demonstrate a remarkable capability to optimally compress the history of sensory inputs while being maximally informative about the future. In this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. In experiments on two datasets, restorative Brownian motion and a hand-drawn sketch dataset, we find that RNNs are sub-optimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. Overcoming this limitation may require alternative training procedures and architectures, or objectives beyond maximum likelihood estimation.

Under review as a conference paper at ICLR 2020
ON PREDICTIVE INFORMATION SUB-OPTIMALITY OF RNNs
Anonymous authors Paper under double-blind review
ABSTRACT
Certain biological neurons demonstrate a remarkable capability to optimally com- press the history of sensory inputs while being maximally informative about the future. this work, we investigate if the same can be said of artificial neurons in recurrent neural networks (RNNs) trained with maximum likelihood. In ex- periments on two datasets, restorative Brownian motion and a hand-drawn sketch dataset, we find that RNNs sub-optimal in the information plane. Instead of optimally compressing past information, they extract additional information that is not relevant for predicting the future. We show how constraining past informa- tion by injecting noise into the hidden state can improve the ability of RNNs to extract predictive information for both maximum likelihood and contrastive loss training.
INTRODUCTION
Remembering past events is critical component of predicting the future and acting in the world, An information-theorrtic quantification of how much observing the past can help predicting the future is given by the predictive information (Bialek et al. (001). The predictive information is the mutual information (MI) between finite set of observations (the past of sequence) and an infinite number of additional draws from the same process (the future of sequence). As a mutual information, the predictive information gives us a reparameterization independent, symmetric, interpretable measure of the co-dependence of two random variables. More colloquially, the mutual information tells us how many bits we can predict of the future given our observations the past Asymptotically, a vanishing fraction of the information in the past is relevant to the future (Bialek et al. 2001), thus systems which excel at prediction need not memorize the entire past of a sequence.
Intriguingly, certain biological neurons extract representations that efficiently capture the predictive information in sequential stimuli (Palmer et al. 2015 Tkačik Bialek) 2016). In Palmer et al. 2015), spiking responses of neurons in salamander retina had near optimal mutual information with the future states of sequential stimuli they were exposed to, while compressing the past as much as possible.
Do our artificial neural networks perform similarly? In this work, we aim to answer this question by measuring artificial recurrent neural networks' (RNNs) capacity to compress the past while retaining relevant information about the future.
Our contributions as follows:
We demonstrate that RNNs are suboptimal extracting predictive information the tractable sequential stimuli used in Palmer et al. (2015).
We thoroughly validate the accuracy of our mutual information estimates on RNNs and optimal models.
We repeat analysis the real-world Aaron Koblin Sheep sketch dataset (Ha & Eck 2017). demonstrating that deterministic RNNs continue to perform suboptimally.
We demonstrate that on small datasets, models trained to compress the past have improved sample quality over naively-trained RNNs.
Because of its symmetry, this is equivalent to the number of bits we reconstruct of the past given observations of the future.

---

Under review as a conference paper at ICLR 2020
2 BACKGROUND AND METHODS
We begin by providing additional background on predictive information, mutual information es- timators, stochastic RNNs, and Gaussian Information Bottleneck. These tools neccessary for accurately evaluating the question of whether RNNs are optimal in the information plane, as we re- quire knowledge of the optimal frontier, and accurate estimates of mutual information for complex RNN models.
2.1 PREDICTIVE INFORMATION
Imagine an infinite sequence of data: X, The predictive information (Bialek et al. 2001) in the sequence is the mutual information between some finite number of observations of the past (T) and the infinite future of the sequence:
pree (T) T(X (uurre) I(( X (X++1....))
For process for which the dynamics not varying in time, this will be independent of the par- ticular time l chosen to be the present. More specifically, the predictive information is an expected log ratio between the likelihood of observing future given the past, versus observing that future in expectation over all possible pasts:
log P(Tfuture (pass))
pl T'uture (Tpass)
T predd
E
log
(2)
p(Atuture)
E post (futuue papst)
sequential model such as an RNN provides stochastic representation of the entire past of the sequence Z ~ p(= (pps)) For any such representation, we can measure how much information it retains about the past, a.k.a. the past information: Tpast T(Z; X ppmt), and how informative about the future, a.k.a. the future information: Tuuture T(Z; X funure Because the representation depends only on the past, our three random variables satisfy the Markov relations: Z + Xpast + X and the Data Processing Inequality (Cover & Thomas 2012) ensures that the information we have about the future is always less than or equal to both the true predictive information of the sequence (Trmure < Ipred) well as the information any particular sequence, there will be frontier of solutions that optimally tradeoff between Jnas and ffuture- A common method for tracing out this frontier through the Information Bottleneck (Tishby
retain about the past (T Ipuss). For
et al. 2000) Lagrangian:
min X(ex) - (I( Xfuuur), ((=(2
where the parameter 3 controls the tradeoff. An efficient representation of the past one that lies on this optimal frontier, or equivalently is a solution to Eqn. [ for particular choice of B. For simple problems, where the sequence is jointly Gaussian, we will see that the optimal frontier can be identified analytically.
2.2 MUTUAL INFORMATION ESTIMATORS
In order to measure whether a representation is efficient, we need a way to measure its past and future informations. While mutual information estimation is difficult in general (McAllester & Stratos] 2019), recent progress has been made on a wide range of variational bounds on mutual information (Poole al. 2019). While these provide bounds and not exact estimates of mutual information, they allow us to compare mutual information quantities in continuous spaces across models. There are two broad families of estimators: variational lower bounds powered by a tractable generative model, or contrastive lower bounds powered by an unnormalized critic.
The former class of lower bounds, first presented in Barber Agakov (2003), are powered by variational generative model: Pl P. pl (((uuuere)
Luturr log
> H (Xiaure + IE [log g(
(4)
generative model provides a demonstration that there exists at least some information between the representation and the future of the sequence. For our purposes, H (I the entropy of the
2

---

Under review as a conference paper at ICLR 2020
future of the sequence is constant, determined by the dynamics of the sequence itself and outside our control. For tractable problems, such as the toy problem we investigate below, this value is known. For real datasets, this value is not known so that cannot produce reliable estimates of the mutual information. It does, however, still provide reliable gradients a lower bound. One example of such generative model is the loss used train the RNN begin with.
Contrastive lower bounds can be used to estimate Fuuure for datasets where building a tractable gen- erative model of the future is challenging. InfoNCE style lower bounds (van den Oord al. [0188] Poole et al. 2019) only require access to samples from both the joint distribution and the individual marginals:
K Σ 1=1 K
(
T(X;2) (Xc 2) & (x,))
(x,*)
K
log
ΣΣ L el(esia))
(5)
Here [1)y, is a trained critic that plays role similar to a discriminator in a Generative Adver- sarial Network (Goodfellow et al. 2014). It scores pairs, attempting to determine if an (x, 2) pair came from the joint (p(x, 2)) or the factorized marginal distributions (p(z)p(zz).
When forming estimates of Lpatt we can leverage additional knowledge about the known encoding distribution from the stochastic RNN p(a(=paa)) form tractable upper and lower bounds without having to learn an additional critic (Poole et al. 2019):
K
p(2 (arpsss)
E
log I K E, (=
(2; X past)
S
E
log
(6)
K 1=1 K-I Ejfr ((21 | (=
We refer to these bounds minibatch upper and lower bounds as they computed using mini- batches of size K from the dataset. As the minibatch size K increases, the upper and lower bounds become tight. When log K < ((Z:Xpaai) the lower bound saturates at log K and the upper bound can be loose, thus we require using large hatch sizes to form accurate estimates of /past-
2.3 STOCHASTIC RNNS
Deterministic RNNs can theoretically encode infinite information about the past in their hidden states (up floating point precision). To limit past information, devise simple stochastic RNN. Given the deterministic hidden state ht, we output stochastic variable 2, by adding i.i.d. Gaussian noise the hidden state before reading are then used to predict the future state: ēt+1 (gatecnder (zt), o.) as illustrated in Figure With bounded activation functions on the hidden state hG, we can use o² to upper bound the information stored about the past in the stochastic latent 24. This choice of stochastic recurrent model yields a tractable conditional distribution p(= (rse) tractable upper and lower bounds on the past information. We will consider two different settings for our stochastic RNNs: (1) where the RNNs are trained deterministically and the noise on the hidden state is added only at evaluation time, and (2) where the RNNs trained with noise, and evaluated with noise.
the outputs: 2, (Mh,, o²). These stochastic outputs
[(,,, o²), which we can use Eq. to form
S0-1 NWW bz)
Ram
Zi
X2
Xay
Future: Xot
Figure 1: Schematic of Gaussian-noisee aug.niseeeemdmmeeddde stochastic RNN,
3

---

Under review as a conference paper at ICLR 2020
2.4 GAUSSIAN INFORMATION BOTTLENECK
To evaluate optimality of RNNs, we first focus on the tractable sequential stimuli of a simple Brow- nian Harmonic Oscillator used in (Palmer al. 2013). A crucial property of this dataset that we can analytically calculate the optimal trade-off between past and future information as is an instance of the Gaussian Information Bottleneck (Chechik al. 2005).
Consider jointly multivariate Gaussian random variables X E Px and Y E Ry,,, with covariance Xi and My and cross-covariance Exy. The solution the Information Bottleneck objective: min (X; :T),
is given by a linear transformation T - AX + with é ~ (O,, The projection matrix A projects along the lowest eigenvectors of Σx1y Exy where the trade-off parameter / decides how many of the eigenvectors participate. Further details be found in Appendix A.1
RESULTS
3.1
BROWNIAN HARMONIC OSCILLATOR
1.5
2.5
SimpleRNN
1.0
2.0
0.5
[HHddenn; Future)
1.5
0.0
1.0
-D.5
-1.0
0.5
1.55
00
20
40
60
80
100
IPaat:: Hidden)
(b) SimpleRNN
(a) Brownian Harmonic Oscillator
2.5
GRU
2.5
LSTM
8
2.0
2.0
((Hidde:: Futurre)
-06
Iittidder Futurel
1.5
-13
1.5
MABP
-18
1.0
-24
1.0
-24
0.5
0.5
9,0
2
4
5 Hidden)
6
7
8
00
2
3
5
I|Past: Hidden)
(c) GRU
(d) LSTM
Figure 2: (a) Example trajectories of Brownian harmonic oscillator over time, with each color rep- resenting a different trajectory. (b, c, d) Estimates of the past information contained in the stochastic variable (x-axis) upper-left perform better future prediction, while compressing the past more, therefore they are of higher efficiency. Points correspond to estimates with a learned critic; bars represent the gap between the lower and upper bounds using the known conditional distribution instead of learned critic: col- represent the noise level, log(noise), added to the hidden state. The stochastically trained RNN is marked as and deterministically trained RNN with post-hoc noise injection is marked as v.
future information (y-axis). The feasible region is shaded. The models to the
We begin by considering the Brownian harmonic oscillator (BHO) dataset, a form restorative Brownian motion, in Palmer al. (205)). This system has the benefit of having fully tractable
4

---

Under review as a conference paper at ICLR 2020
predictive information. The dynamics are given by:
At It + 11
(8)
U+ -TA A. & &.V
where is standard Gaussian random variable. Additional details can be found in Appendix A.2 Examples of the trajectories for this system can be seen in Figure 2a
Given its analytical tractability we can explicitly compare the estimated RNN performance against optimal performance. We compared three major variants of RNNs, including fully connected RNNs, gated recurrent units (GRU, Cho et al.] (2014)) and LSTMs (Hochreiter & Schmidhuber 1997). Each network had 32 hidden units and tanh activations. Full training details are in Appendix A.2
Figure 2 contains the results for both stochastic RNNs and deterministic RNNs with noise added only at evaluation time. By varying the strength of the noise, networks trace out a trajectory on the information plane. We find that networks trained with noise close the optimal frontier (black), in terms of optimally extracting information about the past that is useful for predicting the future. While injecting noise evaluation time produces networks with compressed representations, these deterministically trained networks perform worse than their stochastically trained counterparts.
Training with noise injection can make the networks more efficiently capture the predictive infor- mation than post-hoc noise injection. At the same noise level (the color coding in Figure 2 stochas- tically trained RNNs have both higher T(2; pus and higher [(z; Ttuuure). By limiting the capacity of the models during training, they were able to extract more efficient representations. For this task, we find that more complex RNN variants such as LSTMs are less efficient at encoding predictive information.
Dropout (Srivastava et al. 2014) is a potential alternative way to eliminate information. In Ap- pendix A.S] demonstrate that RNNs trained with dropout extract less information than the ones trained without dropout, and that our simple noise injection technique can find equivalent models. consequence of our choice of maximum likeli- hood (MLE) as training objective, in Appendix AA.4 we demonstrate the effect using contrastive predictive coding (CPC) (Oord et al.) 2018) as the training objective. We find that models trained with CPC loss perform similarly to models trained with MLE, as measured on the information plane. However, this may be due the BHO dataset having Markovian dynamics, and optimizing for one- step-ahead prediction with MLE is sufficient to maximize mutual information with the future of the sequence.
To access whether the observed sub-optimality
3.2 ACCURACY OF MUTUAL INFORMATION ESTIMATES
Our claim that RNNs are suboptimal in capturing predictive information hinges on the quality of MI estimates. Are the RNNs truly suboptimal is it just that our MI estimates are inaccurate? Here we provide several experiments further validating the accuracy of our MI estimates.
Mutual Information Estinations
2.5
2.5
HBAI WinfoNCE)
Information Essimations
Infdatasel sice)
2.0
e0e
(Hidden,, Future) W/BA
2.0
B6 Al E s
2.5
Noise 0.02
Noise
2.5
1.5
20
2.0
1.0
0.5
0.5
0.0
0.0
0.5
%
10'
10*
Noise
WinffNNCE
Samples
Figure 3: (Left) Comparison among critic based estimators, InfoNCE, JS and NWJ. (Middle) Com- parison between estimations from Barber-Agakov and InfoNCE lower bound on future information [XXuuue Z). (Right) Illustration of the convergence for minibatch upper and lower bounds with two noise levels, 0.02 (Blue) and 0.05 (Red). Dashed line is the In number of samples), which is the limit for minibatch lower bounds.
5

---

Under review as a conference paper at ICLR 2020
Comparison among estimators. There many different mutual information estimators. In Fig- ure we compare various mutual information lower bounds with learned critics: InfoNCE, NWJ and JS, summarized in Poooee et al. (019). NWJ and JS show higher variance and worse bias than InfoNCE. The second panel of Higure demonstrates that InfoNCE outperforms a variational Barber-Agakov style variational lower bound measuring the future information. Therefore, adopted InfoNCE as the critic based estimator for the future information in the previous section.
For the past information, tractable likelihood, p ~ (le,,a") In the third panel of Figure we demonstrate that these bounds become tight as the sample size increases. However they require large number of samples before they converge. Fundamentally, the lower bound itself is upper- upper-bounded by the log of the number of samples used.
could generate both tractable upper and lower bounds, given our
3.5
Impact of Training Datasel Size
Mutual Information Estimation
3.0
2.5
9.0 0.5 I[Hidden:: Future)
og
2.0
1.5
1.0
Evel EDiiden; Past)
Thain: imnd Peatt
(riicee:: Funur))
(01 tdaers Pulane
-0.5
4 6
6 10
Training Steps
I(Past: Hidden)
Figure 4: (Left) Estimates of past and future information over training iterations on the training that MI estimates quickly overfit, which remedy here by early stopping. (Right) Impact of training dataset sizes on InfoNCE estimator on more complicated dataset, Aaron's Sheep, introduced Section 3.3 The original dataset has size 7200 for training, and 800 for evaluation. We augment the dataset by random scaling the input values per sequence. The colors indicate the multiples of original dataset size after augmentation.
and testing BHO data. We
Estimator training with finite dataset. Training the learned critic on finite datasets for a large number of iterations resulted in problematic memorization and overestimates of MI. To counteract the overfitting, we performed early stopping using the MI estimate with the learned critic on a validation set. Unlike the training MI, this valid lower bound the true MI. We then report estimates of mutual information using the learned critic on an independent test set, as Figure 4
2.5
45
2.0
I(Hidden; Future)
1.5
1.0
20
15
0.5
0.0
1
2
3
6
7
I(Past: Hidden)
Figure Evaluate mutual information estimators given optimal encoder. For past information estimation, I(Hidden; Past), InfoNCE lower bound (colored points) and MB-Lower and MB-Upper (colored bars) are used; for future information, I(Hidden; Future), only InfoNCE is applied, due to lack of tractable conditional distribution p(Y (T). Heat-color represents the level of trade-off parameter
Analytical Agreement As a final and telling justification of the efficiency of our estimators, Figure demonstrates that our estimators accurately estimate the mutual information in our region of interest,

---

Under review as a conference paper at ICLR 2020
for analytically derived optimal projections. Background for the Gaussian Information Bottleneck in Section 2.4 and details of the calculation can be found in Appendix A.1
3.3 VECTOR DRAWING DATASET
In order to assess whether the sub-optimality of RNNs was an artifact of the BHO dataset, we performed additional experimentson a real world sequential dataset. We used the Aaron Koblin Sheep Sketch D Full experimental details are in Appendix A.3 We adopted the SketchRNN architecture and online data augmentation procedure used in Ha & Eck (207))
Figure (Left) shows the estimates on the information plane for the trained networks. Again, the networks that were trained explicitly with noise dominate on the information plane. For this real world task we no longer know the optimal information tradeoff, but still see that the deterministi- cally trained networks evaluated with noise are demonstrably suboptimal compared to some simple stochastic networks trained with noise injection.
QuickDrew
3.0 3.5
Losuttegeing Log fieellooooo Copeeiie
I[Hldden: Futurel
1.5 2.0
Irsitaaion Lows Traninng
7
8
[(Pest: Hidden))
Noise Level
Figure 6: (Left) Evaluation on Aaron Sheep dataset by comparing training explicitly with noise (o) and post-hoc noise injection after training (V). The color bar shows the noise level in logio scale. (Right) Comparing the training and evaluation loss for noise-trained RNNs.
Besides demonstrating sub- ootimalttll in capturing predictive information the information plane, in (Right,, we demonstrate that the deterministic RNNs are wasteful. Networks with higher levels of compression (marked by higher noise levels) were able to obtain similar average set performance with noticeably lower variance than the deterministic networks. They also show smaller generalization gap. By explicitly training the networks to operate within some constrained information budget reduced variance between runs without substantially reducing test set perfor- mance.
In Figure 7 we show some generated completions from the models trained on the full dataset with varying degrees of noise. The models trained with the highest noise levels show noticeable degre- dation in sample quality.
3.4 SMALL DATASETS
We expect the benefits of training compressed representations to be most marked on limited datasets where the compression can prevent the network from memorizing too many spurious correlations.
To investigate, we repeated the experiments of the previous section with limited dataset with only 100 examples. Figure ( (Upper Left) shows the corresponding information plane points for stochas- tically trained RNNs with various noise levels. Notice that at about 4 nats of past information, the networks future information essentially saturates. While it seems as though the networks don't suffer even when asked to learn richer representations, this is largely an artifact of our training pro- cedure which included early stopping. As can be seen in Figure (Lower Left), all of our networks overfit in terms of evaluation loss, but the onset of overfitting was strongly controlled by the degree of compression.
Available htt http:////.t..hooooaaaee/eeeeeeea aaron_sheep

---

Under review as a conference paper at ICLR 2020
conditioned input
original
conditioned input original
conditioned generation
conditioned generation
0.0
0.0
0.04
noise level
0.04
noise level
0.1
0.2
0,2
Figure 7: Conditional generation of QuickDraw sketches by SketchRNN models trained with dif-
ferent noise levels.
1.0
Conditioned
Original
IHHdden:: Future)
0.8
6.7
-28
Past Information 10X Z)
10
MPast: Hidden)
2.5
1.0
3.0
Etokatton LaG
20
s.0
FERES
1.0
0.5
2000
2000
Figure 8: Left Column. Top: Estimation of past and future information for RNN trained with 100 samples, with color indicating the noise levels logio scale. Bottom: Validation loss for different noise levels. Right Column. Conditional generated samples from models with different levels of past information. The generation is conditioned on 25-step stroke, which taken from a held-out sample. The samples from the model with 4.0 nats past information of better sample quality than models with higher past informations. For more samples, see Figure 12
Most noticeably, in the limited data regime, compressed representations lead to improved sample quality, as seen Figure & (Right). Models with intermediately-sizeee compressed representations show the best generated samples while retaining a good amount of diversity. Models with either too little or too much past information tend to produce nonsensical generations.
4 CONCLUSION AND DISCUSSION
In this work, we have demonstrated how analyzing recurrent neural networks in terms of predictive information can be useful tool in probing and understanding behavior. We find that RNNs trained with maximum likelihood are sub-optimal about the past than is required to predict the future. By analyzing different training objectives and noise injection approaches in the information plane, different models, and identify models that are closer to the optimality demonstrated by biological neurons (Palmer al. 2015).
the information plane, extracting more information
better understood the tradcoffs made by
8

---

Under review as a conference paper at ICLR 2020
While the simple strategy of adding noise bounded hidden state be used constrain infor- mation, setting the amount noise and identifying where one should be on the information plane remains open problem. Additionally, studying the impact of other architecture choices, such as stochastic latent variables in variational RNNs (Chung et al. 2015), or attention-basee Transformers (Vaswani al. 2017) in the information plane could yield insights into their better performance several tasks.
REFERENCES
D Barber and FV Agakov. The IM algorithm: A variational approach to information maximization. In NIPS, pp. 201-208. MIT Press, 2003.
William Bialek, Nya Nemenman, and Naftali Tishby, Predictabilitty complexity, and learning. Neu- ral computation, (3(11):2409-246) 2001. URL
0007070
Gal Chechik, Amir Globerson, Naftali Tishby, and Yair Weiss. Information bottleneck for gaus- Sian variables. Journal of Machine Learning Research, 6:165-188, 2005. URL http::/www.
jmir.org/papezz. http://wccccccccceeeeeee html
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre. Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical ma- chine translation. CoRR, abs/1406. 1078, 2014.
Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Ben- gio. A recurrent latent variable model for sequential data. In Advances neural information processing systems, pp. 2980-2988, 2015.
Thomas M Cover and Joy A Thomas. Elements of information theory. John Wiley & Sons, 2012.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Yee Whye Teh and Mike Titterington (eds.), Proceedings of the Thirteenth Interna- tional Conference Artificial Intelligence and Statistics, volume 9 of Proceedings of Machine Learning Research, pp. 249-242222222 256. PMLR, 13-15 May 2010.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in Neural Infor- mation Processing Systems 27, pp. 2672-2680. Curran Associates, Inc., 2014.
David Ha and Douglas Eck. A neural representation of sketch drawings. CoRR, abs/ htns//704.3377,, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance imagenet classification. In The IEEE International Conference Computer Vision (ICCV), December 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Comput., 983: 1780, November 1997. ISSN 0899-76677.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL /ttett:://axiivi www.xii..rg//bbb.. / 6980
David McAllester and Karl Stratos. Formal limitations on the measurement mutual information, 2019. URL / http://oppeeevvveeeee net
Simon F. Nørrelykke and Henrik Flyvbjerg. Harmonic oscillator in heat bath: Exact simulation of time-lapse-recorded data and exact analytical benchmark statistics. Phys. Rew. E, 83:041 103, Apr 2011.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive pre- dictive coding. arXiv preprint arXiv:1807 0374,, 2018. URL https: / //traa/avvv.///////////aaaaaawav org. abs/ T807. 03778

---

Under review as a conference paper at ICLR 2020
Stephanie E Palmer, Olivier Marre, Michael J Berry. and William Bialek. Predictive information in a sensory population. Proceedings of the National Academy of Sciences, (12((2)):90886611)
2015.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171- 5180, 2019.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12 (1):145-151, 1999.
Nitish Srivastava, Geoffrey E. Hinton, Alex Krizhevsky. llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929- 1958, 2014. URL http::/ cs toronto. edu/ aecuu-rraaaakinn
papers/ https///iivastttaaaaa.aa pd pdf
Naftali Tishby, Fermando Pereira, and William Bialek, The information bottleneck method.
CoRR, physics/0004057, 2000. URL
Gašper Tkačik and William Bialek. Information processing in living systems. Annual Review of Condensed Matter Physics, 7:89-117, 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint arXiv: 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
A APPENDIX
A.1 GAUSSIAN INFORMATION BOTTLENECKS
Consider jointly multivariate Gaussian random variables X Rx and Y E Ry, with covariance Σx and Ey and cross-covariance Σxy. The solution the Information Bottleneck objective:
min [(;;T) (I(YYT))
given by a linear transformation T = AX + with & ~ (O),E.). The projection matrix A projects along the lowest eigenvectors ( E [1, of Exp-Ex where the trade-off parameter 3 decides how many of the cigenvectors participate, v. (i [x]). The projection matrix A could be analytically derived as
= l 02v DDD v Σg =I
(10)
where the projection coefficients of max ((1-2.)-1 0), ri viT'v; with proof in Ap- A
pendix A.I.I
Given the optimally projected states T. the optimal frontier (black curve Figure 5 is:
T(T;Y) X)T:)) log
II (1 AT) 2=1
T2
21(7.X
a ) Cay S I(T; Cuyt1 (11)
where n is the cutoff number indicating the number of smallest eigenvalues being used. The critical points Cay. changing from using n, = N eigenvalues to N + I eigenvalues, can be derived given the concave and C' smoothness property for the optimal frontier, with proof in Appendix A.1.2
N
Cny =
Σ 1=1
log
AN+1 1- X 1 - AN+1
(12)
10

---

Under review as a conference paper at ICLR 2020
A.1.1 PROOF OF OPTIMAL PROJECTION
By Theorem 3.1 of Chechik et al. (005), the projection matrix for optimal projection is given by
(eivT,0,0 ... 0] [anvi,agy 0,
0 0 81
81 < < 82
A
[alv] a2v2 ,21V 02v2
8 S 3 83
(13)
where (i [1, Dx]) are left cigenvectors of Ex sorted in ascending order by the eigenval-
ues A, E [1, (x]]):2 8
x-x. are critical values for trade-off parameter 3; and the projection coef-
ficients are of - ri vi Vi. In practice, noticing that f (1 (1- < 0 when _ <
A.h:
3i we simplify Equation 13) as A [aiv] a2v2 ..,,.,.......................... ODx VDx with a max ((1-As)-1 0).
A,ri
A.1.2 PROOF CRITICAL POINTS ON OPTIMAL FRONTIER
By Eq.15 15 of Chechik al. (2055)
I(::Y) T(T; X)
log (1 X,) + "r
II
21(T-X)
(14)
1=1
where ny is the cutoff the number of eigenvalues used compute the bound segment, with cigenvalues sorted in ascending order.
In order to calculate the changing point, where one switching from choosing n N to N + by C smoothness conditions:
dl Un:=N(T:Y) dln==N+((T aT(T:X)
(15)
dT(T;X)
LHS is
(Aa) $ N
dl (T;:N(T:Y)
(16)
d((T;X)
2(())))
II
X,)
By
2
(17)
21(T:X)
N
1-A:
Thus, Equation 15
could be rewritten
21(7:N)
NT1
(18)
2
2 NTL
N+11
1-11
V
Rewrite RHS of above equation, and noticing
n(x+1) =
nt1
X
21(T:X)
(19)
21(T-X1
N
N+1
1-2=
11

---

Under review as a conference paper at ICLR 2020
The term in lower right corner could be written
N+1 1 1-
AN+1
N+1
(20)
A
À
AN+1
N
X
(1 AN+1
NiN+o
WI-N
(21)
A
A
AN+1
N
Ai
Ax -NN+1)
WiN+1
(22)
A
[iml
Putting RHS together
21T N
21(T:X)
2
2((C:N)
21(7())
N π
(1(11AA44))
NOTTI
((N+1)
(1-X:)AN11
(23)
To let LHS RHS, one trivial solution is
21(7:X e FI
II
N
11 AN+1)
NONFE
(24)
(1 (1-A;) A1)=
Taking log and cancelling out multiplicative factors, one get the critic point to change from n, N
to n, N + happens at
AN+1 1- A
IT;;X)
log
(25)
A 1 AN+1
The original result written in Chechik et al. (2005) missing a factor of 2.
A.1.3 OPTIMAL PROJECTION
The optimal frontier is generated by joining segments described by Equation Figure
11 as illustrated in
100
2.5
90
(Hidden; Future)
2.0
60
1.5
50
Eigenvalues
30
Use Eigenvalues
Use 3 Eigenvalues
20
IZ: X[tuture = 0.990 + IIZ; X|past])
10
Feasible Region
0.0
((Hidden; Past)
Figure 9: Conditionally generated samples from models with different levels of past information.
12

---

Under review as a conference paper at ICLR 2020
A.2 DETAILS FOR BROWNIAN HARMONIC OSCILLATOR
To generate the sample trajectories, we set the undamped angular velocity w - 1.5 x 2 (rad), damping coefficient r 20.0, and the dynamical range of external forces D 10000.0, with
integration time step-size At = 0.01667. The stationary distribution of Equation 8 is analytically
derived in Norrelykke & Flyvbjerg (201T).
We train RNNs with infinite number of training samples, which are generated online and divided into batches of 32 sequences. RNNs, including fully connected RNN, GRU and LSTM, are all with 32 hidden units and tanh activation. They are trained with momentum optimizer (Qian (199) for 20000 steps, with momentum = 0.9 and gradient norm being clipped at 5.0. Learning rate for training is exponentially decayed in stair-case fashion, with initial learning rate 10-4, decay rate 0.9 and decay steps 2000.
The mutual information estimators, with learned critics, are trained for 200000 steps with Adam optimizer (Kingma & Ba 2015) at a flat learning rate of 10 The training batch size is 256, and the validation and evaluation batch sizes are 2048. We use early stopping to deal with overfitting. The training is stopped when the estimation on validation set does not improve for 10000 steps, or when drops by 3.0 from its highest level, whichever comes first. We use separable critics (Poole al.) 2019) for training the estimators. Each of the critics is a three-layer MIP, with [256, 256, 32] hidden units and [ReLU, ReLU, None] activations. The weights for each layer are initialized with Glorot uniform initializer (Glorot & Bengio 2010), and the biases are with He normal initializer (He et al.) 2013). For the minibatch upper and lower bounds, they are estimated on batches of 4096 sequences.
To train the critics, we feed 1t0-step BHO sequences into trained RNN to get RNN hidden states and conditional distribution parameters. From each sequence, we use last 36 steps for the inputs to the estimators, where first 18 steps as Tapstt [t]. t = [1, 2, ... 18], and the other 18 steps as Truture [t]. [19, 20,... 36). .0.....6.. The hidde state 218 is extracted at the last time step of Fpast:
A.3 TRAINING DETAILS FOR VECTOR DRAWING DATASET
We train decoder-only SketchRNN (Ha Eck 2017) on Aaron Koblin Sheep Dataset, as pro- vided in / aaron sheep The SketchRNN uses LSTM as its RNN cell, with 512 hidden units.
com/ har caaarmmaru sket ch- -aah-nnn-aatee dattn-ddtsstsstt. teee watee/maat /
com/ mmmastermmagenaaaaa htt////aaagaameeeeeeeeeeee htttttaammomess///m tasss/kecch. http://eeehhrnnmndeeaeeeeeee py except that we turn off the recurrent drop-out, since drop-out masks out informations and will interfere with noise injection.
For RNN training, We adopt the identical hyper parameeassestrss as in https: wwwp///aggnaa/llo.oo.o
For mutual information estimations, we use the identical hyper-parametert as descibed in Ap- pendix A.2 except that: the evaluation batch size for critic based estimator, InfoNCE, is set to be 4096, and 16384 for minibatch bounds; early stopping criteria are changed that either the esti- mation does not improve for 20000 steps or drops by 10.0 from its highest level, whichever comes first.
Due to the limitation of the sequence length of Aaron's Sheep, we use the samples with at least 36 steps long. The Tpast and Ffuture are split at the middle of the sequences, and each with 18 steps.
Due to the limitation of the dataset size of Aaron's Sheep, we augment the dataset with randomly scale the stroke by a factor sampled from /O, 0.15) for each sequence to generate a large dataset. Figure 4 (Right) shows that the augmentation helps in training the estimator.
A.4 EFFECT OF CHANGING TRAINING OBJECTIVES: MAXIMUM LIKELIHOOD AND CONTRASTIVE LOSS TRAINING
To access whether the observed suboptimality in the information plane was due to the maximum likelihood (MLE) objective itself, we additionally trained constrastive predictive coding (CPC) mod- els (Oord et al. 2018). We used the identical model architecture as described in the Section 2.3 The CPC loss lower bounds the mutual information between the current time step and K steps into the future. For our experiments on the Brownian harmonic oscillator (BHO), we look into K 30 steps future, and use a linear readout from the hidden states time-independent embedding of the
13

---

Under review as a conference paper at ICLR 2020
inputs. As shown in Figure 10 we found that models trained with CPC loss had similar frontiers as those trained with MLE. The loss function does not appear to have a substantive effect.
2.5
2.5
66
00
2.0
I(Hidden: Future)
(IFidden Future
1.5
11
1.0
1.0
0.5
0.5
90
0.0
I(Past: Hidden)
iPast: Hidden)
(a) SimpleRNN
(b) LSTM
Figure 10: The impact of training objectives BHO dataset for SimpleRNN (Left) and LSTM (Right). Models trained with maximum likelihood estimations are marked with o, and models trained with contrastive loss are marked with V. The color bar shows the noise level in log10 scale.
A.5 EFFECT OF DROPOUT
Dropout (Srivastava et al. 2014) is common method applied neural network training to prevent overfitting. We trained fully connected RNNs and LSTMs with different levels of dropout probabil- ity. As shown in Figure II find that RNNs trained with dropout extract less information than the ones without it, but the information frontier of the models does not change, when we sweep dropout rate and additive noise. In other words, can find models that equivalent dropout models by simply adding Gaussian noise to the output at training time.
(a) SimpleRNN
(b) LSTM
2.5
2.5
2.0
2.0
[(Hidden; Future)
I(Hidden:; Future)
1.
04
1.0
1.0
0.5
0.5
0.0
0.0
2
6
I(Hidden; Past)
I(Hidden: Past)
Figure 11: The impact of dropout predictive information capacity for SimpleRNN (Left) and
LSTM (Right). Grey o marks the result of stochastically trained RNN as described in Figure Colored marks the result for stochastically trained RNN with gaussian noise and different keep rate on RNN outputs, with the keep rate colored in heat. V ones are with noise level 0.1, and +s are with 0.5.
14

---

Under review as a conference paper at ICLR 2020
Conditioned
Original
Conditioned
Original
6.
Past Information 1(Xpast 2) 50
Past Information 1(Xpast :2)
0.
80
.0
19
1.0
Conditioned
Original T
Conditioned
Original
Bevi'
96
Past Information 1(Kpast 2)
Post Information ([Xpast
69
50
1.0
Conditioned
Original
Conditioned
Original
Past Information (MPast 2)
Post Information ((Xpust 2)
20
Figure 12: Conditionally generated samples from models with different levels of past information.
15


Reviews:
[{"review_id": "rkxNUnoftB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper investigate how optimal recurrent neural networks (RNNs) are at storing past information such that it is useful for predicting the future. The authors estimated optimality in terms of mutual information between the past and the future. If the RNN was able to retain MI between the past and the future, it then has kept optimal information from the past for predicting the future. The experiments suggest that RNNs are not optimal in terms of prediction of the future. It also suggest that this is due to the maximum likelihood training objective.\n\n\nComments for the paper:\n\n1. Overall, the paper is a very interesting read and it explores and analyzes RNN under a different light. It answers a fundamental question about RNN training.\n\n2. There are a few things that would be nice to clarify. At the end of P3, the authors mentioned that the stochastic RNNs are trained either by a). deterministically during training and noise added during test or b) noise added during training and test. It is not very clear to me how the authors trained stochastic RNNs deterministically during training. It would be nice if this can be clarified. \n\n3. I am also curious how this compares to the training methods for example used in https://papers.nips.cc/paper/7248-z-forcing-training-stochastic-recurrent-networks.pdf. It seems that this would also help with retaining RNN optimality in terms of predicting the future. it would be interesting to include a comparison to this method for example.\n\nOverall an interesting paper. However, I think a few things could be improved and I would be willing to rise the score if the authors could addressed the above points.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "r1l3FOGR9B", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper certainly poses an interesting question: How well do RNNs compress the past while retaining relevant information about the future. In order to quantitatively answer this question, the authors suggest to look at (the optimal solutions of) the Information Bottleneck Lagrangian (IBL). The investigated RNNs need to solve the task of next-step prediction, which can be used to evaluate the IBL. In the paper, the (deterministic) hidden state h is transformed through simple additive Gaussian noise into a stochastic representation which then is utilized to compute the IBL. In general the IBL is not tractable and hence the paper uses approximate computations.\n\nI definitely like the underlying question of the paper. Yet, to me it seems not ready for publication. For one, the presented experimental results look interesting but the suggested method for improvement through adding noise to the latents (during training) is too much of handwaving for such a fundamental problem. Second, shouldn't the results on the BHC be quite surprising in terms of LSTMs performance? Why is that, usually LSTM (or GRU) deliver excellent performance in typical (supervised) sequential tasks. Third, the task itself seems not well described, it seems to be next-step prediction, but how are more future predictions generated -- these seem to be not considered in the equation, but probably should when talking about 'retaining relevant information for the future'? Fourth, recently some researchers started to question whether 'reconstruction' is a good idea in order to learn generative-like models, for example you cite van den Oord 2018. How would such models perform in your metric.\n\nA final remark with respect to your citation for eq. 4, I think you meant Barber, Agakov, \"The IM algorithm...\", 2003?", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Hyxmmk9kcS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This manuscript shows that good ability to compress past information in RNNs help them to predict the future, and that improving upon this ability leads to more useful RNNs. The manuscript first adapts modern mutual-information estimators to mini-batch settings in order to measure the information that an RNN has on the past. It then considers stochastic training, adding Gaussian noise to the hidden states during the training of the RNNs to limit past information. A significant section is dedicated to an empirical study that shows that classically-train MLE RNNs lead to internal representations with a suboptimal mutual-information to the past and the future. For LSTM and GRU architecture, stochastic training actually significantly helps. Experiments on applications such as synthetizing hand-drawn sketches suggest that stochastic training leads to more useful RNNs.\n\nThis work has interesting observations and makes a credible case. The stochastic training does seem useful. However, I would like to understand better how it connects to the set of publications discussing dropout in RNNs. It is already known that stochastic perturbations during training help. In addition, the way stochastic training is introduced in this paper make it seem a bit contradictory with the fact that it actually helps generalization. I have the feeling that the benefit that is not understood and that intuitions that arise from the manuscript may not be as useful as we would like.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: Nice start but unfortunately not ripe.  The issues remarked by the reviewers were only partly addressed, and an improved version of the paper should be submitted at a future venue.
Decision: Reject