Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper certainly poses an interesting question: How well do RNNs compress the past while retaining relevant information about the future. In order to quantitatively answer this question, the authors suggest to look at (the optimal solutions of) the Information Bottleneck Lagrangian (IBL). The investigated RNNs need to solve the task of next-step prediction, which can be used to evaluate the IBL. In the paper, the (deterministic) hidden state h is transformed through simple additive Gaussian noise into a stochastic representation which then is utilized to compute the IBL. In general the IBL is not tractable and hence the paper uses approximate computations.

I definitely like the underlying question of the paper. Yet, to me it seems not ready for publication. For one, the presented experimental results look interesting but the suggested method for improvement through adding noise to the latents (during training) is too much of handwaving for such a fundamental problem. Second, shouldn't the results on the BHC be quite surprising in terms of LSTMs performance? Why is that, usually LSTM (or GRU) deliver excellent performance in typical (supervised) sequential tasks. Third, the task itself seems not well described, it seems to be next-step prediction, but how are more future predictions generated -- these seem to be not considered in the equation, but probably should when talking about 'retaining relevant information for the future'? Fourth, recently some researchers started to question whether 'reconstruction' is a good idea in order to learn generative-like models, for example you cite van den Oord 2018. How would such models perform in your metric.

A final remark with respect to your citation for eq. 4, I think you meant Barber, Agakov, "The IM algorithm...", 2003?
review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: This manuscript shows that good ability to compress past information in RNNs help them to predict the future, and that improving upon this ability leads to more useful RNNs. The manuscript first adapts modern mutual-information estimators to mini-batch settings in order to measure the information that an RNN has on the past. It then considers stochastic training, adding Gaussian noise to the hidden states during the training of the RNNs to limit past information. A significant section is dedicated to an empirical study that shows that classically-train MLE RNNs lead to internal representations with a suboptimal mutual-information to the past and the future. For LSTM and GRU architecture, stochastic training actually significantly helps. Experiments on applications such as synthetizing hand-drawn sketches suggest that stochastic training leads to more useful RNNs.

This work has interesting observations and makes a credible case. The stochastic training does seem useful. However, I would like to understand better how it connects to the set of publications discussing dropout in RNNs. It is already known that stochastic perturbations during training help. In addition, the way stochastic training is introduced in this paper make it seem a bit contradictory with the fact that it actually helps generalization. I have the feeling that the benefit that is not understood and that intuitions that arise from the manuscript may not be as useful as we would like.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: Summary:

The paper investigate how optimal recurrent neural networks (RNNs) are at storing past information such that it is useful for predicting the future. The authors estimated optimality in terms of mutual information between the past and the future. If the RNN was able to retain MI between the past and the future, it then has kept optimal information from the past for predicting the future. The experiments suggest that RNNs are not optimal in terms of prediction of the future. It also suggest that this is due to the maximum likelihood training objective.


Comments for the paper:

1. Overall, the paper is a very interesting read and it explores and analyzes RNN under a different light. It answers a fundamental question about RNN training.

2. There are a few things that would be nice to clarify. At the end of P3, the authors mentioned that the stochastic RNNs are trained either by a). deterministically during training and noise added during test or b) noise added during training and test. It is not very clear to me how the authors trained stochastic RNNs deterministically during training. It would be nice if this can be clarified. 

3. I am also curious how this compares to the training methods for example used in https://papers.nips.cc/paper/7248-z-forcing-training-stochastic-recurrent-networks.pdf. It seems that this would also help with retaining RNN optimality in terms of predicting the future. it would be interesting to include a comparison to this method for example.

Overall an interesting paper. However, I think a few things could be improved and I would be willing to rise the score if the authors could addressed the above points.

review_title: Official Blind Review #1

Meta-review: Nice start but unfortunately not ripe.  The issues remarked by the reviewers were only partly addressed, and an improved version of the paper should be submitted at a future venue.
Decision: Reject