Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The work introduces memUP, a training method that allows to learn long-term dependencies without backpropagating gradients through the entire sequence (unlike RNNs). The method is based on maximizing mutual information between memory states. Then, it is explained how the equations need and can be approximated. This introduces further assumptions leading to possible vulnerabilities to the Noisy-TV problem. This is further analyzed in the experimental sections. This is further simplified to a practical optimization function related to cross-entropy loss that depends on the memory $m$ at a timestep and the model $q$ given the respective input.
The work then continues by describing how memUP is applied to RNNs. The experimental sections cover results for basic sequence tasks (copy, scattered copy, permuted MNIST, and add task), while comparing to LSTMs, SRNNs, and Transformers. Also, it is experimented in RL settings (T-maze, Vizdoom-Two-colors), showing competitive results to existing methods.


strengths_and_weaknesses: The work seems original to be the best of this reviewer knowledge and relevant to the NeurIPS community. Enabling models to learn and process long-range dependencies is indeed a problem of interest. In particular, how to train memory augmented models. The works proposes an interesting method to train such models. 

The idea behind the method is clear and interesting. The paper focuses on deriving the method, but could have make a better job exploring it further. Instead, authors decided to compare against other methods in two settings (supervised learning on sequences and RL). See questions below. In addition, several of these experiments could have compared to newer recurrent methods, and memory augmented methods (both transformers and RNNs). See [1] and [2] as examples. 

The text is clear. Putting more emphasis on how the algorithm is applied to RNNs (and other networks) could really help the reader understand the method and give it more clarity. Defining the tasks in the text instead of the supplementary would be appreciated as well.


[1] Legendre Memory Units: Continuous-Time Representation in Recurrent Neural Networks, Voelker et al., 2019

[2] Not All Memories are Created Equal: Learning to Forget by Expiring, Sukhbaatar et al., 2021
questions: * How is the optimization of the CE loss (eq 5) is solved for both $\theta$ and $\phi$? When is the stop_gradient(m) applied? 
* How would the method needs to be modified to solve regression problems? (for $y_t$ scalar).
* How harder it is to train (and converge) this memory model vs LSTMs with chrono initialization? 
* How does this method perform on more real sequential tasks such as language modeling?
* How does the training effort changes when the tasks are more challenging and more parameters could benefit the solution? Is memUp method still viable? If not, what would need to happen?
* What is the effect of the memory size on the solution? 
* Would it be possible to measure the effect of the memory on the quality of the solution? For example, what is the performance of the model trained with memUp but with the solution measured with $m_t$ always set to 0 (zero) to "remove" the memory information.

limitations: The authors describe and analyze the Noisy-TV problem based on their approximation.
They also assume that dependencies are scarce with the increase of the distance.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this paper, the authors proposed a method called MemUP to help recurrent neural networks learn long-term dependencies better. Specifically, the authors leverage a memory model to predict future outcomes with high uncertainty. By skipping all states with lower uncertainty, the training process takes less computational cost in backpropagation. The framework involves a uncertainty detector model and a predictor to estimate mutual information used in memory model training. The authors show experimental results on both supervised learning tasks and reinforcement learning tasks with improvement comparing with baselines.
strengths_and_weaknesses: The manuscript has the following strengths:
1. The motivation is reasonable, and the authors proposed a framework to address the motivation.
2. The experiments are solid, the authors provide comparible or better results in comparison with several baselines designated to handle long sequences.
3. For such a complicated framework, the authors are able to illustrate clearly.

The manuscript has the following weaknesses:
1. The proposed framework involves training of several models, including a memory model $g_\theta$, a predictor function $q_\phi$ and a uncertainty detector model $d_\psi$. The whole model is too complicated in my point of view. 

Some typos:
1. Title for Figure 2: "final performanse" should be "final performance"
questions: 1. What is the computational cost of the proposed method?
2. By involving auxiliary models to estimate "surprise level", how accurate is the estimation in the proposed framework and experiment?
limitations: The authors have adequately addressed the limitation and potential negative social impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes MemUP (Memory for Uncertainty Prediction), a method to learn long-term dependencies for recurrent models. MemUP learns to keep useful past states for future use, so it saves both computation and memory. The memory module is trained to optimize the mutual information between past states and current prediction targets, and only the steps that have highest "uncertainty" would be used for training to further save memory usage. Empirical experiments on supervisied learning and reinforcement learning tasks show that MemUP significantly boost the results over its baselines.
strengths_and_weaknesses: The solution proposed by the paper is valuable for long-sequence modeling. On the one hand, it reduces the computational overhead of RNN model training; on the other hand, it could preserve distant information, which RNNs would easily forget. The methods are simple, and the intuition behind the methods all make sense to me. The paper is well-written, with details clearly formulated and illustrated. Together with its experiments on both supervised learning and reinforcement learning tasks, I can foresee its potential values in the field of reinforcement learning and NLP.

However, the paper does come with some drawbacks and some confusions:

1.  Lack of background work: Though closed related to memory neural networks, none of them is discussed and compared. For example, J Weston, S Chopra, and A Bordes. "Memory networks." ICLR (2014). Similarly, pointer networks should also be discussed, e.g. See et al. (2017). Memory modules are also popular for long-sequence processing in NLP, e.g. coreference resolution (Toshniwal et al. 2020).
    
2.  Insufficient empirical evidence: Experiments in section 5 are claimed to be long-sequence tasks, while their sequences are not long enough. The lengths of Copy, Scattered copy, and Add are 1000, which can even be handled by some transformers (e.g. T5). The truncation lengths are as small as 10 and up to 60 only, which can be much larger for real applications.
    
3.  Choice of uncertainty detector: While the description of "Reducing the cost of memory training" in section 2 seems intriguing, the method used in training is simplified into a point estimate of prediction error. In this case, it makes more sense to me to simplify the justification in section 2.

5.  While MemUP uses information theory to help the model to keep the most useful information, the basic idea of MemUP is very similar to LSTM, which purely relies on BPTT to learn how to keep memories. The authors show in sections 5 and 6 that MemUP outperforms LSTM, the baselines are presented in a negative light where the rollout lengths are less than 60 steps. The authors may clearly compare the difference between MemUP and LSTM, as well as providing more evidences to show the advantage of MemUP.  
questions: I have a few questions that are not clarified after reading the paper:

1\. In the uncertainty, how do you get p(y\_t | x\_t), given you are training a model to estimate the p(y\_t | x\_t, m_t)? Is it a separate module (L190)?

2\. For the RL experiments, you set r=1. Can you justify this, given it might hurt the performance of other models? I wonder what the results will be if r is much larger.

3\. In equation 2, the objective function for memory training assumes that the current memory m\_t should be helpful for future prediction at step k, while the m\_t will be updated to m_k at that moment. If my understanding is correct, there is a discrepancy between the objective and inference. Is that right?
limitations: I do not foresee any potential societal impacts that may occur.
review_title: null

Meta-review: metareview: The reviewers found the ideas presented in the paper interesting -- the use of mutual information to train memory for a model, and the clear presentation. Some questions were raised about demonstrating on a more elaborate set up such as NLP tasks -- the main experiments aside from the toy experiments of copy, etc algorithmic tasks, seem to be on RL experiments, but the method has been advertised more broadly in the motivation. Another reviewer raised the question of the complexity of training multiple networks. Nevertheless, the reviewers found the paper interesting enough to recommend a weak accept and I support that recommendation.

From a reviewers lens, I was a little surprised that the paper made no mention of prior works on maximizing mutual information between features of neural networks to improve results. As an example, see the following paper [1] that uses a mutual information regularizer between states at different steps of a recurrent neural networks. There is also a rich literature of doing so for convolutional neural networks. It would have made sense to compare how the idea in the paper performed in comparison to these methods (and in a sense the ablation study which looked at randomly choosing time steps, k, (regardless of the uncertainty estimator) is an experiment in this direction). I understand that part of the paper deals with the choice of time points to increase mutual information between, and so its probably more efficient than the other alternatives, but a comparison (or discussion in related works) would have made the paper stronger.

[1] Better Long-Range DependencyBy Bootstrapping A Mutual Information Regularizer. https://arxiv.org/pdf/1905.11978v1.pdf
confidence: Less certain
Decision: Accept