Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new intrinsic reward function for reinforcement learning, building on earlier definitions of "surprise". The proposed idea is to reward not the surprise itself but its novelty. The authors propose a specific approach to computing this intrinsic reward function and present an empirical analysis in a number of domains. 
strength_and_weaknesses: Strengths: The paper is in an important area of research relevant to the conference. The proposed concept for intrinsic reward is intriguing and can be useful. There is extensive empirical analysis, showing improved performance compared to existing approaches. 

Weakness: Experimental analysis has some limitations (see below) with the result that the general applicability and benefits of the approach are difficult to evaluate. The writing is relatively poor. 


clarity,_quality,_novelty_and_reproducibility: Novelty: Intrinsic motivation in reinforcement learning is an active area of research. "Surprise" is a concept that has been used frequently in this literature, and so is "novelty". The current paper builds on this literature to propose a new source of intrinsic motivation, namely the novelty of the surprise, as well as a mechanism for computing a precise value of intrinsic reward .

Clarity: The writing can be much improved. The main paper is repetitive and takes a long time to get to the point and to precise definitions. A large number of acronyms are introduced, which creates unnecessary difficulty for the reader. Page 1 is not the ideal placement for Figure 1, especially in its current form (see further comments on this below). Some important pieces of information are not provided in the main paper (e.g., description of the domains, which is critical for understanding the results). Some important results are relegated to the Appendices, which necessitated a lot of back-and-forth between the main paper and the appendices when reading the paper. Performance metrics should be clearly and explicitly defined in the main paper (e.g., In Figure 5, what precisely is being plotted?). 

The six frames and associated intrinsic rewards in Figure 1 are difficult to understand and evaluate without context. A much more useful figure would show the complete history of rewards through one or more episodes, along with associated frames at certain decision stages (see, for example, Figure 1 by Burda et al, 2018). 

Similarly, in Figure 3, intrinsic and extrinsic rewards can be shown as the agent is interacting with the environment. Currently, the figure shows 7 sample frames, which I did not find to be particularly informative without further context.

Learning curves corresponding to Table 1 would be useful to see in the main paper. 

Quality: The empirical evaluation is extensive but it would be useful to see further interrogation in some areas. One question that arises is the generality of the experimental results to other domains. Much of the discussion and results in the paper are on visual domains. In addition, the domains appear to be either deterministic or stochastic in a particular way (e.g., the stochasticity in the noisy-TV domain is entirely irrelevant to the task.) Another area that deserves further interrogation is the structure of the Surprise Memory. It would be useful to see an exploration of the design choices made here (ideally, with some alternative approaches). The ablation study in section 4 is on a single domain and has a narrow scope. Finally, while I appreciate the existing efforts of the authors to illustrate the behavior of the algorithm (e.g., Figure 4), the main emphasis in the paper is on high-level performance comparison (e.g., total reward obtained), with the result that the proposed computation of "surprise novelty" is not deeply understood. 

summary_of_the_review: The paper presents a potentially useful concept for intrinsic reward, as well as a particular implementation of it. But I struggled to reach a non-superficial understanding of the behaviour of the proposed method and to evaluate its general applicability and usefulness.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new method to manage surprise signals in reinforcement learning. Surprises are stored in a memory for each episode and the memory is wiped after each episode. An autoencoder is used to perform readouts. The memory module can be plugged in existing surprise generators. Benchmark results show that the addition of the memory is almost always beneficial. The main difference with respect to competing memory based approach in RL is that the proposed memory works at the surprise level and not at the state level.


strength_and_weaknesses: Strengths
- The use of memory on the surprise level is novel and well motivated
- The approach can be integrated easily on existing methods

Weaknesses

- Novelty is slightly limited, memory has been used here https://arxiv.org/pdf/2002.06038.pdf as also mentioned in the paper but with a different data to be stored (agent state vs surprise)
- Results show the improvement with respect to three methods but there are no direct comparison with existing sota. Specifically whye the only other memory based approach (https://arxiv.org/pdf/2002.06038.pdf) is not taken into account in the comparison?
In general a table with recent sota methods reported would improve the overall presentation.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, although I believe a better definition of the learning problem onto which the MANN approach is then plugged in would help also non purely RL researchers in benefitting from this work.

Novelty is slightly limited (see above)

The presentation is clear enough for a field expert to reproduce the work.
summary_of_the_review: The approach proposed plugs a memory module into RL system using surprise generation. The use of such module is shown to empirically improve results. Novelty is slightly limited and some comparisons are missing.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This method proposes some modifications to curiosity-based exploration methods to help combat some of the corner cases that they don't deal well with, which also results in increased performance. The motivation for this work is that exploration is still a really large challenge and reinforcement learning and even some of the best-proposed methods have some challenges in being able to explore properly in all types of environments (MDPs). The method proposed in this paper introduces a new metric related to surprise novelty, and the surprise novelty helps calculate and reduce the noisy TV problem for curiosity-based metrics. The paper does introduce some interesting ideas, and there's some evaluation at the end of the paper to show that this surprise novelty metric is less susceptible to some of the issues with normal curiosity base methods.
strength_and_weaknesses: pros
- The method in the paper does make some analysis in order to be able to correct and adjust some of the challenges with applying curiosity-based exploration bonuses in different types of environments.
- An adequate amount of experimental analysis at the end of the paper indicates the potential for such a method to support a contribution over prior curiosity base metrics.

cons
- Generally, some of the conceptual parts and description of the method are challenging to follow. In particular, some of the description around surprise and surprise novelty and what is the mathematical difference between these concepts is not thoroughly explained. This makes it challenging to understand the paper's novelty and reuse the findings of the paper and future work by the community
. There is additional related work that this paper should cite and compare. In particular, if the paper is citing the noisy TV problem and the general aspect that environments can have stochastic elements in them, then the paper should consider comparing to additional methods that use surprise minimization[1,2].

- [1] Berseth, G., Geng, D., Devin, C. M., Rhinehart, N., Finn, C., Jayaraman, D., & Levine, S. (2021). SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments. International Conference on Learning Representations
- [2] Rhinehart, N., Wang, J., Berseth, G., Co-Reyes, J. D., Hafner, D., Finn, C., & Levine, S. (2021). Intrinsic Control of Variational Beliefs in Dynamic Partially-Observed Visual Environments. Advances in Neural Information Processing Systems, 34.
clarity,_quality,_novelty_and_reproducibility: 
- The introduction for this paper is less than simple to understand. there is some immediate discussion between surprise and surprise novelty, yet neither of these concepts are explained in sufficient detail within the introduction. The introduction then goes on to somewhat discuss the importance of considering these two different aspects and why memory is helpful but not why memory is necessary. 
- More information is needed to see why the diagram in figure 2 is an ideal design to be able to memorize and recover surprising novelty. The paper should also cite related work on being able to memorize normal trajectories in some type of sequence. 
- For example, it was not clear until the middle of the second page that surprised novelty might be the second-order version of surprise. 
- It's not obvious if proposition one holds. We don't have much information about what $$X$$ and $$U$$ should be.
- The surprise norm appears to be similar to the prediction mechanism from RND. The RND network is learning to estimate the surprise expectation and then comparing that to the current observation. More details that describe the difference between RND and the prediction comparison mechanism used in this work are needed to understand the novelty.
- The details around the discussion of how the episodic memory is used and trained are difficult to understand. Why is there an additional model for training to predict some of the memory while also that a buffer of the most recent history of the agent is kept in order to be able to compute the surprise novelty? A diagram to further explain the process and necessary components to be able to predict or produce the surprise novelty would be very helpful to the reader.
- The writing in section 2 is also challenging to understand which pieces are the novel pieces being introduced in this work? The authors should consider editing a background or prior methods section. An example to include in this section is the autoencoder training section. Training autoencoders is not new and has been around for many years.
- The comparison on the paper should select more environments that are used in prior (ICM and RND) papers.
- Table 2, and in general for the results of the paper, needs to include the confidence information and statistics over this analysis. How many random seeds are used to run this analysis? How much does the distribution of training in these different methods overlap? Preferably a t-test should be performed over the different methods in the paper so we can understand how statistically confident we can be in these results.
- In addition, the results in figure 5 do not appear to show converged policies. It is possible that prior methods end up outperforming the method proposed in this paper if more training time is given. This makes the results shown in figure 5 difficult to use in the assessment.
- Given that a large motivation for this paper is to deal with undesired or unusual stochastic elements of the environment, this paper should also cite the new line of work on surprise minimization[1,2] that, by default, has a well-conceptualized solution to this problem.

summary_of_the_review: The proposed method for performing better curiosity-based exploration is promising. However, the writing and analysis make it difficult to understand how the method works and if it is novel. These issues will need to be addressed for me to raise my score.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work introduces a new method for intrinsic motivation in RL called Surprise Memory (SM), that in similar fashion to surprise-based intrinsic motivation, provides an intrinsic reward to the agent when it finds a 'new' observation. SM differs in that it takes in account if this 'surprise' was already expected from previous memories and reduce the intrinsic reward if that is the case.

Intuitively, the method consists of a memory module (of fixed size) with previous surprises and an autoencoder (AE) that tries to reconstruct a query with the current surprise and the closest matching surprise within memory. The worse the AE does, the larger the intrinsic reward. 
strength_and_weaknesses: I believe that this is a good work, the method is novel and non-trivial, authors provide abundant theoretical and experimental ground for their approach. The paper is also well presented and easy to follow. 

Weaknesses:
1) The authors only present the strengths of this method, which seem to be many, but I could not see any discussion on limitations or main weaknesses of this approach. The future lines proposed at the end feel a little hand waving in my opinion, it doesn't provide much ground for future researchers to continue this work.

2) The need of the AE, while it is clearly demonstrated in the ablation experiments, it is lacking an explanation of why in the theoretical side -besides that it works better-. Going through the work I could grasp some intuition, but would be good that the authors detail this further in section 2 since intuitively, one could think that the norm between the surprise of this state and the one in memory would already tell you if this is an "expected" surprise.

Extra:
I don't want to point this as a weakness but I would like to ask the authors about how does the SM intrinsic reward evolves through training when you are training for the same objective for long. What I mean is that, if you are training your agent to get a red box at the end of the episode, since the box is exotic it provides high intrinsic reward. However, as you train and train the agent the AE will get better to recover that red box surprise at the end, contrarily, the AE probably cannot learn so well to reconstruct the tv noise. My question is, does the intrinsic reward for that final goal get relatively lower to the one of the tv as training progresses? And if not, could Authors explain why it doesn't happen?

clarity,_quality,_novelty_and_reproducibility: The work is easy to follow and clearly written. It could be fixed following the suggestions I wrote above but it can be easily fixed. The only important thing is that authors didn't share the code neither could I find any mentions to it in the appendix. Would be a good thing to have for reproducibility. 

I couldn't find any flaws in the proofs at the appendix about their theoretical propositions. Also, they provide abundant detail on hyperparameters, pseudocode and experiment details there.

The paper has abundant experiments, ablations, baselines and benchmarks and the results are quite promising. As far as I know the method itself is novel, I only know of [1] that also works with intrinsic motivation and memory. The method presented here is quite different from that one, but probably it is worth to add to the related work.


[1]Fang, Zheng, Biao Zhao, and Guizhong Liu. "Image Augmentation Based Momentum Memory Intrinsic Reward for Sparse Reward Visual Scenes." arXiv preprint arXiv:2205.09448 (2022).
summary_of_the_review: Good paper, novel method, targets one of the main issues of intrinsic motivation with promising results. The work seems solid and sound although could be improved by detailing weaknesses and limitations and with further motivation of the AE mechanism.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This method introduces a specific intrinsic reward for exploration and achieves improvements over methods it extends on small number of domains. However, at the first glance, this reward looks somewhat arbitrary and therefore needs justification. Despite authors giving one, several reviewers had hard time understanding the method or why it should work (I also have hard time following it). I recommend that either the authors work more on explaining the method or provide more extensive experimental evaluation justifying that the method works in general, regardless if it easy or hard to understand why.
justification_for_why_not_higher_score: The motivation is unclear, and either needs better explanation or more extensive experiments.
justification_for_why_not_lower_score: The method works on domains tested.
Decision: Reject