Intrinsic Motivation via Surprise Memory
Authors: Hung Le
Kien Do
Dung Nguyen
Svetha Venkatesh
Venue: Submitted to ICLR 2023 (2023)
Keywords: reinforcement learning
intrinsic motivation
exploration
memory

Abstract: We present a new computing model for intrinsic rewards in reinforcement learning that addresses the limitations of existing surprise-driven explorations. The reward is the novelty of the surprise rather than the surprise norm. We estimate the surprise novelty as retrieval errors of a memory network wherein the memory stores and reconstructs surprises. Our surprise memory (SM) augments the capability of surprise-based intrinsic motivators, maintaining the agent's interest in exciting exploration while reducing unwanted attraction to unpredictable or noisy observations. Our experiments demonstrate that the SM combined with various surprise predictors exhibits efficient exploring behaviors and significantly boosts the final performance in sparse reward environments, including Noisy-TV, navigation and challenging Atari games. 

Under review as a conference paper at ICLR 2023
INTRINSIC MOTIVATION VIA SURPRISE MEMORY
A nonyymon authors
Paper under double-blind review
ABSTRACT
We present a new computing model for intrinsic rewards in reinforcement learning that. addresses the limitations of existing surprise-driven explo- ralions. The reward is the novelty of the surprise rather than the surprise norm. We estimate the surprise novelty as retrieval errors a memory net- work wherein the memory stores and reconstructs surprises. Our surprise memory (SM) augments the capability of surprise-based based intrinsic motiva- tors, maintaining the agent's interest in exciting exploration while reducing unwanted attraction to unpredictable or noisy observations. Our experi- ments demonstrate that the SM combined with various surprise predictors exhibits efficient exploring behaviors and significantly boosts the final per- formance in sparse reward environments, including Noisy-TV, navigation and challenging Atari games.
INTRODUCTION
What motivates agents to explore? Successfully answering this question would enable agents to learn efficiently in formidable tasks. Random explorations such c-greedy inefficient in high dimensional cases, failing to learn despite training for hundreds of million sleps in sparse reward games (Bellemare al. 2016). Alternative approaches propose use intrinsic moliivation Lo aid explorat ion by adding bonuses to the environmentss 's rewards (Bellemare al. 2016) Stadie al. 2015). The intrinsic reward is often proportional to the novelty of the visiting state: it is high if the state is novel (e.g. different from the past ones (Badia or less frequently visited (Bellemare et. al. 2016 Tang al. 2017
al.
2020
2019
Surprise Novelty Surprise Norm
0.7
-0.7
1
2
3
4
Figure 1: Montezuma Revenge: surprise nomelly better reflects the originality of the envi- ronment than surprise norm. While surprise norm can be significant even for dull events such as those in the dark room due to unpredictability, surprise novelty tends to be less (3rd and 6th image). On the other hand, surprise novelty can higher in truly vivid states on the first visit the ladder and island rooms (1st and 2nd image) and reduced on the second visit h4th and 5th image). Here, surprise novelty and surprise norm are quantified and averaged over steps in each room.
Another view of intrinsic motivation is from surprise, which refers to the result. of the experience being unexpected, and is determined by the discrepancy between the expectation (from the gent's prediction) and observed reality (Barto et al.] 2013) Schmidhuber) 2010). Technically, surprise is the difference between prediction and observation representation
vectors. The norn of the residual (Le. prediction error) is used the intrinsic reward.

---

Under review as a conference paper at ICLR 2023
Here, we will use the terms "surprise" and "surprise norm" to refer to the residual vector and its norm, respextively, Recent works have estimated surprise with various predictive models such as dynamics (Stadie et al. 2015), episodic reachability (Savinov et al. 2018) and inverse dynamics (Pathak et al. 2017| 71; and achieved significant improvements with surprise norm (Bunda et al. 20188). However, surprise-hased agents tend to be overly curious about. noisy or unpredictable observations (Itti and Baldi 2005 Schmidhuber [991]. For example, consider an agent watching television screen showing white noise (noisy-TV TV problem). The TV is boring, yet the agent cannot predict the screen's 's content and will be attracted to the TV due to its high surprise norm. This distraction or "fake surprise" is common in partially observable Markov Decision Process (POMDP), including navigalion Lasks and Atari games (Burda et al.] 2018b). Many works have addressed this issue by relying on the learning progress (Achiam and Sastry 2017 Schmidhuber 1991) or random net work distillat iln (R.N)) Burda et al.] 2018bj. However, the former is computationally expensive, and the latter requires many samples to perform well.
This paper overcomes the "fake surprise" issue by using surprise novelty - a new concept that measures the uniqueness of surprise. To identify surprise novelty, the agent needs to compare the current surprise with surprises in past encounters. One way to do this is to equip the agent with some kind of associative memory, which we implement as an autoencoder whose task is to reconstruct a query surprise. The lower the reconstruction error, the lower the surprise novelty. A further mechanism needed to deal with the rapid changes in surprise structure within an episode. As an example, if the agent meets the same surprise at two time steps, its surprise novelty should decline, and with simple antoencoder this will not happen. To remedy this, we add an episodic memory, which stores intra-episode surprises. Given the current surprise, this memory can retrieve similar "surprises" presented earlier in the episode through an attention mechanism. These surprises act as a context added to the query to help the autoencoder better recognize whether the query surprise has been encountered in the episode not. The error between the query and the autoencoder's output is defined as surprise novelty, to which the intrinsic reward is set proportionally.
We argue that using surprise novelty as an intrinsic reward is better than surprise norm. As in POMDPs, surprise norms can be very large since the agent cannot predict its environment perfectly, yet there may exist patterns of prediction failure. If the agent can remember these pattems, it will not feel surprised when similar prediction errors appear regardless of the surprise norms. An imporlant. emergent property of this architecture is that when random observations are presented (e.g,, white noise in the noisy-TV problem), the autoencoder can act as an identity transformation operator, thus effectively "passing the noise through" to reconstruct it with low error. We conjecture that the autoencoder is able to do this with the surprise rather than the observation as the surprise space has lower variance, and we show this in our paper. To make our memory system work on the surprise level, we adopt an intrinsic motivation method to generate surprise for the memory, The surprise generator (SG) can be of any kind based on predictive models and is jointly trained with the memory to optimize its own loss function. To train the surprise memory (SM), we optimize the memory's parameters to minimize the reconstruction error.
Our contribution is to propose a new concept of surprise novelty for intrinsic motivation. We argue that it reflects better the environment originality than surprise norm (see motivating graphics Fig. D]. In our experiments, the SM helps RND (Burda et al., 2018b) perform well in our challenging noisy-TV problem while RND alone performs poorly. Not only with RND, we consist ently demonstrate significant performance gain when coupling three different SGs with our SM in sparse-reward lasks. Finally, in hard exploration Atari games, we boost. the scores of 2 strong SGs, resulting in better performance under the low-sample regime.
2
METHODS
2.1 SURPRISE NOVELTY
Surprise is the difference between expectation and observation (Ekman and Davidson 1994). If a surprise repeats, it is no longer a surprise. Based on this intuition, we hypothesize that surprises can be characterized by their novelties, and an agent's 's curiosity is driven by the
2

---

Under review as a conference paper at ICLR 2023
SG
14
M
I
SM
Figure 2: Surprise Generator Sanneatoo+SSuprisss Surprise Memory (SG+ SM). The SG takes input. I from the environment to estimate the surprise de at. state The SM consists of two modules: an episodic memory (M) and an autoencoder network (W). M is slot-based, storing past surprises within the episode. At any timestep t, given surprise We, M retrieves read-out uf to form a query surprise qi = [uf, uz] to W. W tries to reconstruct the query and takes the reconstruction error (surprise novelty) the intrinsic reward r/.
surprise novelty rather than the surprising magnitude. Moreover, surprise novelty should be robust against noises: it is small even for random observations. For example, watching a random-channel TV can always be full of surprises appear next. However, the agent should soon find it boring since the surprise of random noises reoccurs repeatedly, and the channels entirely unpredictable.
cannot expect which channel will
We propose using a memory-augmented neural network (MANN) to measure surprise nov- elty. The memory remembers past. surprise pal terns, and if surprise can be retrieved from the memorr, it is not novel, and the intrinsic motivation should be small. The memory also be viewed for random, pattern-free surprises, making them retrievable. Surprise novelty has an inter- esting property: if some event is unsurprising (the expectalion-realltyy residual is 0), its surprise (0 with norm 0) is always perfectly retrievable (surprise novelty 0). In other words, low surprise norm means low surprise novelty. On the contrary, high surprise norm can have little surprise novelty as long as the surprise can be retrieved from the memory either through associative ive recall or pass-through mechanism. Another property is that the variance of surprise is generally lower than that of observation (state), potentially making the learning surprise space easier. This property is formally stated as follows.
reconstruction network. The network can pass its inputs through
Proposition 1. Let X and U be random variables representing the observation and surprise at the same timestep, respectively. Under an imperfect SG, the following inequality holds:
Vi: where (xx)² and (o!)² denote the i-th diagonal elements of var(X) and var(U), respectively.
(0!)²
Proof. See Appendix
2.2 SURPRISE GENERATOR
Since our MANN requires surprises for its operation, it is built upon prediction model, which will be referred to as Surprise Generators (SG). In this paper. we adopt many well- known SGs (e.g. RND (Burda et al., 2018b) and ICM (Pathak et al.) 2017)) to predict the observation, compute the surprise ii and its norm for every step the environment. The surprise norm is the Euclidean distance between the expectation and the act act observation:
||u, ||SG (I) Oill
(1)
where un E R" the surprise vector of size n, I the input of the SG at step t of the episode, SG (Io) and O the SG's prediction and the observation target, respectively. The input I specific to the SG architecture choice, which can be the current (st) or previous state, action as). The observation target. O is usnally transformation (can be identical or
3

---

Under review as a conference paper at ICLR 2023
random) of the current state St, which serves as the target for the SG's prediction. The SG usually trained to minimize:
Lsg E, [1|,,]]]
(2)
Here, predictable observations have minor prediction errors or little surprise. One issue is
that a great surprise norm can be simply due to noisy or distractive observations. Next, we
propose a remedy for this problem.
2.3 SURPRISE MEMORY
The surprise generated by the SG is stored and processed by a memory network dubbed Surprise Memory (SM). It consists of an episodic memory M and an autoencoder network W, jointly optimized to reconstruct any surprise. At each timestep, the SM receives surprise We from the SG module and reads content uj from the memory M. {uf, ur) forms surprise query qi to W to retrieve the reconstructed q. This reconstruction will be used to estimate the novelty of surprises forming intrinsic rewards ri. Fig. 2 summarizes the operations of the components of our proposed method. Our 2 memory design effectively recovers surprise novelty by handling intra and inter-episode surprise patterns thanks to M and W, respectively. M can quickly adapt and recall surprises that occur within an episode. W is slower and focuses more on consistent surprise patterns across episodes during training.
Here the query lt can be directly set to the surprise We- However, this ignores the rapid change in surprise within an episode. Without M, when the SG and W are fixed (during interaction with environments). their outputs u, and q stay the same for the same input 4t Hence, the intrinsic reward ri also stays the same. It is undesirable since when the agent observes the same input different timesteps (e.g. J = [2), we expect its curiosity should decrease in the second visit (ri <r²). Therefore, we design SM with M to fix this issue.
The episodic memory M stores representations of surprises that the agent encounters during an episode. For simplicity, M is implemented as first-in-firstoutt queue whose size fixed as N. Notably, the content of M wiped out at the end of each episode. Its information is limited to single episode. M can be viewed as matrix: M N-xdd where d is the size the memory slot. We denote M (j) as the j-th row in the memory, corresponding to the surprise Ut-j. To retrieve from M read-out uf that is close to u4, perform content-based attention (Graves et al. 2014) to compute the attention weight as ur (j) [(.Qiil[[
(G)) The read-out from M is thhen of = w.MV E R.. Here, Q E Raad and V RR are learnahle weights mapping between the surprise and the memory space. To force the read-out close to ut, we minimize:
LM = E, [|]uf - uell
(3)
The read-out and the SG's surprise form the query surprise to W: de = [ug, un] R22n
M stores intra-episode surprises to assist the autoencoder in preventing the agent from exploring "fake surprise" within the episode. Since we train the parameters to reconstruct i, using past surprises in the episode, if the agent visits state whose surprise is predictable from those in M, ||u]- uill should be small. Hence, the read-out context uf contains no extra information than d, and reconstructing qi from W becomes easier as it is equivalent to reconstructing uy In rostrast, visiting diverse states leads to a more novel read-out, uf and makes it more challenging (o reconstruct gt, generally leading to higher intrinsic reward.
The autoencoder network W can he viewed as an associative memory of surprises that. persist across episodes. At timestep t in any episode during training, W is queried with ge to produce reconstructed memory z. The surprise novelty is then determined as:
(4) which is the norm of the surprise residual qe q. It will be normalized and added to the external reward as an intrinsic reward bonus. The details of computing and using normalized intrinsic rewards can be found in Appendix
r = llq: - -ull
We implement W as a ford-forward neural network that learns to reconstruct its own inputs. This kind of autoencoder has been shown to be equivalent to associative memory that.
4

---

Under review as a conference paper at ICLR 2023
(a)
(b)
Returr
0.8
1.5
Surprise Novelty
MNIR
Surprise Norm
0.6
0.5
0.4
0.0
RND
0.2
RND
-1.0
2M
4M
Step
Figure 3: Noisy-TV: (a) mean-normalized intrinsic reward (MNIR) produced by RND and RND+SM at 7 selected steps in an episode. (b) Average task return (mean over runs) over million training steps.
supports memory encoding and retrieval through attractor dynamics (Radhakrishnan et al. 2020). The query surprise is encoded to the weights of the network via backpropagation as minimize the reconstruction loss below:
Cw E [r]] E. (q)) 9:1l
(5)
Here, q. - (Q.). Intuitively, it is easier to retrieve non-novel surprises experienced many times in past episodes. Thus, the intrinsic reward is lower for states that leads to these famillar surprises. On the contrary, rare surprises are harder to relrieve, which results in high reconstruction errors and intrinsic rewards. W like long-term inter-episode asso- ciative memory. Unlike slot-based memories, it has a fixed memory capacity, can compress information and learn data representations. We could store the surprise in a slot-based memory across episodes, but. the size. of this memory would be autonomous, and the data would be stored redundantly. Hence, the quality of the stored surprise will reduce as more and more observations come in. Readers can refer to Appendix A to see the architecture details and how W can be interpreted as implementing associative memory. The whole sys- tem SG+SM is trained end-to-end by minimizing the following loss: L Lsg + L. Lw. Here, we block the gradients from Lw backpropagated to the parameters of SG to avoid trivial ructions of q:- Pseudocode of our algorithm is presented in Appendix B
EXPERIMENTAL RESULTS
3.1 NOISY-TV: ROBUSTNESS AGAINST NOISY OBSERVATIONS
We use Noisy-TV an environment designed to fool exploration methods (Burda et al. 2018b Savinov al. 2018), confirm that our method can generate intrinsic rewards that (1) are more robust to noises and (2) can discriminate rare and common observations through surprise novelty. We simulate this problem by employing 3D maze environment with a random map structure. The TV is not fixed in specific locations in the maze to make it more challenging. Instead, the agent. "brings" the TV with it and can choose to watch TV anytime. Hence, there are three basic actions (turn left, right, and move forward) plus an action: watch TV. When taking this action, the agent will see a white noise image sampled from standard normal distribution and thus, the number of TV channels can be considered infinity. The agent's 's state an image of its viewport, and its goal is to search for red bax randomly placed in the maze (+1 reward if the agent reaches the goal). The baseline is RND (Burda et al. 2018b), a simple yet strong SG that is claimed to obviate the stochaste astic problems of Noisy- TV. Our SG+SM SM model
RND the SG, so we name it RND + SM. Since our model and the baseline share the same RND architecture, the difference in performance must be attributed our SM.
Fig. 3 (a) illustrates the mean-normalized intrinsic rewards (MNIR) I measured at different states es in our Noisy-TV environment. The first two states are noises, the following three states es are common walls, and the last two are ones where the agent sees the box. The
See Appendix for information this metric.
5

---

Under review as a conference paper at ICLR 2023
Task
Baseline
RND/SM
ICM/SM
NGU/SM
AE/SM
KD
0.0.1..0
1..3:126/77.3.1
5.915/44.7.3
64.413/83.411
1.411/ 91.216
DO
-27.0.1..7
-13.6.18770.811 71.0+35/71.1+5
43.6116
4/.6-3/
-5.142/ -5.112/07.5119 29.0+6/ 70.00/00022
67.51
LC
78.0+1.7
56.2+440/ 84.6+1 42.2+40/ 69.3=
Table 1: MiniGrid: test performance after 10 million training steps. The numbers are average lask relurn 100 over 128 episodes (mean over 5 runs). Bold denotes best. results on each lask. Italic denotes that. SG+SM is better than SG regarding Cohen effect. size less Ihan 0.5.
MNIR hars show that both models are aftracted mainly by the noisy TV, resulting in the highest MNIRs. However, our model with SM suffers less from noisy TV distractions since its MNIR is lower than RND's. We speculate that SM is able to partially reconstruct the white- noise surprise via pass-through mechan ism, making the normalized surprise novelty generally smaller than the normalized surprise norm in this case. That mechanism is enhanced in SM with surprise reconstruction (see Appendix D.1 for explanation).
On the other hand, when observing red bux, RND+SM shows higher MNIR than RND. The difference between MNIR for common and rare states is also more prominent in RND+ SM than in RND because RND prediction is not perfect even for common observations, creating relatively significant surprise norms for seeing walls. The SM fixes that issue by remembering surprise patterns and successfully retrieving them, producing much smaller surprise novelty compared to those of rare events like seeing red box. Consequently, the agent with SM outperforms the other by a massive margin in task rewards (Fig. 3 (b)).
As we visualize the number of watching TV actions and the value of the intrinsic reward by RND+SM and RND over training time, we realize that RND+SM helps the agent take fewer watching actions and thus, collect smaller amounts of intrinsic rewards compared to RND. We also verify that our proposed method outperforms a simplified version of SM using counts to measure surprise novelty and vanilla baseline that does not use intrinsic motivation. The details of these results are given in Appendix D.1
3.2 MINIGRID: COMPATIBILITY WITH DIFFERENT SURPRISE GENERATORS
We show the versatility of our framework SG + SM by applying SM 4 SG backbones: RND [Burda et al. 2018b), 1CM (Pathak et al. 2017). J. NGU (Badia et al. 2019) and autoencoder- AE (see Appendíx D.2 for implementation details). We test the models on three tasks from MiniGrid environments: Key-Door (KD), Dynamic-Ohstacles (DO) and Lava-Crossing (L.C) "Chevalier-Boisvert et al. 2018). If the agent reaches the goal in the tasks, receives +1 reward. Otherwise, it can be punished with negalive rewards if it collides with obstacles or takes too much time to finish the task. These environments are not. stochastic as the Noisy-TV but they still contain other types of distraction. For example, in KD, the agent can be attracted to irrelevant actions such as going around to drop and pick the key. In DO, instead of going to the destination, the agent. may chase obstacle balls flying around the map. In L.C the agent can commit unsafe actions like going near lava areas, which are different from typical paths. In any case, due to reward sparsity, intrinsic motivation is beneficial. However, surprise alone may not be enough to guide an efficient exploration since the observation can be too complicated for SG to minimize its prediction error. Thus, the agent quickly feels surprised, even in unimportant states.
Table [ shows the average returns of the models for three tasks. The Baseline the PPO backbone trained without intrinsic reward. RND, ICM, NGU and AE are SGs providing the PPO with surprise-norm rewards while our method SG+SM uses surprise-novelty rewards. The results demonstrate that models with SM often outperform SG significantly and always contain the best performers. Notably, in the LC task, SGs hinder the performance of the Baseline because the agents are attracted to dangerous vivid states, which are hard to predict but cause the agent's 's death. The SM models avoid this issue and outperform the Baseline for the case of (M. Compared to AE, which computes intrinsic reward based on the novelty of the state, SM+SM shows a much higher average score in all tasks. That manifests the importance of modeling the novelty of surprise instead of states.
6

---

Under review as a conference paper at ICLR 2023
Task
EMI
LWM 4
RND
LWM
LWM-SM
RND
RND+SM
Freeway
33.8
30.8
33.3
31.1
31.6
22.2
22.2
Frostbite
7,002
8,409
2,227
8,598
10,258
2,628
5,073
Veuture
646
998
707
985
1,981
1,081
1,119
Gravitar
558
1,376
546
1,242
1,693
739
987
Solaris
2,688
1,268
2,051
1,839
2,065
2,206
2,420
Montezuma
387
2,276
377
2,192
2,269
2,475
5,187
Norm. Mean
61.4
80.6
42.2
80.5
97.0
50.7
74.8
Norm. Median
34.9
60.8
32.7
66.5
83.7
58.3
84.6
Table 2: Atari: average return over 128 episodes after 50 million training frames (mean over
5 runs). is from prior work (Ermolov and Sebe 2020). & is our run. The last two rows are mean and median human normalized scores. Bold denotes best results. Italic denotes that SG+ SG is significantly better than SG regarding Cohen effect size less than 0.5.
To analyze the difference between the SG+SM and SG's MNIR structure, we visualize the MNIR for each cell in the map of Key-Door in Appendix's Figs. 5 (b) and (c). We create synthelic trajectory that. scans through all the cells in the big room the left and, al each cell, uses RND+SM and RND models to compute the corresponding surprise-norm and purprsse-ovvelty MNIRs, respectively. As shown in Fig. 5 (b), RND+SM selectively identifies truly surprising events, where only a few cells have high surprise-novelty MNIR. Here,
visually detect. three important events that receive the most MNIR: seeing the key (bottom row), seeing the door side (in the middle of the rightmost column) and approaching the front of the door (the second and fourth rows). Other less im portant cells are assigned very low MNIR. On the contrary, RND often gives high surprise-none norm MNIR to cells around important ones, which creates a noisy MNIR map in Fig. (c). As result, RND's performance is better than Baseline, yet far from that of RND SM. Another analysis of how surprise novelty discriminates against surprises with similar norms is given in A Appediix's Fig.
3.3 ATARI: SAMPLE-EFFICIEN BENCHMARK
We adopt the sample-effciency Atari benchmark (Kim al. 2019) on six hard exploration games where the training budget. is only 50 million frames. We use our SM to augment. SGs: RND (Burda al.] 2018h) and LWM (Ermolov and Sebe 2020). Unlike RND, LWM uses recurrent world model and forward dynamics to generate surprises. Details of the SGs, training and evaluation are in Appendix D.3 We run the SG and SG+SM in the same codebase and setting. Table 2 reports works, showing SM-augmented models outperform their SG counterparts in all games (same codebase). In Frostbite and Montezuma Revenge, RND+SMMs score is almost twice many as thai of RND. For LWM+-SM gamees such as Gravitar and Venture observe more than 40% improvement. Overall, LWM HM and RND +M achieve the best. mean and median human normalized score, improving 16% and 22% w.r.t the best SGs, respectively. Notably, RND SM shows significant improvement for the notorious Montezuma Revenge.
and representative results from prior
We also verify the benefit of the SM in the long run for Montezuma Revenge and Frostbite. As shown in Fig. A (a,b), RND+SM still significantly outperforms RND after 200 mil- lion training frames, achieving average scores of 10,000 and 9.000, respectively. The result demonstrates the scalability of our proposed method. When using RND and compute the average MNIR in several rooms in Montezuna Revenge (Fig. [ we realive that SM makes MNIR higher for surprising events in rooms with complex structures while depressing the MNIR. of fake surprises in dark rooms. Here, even in the dark room, the movement of agents (human or spider) is hard to predict, leading to a high average MNIR. On the contrary, the average MNIR of surprise novelty is reduced if the prediction error be recalled from the memory.
to
Finally, measuring the running time of the models, notice little computing overhead caused by our SM. On our Nvidia A100 GPUs, LWM and WWM+SMPP average time for one 50M training 11h 38m and 12h 10m, respectively. For one 200M training, RND and

---

Under review as a conference paper at ICLR 2023
(a) Montezuma Revenge
(b) Frostbite
[c] Dynamic Obstacles
(6) 3D Navigation
10000
RND+SM
8000
RND+SM
5M (no 93
RND+SM
Avg. Retura
7500
RND
0.5
Cumm. Retarn
150
RED
6000
5000
Mariy
0.0
100
4000
2500
2000
-0.5
5MPN-1634e-31
50
53554-11 2m13)
0
110
(5-12--44)
100M
200M
100M
200M
10M
50M Slep
100M
Step
Step
Step
Figure 4: (a,b) Atari long runs over 200 million frames: average relurn over 128 episodes. (c) Ablation study on SM's components. (d) MiniWorld exploration without task reward: Cumulative task returns over 100 million training steps for the hard setting. The learning curves are ttdn over 5 runs.
RND SM's average times are 26h 24m and 28h 1m, respectively, These correspond to only 7% more training time while the performance gap is significant (4000 scores).
3.4 ABLATION STUDY
Role of Memories Here, we use Minigrid's Dynamic-Obstacle task to study the role of M and W in the SM (built upon RND as the SG). Disabling W, we directly use l = Ieif, well as the intrinsic reward, and name this version: SM (no W). To ablate the effect of M, we remove of from & and only use q: u, as the query to W, forming the version: SM (no M). We also consider different. episodic memory capacity and sloi size N-d= {32 --4, 4, 128 - 16, 1024 - 64). As N and d increase, the short-term context. expands and more past surprise information is considered in the attention. In theory, big M is helpful to capture long-term and more accurate context for constructing the surprise query.
Fig. (c) depicts the performance curves of the methods after 10 million training steps. SM (no W) and SM (no M) show weak signs of learning, confirming the necessity of both modules in this task. Increasing N-d from 42-44-226 to 1024-00 64 improves the final performance. However, 1024 64 is not significantly better than 128 16, perhaps because it is unlikely to have similar surprises that are more than 128 sleps apart Thus, a larger allent ion span does not provide benefit. As a result, we keep using N 128 and d 16 in all other experiments for faster computing. We also verify the necessity of M and W in Montezuma Revenge and illustrate how M generates lower MNIR when 2 similar event occurs in the same episode in Key-Door (see Appendix D.4)
No Task Reward In this experiment, we remove task rewards and merely evaluate the agent's 's ability to explore using intrinsic rewards. The task is to navigate 3D rooms and get a +1 reward for picking an object. Shhevalier-oiiveert 2018). The state is the agent's image view, and there is no noise. Withhot. task rewards, it. is crucial to maintain the agent.'s 's interest in unique events of seeing the objects. In this partially observable environment, prediccio methods may struggle to explore even without noise due to lacking in- formation for good predictions, leading to usually high prediction errors, For this test evaluate random exploration agent (Baseline), RND and RND in 2 settings: room with three objects (easy), and rooms with one object (hard).
To see the difference among the models, we compare the cumulative task rewards over 100 million steps (see Appendix D.4 for details). RND is even worse than Baseline in the easy setting because predicting causes high biases (intrinsic rewards) towards the unpredictable, hindering exploration if the map is simple. In contrast, RND++SM uses surprise novelty, generally showing smaller intrinsic rewards (see Appendix Fig. 12 (right)). Consequently, our method consistently demonstrates significant improvements over other baselines (see Fig. 4 (d) for the hard setting).
4 RELATED WORKS
Intrinsic motivation approaches usually give the agent reward bonuses for visiting novel states to encourage exploration. The bonus is proportional to the mismatch between the

---

Under review as a conference paper at ICLR 2023
predicted and reality, also known as surprise (Schmidhuber 2010). One kind of predictive model is the dynamics model, wherein the surprise is the error of the models as predicting g the next state given the current state and action (Achiam and Sastry 2017, Stadie et al. 2015). One critical problem of these approaches is the unwanted bias towards transit lons where the prediction target. is a stochastic function of the inputs, commonly found in partially observ- able environments. Recent works focus on improving the features of the predictor's input by adopting representation learning mechanisms such inverse dynamics (Pathak et al., 2017), variational autoencoder, random/ /pixel features (Burda et al. 2018a), or whitening transform (Ermolov and Sebel 2020). Although better representations ations may improve the reward bonus, they cannol. completely solve the problem of si ochaasiicc dynamics and thus, fail in extreme cases such as the noisy-TV problem (Burdaa el. al., 2018b).
Besides dynamics prediction, several works propose to predict other quantities as functions of the current. state by using autoencoder (Nylend] 2017), episodic memory (Savinov al., 2018), and random network (Burda et al. 2018b). Burda et al. (2018) claimed that using deterministic random target network is beneficial in overcoming stochasticity issues. Other methods combine this idea with episodic memory and other techniques, achieving good results in large-scale experiments (Badia et al. 2020: 2019). From an information theory perspective, the nolation of surprise can be linked to information gain uncertainly, and predictive models can be teatedd as parameterized distributions (Achiam and Sastry 2017 Houthooft al., 2016); Still and Precup 2012). Furthermore, to prevent the agent from unpredictable observations, the reward bonus can be measured by the progress of the model's prediction (Achiam and Sastry 2017 Lopes al. 2012 Schmidhuber 1991). However, these met hods are complicated and hard to scale, requiring heavy computing. A different angle to handle strchaatic observations during exploration is surprsie minimization (Berseth al.) 2020: Rhinehart et al. 2021). In this direction, the agents get. bigger rewards for seeing more familiar staes Such a stralegy is somewhat, opposite to our approach and suitable for unstable environments where the randomness occurs separately from the agents' actions.
These earlier works rely on the principle of using surprise as an incentive for exploration and differ from our principle that utilizes surprise novelty. Also, our work augments these existing works with a surprise memory module and can be used as a generic plug-in improvement for surprise-based models. We note that our memory formulation differs from the memory- based novelty concept using episodic memory (Badia et al.] 2019). momentum memory (Fang et. al.) 2022), , or counting (Bellemare el al.) 2016 [Tagg al. 2017) because our memory operales on the surprise level, not the state level. In our work, exploration ion is discouraged not only in frequently visited states but also in states whose surprises can be reconstructed using S.M Our work provides a more general and learnable novelty detection mechanism, which is more flexible than the nearest neighbour search or counting lookup table.
DISCUSSION
This paper presents Surprise Generator-Surprise Memory (SG+SMM) framework to compute surprise novelty as an intrinsic motivation for the reinforcement learning agent. Exploring with surprise novelty is beneficial when there repeated patterns of surprises or random observations. For example, in the Noisy-TV problem, our SG+SM can harness the agent's 's tendency to visit noisy stales such as watching random TV channels while encouraging it to explore rare events with dist inctive surprises. We empirically show that our SM can supplement three surprise-basee based SGs to achieve more rewards in fewer training steps in three grid-world environments. In 3D navigation without external reward, our method significantly outperforms the baselines. On two strong SGs, our SM also achieve superior results in hard-exploration Atari games within 50 million training frames. Even in the long run, our method maintains a clear performance gap from the baselines, as shown in Montezuma Revenge and Fite. If we view surprise as the first-order error between the observation and the predicted, surprise novelty-the retrieval error between the surprise and the reconstructed memory, is essentially the second-order error. It would be interesting to investigate the notion of higher-order errors, study their theoretical properties, and utilize them for intrinsic motivation in our futuure work.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Joshua Achiam and Shankar Sastry. Surprise-hased intrinsic motivation for deep reinforce- ment. learning. Xxx preprint orXi:: 01732, 2017.
Adrià Puigdomenecch Badia, Pablo Sprechmann, Alex Vilvitskyi, Daniel Guo, Bilal Piol, Steven K Kaplurowski, Olivier Tieleman, Martin Arjovsky, Alexander Pritzel, Andrew Bolt, et al. Never give up: Learning directed exploration strategies. In International Conference Learning Representations, 2019.
Adrià Puigdomênech Badia, Bilal Piot, Steven Kaplurowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell. Agent57: Outperforming the atari human benchmark. In International Conference on Machine Learning, pages 507- 517. PMLR, 2020.
Andrew Barto, Marco Mirolli, and Gianluca Baldassarre. Novelty or surprise? Frondiers in psychology, page 907, 2013.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxlon, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Adrances in neural information processing systems, 29, 2016.
Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinehart, Chelsea Finn, Dinesh Jayaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement. learning in unstable environments, In International Conference Learning Representations, 2020.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale sludy of curiosity-driven learning, In International Conference on Learning Representations, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference on Learning Representations, 2018b.
Maxime Chevaleer-Boisvert. gym-miniworld environment for openai gym. https://githhb.
coon//maxiieeebyyyyymmmiiivooooo 2018.
Maxime Chevalier-Biisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld envi- ronment for openai gym. htttp:://gtthn hwom///ximmcb///ooooooooooo gym-minigrid 2018.
Paul Ed Ekman and Richard Davidson. The nature of emotion: Fundamental questions.
Oxford University Press, 1994.
Aleksandr Ermolov and Nicu Sebe. Latent world models for intrinsically motivated explo- ration. Advances in Neural Information Processing Systems, 33:5565-5575, 2020.
Zheng Fang, Biao Zhao, and Guizhong Liu. Image augmentation based momentum memory intrinsic reward for sparse reward visual scenes. arXiv preprint arXiv:2205.09448. 2022.
Alex Graves, Greg Wayne, and Ivo Danihelka. Neural turing machines. arv preprint 0rXiv:1410.54000 2014.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
Laurent Itri and Pierre Baldi. Bayesian surprise artracts human attention. Advances in neural information processing systems, 18, 2005.
Hyoungseok Kim, Jackyeom Kim, Yeonwoo Jeong, Sergey Levine, and Hyun Oh Song. Emi: Exploration with mutual information. In International Conference on Machine Learning, pages 3360-3369. PMI.R, 2019.
Hung Le, Truyen Tran, and Svetha Venkatesh. Learning to remember more with less mem- orization. or Xae preprint or 1mx::1110.00000 1901190013337 2019.
10

---

Under review as a conference paper at ICLR 2023
Manuel Lopes, Tobias Lang, Marc Toussaint, and Pierre-Yve Oudeyer. Exploration in model-based reinforcement learning by empirically estimating learning progress. Advances in neural information processing systems, 25, 2012.
Mikkel Sannes Nylend. Data efficient deep reinforcement learning through model-based intrinsic motivation. Master's thesis, NTNU, 2017.
Deepak Pathak, Pulkit. Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven ex- ploration by self-supervised prediction. In International conference on machine learning, pages 2778 2787. PMLR, 2017.
Adityanarayanan Radhakrishnan, Mikhail Belkin, and Caroline Uhler. Overparameterizee neural networks implement associative memory. Proceedings of the National Academy of Sciences, 117(44):7102-22777, 2020.
Nicholas Rhinehart, Jenny Wang, Glen Berseth, John Co-Reyes, Danijar Hafner, Chelsea Fimn, and Sergey Levine. Intrinsic control of variational beliefs in dynamic partially- observed visual environments. In ICML 2021 Workshop on Unsupervised Reinforcement Learning, 2021. URL https://openreviier net. hett//orruI1d-gbb00ccll
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphael Marinier, Mare Pollefeys, Tim- othy Lillicrap, and Sylvain Gelly, Episodic curiosity through reachability, In International Conference Learning Representations, 2018.
Jürgen Schmidhuber. Curious model-bnilding control systems. In Proc. international joint. conference on neural networks, pages
1991.
Jürgen Schmidhuber. Formal theory of creativity, fun, and intrinsic motivation (1990-2010).
TEEE trausactions on automomous merdal development, 2(3):230-22 247, 2010.
Bradly Stadie, Sergey Levine, and Pieter Abbeel. Incentivizing exploration reinforce- ment learning with deep predictive models. arXiv e-prints, pages arXiv-1507, 2015.
Susanne Still and Doina Precup. An informatiion-hheortiic approach to curiosity-driven reinforcement learning. Theory in Biosciences, 131(3):139 148, 2012.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: study of count-based exploration for deep reinforcement learning. Aduances in neural information processing systems, 30, 2017.
11

---

Under review as a conference paper at ICLR 2023
APPENDIX
A W AS ASSOCIATIVE MEMORY
This section will connect the associative memory concept neural networks trained with the reconstruction loss as in Eq. 5 We will show how the neural network (W) stores and retrieves its data. We will use I-layer feed-forward neural network W to simplify the analysis, but the idea can extend to multi-layer feed-forward neural networks. For simplicity, assuming W square matrix, the objective is to minimize the difference between the input and the output of W:
For simplicity, assuming W is a square matrix, the objective is to minimize the difference between the input and the output of W;
L = 2
(6)
Using gradient descent, we update W as follow,
W W Oc
OW
+ W 2a (Wx - x))
- W - 2a. + 20xF.'
W (I 2xxr 1) + 20.x..
where the identity matrix, r is the column vector. a batch of inputs { is used in computing the loss in Eq. 6 at step t, we update W as follows,
W, Wi-1 (I axi) aX; From 0, after T updates, the weight becomes
where X,
zıo 22
T
WT = Wo II (X) a² Σ 1m1
X.X. t=
II (I- axx) + Σ X
(7)
k=t+1
Given the form of Xt: X, is symmetric positive-defnitt. Also, as a is often very small (0<a & 1), we can show that ax < Amin (aX,) < 1. This means as T oc I'o (I- (I-)x,) + and thus, WT - a² Σ" X,X, IT ((-axx)) + ET X₁ independent from the initialization Wo. Eq. shows how the data (Xi) integrated into the neural network weight. W. The other components such as a² "' X,X,-1 H I1 1 (I xxx)) can be viewed as additional encoding noise. Without. these components (by assuming a small enough),
T
WT R
X
t=1
or equivalently, we have the Hebbian update rule:
+ W + + 8 Z4,t
where W can be seen the memory, & is the outer product and T5 is the data item stored in the memory. This memory update is the same as that of classical associative memory models such Hopfield network and Correlation Matrix Memory (CMM).
12

---

Under review as a conference paper at ICLR 2023
Given query 9, retrieve the value in W as output of the neural network:
d 7
T
=9 R + a
q.tt
t=1
B
q R + 2a
[=1
where == Wo II (1 ax) a2> X,X,
X (a-axs) If is presented to
the memory W in the past some Ij, can be represented as:
=
R+ 20
Σ
q xi : 200 (qq
t=1 (=1,11
T
R 20 Σ
Σ
q Fust, +2x llall
1=1
noise
cross talk
Assuming that the noise is insignificant thanks to small a, we can retrieve exactly g given that all items in the memory orthogona P
As result, after scaling o' with 1/2a, the retrieval error ( a/| is O. If q is new to W, the error will depend on whether the items stored in W are close to Usually, the higher the error, the more novel q is w.r.t W.
B SM'S IMPLEMENTATION DETAIL
In practice, the short-term memory M is a tensor of shape [B, where B is the number of actors, N the memory length and d the slot size. B is the SG's hyperparameters and tuned depending on tasks based on SG performance. For example, for the Noisy-TV TV,, we tune RND the SG, obtaining B 64 and directly using them for M.. N and are the special hyperparameters d 16 in all experiments. As B increases in large-scale experiments, memory storage for M can be demanding. To overcome this issue, we can use the uniform writing trick to optimally preserve information while reducing N (Le al. 2019).
our method. As mentioned in Sec. 3.4 we fix N 128 and
Also, for W, by using a small hidden size, we can reduce the requirement. for physical implement. W as 2-layer feed-
memory significantly. Practically, in all experiments, forward neural network with a hidden size of 32 (2n - 32 + 2n). The activation is tanh. With 512 d 16, the number of parameters of W is only about 65K. Also, E gaxd and V E _dx= have about 8K parameters. In total, our SM only introduces less than 90K trainable parameters, which are marginal to that of the SG and policy /value networks (up to 10 million parameters).
The join training of SG+SM is presented in Algo. 2 We note that vector notations in the hm are row vectors. For simplicity, the algorithm assumes actor. In practice, our algorithm works with multiple actors and mini-batch training.
INTRINSIC REWARD NORMALIZATION
Following (Burda al. 2018b to make the intrinsic reward on consistent scale, we nor- malized the intrinsic reward by dividing it by a running estimate of the standard deviations
"y certain mannsformatio,, this condition he reduced to linear independence.
13

---

Under review as a conference paper at ICLR 2023
A Algorithm 1 Intrinsic rewards computing via SG--SM framework.
Require: Wz, and our surprise memory SM consisting of a slot-hased memory M, parame- ters Q, V, and a neural network W
1: Compute Lsc = ||u,|l
2: Query M with ue, retrieve ut = WMMV where wa is the attention weight
3: Compute LM llug un u,.detach())
4: Query W with dt [ug, wrl, relrieve = W(+:)
5: Compute intrinsic reward ri = Lw = 11 ge.eelac
6: return LSG, LM, Lw
Algorithm 2 Jointly training SG and the policy.
Require: busfer, policy æp, surprise-hased predictor SG, and our surprise memory SM
consisting of slot-hased memory M, parameters Q, V, and neural network W
1: Initialize Tæ, SG, Q, W
2: for iteration 2.. do
3: for t 1, 1, do
4: Execute policy me to collect St, at, rt, forming input I, St,.' and target O
5: Compute surprise u, = SG (I) Oe.detach() (Eq.
6:
Compute intrinsic reward rł using Algo.
7:
Compute final reward < Tt + Bri/rttd
8: Add (4, St-1, ae, r() buffer
9: Add u,Q to M
10: if done episode then clear M
11:
end for
12:
for 2.,, do
13:
Sample I,, O2 from buffer
14:
Compute surprise i, = SG (I) O,.etacc() (Eq.
15:
Compute Lsc, LMs Lw using Algo. I
16:
Update SG, Q and W by minimizing the loss L. = Lsa + LM + Lw
17:
Update mo with sample (st-1, St, 4z Tz) from buffer using backbone algorithms
18:
end for
end for
of the intrinsic relurns. This normalized intrinsic reward (NIR) will used for training, In
addition, there is hyperparameter named intrinsie reward coefficient to scale the intrinsic contribution relativly to the external reward. We denote the running mean's standard deviations and intrinsic reward coefficient. rętd and B3 respectively, in Algo. 2 In our experiments, otherwise stated, 3 1.
We note that when comparing the intrinsic reward at different states in the same episode (as in the experiment section), we normalize intrinsic rewards by subtracting the mean, followed by a division by the st staddrrd deviation of all intrinsic rewards in the episode. Hence, the mean-normalized intrinsic reward (MNIR) in these experiments is different. from the one used in training and can be negative. We argue that normalizing with mean and std. of the episode's intrinsic rewards is necessary to make the comparison reasonable. For example, in an episode, method assigns all steps with intrinsic rewards of 200; and method B assigns novel steps with intrinsic rewards of I while others O. Clearly, method A treats all steps in the episode equal, and thus, it is equivalent to giving no motivation for all the steps in the episode (the learned policy will not motivate the agent to visit novel states). On the contrary, method B triggers motivation for novel steps in the episodes (the learned policy will encourage visits to novel states). Without normalizing by mean subtraction, it tempting to conclude that the relative intrinsic reward of method A for a novel step is higher, which is technically incorrect.
14

---

Under review as a conference paper at ICLR 2023
22
4
(a)
(b)
(c)
Figure 5: Key-Door: (a) Example map in Key-Door where the light window is the agent's view window (state). MNIR produced for each cell in manually created trajectory for RND+SM (b) and RND (c). The green arrows denote the agent's direction at each location. The brighter the cell, the higher MNIR. assigned to the corresponding state.
D EXPERIMENTAL DETAILS
D.1 NOISY-TV
We create the Noisy-TV environment by modifying the Maze environment (MazeS3Fast-vol in the MiniWorld library (Apache License) (Chevalier- -hrvvsller-Boisvertt 2018). The backbone RL algorithm is PPO. We adopt a public code repository for the implementation of PPO and In this environment, the state is image of the agent's viewport. The details of architecture and hyperparaneters of the backbone and RND is presented in Table Most. of the setting is the same as in the repository. We only tune the number of actors (32, 128, 1024), mini-batch size (4, 16, 64) and e-clip (0.1, 0.2, 0.3) to suit our hardware and the task. After tuning with RND,
RND (MIT License [
the same setting for our RND SMM
Fig. 6 reports all results for this enviromment. Fig. (a) compares the final intrinsic reward (IR) generated by RND and RND++SM ove training time. Overall, RND's IR. is always higher than RND+SM's, indicating that our method is significantly reduces the attention of the agent. to the noisy TV by assigning less TR to watching TV. Fig. (b) compares the number of noisy actions between two methods where RND+SM consistently shows fewer watching TV actions. That confirms RND+SM agent is less ricted by the TV.
As mentioned in the main text, RND+ + SM is better at handling noise than RND. Note that RND aims to predict. the transformed st alaes by minimizing ||SG (s.) (a(x)) where I' fixed neural neiwork init ialized randomly. If RND can learns the transformai ion, it can pass- through the state, which is similar to reconstruction fR can be harder and require more samples than learning identity transformation since fR non-linear and complicated. Hence, may be more challenging for RND to pass-throngh the noise than SM.
an autoencoder. However, learning
Another possible reason lies in the operating space (state vs. surprise). we treat white noise random variable X, surprise generator (SG) can most learn predict the mean of his variable and compute the surprise U = [X]] X where Y is random factor that afTecis the training of the surprise generator. The factor Y makes the SG produce imperfect reconstruction E [X]Y[]] Here, SG and SM learn to reconstruct X and U, respectively. We can prove that the variance of each feature dimension in is smaller than that of X (see Sec. Ep. Learning an autoencoder on surprise space is more beneficial than in state space since the data has less variance and thus, it may require less data points to learn the data distribution.
Fig. (c) reports performance of all baselines. Besides RND and RND+SM we also include PPO without. intrinsic reward as the vanilla. Baseline for reference. In addition, we investigate a simple implementation of SM using count- based method to measure surprise novelty. Concretely, use SimFlash algorithm to count the number of surprise c(u) in similar manner as (Bellemare et al. 2016) and name the baseline RND M (count). The
https: h/tg:://g.th..... cato..ccooeeeeeeeee.....e..
ne wornk-dsstllattee pyttrch
In this case, the perfect reconstruction is R[X]
15

---

Under review as a conference paper at ICLR 2023
(a)
(b)
(c)
1.0
RND+
Watching Actions
Intiasic Reward
Avg. Return
4000
0.5
2000
RMD+SM
RXDI-SM tosunt)
0.0
1M
4M
4M
2M
4M
Step
Step
Step
Figure 6: Noisy-TV's learning curves over training steps: (a) Average NIR. (b) Number of watching actions. (c) Average return. All curves are averaged over 5 runs (mean
intrinsic reward is then /cwu) We tune the hyperparameter 3 = {0.5, 1,5) and the hash matrix size hn (32, 64, 128, 256) and use the same normalization and training process to run this baseline. We report the learning curves of the best. variant. with / = 0.5 and ky = 128. The result demonstrates that the proposed SM using nemory-anggeent networks outperforms the count-based SM by a significant margin. One possible reason is that count-based method cannot handle white noise: it always returns high intrinsic rewards. In contrast, our SM can somehow reconstruct white noise via pass-through mechanism and thus reduces the impact of fake surprise on learning. Also, the proposed SM more flexible Ihan the count-based counterpart. since it learns to reconstruct from the data ralher than using a fix counting scheme. The result. also shows that RND+-MM outperforms the vanilla Baseline. Although the improvement is moderate (0.9 vs 0.85), the result is remarkable since the Noisy-TV designed to fool intrinsic motivation methods and among all, only RND+SM can outperfor the vanilla Baseline.
neural
D.2 MINIGRID
The tasks in this experiment are from the MiniGrid library (Apache License) (Chevalier- Boisvert al. 2018). In MiniGrid environments, the state is description vector repre- senting partial observation information such as the location of the agents, objects, moving directions, etc. The three lasks use hardest. maps:
DoorKey: MiniGrid-Doorker MiniGrii-DDorrXyy-166666--0
LavaCrossing: MiniGrid- MhniChiicLLawaerrossnggggiL..
DynamicObstackes: MiniChiidDDyaaaiii-Ossaalee----
The SGs used in this experiment are RND (Burda et al. 2018b), ICM Pathak et al. 2017) NGU (Badia et al. 2019) and AE. Below we describe the input-output structure of these SGs.
RND: 1, = & and O, fn (8x) where 8, is the current. state and In is a neural network that has similar structure as the prediction network, yet. its parameters are initialized randomly and fixed during training.
1CM: 4 = (s:-1, and O: = st where s is the embedding of the state and a the action. We note that in addition to the surprise loss (Eq. 2), ICM is trained with inverse dynamics loss.
NGU: This agent reuses the RND as the SG (4, = 5, and O, = fR (s:)) and combines the surprise norm with an KNN episodic reward. When applying our SM to NGU, we only take the surprise-base based reward as input to the SM. The code for NGU is based this public repository https: //gi thub. wawp.//////ooonnaaalloa-aaaa.eee Dingaa/
and O, 3, where is the embedding of the state. This SG can be viewed as an associative memory of the observation, aiming to remember the states. This baseline is designed to verify the importance of surprise modeling. Despite sharing a similar architecture, it differs from our SM, which operates on surprise and have augmented episodic memory to support reconstruction.
AE: I,
16

---

Under review as a conference paper at ICLR 2023
Hyperparameters PPO's state
ICM 3-layer weedforward net (Tanh, h h- 256) 4- 4-layer feedforward net (ReLU, h h-512)
AE 3-layer feedforward net (Tanh, h h- 256) 3 l-layer feedforward net (Tanh, h=
encoder
SG's surprise predictor
Intrisic Coef.
1
1
Num. Actor B Miníhatch size
64
64
Horizon T
128
128
Adam Optimizer's s
10-4
10-4
Discount
0.999
0.999
Iutrinsic 7
0.99
0.99
GAE A
0.95
0.95
PPO's clip e
0.2
0.2
Table 3: Hyperparameterss of TCM and AE (PPO backbone).
AE
ICM
NGU+ SM
AE+ SM
ICM+ SM
RND
Baseline
NGU
RND SM
Key Door
Dynamic Obstacles
Lava Crossing
1.00
0.8
Relurn
0.75
0.5
0.50
0.0
0.4
0.25
-0.5
0.2
0.00
-1.0
0.0
5M
10M
0
5M Step
10M
0
5M
10M
Step
Step
Figure 7: Minigrid's learning curves over 10 million training steps (mean over 5 runs).
The backhone RT. algorithm is PPO. The code for PPO and RND is the same as in Sec. D.1 We adopt public code repository for the implementation of 1CM (MIT License L We implement AE ourselves using a 3-layer feed-forward neural network. For the SGs, we only tune the number of actors (32, 128, 1024), bitc- size (4, 16, 64) and e-clip (0.1, 0.2, 0.3) for the DoorKey task. We also tune the architecture of the AE (number of layers: 1,2 or 3, activation tanh or ReLU) on the DoorKey task. After tuning the SGs, we the same setting for our SG+SM. The detailed configurations of the SGs for this experiment are reported in Table 3 and Table 4
The full learning curves of the backbone (Baseline), SG and SG+ SM are given in Fig. To visualize the difference between surprise and surprise residual vectors, we map these in the synthetic trajectory to 2-dimensional space using t-SNE projection in Fig. The surprise points show clustered patterns for high-MNIR states, which confirms our hypot that there exist familiar surprises (they are highly surprising due to high norm, yet repeated). In contrast, the surprise residual estimated by the SM has no high-MNIR clusters. The SM transforms clustered surprises lo scalter surprise residuals, result ing in broader range of MNIR, t hus showing significant. discriminatinn on states that have similar surprise norm.
D.3 ATARI
The Atari 2600 Games task involves training an agent to achieve high game scores. The state is 2d image representing the screen of the game.
17

---

Under review as a conference paper at ICLR 2023
Surprise
Surprise Residual
0
0
-10
-5
0.8
0.8 2.4
20
0.0
.3.0
-10
0.0
1.6
1.5
+4.5
10 -5
5
0
20
40
Figure & Key-Door: I-NSE 2d representations of surprise (u,) and surprise residual (4r 18--gh (b)) Each point. corresponds to the MNIR some step in the episode. Color denoles the MNIR value (darker means higher MNIR). The red circle on the left picture shows an example cluster of surprise points. Surprise residuals of these points are not clustered, as shown in 6 red circles on the right pictures. In other words, surprise residual can discriminate surprises with similar norms.
Hyperparamoters
MiniGrid
Noisy MiniWolld
Atari
PPO's state encoder
3-layer feedforwan ard net (Tanh, b=256)
3-layer Leaky-ReLU CNN with
3-layer Leaky-ReLU CNN with
kernels
kernels
112/2//8/4, 64/84///////////1/
(4 4//32/6/4432////////// 64/64/3/1
+2-layer feedforward
+2-layer feedforward
(ReLU, h=256)
(ReLU, h=256)
RND's surprise predictor
layye foodforward net (Tanh, h h-512)
3-layer Loaky BeL. CNN with kernels
A-layer Teaky CNN
kernel
(1/32/8/4, 32/61/4/2222//// 32/11//333/2////////////3333 61/61/3/13
(L/32/8/4.32//6//11 61/64/3/13
2-layer feedforward net
2 2-2layee feedforward net
(ReLU, h=512)
(ReLU, b=
Intrinsic Coef. 8
1
1
Num. Actor
Minihatch size
64
64
Horizon
128
128
128
Adam Optimizer's Ir
10-4
10-1
10 10-4
Discount ?
0.999
0.999
0.999
Intrinsic 7
0.99
0.99
0.99
GAE
0.95
0.95
0.95
PPO's clip
0.2
0.1
0.1
Table Hyperparameters of RND (PPO backbone).
18

---

Under review as a conference paper at ICLR 2023
Feeway
Frostbite
Venture
30
10000
LWM LWM-EM
150M LM
Avg. Return
7500
BND++M
1000
RND-SM
20
5000
500
10
2500
RN SM
0
20M
40M
0
20M
40M
0
20M
40M
Step
Step
Step
Gravitar
Solaris
Montezuma
LMM
6000
Avg. Return
1500
LWM- SM
RND
3000
LMM
RND-SM
BND+55 SM
1000
2000
500
2000
1000
20M
40M
0
20M
40M
0
20M
40M
Step
Step
Step
Figure 9: Atari low-sample regime: learning curves over 50 million frames (mean over 5 runs). To aid visnalization, we smooth the curves by taking average over a window sized 50.
SG and RL backbone implementations We use 2 SGs: RND and LWM. RND uses PPO backbone as in previous sections. On the other hand, LWM uses DQN backbone with CNN-based encoder and GRU-based value function. The LWM SG uses GRU to model forward dynamics of the environment and thus its input is: I, = (8t-1, 0t, h+-1) where 8--1 is the embedding of the previons state, Oş the current action, and h₂-1 the hidden state of the world model GRU. The target O, the embedding of the current state St.
RND follows the same implementation as in previous experiments. We use the public code of LWM provided by the hors to implemeent LWM. The hyperparameters
RND and LWM are tuned by the repository's owner (see Table 4 for RND and refer to the code or the original paper (Ermolov and Sebe 2020) for the details of LWM implementation). We augment. them with our SM of defanlt hyperparameters N 128, d = 16.
Training and evaluation We follow the standard training for Atari games, such as stack- ing four frames and enabling sticky actions. All the environments are based on OpenAl's gym-atariis NoFrarneski-vv variants (MIT Liscence
Afterr training, we evaluate the models by measuring the average return over 128 episodes and report the results in Table. Depending on the setting, the models are trained for 50 or 200 million frames.
2
Results
Fig. M demonstrates the learning curves of all models in Atari games under the low-sample regime. LWM+SM and RND+SMM clearly outperfrom LWM and RND in Frost- bile, Venture, Gravitar, Solaris and Frostbitee, Venture, Gravitar and MontezunaBevnng. respectively. Table reports the results of more baselines.
D.4 ABLATION STUDY
Role of Memories We conduct. more ablation studies to verify the need for the short M and long-term (W) memory in our SM. We design additional baselines SM (no W) and SM (no M) (see Sec. 3.4h, and compare them with the SM with full features in Montezuma Revenge and Frost task. Fig. 10 (a) shows that only SM (full) can reach an average score of more than 5000 after 50 million training frames. Other ablated baselines can only achieve around 2000 scores.
hhttpa://githbb.ccm
htttps://giiii h/tp:/gggtttt.../// hwtpimn..omm//p//// www.coom.cma..ee oppeai/gnnaggmmmmmmm
19

---

Under review as a conference paper at ICLR 2023
Task
EMI
EX2
ICM
AE S11
LWM
RND
IWM
IWM+SM
RND
RND+SM
Freeway
33.8
Frosthite
7,002
3,387
4,465
5,214
8, ,409
2,227
8,598
10,258
2,628
5,073
646
1,381
1,081
1,119
Gravitar
558
550
424
482
1,376
546
1,242
1,093
749
987
Solaris
2,688
2,276
2,453
4,467
1,268
2,051
1,839
8,065
2,206
8,4 2.
Montezuma
387
161
75
2,276
377
2.192
2.209
2,473
6,187
Norm. Mean
61.1
10.5
16.1
52.1
80.6
42.2
80.5
97.0
50.7
74.8
Norm. Median
31.9
32.3
33.3
32.7
66.5
83.7
58.3
84.6
Table Atari: test performance after 50 million training frames (mean over 5 runs). is from prior work (Ermolov and Sebe 2020). 4 is our run. The last two rows are mean and median human normalized scores. Bold denotes best results. Italic denotes that SG+SM significantly better than SG regarding Cohen effect size less than 0.5.
Montezuma Revenge
Frostbite
Venture
6000
SM (no M]
1000
Avg. Retur
4000
SM (no WI
4000
SM (no
750
SM (so W1
500
2000
2000
250
20M Step
40M
20M
40M
20M
40M
Step
Step
Figure 10: Ablation study: average returns (mean over 5 runs.
We also shows the impact of the episodic memory in decreasing the intrinsic rewards for similar states as discussed in Sec. [.33 We select states in the MiniGrid's KeyDoor task and computes the MNIR for each state, visualized in Fig, [I] AI the slep-1 stale, the MNIR low since there is nothing special in the view of the agent. the step-1 state, the agent. first sees the key, and get high MNTR. At the step-28 state, the agent drops the key and sees the key again. This event still more interesting than the step-1 state, However, the view is similar to the one in step 15, and thus, the MNIR decreases from 0.7 to 0.35 as expected.
No Task Reward The tasks in this experiment are from the MiniWorld library (Apache License) Chevalier-Boisver 2018). The two tasks are:
Easy: MiniWWold-Picknphiii...
Hard: Mini Worooo--FooRRoossseee
The backbone and SG are the same as Sec. D.1 We remove the task external reward in this experiment. For the Baseline, without task reward, it receives training signal
step 1: -0.66
step 15: 0.70
step 28: 0.33
Figure 11: MiniGrid's KeyDoor: MNIR of SM at different steps in an episode.
20

---

Under review as a conference paper at ICLR 2023
Easy
Hard
Easy
Hard
Cam. Return
400
300
150
anD
Aeg. Linttoss Reward
1.00
0.75
EXDHAM
1,00
A0D1-M
200
100
0.50
100
0.25
0.00
50M
100M
50M 100M
50M Step
100M
50M
100M
Step
Figure 12: MiniWorld: Exploration without task reward. Left: Cumulative task returns ove 100 million training steps for two setting: Easy (1 room, 3 objects) and Hard (4 rooms, object). Right: The average intrinsic reward over training time. The learning curves are taken average (sta)) over runs
and thus, showing similar behavior random agent. Fig. 12 illustrates the running average of cumulative task return and the intrinsic reward over training steps. In the Easy mode, the random Baseline even perform better than RND, which indicates that both modes, shows superior
biased intrinsic reward not always helpful. RND+ performance, confirming that its intrinsic reward is better to guide the exploration than that of RND.
THEORETICAL PROPERTY OF SURPRISE SPACE'S VARIANCE.
Let. X he random variable representing the observation at some timestep, surprise generator (SG) can at most learn to predict the mean of this variable and compute the surprise U E [X[[]] - where is a random factor that affect the prediction of SG and makes it produce imperfect reconstruction E [X instead of E [X]. For instance, in the case of an autoencoder AE as the SG, and are stand AE(st) St, respectively.
Let. denote Z EX(Y), then E. [2 7 and E [ 22. We have
var (X) -varaX X - Z + 22
var(X Z) var(2) + 2eov(X Z)
vvr(XX - 2) - v + 2E[(X Z)Z] - Z|
Using the Law of Iterated Expectations, we have
E[X- 2 = E _|
E Y E [Z
E z 2 =0
and
[ 2) E Z)
= E|EXX Z2|2]]
= EXE(XZ)) - (Z*
= -E|ZE - 2
E|22 - 2
Therefore,
var (X) var(X Z) + var(Z)
Let C, -x and C denote the diagonal entries of these covariance matrices, they the variances of the components of the random vector X, Z and Z, respectively. That is,
21

---

Under review as a conference paper at ICLR 2023
(o:²) 2 (0 20 (o?9) x (o 2 (o 20= (0!))
+ =
In our setting, X and U represents observation and surprise spaces, respectively. Therefore, the variance of each feature dimension in surprise space is smaller than that of observation (o²)² 0 or E (X(Y) = (X). That is, the SG's prediction is perfect, which is unlikely happen practice.
space. The equality is obtained when
LIMITATIONS
Our method assurnes that. surprises have patterns and can be remembeere by our surprise memory. There might. exist environments beyond those studied in this paper where this assumption may not hold, or surprise-based counterparts already achieve optimal explo- ration (e.g., perfect SG) and thus do not need SM for improvement (e.g., Freeway game), In addition, M and W require additional physical memory TRAM/GPU than SG meth- ods. Finally, a plug-in module like SM introduces more hyperparameters, such as N and d. Although find the default values of N-1 and d-16 work well across all experiments in this paper, we recommend adjuslments users apply our method to novel domains.
22


Reviews:
[{"review_id": "pMuBxt4lIle", "reviewer": "Reviewer_ZXKP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This method proposes some modifications to curiosity-based exploration methods to help combat some of the corner cases that they don't deal well with, which also results in increased performance. The motivation for this work is that exploration is still a really large challenge and reinforcement learning and even some of the best-proposed methods have some challenges in being able to explore properly in all types of environments (MDPs). The method proposed in this paper introduces a new metric related to surprise novelty, and the surprise novelty helps calculate and reduce the noisy TV problem for curiosity-based metrics. The paper does introduce some interesting ideas, and there's some evaluation at the end of the paper to show that this surprise novelty metric is less susceptible to some of the issues with normal curiosity base methods.", "summary_of_the_review": "The proposed method for performing better curiosity-based exploration is promising. However, the writing and analysis make it difficult to understand how the method works and if it is novel. These issues will need to be addressed for me to raise my score.", "strength_and_weaknesses": "pros\n- The method in the paper does make some analysis in order to be able to correct and adjust some of the challenges with applying curiosity-based exploration bonuses in different types of environments.\n- An adequate amount of experimental analysis at the end of the paper indicates the potential for such a method to support a contribution over prior curiosity base metrics.\n\ncons\n- Generally, some of the conceptual parts and description of the method are challenging to follow. In particular, some of the description around surprise and surprise novelty and what is the mathematical difference between these concepts is not thoroughly explained. This makes it challenging to understand the paper's novelty and reuse the findings of the paper and future work by the community\n. There is additional related work that this paper should cite and compare. In particular, if the paper is citing the noisy TV problem and the general aspect that environments can have stochastic elements in them, then the paper should consider comparing to additional methods that use surprise minimization[1,2].\n\n- [1] Berseth, G., Geng, D., Devin, C. M., Rhinehart, N., Finn, C., Jayaraman, D., & Levine, S. (2021). SMiRL: Surprise Minimizing Reinforcement Learning in Unstable Environments. International Conference on Learning Representations\n- [2] Rhinehart, N., Wang, J., Berseth, G., Co-Reyes, J. D., Hafner, D., Finn, C., & Levine, S. (2021). Intrinsic Control of Variational Beliefs in Dynamic Partially-Observed Visual Environments. Advances in Neural Information Processing Systems, 34.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Sh4hg_eMZh", "reviewer": "Reviewer_Qsrw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors propose a new intrinsic reward function for reinforcement learning, building on earlier definitions of \"surprise\". The proposed idea is to reward not the surprise itself but its novelty. The authors propose a specific approach to computing this intrinsic reward function and present an empirical analysis in a number of domains. ", "summary_of_the_review": "The paper presents a potentially useful concept for intrinsic reward, as well as a particular implementation of it. But I struggled to reach a non-superficial understanding of the behaviour of the proposed method and to evaluate its general applicability and usefulness. ", "strength_and_weaknesses": "Strengths: The paper is in an important area of research relevant to the conference. The proposed concept for intrinsic reward is intriguing and can be useful. There is extensive empirical analysis, showing improved performance compared to existing approaches. \n\nWeakness: Experimental analysis has some limitations (see below) with the result that the general applicability and benefits of the approach are difficult to evaluate. The writing is relatively poor. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5Ut7Gy0kPh", "reviewer": "Reviewer_oPBQ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces a new method for intrinsic motivation in RL called Surprise Memory (SM), that in similar fashion to surprise-based intrinsic motivation, provides an intrinsic reward to the agent when it finds a 'new' observation. SM differs in that it takes in account if this 'surprise' was already expected from previous memories and reduce the intrinsic reward if that is the case.\n\nIntuitively, the method consists of a memory module (of fixed size) with previous surprises and an autoencoder (AE) that tries to reconstruct a query with the current surprise and the closest matching surprise within memory. The worse the AE does, the larger the intrinsic reward. ", "summary_of_the_review": "Good paper, novel method, targets one of the main issues of intrinsic motivation with promising results. The work seems solid and sound although could be improved by detailing weaknesses and limitations and with further motivation of the AE mechanism. ", "strength_and_weaknesses": "I believe that this is a good work, the method is novel and non-trivial, authors provide abundant theoretical and experimental ground for their approach. The paper is also well presented and easy to follow. \n\nWeaknesses:\n1) The authors only present the strengths of this method, which seem to be many, but I could not see any discussion on limitations or main weaknesses of this approach. The future lines proposed at the end feel a little hand waving in my opinion, it doesn't provide much ground for future researchers to continue this work.\n\n2) The need of the AE, while it is clearly demonstrated in the ablation experiments, it is lacking an explanation of why in the theoretical side -besides that it works better-. Going through the work I could grasp some intuition, but would be good that the authors detail this further in section 2 since intuitively, one could think that the norm between the surprise of this state and the one in memory would already tell you if this is an \"expected\" surprise.\n\nExtra:\nI don't want to point this as a weakness but I would like to ask the authors about how does the SM intrinsic reward evolves through training when you are training for the same objective for long. What I mean is that, if you are training your agent to get a red box at the end of the episode, since the box is exotic it provides high intrinsic reward. However, as you train and train the agent the AE will get better to recover that red box surprise at the end, contrarily, the AE probably cannot learn so well to reconstruct the tv noise. My question is, does the intrinsic reward for that final goal get relatively lower to the one of the tv as training progresses? And if not, could Authors explain why it doesn't happen?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "47DeO23xyXd", "reviewer": "Reviewer_422A", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method to manage surprise signals in reinforcement learning. Surprises are stored in a memory for each episode and the memory is wiped after each episode. An autoencoder is used to perform readouts. The memory module can be plugged in existing surprise generators. Benchmark results show that the addition of the memory is almost always beneficial. The main difference with respect to competing memory based approach in RL is that the proposed memory works at the surprise level and not at the state level.\n\n", "summary_of_the_review": "The approach proposed plugs a memory module into RL system using surprise generation. The use of such module is shown to empirically improve results. Novelty is slightly limited and some comparisons are missing. ", "strength_and_weaknesses": "Strengths\n- The use of memory on the surprise level is novel and well motivated\n- The approach can be integrated easily on existing methods\n\nWeaknesses\n\n- Novelty is slightly limited, memory has been used here https://arxiv.org/pdf/2002.06038.pdf as also mentioned in the paper but with a different data to be stored (agent state vs surprise)\n- Results show the improvement with respect to three methods but there are no direct comparison with existing sota. Specifically whye the only other memory based approach (https://arxiv.org/pdf/2002.06038.pdf) is not taken into account in the comparison?\nIn general a table with recent sota methods reported would improve the overall presentation.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject