Novelty Detection in Reinforcement Learning with World Models
Authors: Geigh Zollicoffer
Kenneth Eaton
Jonathan C Balloch
Julia Kim
Wei Zhou
Robert Wright
Mark Riedl
Venue: ICML 2025 spotlightposter (2025)
Keywords: Anomaly Detection
Safety Mechanisms
Primary area: reinforcement_learning->online
TL;DR: Designing straightforward techniques to improve deployment safeguards for RL systems

Abstract: Reinforcement learning (RL) using world models has found significant recent successes.
However, when a sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline.
We refer to the sudden change in visual properties or state transitions as novelties.
Implementing novelty detection within generated world model frameworks is a crucial
task for protecting the agent when deployed. In this paper, we propose straightforward bounding approaches to incorporate novelty detection into world model RL agents by utilizing the misalignment of the world model's hallucinated states and the true observed states as a novelty score.  
We provide
effective approaches to detecting novelties in a distribution of transitions learned by an agent in
a world model. Finally, we show the advantage of
our work in a novel environment compared to traditional machine learning novelty detection methods as well as currently accepted RL-focused novelty detection algorithms.

Novelty Detection in Reinforcement Learning with World Models
Geigh Zollicoffer!
Kenneth Eaton Jonathan Balloch
Julia Kim
Wei Zhou?
3 Robert Wright? Mark Riedl 3
Abstract
can undergo distributional shifts in the underlying transition dynamics. In this paper, address novelty detection, form of anomaly detection, which is relatively unexplored in reinforcement learning (Müller et al., 2022; Nasvytis et al., 2024). Novelties are sudden changes to the observation space or environment state transition dynamics that occur at inference time that are unanticipated (or unanticipatable) by the agent during training (Balloch et al., 2022). Nov- elties represent permanent shift in observation space or environment state transition dynamics, as opposed to one- off sensory or transition errors; when novelty occurs it may not be encountered immediately by an agent, and be- cause transition dynamics may be uncertain, only become apparent after several actions (Balloch et al., 2022).
Reinforcement learning (RL) using world mod- els has found significant recent successes. How- ever, when sudden change to world mechanics or properties occurs then agent performance and reliability can dramatically decline. We refer to the sudden change in visual properties or state transitions as novelties. Implementing novelty de- tection within generated world model frameworks is a crucial task for protecting the agent when deployed. In this paper, we propose straightfor- ward bounding approaches to incorporate novelty detection into world model RL agents by utiliz- ing the misalignment of the world model's hal- lucinated states and the true observed states as a novelty score. We provide effective approaches to detecting novelties in distribution of transi- tions learned by an agent in world model. Fi- nally, we show the advantage of our work in Mini- Grid, Atari, and DeepMind Control environments compared traditional machine learning novelty detection methods as well as currently accepted RL-focused novelty detection algorithms.
To motivate the work, consider Table 1. The top row shows what agent sees when training, when novelty is intro- duced at inference time, and post-novelty. The bottom row shows the reconstructed image from the world model's pre- diction of the next state. Post-novelty, the world model's predicted state begins differing radically from the ground truth. In cases where there is novel change, the RI. agent's s converged policy no longer reliable, and the agent can make catastrophic mistakes. The agent may flounder inef- fectually, or, worse, the agent can mistakenly take action that put itself or others in harms way.
1. Introduction
There are several ways of addressing inference- novelty depending on the nature of the agent's 's task. One may halt execution because continuing to run the policy has become potentially dangerous, for example in the case of a robotic platform or agent interacting with people, financial systems, or other high-stakes situations. Once halted, an operator fig- ures out the best way to retrain the agent. Alternatively, one may attempt novelty adaptation where the agent attempts to update its own policy during online inference time (Zhao et al., 2019; Wilson & Cook, 2020).
Reinforcement learning (RL) using world models has found significant recent successes due its strength in sampling efficiency and ability incorporate well-studied Markov Decision Process techniques (Moerland et al., 2020; Robine et al., 2021). A world model (Ha & Schmidhuber, 2018b) is a model that predicts the world state given a current state and the execution that was executed (or will be executed).
While RL agents are often trained and evaluated in environ- ments with stationary transition functions, the real world
In either case, one must detect that novelty has occurred prior to halting or adapting. The standard approach- as exemplified by Recurrent Implicit Quantile Networks (RIQN)' (Danesh & Fern, 2021)-is to treat novelty as a distribution shift in either the observation space or the state- action transition probability. This is traditionally accom-
'Deparmennt of Mathematics, Georgia Institute of Technology, Atlanta, United States of America GGerriaa Tech Research Insti- tute, Atlanta, United States of America 'Department of Computer Science, Georgia Institute of Technology. Atlanta, United States of America. Correspondence to: Mark Riedl <riedl @cc. wowool@e....tce..eeeloo eduu, Geigh Zollicoffer <gzollicoffer3 @ gatech.edus.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
'RIQN is the haseline method used in Out-of-Distribuue Dy- namics Detection: RI-Relevant Benchmarks and Results.

---

Novelty Detection with World Models
Humanoid
Freeway
Misigrid
Ground Truth
Posterior
Training
Novelty
Post-Novelty
Training
Novelty
Post-Novelly
Training
Novelty
Post-Novelyy
Table 1: Examples of model collapse when encountering novelty in three environments: DeepMind Control Suite Humanoid (noise introduced), Atari Freeway (cars become invisible), and MiniGrid (lava introduced). The Ground Truth row shows actual screens during training, when a novelty is introduced, and post-novelty. The posterior row shows the world model's reconstructed prediction, which deviates from the ground truth.
plished by setting a threshold hyperparameter to determine when the distribution shift is significant enough to be sidered novelty. Setting such a parameter requires at least implicit foreknowledge of anomalies that can occur, which violates our assumptions that novelties be unanticipatable and unencountered during training.
niques in reinforcement learning, we compare our method to the state-of-the-art RL novelty detection technique, RIQN (Danesh & Fern, 2021). We ablate technique to more closely resemble standard assumptions about detec- tion to show the possible dangers of finetuning a threshold fit for a novelty detector. While our primary result built on top of the Dreamer world model (Hafner al., 2021), we also show that our novelty bound technique can be applied to other types of world models with similar success.
We introduce a novel technique for novelty detection with- out the need to manually specify thresholds, and without the need for additional augmented data. Our approach builds the capabilities of world models to predict the state of the world given the current state and an action. When novelty occurs, our insight is that the predicted state and actual state after an action will deviate. Our technique calculates a nov- elty threshold bound without additional parre-parameters by considering how much the actual world observation de- viates from the distribution of world observations that the agent predicts it will encounter.
2. Related Work
Novelty detection has taken on different names depending on the context (Pimentel et al., 2014). There are important applications that can benefit from RL. novelty detection frameworks (Fu al., 2017), yet novelty detection has been well-studied in the realm of RI. (Nasvytis al., 2024).
Many complications arise from applying traditional ma- chine learning novelty detection methods to an online set- ting (Müller et al., 2022). Generalization techniques exist, such as procedural generation to train agents to be more robust novel situations (Cobbe al., 2019) and data augmentation (Lee et al., 2019) to better train the agent, however these techniques suffer from high sample complex- ity (Müller et al., 2022). Furthermore, there may not be sufficient representation of the agent's 's evaluation environ- ment to be able to detect novel transitions. Recent research shows that learning an OOD model is a difficult problem to solve and may even prove to be impossible in certain situations (Zhang et al., 2021; Fang et al., 2022). In addition, novelty detection techniques in the RL domain need to consider the current context of the situation. Not considering context becomes problematic when assuming that data is independently and identically distributed (i.i.d),, as it overlooks dependencies and contex- tual information that are often crucial for accurate detection. Thus it not simple apply standard detection techniques such as (Angiulli et al., 2024) in the RL domain.
Our technique draws from the notion of Bayesian sur- prise (Itti& & Baldi, 2005). Specifically, as the agent interacts with the environment, tracks the KL divergence between the predicted latent world state representation given a hidden state and embedded image relative the predicted latent world state given just the hidden state (without the embed- ded image). To develop the bound, observe that, under nominal conditions, any divergence should be smaller than that of the predicted latent world state computed with the initial hidden state, as the latter prediction becomes increas- ingly inaccurate. But, as we will show, novelty can flip this relationship. When this happens, we flag the violation of inequality of divergences.
(OOD)
We evaluate our method by injecting novelties into Mini- Grid (Chevalier-Bee -Biivvrrr et al.. 2018), Atari (Machado et al., 2018), and continuous Deep Control (DMC) (Tunyasu- vunakool et al., 2020) environments. Specifically, we use the NovGrid (Balloch et al., 2022), HackAtari (Delfosse al., 2024), and RealWorld RL Suite (Dulac-Arood al., 2020) that provide novelties to their respective base environments. Due to the dearth of established novelty detection tech-
2

---

Novelty Detection with World Models
Some methods attempt to use a measure of reward signal deterioration way to detect if novelty has occurred, but that can prove to be dangerous in high stakes envi- ronment where it may be too late to recover from nega- tive Signal(Greenbery & Mannor, 2020). This method may also prove to have an incorrect mapping of the novelty that caused the reward shift.
St (h2, 2,) is the agent's compact model state.
Given each representation of state st, DreamerV2 defines six other learned,
transition dis- tributions given by the trained world model:
Recurrent model:h, = fo (h 21-1, as-1) Representation model: prl [/s, ax)) Transition prediction model: po (&x [tt)
The RIQN (Danesh & Fern, 2021) framework has shown to be successful by focusing the aleoric uncertainty of the agent and draws upon the generated distributions of Im- plicit Quartille networks (Dabney et al., 2018). RIQN is an ensemble based method that detects novelty by predicting action values from state-action pairs as input, and predicts the feature distribution at each time step based on the pre- vious values of the features. First, given an ensemble of dynamic models and time t, RIQN computes anomaly score through predicting samples. Given these samples an anomaly score is computed through the average L1 dis- tance for each observed feature between the samples and the actual observation at time t. The anomaly scores for each feature in the observation are then individually used as values for the cusum algorithm (Page, 1954) to detect any disturbances. However, RIQN is also sensitive to the selection of threshold X, and drift A which can vary from environment to environment and must be tuned; which re- quires a priori belief of whether anomalies will result in large or small shifts in alcoric uncertainty.
DreamerV2:
Image prediction model: pa (a&a he, 2x) Reward prediction model: pa h4g zz)
Discount prediction model: pe (fellte zt)
(1)
where & describes the parameter vector for all distributions optimized. The loss function during training (Hafner et al., 2021) is:
[ = E
Σ Inpp( 21)
polz|a,x)
Impp 20) - Inpo(Tellee, 21)
+ BKL (po mollooo
(2)
where 3K L proving the prior dynamics towards the more informed pos- terior through KL Balancing (Hafner et al., 2021).
is minimized by im-
3. Background
The goal of the DreamerV2 world model is to learn the dynamics and predictors for the observation 2t, reward, and discount factor of the training environment. We consider single agent online RL setting, where an agent time & will traverse through each state 8; in which observations are represented in the form of xg, The agent relies these along with r. to construct its belief state and make decisions to achieve the optimal discounted sum of rewards.
Partially observable Markoy Decision Processes We study episodic Partially Observable Markov decision processes (POMDPs) denoted by the tuple M = (S,A, T, r, 22 7). where S is the state space, A is the ac- tion space, T is the transition distribution [s 4(-1), r the reward function, O is the observation space, Ω is an emissions model from ground truth states to observations, and y is the discounting factor (Åström, 1965).
We consider an agent that has trained in a stationary training environment and then been deployed to a non-stationn non-stationary evaluation environment that undergoes a change that is priori unknown and unanticipated during agent development and training.
DreamerV2 V2 World Model We conduct experiments ap- plying our methods to state-of-the-art DreamerV2 (Hafner et al., 2021) world model framework due to its VAE and history component, which is similar to traditional world model architectures (Ha & Schmidhuber, 201 2018a;b). Dream- V2 learns a policy by first learning a world model and then using the world model to roll out trials to train a policy model. The framework composed of an image autoen- coder and a Recurrent State-Space Model (RSSM). Relevant DreamerV2 components we will refer to throughout are:
For latent- based detection, we investigate the world model and its learned probabilities. We construct a bound that is not dependent on additional the world model's s learned logic given the effect of the ob- servation 2; at time t. Our goal is to inform the agent in response to all possible Markovian novel observations. We do not consider techniques that are reliant on reward deteri- oration as those methods can involve temporally extended action sequences to be conducted between sparse rewards. Even with dense rewards, novelty may only be detected after a meaningful amount of time of reward decrease.
papee-prrameters and test
Zt is the current image observation.
h is the encoded history of the agent. 24 is an encoding of the current image 2, that incorpo- rates the learned dynamics of the world.
3

---

Novelty Detection with World Models
We also do not consider generalization techniques that try to model all possible novelties that the agent may experience, as it is generally impossible to construct model that gener- alizes towards every unseen stimuli; to do so one must be able to categorize possible novelties in advance and generate novelties during training time (Wang et al., 2020).
(ppo(a Lalhe (y)) Po x;)) is not minimized with re- spect to the performance of Hps (2, Po(s)(()))))) the threshold (the right-hand side of the equality in (4)) will decrease. But it is also expected that the left-hand side, K [. lhe ahe ( II (zx [bx)]] reaches extremely high values, depending on the current reliance of h4-
The KL loss equation in (2) implies that, as training pro- gresses, the divergence between the latent representation of the hidden state given the latent current state repre-
Thus, there are two cases of detection arise upon seeing an observation Zt:
Proposition 3.1. If the cross entropy score comparison becomes negative when introducing the vector Us, then the right side (4) will become negative, which immediately flags 24 as novelty due to the property of non-negativity of the left side KL divergence.
sentation with and without the direct observation goes to zero for local transition space (Wang et al., 2019), i.c., [ zz) (lp [t]] S c. Given the current train- ing mechanism in place, learning the transition distribution may be difficult due to the task of avoiding regularizing the representations toward a poorly trained prior (Hafner et al., 2021). A direct consequence of using common KL balancing techniques such as Equation (2) to bias the loss towards the prior error is that the world model is initially reliant on ground truth 2e in contrast to a noisy hr- This stabilizes (2 (he, for proper training of po he) and Po ( lh: z+) (Asperti & Trentin, 2020) until the model un- derstands the role of history h4- Since the world model does not directly measure this relationship with Ft, we introduce the cross entropy score comparison:
Proposition 3.2. the cross entropy score comparison be- comes nonnegative when introducing the vector 24, then 4 defines a decision boundary in the latent space over the mea- sure of robustness to partial destruction of the input (Vincent et al., 2008; Srivastava et al., 2014). i.e.:
K (2x (h,, [ein
K KT (he, 24) (zi S K LL x (2)o(())))))
(5)
H H(po(z (ha, 21). Po ho))
(3)
where the model is tasked to simultaneously have suf- ficiently low KL divergence with the dropout of x and with the dropout of h4- Therefore, our bound is both combating posterior collapse, i.e., Po(ze (rr)) = Po (2+ ho), as well as measuring possible over-fitting i. K L(p (ze [b,, Polloooooo/oooo xx)] >> e for some large c.
- (p (1), polzello== ))
where bg is simply an empty hidden state passed to the model in order to simulate the dropout of ht and compute the influence that the ground truth x; has on the final predic- tion of the distribution of latent state žz- This gives empirical measure of the world model's current reliance and improved performance based on the ground truth observa- tion Iz. since the cross entropy score comparison increases we find the entropy to be minimized with respect to hg-
We find the above empirically to be true, as illustrated in Figure 1. As the agent trains in a stationary (no novelty) en- vironment, the surprise (orange) (Itti & Baldi, 2005) settles and steadily decreases. Initially, the difference of KL diver- gences (blue) is below the orange line, and all new stimuli are flagged as novel, but the world model quickly learns that the environment is predictable. As the agent progresses through training, the green line represents how effective inputs (ha. xx₂) are to the model in terms of predicting dis- tribution that minimizes K L|ss (zel/t, xx) ) (Pos /ho, 2z)]. Initially this value is close to the orange line because the model has learned to predict what will happen based mostly on the observation x₁. At some point the model discovers that incorporating history ht further drives loss decrease, and the KL divergence with the history dropout rises. A properly trained model is one in which its learned represen- tations, h, and 22, are useful in the prediction of distribution po(2 he, Figure illustrates that failure to do so may result in I2 being interpreted as a noisy signal (blue differ- ences collapse to orange) or the transition prediction model with h2 has not yet met sufficiently low KL divergence (or- ange exceeds blue).
As the divergence between latent predictions with and without the ground truth observable decreases, K Llpp (=)lllp(aa(tt)))) + e, we expect a reduction of the impact of zt. We model this relationship with the novelty detection bound:
K KLlpo(zelMhe (h, x;) (=))lpl S K KL (ze (he, 2:)1 (z
(4)
K [ppo hh, 2x) (2
where we substitute KL divergence in place of cross entropy loss to use the world model loss equation (2) (See Appx. D for a derivation).
The intuition behind the bound is as follows. If this relationship becomes disturbed given an unforeseen ob- servation ب that is, if the model is incapable of con- structing a mapping such that the cross entropy of
4

---

Novelty Detection with World Models
traversed and all detection methods the same policy.
Kilptaih. KLLote//h, Ailljp22//s.x xill
KLipiz/hh.x/ljoi2/hb))
60
Kulptath..xlllleeeb....
4.1. Baselines
RIQN We compare our bound against the Recurrent Implicit Quantile Network anomaly detection model (RIQN) (Danesh & Fern, 2021). a traditional and accepted RL-focused novelty detection approach, taking note of some of the practical desiderata discussed in (Müller et al., 2022) to ground our evaluation techniques. We explicitly train the RIQN model using 10 nominal transitions given by the trained policy as recommended (Danesh & Fern, 2021). We test the trained RIQN algorithm with the recommended syper-parameters, well use the recommended cusum algorithm to detect novelties (Page, 1954), and the same trajectories used by the world model. In addition, we adjust the threshold X, and drift A (value listed next to method name in tables) to improve the performance on larger obser- vation dimensions. We use an ensemble size of 5 within the framework. RIQN represents the class of ensemble-basee baselines when correcting for the RL setting.
Divergence
40
0
sazea
17500
Training Progression (Steps)
Figure I: Visualization of the average levels of divergence given 2y samples from the nominal MiniGrid Sntevalier-Boisver et al.,
2018) environment as training progresses: proposed bound (Blue); divergence of the RSSM predicted distributions given (he) and (he, 2x.) (Orange): divergence between the RSSM given (h2, 21)
and the RSSM given only the image (ho. 21) (Green line) the Nominal environment; and the divergence of the RSSM given (ht, 2re)) and receiving zero input (ha) (Red) as training progresses. For a training time-step f, given T; is said to he normal observation if the value of K [p ( xx [a (hs)) is below blue (within the green shaded area), otherwise i, is said to novel obser- vation given the current context. See Appendix C.I for similar visualizations corresponding to other environments.
PP-Mare We derive a novel ablation reconstruction error technique over the induced DreamerV2's RSSM prior and posterior reconstructions for a single realized step. This is a straightforward application of a world model wherein reconstructed states are directly compared; is simple, but effective, way detect novelties and anomalies. However, it does require a tuned hyperparameter We derive the method by first observing that locally successful training of the world model implies 3K L.pp (zx laa, he) ho) ( [hx)]] s e. To properly utilize reconstruction error for trained world models, we induce po (zx he, fa) from Eq. where ht = fe 2t-1, a--1), 2t-1 Po(22-1lhe-2 21-1), and εg (4)) to compare the pixel reconstruction losses between the generated 1-step prior and posterior I, observa- tions and bound the difference by a small À defined by the user during deployment:
4. Experiments
To empirically evaluate our bound, experiments are con- ducted in the MiniGrid (Chevalier-Boisvert et al., 2018), Atari (Machado et al., 2018), and continuous Deep Mind Control suite (DMC) (Tunyasuvunakool et al., 2020) We introduce novelties by, after certain amount of time, generating novel observations from alternative environ- ment configurations from NovGrid (Balloch et al., 2022), HackAtari (Delfosse et al., 2024) and the RealWorLdRL Suite (Dulac-Arnon et al., 2020). We detail each of novel environments in Appendix E. We use the exact hyper- parameters introduced in Hafner et al. (2021) for training DreamerV2, our base world model (c.f. our Appendix F).
Σx | L posterior
<A A
(6)
We first train an agent to leam e-optimal policy in the nominal environment then transfer the agent to one of the novel environments during testing time and let the trained agent take steps in each novel environment, capturing 300 independent and identically distributed episodes. We track each time step where novelty is detected to imitate possi- ble agent halting situations. The agent's 's trained policy is expected to be sub-optimal in the novel environments but it is not trairre at any point adapt to the novel environ- ments' dynamics. A novelty is experienced in every episode
(oe &t), and Exposterior po (f 2,). This removes direct dependence on the replay buffer by using only the final performance of the encoder and the current state (2,, he). As discussed further in Ap- pendix A (see Figure 3 in the Appendix in particular), the prior and posterior observations can have dramatic changes in representation when the predicted latent representation 2 is revealed.
where Eeprior
We tune À (value listed next to method name in tables) to improve the performance of our ParM baseline during all experiments. PP-Mare represents the class of observation- based baselines when correcting for the RL. setting.
p DMC experiments are conducted in pixel space, whereas most common usage is via position, velocity and orientation vectors.
5

---

Novelty Detection with World Models
Atari
Boxing
Kangaroo 10-2
Freeway < 10
SeaQuest
A
AUCT
ADE
AUCT
ADE
AUC
KL Bound
< 10
10--
PP. Mare
04
10
10
10-2
Boxing
OneAm
BadySwitch
Doppleganger
RIQN
.17
.9
24
.43
KL Bound
52.6
708
$ 10
2 .99
$ 10
2 .99
DMC
KL Bound
Cartpole-B.
<
2
Quadruped-R.
< 10
2
Humanoid-S.
< 10
Walker-W.
2
RIQN (10 8, 10 ?)
PP-Mare (2)
347.5
22.5
.055
-005
509.3
6.30
.30
915
103.1
5.4
.01
862
PP-Mare
.01
011
10 2
10-2
Kangaroo
Floorswap
Difficulty+
DisahleMonkey
RIQN
.68
.19
055
.18
KL Bound
9.9
.777
S 10
2 .99
.60
2 .99
PP-Mare (.5)
85.2
.281
42.5
41.3
.937
Table 2: False positive rates (lower is hetter) for nominal (no novelty) environments. PP-Mare and RIQN hyperparameters are tuned for each environment and correspond to those reported in Tables 3. 4, and 4 for each environment.
RIQN (10-),10))))) 3,10-3)
166.3
.541
94.2
.99
93.1
.710
Freeway
InvisibleCars
ColorCars
ProzenCars
KL Bound
< 10
2
> .99
< 10 2
2
.99
< 10 2
985
PP-Mare (.5)
2.33
.99
1.84
2.60
.931
RIQN (10"7, 7,10
2.87
.938
2.62
.99
.502
.900
SeaQuest
DisableEnemy
Gravity
UnlimitedOkygen
4.2. Metrics
KI. Bound PP-Mare (.7) RIQN (10-2 .10-2)
.202
962
45.8
.499
< 10
2 .90
We use the following metrics. Novelty detection delay error the difference between the earliest time step that novelty is first observable and the time step that the novelty is detected. Average delay error (ADE) is the measure of how many steps off the detection method is from the true environment time step of the novelty. We calculate the agent's 's average delay error by averaging the novelty detection delay error across all novel environment episodes. False positives: We use the original nominal training environment to test for false positives. We do not consider false negatives when in the training environment. Ideal performance primarily mini- mizes the average delay error and false positive rates (Müller et al., 2022). In addition to average delay error, we also mea- sure the real time inference speed to compare the raw speed of each method when computational resources are held fixed. Finally, we measure AUC evaluate the discriminative abil- ity of anomaly scores (left-hand side of Eqn. 4) generated by each method.
111.6
.678
24.1
.882
3.73
.938
45.3
.272
157.3
.390
16.0
.701
Table 3:
Average Delay Error and AUC results for Atari environ- ments, with best tuned parameters when appropriate.
for RIQN, our proposed KL and PP-Mare methods achieved lower or comparable average delay error (ADE) and higher AUC to RIQN across all MiniGrid and Atari environments, while maintaining competitive performance in the DMC do- main. RIQN's faster ADE scores in DMC may be achieved by its willingness to trade-off higher false positive rates in certain environments, such as Ca Cartpole - 1 ance where the false positive rate is as high as 68%, but never lower than 5% and usually somewhere in the middle. Ar- guably, high false- colle-positive anomaly detection rates are un- desirable. Additionally, RIQN exhibits greater sensitivity to subtle observation changes, as reflected in its strong perfor- mance in low-noise environments.
4.3. Results
PP-Mare generally falls behind our KL bound, likely due to reconstruction error not being a measure optimized for computing semantic differences between the observations. Further, PP-Mare assumes a false correlation between pixels with high reconstruction error and novel regions of input im- ages (Feeney & Hughes, 2021). In order achieve a greater AUC for the PP-Mare method, better similarity/difference metric would be needed to quantify the differences between the observations.
False Positives Table 2 shows false positive rates for Atari and DeepMind Control suite environments. The false posi- tive rates for Minigrid oi -Dooroyy-- 6x6 is S 10-2, 100, and .52 for the KL bound method, PP-Mare, and RIQN.
The KL bound method false positives rate is S 10-2 across all environments tested, allowing for minimal confronta- tion from a potential user. We hypothesize that KL's high performance in avoiding false positives coincides with the behavior observed from Figure I and is direct result of the bounds formulation. RIQN's high false positive rate was also observed by Danesh & Fern (2021).
The AUC score also presents possible improvements that can be made over each method, as at times RIQN's cusum detection was unable to take advantage of decent AUC scores generated by each observation feature. The KL bound was effectively able to translate its higher AUC scores far more often than RIQN, despite no user interference.
Average Delay Error and AUC Scores Average Delay Error and AUC Scores are reported in Tables 3 (Atari), 4 (DeepMind Control Suite), and 5 (Minigrid). The KL bound appears to demonstrate strong potential as a versa- tile approach for novelty detection, across all environments, particularly due to its ability to detect nuanced differences in anomaly score scales. Despite tuning À and A parameters
Real Time Halting Speed We analyze the real time halt- ing speed to put ADE and FP rate into perspective for real world detection scenarios. Table 6 presents the real time halting speed increase in comparison to the RIQN algorithm.
6

---

Novelty Detection with World Models
ADEL
AUC*
ADE
AUCT
ADEL
AUCT
ADE
AUCT
ADE
AUCT
ADE AUC1
Caripole-3D-Bable
LowPerterh
HighPerterb
LowNoise
DoorKey-6x6
LavaGap
BrokenDoor
DoorGone
KL Bound
51.9
.799
49.2
.744
24.1
.833
KL Bound
.110
.732
$ 10
2
939
.120
940
PP- PP-Mare (77)
45,4
.299
43.7
.522
61.3
.600
(1)
.170
.077
.784
.080
RIQN (10 1,10 3)
6.20
915
5.75
890
3.67
2 .99
RIQN (10 2, 10 3)
< 10
.760
2.56
920
.066
600
HighNoise
LowDamp
HighDamp
Teleport
ActionFlip
KL. Bound
2.86
2 .99
20.2
.844
25.0
.770
KT. Bound
10 2
.922
S 10 2
991
PP-Mare (0.7)
22.7
840
110.8
.395
94.5
448
PP-Mare (1)
.080
.959
.105
962
RIQN (10 1,10)) ))
3.33
.923
5.33
.543
1.66
.520
RIQN (10 10-
6.29
.950
2.34
890
Quadruped-3D--m
LowPorterd
HighPorterb
LowNoise
Table S: Average Delay Error and AUC results for Minigrid environments, with best tuned parameters when appropriate.
KI. Bound
1.00
908
1.30
.011
24.8
.839
PP-Mare (1.5)
1.00
.793
1.26
830
17.3
645
RIQN (10 -.000"1000))
13.3
.006
12.7
883
4.70
850
HighNoise
LowDamp
HighDamp
Method
Average False Positive Rate
Inference Run-time Speedup
KL Bound
3.82
2 .99
89.8
.512
112.9
.507
DMC
Atari
DMC
Atari
PP-Mare (1.5)
3.94
.211
49.1
.488
52.86
.155
RIQN (10 110
4.02
.900
94.3
.266
103.0
489
RIQN
.275
.308
x1
x1
PP-Mare
< 10-2
< 100-
x1.66
102
x4.45 10'
Homaanidd-3DSStame
LowPerterh
HighPerterb
LowNoise
KL bound
< 100
< 10-2
x1334
10
x5.12 10a
KL Bound
44.4
925
10
2
>
.99
9.05
.755
PP-Mare
17.9
.833
15.2
.899
157.0
3633
Table 6: Real time performance of RIQN versus our proposed RL-specific detection methods, in DeepMind Control Suite and the Arcade Learning Environment (Atari) under all respective tested novelties.
RIQN (000".107))
64.8
854
32.3
.983
2.16
919
HighNoise
LowFriction
HighPriction
KL. Bound
10-2
2
59.8
.748
18.75
PP-Mare (4)
258.5
.32
17.3
853
8.10
822
RIQN (10- _1
1.50
.982
53.1
.549
12.7
.800
Walker-3D- Maneer-D-Waarl Walk
LowPotter
HighPorterb
LowNoise
model architectures, We report ADE, FP, and AUC in Ta- ble 7 for the Atari environment.
KL. Bound
< 10-"
2 .99
10
.99
.573
PP-Mare (7)
18.7
.755
7.16
.998
50.9
407
RIQN (10- 1,10
8.11
.743
7,80
.715
2.73
883
The KL bound works with transformer-basee world mod- els and has very low false-positive rates as consistent with the bound on other architectures. The transformer-bane IRIS (Micheli al., 2023) is slower to build evidence of the novelty, which is consistent with observations by Micheli et al. (2023) (Appendix B) about under-sampled states on the At AtaariFFii Meeri-Freeet FFeeway environment; the effect on novelty detection is described further in our Appendix C.2. Our KL bound method cannot be used with diffusion-based world- models that directly generate prior and posterior images instead of latent hidden states. Instead, we use PP-Mare, the closest ablation that operates directly on predicted observa- tion differences.
HighNoise
LowFriction
HighFriction
KL Bound
1.48
.911
17.25
.911
5.93
> .99
PP-Mare (7)
21.1
.546
2.37
.943
1.98
.962
RIQN 1107)
2.10
.966
10.55
Table 4: Average Delay Error and AUC results for DeepMind Control environments, with best tuned parameters when appropri- ate.
We observe that the KL computation increases the speed performance in both the Atari and DMC domains, by magnitude of 102 and 1 respectively. Despite the heavier architecture of world model, the advantage of using KL or PP-Mare over RIQN appears to be the removal of the need to compute an individual anomaly score over each m feature (of the observation) for e samples, as well as operating the cusum calculation over all m features for each transition. In- deed, we see the increase in observation dimension between Atari and DMC vastly slows the RIQN algorithm as feature dimension begins to rise.
Exploration Effects We turn our attention toward the hypothetical KL divergence of:
K Klp(ze (he, || po(2z xx)] S.
where /e x2) is the theoretical true distribution of the training environment.
If a world model is trained alongside a policy, the world model is restricted to train on the local transition space as the agent zeros in on particular trajectories of states with high value the policy improves (Kauvar et al., 2023), This can result in the phenomenon where the policy restricts what the world model is able to observe and learn from. In these cases, aspects of the environment that have not been observed enough may appear novel even. This is
Alternative World Models In this section we explore world model architectures other than the RNN-based Dream- erV2 architecture. World models can alternatively be built on top of diffusion (Ho et al.. 2020; Alonso et al., 2024), and transformer (Vaswani et al., 2017; Micheli et al., 2023) models. We provide detailed instructions (see Appendix B for details) required to replicate results on different world
7

---

Novelty Detection with World Models
ADE
AUCT|
ADE AUC
ADEL AUC
FP/
Atari Freeway
InvisibleCars
ColorCars
FrozeCarr
RNN (2reamerVV)
< 10
2 .9
< 10
2 99 S 10
985
< 10
Diflusion based
10
978
10-2
2
977
10
988
10-2
Transformer hased
2.8
.866
134
681
100-
Table 7: We compare our KL-bound results implemented on RNN- based DreamerV2 (Hafner et al., 2021) to alternative world model architectures: diffusion-barn based (Alonso et al., 2024) and transformer-base (Micheli et al., 2023).
Figure 2: Minigrid full render of a simple FakeGoal environ- ment. Here empirically observe when the agent detects novelty in an task that has already been completed by disabling the goal. For this ablation experiment, we tentatively define ground truth novel transitions as transitions that interact with the fake goal. The most common transition initially flagged is from left to right, We suspect that this is due to the observation (light gray) that there appears to be nothing on the other side of an open door.
Table 8: Metrics for Fake Goal Environment
Method
Accuracy
5. Limitations
KL
0.49
PP-Mare
0.46
Experimentation with novelty detection algorithms can be difficult it requires environments that have novel alter- native environments. To detect a novelty, one must be able to alter the dynamics of the environment. This creates conundrum where experimentally we know the alternatives but must not allow data about the alternatives to inform the training of agents in the nominal, pre-novelty environment. Experimenter bias is also a risk faced by those that research novelty detection- our PP-Mare and KL-Bound methods were developed before we identified Atari and DeepMind Control Suite sources of environments with alternatives.
RIQN
0.21
KLExplored
0.91
technically correct because the agent is observing aspects of the environment that it failed to learn about during training. However, is also not always what intended because the novelty detection is triggering off of the wrong aspects.
To give concrete example, in MiniGrid DoorKey-6xt 6x6, an agent that is trained to always go from the left-hand side of the map to a goal on the right-hand side of the map will never need to turn to look at the back side of door. However, if the novelty is that it starts on the right side of the door, it will see the back side of the door, in Figure 2. This transition was possible during the training of the world model, but it was illogical to consistently explore looking backwards and the policy quickly learned to discourage that behavior, depriving the world model of observations that should have been able to occur naturally. After the novelty, the world model's 's locally learned dynamics classifies this transition as a novelty.
While the research challenge of novelty detection is inspired by the real world, where sudden, unanticipatable, and per- manent changes to how the world works occur relatively frequently, experimentation in the real world is often infea- sible, requiring virtual surrogate testbeds.
While our proposition using KL divergence has proven ef- fective empirically, future work might explore alternative divergence measures with metric properties, such as the Jensen-Shannon divergence or Wasserstein distance, which could provide stronger theoretical guarantees. Additionally, exploring the relationship between latent space dimension- ality and the reliability of variational approximations (of which our bound is based see Appendix D) could yield insights into optimizing model architecture for novelty de- tection.
To illustrate that it is an artifact of how the agent is learned, we provide an alternative training paradigm, KLExplored where we train the agent from both sides of the map pre- novelty. Table 8 shows that this increases novelty detection accuracy because this transition no longer flagged. We leave the decision of declaring novel transition as novel based on the local environment dynamics or the agent's 's beliefs for entirely different discussion (Miljković, 2010; Balloch et al., 2022). This highlights the significance that novelties are unanticipatable because a training paradigm that is prepared for this case will be more accurate. However, we assume that novelties cannot be anticipated in advance and thus factored into the training paradigm.
6. Conclusions
Novelties are sudden changes to the observation space or environment state transition dynamics that occur infer- ence time that unanticipated (or unanticipatable) by the agent during training. Novelties are permanent distribution shifts, distinguishing them from anomalies that are local- ized one-time time out-of-domain occurrences. Novelties happen frequently in the real world and this paper addresses the detection of novelties, as defined above, but leaves the re-
8

---

Novelty Detection with World Models
sponse novelties as out of scope.
References
This paper proposes novel way to detect novelty in re- inforcement learning settings using world model. The KL bound method we introduce is demonstrably resilient to false positives while simultaneously detecting novelty quickly and accurately. Crucially, our KL bound method does not require thresholds other hyperparameters. This essential because tuning thresholds and hyperparameters requires some priori intuition about the nature, scope, and scale of anomalies, which is an assumption that is disal- lowed in our research setting on novelty.
Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., and Fleuret, F. Diffusion for world model- ing: Visual details matter in atari. 2024.
Angiulli, F., Fassetti, F., and Ferragina, L. Reconstruction error-based anomaly detection with few outlying exam- ples, 2024. URL
Asperti, A. and Trentin, M. Balancing reconstruction er- ror and kullback-leibler divergence in variational autoen- coders, 2020.
Our paper supports the overall value world model based RL implementations because the world model can be re- purposed to anomaly and novelty detection. Our method appears robust enough provided the basis for future re- search on addressing inference-time novelties such as those that occur in the real world.
Áström, K. Optimal control of markov processes with incomplete state information. Journal of Mathematical Analysis and Applications, 10:174-205, 1965.
Balloch, J., Lin, Z., Hussain, M., Srinivas, A., Wright, R., Peng, X., Kim, J., and Riedl, M. Novgrid: A flexible grid world for evaluating agent response to novelty, 2022. URL http://aaiixorr//b//20011111111
Acknowledgments
Chevalier-Boisverr M., Willems, I.., and Pal, S. Minimal- istic gridworld environment for gymnasium, 2018, URL https:/github..co/aaaaaaa Ptw.:/aaaaaamauuuuaaatttiiiiiii
We gratefully acknowledge Minh Vu, Manish Bhattarai, and Sebastián Gutiérrez Hernández for their valuable insights and stimulating discussions, which significantly contributed to the development and refinement of the ideas presented in this work. Their thoughtful feedback and engagement were instrumental in helping shape certain areas.
Cobbe, K., Hesse, C., Hilton, J., and Schulman, J. Lever- aging procedural generation to benchmark reinforce- ment learning. CoRR, abs/1912.01558, 2019. URL http:/aarii.oorgabs//1122..5.
Impact Statement
Dabney, W., Ostrovski, G., Silver, D., and Munos, Im- plicit quantile networks for distributional reinforcement learning. 2018. URL
This work advances the field of reinforcement learning by exploring methods for detecting unanticipated changes in the environment, with deliberate emphasis on minimiz- ing reliance on additional hyperparameters. By leveraging world models in conjunction with a novel KL bound tech- nique, we present a reliable and efficient approach to identi- fying persistent distributional shifts. Our findings highlight the broader utility of world models beyond policy learning, demonstrating their potential for robust deployment in dy- namic, real-world settings where adaptability and resilience are essential. This contribution establishes a foundation for future systems capable of not only detecting but also responding to novelty with minimal human oversight.
Danesh, M. H. and Fern, A. Out-of-distribution dynamies detection: Rl-relevant benchmarks and results. In Inter- national Conference on Machine Learning, Uncertainty & Robustness in Deep Learning Workshop. 2021.
Delfosse, Q., Blüml, J., Gregori, B., and Kersting, K. Hackatari: Atari learning environments for robust and continual reinforcement learning. arXiv preprint wwXh::2406003997 2024.
Dulac-Arnold, G., Levine, N., Mankowitz, D. J., Li, J., Paduraru, C., Gowal, S., and Hester, T. An empirical in- vestigation of the challenges of real-world reinforcement learning. 2020.
Fang, Z., Li, Y., Lu, J., Dong, J., Han, B., and Liu, F. Is out-of-distributie detection Icarnable? In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022. URL http:///opaarriieww.e//oommttmdt 7ZZGXOE.
Feeney, P. and Hughes, M. C. Evaluating the use of re- construction error for novelty localization, 2021. URL https//ariiv.oo//aa/2/10...33

---

Novelty Detection with World Models
Fu, J., Co-Reyes, J. D., and Levine, S. EX2: exploration with exemplar models for deep reinforcement learning. CoRR, abs/1 2017. URL http://arxxvv.rr//bbs/ 1703.01260.
Miljković, D. Review of novelty detection methods. In
The 33rd International Convention MIPRO, pp. 593-598, 2010. ISBN 978-1-42444-773-0
Mocrland. T. M., Broekens, J., Plaat, A., and Jonker, C. M. Model-based reinforcement learning: A survey, 2020. URL https:/aaxii..rg/ab///00..... 16712..
Greenberg, I. and Mannor, S. Detecting rewards deterio- ration in episodic reinforcement learning, 2020. URL
https://arxi..orr/////00001.... 11660.
Müller, R., Illium, S., Phan, T., Haider, T., and Linnhoff- Popien, C. Towards anomaly detection in reinforcement learning. In Proceedings of the 21 International Con- ference on Autonomous Agents and Multiagent Systems, AAMAS '22, pp. 1799-1803, 1033, Richland. SC, 2022. Inter- national Foundation for Autonomous Agents and Multia- gent Systems. ISBN 9781450392136.
Ha, D. and Schmidhuber, J. Recurrent world models facili- tate policy evolution. In Advances in Neural Information Processing Systems 31, pp. 2451-2463. Curran Asso-
ciates, Inc., 2018a. URL http://aaprrs.nisscc/aaaa 7512-recurrent- world-models- facillitte-i cecllittte-poicc--eee policy-evolution. https://worldmoodls...iii gitb.u...
Ha, D. and Schmidhuber, J. World models, March 2018b. URL tive version of the article at https:/wortdmoodekkgghhe
Nasvytis, L., Sandbrink, K., Foerster, J., Franzmeyer, T., and de Witt, C. S. Rethinking out-of-distribution de- tection for reinforcement learning: Advancing meth- ods for evaluation and detection, 2024. URL https:
Interac-
Hafner, D., Lillicrap, P., Norouzi, M., and Ba, J. Mas- tering atari with discrete world models. In International Conference Learning Representations, 2021. URL http:://opnneeveeeeeefffommmm//" wwnee/ffoummmddddoobaawbbb
/arxiv.org/bb/24444077999
Page, E. S. Continuous inspection schemes. Biometrika, 41(1/2):100-115, 1954. ISSN 00063444. URL http:
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851 Curran Associates, Inc., 2020. URL http://iinnnneeeerec.c//ppppeeee flc4455baccc554441000667771111.2 oow.:///haa.0.0ttttoo00o0000ooao
/www.jstorog/sabbe233309999
Pimentel, M. A. F, Clifton, D. A., Clifton, L. A., and Tarassenko, L. review of novelty detection. Signal Process., 99:215 249, 2014.
Robine, J., Uelwer, T., and Harmeling, S. Smaller world models for reinforcement learning, 2021.
Itti, L. and Baldi, P. Bayesian surprise attracts human attention. In Weiss, Y., Schölkopf, B., and Platt, J. (eds.), Advances in Neural Informa- tion Processing Systems, volume 18. MIT Press, 2005. URL https://proceedingg. newrrrrieddings..nnns....... cc/paper/2005/ 110011222291a1844468555555312222 Pa.eer..d.
Schmidhuber, J. Making the world differentiable: on using self supervised fully recurrent neural networks for dynamic reinforcement learning and planning in non-stationary environments. Forschungsberichte, TU Munich, FKI 126 90:1-26, 1990. URL https://api. senamtischharrooo/oopusi:::::-/:
Kauvar, I., Doyle, C., Zhou, L., and Haber, N. Cu- rious replay for model-based adaptation. In Interna- tional Conference Machine Learning, 2023. URL
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, Dropout: simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929-13)) 1958, 2014. URL http: wtwt://stwwsssiiiaaaaaaavaaaaaaa | htm.
Lee, K., Lee, K., Shin, J., and Lee, H. A simple random- ization technique for generalization in deep reinforce- ment learning. CORR, abs/1910.05396 2019. URL http://arrivoog//bb//91..0099
Tunyasuvunakool, S., Muldal, A., Doron, Y., Liu, S., Bo- hez, S., Merel, J., Erez, T.,, Lillicrap, T., Heess, N., and Tassa, Y. dm control: Software and tasks for continuous control. Software Impacts, 6: 100022, 2020. ISSN 2665- 9638. doi: https://doi.org//00000666 20000200.000000 100022. URL http://wwww..cccccddcccccoootten $2665963820300099.
Machado, M. C., Bellemare, M. G., Talvitie, E., Veness, J., Hausknecht, M. J., and Bowling, M. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61 ::523-562, 2018
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polo- sukhin, I. Attention is all you need. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus,
Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample-efficient world models. In The Eleventh Interna- tional Conference on Learning Representations, 2023.
10

---

Novelty Detection with World Models
R., Vishwanathan, S., and Garnett, R. (eds.), Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https//ppocccdingsneuuri....pppp httppe/ppe/ep///2211777 flle3t55e243347ddeee1 ftbd033elc4aBeee
Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, P.- A. Extracting and composing robust features with de- noising autoencoders. In Proceedings of the 25th Inter- national Conference on Machine Learning, ICML '08, Pp. 1096-1103, New York, NY, USA, 2008, Association for Computing Machinery. ISBN 67816055800 doi: 10. 10.1145/139016 1390294. URL https///doi.rr//0..1144 1390156.139900
Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., and Stanley, K. O. Enhanced poet: Open-ended reinforce- ment learning through unbounded invention of learning challenges and their solutions, 2020.
Wang, T., Bao, X., Clavera, I., Hoang, J., Wen, Y., Lan- glois, E. D., Zhang. S., Zhang, G., Abbeel, P., and Ba, Benchmarking model- reinforcement learning. CoRR, abs/ 907.02057, 2019. URL http://arxiv.or//abs/ 1907.02057.
Werbos, P. Learning how the world works: Specifications for predictive networks in robots and brains. In Proceed- ings of IEEE International Conference on Systems, Man and Cybernetics, N.Y., 1987.
Wilson, G. and Cook, D. J. survey of unsupervised deep domain adaptation. ACM Trans. Intell. Syst. Technol., (5), jul 2020. ISSN 2157-6904. doi: 10.1 10.1145//400066. URL https://dii.org/1... 1.1144444114666
Zhang, L. H., Goldstein, M., and Ranganath, R. Understand- ing failures out-of-distribution detection with deep generative models. CoRR, abs/2107.06908 2021. URL http::/arxiv.org/ab//000....0.00 020.006908.
Zhao, H., des Combes, R. T., Zhang, K., and Gordon, G. On learning invariant representation for domain adapta- tion, 2019.

---

Novelty Detection with World Models
A. Reconstruction Error Comparisons
Figure 3: From left to right, Eiprior: ot posteriir and 2t observations of agent trying to open door in the BrokenDoor custom minigrid environment where the door fails to open despite having the correct key. The intuition of PP-Mare is to distinguish high reconstruction loss between samples.
Trained Threshold
Trained Threshold
Random Threshold
3000
Combination Threshold
Combination Threshold
800
Number Samples
Number of Samples
600
2000
1500
400
1000
500
60
Train
Train Loss
Figure 4: Reconstruction alongside the three tested thresholds: Random Model, Trained Model and Combination model. Reconstruc- tion error from nominal MiniGric (Right). Each vertical line refers to the cut off value for the corresponding threshold. Samples to the right the line are classified as novelty. Reconstruction Error was generated from trained agent. It appears that no threshold (even tuned) would separate the space.
Gr LenniiGrd-DDooooo dei--Dooroyy-xx
environment (Left), and Reconstruction error novel LavaGap environment
B. Novelty Detection For Other World Model Frameworks
We consider world models architectures that learn representations of past and present states, alongside a distinct predictive model the future distribution states (Werbos, 1987), preferably a powerful predictive model implemented on general purpose computer such as a recurrent neural network (RNN) (Schmidhuber, 1990) (Ha & Schmidhuber, 2018a; Hafner al., 2021). We show how our technique can be used on a RNN-based, Transformer-basee and Diffusion-based World Model respectively:
IRIS: (Micheli al., 2023) At high level, the Transformer captures the environment dynamics by modeling the language of the discrete autoencoder over time. We expect that the categorical probability distribution prompted from the predicted observation logits when generating can be used compute the divergence between two outputs. Since the two world models are sufficiently different we briefly experiment with possible interpretation of KL to the IRIS framework. We first construct the bound similarly to Eq 4. We model Ps (s(zeho xx) as an auto-regressive prediction of the logits zf to 2f where zf is predicted from the true observation x4. We disable kv caching to mimic hg. model pa (z+(ho) as an auto-regressive prediction of the logits z! to zt where = predicted from the zero vector substituting for Fe in the IRIS framework, we also disable kv caching to mimic hg. We experiment with interpreting P [b,, 2x) as po(zelho, x2) except Ps (zel/t,, zx)) utilizes kv caching. For our results, we experimented with bounding by K KL[ps(2 (Pp(2e)l (e, 2x)lp
(ho))) hoo] K LLpop (h, xx)] Po(2 22)]. We interpret the Bayesian surprise (. where Ê is generated from previous sampling of Pa pa(zel/u, xr)..
as K KL L.po(= (sgr) ) [llllplaa (2,(,.11, For our results, we experimented with utilizing KL LL ac:)) (2) E5t-2, ag:)] as a score for each auto-regressive step. Although our initial interpretation had moderate success, we expect that future work should be able to improve by adjusting the bound as well as addressing the exploration concerns given the exploration trick used in (Micheli et al., 2023), of which the "up" action was heavily sampled in favor of helping the agent complete the game,
12

---

Novelty Detection with World Models
rather than improving the world model performance. Ultimately, we leave the task of enhancing novelty detection in the IRIS world model for future work.
Diamond:
(Alonso al., 2024) constructs a conditional generative model of the environment dynamics, Po(#e+1 (gge a.ge), and considers the general case of POMDP, in which the Markovian state 8t is unknown and can be approximated from past observations and actions. To construct p, the authors condition diffusion model on the history, to estimate and generate the next observation directly. Since Diamond operates primarily in the observation space, we use PP-Mare to employ a interpretation of our technique. We utilize po(I (Zge, age) as our prior, and I ~ P& (# ag++1) = p( as our posterior at each step.
C. Policy Exploration and Uncertainty Effects On Detection
C.1. Base Divergences in the Nominal Environment as Training Progresses
Figure 5 shows how the KL bound evolves with training for Atari environments. Each row is equivalent to Figure but with each component of the bound split into a separate graph:
(Tlpo((al44,:==)llo<01///// - KT. [T (za lh, x,) Ilpo rt)]. right-hand side of Eqn 4.
(Lipg(za/ht,_xx)lppp0022 left-hand side of Eqn 4
(Zipp(2a)lb,==>)lbbbb /ho. 24)], the subtracted component of the left-hand side of Eqn 4
(2 Ips [. the component of Eqn 4 that is subtracted against.
Note that the scale of the y-axes differ. The significant observation that in the nominal environment, the right-hand side of the bound is always higher than the left-hand side of the bound.
C.2. Exploration effects
We turn our attention toward the hypothetical KL divergence of: K the theoretical true distribution of the training environment. If a world model is trained alongside policy, the world model is restricted to train on the local transition space the agent zeros in particular trajectories of states with high value as the policy improves (Kauvar et al., 2023).
x:)] S where p(zel (ee x4g) is
This can result in the phenomenon where the policy restricts what the world model is able to observe and learn from. In these cases, aspects of the environment that have not been observed enough may appear novel even. This is technically correct because the agent observing aspects of the environment that it failed to learn about during training. However, it is also not always what is intended because the novelty detection is triggering off of the wrong aspects.
To give concrete example, in MiniGrid -erridd--Doo------oe 6x an agent that is trained to always go from the left hand side of the map to goal on the right hand side of the map will never have need to turn to look at the back side of a door. However, if the novelty is that it starts on the right side of the door, it will see the back side of the door, as in Figure 2. This transition was possible during the training of the world model, but it was illogical to consistently explore looking backwards and the policy quickly learned to discourage that behavior, depriving the world model of observations that should have been able to occur naturally. After the novelty, the world model's locally learned dynamics classifies this transition as novelty. To illustrate that it is an artifact of how the agent learned, we provide an alternative training paradigm, KLExplored where train the agent from both sides of the map pre-novelty. Table 8 shows that this increases novelty detection accuracy because this transition is no longer flagged. We leave the decision of declaring novel transition as novel based on the local environment dynamics or the agent's 's beliefs for an entirely different discussion (Miljković, 2010; Balloch et al., 2022). This highlights the significance that novelties are unanticipatable because a training paradigm that is prepared for this case will be more accurate. However, we assume that novelties cannot be anticipated in advance and thus factored into the training paradigm.
C.3. Policy Behavior against KL Divergence
World Model vs Agent Uncertainty Since the policy is configured to make decisions solely on the world model's 's predicted belief states (z+, ht), in Figure 6 we analyze the effect of the world model on the policy's 's entropy (the degree of
13

---

Novelty Detection with World Models
Atari-Boxing
0.1
015
031
01 05 07
11
020004
Atari-Seaquest
0.3
045
0.5
02
005
003
194 25/ 291 00 02
0M 944 1M12M
14 2M do 2M do du Sur MMR2MS
M.C 1M2M
Atari-Kangaroo
03
0.15
0.25
0.1
(A1
0M
144
14
291
10M
011
891
10M
1.0M
431
Atari-Freeway
1.5
30
225
23
30
0.0
32
0
Figure Divergences during training (Note scale of the y axis)
Figure Novelty score (X-axis) against the entropy of the policy (y-axis) during different stages of training. Here we show that regardless of novelty being predicted (Top) or not (Bottom), the entropy of the policy does not appear to be strongly affected by the divergence of the posterior and prior, regardless of current training performance.
14

---

Novelty Detection with World Models
Cartpole
Quadruped
0.3
Homelized
0.2
6.1
12
5.
1.-
Timestep
Timentop
Timeatop
Timeatap
Kangaroo
Freeway
0.7
0.5
W
6.3
0.4
1.4
64
0.2
02
8.1
14
Timeatop
Timestop
Figure The normalized anomaly score trend of each detection method as the episode progresses 50 time-steps in the nominal environment.
disorder or uncertainty in system) during the case of novelty detection and vice versa. From our experiments, it appears that regardless of the measured surprise from the world model, the policies entropy maintains weak relationship with the world model throughout the entirety of training. This suggests that simply relying the policy may be powerless in determining if novel transition has occurred.
Detection Score Trend Over Time Figure 7 presents the normalized anomaly scores, (the left-hand side of Eqn 4 for KL, and Eqn for PP-MARE) for each environment in the Atari and DMC domains as timestep progresses in the nominal, novelty-free enviromments. KL Bound and PP-Mare detection methods are expected to have higher anomaly scores in the first few steps of a training episode because the world model struggles generate reasonable predictions due to its arbitrary prior initialization. Anomaly detection drops rapidly or was never high begin with. This is in contrast RIQN, where the anomaly score fluctuates and is higher for all environments except one. appears that leveraging learned surrogates conditioned on the agent's actions appears to provide more informative signals of normality in higher- higher-dimensionall spaces, enabling the detection of novelties with minimal delay error.
15

---

Novelty Detection with World Models
D. Derivations
Mapping from cross entropy to KL divergence:
H (ppozeelee vz), Po(zelho)) -
rz)) (K (2 2 (oggall xz)),
- Lp (e, a.)ll (oo
Proposition 3.1 (restated) If the cross entropy score compurison becomes negative when introducing the vector 2u, then the right side of (4) will become negative, which immediately flags 24 as novelty due the property of non-negativity of the left side KL divergence.
((po at). - pall x+)) <0 =
K KL ( (TAp K KTTlpo (atp:a/aa))aall///aaaaaa/aa/aaa/ mz)lllss < 0. Then, K ( hy, on ())lppp(l
KT. KIlppo((2(hh.
K [plps zz) (z)lt()))))))) < 0; Which is impossible.
Proposition 3.2 (restated) If the cross entropy score comparison becomes nonnegative when introducing the vector It then 4 defines decision boundary in the latent space over the measure of robustness to partial destruction of the input (Vincent al., 2008; Srivastava al., 2014), Le:
KL [,,
KL KL 2, (2
H ((po(zcle,, (.), p2(2)(ho)) H (p+ 2, 2x)) 0:
H (2. (2/ x htgp://4hhhhohlaoocc..o/.o///.aa
m.) (t)) - (mo ||gp://.ehhaaapooooaoa///////a/a x2)) 2
[rap::/--KKKkaaaaaaaaaaaaaaaa1a1 21)) 2
KL KL
http://///aaaaalllooo/oo./////// 21) 2
I.)I
K LL (2,
2 KL Nr 2;) 11)p
KLLpa (x)o(
(Iff the local training of K ( /4, x4) | (2 [44]] < holds for some e)*
Let the full posterior (2 |x,, h₂) be the most informed distribution and the desired distribution with support over the latent space Z any given and let So map the representation of distribution with the support of well.
Utilizing notation similar that in Vincent al. (2008), denote (h,, as the clean input (not to be confused with xt), and define three variants of a': -k)) o (-a)): * (,,, .)> which denote the removal of features (hz), (2t), and (he, 22) from the clean input x, respectively.
Since the loss is at the latent level, the robustness to partial destruction of single input x for a desired distribution is measured as:
L(p hes 21), fa()))
Therefore, if chosen to be the KL divergence, and x' becomes explicit, then Proposition 3.2 as can be rewritten as:
[(po(zelM) 24). po (ho)) L(po 2.), 2.), (/ 2t) (p(e(ho. K KT Po (b)) > K (T(ps xx) pa (h,)) + (T,
where Ps corresponds to the distribution modeled by fe given some x' and ho represents the state when no conditioning information available.
16

---

Novelty Detection with World Models
Theoretical Guarantee of Equation 4 Assumption: Suppose the objective defined by Equation 2 is minimized such that:
8KL [oo(ze)),, m2) (ps 0,
implying that is conditionally independent of xi given hT. Furthermore, assume that the model distribution po is correctly specified, meaning there exists set of parameters 0" such that:
Po (-I+)) = p-(-.))
where (+--) denotes the well defined true conditional distribution. Then we can rewrite Proposition 3.2 as:
KL(po(zelhe 24) II Po(2e)ho)) KI(po(z,lh, 24) Pa(= 27))
Note that the explicit definition is:
KL (polze her xr) ppole ho)) = (pp(eelhirn))
log (= 1e, wir (2 h2o)
KL ppl(z I he, 22) ppo ho. = (pp(Eclhirr))
log Pol 11,, xe.)
(2t hos xte)
Now consider the difference:
A = KL (polee 2e, (ooee)/lole l hol) - KL (polze me. pples hos ze))
log Po(2 he,
log pole he, 24) po(ze ho xt))
A Exp(ae (Ma,as)
E Epo(zeleerra)
(2r ho)
log Pol [t he,
log p=( he, 224) po(2 ho: Iee)
Ps 22 ho)
Po (zc ho, 221)
log
p0(zt ha)
Thus, we have key result:
log Poler (gg22)) palze ho)
Epo(zc
The inequality
KL(polz he, (tssze)/pose ho)) 2 KL <pollz bessei/looee no, 21)
thus holds whenever
(xe(beebe)))))) Log Polz ho, ke) Po(ze ho)
2 0.
Which can be interpreted as holding iff the Expected Information Gain (EIG) of x, nonnegative under the distribution of Ps(ze] xz).
Discussion on KL Divergence Properties We briefly note that KL divergence is not true metric. The bound in Proposition 3.2 compares the posterior to three different priors and provides useful approximation for detection, rather than strict mathematical guarantee. Formally, for distributions P. Q, and R, the KL divergence does not generally satisfy DKI (P || R) < (P || Q) + Dxr(Q || R). This means our bound might be violated not only due to novelty but potentially due to the inherent non-metrie behavior of KL divergence, particularly in high diggh-iimssionall or complex latent spaces.
17

---

Novelty Detection with World Models
E. Novel Environments
Below we state each novel environment alongside what timestep was used to evaluate our method (Note that frame number is different from 7):
MiniGrid
Dooorey-6x6
BrokenDoor: The door no longer opens, even with the key. T is set to the timestep when the agent attempts to open the door.
ActionFline All movement actions set to the opposite direction. T is set to when the agent chooses to rotate.
Teleport: Teleport the agent randomly then perform the selected action. T is set to 5.
HeavyKey: The key can longer be picked up. T is set when the agent tries pick up the key.
LavaGap: A lava gap is introduced around the goal. set to when the lava the agent's sight.
DoorGone: The door is removed from the environment and is replaced with empty space. T is set when the empty space is in the agent's sight.
Atari
Freeway
InvisibleCars: All cars are invisible but still move. T= 100.
ColorCars:
All cars are changed to black. T=100.
FrozenCars: All cars are suddenly frozen, reset, and do not move. T=100.
- Kangaroo
FloorSwap:
The agent suddenly switches onto different floor. 300.
Difficulty+; The games difficulty switches to the hardest. = 300.
DisableMonkey: All monkeys are removed from the game. T is set to when the monkeys are fully on the screen.
SeaQuest
DisableEnemy: Enemies are removed from the game. T is set when the enemies are fully on the screen. T
=30..
UnlimitedOxygen: The player suddenly gains unlimited oxygen. = 500.
GravityDrift: The gravity begins become stronger. T= 300.
Boxing
OneArm: One of the agent's arms are disabled. set to the time-step when the agent first attempts to use the disabled arm.
BodySwitch: The agent and the opponent switch places visually. T 300.
Doppleganger: The agent and opponent look exactly the same. T= 300.
Deep Control Suite
All environments use the following:
Perturb: The agent suddenly has a limb length increase. T
Walker: Thigh length increase to .3 (Low) or (High).
Humanoid: Head size increase to .2 (Low)
(High).
Cartpole: Pole mass increase to (low) or 10 (High).
Quadruped: Shin length increase to (low) or 2 (High).
Noise: The agent's observations begin to be filled with gaussian noise. Standard Deviation: 5 (Low) or Standard deviation: 30 (High). 30.
Friction: The agent's environment has 6 (Low) or 10 (High) increased friction. T 1.
Damping: The agent's joints have (Low) (High) increased damping. T= 30.
18

---

Novelty Detection with World Models
We first train agent learn e-optimal policy in the nominal environment, then transfer the agent of theenvironments with novelties during testing time. We then let the trained agent take steps in each novel environment, capturing 50,000 steps within various independent and identically distributed episodes; tracking each time step where novelty is detected to imitate possible agent halting situations. The agent's trained policy is expected to be sub-optimal in the novel environments and is not re-trained at any point to adapt to the novel environments dynamics. All novelties are experienced in every episode traversed.
F. Hardware Requirements
All experiments can be sufficiently reproduced utilizing NVIDIA GeForee GTX 1080 GPU with at least GB of VRAM for environment complexity, AMD Ryzen 5 5600X 6-Core Processor and at least 50 MB for files, excluding training data which is dependent on environment and model hyper-parameters.
Future work would likely go beyond the scope these hardware requirements, and we expect that cloud computing is a necessity to experiment in larger domains.
F11 Dreamer World Model Training Parameters
Name
Symbol
Value
World Model
Dataset size (FIFO)
2 106
Batch size
B
50
Sequence length
L
50
Discrete latent dimensions
32
Discrete latent classes
32
RSSM number of units
1024
KL loss scale
1
KL balancing
0.8
World model learning rate
2 10
Reward transformation
tanh
Behavior
Imagination horizon
H
15
Discount
0.99
X-target parameter
0.95
Actor gradient mixing
Actor entropy loss scale
1
1 10-3
Actor learning rate
4 10-
Critic learning rate
10-02
Slow critic update interval
100
Common
Policy steps per gradient step
4
MPI. number of layers
MPL number of units
400
Gradient clipping
100
Adam epsilon
10
Weight decay (decoupled)
10-6
Table 9: We utilize the default training parameters specified in trained could help understand the sensitivity of the bound.
Manipulating how the model
19


Reviews:
[{"review_id": "36HVYk8GLH", "reviewer": "Reviewer_iyHf", "summary": "The work proposes a principled method for detecting novelty in RL agents that use latent dynamics models, such as DreamerV2. The central idea is that when an agent encounters novel observations or dynamics, the latent state inferred from the current observation (posterior) will significantly differ from the one predicted by its internal dynamics (prior). This mismatch is captured through the KL divergence between the posterior and prior latent distributions. Rather than relying on heuristic thresholds, the authors introduce theoretically motivated bounds that relate this KL divergence to priors conditioned on reduced information — namely, with no history or only the current observation. These bounds are formalized through Propositions 3.1 to 3.2, enabling novelty detection without supervision or tuning.\n\nThe paper evaluates this method across several domains, including DeepMind Control Suite, MiniGrid, and Atari, introducing both observation and transition-based novelties mid-trajectory. The results show that the proposed KL-based approach reliably detects novelty with low false positive rates and sharp response times, outperforming baselines like RIQN. A key contribution is the demonstration that a well-trained latent world model can serve as a robust internal monitor of novelty, offering a unified, efficient, and interpretable mechanism to detect both perceptual and dynamic anomalies in reinforcement learning environments.\n\n### Update after rebuttal\n\nI am confident in my assessment of accepting this paper, It's a good work.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time in working to improve our presentation and their encouraging feedback. We are delighted that you took the time to assist the quality of our work. In response, we outline provisional revisions below:\n* We agree that the discussion of C.2 may be more informative to the reader than the section *Detection Score Trend Over Time*, since the world model's final performance is directly tied to the exploration of the environment. Therefore, we have moved section C.2, Figure 7, and Table 8 into the main manuscript, and have moved *Detection Score Trend Over Time*, and Figure 2 into the appendix for those interested in visualizing the anomaly score trend. Section C.2 now appears after the section *alternative world models*.\n* We agree that it is important to make a note to the reader about the downsides of not using a proper metric such as the KL divergence, and the method's dependence on variation approximations. In response, we've extended the derivations section to discuss explicit details of the dependence on variational approximations as well as assumptions on the properties of the model itself. We state: “We briefly note that KL divergence is not a true metric. The bound in Proposition 3.2 compares the posterior to three different priors and provides a useful approximation for detection, rather than a strict mathematical guarantee.\nFormally, for distributions P, Q, and R, the KL divergence does not generally satisfy $D_{KL}(P \\parallel R) \\leq D_{KL}(P \\parallel Q) + D_{KL}(Q \\parallel R)$. This means Eq 4 might be violated not only due to novelty but potentially due to the inherent non-metric behavior of KL divergence, particularly in high-dimensional or complex/untrained latent spaces (see exploration effects)”. In addition, we also redirect the reader to this discussion as we note the reliance of functional approximators in the limitations section, which reads:\n*While our proposition using KL divergence has proven effective empirically, future work might explore alternative divergence measures with metric properties, such as the Jensen-Shannon divergence or Wasserstein distance, which could provide stronger theoretical guarantees. Additionally, exploring the relationship between latent space dimensionality and the reliability of variational approximations (of which our bound is based on see Appendix D) could yield insights into optimizing model architecture for novelty detection.*\n* We have summarized the contents of Appendix H in Appendix B to improve the self sufficiency of the paper, (which ties into C.2 Exploration effects) of which we discuss the exploration trick of weighing a single action heavily that IRIS incorporated to successfully earn a higher score (which could affect the performance of the world model). Specifically, we add to Appendix B:\n*Although our initial interpretation had moderate success, we expect that future work should be able to improve by adjusting the bound as well as addressing the exploration concerns given the exploration trick used in IRIS, of which the \"up\" action was stated to be heavily oversampled in favor of improving the agent performance, rather than improving the world model performance. Ultimately, we leave the task of enhancing novelty detection in the IRIS world model for future work.*", "claims_and_evidence": "The claims made in the submission are largely supported by clear and convincing empirical evidence. The authors claim that their KL-based novelty detection method, grounded in the latent space of a Dreamer-style world model, is capable of reliably identifying both observation-level and dynamics-level novelties across a range of environments. This is substantiated through a series of well-designed experiments on DMC, MiniGrid, and Atari, where the proposed method outperforms baselines. The per-timestep KL score clearly spikes in response to injected novelties (can be seen even during training). Tables 2-6 show clear quantitative results where the method is able to almost instantaneously detect a novelty in most cases. \n\nThat said, while the empirical results are strong, some of the theoretical claims—particularly Propositions 3.2 —rest on assumptions about the behavior of KL divergence that may not always hold in practice (discussed later), such as its non-metric nature and dependence on variational approximations. This might not be a major flaw but should be talked about in the paper in my opinion. Nonetheless, the main claims regarding the method’s novelty detection performance are well-supported by the presented evidence across a variety of setups.", "methods_and_evaluation_criteria": "Yes, the paper uses a sufficient set of baselines, datasets, and methods for the problem at hand. The baselines are good, covering a broad spectrum of novelty detection strategies. The evaluation spans three well-established benchmarks—MiniGrid, DeepMind Control Suite, and Atari—which together capture a wide range of perceptual and dynamic challenges. The proposed method is integrated naturally into DreamerV2, and its performance is compared rigorously across these environments, demonstrating the method’s generality and robustness. They also test the effectiveness of their approach on other world model approaches and the algorithm works well (PP-Mare variant) on diffusion based but struggles on Transformer based which was also an observation by a previous work.  \n\nAn important issue here is: **authors mention an Appendix H but there is none present**, that possibly discussed transformer based world models and KL bound for novelty detection. I will recommend authors to add that for more context.", "theoretical_claims": "The paper includes several theoretical claims in the form of Propositions 3.1 and 3.2, which are used to justify bounds for novelty detection based on KL divergence between posterior and prior latent state distributions. The propositions are intuitive and well-motivated, but some rely on assumptions that are not fully formalized. \n\nWhile Proposition 3.2 presents an intuitive inequality that resembles a triangle inequality over KL divergences, it is important to note that **KL divergence is not a true metric and does not satisfy the triangle inequality** in general. The bound in Proposition 3.2 compares the posterior to three different priors—one conditioned on memory, one on the current observation, and one on neither—and assumes that if the posterior is close to each of the simplified priors individually, then it should also be close to the full prior. However, this assumption is structurally similar to triangle inequality reasoning, which is not formally valid in KL space due to its asymmetry and lack of metric properties. As a result, the inequality may not hold even when all distributions involved are close in distributional terms. This weakens the theoretical guarantee of the proposition and suggests that violations of the bound might occur not only due to novelty, but also due to the non-metric behavior of KL divergence, particularly in high-dimensional or undertrained latent spaces. However, empirically, I don't think this is a big issue but I will still recommend authors to discuss a bit more on this, especially in the derivations.", "experimental_designs_or_analyses": "The experimental design and analysis appear generally sound and well-aligned with the paper’s goals. The authors evaluate their method across three diverse benchmarks—MiniGrid, DMC, and Atari—with injected observation and transition novelties that simulate realistic distribution shifts. The novelty is introduced mid-trajectory, allowing for precise analysis of detection timing. The use of per-timestep KL divergence as a novelty score, along with aggregated trajectory-level evaluations (e.g., AUC, false alarm rate, detection delay), provides a decent measure of novelty detection. The comparison against multiple baselines is thorough ensuring a fair and comprehensive evaluation.  Overall, the key metrics and qualitative analyses convincingly support the claims, and no major flaws are apparent in the experimental methodology.", "relation_to_broader_scientific_literature": "The paper builds directly on the growing body of work in model-based reinforcement learning, particularly approaches using latent world models such as DreamerV2, and extends these frameworks to address the underexplored problem of unsupervised novelty detection. While prior works like Dreamer and PlaNet focus on learning compact latent dynamics for control and planning, they do not consider the challenge of identifying distributional shifts during deployment. This paper shows that the KL divergence between the posterior and prior latent distributions—already computed internally by such models—can be leveraged as an effective and lightweight novelty signal. Importantly, unlike reconstruction- or prediction-based baselines, this method operates entirely in the latent space, making it more stable and closely aligned with the agent's internal belief state.\n\nRelative to traditional novelty detection methods such as autoencoders, latent prediction models, or uncertainty-based approaches like RIQN, the proposed method introduces a novel theoretical framework based on KL divergence bounds derived from priors conditioned on partial information (e.g., without memory or observation). These bounds are used to derive threshold-free detection criteria rooted in variational inference, inspired by dropout-based generalization tests. A key advantage of this approach is that it requires **no additional tuning or threshold hyperparameters**, making it both more interpretable and easier to deploy in practice.", "essential_references_not_discussed": "None to the best of my knowledge.", "other_strengths_and_weaknesses": "I don't have many concerns barring a suggestion to include some theoretical clarifications which I stated earlier as well, I think the paper is well written and evaluations are exhaustive. The need to not tune per environment makes the method extremely versatile. The reproducibility is as good as it can be.\n\nHowever, one important weakness I think is: the approach's results will be as good as the world model themselves, i.e., their applicability is limited in cases where interactions with the environment is expensive or potentially dangerous.", "other_comments_or_suggestions": "I think Appendix H is missing (or that is meant to be linked with appendix B, IRIS). \n\nI think the discussion on exploration should be included in the main paper and \"Detection Score Trend Over Time\" + Figure 2 can be shifted to appendix. An under-explored policy directly impacts the reliability method. Additionally, as future work, authors may want to incorporate uncertainty-aware priors to distinguish novelty from ignorance to tackle insufficient exploration (though I understand, that possibly means introducing changes to the world model, which is not the point of the paper.)", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o3Nj4rXmYN", "reviewer": "Reviewer_oBkv", "summary": "The paper seeks to determine when there is novelty in an environment by using world models, and particularly when there is high prediction error with such world models. Such an approach aligns strongly with existing neuroscience work. The paper demonstrates with strong results their approach, exceeding the performance of most existing approaches.\n\n## Update after rebuttal\nThe author's successfully responded to my questions and the questions of the other reviewers. I feel confident in my score of a 4 and in the broader merit of the paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review, hopefully we can interpret your questions and clarify as well as possible:\n* Why is the ground truth for Table 1 so noisy?\n   * The ground truth component of the table shows the true observation given to the agent, specifically the noisy observation is a sample from the noisy environment that occurs when we inject the novelty ‘noise high’ in the humanoid setting. \n* I don't understand this point \"in nominal conditions, any divergence should always be smaller than the divergence of the predicted latent world state computed with the initial hidden state instead of the current initial hidden state; the latter prediction should be increasingly inaccurate.\" Can the authors elaborate on this?\n   * Thanks for pointing out that this statement might be confusing to the reader, with that statement we are attempting to describe our observation in Figure 1. In response we have rephrased the statement to be: “to develop the bound, we observe that, under nominal conditions, any divergence should be smaller than that of the predicted latent world state computed with the initial hidden state, as the latter prediction becomes increasingly inaccurate.” \n* The authors state \"We introduce a novel technique for novelty detection without the need to manually specify thresholds, and without the need for additional augmented data.\" but then they later state \"Our technique calculates a novelty threshold bound\" so there seems to still be a reliance on the novelty threshold bound? I'm unsure clarification would be helpful.\n   * No problem. To clarify, we are stating that the user does not have to be mindful of explicitly/manually setting a measurement of uncertainty $\\lambda$ beforehand to trigger the detection of novelty. Our technique will simultaneously handle the bound (as well as the detection), while minimizing false positives.\n* The authors used an older version of dreamer (V2 instead of V3) which begs the question--does the proposed approach work for newer MBRL based approaches? … Why did they use DreamerV2 instead of the newer DreamerV3?\n   * We consider our work on the V2 algorithm to be mainly based on the RSSM component of the world model, and although the major contributions of going from the Tensorflow V2 to the JAX V3 introduced faster optimizations, this move would have made the experimental design more complex despite both V2 and V3 being centered around the core RSSM framework. Therefore, for newer MBRL approaches, we believed it was more impactful to take time to test the newer Diamond (2024) [1], and IRIS (2023) [2] world models, which unlike DreamerV3 or V2, do not use the RSSM framework.\n* The proposed approach is not compatible with raw inputs (i.e. when using diffusion) or non probabilistic latents. I also see this as a small weakness. \n   * For diffusion based approaches, we found that the PP-MARE method (see Table 7, Section 4 PP-MARE, and Appendix B paragraph Diamond) could successfully be used directly on raw inputs. As for non-probabilistic latents, (possibly used if it is expected that the environment is deterministic and fully observable), then our setting of being in a stochastic partially observable setting no longer holds. However, we hypothesize that a subtle extension could be that the KL loss could be substituted with a binary operator (or suitable deterministic metric) over the posterior and prior representations, when conditioned on different information. \n* While the approach is simple, that also means the contribution is rather limited. I also see this as a small weakness\n   * Thanks for the feedback. We purposely crafted the main method to be simple, but to the best of our knowledge, we are the first to test novelty detection in both Atari and 3D-DMC environments entirely in the image space (see footnote 2). We also introduce PP-MARE for those who are interested in working outside of the latent space, and are okay with specifically defining their threshold. \n```\n[1] Alonso, E., Jelley, A., Micheli, V., Kanervisto, A., Storkey, A., Pearce, T., and Fleuret, F. Diffusion for world modeling: Visual details matter in atari. 2024.\n[2] Micheli, V., Alonso, E., and Fleuret, F. Transformers are sample-efficient world models. In The Eleventh International Conference on Learning Representations, 2023.\n```", "claims_and_evidence": "The claims in the submission are very strongly supported by good evidence. The authors used strong baselines and benchmarks and I did not find any issues with any claims/experiments.", "methods_and_evaluation_criteria": "The benchmark datasets and evaluation criteria used was standard and makes sense for the problem.", "theoretical_claims": "I checked the correctness of all proofs in the main paper.", "experimental_designs_or_analyses": "I checked the validity of all experiments in the main paper and did not find any issues.", "relation_to_broader_scientific_literature": "The paper is broadly related to world models, Reinforcement Learning and particularly Model Based reinforcement learning, as well as novelty detection/uncertainty.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "## Strengths\n- The approach is very simple and elegant, aligning with neuroscience ideals of uncertainty/novelty detection guided by high prediction errors\n- The results strongly support the claims of the authors, and the metric has several potential downstream use cases for world modeling, including determining whether the world model is properly using history (Figure 1) as well as whether the world model has converged or not.\n\n## Weaknesses\n- While the approach is simple, that also means the contribution is rather limited. I also see this as a small weakness\n- The authors used an older version of dreamer (V2 instead of V3) which begs the question--does the proposed approach work for newer MBRL based approaches?\n- The proposed approach is not compatible with raw inputs (i.e. when using diffusion) or non probabilistic latents. I also see this as a small weakness", "other_comments_or_suggestions": "- The authors state \"We introduce a novel technique for novelty detection without the need to manually specify thresholds, and without the need for additional augmented data.\" but then they later state \"Our technique calculates a novelty threshold bound\" so there seems to still be a reliance on the novelty threshold bound? I'm unsure clarification would be helpful.", "questions_for_authors": "- Why is the ground truth for Table 1 so noisy? \n- I don't understand this point \"in nominal conditions, any divergence should always be smaller than the divergence of the predicted latent world state computed with the initial hidden state instead of the current initial hidden state; the latter prediction should be increasingly inaccurate.\" Can the authors elaborate on this?\n- Why did they use DreamerV2 instead of the newer DreamerV3?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gmJAkM3eUp", "reviewer": "Reviewer_izpy", "summary": "This work proposes a novelty detection technique in model-based reinforcement learning called world models. It refers to sudden changes in the world model's visual properties or dynamic system as novelties. They use KL divergence between latent predictions with and without observable ground truth to design the novelty detection bound. The proposed technique is not dependent on additional hyper-parameters, and experiments are carried out on NovGrid, HackAtari, and RealWorldRL Suite.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their suggestions on improving our work. In response, we outline provisional revisions below:\n \nFirst, we note that we have fixed each of the typos and syntax errors that were identified.\nSpecifically, we have:\n* Moved the period to after the citation.\n* Rephrased Ln 92 to 96 - col 1 to: \n\tTo develop the bound, we observe that, under nominal conditions, any divergence should be smaller than that of the predicted latent world state computed with the initial hidden state, as the latter prediction becomes increasingly inaccurate.\n* Rephrased Ln 215 - clo 2 to: (blue differences collapse to orange).\n\nRegarding the following statement:\n```\nReferencing Srivastava et al. 2014 for Proposition 3.2 is not enough, a little bit more explanation is needed to make the paper self-sufficient.\n```\nWe first reword Proposition 3.2 to align with terminology used by other works [2]:\n*If the cross entropy score comparison becomes **nonnegative** when introducing the vector $x_t$, then 4 defines a decision boundary in the latent space over the measure of robustness to partial destruction of the input (Vincent\net al., 2008; Srivastava et al., 2014), i.e.:*\n\n$KL[p_{\\phi}(z_t|h_t,x_t)||p_{\\phi}(z_t|h_t)] +\nKL[p_{\\phi}(z_t|h_t,x_t)||p_{\\phi}(z_t|h_0,x_t)] \n\\leq\\\\ \nKL[p_{\\phi}(z_t|h_t,x_t)||p_{\\phi}(z_t|h_0)]$.\n\nand have extended the derivations section in the appendix to include a more explicit tone:\n\nLet the full posterior $p_{\\phi}(z_t|x_t,h_t)$ be the most informed distribution and the desired distribution with support over the latent space  $\\mathrm{Z}$ at any given $t$, and let $f_\\theta$ map to the representation of a distribution with the support of $\\mathrm{Z}$ as well.\n\nUtilizing notation similar to that in Vincent et al. (2008), denote $(h_t,x_t)$ as the clean input $x$ (not to be confused with $x_t$), and define three variants of $x'$: $\\left\\lbrace x_{(-h_t)}', x_{(-x_t)}', x_{(-h_t,-x_t)}' \\right\\rbrace$, which denote the removal of features $(h_t)$, $(x_t)$, and $(h_t,x_t)$ from the clean input $x$, respectively.\n\nSince the loss is at the latent level, we measure the robustness to partial destruction criterion of a single input $x$ for a desired distribution as: $L(p_{\\phi}(z_t|h_t,x_t), f_{\\theta}(x'))$\n\nTherefore, if $L$ is chosen to be the KL divergence, and $x'$ becomes explicit, then we can derive Proposition 3.2 as:\n\n$L(p_{\\phi}(z_t|h_t,x_t), p_{\\phi}(z_t|h_0)) \\geq L(p_{\\phi}(z_t|h_t,x_t), p_{\\phi}(z_t|h_t)) + L(p_{\\phi}(z_t|h_t,x_t), p_{\\phi}(z_t|h_0,x_t)) =$\n$$KL(p_{\\phi}(z_t|h_t,x_t) \\parallel p_{\\phi}(z_t|h_0)) \\geq KL(p_{\\phi}(z_t|h_t,x_t) \\parallel p_{\\phi}(z_t|h_t)) + KL(p_{\\phi}(z_t|h_t,x_t) \\parallel p_{\\phi}(z_t|h_0,x_t))$$\n\n\nwhere $p_{\\phi}$ corresponds to the distribution modeled by $f_{\\theta}$ given some $x^{'}$, and $h_0$ represents the state of $h$ when no conditioning information is available.\n\n```\nThe quality of the paper can be improved further if authors come up with some guarantee on the bound.\n```\nFinally as suggested, we agree that we should attempt to give the reader intuition surrounding our method by extending the appendix with the following theoretical guarantee which shows that for a theoretical perfectly trained world model  (i.e: $z_t \\perp x_t \\mid h_t$ and there exists a set of parameters $\\phi^* $ such that:\n$  p_\\phi(\\cdot|\\cdot) = p^*(\\cdot|\\cdot)$\nwhere $p^*( \\cdot | \\cdot)$ denotes the well defined true conditional distribution) then EQ 4 holds if:\n$\nE_{p_\\phi(z_t \\mid h_t,x_t)}\\left[\\log \\frac{p_\\phi(z_t \\mid h_0,x_t)}{p_\\phi(z_t \\mid h_0)}\\right] \\ge 0.\n$\nWhich can be interpreted as holding if the Expected Information Gain (EIG) of $x_t$ is nonnegative under the distribution of $p_\\phi(z_t \\mid h_t,x_t)$.\n\nIdeally, although as an informal side note, we would like to intuitively interpret $KL[p(z_t |h_t x_t )|| p(z)]$ as proportional to $I(z_t;h_t,x_t)$ (which is often a substitution in practice [1]). Where \n* $KL[p(z_t |h_t, x_t )|| p(z_t)]$  is substituted by $I(z_t;h_t,x_t) = H(z_t) - H(z_t|h_t,x_t)$\n* $KL[p(z_t |h_t, x_t )|| p(z_t | h_t)]$ is substituted by $I(z_t; x_t | h_t) = H(z_t|h_t) - H(z_t|h_t,x_t)$\n* $KL[p(z_t |h_t, x_t) || p(z_t | x_t)]$ is substituted by $I(z_t;h_t | x_t) = H(z_t|x_t) - H(z_t|h_t,x_t)$\n\nTherefore, if showing (EQ 4) was somehow related to showing:\n$H(z_t) - H(z_t|h_t,x_t) \\geq H(z_t|h_t)- H(z_t|h_t,x_t) + H(z_t|x_t)- H(z_t|h_t,x_t)$\n\nThen simplification would show:\n$H(z_t) - H(z_t|h_t) \\geq H(z_t|x_t) - H(z_t|h_t,x_t) \\implies$\n\n$I(z_t;h_t) \\geq I(z_t;h_t|x_t) (*)$\nWhere a potential key result would lie: EQ 4 holds if knowing X reduces the mutual information between Z and H.\n```\n[1] Zhao, S., Song, J., & Ermon, S. (2019). InfoVAE: Balancing Learning and Inference in Variational Autoencoders. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01), 5885-5892.\n[2] Vincent, P., Larochelle, H., Bengio, Y., and Manzagol, Extracting and composing robust features with denoising autoencoders. ICML 2008\n```", "claims_and_evidence": "Mostly, claims are supported by experimental results i.e.,\n\n1. Even though Proposition 3.1 is supported by explanation and visualization of novelty detection bound in Figure 1. The quality of the paper can be improved further if authors come up with some guarantee on the bound.\n\n2. Referencing Srivastava et al. 2014 for Proposition 3.2 is not enough, a little bit more explanation is needed to make the paper self-sufficient.", "methods_and_evaluation_criteria": "The proposed novelty detection bound has been evaluated on NovGrid, HackAtari, and RealWorldRL Suite, these environments are suitable for the proposed method.", "theoretical_claims": "The claims look ok, apart from the issues listed in the Claims And Evidence section.", "experimental_designs_or_analyses": "The experimental designs for novelty detection of visual properties and dynamic systems are ok. Authors use the false positives rate, average delay error, AUC scores, and inference run-time speedup for evaluation of proposed novelty detection technique.", "relation_to_broader_scientific_literature": "Most of the prior novelty detection methods need environment specific parameter turning. The proposed method doesn’t need the environment specific parameters turning, which is a good advantage. Further, novelty detection is under explored in the model-based RL settings. Hence, this work is relevant to the community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The main strength of the paper is that the proposed technique doesn’t require environment specific parameters/threshold tuning.\n\nThe main weakness, as noted above, is that the authors don’t provide rigorous proof of the novelty bound.", "other_comments_or_suggestions": "Typos or errors,\n1. Ln 112 - col 1: signal.(Greenberg & Mannor, 2020) This\n2. Ln 92 to 96 - col 1: complex sentence to get the meaning, so rephrase or at least give ref to eq 3.\n3. Ln  215 - clo 2: (divergence of blue collapses to red)? or blue / differences collapse to orange?", "questions_for_authors": "Please see above, including the Claims and Evidence section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a theoretically motivated approach for novelty detection in model-based reinforcement learning by utilizing the KL divergence between latent posterior and prior distributions within world models. Across all three reviewers, there is agreement that the method is simple, elegant, and strongly supported by empirical evidence across diverse benchmarks. Theoretical propositions are appreciated for their intuition, though some reviewers noted that assumptions about KL divergence’s properties may weaken the formal guarantees--an issue the authors addressed thoughtfully in the rebuttal. The approach’s key strengths include threshold free deployment, strong detection performance, and a clear link to internal model uncertainty. While some concerns were raised about limited applicability to newer architectures (like DreamerV3 or non-probabilistic latents), these were adequately acknowledged and contextualized in the response. Overall, the paper received solid endorsements, with all reviewers recommending acceptance. Please consider adding impact statement in final version.
Decision: Accept (spotlight poster)