Variational Learned Priors for Intrinsically Motivated Reinforcement Learning

Authors: Jessica Nicholson, Joseph S Goodier, Akshil Patel, Özgür Şimşek
Primary area: reinforcement learning
Keywords: reinforcement, learning, intrinsic, motivation, exploration, curiosity

Abstract:
Efficient exploration is a fundamental challenge in reinforcement learning, especially in environments with sparse rewards. Intrinsic motivation can improve exploration efficiency by rewarding agents for encountering novel states. In this work, we propose a method called Variation Learned Priors for intrinsic motivation that estimates state novelty through variational state encoding. Specifically, novelty is measured using the Kullback-Leibler divergence between a Variational Autoencoder's learned prior and posterior distributions. When tested across various domains, our approach improves the latent space quality of the Variational Autoencoder, leading to increased exploration efficiency and better task performance for the reinforcement learning agent.

Decision: reject
Reviewer scores: 5, 5, 6, 5

Reviews and discussion:
<thread>
creation date:
1729028177172

thread_type:
official_review

<thread_object>
creator: reviewer_zxck

<content>
summary:
This paper investigates the problem of prior/aggregated-posterior mismatch in the setting of VAE-based intrinsic motivation exploration. The authors argue that when measuring state novelties, using KL divergence between the posterior of a state and the prior can be misguided when the prior does not faithfully represent the aggregated posterior. The authors then proposed VaLP, which utilizes learned flexible priors, to mitigate this problem. Through extensive experiments, the authors showed that variants of VaLP (with differently parametrized priors) can outperform other exploration methods.

strengths:
1. Extensive experiments in various domains, including static datasets (MNIST/FashionMNIST), toy RL environments such as DeepSea and Minigrid, and the more popular environments such as MuJoCo and Atari.

2. The experiments with toy RL environments are especially helpful for visualizing different behaviors

weaknesses:
1. One challenge of exploration in RL is that data distribution can shift over time as the agent’s policy continues to improve, and I think it’s important to see whether the misalignment problem would also appear for VaLP. Currently, the misalignment figures (Fig1 and Fig2) both focus on fixed datasets, and it is not clear whether VaLP can solve the misalignment problem in the RL setting. One can test this by, for example, repeating the experiments in Fig1 every T steps using the states in the replay buffer.

2. The MuJoCo experiments are not really informative. Most of the algorithms seem to achieve similar performance, making it difficult to judge the effectiveness of the proposed method. Perhaps exploration is not the main bottleneck for these environments. Also, the TD3 baseline seems to underperform other open-source baselines (e.g., CleanRL [2]). For example, CleanRL’s TD3 can achieve 9583.22 ± 126.09 on HalfCheetah at 1M steps, but the scores reported by the authors are all below 8000.

3. The Atari experiments are also problematic. For example, [3] reported that random uniform policies can achieve average scores of 1.70 on Breakout, 173.0 on Gravitar, and 24.90 on Private Eye. Yet, the authors reported scores close to or worse than the random baselines. Ultimately, I think comparing exploration methods with such a tight training budget (100k interactions) is not really meaningful. As the hard exploration problems often take millions if not billions of frames to solve even for SOTA algorithms (see [4] for example).


[1] Fujimoto, Scott, Herke Hoof, and David Meger. "Addressing function approximation error in actor-critic methods." International conference on machine learning. PMLR, 2018.

[2] https://docs.cleanrl.dev/rl-algorithms/td3/#experiment-results

[3] Van Hasselt, Hado, Arthur Guez, and David Silver. "Deep reinforcement learning with double q-learning." Proceedings of the AAAI conference on artificial intelligence. Vol. 30. No. 1. 2016.

[4] Saade, Alaa, et al. "Unlocking the power of representations in long-term novelty-based exploration." arXiv preprint arXiv:2305.01521 (2023).

questions:
1. How does the misalignment problem evolve over training steps in the RL setting?
2. Can the authors explain the performance of VaLP on MuJoCo and Atari?

Minor suggestions:
1. On line 95, the expectation is missing for the definition of expected return.
2. For the Deep Sea experiments, I think it would make sense to also report the training curves (cumulative rewards) to demonstrate the exploration-exploitation tradeoff, as the environment is not purely explorative (the extrinsic rewards are not zero).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730342118475

thread_type:
official_review

<thread_object>
creator: reviewer_gfff

<content>
summary:
The paper proposes Variational Learned Priors (VaLP) for intrinsic motivation in reinforcement learning. VaLP improves exploration by using the Kullback-Leibler divergence between a learned prior and posterior distributions in a Variational Autoencoder. Unlike standard fixed priors, the learned priors dynamically adjust to the agent's experiences, creating a more accurate latent space that enhances novelty estimation and exploration. The authors introduce four types of priors—Mixture of Gaussians, Generative Topographic Mapping, VampPrior, and Flow-based prior—and validate VaLP across environments, demonstrating improved exploration and task performance over traditional intrinsic motivation methods in both discrete (Atari) and continuous (MuJoCo) settings.

strengths:
1. Interesting qualitative results are presented, demonstrating the effectiveness of the proposed method.
2. Hyperparameters and detailed experiment settings are provided.

weaknesses:
1. Some parts of the paper are hard to understand: the authors spend a lot of words explaining Figure 1 in the introduction section but it's still confusing to me. For example, what is latent space coverage? The authors link it to "how well the prior aligns with the encoded states" but it's still quite abstract - more intuitive explanations will help before talking about prior, posterior and latent space. MoG, GTM etc. are not introduced until section 4 which may make the readers wonder if these are previously proposed methods and the paper proposed a new method that outperforms these. In Section 4, more explanations of the fundamental difference between the four models will also make the paper more readable.

2. As the proposed intrinsic reward is just investigating different existing ways for modeling priors I would expect a thorough experiment section. The presented qualitative results are impressive, but I would like to see the quantitative results on more hard-exploration domains. A typical example is Ant-maze, which has been investigated by many prior exploration papers. Experiments on Ant-maze are valuable also because the presented results on mujoco tasks seem to be mixed in the paper.

questions:
For Figure 3 and 9, what are the standard deviations for the results? Only presenting the means may make the conclusions from these results highly biased.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490233537

thread_type:
official_review

<thread_object>
creator: reviewer_xtsg

<content>
summary:
The paper revisits an earlier intrinsic motivation algorithm in reinforcement learning (RL), identifies a limitation of this algorithm and proposes and tests a solution for this limitation. 

In particular, the paper considers an algorithm where a variational autoencoder (VAE) of the observations of an agent is learned in parallel with its policy. The reason to learn the VAE is to use the Kullback-Leibler divergence term in its ELBO loss as an intrinsic reward for the agent. The idea is that if a novel state is observed, the associated latent representations will not be captured by the latent prior distribution, that is, the divergence between the sample posterior and the prior will be large, and so the agent will be rewarded to visit again the atypical state. In this manner, the KL divergence rewards the novelty of a state. The limitation that the paper points out is that the standard VAE has a fixed prior that does not reflect the knowledge of the agent and this affects the quality of its sense of novelty. 

Following an existing solution in the more general setting of generative models, the paper proposes to update the prior with an approximation of the average of the sample posterior distributions. For this, four different types of generative models are considered, including mixtures of Gaussians and flow-based density estimators.  

The paper proceeds to test the proposed solution. First, as a sanity check, it shows how good are the generative models to fit an average of sample posterior distributions in a non-RL task in contrast with the fixed normal prior. After showing that the proposed models are useful, it provides evidence of the improvement obtained in the first state visitation or state space coverage in three exploration tasks. Finally, results in MuJoCo and Atari 100K are provided to show that the extrinsic reward can be maximized as a result of improved exploration.

strengths:
Originality:
- The paper revisits an algorithm with potential but which had no major impact nor visibility. In this sense, it is original to identify the relevance of the algorithm and to propose a way to make it fulfill its potential. 

Quality:
- The paper is thorough in its empirical evaluation, including not so popular tabular environments and challenging and popular deep RL environments. It also includes multiple intrinsic reward algorithms as baselines, showing that the proposed solution works across environments and that it is competitive with existing techniques. 
- The paper provides strong evidence to conclude that using a fixed prior to assess state novelty with VAEs is inappropriate.
- The paper includes relevant references that allows the reader to assess its novelty.

Clarity:
- The paper is well organized. The earlier work, its limitation, and the proposed solution are well motivated and follow a logical order.

Significance:
- The paper provides some evidence for the final algorithm, VaLP, being competitive with other state-of-the-art exploration algorithms like RND.

weaknesses:
I think it is fair to say that the paper is very incremental. It focuses on a limitation of VAEs that has been identified and solved already, as the Background section explains. In this sense, most of the work should lie on proving that the already existing solution has a significant impact in the context of intrinsic motivation. While the tabular experiments are conclusive regarding the inappropriateness of using fixed priors, I consider that the paper overclaims in the interpretation of the results and that the deep RL experiments are insufficient.

Why is the paper overclaiming?
- In Section 6.3, it is stated that VaLP$ _{\text{Flow}}$ covers a "wide range of states" while RND "trails behind the learned priors VaLP$ _{\text{Flow}}$". Without seeing the plot, some reader could infer that RND is a worse exploration algorithm. However, looking at Figure 6, the *only* method that leads to explore more than RND is VaLP$ _{\text{Flow}}$, it does only cover around 70% of the state space, and it is hard to tell if much can be inferred from visiting 6 more states after 500000 steps in a single seed.

- In Section 6.3.1, it is stated that "VaLP$ _{\text{Flow}}$ demonstrates strong exploration across both spirals, avoiding detachment". As per my understanding of the phenomena of detachment, this conclusion is confusing state space coverage with even visitation of states throughout an agents life. Figure 7 has no conclusive evidence regarding how even is exploration and, if anything, results seem to suggest that VaLP$ _{\text{Flow}}$ concentrates in one of the spirals in the last steps.

- In Section 6.4, is is stated that VaLP methods "demonstrate significant and reliable improvements over the Standard prior and baseline methods", that "in Hopper, VaLP$ _{\text{Flow}}$ shows dominant performance throughout training" and that "in Ant-v4, VaLP$ _{\text{Flow}}$ exhibits the strongest performance, with a final mean reward approximately 1000 points higher than (...) baseline methods such as RND and DRND". Again, from these claims the reader might conclude that VaLP$ _{\text{Flow}}$ is superior to any other exploration method. However, looking at Figure 8, this is very unclear:

    1. There are only 5 seeds per environment, which makes it difficult to make any strong statistical claim. 
    2. The shaded regions considering the standard error of the mean greatly overlap and it is difficult to tell whether a curve is inside or outside of a shaded region for most tasks. 
    3. Performance in Hopper is only 'dominant' after 500K steps.
    4. DRND arguably has a similar final mean as VaLP$ _{\text{Flow}}$ in Ant-v4 and any difference certainly lies under the statistical error.

- Similarly to previous cases, in Section 6.4.1, the VaMP models are described by qualifiers like "achieves the highest rewards", "leads", "outperform", and "top performer". However, I just see the same issues as before and new ones. Not only 5 seeds were used again, but there is no comment on statistical significance. In addition, it is not clear what kind of scale Figure 9 is using. If the original rewards are being used, most scores are significatively low, considering the average Human scores reported in previous papers like the following:\
    a. Kaiser Ł. et al., Model-Based Reinforcement Learning for Atari, ICLR, 2020.\
    b. Schwarzer M. et al., Bigger, Better, Faster: Human-level Atari with human-level efficiency, ICML, 2023.\
Added to this, the paper introducing RND shows that solving tasks to an acceptable level like Gravitar requires approximately 500K steps. This leads me to conclude that the results shown might correspond to noise at the beginning of a learning curve and that the obtained behavior in the games is not satisfactory for any algorithm in most of the games being considered.

- Finally, in the Discussion and Future Work section it is stated that "this adaptability enables VaLP$ _{\text{Flow}}$ to excel across diverse tasks, leading to superior exploration and policy optimization in environments like MuJoCo and Atari". Considering the previous points, saying that the method excels and is superior is a big stretch.

Proposed actionables:
1. For both MiniGrid (MultiRoomN6 and Double Spiral) and Atari I suggest running longer experiments. The fact that novel states in the edges of what has been visited are being discovered in the last steps in MiniGrid tells me that none of the algorithms have finished exploring. In Atari, 100K steps might be insufficient to discover any useful behavior, as suggested by the low scores and the learning curves reported in the RND paper.
2. Include tables with final performances (mean and standard deviation of the mean) for MuJoCo and Atari to be able to more clearly assess them.
3. For the MuJoCo learning curves, shift the VaLP curves by the amount of pretrain exploratory steps needed to pretrain the VAE. Also add this quantity somewhere else.
4. If my understanding is right and there is only 1 seed for MiniGrid, run for more seeds.
5. Add details about hyperparameter selection. If no hyperparameter selection was performed for the baselines, the results of the baselines could be unfair.
6. For MiniGrid, include coverage metrics to assess more definitively how significant is the difference between RND and VaLP$ _{\text{Flow}}$.
7. More seeds would be better for MuJoCo, as the differences do not seem to be significant. 
8. Include videos or images of the behavior of agents in Atari that allow determining if something useful is being learned.
9. Add some metric of "evenness" for the MiniGrid Double Spiral experiment.

questions:
Questions:
- In the Atari 100K results, are the scores normalized against the average human scores? My guess is no since there are negative values.
- In the first-to-visit plots, why not just add a different color to not visited states? The value of 1000 seems arbitrary and might affect the conclusions about the exploration abilities of each model.
- Why does the scale in Figure 7 show values not present in the subplots?
- Why is it relevant to show the non-RL results? The fact that these models can approximate distributions has already been extensively studied, including the papers mentioned in the Background.
- Can you explain what is the difference between the results 6.2, 6.3, and 6.3.1? To me, the only difference is that a different tabular environment is considered. Given that the reward for DeepSweep is so sparse, I am not convinced that there is much difference with the other two cases either.
- In the Introduction, what does "latent space coverage" mean exactly? What is the formula to calculate it?
- In Line 518, in Page 10, why are non-linear dynamics mentioned? How are dynamics related with anything else in the manuscript?
- In the last paragraph, why would the learned priors be transferable? Also, what does it mean to identify the most effective learnable prior? This sounds just like a matter of using better generative models and following hyperparameter tuning.

Additional minor suggestions:
- Improve the explanation of the notation in the VAE subsection of the Background. In particular, there is unexplained or inconsistent notation.
- Unify the color palettes for Figures 4, 6, and 7.
- The last line in Page 1, "this weakens the intrinsic reward, which can result in inefficient exploration.", should start with uppercase.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707783026

thread_type:
official_review

<thread_object>
creator: reviewer_wejj

<content>
summary:
This paper introduces an approach to provide intrinsic rewards based on state novelty for exploration in reinforcement learning. Previous work had proposed using the KL divergence between the posterior encoding of the state (learned through a VAE with ELBO) and a fixed prior (standard Gaussian). This paper identifies the misalignment of the fixed prior with encoded states as a crucial limitation of the previous work. It suggests a solution by using VAEs with a learnable prior. The paper studies many approaches to represent and learn more expressive priors, enabling better exploration than using a standard Gaussian prior VAE.

strengths:
**S1.** The idea of Variation Learned Priors (VaLP) is simple and motivated well through latent space visualizations in simple settings. Compared to many complex ways to generate intrinsic rewards, this approach should be simple to implement and integrate with RL algorithms.

**S2.** The paper does a great job of comparing various approaches (flow-based, mixture of Gaussians, VAMP, GTM) to learning expressive priors with VAEs. Experiments clearly show the learned prior variants outperforming the bonus from VAEs with the standard Gaussian prior.

**S3.** The paper is well-written, and the key concepts are clearly explained and flow nicely.

weaknesses:
**W1.** While the experiments clearly show the benefit of VaLP over using a VAE with the standard/fixed prior, it is unclear if this approach outperforms ICM, RND, and DRND as claimed in Lines 500 and 501. 

In the MiniGrid experiment, all approaches seem to have similar coverage (except VaLP$_{Flow}$, which does a bit better), and none are near the final room. Could these experiments be run for longer (more interactions) to gauge how the approaches perform further? 

In the Mujoco experiments, many intrinsic rewards do similarly well in HalfCheetah and Walker (2/4 environments). The caption for Figure 8 mentions, “in Ant-v4, $VaLP_{Flow}$ exhibits the strongest performance, with a final mean reward approximately 1000 points higher than the Standard prior method and baseline methods such as RND and DRND.” however, DRND performance seems to be quite close to $VaLP_{Flow}$. 

It is also unclear how informative the Atari 100K experiments are with the chosen environments. Some of the environments (Gravitar, Pitfall, and PrivateEye) are probably too challenging in this setting for all considered approaches, and performance at 100K with DQN with intrinsic rewards may not convey how the approaches truly explore and will fare in the longer run.

On a separate note (but related to the Minigrid comment), it would also be helpful to see how the VaLP agents perform in the detachment spiral experiment when it is run for longer, as no agent is close to finishing the spirals yet. 

**W2.** The paper motivates using a learned prior through the observation that a standard Gaussian prior might lead to over-regularization. With that in mind, it would be important to know if the added complexity of VaLP can be avoided by using a larger latent dimension or a more expressive decoder architecture with standard prior VAEs. Are there any experiments/arguments showing that the above approach is insufficient? Or would using learned priors continue to help even with a larger capacity?

**W3.** While not strictly necessary, it would have helped to include pseudo-count density-modeling bonuses [1, 2] in the empirical comparison due to the similarity of using a generative model of observations for deriving intrinsic rewards.

Overall, I appreciate the simplicity and clarity of the ideas presented in the paper. I remain open to increasing the score should the weaknesses and questions be adequately addressed/clarified.

### Minor comments/questions

- Figure 8, Hopper-v4: It has VaLP variants as a dashed line, but it should be solid.

- I guess “decision stages” refers to the number of interactions (or steps) with the environment. Could you confirm that? I wasn’t familiar with that terminology, so clarifying that in the paper may be helpful.

- Are the detachment spiral results in Figure 7 presented for multiple seeds of each VaLP variant? I couldn’t find if/where that information was provided in the paper. Similarly, while the paper mentions that MiniGrid coverage visualizations are computed with multiple seeds, I couldn’t find how many runs were used to generate the results.

### References

[1] Bellemare, M., Srinivasan, S., Ostrovski, G., Schaul, T., Saxton, D., & Munos, R. (2016). Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29.

[2] Ostrovski, G., Bellemare, M. G., Oord, A., & Munos, R. (2017, July). Count-based exploration with neural density models. In International conference on machine learning (pp. 2721-2730). PMLR.

questions:
Included in the Weaknesss section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732099015914

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment to Everyone

comment:
Dear Reviewers,

We sincerely thank the reviewers for their constructive feedback and thoughtful evaluations of our submission.  We are pleased that our contributions have been recognized for their novelty, presentation, and soundness.  We are grateful for the reviewers' recognition of several key strengths in our work. These include the simplicity and clear motivation behind VaLP, supported by strong evidence that learned priors (e.g., Flow, Vamp, GTM, and MoG) effectively address the limitations of fixed Gaussian priors. The thoroughness of our empirical evaluation has also been highlighted, spanning diverse environments such as static datasets (MNIST), tabular toy RL environments (DeepSea), and deep RL benchmarks like MuJoCo and Atari, complemented by visualizations that showcase different agent behaviors. Additionally, reviewers acknowledged the use of multiple intrinsic reward baselines that demonstrate VaLP’s competitiveness with state-of-the-art exploration methods, as well as the inclusion of detailed experimental settings and hyperparameters to ensure reproducibility.

Below, we summarize the key points raised in your feedback and outline the revisions and clarifications we have made in response:

**1. Emphasis on Our Main Contributions**

Our primary contribution is the integration of **learned priors** (e.g., Flow, Vamp, GTM, and MoG) in a VAE's KL divergence to create a novel, easy-to-use intrinsic reward;  demonstrating their superiority in addressing the limitations of a fixed prior in Variational Autoencoders.

**2. Clarification of MuJoCo and Atari Results**

We appreciate the reviewers' interest in our MuJoCo and Atari results. We have included tables with the final results (with Mean± SEM)  for added clarity (Table 4 + Table 5 https://bit.ly/4fAbViG). 

- **MuJoCo**: Table 4 shows that $\rm{VaLP}\_{\text{Flow}}$  consistently outperforms other methods, achieving the highest rewards in Hopper (3403.36 ± 354.63), Walker (4188.37 ± 309.09), and Ant (4611.13 ± 642.27). $\rm{VaLP}\_{\text{Vamp}}$  excels in HalfCheetah (7325.67 ± 386.67), leading the rewards in this environment. Across all tasks, VaLP methods outperform the Standard prior highlighting their effectiveness in exploration and policy optimization.
- **Atari**: Table 5  shows that on Atari, $\rm{VaLP}\_{\text{Flow}}$ demonstrates strong performance in sparse-reward environments, achieving the highest rewards in Pitfall (-0.10 ± 0.13) and Gravitar (0.18 ± 0.04), while $\rm{VaLP}\_{\text{Vamp}}$  excels in Breakout (2.12 ± 0.5) and Private Eye (0.28 ± 0.39). Across most tasks, VaLP methods outperform the Standard prior and intrinsic motivation baselines, showcasing their adaptability and efficiency in guiding exploration within constrained interaction budgets.
- Statistical Significance and Variability: We enhanced the statistical reliability of our results by including tables with the final results (Mean ± SEM) and updating the figures to provide clearer evidence and a more comprehensive evaluation of the $\rm{VaLP}$ methods' performance (see Tables 4 + 5 https://bit.ly/4fAbViG).
- 100K Interaction Budget: While the constrained interaction budget in Atari was intentional to assess early-stage exploration efficiency, we have revised the text to better contextualize the results as initial exploratory performance rather than ultimate task proficiency

**4. Minigrid**

While running MiniGrid experiments for longer to allow agents to fully explore the environment would be valuable, our methods are primarily focused on evaluating early exploration efficiency. To strengthen the current evidence, we have included two additional metrics: Coverage (%) and the Coverage Balance Index (CBI), which provide a more detailed assessment of exploration performance within the constrained interaction budgets used in our study.

The coverage percentage was calculated as follows:
$$
\text{Coverage} = \left( \frac{\text{Number of visited states}}{\text{Total number of available states}} \right) \times 100
$$
This metric evaluates how effectively each algorithm explores the environment, with higher values indicating more comprehensive state visitation.

The Coverage Balance Index (CBI) was calculated as follows:
$$
\text{CBI} = \left| \frac{\text{left visits}}{\text{total visits}} - \frac{\text{right visits}}{\text{total visits}} \right|
$$
Where: left visits is the total first-time visits in the left region, right visits is the total first-time visits in the right region, and total visits = left visits + right visits. A CBI of 0 has perfectly balanced exploration. A CBI of 1 has completely imbalanced (all visits in one region). Lower CBI values indicate better evenness of exploration, avoiding detachment between regions. This quantifies the evenness of exploration across two regions of the environment (e.g., left and right spirals). Results for both Coverage Percentage and CBI can be found in Tables 2 + 3 ([link to tables](https://bit.ly/4fAbViG)).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes Variational Learned Priors (VaLP) to address the difficulty of exploration in reinforcement learning. VaLP uses VAE to capture state novelty and was evaluated on popular benchmarks.

Strengths:  
The paper is well-written. The algorithm is simple and has been evaluated across various environments and tasks.

Weaknesses:  
Most reviewers express concerns about the empirical evidence. Additionally, some reviewers question the writing quality and scientific contribution.

Currently, the paper has significant room for improvement, particularly in its experimental section, as noted by reviewers. Therefore, I do not believe this paper is ready for publication at ICLR.