A Temporally Correlated Latent Exploration for Reinforcement Learning

Authors: Sumin Oh, WANSOO KIM, HyunJin Kim
Primary area: reinforcement learning
Keywords: Colored Noise, Intrinsic Reward, Latent State Representation, Reinforcement Learning, Temporal Correlation

Abstract:
Efficient exploration remains one of the longstanding problems of deep reinforcement learning. Instead of depending solely on extrinsic rewards from the environments, existing methods use intrinsic rewards to enhance exploration. However, we demonstrate that these methods are vulnerable to Noisy TV and stochasticity. To tackle this problem, we propose Temporally Correlated Latent Exploration (TeCLE), which is a novel intrinsic reward formulation that employs an action-conditioned latent space and temporal correlation. The action-conditioned latent space models the probability distribution of states, thereby avoiding the assignment of excessive intrinsic rewards to unpredictable states and effectively addressing both problems. Whereas previous works inject temporal correlation for action selection, the proposed method injects it for intrinsic reward computation. We find that the injected temporal correlation determines the exploratory behaviors of agents. Various experiments show that the environment where the agent performs well depends on the amount of temporal correlation. To the best of our knowledge, the proposed TeCLE is the first approach to consider the action-conditioned latent space and temporal correlation for curiosity-driven exploration. We prove that the proposed TeCLE can be robust to the Noisy TV and stochasticity in benchmark environments, including Minigrid and Stochastic Atari.

Decision: reject
Reviewer scores: 3, 5, 6, 3

Reviews and discussion:
<thread>
creation date:
1730547667729

thread_type:
official_review

<thread_object>
creator: reviewer_ttzz

<content>
summary:
The paper proposes a method for improving exploration in RL using intrinsic motivation. Similar to ICM, a state embedding is learned with respect to an inverse dynamics loss and the error between a predicted embedding of the next state and the actual embedding of the next state serves an auxiliary reward. However, unlike ICM, the predicted embedding of the next state is obtained by a forward model but by reconstructing the next state embedding using a VAE. The paper proposes to use an action-conditioned C-VAE and to use colored noise while sampling the latent variable of the C-VAE to induce temporal correlation (or anti-correlation). The method is evaluated on suitable gridworlds (DoorKey) and Atari environments, where it achieved better results than the baselines (ICM and RND).

strengths:
- The description of the approach is clear.
- While colored noise was previously used for sampling temporal correlated actions for better exploration, injecting temporal correlations while sampling the latent of the VAE is novel.

weaknesses:
- The contributions are quite limited. The main contribution seems to be sampling the VAE latent with colored noise.
- The use of colored noise while sampling the VAE latent is not well motivated.
- Related work is insufficient discussed. For example, Kubovcik et. al used the reconstruction of an auto-encoder as intrinsic reward, Klissarov et al. also used a variational autoencoder, but used the KL between the posterior and prior for intrinsic reward, and, recently, Yan et al. also proposed to use the reconstruction error of a VAE for intrinsic reward but extended it with adaptive exploration mechanism, based on a discriminator.
- The empirical evaluation is insufficient. Table 1 does not show confidence intervals or baseline results. It is not clear how important hyperparameters (e.g. weight of the intrinsic reward) for the baseline were selected and which values were used, and, furthermore, important baselines are missing, e.g. the above mentioned related works.
- The writing could be improved and is often inaccurate. For example the abstract states that the latent space "models" the probability distribution of states, and that the environment where the agent performs best depends on the amount of temporal correlation. Some claims are also questionable, e.g. that the existing curiosity-driven methods are vulnerable to Noisy TV problems (some are, but not all of them)  

References:

Martin Kubovcik, Iveta Dirgova Luptakova and Jiri Pospichal. Signal Novelty Detection as an Intrinsic Reward for Robotics. Sensors 2023.

Martin Klissarov, Riashat Islam, Khimya Khetarpal and Doina Precup. Variational State Encoding as Intrinsic Motivation in Reinforcement Learning. Task-Agnostic Reinforcement Learning Workshop at ICLR. 2019.

Renye Yan, Yaozhang Gan, You Wu, Ling Liang, Junliang Xing, Yimao Cai and Ru Huang. The Exploration-Exploitation Dilemma Revisited: An Entropy Perspective. ArXiV 2024.

questions:
Could you extend on your motivation for colored noise? I understand that temporal correlation and anti-correlation will affect the smoothness of the intrinsic reward. But how does it relate to the noisy TV problem?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730660584523

thread_type:
official_review

<thread_object>
creator: reviewer_gn7s

<content>
summary:
This paper proposes Temporally Correlated Latent Exploration (TeCLE), a novel curiosity-driven exploration method for reinforcement learning that uses action-conditioned latent spaces and temporal correlation, for addressing the noisy-TV issue, a long-standing problem in efficient exploration in RL. The authors propose to use action-conditioned latent space to model state distributions and compute intrinsic rewards based on reconstruction error. Temporal correlation is induced through colored noise in the latent sampling process, and different noise profiles lead to different exploratory behaviours. The method is evaluated on Minigrid and Stochastic Atari environments, showing improved robustness to the Noisy TV problem and stochasticity compared to baselines like ICM and RND.

strengths:
- The proposed method is a novel combination of action-conditioned latent space and temporally correlated signals for addressing the noisy-TV problem.
- The empirical evaluations is comprehensive, with extensive experiments and ablation studies across multiple environments.
- The propose method empirically exhibits robustness with respect to noise.

weaknesses:
- Empirical evaluations on hard exploration tasks (e.g., Montezuma's Revenge) show modest improvement.
- The theoretical grounding between the injected temporal correlations and resulting exploratory behaviour requires further justification.
- The proposed method induces significant computational overhead compared to simpler methods.

questions:
See Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730693270551

thread_type:
official_review

<thread_object>
creator: reviewer_kd3w

<content>
summary:
This paper proposes to add temporal correlation to intrinsic rewards computation by learning an action conditional C-VAE for regularizing the representation of the state representation. The experiments in a few MiniGrid and Atari games show some improvements.

strengths:
The connection between temporal correlation and the NoisyTV problem could be interesting.

weaknesses:
## Writing
Overall, this paper only states how they implement the method but barely talks about the motivation behind them, which will make the reader lose. For example, in Line 48, the author is supposed to say how they address the noisy tv problem mentioned in the previous paragraph, but the author directly jumps in the implementation details of their method and never mentions why temporal correlation can mitigate NoisyTV problems. There are more examples like this, but it will take forever to enumerate all of them.

## Experiments
This paper lacks a lot of stronger baselines in the chosen environments (e.g., [1]). RND and ICM are not state-of-the-art in those environments. Moreover, only a fraction of Atari games are chosen, and the most classic hard-exploration problem, Montezuma Revenge, is not studied, which makes the results questionable.


[1] Zhang, Tianjun, Huazhe Xu, Xiaolong Wang, Yi Wu, Kurt Keutzer, Joseph E. Gonzalez, and Yuandong Tian. "Noveld: A simple yet effective exploration criterion." Advances in Neural Information Processing Systems 34 (2021): 25217-25230.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720204813

thread_type:
official_review

<thread_object>
creator: reviewer_2ara

<content>
summary:
This paper aims to develop an approach to providing intrinsic rewards more robust to stochasticity (and noisy-TV scenarios) than previous approaches. The proposed approach–TeCLE–uses inverse dynamics modeling to learn a state representation and uses the action-conditioned state reconstruction loss (through a conditional VAE) as an intrinsic reward. While sampling the latent representation from the encoder’s output, temporally correlated noise is added (modulated by $\beta$). Empirical analysis shows that TeCLE outperforms bonuses such as RND and ICM in Minigrid environments with noisy-TV and some stochastic Atari environments. The authors also illustrate qualitative differences in exploration behavior based on the color of noise added.

strengths:
S1. Using temporally correlated noise on a latent representation of inverse dynamics model features appears novel (but also see W2).

S2. The approach outperforms popular intrinsic bonuses like RND and ICM in Minigrid environments with noisy TVs and also in many of the considered atari environments.

weaknesses:
W1. From the aggregated and normalized results in Figure 2, all noise levels seem to perform similarly (high overlap in standard errors), even white noise with $\beta=0$. This seems to indicate that colored/temporally correlated noise may not be so important to the performance of the proposed approach, which is a major theme of the paper.

W2. While the paper describes approaches that add noise in the action space, it misses comparisons with approaches that add noise in parameter space [1, 2], which would be closer to adding noise in some latent space. Similarly, another work adds temporally correlated noise to the latent state for exploration [3]. It would be helpful to compare the differences with the approach introduced here. Other forms of temporally correlated exploration, like temporally extended epsilon-greedy [4] could also be interesting to consider in the atari evaluation.

W3. The clarity of the paper requires improvement. Section 4A describes the inverse dynamics modeling task used for feature embedding. As far as I can tell, this is the same idea of feature embedding used in ICM. While the authors describe the ICM paper in the Background section, the current presentation of feature embedding in Section 4A is more likely to be incorrectly interpreted as a novel contribution of this paper. It should be clearly mentioned that the representation/feature learning part is the same as that of ICM. If there is a difference, please clarify what is new in that section. 

Further, since ICM and TeCLE use the same feature embedding (which captures the same information about states), the difference in robustness to stochasticity between the two approaches must arise from the tasks used to generate intrinsic rewards (model error vs. reconstruction error), and it is not clear why reconstruction error would work better. Perhaps the authors could comment and elaborate on why TeCLE works better than ICM.

Please also see the “Questions” section for additional details that weren’t clear to me. I remain open to increasing the score should the weaknesses and questions be adequately addressed/clarified.

### References

[1] Meire Fortunato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Ian Osband, Alex Graves, Vlad Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, et al. Noisy networks for exploration. arXiv preprint arXiv:1706.10295, 2017


[2] Rückstiess, T., Sehnke, F., Schaul, T., Wierstra, D., Sun, Y., & Schmidhuber, J. (2010). Exploring parameter space in reinforcement learning. Paladyn, 1, 14-24.

[3] Chiappa, A. S., Marin Vargas, A., Huang, A., & Mathis, A. (2023). Latent exploration for reinforcement learning. Advances in Neural Information Processing Systems, 36.

[4] Dabney, W., Ostrovski, G., & Barreto, A (2021). Temporally-Extended ε-Greedy Exploration. In International Conference on Learning Representations.

questions:
- In Ablation Study 1, the authors show that the CVAE's action conditioning helps one of the considered problems but not the other. In lines 503-506, the authors mention that this is because of better self-supervised learning. It is not clear to me precisely what that statement means. Could the authors elaborate on why action conditioning is helpful?  

- Is there a reason to use the Number of Updates as the x-axis in Figures 5/7 and the Number of Steps in Figures 6/8? Using the Number of Steps in all cases could be better, or specifying the correspondence between updates and steps (and frames in Atari) in the main text.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732285346520

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Review of the Rebuttal

comment:
Dear Reviewers,

We have prepared this rebuttal by addressing your comments from two days ago. We understand you have many commitments and a busy schedule, but we hope to discuss our responses to your comments. If our rebuttal sufficiently addresses your concerns, we kindly ask you to reconsider your ratings.

Thank you.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732454964087

thread_type:
official_comment

<thread_object>
creator: area_chair_qgu2

<content>
comment:
Dear Reviewers,

thank you for your service. As the deadline for the discussion period is approaching, I strongly encourage you to submit your response to the authors soon. At least, an acknowledgment of having read the rebuttal should be submitted.

Best regards,

AC
</content>
</thread_object>

</thread>
<thread>
creation date:
1732707429206

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Gentle Reminder for deadline of PDF modification

comment:
Dear Reviewers,

As the deadline for finalizing the revised version of the paper's PDF approaches, further modifications to the PDF are no longer possible. If reviewers raise additional discussions or concerns, we will do our best to address them thoroughly in the rebuttal. To respond to the concerns raised by the four reviewers, we have added several appendices and made revisions and additions to the figures and tables. 

Please review these sections, and **if you find that the concerns you raised have been adequately addressed, we kindly ask you to reconsider your rating.**
 
Once again, we sincerely thank the reviewers for their valuable feedback.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733192705779

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final comment for all reviewers

comment:
We would like to thank the reviewers for their comments and help in improving the paper. As the discussion period for the paper is nearing its end, we have written the following message to encourage reconsideration and reflection on any misunderstandings regarding our contributions and achievements.

  

In this paper, we proposed a novel curiosity-based intrinsic reward method, TeCLE.

  

In our proposed method, the intrinsic reward is calculated based on **the state reconstructed by the cVAE,** and the exploratory behavior of the agent is determined by applying colored noise to the cVAE. Extensive experiments on Atari and Minigrid environments show that TeCLE has good exploration abilities and is robust to NoisyTV.

  

---

  

**Common concerns of reviewers with the TeCLE proposed in this paper are as follows:**

  

  

* It is not clear why reconstruction error and temporal correlation are better than prediction error (ICM, RND).

  

  

* Limited comparison between other methods.

  

  

* Clarification is needed on how color noise determines the exploratory nature of the agent.

  

  

* Contribution is limited.

  

However, we have tried to show that the proposed TeCLE has the following contributions and outstanding works as:

  

---

Firstly, in agreement with the reviewer’s concerns, we have revised the paper. **A comparison between the draft and revised versions of the submitted paper is as follows:**

  

* We have added experiments with **three additional Minigrid environments** to Table 1, thereby revising Figure 1.

* We have supplemented the Related Works section in the revised version of the paper.

* We have conducted **additional experiments** and **Appendix F** has been added, providing a comparison of intrinsic rewards with baselines to analyze how TeCLE outperforms baselines while successfully learning optimal policy networks.

* We have added the **description for the Minigrid experiment results** in the revised version of the paper.

* We have modified some ambiguous sentences and statements, which were pointed out by reviewers, to be clearer.

  

---

  
  

Besides, while not in the revised version of the paper, here are some experiments we conducted during the review period to address reviewer concerns and the results of those efforts:

  

* We have compared **the intrinsic rewards of RND and TeCLE before and after the Noisy TV states.** This explains why TeCLE is robust against Noisy TV and has achieved relatively good scores even in the most sparse compensation environments, as opposed to baselines (for a detailed explanation, please refer to the rebuttal for reviewer KD3W).

* We have measured the computational overhead of ICM, RND, and TeCLE. **The results clearly show that TeCLE has less computational overhead compared to ICM and RND.** Regarding the model structure, we chose CNN over MLP in the cVAE since it enables more efficient processing and better abstraction. To further reduce computational cost, we minimized the number of channels in convolutional layers. These designs were made to achieve a balance between efficiency and performance (for a detailed explanation, please refer to the rebuttal for reviewer GN7S).

* We have conducted **additional experiments to compare TeCLE with AMA and NovelD,** which are the strong baselines. Experimental results show that TeCLE outperforms AMA and NovelD (for a detailed explanation, please refer to the rebuttal for reviewer KD3W).

  
---
**We hope that the answers and further experiments during the review period helped address your suggestions and concerns. We sincerely thank you all reviewers.** Thanks again.

  

Best Regards,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces TeCLE, a new method for improving exploration in reinforcement learning by incorporating temporally correlated noise into the intrinsic reward. The core idea involves using a conditional VAE to reconstruct state embeddings, with colored noise injected during latent sampling. This approach aims to address the "noisy TV problem" and enhance robustness to stochasticity in environments.

Strengths
-----------
- **Novelty:** The use of temporally correlated noise within a VAE-based intrinsic reward mechanism appears to be a novel contribution.
- **Performance:** Empirical results suggest TeCLE outperforms established baselines like ICM and RND in certain Minigrid and Atari environments, particularly those with noisy TVs.

Weaknesses
--------------
- **Limited contribution and motivation:** Several reviewers argue that the core contribution of injecting colored noise into the VAE latent space is somewhat incremental. The motivation behind using colored noise and its connection to improved exploration is not clearly articulated.

- **Empirical evaluation:** Concerns are raised about the sufficiency of the empirical evaluation. Specifically: missing confidence intervals and baseline results in Table 1, lack of clarity regarding hyperparameter selection for baselines, absence of key baselines and challenging exploration tasks like Montezuma's Revenge.

Moreover, Reviewers expressed concerns about the quality of writing, inaccurate statements, and questionable claims.

Reviewers agree that TeCLE presents an interesting idea with some promising empirical results; however, the paper suffers from several weaknesses. The novelty and impact of the contribution appear limited, and the motivation and theoretical grounding require further elaboration.  The empirical evaluation needs strengthening through the inclusion of missing information, additional baselines, and more challenging environments. I encourage the Authors addressing the Reviewers' concerns in a future submission.