Kati8/ICLR_2023_2025 huggingface dataset:
authors: Shih-Min Yang
authors: Martin Magnusson
authors: Johannes A. Stork
authors: Todor Stoyanov
reviews: 
Comments: Reinforcement Learning tasks with sparse rewards are always hard to solve. Curiosity-based approaches are derived to facilitate the exploration. However, Soft Actor-Critic (SAC) does not perform well even with curiosity-based exploration. To address this issue, this paper introduces a new exploration strategy KEA (Keeping Exploration Alive), which combines entropy-based exploration and curiosity-based exploration strategies by introducing a simple switching mechanism between two policies with different variance. This mechanism decides which policy to deploy based on the intrinsic rewards.

For experiments, the authors concluded a simple 2D discrete navigation tasks and analysed the learned policy’s entropy and novelty. Results show that their method can improve performance or learning speed for NovelD and RND. They also tried the methods on a set of sparse continuous control tasks. KEA outperforms baseline algorithms. An ablation study of different UTD (Update-to-Data), KEA is more consistent than other baseline methods.
Strengths: 1. This paper addresses an important research questions. Applying Off-policy to tasks with sparse rewards is always tricky and existing curiosity-based methods are not working out of the box.
2. The paper is well motivated via a good visualization. The interplay between policy entropy and state novelty is interesting to look into.
3. The proposed algorithm is simple to implement with a minimal extra computing budget. The switching mechanism is clearly written.
4. The ablation study of various UTD ratios is also interesting. 
5. Relevant literature is good summarised.
Weaknesses: 1. The scope of the paper is rather limited as the authors only tackle Soft Actor-Critic (SAC) in their paper. This paper can be extended to other off-policy algorithms as they all share the issues of premature convergence of the exploration and the influence from different UTD ratios.
2. The experiments are also not very convincing. First, the proposed 2D navigation task is too simple and the performance improvement is also not significant. Second, NovelD and RND both are used as baseline algorithms and NovelD has achieved better performance compared to RND. Yet, the latter experiments are all about RND. Thirdly, in the continuous tasks, the three tasks are not trained for enough time. The performance imporvement is also limited.
3. The presentation/clarity of the paper should also be improved. For example, I  didn't fully get Fig. 1's layout in the beginning. Some sentences are also written in a vague way without further explaination (See questions section). The training of the co-behavior policy is not mathematically defined.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 3
Clarity: 2
reviews: 
Comments: The paper introduces KEA (Keeping Exploration Alive), a framework designed to enhance exploration efficiency in reinforcement learning, specifically for sparse reward environments. The core contribution of the paper is a novel method that combines the Soft Actor-Critic (SAC) approach with a co-behavior agent, designed to improve exploration by proactively coordinating multiple strategies. This coordination is facilitated by a dynamic switching mechanism, which alternates control between the SAC agent, which includes a curiosity-driven exploration method, and the co-behavior agent based on the novelty of the state being explored. This mechanism enables KEA to maintain high stochasticity in high-novelty regions and avoid premature convergence, thereby improving both learning efficiency and robustness.
Strengths: - The method is easy to understand and contributes to alleviating an important issue in the field of reinforcement learning. 

- Proposes a unique combination of SAC and a co-behavior agent with a dynamic switching mechanism.

- Shows substantial performance gains in benchmarks like 2D navigation tasks and robot control.

- Paper well written and easy to follow
Weaknesses: - The paper should discuss how the switching mechanism threshold is tuned and its effect on the performance. This appears to be a key mechanism but somehow ablation studies on this component are limited. 

- The evaluation lacks comparison with papers that address a similar issue, such as [Never give up: Learning directed exploration strategies, Badia et al., 2020]. I would suggest adding stronger baselines to validate the proposed method.

- The paper’s proposed method, KEA, is designed specifically with SAC in mind, leveraging SAC’s ability to handle exploration-exploitation trade-offs in off-policy settings. However, the authors do not discuss how KEA might generalize to other off-policy algorithms. 

- Finally, one of my main concerns comes from the switching mechanism, which utilizes a fixed threshold to decide when to switch between the SAC agent with curiosity-driven exploration and the co-behavior agent. While this approach appears effective in simulations, using a fixed threshold might lack the flexibility needed for different environments or tasks, especially those with varying reward sparsity or novelty patterns. This inflexibility could limit KEA’s adaptability, potentially leading to less effective exploration in more complex or dynamic scenarios
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3
reviews: 
Comments: This paper proposes Keeping Exploration Alive (KEA), an algorithm that switches between exploration policies depending on the value of the intrinsic reward in order to reduce sub-optimal performance due to explore-exploit tradeoffs.
Strengths: This paper is generally well written, and tackles a valid problem in RL, how to gracefully handle the explore-exploit tradeoff. The core results are clear, and the resulting algorithm seems to outperform the baseline by a significant amount on the evaluation task suite. The approach is, to my knowledge, original, though other methods have tackled this problem before in various ways.
Weaknesses: The core issue I have with this paper is that there's very little exploration of the proposed algorithm beyond claims of superior performance. The core concept is simple (not a bad thing), but there's no real consideration given to what makes it good or useful. Seeing as the core contribution of this work is very simple, it should be straightforward to do deeper analysis and benchmarking, but this paper doesn't do much of that, and largely just concludes "it works."

As a result, I can't help but feel this paper is just sort of insubstantial and doesn't contribute that much to the field. It proposes an idea that is novel, but straightforward as an extension of past work, and doesn't do anything with the idea beyond show that it outperforms the baseline. If accepted as is, I'm not sure how much this will actually move the topic of exploration forward.

As such, I'm inclined to recommend rejection, but would like to see an expanded version of this work submitted to a future conference, because I do like the core idea of dynamically switching between explore and exploit policies for off-policy RL.
Recommendation: 5
Reviewer confidence: 4
Soundness/correctness: 2
Clarity: 3