Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: The paper tries to improve exploration in continuous control tasks by augmenting Soft Actor-Critic (SAC) with a curiosity module that increases the target entropy for unfamiliar states and decreases the target entropy for familiar states. The curiosity module is implemented using RND combined with a contrastive loss function (X-RND). To incorporate curiosity into SAC, the entropy coefficient is learned and dependent on the prediction error (coming from X-RND) of the state. The overall method (CAT-SAC) is tested on mujoco and swiss roll maze. Additionally, the paper includes ablation studies showing the benefit of X-RND and that of discretization of curiosity.

Novelty: The main novel component is incorporating curiosity into entropy to increase the target entropy in unfamiliar states and decrease it in familiar states. 

Pros: CAT-SAC does improve over SAC on mujoco domains and it helps to have X-RND component instead of vanilla RND.

Cons: The ablation studies and the evaluation feels incomplete. 
(1) Rather than adding curiosity to the entropy, what if we added it to the reward function? How well that perform? That should be a baseline. 
(2) RND doesn’t work that well with feature inputs. What about other notions of curiosity? How does dynamics model prediction error work in input space? There should be 2 baselines: (i) how well dynamics model prediction error work when added to reward function? (ii) how well it works when incorporated into entropy?
(3) The method should be evaluated on harder exploration tasks (eg: ant gather and swimmer gather from Variational Information Maximizing Exploration (VIME; Houthooft et. al.) paper) and compared with VIME (or other exploration algorithms eg: why does hierarchy work so well? (nachum et. al.))

Reasons for score: Weighing the above pros and cons, I vote for rejecting. 

Questions: Given c(s_t)is a scalar (as it’s prediction error), is g_{\delta}(c(s_t)) just m(c(s_t)) + b?

I am happy to reconsider my score if the above concerns are addressed. 
review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper introduces an algorithm that augments SAC with Curiosity-Aware Temperature, to enable more efficient exploration. Previous versions of SAC had a fixed entropy temperature which had to be tuned or an automatic tuning mechanism that was not state-specific. The paper proposes that exploration can be improved if temperature is state-dependent, based on curiosity, or unfamiliarity of the state.

Authors introduce curiosity to the target temperature such that the entropy is large in unfamiliar states, promoting exploration, and small in familiar states, encouraging more exploitation. To enable this, the authors introduce three components: 1) target entropy that is augmented with curiosity, 2) curiosity and hence state based entropy, 3) X-RND that adds contrastive loss to ensure more robust computation of curiosity.

Curiosity is based on prediction error of states, using the idea from Random Distillation Network (RND) (Burda et al. 2018b). This curiosity term is normalized such that in expectation it corresponds to the original target entropy. The instance-level temperature also uses this curiosity to map states with similar level of unfamiliarity to similar temperature value. X-RND is a technique that the authors develop in order to overcome previous difficulties that RND had on feature inputs. 

In their benchmark experiments they show that their method CAT-SAC shows superiority compared to SAC as well as other baseline methods. They also show results on a toy domain how X-RND can estimate curiosity more robustly than RND.

The authors have done a fair job introducing curiosity to enable better exploration by varying the target entropy at state-level. Disregarding minor grammar errors I think it is structured nicely. The idea is interesting but I would recommend reject as the experiments need to be conducted more rigorously. 

Benchmark experiments are conducted only with 4 runs on Mujoco environments that are known to have high variance based on random seeds. Henderson et al. 2018 (https://arxiv.org/pdf/1709.06560.pdf) show that on HalfCheetah, the same algorithm can have significantly different performance, between two groups of 5 random seeded runs. More runs need to be conducted in order to show credible performance improvements.
Furthermore, currently CAT-SAC surpasses SAC in all four Mujoco domains at 200k steps of training, but SAC results in the original paper (Haarnoja et al. 2018b) show SAC reaching avg return of 2000 in Hopper and 3200 in Walker. This also shows that current results where SAC performs worse than CAT-SAC could have been due to variance in random seeds.

Comparison of CAT-SAC to other baseline method performance seems less appropriate too. Results are based on SUNRISE paper (Lee et al. 2020) which has not been peer-reviewed and which have also done only 4 runs each. 

Lastly, demonstration of X-RND seems to show that it is able to remember states it has visited and keep curiosity for remaining unvisited states. It looks like after the state is visited once the curiosity drops from 0.2 to 0.02 immediately. I’m not sure if it would be desirable to have curiosity drop suddenly after visiting it only once. To me X-RND seems like a mechanism with a replay buffer that limits neural networks from generalizing and making it tabular-like to output low curiosity only for states it has visited. X-RND certainly performs better than RND in this toy domain but I’m not sure if it adds that much value to the overall paper, where it is about having instance-level entropy that encourages exploration.

For other minor details, I think the plots and labels in Figure 2(b) (c)  are confusing.I think the x-axis should be index of each state, and prediction error of each state the label for the colormap. 
review_title: Interesting idea but needs more rigorous experiments

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary: 

This work proposes to incorporate curiosity into the entropy temperature of Soft Actor-Critic, and applies a modified version of Random Network Distillation as their curiosity model. Its key insight is that the entropy temperature in Soft Actor-Critic should encourage the agent to explore unfamiliar states more and familiar states less, rather than globally encourage some expected target entropy. To this end, they make the target entropy and temperature both state-dependent, using a curiosity model based on Random Network Distillation. 



Strengths: 

- The insight is described well and motivates the need for a curiosity-aware entropy temperature for further sample-efficiency.

- The paper demonstrates good experimental results across four OpenAI Gym tasks evaluated after 200K environment steps. There is also a thorough ablation study to illustrate the importance of each component of the approach.



Weaknesses:

- The justification of the instance-level entropy temperature in Section 4.2 could be clearer. As is, it’s unclear why the instance-level entropy is not redundant after the curiosity augmented target entropy, and vice versa. It’s also not well justified why the curiosity value is rounded and then re-scaled to form the entropy temperature. 

- While the method achieves good results on the standard tasks, I am still curious about its performance on pure exploration tasks, and hard exploration tasks such as sparse-reward settings. In the former, it would be great to quantitatively measure the unique states visited and to provide a comparison to RND.

- While the modification to RND is intended to correct for curiosity at unseen states, the selection of unseen states is based on a heuristic: unseen states are created as combinations of pairs of seen states.



Recommendation:

I am recommending to reject this paper. I think parts of the method can be more clearly explained and justified. There is also potential to better highlight the strengths of the proposed method in the experimental results, i.e., by including pure/hard exploration tasks.



Questions:

- The learning curves in Figure 1 evaluate SAC and CAT-SAC for 200K environment steps. I’m curious about the behavior after more steps & at convergence, and how many steps are necessary for convergence.

- Are beta and m tuned for each task?

- What does “the instance-level entropy temperature may weaken the connection across states” in Section 4.2 mean?



Updates after Reading Authors' Response:

Thank you for the detailed clarification and the new results. From the response, it seems that the proposed modifications to SAC do not and are not meant to solve the exploration problem in sparse-reward tasks. Instead, its aim is to improve sample efficiency on standard dense-reward tasks. The impact of the work then feels quite limited: the proposed modifications are specific to SAC and do not meaningfully improve performance on the sparse-reward task. For these reasons, I will keep my original evaluation.
review_title: Interesting way of incorporating curiosity but requires clarification and more experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: **Contribution**: For better exploration, the authors propose to use curiosity to set state dependent target entropies with SAC, with the goal of inducing more diverse behavior at unfamiliar states. They use RND to provide a curiosity score, which after normalizing, is used to adjust the state dependent target entropy. Due to RND performing poorly as a curiosity measure with state-based representations instead of image based, introduce a variant X-RND which additionally uses a contrastive loss to improve the curiosity mechanism. They demonstrate benefits over regular SAC on standard Mujoco Gym benchmarks.

**Prior work in exploration**: While increasing policy entropy in regions of low  confidence makes intuitive sense, it is not actually clear simply being noisier in the face of uncertainty actually leads to good exploration (for example https://arxiv.org/abs/1306.0940 argues that these "dithering" style exploration is inefficient). It would be good to add additional comparisons to other methods for augmenting SAC to perform better exploration. For example, OAC https://arxiv.org/abs/1910.12807 learns upper confidence bounds on Q-values and uses the optimistic Q values for an exploration policy. It could be interesting to also explore how well CAT-SAC compares to just using the (X-)RND curiosity score directly as a bonus for exploration. In general, the related work is lacking in discussion of classical RL exploration methods like UCB-style bonuses and posterior sampling.

**Conclusion**: Overall, I like the work. It appears technically sound, and presents a fairly simple and intuitive way to adjust exploration with SAC to better handle unfamiliar states (as far as any undirected exploration methods do at least). I would like to see a few more comparisons against other exploration techniques in deep RL, and perhaps some experiments ont asks outside of the standard gym benchmarks that focus more on the exploration problem itself rather than control (for example some sparse reward tasks). 

Fairly minor points: 
LaTeX error: There are several instances of log in math mode that should use \log instead.
I would also like to see extended learning curves of CAT-SAC vs SAC, particularly until we see the performance of each method saturate. It would be interesting to see if the better exploration from CAT-SAC allows it to converge to higher performing policies as well as learning faster.
review_title: Intuitvely reasonable and empirically beneficial exploration strategy

Meta-review: The concept of increasing entropy in novel states to promote exploration appears to be quite interesting. I do appreciate this idea, and I would encourage the authors to study it further. I think the reviewers also agree with this. Unfortunately, the paper as written has a number of issues: (1) the theoretical motivation for this is quite weak -- it intuitively makes sense, but for a published paper, we need more than intuition; (2) the empirical results are not especially compelling, it seems like the authors have to kind of thread the needle in arguing that they are concerned specifically with dense reward exploration -- a less chartable view is that the method just doesn't work all that well compared to other exploration settings in problems that present a major exploration challenge. The reviewers generally found the evidence in favor of the method to be a bit questionable. Therefore, in the balance, while the paper presents what I think is a really nice idea, it still needs work to justify both theoretically and empirically. I would encourage the authors to flesh out this work more -- I think with a bit more work, it could be a really nice paper, but for now it's probably not quite ready.

A few more comments (which did not influence the decision, but I recommend addressing in the future):

- Presenting exploration and efficiency results in a table, like Table 1, is not good. It hides actual patterns in performance, especially if you are talking about exploration. To the authors' credit, it appears that this trend was started by prior work (e.g., Kimin Lee et al.), but it was bad scholarship then, and it's bad scholarship now -- it's quite easy to pick a checkpoint where a given method looks better than all the other methods (which might be why some prior work opted for this format), but it's misleading to the reader and should not be done.

- I don't agree with Reviewer 2's comments about random seeds. Certainly it's better to have more random seeds than less, but this doesn't appear out of line with the standard in the field. That said, the results in Figure 4 do look quite close, so trying an actual statistical significance test might be a good idea (again, this didn't influence my decision -- the standard in RL holds that 4-6 seeds is plenty, and we have to review work by the current standard in the field).
Decision: Reject