Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors present a novel exploration method wherein an additional Q-function/policy is learned that treats abs(TD-error) of the standard Q-function as its reward function. Both policies are executed in parallel and experience is shared between them for off-policy learning. They demonstrate their method's superiority on a few continuous control tasks relative to RND. Overall, I thought the method was interesting and novel, but several concerns prevent me from endorsing its publication.

1) Incorrect or untuned RND baseline.
As the tasks under consideration are quite different from those in the RND paper, RND must be adapted significantly. It is therefor troubling that only a single RND-specific hyper-parameter is reported, when there should be many (e.g. ratio of intrinsic to extrinsic rewards, see Table 5 in their paper for many more).

It could be that you're adapting RND to be more like an ablation of your own approach, such as replacing the Qx TD-error rewards with random network distillation rewards. This is fine (and I'd suggest doing this as an additional baseline if not), but it should be labeled as such.

2) No comparisons to published RND baselines. 
All of these problems could be avoided if the authors chose to run their method on tasks with published baseline results. Indeed, the lack of a Montezuma's Revenge run is particularly glaring. The author's are right to point out that Atari is perhaps not ideal for exploration since most novel states are rewarding, so I'm not expecting it to outperform RND per se. But the authors also claim that in these situations it won't do worse than RND, and without a Montezuma's Revenge run, this claim isn't well founded empirically.

3)No comparison to value uncertainty methods.
This is also made more frustrating when contrasted to the author's claim that "to our knowledge posterior uncertainty methods have thus-far only been demonstrated in small MDP." Osband, Aslanides, and Cassirer (2018) has results on a continuous control task and Montezuma's Revenge. As QXplore is related to both RND and this class of value uncertainty methods, is feels as though the latter should also be included.

Smaller points:

* No mention of prioritized experience replay. This method utilizes TD-error to bias replay rather than act explicitly, but there seems to be an interesting connection (acting to obtain more TD-error vs over-sampling high TD-error transitions) that was disappointingly ignored in this work.

* I'd suggest dropping the "biologically inspired" bits from the paper. The evidence you're citing (dopamine represents TD error + dopamine seeking behavior) could be used to justify any approach that optimistic with respect to value function uncertainty (e.g. thompson sampling on value function posterior), and doesn't really add anything.

* While the Fetch results are impressive, they should be contextualized by the results obtained in the HER paper. Obviously, HER isn't as general as this method, but it is worth reminding the reader that there is still a large performance gap between QXplore and the SOTA on these tasks.

* noisy TV problem also isn't a problem for RND. Either compare to prediction-error based approaches and show that you're more robust to this issue, or drop it entirely.

Rebuttal EDIT:

I appreciate the authors' responsiveness to my feedback on the text; the framing of biological plausibility and the connection to HER performance are both handled quite well now.

I also appreciate the efforts the authors have demonstrated in adding new baseline results. That said, GEP and DORA results seem preliminary still (which is understandable given the timeframe), and would benefit from tuning and immediate results reporting in the case of GEP.

I still find the treatment of RND problematic. "Where possible, we used the parameters specified by RND in experiments (i.e. fixing intrinsic/extrinsic reward weight at 2)" -- this is not the right attitude when testing such different domains and reinforcement learning algorithms. All of these hyper-parameters should be swept over in a manner analogous to your own approach before this comparison is valid.

I acknowledge the Montezuma's Revenge (MR) isn't representative of exploration problems by itself -- this could be a good place to raise the bar and test against all 6 or so exploration-heavy problems in the Atari Suite. But even just running on MR would be better than introducing new tasks without published results from prominent exploration methods like RND. At the very least adding MR to your set of tasks would allow for comparisons against prior exploration results, which is the primary thing preventing me from raising my score.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes an exploration algorithm for reinforcement learning agents based on learning a separate Q-value function which treats TD-errors as rewards. Experiments on continuous control tasks with a difficult exploration component are used to highlight the effectiveness of the approach. 

The proposed algorithm is interesting and has an intuitive appeal, the magnitude of the TD-error as an "auxiliary reward" seems like a natural choice to guide exploration. The experiments are comprehensive, including some ablation studies and robustness tests, with one caveat. Due to some experimental details (particularly concerning the environments), the empirical results may be invalid and it is difficult to assess them. There are also some certain parts that are unclear in the experimental results. Overall, because of these issues, I cannot recommend acceptance although I would be willing to increase my score if my concerns are addressed.

Concerning the SparseHalfCheetah task, from Appendix F: "...baseline reward level is -1 while a successful state provides 0 reward, but report reward values on a 0 to 500 scale for direct comparison with previous work". This reward function is not sparse since it gives -1 at every step except at the goal which would incentivize the agent to explore naturally if it were acting greedily (or approximately greedily) with respect to the value function. For a sparse reward task, I would expect the reward to be 0 everywhere except at the goal state(s), where it would have some positive value. This is the case for the original reward function for SparseHalfCheetah described in [1]. This change to the reward function makes any comparisons to previous work questionable (Table 1) and could also affect the qualitative results in section 4.5. 

Similarly, the other 3 tasks also do not have a sparse reward structure as the agent receives -1 at every timestep. This again is a confounder for assessing the exploration capabilities of the algorithm. It would be preferable to use other environments such as those in [1], which indeed have sparse rewards. 

For these sparse reward problems where most rewards are zero, then before any nonzero reward is observed, it seems like Qx would mostly behave like previous algorithms based on a state novelty term (as discussed in sec 3.4). This could happen since the agent would only be observing rewards of 0 at this point during training. It is unclear to me then which benefits QXplore could have compared to previous algorithms such as DORA.

Other points:
1) Some discussion of possible pitfalls for the algorithm could be added. For example, while the noisy TV problem may be avoided, there could be a noisy reward problem now. The absolute TD error could be high simply due to stochasticity in rewards. The agent could also be vulnerable to scenarios in which a stochastic transition brings the agent to either a state eventually leading to very high reward or to a state leading to no reward as this would cause the TD error be high (even if the value function converges).
2) The sentence before sec 3.3, "Further TD-error-based exploration with a dedicated exploration policy removes the exploitation-versus-exploration
tradeoff that ... to an optimal Q-function." Could the authors clarify how TD-error-based exploration avoids the exploitation-versus-exploration
tradeoff? I do not see the connection here.
3) In the experiments, how are actions chosen for the epsilon-greedy baseline since the action space is continuous (and not discrete)? 
4) I am bit confused as to what the difference between the lines Q and Qx are in Fig. 3.
5) Why was DORA not included as a baseline algorithm? It would seem to be the most closely related to QXplore when rewards are mostly constant (or zero).
6) Are two separate buffers necessary instead of a single shared buffer (with uniform sampling)? It seems to needlessly introduce complexity and additional hyperparameters (the ratios of samples from each buffer). If this is a key choice, then it should be mentioned in the paper.
The text could also be clarified to indicate there are two buffers. In some places the writing suggests there is only one, e.g. sec3.3 "...policy with a replay buffer shared between Q_\theta andthe Q-function maximizing rx, which we term Qx."
7) In section 4.4, for the "Single-Policy QXplore", why was Q(s,a) replaced by V(s)? If I understand correctly, this experiment tries to test a variant where the TD error is used as a reward bonus. If this is the case, it seems like the best comparison would be to leave the original Q(s,a) which is learned by Q-learning instead of changing it to a state-value function. 

Suggestions (did not impact score)
- I wonder if there is a connection between Qx and the variance of the returns since the latter can be learned by using the squared TD-errror as a reward (see the Direct Variance algorithm in [2]) and the absolute TD-error is a similar quantity. In this way, it could be possible to frame QXplore as following a type of risk-seeking stategy (see [3] for an algorithm that makes use of the variance). 
- I am not sure 'adversarial' is the right term to describe Q and Qx since the two policies are not in direct competition with each other.

[1] "VIME: Variational Information Maximizing Exploration" by Houthooft et al.
[2] "Directly Estimating the Variance of the λ-Return Using Temporal-Difference Methods" by Sherstan et al. 
[3] "Deep Reinforcement Learning with Risk-Seeking Exploration" by Dilokthanakul and Shanahan

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper proposes an exploration method based on the TD-error as an adversarial reward signal. The authors show that this reward signal has interesting exploration properties. They compare it empirically to RND and epsilon-greedy, showing better performance. Besides, they perform additional ablation studies to better investigate the properties of their approach.

Though the paper proposes an interesting concept, it suffers from many weaknesses, some easy to fix and some which will require more work.

Here are some remarks, in random order:

- at the end of the introduction, the authors mention inspiration from computational neuroscience models, but they do not come back to this aspect in their work. To me, these remarks should be removed from the paper and kept for another paper about the biological significance of the model. In the conclusion, the authors come back to the "biological concepts of curiosity boredom, and exploration", I would rather say they are psychological concept, and the authors should have a look at developmental psychology and developmental robotics if they really want to contribute in this respect (but not for this paper).

- some related work references are dispersed in the introduction, in Section 2, in the beginning of Section 3.2, in the end of Section 3.3 and in a few other places. The authors should build a proper "related work" section. Globally, the paper is poorly organized, e.g. Section 3.2 refers to Section 3.4 etc.

- given that Q_x's reward function is the unsigned TD-error, I would like to see whether QXplore can deal with problems taking both some positive and negative rewards.

- the authors mention RND and DORA as baselines, but only compare to RND. What about DORA? Despite the excitement it generated when published, I suspect RND is a rather weak baseline. There are many other exploration frameworks, the paper would be much stronger if the comparison was with respect to many other methods, such as GEP-PG (Colas et al. , ICML 18), Novelty search approaches, etc. To me, the weak comparison is the biggest weakness of this work.

- the authors compared themselves to approaches based on TRPO while they were using TD3 (this information is hidden in Appendix A and should be moved in the main paper. But then, is the difference in performance due to using TRPO which is known to be less sample efficient? Again, this makes the comparison very weak, the authors should rely on the same algorithm from both sides.

- the caption of Table 1 should be explicit about which algorithm comes from which paper.

- I found Section 4.5 very weak, it looks like mere "handwaving" and would deserve a proper quantitative study if the authors want to keep it. In my opinion, the authors should remove it for now and move Appendices E and F to the main paper instead. As is, Appendix E is very poor (by the way, the caption and the figure legend do not match, so we don't know which is which) and I had to look for the number of seeds until I found it hidden in Appendix F.

- Appendix C shows that, though the authors try to minimize the importance of this fact, their algorithm is very sensitive to initialization. It is very honest of them to have kept this study in their paper, but I'm afraid it strongly speaks against the algorithm.

- As described, SparseHaflCheetah is not that sparse (-1 everywhere and 0 when you succeed, as this reward scheme already favors exploration). It would be more informative to use 0 everywhere and 1 when successful.

typos:

Eq (1) and (3) should finish with a dot as it is the end of a sentence.
p3: MDP's => MDPs
Section 3.3, second sentence: avoid starting a sentence with a symbol.
Section 4, the second sentence is not a sentence (no main verb)
p14: is also no subject to it => not 
review_title: Official Blind Review #2

Meta-review: There is insufficient support to recommend accepting this paper.  Although the authors provided detailed responses, none of the reviewers changed their recommendation from reject.  One of the main criticisms, even after revision, concerned the quality of the experimental evaluation.  The reviewers criticized the lack of important baselines, and remained unsure about adequate hyperparameter tuning in the revision.  The technical exposition lacked a sober discussion of limitations.  The paper would be greatly strengthened by the addition of a theoretical justification of the proposed approach.  In the end, the submitted reviews should be able to help the authors strengthen this paper.
Decision: Reject