Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The paper considers the problem of approximate value estimation in continuous-action actor-critic architectures, raising the argument that the overestimation bias is not always (in all environments) bad. In particular, they accord to the *optimism in the face of uncertainty* principle to argue that overestimation helps in some domains due to promoting exploration, while in others the negative impacts of such bias outweigh the exploratory benefits (which is why pessimistic approaches such as TD3 have improved performance in many domains). Based on this argument the direction of the paper is a natural one: create a method that adapts the agent's degree of optimism in a single training lifetime. The authors propose such an approach where (at least) two action-value critics are trained, based on which a measure of *epistemic uncertainty* can be obtained. Based on this epistemic uncertainty, the agent's value estimation can be adjusted to be more optimistic or pessimistic. To isolate this measure from *aleatoric uncertainty*, they use distributional critics. There is only a single scalar parameter, $\beta$, that determines the degree of optimism, and the proposed method applies a multi-armed bandit formulation across episodes to optimize this parameter (over discretized values of the parameter in the range of $[-1, 0]$).
main_review: I salute the authors for this fine paper. The paper packs a lot of information but so concisely and clearly. 

The paper presents a fresh take on optimism/pessimism in approximate value estimation. Most successful methods in continuous control address the overestimation of values with pessimism, whereas Ref. [13] proposes an optimistic actor-critic approach that achieves better performance. As such, the present work addresses a natural gap by maintaining the best of both worlds adaptively in the context of the environment. 

The paper is written beautifully. The ideas are clearly presented. The extensions needed to implement the method are minimalistic. I like the fact that the paper doesn't try to use meta-learning, or other more complicated settings, to showcase the base ideas. This makes the paper easy to follow, without confounding effects that would make validating the fundamental ideas difficult.   

The arguments are well motivated with experiments in Hopper and HalfCheetah domains (see Fig. 5). The experiments are thorough (the number of seeds is sufficient, baseline set is reasonable and up-to-date), showing that improvements across all domains can be obtained with a single algorithm. Also, one interesting finding is that (Fig. 4 left) the degree of optimism need not be constant or monotonic during the training course; this further illustrates the utility of using adaptive optimism during training.

**Minor:**
- L116: This is really Q-learning rather than SARSA because $\pi(s_{t+1})$ is not generally the action that the agent has executed in the environment; it's critical to off-policy learning in DPG-style methods to use a Q-learning-style Bellman update on the critic.
- Why focus only on continuous-action algorithms? Doesn't the same ideas hold in discrete-action methods? E.g. Q-learning vs Double Q-learning?
- Why not use $\beta > 0$ in experiments? 
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper is motivated by the two (seemingly contradicting) views of learning efficiently in the RL setup, that of pessimism and optimism. It touches upon why pessimism has been useful in computing better value estimates when deploying function approximation. It then argues that pessimism also leads to reduced exploration and contradicts the optimism favouring view predominant in RL theory. Finally, it claims that the right levels of pessimism/optimism are problem dependent and thus proposes adding a bandit algorithm to adaptively choose either optimism or pessimism for individual problems. This algorithm (called TOP) is tested on both state-based and pixel-based MuJoCo benchmarks. There is a decent amount of performance improvement seen in both cases when comparing against popular baseline methods.
main_review: Originality: The motivation of the work is solid in my opinion. The main question the authors ask is indeed important and I do not know of any other works that empirically test for the trade-off between pessimism and optimism (though I do not follow this exact topic in too much depth). In terms of the solution proposed, I do not rate it too highly in terms of originality. 

Quality: The overall quality of the work is quite high. The experiments are well executed and observations are nicely reported. 

Clarity: The paper reads quite clearly overall. The flow of content is pretty nice as well.

Significance: Since the main contribution of the paper is empirical, there could be two ways to evaluate the significance in my opinion: 1) how much insights do the experiments shed on the trade-off between pessimism and optimism depending on the problem in hand, and 2) how much empirical gains can one get from the proposed solution + how simple would the method be to implement. I strongly feel that this paper makes contributions in the second category. In that regard, I do not completely get why the distributional framework is necessary here. It seems that the pessimism/optimism trade-off can be modelled without bringing in the distributional framework, as is done in the ND TOP-TD3 method. Therefore, to me it seems as if it was added just to boost the performance further and add more content for the paper. That is fine in general but I would argue that it then hampers the simplicity component, and does not highlight as well how much difference the bandit algorithm can provide in and of itself.


Questions:

1. Is there any special reason to choose to work with TD3 instead of SAC? I believe both use the pessimistic form for updating the value functions.

2. The authors say that the code is attached but I could not find any. Can the authors please share the code (with a read-me)?

3. Was there a similar bifurcation done for the pixel-based case like in the state-based case? i.e. ND TOP-TD3 and QR-TD3 for the pixel-based method?

4. I would really like to see error statistics for all methods, instead of just for the TOP method in the tables.

5. Based on the above comments, I am inclined to look at the ND TOP-TD3 column as the main results. Is that a fair way to interpret the results? Can the authors explain more about the inclusion of the distributional framework in this respect?


limitations_and_societal_impact: Suggestions: I believe making contributions in the first category (as described above) would be really valuable. In particular, if we can have an experiment with a simple task such as Mountain Car, and show that the amount of exploration directly varies with the tradeoff in optimism/pessimism. The effect pessimism and optimism have on the exploration seems to be the key to better performance (maybe there is something else as well? maybe they directly affect the representations learnt?) and this does not come forth in the experiments with more complex domains. Furthermore, going deeper into why/how the representations get better when we apply the proposed adaptive strategy can give even more insights.


I do not think there any explicit negative societal impacts of this work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a bandit algorithm to adaptively change optimisim/pesimism in the actor critic algorithms

main_review: In many ways, it looks like the paper builds on top of “K. Ciosek, Q. Vuong, R. Loftin, and K. Hofmann, Better exploration with optimistic actor-critic, In Advances in Neural Information Processing Systems”. While that work considers only optimistic setup (where beta is positive), this work allows for both positive and negative beta.
The idea is to use (discretized) beta in a multi-arm bandit problem to adapt the optimism/pesimism per task (and also during training) 
 
The paper is well written and clear. Previous work seems to be well cited. The motivation is well explained, and further supported by the experiments in the last (sub)section.
Authors also include quite a few Mujoco tasks to evaluate the method and compare it to previous ones. The evaluation is also done very well, 
 
I like the paper, and I see only two negative points here. First, the resulting comparison to previous methods is not that amazing - it mostly does better (to be fair, that is usually the case for many algorithms, it’s very rare to substantially do better in all domains). Second, the work is very much incremental to prior methods.




limitations_and_societal_impact: Sufficiently addressed
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: **Summary and contributions**

This paper studies an interesting contradiction in the reinforcement learning literature, which suggests that both optimistic and pessimistic attitudes toward exploration can lead to high performance. The paper argues that no single attitude applies equally-well to all domains and to all stages of learning; for example, it can sometimes be beneficial to explore pessimistically in the beginning and then slowly change to adopt an optimistic attitude after some amount of experience.

 To resolve the apparent contradiction, the paper proposes the algorithm called TOP. TOP estimates epistemic uncertainty and uses it to bias sample estimates of a distributional reinforcement learning algorithm. The magnitude of bias is adapted online using one of the EXP bandit algorithms, with meta-like feedback in the form of differential episodic returns. Thus, the system will choose the exploration strategy to improve the utility of its learning process.     

The paper applies TOP to algorithms based on the actor-critic architecture, and it evaluates the algorithm extensively in continuous control domains, showing positive results to suggest the extra adaptation is beneficial. 
main_review: **General impressions**

I quite like this paper. I think that it contains many interesting ideas, and it presents solid empirical evidence that support its main claims. But with that said, it currently feels like a stretch to call the paper complete. I have a few issues with the writing and presentation, and there are some technical details that I’d like to clarify with the authors before committing to accept the paper.

**Further details**

In Section 1, the paper does a good job of motivating its research problem by presenting an apparent contradiction. But it follows this up with a vague hypothesis (Lines 36-37). This can be fixed by making the statement more specific and using terms that have been previously introduced. Remove loose language such as “function of the environment”, and “overall context in which a learner is embedded”. The content of a hypothesis needs to be specific and unambiguous enough to be falsifiable.  These comments also apply to the claim on line 44.

In general the language needs to be improved. 
* “estimates the optimal approach” (Line 48). This algorithm can’t claim to be optimal, but it can claim to be good or useful.
* “Learning-theoretic” (Line 43): This is a misuse of the term. 
* I think it’s wrong to say pessimism and optimism are in balance, because choosing one doesn’t come at the expense of the other. 
* “A Bayesian representation of aleatoric uncertainty” is a contradiction in terms. As more information is collected, a Bayesian representation of uncertainty will eventually vanish. But aleatoric uncertainty will persist despite the amount of experience the system has seen.
* “Degree of optimism” is vague. The system has an attitude toward exploration, which can be pessimistic or optimistic. Choosing an attitude gives the system an exploration strategy. 
* The main contributions need to be more clearly stated, without ambiguous language. 

The way epistemic uncertainty is discussed and estimated is confusing (Lines 189--196). It took me several passes to realize that epistemic uncertainty referred to the estimation error of the return distribution’s quantile. The notation and text do not make this clear. 

The random return does not have to be modeled as a Gaussian. One can decompose a random variable into its mean and zero-mean residual. I believe this is what is done in Equation 8. Saying this to start with and removing the language around Gaussians will simplify the exposition.

Proposition 1 seems obvious and potentially not helpful. Unless the authors can justify its purpose, I suggest it be removed. 

The notation is a bit of a disaster. Random variables and distributions are denoted the same way. Equation 5 uses a random variable as the subscript of the quantile function, but Equation 8 uses the belief distribution as the subscript. Equation 4 should reflect equality in distribution. Indices are used without defining their range. I strongly recommend taking another editing pass to simplify and clarify notation. 

How important are having multiple estimates of the quantile function for epistemic uncertainty? Right now you have two.

Can you justify the choice for the baselines and describe what the differences between the proposed algorithm and these represent?

**Other feedback / questions**

* Why is some text of your algorithm red? 
* Aleatoric and Epistemic uncertainty are introduced as terms without definition. It is better to describe the idea of a concept before the term, if it’s not going to be defined.
* How was significance determined in Table 1?

limitations_and_societal_impact: NA
review_title: null

Meta-review: This is a solid piece of work and the reviewers agreed in the end. The paper is well written, it unifies two views on exploration, the experiments are well done, and the authors did a good job both (a) responding to the reviews and (b) improving on the previous submission. There were some concerns about the working being incremental, but consensus emerged that the work exhibited a high degree of correctness (which is critical and not as common as should be) that the paper provides useful insights and foundations of future work. Another reviewer was generally favourable after discussion but did not update their score to reflect that. Given the relevance and difficulty of efficient exploration this work will make a welcome addition to the neurips program.

There were some important missing details and notational issues flagged by one reviewer. I am confident these can be addressed easily based on the author response.
Decision: Accept (Poster)