Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a new class of policy evaluation algroithms, called AsymQ, for a lightweight and effective method to control over-/under-estimation. Different from the previous algorithms that rely on value function ensemble, the key idea of AsymQ is to adopt softmax MSE (SMSE) rather than conventional symmetric loss functions based on a proxy of over-/under-estimation bias. This transforms the landscape to emphasize the policy evaluation where overestimation occurs. The automatic adjustment of softmax temperature is designed based on a pre-defined range of effective batch ratio (EBR). The proposed method is evaluated in MuJoCo continuous control tasks, showing the effect of AsymQ in controlling value function bias under different temperature, comparable performance of DDPG-SMSE to TD3/SAC and the time cost of AsymQ. The proposed method is also evaluated based in Atari and MinAtar, demonstrating the effectiveness when applied to DQN.
strength_and_weaknesses: $\textbf{Strengths:}$
+ The writing and the presentation of the proposed method is almost clear.
+ The proposed method shows effective value function bias control without a value function ensemble.
+ The proposed method is evaluated based on multiple algorithms and in the environments with continuous and discrete actions.
+ In addition to performance comparsion, results from multiple aspects are provided, e.g., loss function landscape, value function bias comparison.

&nbsp;

$\textbf{Weaknesses:}$
- Closely related works [1-4] are not included in this paper. SUNRISE [1] also adopts an asymmetric loss function based on uncertianty quantification (which actually has an underlying connection to value function bias). [2-4] are SOTA bias control algorithms which achieve significant performance improvement over TD3 and SAC. At least, they should be included in this work and discussed.
- Although the proposed method is clear, I have a concern on using 1-step td bootstrapping target as the proxy of ground true value. I know it is non-trivial to obtain a high-quality proxy and 1-step td bootstrapping target is indeed a convenient and conventional chocie (as also used in a few prior works).
  - In my personal opinion, I do not consider that 1-step td bootstrapping target is a good proxy. As the authors also mention, “the constructed supervised target usually depends on bootstrapping as in Eq (4) and the loss is not the distance from the ground truth value function of the policy”.
  - Especially, when applying SMSE to DQN, it becomes more confusing. This is because value-based RL algorithms like DQN, follow Value Iteration rather than Policy Evaluation. In another word, 1-step td bootstrapping target is a value estimate of the improved policy rather than the current value function. I think this may also be a potential reason to the non-robust performance of DQN-SMSE as mentioned by the authors.
  - For another point, I have a little concern on the coupling of using $\delta$ for both the softmax weights and the MSE loss calculation. Since the td error is used for a proxy of under-/over-estimation bias of a specific $Q(s,a)$, I think dependent estimation may be necessary for the proxy of under-/over-estimation bias (similar to the double sampling problem).
- It is good to see the automatic temperature adjustment. However, the range of EBR needs to be predefined. The choice of the range needs to be considered in different domain. How different choices influence the learning performance is unknown. Moreover, it also introduces an additional hyperparameter $\beta_{multiplier}$.
- The proposed method DDPG-SMSE* performs infavorably to TD3/SAC-D in an overall view.

&nbsp;


Reference:

- [1] Kimin Lee, Michael Laskin, Aravind Srinivas, Pieter Abbeel. SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning. ICML 2021
- [2] Xinyue Chen, Che Wang, Zijian Zhou, Keith W. Ross. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. ICLR 2021
- [3] Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, Dmitry P. Vetrov. Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics. ICML 2020: 5556-5566
- [4] Litian Liang, Yaosheng Xu, Stephen McAleer, Dailin Hu, Alexander Ihler, Pieter Abbeel, Roy Fox. Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks. ICML 2022: 13285-13301


&nbsp;

$\textbf{Questions:}$
1) I notice that in Appendix B.5, the MC estimates of ground true value clip the trajectories at 500 steps. Can the authors justify this?
2) Also in Appendix B.5 and Algorithm 2, the text says “10,000 state-action pairs that are uniformly sampled from the replay buffer” and the buffer B is collected by the policy to evaluate. So, the buffer is an on-policy buffer for the policy evaluate, right? It also means that each policy (checkpoint) evaluates its value function bias over its own on-policy state-action supports. Do I understand it right?
3) According to Figure 3c,d and Figure 4c,d, it seems that the range of EBR [0.95, 0.98] results in a slight overestimation. Can the authors provide more explantion on this point?
4) How many seeds are used for the results in Figure 4? It seems that the stds are large since only 0.25 std is shown.

clarity,_quality,_novelty_and_reproducibility: The writing and presentation of the proposed method is almost clear. The proposed method is simple and effective in controlling value function bias. The proposed method is somewhat novel but similar techniques are proposed in previous related work. The advantage of the proposed method in learning performance is not significant when compared to conventional baselines, i.e., TD3/SAC.

For reproducibility, most experimental details are provided in the appendix. The source codes are also provided.

summary_of_the_review: According to my detailed review above, I think this paper is clearly below the acceptance threshold.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper introduces an asymmetric loss called Softmax MSE to address the overestimation issue in value-based off-policy learning. Instead of taking a minimum of an ensemble of Q-functions, as was proposed in TD3, the authors propose penalizing overestimation in TD learning.
strength_and_weaknesses: # Strength
1. The idea is exciting and novel. I haven't seen prior work that mitigates the overestimation issue via an asymmetric loss instead of an ensemble.
2. The experimental evaluation is adequate and demonstrates the advantage of the method.

# Weaknesses
1. TD3 natural decouples dynamics and model uncertainties. However, this method might also penalize good trajectories in the case of stochastic dynamics.
2. Prior work [1] has demonstrated that overestimation bias might be domain-specific. While Clipped Double Q-learning introduced in TD3 helps in some instances, it also negatively affects results. The paper can be improved by extending the experimental evaluation to these tasks.


[1] Tactical Optimism and Pessimism for Deep Reinforcement Learning
Ted Moskovitz, Jack Parker-Holder, Aldo Pacchiano, Michael Arbel, Michael I. Jordan
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. It provides enough details to reproduce the results.
summary_of_the_review: Overall, this is an interesting paper, and I recommend it for acceptance. There are some minor flaws that I hope the authors will address in the next revision; however, these flaws do not affect my assessment of the submission.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to use asymmetric Bellman losses for TD-learning in algorithms such as DDPG, SAC, etc. The goal is to resolve overestimation bias, and the idea of the paper is intuitive. They further propose an auto-tuning algorithm for the temperature hyperparameter in their algorithm.
strength_and_weaknesses: Strengths: the method does appear effective, in that it does reduce overestimation bias.

Weakness:

- it is unclear if the method really improves performance. In particular, the SAC performance with two critics is lower than typical SAC performance, sot he MuJoCo resutls are not trustworthy. For DQN results, DQN is now a 10 real old algorithm. Can you show that this helps with Rainbow?

- It is not clear if fixing over-estimation should lead to better performance. Also, what about using other methods such as truncated distributional critics? How do they perform in comparison?

- No clear significance: to be honest, I am not sure what the point of the result DDPG + SMSE == SAC is. In particular, SAC is so well adopted, that unless we improve over existing state of the art algorithms, or bring new insight to the table, it is not obvious why this result should be significant in the long run.

- More experimental settings: it would be worth studying this in settings where over-estimation is likely to hurt more, like in offline RL, or in settings where we must take more gradient steps per environment step.

clarity,_quality,_novelty_and_reproducibility: The paper is clear, and the quality of writing is OK (though it can definitely be improved). The authors address reproducibility at the end of the paper.
summary_of_the_review: In summary, I think the empirical evaluation of the paper needs a lot more work. The paper needs to convince the reader why their method is significant, and why we should take note of it as a community, and this is where the paper falls short. I am opting for a reject score due to these reasons. Besides, it would also be good to see an analysis on why overestimation matters.

___

## After Rebuttal

Thanks for the responses! I am still not convinced about the significance in the long run, but I will move up my score to a 5, in light of the rebuttal.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new method to better mitigate estimation bias. Essentially the method will allow weighted gradient updates when training the critics, and (depending on the temperature hyperparameter) give a higher weight to gradients that will lower Q value estimates and a lower weight to gradients that will increase Q value estimates. In this way a more fine-grained bias control can be achieved. A major benefit of this method is it lowers computation and memory consumption. Empirical results on Gym and some Atari tasks show it can outperform other baselines. Toy example and other empirical analysis and some theoretical support are also provided. 
strength_and_weaknesses: Strength: 
- Computation efficiency: The method imposes small computation and memory overhead. 
- Easy to apply: Proposed method can be applied to existing methods and does not require extensive modifications.
- Some significant results: Results on 4 Atari tasks show significant performance gain over DQN and DDQN baselines
- Technical details: many technical details are provided for better reproducibility, the computation comparison figure is great and there are studies on gradient clipping and the temparture. 

Weaknesses: 
- Significance of results: in Gym tasks, the proposed method can be stronger than other methods in some cases, but it is not that consistent, in some tasks, the performance gap is really small. In Atari tasks, there are only 4 tasks shown. It seems quite reliable that the proposed method can greatly improve DDPG, DQN, DDQN on given tasks. But these methods are a bit old and it is unclear how it compares to more recent methods? For example REDQ (Randomized ensembled double Q learning, ICLR 2021) on gym tasks and Rainbow variants on Atari. 
- Would like to see how your method compares to SOTA algorithms on gym, for example if you follow what's in the REDQ paper, use a higher update-to-data ratio, maybe your method will have a much better performance compared to competitors? If you can reach SOTA performance with much lower computation, then that will make the performance claim much stronger. 
- I believe you are tuning for each task. In that case why is it that DDPG+SMSE cannot beat DDPG in some tasks? Not sure if that really makes sense? The other thing is for the methods you are comparing to, are they also fine-tuned to each task individually? 
- Figure 1 the figure is OK but you might want simply make your discussions a bit more explicit (maybe just explicitly point out early in the paper that the proposed method can be seen as providing weighted update and will make Q values faster to go down and slower to go up and be explicit about the intuition you are trying to convey in the figure)
- For figure 2, the plot is nice but the discussion on how you set up the toy example experiment is a bit vague, maybe you can rewrite some of the explanation to improve clarity. Is each curve in the figure showing what happens after many updates? And would be good to add a summary sentence in the end, just to make your insights more explicit for your reader. 

clarity,_quality,_novelty_and_reproducibility: - Clarity: Most of the paper is clear, some parts can be refined
- Quality: overall quality is good
- Novelty: the proposed modification is not super different from existing methods but can be considered somewhat novel
- Reproducibility: good, adequate details are provided 

summary_of_the_review: The proposed method is interesting, and a lot of technical details are provided which is great. My main concern is on the significance of the performance of the proposed method. All the methods compared are quite old, so it is unclear how the proposed method will perform against some of the current SOTA methods in bias reduction.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes the use of asymmetric losses to reduce the overestimation bias in off-policy reinforcement learning. The proposed approach is novel and elegant, but weaknesses in the supporting experiments led to reviewers questioning the effect size of the approach particularly in comparison to more recent baseline algorithms. Whilst the current experiments were not fully convincing to the majority of reviewers, there was broad acknowledgement of the novelty of the approach and early signs that this could be a promising direction for future work.
justification_for_why_not_higher_score: Insufficient experimental results to support claims
Decision: Reject