Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper addresses the foundamental problem of Bellman operator in RL. The authors proposed a proximal Bellman operator that updates that incentivize the online network to remain in the vicinity of the target network. Theoretical analysis and toy task results suggested that the proximal Bellman Operator can accelerate convergence in the presence of high noise in value function approximation. Then the proximal value update is implemented on DQN and Rainbow agents and showed performance gain in many Atari games.
strengths_and_weaknesses: [Strength]
1. The work tackles a generally interested problem in deep RL about target value network.
2. The paper is basically well-written and easy to follow.
3. Both theoretical analysis and experimental results are well-presented
4. The proposed method is simple and effective.

[Weakness]
1. A new hyperparameter $\tilde{c}$ is introduced and need be to tunned to achieve best performance.
2. It is unclear that how can proposed PMPI generalize to continuous action case.
questions: Questions:
1. I understand $\alpha$ as the learning rate same as the orginal algorithm, is this correct?
2. While the experiments are based on Atari which are mostly deterministic environments,  I wonder whether the proposed PMPI works even better in noisy environments. May the author provide some insights?
3. Fig. 1, is $\epsilon$ equal to $\epsilon_k$ ?

Suggestions:

- Fig.2 and 5 are not that friendly to color-blind readers. Some efforts could be made to improved the plots.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose to improve reinforcement learning robustness by essentially adding a regularization term to the Q updates, so that the weight of the online Q network will stay closer to the weight of the target Q network. 
The authors provide both theoretical and empirical results and show that when the proposed method is added to DQN and Rainbow baselines, it improves performance quite significantly on Atari tasks. 
strengths_and_weaknesses: Strengths:
- the simplicity of the method: as the authors discussed, the proposed method is quite simple and effective. And the simplicity brings a number of benefits. 
- clean presentation and adequate technical details: the authors did a very good job presenting the method and also the many technical details, overall, the results seem to be reliable. 
- interesting ablations: very important ablations to show that the effect of the proposed method is not the same as alternatives such as polyak averaging and learning rate decay. 
- theoretical support: the theoretical results are good. 
- significant results: although the proposed method did not improve on all tasks, but overall, the results show significant improvement in performance, and seems to be reliable results given the ablation provided and the discussion on hyperparameters. 

Weaknesses:
- the novelty of the idea: the idea to make network weight stay close to the weight of an older version of the same network is not new. However, it would seem that this particular simple design will have a similar effect to changing polyak or learning rate decay, but the authors show that this is not the case and those ablations are very interesting. So probably should not be considered a major weakness. 
- would love to see some more discussion on how exactly the proposed regularization can have superior effect compared to alternatives such as using polyak and smaller learning rates. Currently, there are empirical results that indicate the effects are very different, but why? For example, if we use polyak instead, will it perhaps... always produce a much larger shift in network weights or maybe the output Q values? I think it will be great if the authors can dig deeper and provide a hypothesis on what is the fundamental difference here that allowed the proposed method to achieve much stronger performance than alternative regularizations. 
- would love to see some additional analysis on environments where the proposed method did not do well, in these cases, is it because the Q networks are now updated much slower (due to the fact that a consistent hyperparameter is selected for all environments)? Then for example, will increasing the learning rate in these environments recover the performance? Not a major point but would be good to see how that works. 
- would be great if we can see results also for tasks with continuous action spaces, such as MuJoCo, where there seems to be a larger problem with bootstrapping. But I understand this might be significant extra work. 


very minor typos:
- line 195 xCan

Summary:
Overall, I like this work, it is presented in a very clean fashion, with adequate details and good results. My current major concern is probably on the lack of explanation on how maintaining vicinity in the weight parameter space will give a very different effect compared to alternative methods? I am willing to increase my score if the rebuttal can address my concern. 
questions: - My main question is: How is the proposed method fundamentally different from the alternatives? Why is it that it can achieve good performance improvement while others fail? (I also mentioned this in the previous section). In the current version of the paper, I don't quite find a very clear and comprehensive explanation for this. 
limitations: I saw the authors indicate they have discussed the limitations of the work, but I did not seem to find where the discussion is, maybe I missed it? Would be great if you can point it out for me.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors consider the class of deep reinforcement learning algorithms that are based on an online network that is updated at every interaction and a target network that is updated periodically. This technique, in general, ensures that the slower moving target network stabilizes the learning. 

The paper proposes a type of regularization which keeps the online network in the proximity of the target network (this technique is referred to as the "slowing down" in the title of the paper).

The paper considers the DQN and Rainbow algorithms, and modifies them with the proposed approach. Experiments on the OpenAI Gym Atari game benchmarks show a significant improvement compared to the baseline algorithms. Experiments show that the performance improvement generated by technique depends on the level of noise in the reward function of the RL process.

strengths_and_weaknesses: + The paper is written very clearly. 
+ Whenever feasible, theoretical motivations of the approach are given. 
+ The paper clearly points out the parts where the current level of understanding does not allow for theoretical proofs.
questions: The paper provides sufficient information to make an informed opinion. 

limitations: The paper adequately addresses the limitations of the work. The theoretical nature of the paper does not raise issues of negative societal impact.
review_title: null

Meta-review: metareview: This paper proposes a a simple method to improve the robustness and sample-efficiency of value-based deep RL methods. The idea is to regularize parameters not to deviate much from target-network parameters. The paper also provides theoretical results showing its convergence property and justifying why the proposed method can accelerate convergence. The results across all Atari games are strong. All of the reviewers appreciated the simplicity and the effectiveness of the method. Some of the minor concerns were addressed during the rebuttal period. Thus, I recommend accepting this paper.
confidence: Certain
Decision: Accept