Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a reward shaping mechanism that trades of exploitation/exploration, by considering adding a negative vs. positive intrinsic constant leads to curiosity driven and conservative exploitation respectively, also they  study the effect of positive constant reward multiplication acts as a different learning rate. Based on this simple remark they study how offline and online RL can benefit from this shifting approach.



strengths_and_weaknesses: Strengths:
This paper is very well written and clear it studies a simple yet widely applicable problem in RL and curiosity driven approaches for exploration. It provides a derivation to how reward shifts affect the optimal value function, and their equivalence to different initialisations.
It tackles an important problem in exploration and provides interesting insights.

Weaknesses:
The paper does not show case the ranges under which this constant shifts work well, for too large gaps the the agent will never exploit or explore depending on sign? 
There is also a trade-off between the multiplicative weight k and the bias? Can the authors clarify how these interact with each other?
How to best estimate a good b that provides a LB or UB? Can these parameters be learned? 


questions: Questions:
(See above)
. can the Q_UB-Q_LB provide a good uncertainty estimate locally? 
Minor:
. missing meaning of each line if both fig.3 and fig. 2 
. yellow in line166 (no yellow lines exist)

limitations: - The paper studies how constant bias shift affect the value based learning agents as the constant is the same for all action/state space. In intrinsically motivated exploration this is not the case and it is a different one per state/action. Can the authors elaborate on insights of what this means and whether the current approach can make certain cases blind to exploration since the gap now is too big or small.
- how other OFU methods apply in this setting, such as Rmax.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper first shows that reward shifting is equivalent to diversified Q-value network initialization in deep-RL. This kind of phenomenon can influence learning efficiency. Then, using the key insight, the paper presents three scenarios to show the benefit of reward shifting. Finally, the paper conducts experiments to prove the key insight in three scenarios.
strengths_and_weaknesses: Strengths:
1. The paper shows that reward shifting is equivalent to diversified Q-value network initialization in deep-RL. This is the key point throughout the paper.
2. In 4.3.1, the paper proposes that Q-value can be written in the form of a linear combination.
3. The experiments compare many conditions in one environment. 

Weakness:
1. The experiment results are not reliable enough. In Figure 4, the paper shows four kinds of Hopper environment and one kind of Walker environment. And the result in the Walker environment is not so obvious. More environments and more experiments in each environment are needed to compare. The experiment shown in Figure 6 has the same problem. In Figure 5, only 1e6 steps of environmental interactions are used, resulting in policies that have not yet converged in most tasks.
2. The theoretical proof is not good insufficient. We should be more concerned about how to find the balance of the conservative and optimistic exploration, rather than attempting different reward shifting settings for each environment.
3. The innovation of this paper requires further discussion. This paper is based on an intuitive idea to optimize multiple classes of methods such as offline and online. This paper treats each class of methods differently, and there is only one baseline algorithm for each class of methods, which makes me worry about the universality of this optimization.
questions: 1. In lines 40-41, is there any theoretical proof of this conclusion? Can you make sure in most of the conditions, the key insight is true?
2. Why the baselines in 5.2 and 5.3 are different?
3. How about the total amount of compute between the baselines and the algorithms in this paper?
4. What are the advantages and disadvantages of the proposed method compared to similar exploration enhancement algorithms? Lack of description or theoretical comparisons.
5. Why does EnsemleTD3 perform worse than TD3 in Figure 5?
6. How can fairness be guaranteed by using only 3 Q networks in Figure 5?
7. There is no doubt that negative reward shifting will lead to more exploratory behavior, but is the Reward Shift approach better than Count-Based or other approaches? In Figure 1, Count-Based and epsilon-greedy can also be adjusted for exploration ability by hyperparameters, are these adjusted to be optimal in the experiment?
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies how reward shifting can be used to encourage or discourage exploration without modifying optimal behavior in deep Q-learning algorithms. The authors demonstrate how reward shifting can be used in different contexts where exploration is desirable or undesirable and show that it can improve performance on a range of tasks.
strengths_and_weaknesses: The core idea proposed by this paper, reward shifting, is shown to be both simple and effective. A simple method (both conceptually and to implement) is more likely to see wider adoption and further work, so that is a significant strength of this paper.

I do feel there are a number of significant issues to be addressed to make this paper great, however. To be clear, I like this work, but I find it hard to recommend it be accepted without substantial modifications (particularly the points below). If these issues can be addressed I think this will be a good paper and a valuable contribution to the field.

-The clarify of the writing is poor. I found the paper generally understandable, so this isn't a fatal flaw, but it made understanding more difficult than it needed to be. Apologies if I'm mistaken, but this paper comes across as written by scientists not fluent in English, and regardless of acceptance status I strongly encourage the authors to take a careful editorial pass (preferably with an English-fluent colleague's help) to improve the language. This will greatly improve the presentation and lead to more people reading and citing this paper in the future.

-The paper feels somewhat padded. Given the simplicity of the core idea it doesn't take much space to explain how reward shifting works, and the paper as is (IMO) spends more space than needed on it. This space could be better spent addressing some of the points below, or increasing the size of the results figures (which are a little too small to read without zooming in).

-The idea of reward shifting is not novel, though I do not know of any work that explores it in the way this paper does. I think the novel contribution is significant and worthy of publication here, but a longer discussion of related work seems important for a paper like this one that applies an old idea in a new way to good effect.
 --There is related work using value function initialization, which is equivalent to reward shifting, to perform exploration. For example DORA, which is cited in this paper but not discussed, uses biased initialization for exploration, which should perform similarly to reward shifting, and seems like a natural point of comparison.
--Similarly, biased value ensembles have been proposed before, but such methods don't seem to be discussed (Some related works: "A distributional code for value in dopamine-based reinforcement learning" by Dabney et al, "UCB Exploration via Q-Ensembles" by Chen et al)

-The experiments cover several cases and domains, which is good, but the comparison to RND seems odd as it focuses on simple minigrid environments and doesn't include any hard exploration continuous MDP tasks where baseline naive exploration methods perform poorly, which were the intended use case for RND. For example, the Sparse HalfCheetah environment proposed by VIME or other continuous control exploration tasks similar to the MuJoCo gym tasks already tested would be easy to compare on without additional modification to the algorithm.

-I'd like to see some exploration of the limitations of reward shifting (see limitations section).
questions: Some other suggestions (less critical stuff):

-Consider using a non-gridworld example for figure 1. This paper is mostly concerned with deep RL and continuous MDPs, so leading with gridworld results seems odd.

-Figures 2 and 3 I found kind of confusing. I see and like the intent, but it's hard to piece together what's happening based on the caption and equations in the panels. Consider cleaning up the captions and labeling the lines with descriptive labels rather than equations/symbols that haven't been fully introduced/explained at that point in the paper.

-I don't like table 1. It's a matter of preference, but I don't find it informative. Consider focusing on the domains/use-cases for reward shifting instead of the comparison to previous methods.

-I didn't understand the point about RND-1.0 being compared to vanilla DQN. RND has a variable reward bonus, so I'm not sure why it should neatly cancel out when reward shifted.


limitations: This paper doesn't discuss the limitations of reward shifting, which I think is a significant issue. The experiments presented show reward shifting to work well on a variety of tasks, but when does it perform poorly? 

I think it might have issues with stochastic rewards, for example (offhand it seems like it might work but with worse bounds?)

How does reward shifting perform on harder exploration tasks, or larger/more complex off-policy datasets? Even if it under-performs compared to dedicated methods there is value in a simpler and more general but less specialized algorithm, so I'd love to see the results regardless of outcome.

Regarding societal impact, I don't think this work has any direct negative impact risks.
review_title: null

Meta-review: metareview: This paper proposes a simple but general way to improve exploration in RL based on the equivalence between reward shifting and the initialization of value function. The paper shows that it is straightforward to implement conservative exploitation and curiosity-driven exploration based on this idea. The results on a variety of offline/online RL settings show that a properly initialized value function (i.e., shifted reward) can achieve a better exploration/exploitation and improve the performance of existing RL algorithms as a result.

In general, most of the reviewers found that the proposed method and the results are quite interesting enough to be presented at NeurIPS. While some reviewers had a concern about the lack of challenging tasks, the authors addressed it with updated results in the appendix. The only reviewer with a negative score did not respond during the discussion period. Thus, I recommend this paper to be accepted. In the meantime, there are still remaining (minor) concerns about the presentation of the paper (e.g., grammar, lengthy description of a simple idea, etc) and the lack of discussion on limitations and related work. I highly recommend the authors to improve them by reorganizing the paper (e.g., omit some details and move some important discussion from the appendix to the main text) for the camera-ready version. 
confidence: Less certain
Decision: Accept