Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper studies the problem of learning informative rewards, without domain knowledge or external supervision, that will accelerate learning in extremely sparse reward settings with noisy distractions. To address this problem, the paper proposes Exploration-Guided Reward Shaping (ExpRS) that balances exploration and explotation of environment rewards. The paper derives the update rules for ExpRS in a general setting, not specific to policy gradients. The paper then shows results on three environments comparing to other reward shaping methods. 
strengths_and_weaknesses: Strengths:
* The proposed method is intuitive and to the best of my knowledge novel. The paper discusses benefits of both an intrinsic shaped reward and intrinsic bonus rewards and describes how to combine both into one algorithm. The paper also derives an update rule that works for general RL algorithms, not just policy gradient methods.
* Theorem 1 with the analyzed chain environment is helpful for gaining insight into the utility of ExpRS. 
* ExpRS is more robust to distractors in the environment than baselines. The results demonstrate that distractors hurt the performance of all baselines, whereas ExpRS is more robust and affected less.

Weaknesses:
* The simple environments are not enough to verify the algorithm. While these environments are still helpful, results on more complex environments are important to verify the utility of ExpRS. For example, LIPRG [1] a work cited throughout this  paper and published at NeurIPS 2018, shows results on pixel-based Atari games. My concerns are exacerbated with the performance on the LineK task, which is the only task that demonstrates a neural network policy, yet the gap between ExpRS and baselines is small. In LineK, ExpRS is only slightly more sample efficient than the regular sparse reward function. This concerns me that ExpRS cannot scale to neural policies and rewards in complex environments.
* How does ExpRS balance the SelfRS and count-based bonus reward? Since the main contribution of the paper is combining these two rewards, studying their balance is importance. On line 183, the paper shows the count-based reward is weighted by parameter $\lambda$. The authors should analyze the impact of different reward weightings to see the impact on performance. 
* The SORS baseline is not explained enough in Section 4.1. SORS is briefly explained on L114, but a more detailed description would be useful in Section 4 for how the SORS reward is obtained and how it compares to ExpRS.
* The paper might also be lacking some important baselines, which I discuss in the Questions section. 

[1] Zheng Zeyu et al. On learning intrinsic rewards for policy gradient methods. In NeurIPS, 2018.
questions: * As the authors state, [1] compares on the same set of environments. However, [1] uses a exponential scale on the x-axis. This is important because it shows that the method from [1] takes exponentially less experience to learn. Why did this work not use an exponential x-axis scale? 
* In Theorem 1, it seems using an exploration reward contributes more to the cost than the self-supervised reward shaping. Why is this the case?
* Given LIPRG [2] was mentioned several times throughout the paper, why not compare to LIPRG as well in the REINFORCE experiments?
* Why not compare to ExpRD from [1] given that work shows results on the same set of environments? 

* The derivations in Eq. 1 detract from the description of the algorithm. I recommend moving the derivation to the supplementary and stating the final result in the main paper.
* The supplementary material also includes the main paper, reformat to only include supplementary material or move the supplementary to the main paper. 

[1] Devidze Rati et al, Explicable reward design for reinforcement learning agents. In NeurIPS, 2021.

[2] Zheng Zeyu et al. On learning intrinsic rewards for policy gradient methods. In NeurIPS, 2018.
limitations: The paper acknowledges that it would be interesting to extend the evaluation to more complex tasks. The paper also discusses the limitations of not having a rigerous theoretical analysis of the method in terms of convergence speed and stability. Despite this discussion, the paper does not sufficiently analyze limitations of the method. In what settings would ExpRD not perform as well? What are the limitations of learning a more complex reward in ExpRD?
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a new method for reward shaping (called EXPRS) that leverages intrinsic rewards from self-supervised reward shaping techniques, as well as exploration bonus in the shaped reward. The authors first describe the general setting of reward shaping followed by existing reward techniques and their shortcomings, specifically self-supervised reward shaping and exploration bonus based reward shaping. Subsequently the authors introduce their reward shaping formulation, which includes both the intrinsic self-supervised reward and the exploration bonus in a combined formulation. The authors then derive gradient update formulas for their proposed algorithm, in a general and empirical setting, and outline the steps in their EXPRS method.

Following the definition of EXPRS, the authors perform a theoretical analysis of how EXPRS should perform in a chain environment compared to other methods and then show experimental results of EXPRS and those methods in a chain environment, as well as a Room environment and a LineK environment. The paper then concludes with a brief discussion of the results and some directions for future work.
strengths_and_weaknesses: **Originality:**
- Strengths: The paper proposes a new method for automated for reward shaping in RL that draws upon two prior methods and combines them into a single formulation. Relevant work is cited and compared against in the beginning of the paper.
- Weaknesses: The authors could have provided more detail on the limitations of their method, which are only briefly discussed in the conclusion.

**Quality:**
- Strengths: The paper provides a good amount of detail in the derivation steps related to the method and is generally very thorough in description of relevant details.
- Weaknesses: The number of experiments that showcase the capabilities of the method is somewhat limited.

**Clarity:**
- Strengths: The paper is generally well organized and includes relevant figures and diagrams for the environments used in the experiments.
- Weaknesses: The authors could have helped guide readers by providing summaries of important points/messages throughout the paper, especially after sections with heavy text and equations. For example, after Section 2 it could be helpful to have a small table outlining the prior methods and outline what they can do (e.g. with checkmarks) and how EXPRS has capabilities beyond those method. In Section 3.2 it could be helpful to state in the beginning what the missing terms in the gradient computation are and how the derivation leads to them being available when describing the final algorithm. 

**Significance:**
- Strengths: The paper proposes a new method for exploration based in reward shaping, which is a relevant problem in RL.
- Weaknesses: The environments and algorithmic varieties for which the method was tested were somewhat limited and provide little insight into how the method may perform in more challenging settings.



questions: **General Questions:**
- Could you define "b" in Line 167? This is mainly for completeness since I did not see it defined (I might have missed it)
- How general is the gradient derivation to different RL algorithms? For example, if someone wanted to apply EXPRS to PPO, would they have to derive empirical updates just for PPO? It would be good to get a better sense of how easy/difficult it is to plug in EXPRS to different RL algorithms
- Is the theoretical analysis in Section 3.3 mainly limited to the chain environment or does the general utility hold beyond it? I noticed that the appendix includes further details on how you arrived at cost for the different cases, which would be good to mention in the paper.
- What is the reasoning behind using the baselines that you show in your experiments? Also, is LIPRG included there (it could be that SELF-RS is similar to LIPRG but that is currently unclear)?
- It looks like for many of the experiments that you performed, EXPRS and some of the baselines fall within the deviation of each other. Could you comment on where you see clear advantages of EXPRS and where it is more difficult to distinguish its performance from other methods.
limitations: I think the paper could be improved by a more thorough discussion of the limitations of EXPRS, which are only briefly touched upon in the conclusion. It would be good to get more detail on what the authors anticipate would be needed (in general terms) to perform the extensions that they mention (continuous spaces, converge, other agents). Given space limitations, it would be OK to include this in the appendix even though I would recommend it to be in the main paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper takes a holistic approach to reward shaping in RL, and has two main contributions: (1) simplified meta-gradients for updating a parameterized intrinsic reward function, which generalizes beyond policy gradients algorithms, and (2) the combination of said intrinsic reward function with an intrinsic exploration bonus. The approach is self-supervised, and does not require expert knowledge. The combination of different intrinsic signals is studied theoretically in a chain environment, and then evaluated empirically on several low-dimensional MDPs, both in tabular settings and with function approximators.
strengths_and_weaknesses: **Strengths**

* The paper is well-written and easy to follow, despite the large quantity of preliminary derivations.
* The integration of both reward shaping terms is an interesting concept, and the analysis in Appendix D sheds nice insights on the synergy of the two.
* Empirical performance appears strong across the environments considered in the paper, and the evaluation protocol seems robust.

**Weaknesses**
* In its main body, the paper does not always clarify which derivations are original, and which are reported from previous works (see following comments).
* A number of separate concerns on the introductory writing and baselines would need to be addressed to further improve the quality of he paper (see following section, main comments). 
questions: **Main comments**
* The title is not very informative. I would suggest to consider something that clearly establishes the existence of two shaping terms, as well as the generalization beyond policy gradient algorithms.
* Similarly, the abstract is also vague. I would recommend to include the concrete contributions of the paper, especially mentioning that the authors generalize self-supervised reward shaping techniques to general RL algorithms. Moreover, the sentence on line 8 does not clarify that two different terms are added to the environment's reward signal.
* Despite generally stating in the abstract that they would 'theoretically showcase the utility of our reward shaping framework', the authors only present a limited theoretical analysis on a particular family of MDPs. Although this is not negative per se, it should be made clear right from the start.
* Is there a reason why LINRPG is not included in the empirical evaluation on REINFORCE? In would be particularly interesting to see how SELFRS compares against it.
* Section 3 could further benefit by clarifying that the derivations until line 163 are reported from existing works. Moreover, it should anticipate that the derivation will skip non-trivial steps, which are however reported in the Appendix.
* On line 190, the paper mentions "distractive" rewards, but I fail to see how the rewards in this particular environment are distractive.

**Minor comments**
* line 20: To the best of my knowledge, transfer learning approaches do not need domain-knowledge, and can rely solely on data. Can this be clarified?
* line 131: The authors should anticipate that superscripts (e.g. SelfRS) are dropped from this point on, as they do in the Appendix.
* line 157: Where is this smoothness condition derived from? is it an assumption?
* line 209: I would suggest rewriting the bound as $2^{n_1} - 1$ for ease of reading.
* possibly due to my lack of familiarity, I would ask the authors to explain why the equality between the last line on page 17, and the first line on page 18 holds.

**Nitpicks**
* line 69: The notation for T does not clarify that its output is a probability distribution over the action space (i.e. integrates to 1).
* line 234: green-star -> blue-star
limitations: Authors present a sufficient discussion of limitations in their conclusion.
review_title: null

Meta-review: metareview: This paper proposes a self-supervised reward shaping framework. This is an interesting contribution, that is well validated theoretically and empirically. The paper is generally well-written.

I commend the authors for their helpful and positive interaction during the rebuttal period. I believe that this sufficiently addressed most of the concerns of the reviewers, particularly after the updated version. The only remaining concern for me is the lack of experiments on larger domains more commonly used in the field, but I do not consider this a critical flaw of the work. On the other hand, this paper introduces novel ideas which are well formulated and supported. 

My view is that this would be a positive contribution to NeurIPS.
confidence: Certain
Decision: Accept