Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper focuses on the exploration-exploitation trade-off in reinforcement learning. In particular, the authors aim to devise a method that automatically adjusts the importance of the intrinsic reward component (exploitation) and the extrinsic reward component (exploration). To this end, they pose an (extrinsic optimality) constrained policy optimization. Then, they write the Lagrangian dual problem and solve it iteratively. For implementation purposes, they leverage tools from TRPO/PPO literature. The proposed EIPO method is compatible with any intrinsic reward method; for experiments, they use RND. Finally, they conducted an extensive empirical investigation of their proposed method on all 61 Atari games.    
strengths_and_weaknesses: Strengths:

The proposed method is interesting and novel. The existing intrinsic reward-based exploration methods (e.g., RND) need to heavily tune the parameter $\lambda$ in $r_E + \lambda r_I$ for each environment to perform better. EIPO alleviates this manual tuning effort and automatically adjusts the importance scale $\alpha$ between the intrinsic and extrinsic rewards via optimization. Further, EIPO is compatible with any intrinsic reward method. In this work, the authors do not directly learn or optimize the intrinsic reward $r_I$.  

The paper is overall well written; the motivation and the story are well conveyed. It was easy to follow.

On the empirical side, the authors have done a large-scale study on all 61 Atari games and reported the results using rigorous metrics as prescribed in [18]. In particular, the empirical results validate the following: (i) the probability of improvement EIPO over PPO is higher than the baselines (variants of RND), (ii) EIPO performs better than all the baselines in the majority of games (as well as in terms of strict probability of improvement), (iii) EIPO performs comparable/better than heavily $\lambda$-tuned RND (for each environment). These empirical findings make the paper strong. 

****

Weakness:

The paper is missing a detailed discussion of related work. 
questions: Question: 

How does EIPO relate with (or compare against) the line of works in [Zheng et al. 2018; Hu et al. 2020] that also consider automatic learning/adjustment of intrinsic reward component? 

[Zheng et al. 2018] Zheng et al. On learning intrinsic rewards for policy gradient 376 methods. 2018.

[Hu et al. 2020] Hu et al. Learning to utilize shaping rewards: A new approach of reward shaping. 2020.
limitations: Their proposed method does not seem to cause any direct potential negative societal impacts.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors pose the problem of exploration using intrinsic motivation as one of constrained optimization.
Naturally, the goal of the agent maximizing the intrinsic+extrinsic reward should be to perform AT LEAST as good as an agent simply maximizing the intrinsic reward.

As such, they present a constraint where the E+I agent needs to perform at least as good as the E agent when measured only on E (extrinsic) rewards.

Empirical evidence shows this method indeed works.
strengths_and_weaknesses: Strengths:

- Motivation is clear. The approach is solid.
- Empirical results look very good.
- Paper is overall well written.

Weaknesses:

- Wouldn't expect strong theoretical justification, but overall this is an optimization objective with 3 parts that can be presented as a 2-3 time-scale stochastic optimization scheme and convergence can be analyzed. Convergence analysis might shed some light on section 3.3 which seems a bit hard to comprehend.
questions: My main question is around section 3.3 (and 3.2)

Why collect data using pi_{E+I} when optimizing pi_{E}? Why not optimize pi_{E} when using data collected by pi_{E}?
Why not optimize both agents in parallel at each time? Is there a restriction against optimizing pi_{E} and pi_{E+I} in both max and min stages, just making sure to use the appropriate importance sampling ratios?
limitations: -
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: Difficult exploration tasks which rely on sparse rewards can be particularly challenging for traditional RL methods. Injecting intrinsic reward functions can increase reward density, but also introduces bias that can hinder performance in more tractable exploration tasks. Regulating this bias often reduces to a balancing act between intrinsic vs extrinsic reward influence using manual hyperparameter tuning. This paper proposes an objective function that explicitly minimizes the difference between the learned, combined intrinsic + extrinsic policy and an optimal policy under only the extrinsic reward. The objective function is reformulated using lagrangian duality and the policy learning is constrained using TRPO to provide tractability. The method is then evaluated on 61 Atari exploration tasks that vary in difficulty. Results show that the proposed method can match state of the art performance on difficult tasks while still achieving PPO-level performance on simpler tasks, all without manual tuning of the reward balance.
strengths_and_weaknesses: ## Originality

The core concept is simple but elegant.  Formulation of the minimization constraint between mixed- and extrinsic-trained policies, reformulation using langrange duals, and the use of a TRPO-like objective to improve trainability were all creative. Overall, I like that the idea is approachable but very effective.

## Quality

The overall quality of the paper is excellent, with a few minor caveats:

- I thought the abstract was a little weak. It's more pronounced because the rest of the paper is very solid, so the abstract feels thin by comparison. A few suggestions:
	- Some may not be familiar with "intrinsic rewards," so maybe highlight the link with reward sparsity. 
	- Maybe include a few more details about your approach beyond just "constrained policy optimization"
- Very minor, but each word in your title should be capitalized except for "via".

On the positive:

- As mentioned above, the core concept is solid and well-presented.
- Detailing of the methodology was excellent, from preliminaries through implementation details.
- The number of experiments and the detail of the analysis were great.

## Clarity

The paper was clearly written and approachable. The description of the problem and the formulation of the solution follow a coherent narrative and are straightforward to understand.

## Significance

I don't feel like anything in this work is particularly ground breaking, but rather an elegant solution to a specific problem. That said, I do appreciate the breadth of evaluation and it will be interesting to see this approach applied to more real-world scenarios.
questions: I have no major questions or suggestions, other than the minor suggestions provided above.
limitations: I would have liked to see more discussion around the limitations of this approach, or possible applications where this may not be appropriate.
review_title: null

Meta-review: metareview: Balancing between extrinsic rewards and intrinsic rewards is an important challenge for exploration in RL. This paper proposes a simple yet effective way to automatically adjust the balance between them. The large-scale empirical result across 61 Atari games shows a strong improvement over the baseline approaches. All of the reviewers agreed that the proposed method is novel, and the empirical results are convincing. The reviewers had no major concern about the paper. Thus, I recommend accepting this paper.
confidence: Certain
Decision: Accept