Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a method to learn interest state function via meta-gradients in order to do emphatic RL. This interest function essentially simplifies the credit assignment problem and highlights 'more important / interesting' states.

The authors study their method in multiple settings: toy domain, continuous control and discrete domains as well as in the transfer learning setting. The signal is quite clear, the method which authors propose seems to always provide the benefit compared to standard RL approaches.

The authors study different ways of constructing interest function and provide informative ablations over these, with somewhat limited explanations.

Overall, the paper is clearly written, the ideas are easy to understand and the results look good.
strengths_and_weaknesses: Strengths:
* Clearly written paper
* The authors studied a novel approach for RL and credit assignment via meta-learning of the interest function and provided solid experimental evidence that the approach can improve RL
* The experiments are quite broad, demonstrating the applicability of the method in different scenarios: discrete control, continuous control, transfer learning.

Weaknesses:
* It is not clear why the authors didn't provide `TD error interest` for MinAtar. Given the results on the toy domain, this approach may actually perform better than meta-gradients.
* Minor writing / visualization details. The legend colours on the Figure 2 are very hard to read. 
questions: * How does the TD error interest method perform on MinAtar?
* Why do the authors claim that meta-gradient performs better on Figure 2, where it is clear that TD error interest converges faster and to a better value?
limitations: Limitations are clear from the paper
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper studies the problem of learning the interest function, which emphasizes the importance of different states differently in RL objectives. The interests of different states are typically set to be all 1 in existing approaches. And this paper seems to pioneer the study of the adaptive interest function (the paper didn't claim it but from its discussion of existing works it appears to me that the authors believe that no other works have studied this problem before).

The paper proposed a meta-learning approach to the problem. Specifically, the interest function is learned to maximize an objective, which is an off-policy control objective. The paper proposed an off-policy prediction algorithm and an on-policy control algorithm. Empirically, the off-policy prediction algorithm almost matches the performance of an ETD algorithm with TD error as the interest. The on-policy control algorithm is significantly better than the PPO algorithm and the other Meta-learning algorithm.
strengths_and_weaknesses: Strengths:
In general, the paper is easy to follow.
Adapting the interest function seems to be interesting and important, and is not well understood in the literature.
The proposed algorithm uses an elegant idea.
Empirical results show some interesting patterns, which would be helpful for understanding the proposed algorithm.

Weakness:
The motivation of the idea, the description of the algorithm, and the explanation of the empirical results are not clear enough. Some important questions need to be addressed to evaluate the paper's contribution. See the Questions section for details.

-------------------------------------------------------------------------------

I have read the authors' responses and feel that most of my questions were answered, including the key question about explaining how interest function influences the learning process. Therefore I decided to increase my score to 6.

questions: Overall, could you explain how different interests influence the learning process in prediction and control settings respectively?

line 21 A possible solution could be to selectively emphasize certain updates, for example through a state-dependent interest function.
Why is this a possible solution?

Are there any existing methods that adaptively choose the interest? Is your work the first one studying the adaptive interest function?

It is not clear from the introduction section the prediction or the control setting or both is considered in the paper.

When designing a fixed interest function for SARSA with emphatic weighting, it would be advantageous to emphasize the states on the left and de-emphasize the states on the right, as a way to try and avoid the sub-optimal solution.
Why?

Can interests go arbitrarily large?

In figure 1, no algorithm eventually converges to the optimal policy (left), right? Why is that?

In figure 2, how did you compute the MSVE?

Isn't figure 2 showing that the proposed method is slightly worse than the ETDLB method with the TD error as the interest in the tested problem (c.f. subfigure c)? 

I didn't find in the paper a description of the off-policy policy evaluation algorithm (the one you used to produce figure 2).

It seems that the interest function is shared across all sub-tasks? If so, why is it reasonably to share it? Why do you consider the multi-task setting instead of the single one?

Is your off-policy policy evaluation algorithm a batch algorithm? Are TDRC, Vtrace, and ETDLB online algorithms or batch algorithms?

In figure 3, I see that during the entire training period, cells near the hallway cells have high interests. I can also see that, at the end of training, the further a state is from from the hallway state, the lower interest the state is assigned. The paper explains that "as the target policy does not visit them often and not many states bootstrap from them, it is less important for them to be accurate". This is a reasonable explanation. But given that the pattern looks pretty close to the value function, it can also be hypothesized that higher state value results in higher interest. If a state has a high value but is not frequently visited by the target policy, would the interest be high?

What is the on-policy control algorithm? 

When comparing with PPO and the Meta Learned Target algorithm, do you use the same amount of buffer to store past experience and the same amount of computation?
limitations: Overall I would need the questions listed above to be addressed to tell the paper's limitations. The paper does not have a potential negative societal impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper extends work focusing on emphatic TD learning. While the original formulation of emphatic TD learning included an interest function that prioritises updates to certain states, in practice this was always made uniform or required expert knowledge to set. The idea here is to (meta) learn the interest function as training progresses - an inner loop follows regular emphatic TD learning with the learned interest function, while an outer loop next runs to update the interest function's parameters. Empirical results are provided in several domains, including pixel-based ones, continuous control tasks and linear function approximation settings. Results indicate that an adaptive interest function learned over time seems key to applying these approaches to on-policy learning, while the method also outperforms a fixed interest function, as well as other meta-parameter learning approaches. 

strengths_and_weaknesses: \+ I found the paper to be extremely well-written and well-motivated. Every time a sentence raised a question in my mind, the very next sentence explained or justified it to me! 

\+ The main idea is conceptually simple (in a good way) and the paper does a good job of evaluating it in several different kinds of domains. 

\- I realise that space is a large issue here, but I found that I had to jump back and forth between the appendix and the main paper quite often. While much of this is extra discussion, a few times it was for key information, such as the details of the domain. In particular, I'm still not 100% certain if the reward function for the four rooms domain is a sparse one (and what the exact values are), even having consulted the appendix.  
questions: 1. In the MinAtar experiments, I was expecting to see the TD-error heuristic which was used in the chain walk (and popped up again in MuJoCo). Is there a reason that it wasn't included in this set of experiments?

2. On Line 292 it states: "We explain in detail this baseline in the App. F and theoretically show that learning the interest function is not simply (sic) special case of their approach". However, I cannot find a discussion of this anywhere. 

3. In the motivating example, I'm not entirely clear on why the choice of base learning algorithm was SARSA. Is this because the focus, later on, is on-policy learning? Otherwise, it seems strange to apply the various interest strategies on top of a learning algorithm that doesn't quite fit. 

4. Is there any intuition for why the MSVE should diverge for larger learning rates with the TD error interest on H4R8T, but not 4R8T?

As an aside (certainly not necessary to include here), the focus on the robustness of hyperparameters like learning rate could potentially benefit from applying the methodology of [1].

Minor comments:

1. It would be helpful to move Figure 4 to the top of the next page. Currently, it appears before Fig 3 and you have to jump between pages to read the associated text. 
2. Without knowledge of the exact reward function, Figure 3 takes some puzzling out. My guess is that it is sparse (1 at the goal, 0 elsewhere) but that is simply a guess because I do not see it listed anywhere

3. Many of the references list the arxiv or preprint versions of papers. Please check these all, since many have been published. For one example: [59] appeared at NeurIPS 2019. 

Minor typos:
L184: "as THE inner objective"
L239: algorithm -> algorithms

[1] Jordan, Scott, et al. "Evaluating the performance of reinforcement learning algorithms." International Conference on Machine Learning. PMLR, 2020.
limitations: N/A
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes to use meta-gradient to adapt the interest function in emphatic RL algorithms. Empirical study covering off-policy policy evaluation with linear function approximation, on-policy control with nonlinear function approximation, as well as transfer learning confirm the efficacy of the proposed method.
strengths_and_weaknesses: Strength:
1. This paper is well motivated and targets an important problem in RL. The use of meta-gradient is straightforward but appears novel to me.
2. The paper is well organized and easy to follow. Empirical study covers a wide rage of settings. 

Weaknesses:
1. The jump from off-policy prediction to on-policy control seems a bit abrupt to me. Emphatic methods are originally designed for off-policy prediction so starting with Section 4.1 seems natural to me. But is there any particular reason that the control experiments are in the on-policy setting, especially given the success of [22, 23], the motivating example in Section 3.1, and the derivation in Section 3.2. I feel adding more off-policy control experiments would significantly strengthen this work.
2. The meta-objective in Equation (8) seems not well motivated. Why not simply use the episodic return \sum_s p_0(s) v_\pi(s) where p_0 denotes the initial distribution? After all all the domains used for control is episodic instead of continuing and the agents are evaluated against episodic return. Is the ease of optimization the main consideration for the use of the excursion objective? I feel more discussion about this would also strengthen this work.
questions: See "Weaknesses"
limitations: n/a
review_title: null

Meta-review: metareview: Well-written and interesting paper that meta-learns the interest function in emphatic RL, rather than using a fixed interest function. The idea is well-motivated and a comprehensive empirical study is performed. There was an an active discussion among the reviewers and authors, in which the authors addressed the various reviewer questions effectively and which resulted in an updated presentation and an increase in scores. Overall, a clear accept. 
confidence: Certain
Decision: Accept