Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a new active reward learning method called IDRL, designed to select the most informative query for identifying an optimal policy among a set of plausibly optimal policies. First, IDRL selects two candidate policies that maximize the entropy of the difference in expected returns given past queries. Then IDRL chooses queries that reduce the uncertainty of the return difference most among a set of candidate queries. Updating the reward model using the selected queries rather than queries maximizing the information gain about the reward stimulates identifying the optimal policy. Experimental results show that IDRL requires fewer queries than the considered baselines and works with different query types.
main_review: (1) Originality and significance: \
This work changes the main focus of active reward learning from achieving a low estimation error of the reward to directly improving the policy which the reward model induces. This approach can be a new direction for further research in the active reward learning area.

(2) Quality: \
This paper is supported by theoretical background on Gaussian distribution and GP. IDRL scales to Deep RL setting with approximation the posterior as Gaussian using Laplace approximation, making the algorithm tractable (including computation of entropy and information gain).

Here are some related questions. \
Q1. Can you show any statistical test for the results in Figures 3 and 4? \
Q2. What is the performance tendency on the number of candidate policies (although too many candidates cannot be used due to the computational cost)?

(3) Clarity: \
Here are some questions for clarification. \
Q3. In Section 7.5, "it(IDRL) generates the candidate queries rolling out the currently optimal policy and the candidate policies(356-357)." On the other hand, Algorithm 5 of Appendix E.1 uses the currently optimal policy only (line 23)." Does the author say that the former is IDRL and the latter  'IDRL without candidate policy rollouts'?\
Q4. In Algorithm 5 of Appendix E.1, does line 25 mean that the Hessian is calculated after (virtually) updating DNN with $q_i$ and its predicted response $y_i$? \
Q5. In Section 7, "the agent's queries are answered with simulated feedback based on an underlying true reward function(219)." Could you clarify how 'simulated feedback(expert)' runs (or is implemented) in detail?


---------------- Post Rebuttal --------------------

I read the authors' response carefully and appreciate the detailed explanations. I am inclined to accept this paper.



limitations_and_societal_impact: The authors have addressed the limitations and potential societal impact.
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors propose a Bayesian method for learning the reward function in an RL setting.  They study the setting in which the algorithm can query experts.  The form these queries can take is quite general.  They analyse their algorithm theoretically and empirically.  Their key insight is that queries should be focused on enabling the reward model to be used to find the best policy possible (as efficiently as possible), rather than decreasing the reward model’s uncertainty in general.

main_review: --BEGIN UPDATE--

After reading the other reviews and the author responses, I maintain my score; this is an outstanding paper.  I think the improvements the authors have promised will make it even stronger.

--END UPDATE--


The amount of details included about the experimental setup is outstanding (main paper and Sections D-F) and the additional experiments/theory in the appendix are very nice.  The experiments have outstanding breadth and depth and the results in Section 7.3 in particular were interesting and well-thought-out.

Small issues/edits:

-\mathcal D is first used about 5 lines before it is defined, it would be easier to read if the definition was moved before the first use.

-There are some definitions that can only be inferred or are stated informally, and should be more formally defined.  For example, the belief about the optimal policy, y hat and the set within which it exists, G hat (see next issue below; I’m not sure which “G” “G hat” estimates, a very literal reading makes me think it estimates the first definition, but the context makes me think it estimates the second definition), etc.

-The definition of G(pi) is overloaded; this is confusing and should not be done.  (This maybe contributes to the confusion in the point above.)  Edit: On my second read-through, it occurs to me that the second “definition” of G(pi) (on line 128) is actually not a definition, and that that notation denotes the dot product.  This would explain a lot of my confusion above. Two suggestions if this is the case: 1) use \coloneqq for definitions (for example, the definition of G in Section 3) to clearly distinguish definitions from equalities, and 2) tell the reader on line 128 that that notation denotes the dot product (at first, I assumed it was a vector or a tuple or a contention of vectors, or something like that; in other words I thought that G was being overloaded, and the new definition included f^pi and r rather than just their scalar dot product).

-Line 8 of algorithm 1 (I think it’s the update to the belief about the reward function) is a bit confusing to me at a high level; high-level intuition/clarification in the pseudocode or in the relevant text (the second to last paragraph of Section 4) would be nice.

-The error bars of Figure 2 are not specified (I assume they are standard error based on the other figures).

-Please specify the number of trials (random seeds, or “runs”) used for Figure 2.  (This would be a major issue, except that the bars, which I assume are standard error, see above, seem to indicate that there is a reasonable amount of statistical significance in this experiment.)

-The caption of Figure 6 in the appendix has a dead reference (to a section or a figure perhaps).

Major issue: Line 114: p^* is completely undefined. “I” (the mutual information?) is also undefined. Because of these undefined symbols, I’m not following the next parts (e.g., the “two undesirable properties”) as well as I would like.  Related minor suggestion: Assuming the latter is mutual information, perhaps there should be a little background on this; I suspect readers are more likely to know the basics of MDPs than the basics of information theory (and the authors have rightly included MDP basics).  This can be a short appendix section if there’s no room in the main paper.

Question/suggestion: Throughout the paper, you assume unlimited and cheap access to the environment and its transition function (minus the reward function), correct?  There’s nothing with this problem setting, but it might be worth explicitly mentioning this point in Section 1 or 3.

In summary, I strongly recommend an accept.  This recommendation is contingent on the authors addressing the (easily-fixed) issues I am concerned about.  However, my confidence in my assessment is low; so my assessment may change depending on the feedback of/discussion with the other reviewers.

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors introduce a query selection strategy for reward learning in reinforcement learning. This strategy can use different query types and is not restricted to pairwise trajectory queries, as many other approaches. Additionally, the strategy is also considering the environment dynamics. The papers main contribution is a method, that reduces the number of required queries, besides the aforementioned aspects. The authors also show, that the approach is also applicable to Deep RL techniques.
main_review: The algorithm is based on well known entropy-based learning approaches, but combines this idea with policy awareness. This is an interesting idea and a relevant step forward, despite the fact that the elements themselves are not novel. As this algorithms builds on principles with well established theory, it can be considered sound and is clearly described. Related work is sufficiently covered, differences mentioned and the experimental section is also mostly well structured and understandable. 

The novel sections (wrt. "Improvements Made") are a clear improvement of the paper, but could be improved. As example, the algorithm 5 from the appendix is relevant for understanding the DNN setup and should be moved to the main paper, or at least, explained in more detail. A more substantial limitation is the discussion of the results from the Deep RL setting (7.5). From the appendix, it is clearly visible that IDRL is subject to failure modes (e.g. Swimmer-v3) or premature convergence (Ant-v3, InvertedDoublePendulum-v2). The same is true for most other approaches, but potential reasons are not really discussed. The section (together with 7.4) is sufficient for fulfilling the claim, that it is computationally possible to scale the approach to higher dimensional problems, but performance remains in question. 

The rest of the evaluation was performed on simpler domains, enabling the authors to disregard the scalability and exploration issues of the RL approach (by fully recomputing the candidate set multiple times). This is very interesting, as it allows to specifically determine the effect of the query selection strategy, which is the papers main contribution. It should also be positively considered, that they compare against a broad range of baseline methods. However, the capability of dealing with non-linear reward queries was not evaluated, but it is clear, that it is theoretically possible.


limitations_and_societal_impact: The authors addressed several limitations, but there is quite a bit room for improvement. This mostly relates to section 7.5 (as stated in the main review). It is especially interesting if the mentioned problems are induced by the DNN approach,  the limited candidate update rate or the higher dimensionality. Multiple ideas come to mind: Using scalable, exact GP approaches [Exact Gaussian Processes on a Million Data Points, Wang, 2019], Kernel Neural Networks with GPs [Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awareness, Liu, 2020] or running SAC from scratch often. However, i acknowledge, that this would introduce substantial runtime demands, making it an interesting study for future work.
Lastly, evaluations concerning the non-linear query capabilities should be added
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Information Directed Reward Learning (IDRL) actively selects queries that are most informative at distinguishing policies in terms of their value. By contrast, prior work has typically tried to maximize information gained about the reward function itself -- but some parts of the state space may be unreachable, or never visited by any plausible optimal policy, and so learning about them is irrelevant for task performance. IDRL has a pleasing theoretical motivation, although practical implementations of it require considerable use of approximations and heuristics. Additionally, IDRL is compatible with a broad variety of reward learning queries (e.g. preference comparisons, labeling trajectories with returns) whereas most prior work focuses on a single feedback modality. Experimental results show significant improvements in sample efficiency from use of IDRL relative to a variety of baselines.
main_review: The paper tackles an important problem, sample-efficient reward learning, with a novel approach that has some theoretical justification and promising empirical results. The presentation is in general very clear, and the experiments seem to be carefully executed: e.g. I was pleased to see the environment termination conditions were removed in section 7.5, which is a common pitfall for work in this area.

My main hesitation lies in the scalability of the approach. The GP approach, while having pleasing theoretical properties, seems intractable to scale without significant hand-designed feature engineering -- e.g. section 7.4 required pre-training fixed policies for Ant, and there was a relatively simple mapping from observations to reward. The deep RL approach seems potentially more scalable, but the approximations used are so substantial that the method loses much of its theoretical justification. The empirical performance of deep IDRL goes some way to allaying this concern, but the improvements are modest and inconsistent between environments, on occasion even doing worse than random sampling.

Overall I vote to accept the paper. In particular, the key idea of choosing queries based on how this affects the return achieved seems like an insight that will prove useful in the long-term. I am unsure if the IDRL method itself will see adoption, but it is certainly plausible, especially as Bayesian NNs improve the deep IDRL method may be able to scale.

# Originality

Moderately original. The algorithm is new, but operates in the existing (but understudied) paradigm of active reward learning. The key idea of focusing on information gain on policy-relevant factors rather than reward itself has been considered in other contexts before (e.g. Russo and Van Roy, 2014 which is cited in the appendix). But as far as I know it has not been applied to reward learning in general sequential decision making problems before, and the focus on difference in return between policies is new (although I'm not convinced it's quite the right objective).

# Quality

The paper generally seemed to be carefully executed. I did not check the proofs in the appendix but all claims and derivations seemed plausible. Details of the experimental setup were for the most part clear, the baselines were reasonable and when a direct comparison could not be made the authors highlighted this limitation.

# Clarity

While the paper was in general easy to read, there were a few parts that seem confusing or ambiguous:
  - Use of the phrase "potential reward function" at line 248 and elsewhere is ambiguous. I think you mean something like a candidate reward function. But it could also be misunderstood to mean a reward function based on potential shaping. I would suggest rephrasing.
  - It is unclear what the reward function in section 7.4 actually is -- can you add more detail or least reference a relevant section in the appendix? In particular, I am not sure what is meant by a "one dimensional grid". Furthermore, you say it is "different across the grid" -- different in what way?
  - You say IDRL supports multiple query types, but it's unclear to me exactly how flexible it is. Could you choose between queries of different types (e.g. preference comparisons vs reward labels) on the basis of informativeness? If so, why does no experiment do this? Multi-modal feedback seems particularly promising to me, as some feedback sources (e.g. demonstrations) are likely to be more sample-efficient at the start of training when uncertainty everywhere is high, whereas other feedback sources (e.g. preference comparison) are likely to be more informative towards the end of training when it is necessary to disambiguate specific areas. This could be a key advantage of IDRL that I don't see any mention of currently.

There were also a few more minor points, which I discuss at the end of the review.

# Significance

The problem domain is important. I find it unlikely that IDRL will be of direct use to practitioners today due to the computational cost and complexity of implementation -- it may often be cheaper to just gather more data -- but I expect this to improve over time.

-----
# Minor typos etc

- Figure 1: overall I really liked this figure. However I initially didn't parse the square around the pear as being an additional wall, and so was confused why there were only two plausible optimal policies. It didn't take long to figure out, but it might be worth being explicit in the caption: e.g. “It can only move 4 timesteps in the gridworld, *cannot move through walls (in black)* and collecting more food is always better.”

- Line 72: “(MDPs, Puterman, 2014)” is confusing – is MDP an author? Use a semicolon instead to separate: (MDPs; Puterman, 2014)”.

- Line 79: $r_t$ is not defined. Although clear from context, this is still worth defining, e.g. you could add it in the sentence starting “In an MDP, ...”

- Line 114: what is $\hat{p}^*$? I think this is meant to be $\hat{\pi}^*$?

- References: some capitalization issues, likely due to missing {} around words to preserve capitalization in your .bib file. Line 396: “Openai gym”, line 423: “atari”. Please proofread this section.
limitations_and_societal_impact: One significant limitation of the work that is unfortunately not discussed is that it may harm generalization relative to other active learning criteria, such as maximizing information gain over the reward. You highlight generalization as a significant advantage of reward learning (line 44 onwards): "policies tend to generalize poorly between environments" while "reward functions are a more robust representation of desired behavior". But if you seek only information that lets you discriminate between policy return, you are liable to lose much of this generalization benefit. Indeed, consider Figure 1: IDRL specifically avoids querying about the pear. If at deployment the wall around the pear is removed, an IDRL agent will be highly uncertain what to do, whereas random sampling or better yet EVR would have learned about the pear.

There's a fundamental tradeoff here, and I think IDRL may well be the right choice for many applications, but it seems disingenuous to simultaneously claim the benefits of reward learning in terms of generalization and systematically avoid gathering information that would help generalize.  I did find it interesting that the cosine similarity (Figure 3) nonetheless was higher or comparable to existing method, but I would need to see comparison of learned to ground-truth rewards in more environments to be convinced, or explicit test of transfer to new environments.
review_title: null

Meta-review: This paper proposes a new active reward learning algorithm, where where the agent is required to reason about unknown reward function by querying an expert. The main idea is to select queries that maximize the information gain about the difference in return between policies with high uncertainties of their return difference. The results show that the proposed method can achieve similar or better performances with fewer queries and can work with different types of queries unlike the prior work.

All of the reviewers appreciated that the proposed objective that focuses on improving policy as opposed to reducing reward model error, which has been done in the prior approaches, is novel and technical sound. The paper also presents both a theoretically exact implementation and a scalable approximation for Deep RL. The results look solid in that the proposed method is much more query-efficient and can work with more types of queries compared to the baseline approaches. Overall, this is a solid and neat paper. Therefore, I recommend to accept the paper.
Decision: Accept (Poster)