Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work analyzes the effect of model accuracy on reward inference accuracy when fitted to human behavior. Specifically, it shows it is possible that a small model error leads to a very large error in inferring the reward. However, this scenario is unlikely. The authors backed their claims with simulations and real human data.
strength_and_weaknesses: Strength: 
The paper tackles an important problem. It contains both theoretical and empirical results. Moreover, the experiments contain both simulation and real data.

Weakness:
 My main concern: I did not find IRB approval information on the human experiment. If there is, it should be mentioned, if not the authors should explain why it is not necessary in this case (and should be validated with the conference chairs). Also, the details of the experiment and instructions to the demonstrators should accompany the paper. 

Worst case v.s. average (end of page 3, beginning of page 4): If I understood correctly the worst-case error is based on all states but the average is based on available states in the data. This looks inconsistent, what is the rationale behind it? 

clarity,_quality,_novelty_and_reproducibility: The paper is clear in most parts (except the details of the human experiment). I also like both theoretical and experimental contributions. Given the code, simulations are reproducible, but I did not find any statement on the availability of human data. 
summary_of_the_review: It is a good solid paper to me both in terms of theoretical analysis and experimental results, but human data experiment should be more clear.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper addresses the question of whether misspecification of human behavior models for IRL can lead to detrimental effects on reward inference accuracy, and provide theoretical guarantees showing that while adversarial situations can be constructed such that small human behavior models lead to "catastrophic" inference errors, under a log-concavity assumption the effect is more stable. The authors then provide empirical results showing a correlation between human model accuracy and reward inference accuracy.  
strength_and_weaknesses: 
Strengths: the paper is very well-written, and I appreciate the attempt to explore different types of human irrationality (e.g. myopia), and the experiments include real human data (albeit w/ a very simplified form of misspecification via simple re-weighting)

- I am a bit confused why measuring reward misspecification by distance in parameter space works except for very restrictive, simple linear reward models. If the reward function is a NN, then it is well known that two networks that behavior similarly can have widely different parameters. Any necessary assumptions here should be clearly stated. 
- I am likewise confused why KL diverge between demonstration policy and the human model is the right way to measure distance. Given the motivations discussed in the paper of modeling humans via noisy-rationality, systemic biases, etc., it seems like humans may deviate from an idealized model in ways much stronger and more state-specific than the KL measure assumes. 
While I like the idea of fitting certain types of irrationalities (myopia and illusion of control) into this framework, it feels a bit cherry-picked to only study irrationalities that satisfy the specific notion of distance the authors chose. 
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is very well written
Novelty: There is no new methodology to evaluate novelty for, so it is mainly the question (how harmful misspecified human models are for reward inference) that is novel
Reproducibility:  Experiment details are included in the appendix. 
summary_of_the_review: 
My primary concern and reason for rejection is the rather simplified assumptions the authors make in order to "get the theory to work".  It is also unclear what the ultimate takeaway here is - if the goal of studying the sensitivity of reward inference is to make some kind of claim on whether reward learning is a valid direction, then only studying restricted forms of human model misspecification does not really lead to any meaningful insights. I think a more clear, isolated section stating the author's assumptions is necessary for acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Inferring human reward functions requires 'human models' which specify how a human acts given their reward function. This paper studies how errors in the human model translate into errors in the inferred reward. First it shows a pessimistic result: in an adversarial setting, a small error in the human model can lead to a large error in the inferred reward. However, under some assumptions the paper shows that the reward error can be bounded linearly in the human model error (for particular choices of error metrics). The paper show that this result applies to several known human biases. The result matches up with experiments in two environments. The authors thus make an optimistic conclusion: reward inference with misspecified human models is not very difficult. 
strength_and_weaknesses: The paper studies an important problem: To infer reward functions, a human model is necessary, but it is infeasible to infer it accurately. This is because inferring it would at least require the human's reward function, which is initially unknown. This problem becomes more important as AI systems attempt more complex tasks where human behavior is nowhere near optimal and cannot be modeled as optimal.  If we know the relationship between reward errors and human model errors, we can better understand if we can improve reward learning by learning better human models. This understanding also helps clarify if reward learning is a viable strategy at all for complex tasks where humans are increasingly suboptimal. Both the optimistic and pessimistic result yield useful insight on these questions. The results are clear and succinct and to my knowledge novel, providing a clearly significant improvement in our knowledge of the important question at hand. 

Weaknesses

In the first and last section, the authors make an optimistic conclusion overall, saying that their results imply that reward inference with misspecified human models is not very difficult. The positive conclusion needs some strong caveats stated in the first and last section to avoid false impressions. Though I think these caveats should be stated, I don't think they detract, and in fact add to, the value of the paper. By order of importance:

Firstly, the introduction speaks of mild assumptions for the positive conclusion that a slightly misspecified human model will lead to small errors in the inferred reward. However there is an assumption baked into the choice of error metric for human models: the KL(human_policy | model_policy). To assume that the human model error is small, this KL divergence assumes that the human policy would *never take an action that the human model places ~0 probability on*. Otherwise, the KL will be large, allowing large reward errors, and challenging the paper's positive conclusion.

Aside from noting this caveat, the authors could reproduce their results with a sufficiently different metric such as the reverse KL. Furthermore, the authors could note that having a small forward KL requires that the human model 'covers' the support of true human, as otherwise the KL would be large. This is an interesting result, as it means that, all we need is to learn human models with good coverage. (And it is not a problem if the human model sometimes takes actions that humans would not take). 

Second, the authors construct a case where the inference error is large, but then argue that their construction is unlikely to occur in practice and that this supports their optimistic conclusion. However, they have not ruled out that other constructions exist. Some of these may be more likely in practice. 

Third, the paper could justify its choice of reward distance metric, the squared distance between reward parameters. In particular, a small error in inferring the reward parameters might still lead to a large error in the resulting return and policy. 



clarity,_quality,_novelty_and_reproducibility: The writing is clear and the theory well motivated which was helpful. However, log-concavity could be better explained, e.g. with (counter)examples in the main text. (For example, I wonder if log-concavity is always satisfied by optimal policies?) Furthermore, it would be useful to summarize the intuitions for Theorem 2 and 3 without reference to the appendix. 

In the literature on reward inference, this appears to be the first paper to study the relationship between errors in the human model and inferred reward quantitatively. (Though some similar work has seemingly been done in inverse problems). However, the relationship to arXiv:1712.05812 should be discussed. That work implies that inferring the human model (without knowing the reward function) is difficult, meaning that large errors are possible. 

Minor issues:

The abstract and introduction should probably be shortened for easier readability. 

The supplement contains an old and unfinished version of the paper which was confusing; please check the submission carefully. 

The terminology of "human model" is overloaded since it can also refer to the human policy (mapping states to actions rather than reward functions to actions). Other possibilities include human decision model, decision model, decision rule, or decision algorithm. 

If one exists, it would be helpful to discuss any known human bias which breaks the assumptions of Theorem 3. 

summary_of_the_review: The paper makes a significant contribution to filling an important knowledge gap: can we accurately infer reward functions with a slightly misspecified human model? The theoretical results are clear and useful. While the overall positive conclusion seems to need caveats, these do not affect the strength of the contributions. If the caveats can be clarified, I would consider raising my score since I believe the paper will be among the more useful ones accepted to ICLR. 

EDIT: The authors partially addressed the suggestions I made and argued for the generality of the results (for example by noting that the KL upper bounds the TV distance). This is one of the somewhat rare papers in modern ML where a clean theoretical result provides a clear and non-obvious practical benefit. In addition, the benefit pertains to a problem that is becoming increasingly important. Assuming the authors make the minor changes I suggested, I think the paper should be considered for an award.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper theoretically and empirically targets the question how/if small errors in learned (human) reward models can lead to large inference errors, thereby making them unusable. The goal then becomes to try to bound the inference error by a function to guarantee usability. The authors argue that it is theorerically provable that the inference error (using maximum likelihood estimation) can be arbitrarily large for sufficiently close human reward functions (learned on finite datasets) for worst-case policy divergence, but that the case is unlikely for "non-adversarial" sampled datasets compared under a weighted policy divergence. The paper identifies central assumptions for the human reward functions, which lead to a possible bound of the inference error. The authors also provide an empirical evaluation for diagnostic grid world domains and LunarLandor for generated as well as real human biases. All experiments support the claims of a bounded inference loss.
strength_and_weaknesses: Pros:
* High clarity wrt motivation, related work, assumptions and conducted theoretical and experimental work
* Important topic - it is a baseline for numerous human-centered RL approaches focussing on AI alignment
* Careful evaluation

Cons:
* Minor open questions to reward comparison / generalization to other reward learning settings. How, for example, does recent work on  equivalent-policy invariant comparison fit compared to the chosen reward similarity? Would it be possible derive similar results here or are such approaches prone to get caught in problems for adversarial samples? Is the derived bound dependent on chosen IRL methods / methods that assume a dataset of expert states and actions or would it transfer to reward functions learned based on trajectory comparisons or other kinds of datasets?
* The related work could also other works on reward learning, e.g. from human preferences
clarity,_quality,_novelty_and_reproducibility: Clarity: The paper uses a highly clear presentation of the motivation, method and results. The problem is nicely defined and situated into available work. The theoretical and empirical results are clearly described as well.
Quality: The analysis and derived bound is clearly motivated and formalized. The taken assumptions, possible limitations as well as important next steps are also highlighted. 
Novelty: The paper provides a novel analysis on reward functions, focussing on bounding the inference error for close reward functions
Reproducibility: The empirical analyses are well-documented, such that evaluations with biases on the mentioned environments can be reproduced.
summary_of_the_review: The paper clearly describes and situates the targeted problem for learned reward models and inference and provides thorough theoretical and empirical analyses. The problem at hand is important and the contribution has sufficient impact. I am still wondering if/how the results can be applied to other forms of reward learning.

## Post author response
I want to thank the authors for their answers to my questions. Also based the other reviews for the paper, my positive evaluation of the paper remains.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:

Inferring human reward functions requires 'human models' which specify how a human acts given their reward function. This paper studies how errors in the human model translate into errors in the inferred reward. First it shows a pessimistic result: in an adversarial setting, a small error in the human model can lead to a large error in the inferred reward. However, under some assumptions the paper shows that the reward error can be bounded linearly in the human model error (for particular choices of error metrics). The paper show that this result applies to several known human biases. The result matches up with experiments in two environments. The authors thus make an optimistic conclusion: reward inference with misspecified human models is not very difficult.

Strengths:

- The writing is clear and the theory well motivated.
- this appears to be the first paper to study the relationship between errors in the human model and inferred reward quantitatively
- This is one of the somewhat rare papers in modern ML where a clean theoretical result provides a clear and non-obvious practical benefit. In addition, the benefit pertains to a problem that is becoming increasingly important. Assuming the authors make the minor changes I suggested, I think the paper should be considered for an award.
- High clarity wrt motivation, related work, assumptions and conducted theoretical and experimental work
- Important topic - it is a baseline for numerous human-centered RL approaches focussing on AI alignment
- Careful evaluation

Weaknesses:

- In the first and last section, the authors make an optimistic conclusion overall, saying that their results imply that reward inference with misspecified human models is not very difficult. The positive conclusion needs some strong caveats stated in the first and last section to avoid false impressions. Though I think these caveats should be stated.

- log-concavity could be better explained,

Recommendation:

A majority of reviewers vote strongly for acceptance. I, therefore, decide to accept the paper. I encourage the authors to use the feedback provided to improve the paper for its camera ready version.

justification_for_why_not_lower_score: - this appears to be the first paper to study the relationship between errors in the human model and inferred reward quantitatively

- This is one of the somewhat rare papers in modern ML where a clean theoretical result provides a clear and non-obvious practical benefit. In addition, the benefit pertains to a problem that is becoming increasingly important. I think the paper should be considered for an award.
Decision: Accept: notable-top-5%