Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work studies causal confusion in the context of reward learning from preferences. It is shown that rewards learning from preferences achieve poor performance in terms of learning a policy in three robot learning benchmarks. Several factors for causal confusion are identified, including distractor features and noise in stated preferences.
strength_and_weaknesses: The paper is well-written and the ideas are exposed clearly.

However, the scope of this study is very limited. The experimental setup only includes continuous robot learning benchmarks, with three similar tasks, and is not extended to additional, potentially very different, environments. Besides, a specific model is used for the preferences. It might also be interested to compare the results here with alternative preference-based reinforcement learning approach, including additional value-based approaches and relation-based techniques (see for example the review: "A Survey of Preference-Based Reinforcement Learning Methods", Christian Wirth et al.). Therefore, it is not clear if the conclusions reached in this study may be extended beyond this narrow scope.

Another limitation of this work is the use of only synthetic preferences, computed from the real reward using a noise model. Would the same conclusions hold with real human raters preferences? 

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and clear.

However, the novelty of the work is limited as well as its scope, which lessens the contributions.

In addition, some claims in Section 4 seem to not be fully supported. For example, in Section 4.1., "in incentivizing the Reacher robot to spin fast, it leads the RL optimization toward states that were not seen during reward learning" ; might an exploration issue be the cause of the poor performance? Another example in Section 4.3., where it is stated that adding "preference data when noise is present actually has a negative effect"; might this be a specific phenomenon that happens with the selected noise model? Does this happens under additional models? We would expect noise to be averaged over different samples so that adding more data would increase performance.

A few small typos need to be corrected: e.g., "to have a access" in Sec. 1 -> an access , "Note that later sections, we observe" -> in late sections, in Sec. 3
summary_of_the_review: This work considers an interesting idea. However, the narrowness of this scope of this study as well as the lack of full support for certain claims limits the contribution of this work. It seems that it is not yet ready for publication and a larger scope would improve considerably its contribution.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work analyzes the occurrence of spurious correlations in learning reward functions from preferences in the commonly used Bradley-Terry model. The authors show that picking up such correlations can have a drastic effect on the performance of an RL agent that is trained using the learned reward. In their empirical analysis, the authors examine the effect of distractor features, noise, partial observability and model capacity. The paper quantifies how these factors affect the trained RL agent on the basis of several different metrics.
strength_and_weaknesses: **Strengths**

- Assistive robotics is a very interesting and relevant problem domain.
- The analyzed failure cases are of high practical relevancy.
- The empirical analysis is easy to follow.
- Videos on the companion website qualitatively illustrate the failure modes of trained RL agents very well.
- I appreciated the authors making a concerted effort to convey their findings graphically.

**Weaknesses**  
My main concerns regarding the paper are:

- The paper has limited novelty. While the setting is new and interesting, the methods used by the authors to measure causal confusion are not novel, except for a method to approximate the KL divergence between two distributions of state-action pairs.
- No proposed solution method. The paper reads like a very nice and thorough exposition of an important problem but then it "abruptly" ends. Once the authors introduced their method to measure the KL divergence between state-action distributions, I was immediately thinking of whether this could be used to penalize learned rewards and thereby encouraging agents to stay within the "known" region of the state-action space. Even negative results on "straightforward" (devil's in the details of course) remedies like this would be nice.

More minor weaknesses:

- Noise evaluations could be more detailed. I agree with the authors that user noise is inevitable, but in my opinion the setting they evaluate ($\epsilon$-greedy action noise) is rather uninteresting. "*B-Pref: Benchmarking Preference-Based Reinforcement Learning*" (Lee et al.) introduces several types of noise based on known cognitive biases. It would be interesting to see how such structured noise affects performance and I also think evaluating it would mesh very well with the practical relevancy of the author's work.
- It would have been nice to have an explanation of the noise procedure in the main paper as I think it's relevant to understanding.
- The authors evaluate against the ground-truth reward functions very often. It's frustrating that these are neither stated in the main paper nor in the appendix. 
- Hyperparameters for the SAC and PPO agents where missing. 
- Evaluation methods are not used consistently for all settings. There are no saliency maps and EPIC distances for itch scratching. They would help paint a more complete picture even though I presume they are omitted because the agent fails to learn in most settings anyway.
- In A.4: "*With the trained model, we calculate $D_{\text{KL}} (p \parallel q)$ by taking the negative mean return/logit of all
reward learning observation-action pairs*". I assume it's the mean return according to the reward model? Would still be nice to have it explicitly stated here.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

The presentation is generally clear and concise. Sometimes a lot of flipping between text and appendix is required. I feel like the plots could be organized more coherently to be on the pages where they're mentioned. The plots themselves are simple, easy to understand and convey the desired message well.

**Quality**

Quality of writing is very high. It has hardly any typos/grammatical errors. 

**Novelty**

This work's main novelty lies in the application of existing methods for analysis to an interesting problem setting. Apart from this, the authors provide a method for approximating the KL divergence between two data distributions. Novelty is thus a weakness of the paper.

**Reproducibility**

The authors state that they will release source code and datasets. Reward functions are not stated in the paper but the environments used by the authors are open source. No hyperparameters for the RL agents are given. With a release of those I think the work can be fully reproduced.
summary_of_the_review: Overall, I'm torn on this paper. On one hand, the authors analyze a relevant problem in a very interesting and relevant problem domain. The writing is clear and easy to understand and comes with detailed empirical analysis that follows a coherent structure.  
On the other hand, this feels like reading only the first half of the paper: After the problem has been clearly identified, a (partial) solution is missing. Of course that's not necessary for each paper, but without a solution method the limited novelty and in points not 100% clear presentation weigh more heavily.  
For me, the lacking novelty currently outweighs the intriguing setting and therefore I'm voting for weak reject. I'm looking forward to an improved version of the paper (even just addressing some of the weaknesses) because there's obvious potential here.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper studies reward hacking that happens when a policy is trained using a reward model that is learned from human preference data. In reward hacking, the predicted reward of the learned policy is high but the actual reward is low. The paper shows multiple lines of evidence that the observed reward hacking stems from spurious/non-causal correlations: Firstly, reward hacking disappears when non-causal features are removed from the input space. Secondly, the predicted reward is sensitive/salient to non-causal features that we know do not cause ground truth reward. Thirdly, after learning a policy, the state-action distribution differs from the training distribution, more than it does when non-causal features are not present. Thus, non-causal features exacerbate distribution shift, breaking spurious correlations even more. Then, the paper shows (a little bit of) evidence that causal confusion is exacerbated by noisy preferences, greater model scale, and unobservability of features that cause reward. 
strength_and_weaknesses: RL from preferences has increasingly high-impact applications and great promise to be scaled further. This paper addresses reward hacking which is a common and severe difficulty with RL from feedback. Therefore it is important to understand the causes of this reward hacking. This paper identifies and analyzes one cause: spurious features that correlate with reward during training, are therefore rewarded by the reward model, which leads to troublesome behavior when training a policy with this reward. The paper also identifies exacerbating factors. Showing empirical evidence for these is a useful contribution since the problem is important and understudied. 

However, the evidence for key findings is limited, which makes it hard to know how much the findings will generalize to other settings. Some claims also lack an explanation. I expand on this below. 
clarity,_quality,_novelty_and_reproducibility: __Novelty__

This appears to be the first paper that focuses primarily on hacking of reward functions that are learned, not programmed. Some papers have studied hacking of learned reward functions as a side-note, and anecdotally identified causal confusion as the culprit but not studied it in detail. The authors should discuss the relationship to goal misgeneralization (Langosco et al and Shah et al) as these papers also empirically study goal/reward functions that lead to good performance on the training distribution but not elsewhere, due to spurious reward correlations (one difference is that their reward function is learned implicitly by the policy rather than with a reward model). The authors should also discuss to what extent the causal confusion problem in imitation learning is similar or different from the problem in preference learning. This would help me understand how novel the findings are. 

__Quality__

Most claims appear to lack strong evidence.

Claim 1: reward hacking is due to spurious features that are being rewarded. This claim is the focus of Section 3. 
- The first piece of evidence is that noise worsens final performance, and the paper suggests that this means causal confusion is the culprit. However, no explanation is given why independent noise should worsen causal confusion. (Previous work only gave an explanation for temporally correlated noise, but here the noise is independent.)
- The second piece of evidence is that removing non-causal features improves performance. This is valid but only shown in one environment (Reacher). In the other environments, results are unclear or not visible (Fig 7). 
- The third piece of evidence is that a saliency map shows that features are rewarded that are not causally related to ground truth reward. However, it must also be shown that these features are correlated with reward on the training data, otherwise no spurious correlation has been shown. There may be a spurious correlation in the feeding environment since the spoon is never behind the head during training. However, for other environments it is not shown that the rewarded features actually correlate with reward on the training data. 

Claim 2: a) preference noise worsens causal confusion and b) this becomes worse with more data. a) is not shown; the paper only seems to show that adding noise worsens task success, not causal confusion. b) is not given a convincing explanation, which means we need strong empirical evidence, ideally from more than two environments (Reacher and Feeding).

Claim 3: greater model size worsens causal confusion. Figure 9 shows that this sometimes happens and some times the opposite happens. I don't think there is enough evidence here to make a claim that belongs in the abstract. 

Claim 4: when causal features are unobserved, causal confusion can get worse. This is shown in one environment (Feeding), which is acceptable since this claim is unsurprising. 

To improve the paper, the authors could collect more evidence for each claim or reduce the number of claims and focus on the ones that matter most. I believe the paper would be strong enough by making only a subset of these claims, but supporting them well. 

__Details__

Section 4.5: The experiment simply shows that the learned policy fails. However, it does not show that this is due to causal confusion, reward hacking, and/or distribution shift. So the value of this experiment is unclear. 

Section 4.3: "increasing the data size without increasing data diversity results in data points that are more similar to each other" --> Why would trajectories become more similar when more samples are drawn but the distribution they are drawn from does not change? This claim needs to be supported or removed. 

- The figure presentation could be easily improved. 
    - 1) Figure 3 is not self-explanatory. Figures (incl. caption) should be self-explanatory if possible because most readers will skim them without reading the text, or skim the figures before reading the text.
    - 2) In Figure 1a, it is hard to understand what is happening in the top right image, especially what the meaning of the arrow is. It is also unclear if this is meant to show an OOD situation. Furthermore it should be clarified that the < refers to what preferences learned by the model, not ground truth. It should also be clarified that the right column refers to OOD situations which are caused by imperfections in the reward model, as this is not clear from the figure alone. 
    - 3) Figure 1b is used in the introduction but only appears on page 5; ideally it would appear much earlier. 
    - 4) Figure 1c: again, the meaning of the arrow is unclear without reading the main text. Perhaps use a circular arrow to show rotation. 
    - 5) Table 2: the first sentence of caption is ambiguous and therefore difficult to follow, please rewrite. Furthermore, the text references Table 2c which does not exist. 
    - 6) In Figure 5 it is hard to see the food, perhaps zoom in. 

Lastly, I was unsure what the main contributions are. Section 3 suggests that a key contribution is to show that the reason for reward hacking is causal confusion. This is also implied in the introduction: "we demonstrate the failure of preference-based reward learning to produce causal rewards". However, the abstract does not mention this contribution. 

The presentation is adequate but the paper tries to prove too many things which can be confusing. 

summary_of_the_review: The paper addresses an important, interesting, and understudied problem by showing a variety of empirical evidence for spurious correlations and factors that exacerbate it in RL from preferences. To fully recommend acceptance it would need stronger evidence for its conclusions, or focus on conclusions that can be supported well. 

EDIT: my concerns are sufficiently addressed now that I score the paper as marginally above acceptance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper focus on causal confusion in preference-based reward learning. The study was conducted by performing a series of sensitivity and ablation studies on three robot control benchmarks, where the preference-based reward learning methods fail to generalize to out-of-distribution states.  The study indicated some sources that exacerbate causal confusion. Moreover, a set of methods were identified to interpret causally confused rewards. 
strength_and_weaknesses: ## Strength

+ This paper provides the first concrete and systematic study of causal confusion for preference-based reward learning on three robot control benchmarks. 

+ Several techniques including saliency maps, EPIC reward distance and Kullback-Leibler divergence were used for evaluating the learned reward functions. This study tried to provide evidence of causal confusion in preference-based reward learning, and identify several factors that lead to causaul confusion.


## Weakness

+ There is no formal definition of causal confusion in preference-based reward learning. How to distinguish between the cuasal confusion and other factors that result in similar performance in the benchmarks, for example, overfitting or distribution shift? 

+ EPIC distance might no be a proper method for evaluating the learned reward in distribution shift cases. A recent work [1] has identify that EPIC avoids policy optimization, but in doing so requires computing reward values at transitions that may be impossible under the system dynamics. This is problematic for learned reward functions because it entails evaluating them outside of their training distribution, resulting in inaccurate reward values that we show can render EPIC ineffective at comparing rewards. 

+ It is not clear how to use the findings in this study to avoid causal confusion in reward learning. It would be more interesting if this work can provides some suggestions or algorithms to do so. 

[1] Wulfe, Blake, et al. "Dynamics-Aware Comparison of Learned Reward Functions." In ICLR 2022.






clarity,_quality,_novelty_and_reproducibility: ## Clarity
Can be improved. First, there is no formal definition of causal confusion in preference-based reward learning. Second, some part of the logic is jumped and need more connections between sections. For example, this work provides evidence of causal confusion in section 3. and factors in section 4. But there is no clear connections between the evidence and the factors. Finally, there is no showcase on how to leverage the findings in the study to avoid causal confusion in reward learning, which redues the contribution of this work. 


## Quality
From the study perspective, the quality is good. This study was conducted on three robot control benchmarks for preference-based reward learning, using diverse evaluating approaches and indentify several factors that lead to causal confusion. 

## Novelty
The study is novel.

## Reproducibility
Need more details to reproduce the results. 

summary_of_the_review: Overall, this work provides a novel study for causal confusion in preference-based reward learning. To make it stronger, I suggest the authors to provide formal definition of causal confusion in reward learning, and showcase on how to leverage the findings in this study to avoid the causal confusion (if it is true) in reward learning.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an analysis of reinforcement learning reward functions learned from preference labels over trajectory pairs.  In particular, the authors show that such learned reward functions are prone to pick up on spurious correlations in the features, to noise in the preference labels, and to distribution shifts when optimizing the reward model.  Furthermore, the authors demonstrate that is not simply an over-fitting problem as reward models can reliably predict the preferred trajectory within a pair of trajectories in validation and test data, but rather that the policy is unable to exploit the reward as the policy shifts from the training distribution.
strength_and_weaknesses: +The paper drives home the message that being able to predict preferences does not necessarily translate to quality task completion. 
+The experiments are thorough and consider various issues that could be the cause of the failure with learned reward functions.
+The test environments offer a range of difficulty, and the findings are consistent across tasks.

-The paper reads as though the policies and rewards are trained independently.  First train a policy using the ground-truth environment reward, sample behaviors and add noise to form the dataset for reward learning, optimize the learned reward function, then finally train a new policy on this reward.  However, as noted in Cristiano et al. (Section 2.2), it is important the the policy and the reward are learned together.
-The number of pairs in the datasets is somewhat large.  For Table 1, the large dataset is >50k pairs.  For a real problem this is too large to imagine being labelled by people.
clarity,_quality,_novelty_and_reproducibility: When creating the synthetic preference dataset, varying amounts of noise is added to the behaviors from the policy rollout.  When optimizing the reward do you sample behaviors across all noise levels?   If so I would be interested in seeing how this compares to iteratively optimizing the policy and the reward as over time the quality of the policy and reward improves, and so there should be less noise further in training.  How would this impact training and the trained models?

It would be useful to include an overview of the algorithm in an appendix to help properly understand the interaction between the policies and the rewards during optimization — e.g., confirm that you do not alternate between policy updates and reward updates, and to specify how the behaviors are sampled — e.g., randomly drawn from all noisy levels at each step, or begin with \epsilon=1 and decay towards \epsilon=0 to simulate the performance of the policy training?

Nit-pick:  throughout the paper, the incorrect opening quote is used.

summary_of_the_review: Preference-based reward learning is becoming increasing popular, especially for domains in which it is challenging to hand-define a reward function.  Understanding the impact and limitations of learned rewards is important, and the work in this paper provides insights towards that.

I have some concern about the flow of the training — it’s not that what is done here is wrong, but rather there are better ways to train the policies and rewards together.  However, with the authors open sourcing their code, others can extend the work to consider how they specifically go about training.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of causal confusion in preference-based reward learning. In particular, the setting in which low test error is achieved with the learned reward function but out-of-distribution-generalization fails, is analysed on several benchmark datasets. The authors identify several factors that can exacerbate causal confusion and demonstrate their impact in experiments.

Strengths of the paper:
* The paper is mainly well-written and most ideas and thoughts are clearly expressed.
* The considered problem is important and the study goes deeper than existing studies.
* The analyzed failure cases are of high practical relevancy.

Weaknesses of the paper:
* Limited number of experiments, only on the robotics domain.
* Only a single preference model is considered; the used noise model is of limited interest.
* Uses only synthetic preferences, computed from the real reward using a noise model.
* There is no conclusion in the sense of providing a method or clear directions for resolving the identified issues (if this is possible).


----

Discussion and recommendation:
The paper received mixed scores and, in most cases, a discussion between reviewers and authors took place. The authors improved their paper already addressing several of the reviewers' comments. In a final call, we came to the consensus that despite some shortcomings, the paper contains quite some material relevant to the community working on reward-identification and causal confusion. Hence I am recommending acceptance of the paper. Nevertheless, the authors are strongly encouraged to update their paper in line with the reviewers comments. In particular, regarding the following the claims (copied and adjusted from the corresponding reviewer's comments):

* Remove/clarify claims about noise that the authors have to be incorrect
* Tone down claims that reward misidentification is due spurious correlations / causal confusion since the evidence is limited. In particular, where the paper talks about 'causal confusion / spurious correlations', it should often refer to reward misidentification instead.
* Remove the claim that greater model size worsens causal confusion. Figure 9 (original submission) shows that this sometimes happens and some times the opposite happens. There's not sufficient evidence to clearly support that claim.

But also the other points raised by the reviewers are important and should be incorporated in the camera ready submission.


justification_for_why_not_higher_score: Several issues which should be addressed in the camera ready version of the paper. Analysis would have to be extended to warrant a higher score.
justification_for_why_not_lower_score: The paper contains material interesting to the community.
Decision: Accept: poster