Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an exploration method for "preference based Reinforcement Learning" methods, where human feedback is incorporated to the training regime. The authors use an ensemle of learned reward models and add an intrinsic reward based on disagreement (or uncertainity). The authors test the idea on robotic manipulation tasks from meta-world. The agent learns purely based on the feedback from a teacher that provides preference of trajectory one over another. The tasks presented are "door close", "door open" and "drawer open".
The authors combine their exploration strategy (RUNE) with the preference based learning method PEBBLE and compare it with PEBBLE with some other exploration strategies. The results show that the proposed method provide some improvement over others. The authors also compare with PEBBLE by using 700 feedback instead of 1000. The results show minor improvement.
main_review: The idea of efficient exploration is very important for reinforcement learning, especially for cases where human feedback is used. The authors propose an interesting approach of looking at reward model ensemble. At a high level, this idea makes sense, but looking at the evaluation, the effect of the algorithm on the results is very minor. In many of the plots, it is not easy to see a big difference between different methods. In addition, the evalutation is very limited. The authors use one problem (manipulation) and 3 tasks from that problem. 

The paper is well written, but same citations are repeated over and over. This is both unnecessary and distrupts reading flow.
summary_of_the_review: The paper tackles an interesting problem for RL, the proposed method is incremental but partially novel. On the other hand, the evaluation is limited and the results are not significant.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: With this work, the authors propose a bayesian active learning approach to the problem of reinforcement learning from preferences. To do this, they model the epistemic uncertainty of the reward function to intrinsically motivate the RL agents to explore. The solution demonstrates improved sample efficiency.
main_review: strengths: A simple solution to the problem of epistemic uncertainty driven exploration for RL from preferences. The experimental results demonstrate improved sample efficiency over other baselines. Also, the writing and the structure of the paper is clear.

weaknesses: The paper suggests that capturing the epistemic uncertainty at the reward prediction level is better than capturing it at the level of the dynamics. Although I find this intuition correct, you could show this theoretically as well since the reward uncertainty incorporates the state transition uncertainty as well. I'd like to see some theoretical analysis that would support the experiments. Also, I see this paper as a direct application of the epistemic uncertainty driven exploration in RL from preferences that make its novelty small.

C1: "We take advantage of an ensemble of reward models ... which is not available in other traditional RL settings" - What is the challenge here? To have a reward model or an ensemble of reward models? I don't think that any of those is challenging. The disagreement method doesn't need to happen over the forward dynamics model but can also happen over the rewards. In fact, the intuition here is that you model the epistemic uncertainty of the predictor of interest. If you want to cover more states you model the epistemic uncertainty over the forward states, if you want to cover more downstream task-related states you model the epistemic uncertainty over the reward model.

C2: "Quality of learned reward functions" - The plot doesn't seem to add anything valuable. The scales are different (you explain why) but that could also have been the result of learning a wrong reward function. What you want perhaps to show instead is that for sampled $s_t, a_t$ pairs the predicted reward and the true reward is correlated.
summary_of_the_review: I find the paper clearly written and a direct application of epistemic uncertainty driven exploration in RL from preferences. Although simple, its novelty is marginal. Also, it lacks theoretical justification (e.g. bayesian RL intuitions, why uncertainty over the rewards might work better than uncertainty over dynamics, aleatoric vs epistemic uncertainty, etc.). Finally, the experiments, although they support the argument, there are experiments that don't contribute to the discussion (C2)
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This work proposes an ensemble-based intrinsic reward to improve exploration in preference-based RL. The key idea is to incorporate uncertainty in teacher preferences as an intrinsic reward. An ensemble of reward functions is used to capture this uncertainty. The paper discusses experiments on three robot tasks with ablation studies to investigate the method's performance. 
main_review: Exploration strategies for preference-based RL is underexplored and this paper takes a step towards practical RL methods that can learn with humans-in-the-loop. Although the idea is not groundbreaking given existing work, RUNE is simple, scalable, and easy to implement. Adding an uncertainty-based intrinsic reward to guide preference-based RL is intuitive and at its core, I think the underlying idea can improve existing methods. 

However, I do have concerns about this particular instantiation of the approach. First, the method can be better explained and it's unclear how RUNE is trained:

- Is each model in the ensemble trained using the same set of queries and feedback? Are the models trained independently given their respective data? 
- In general, there could be multiple reward functions that give rise to the same preference feedback (e.g., a translation of the reward function). In this case, each model in the ensemble might learn correct but different reward functions, yet the standard deviation of the state rewards would be non-zero? Is this still a meaningful estimate of uncertainty?
- In the above case, is the average reward indicative of the actual extrinsic reward? 

Second, the main claims are not well-validated by the experiments. While I appreciate the authors used rather complex robot domains, it is difficult to understand if RUNE actually increases exploration:

- Iit is unclear if the better performance is due to extra exploration or simply the use of a better reward estimator, i.e., the ensemble. I encourage the authors to quantitatively compare the amount of exploration when RUNE is used with PEBBLE. If the current domains are too complex, a simple gridworld-type domain (or small-scale continuous domain) can be used. 
- If my interpretation is correct, Fig 5b seems to indicate that less exploration is performed with RUNE. We would expect that disagreement to be higher at the middle portions as the method explores uncertain states?  
- For the experiments, how many runs are performed for each method? Is there noise in the simulation human feedback? Does the simulated human prefer one trajectory above all others or is there some ambiguity? These details are important to ascertain the robustness of the method and results. Also, please use a different color for the methods in Fig 4; I am unable to compare the standard deviations. 
- The analysis on the quality of learned rewards can be improved. Normalizing the rewards to a standard range, e.g., [0,1], can aid comparisons in Fig 5.c. Again, using a gridworld here may help to better illustrate that a reasonable reward function is learnt. At present, the figure shows the rewards from a single trajectory rather than the state space. 



summary_of_the_review: Exploration for preference-based RL is an interesting problem, but the current paper has multiple issues that should be addressed. I hope the authors can provide some clarification my questions above.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors introduce a method for integrating reward uncertainty into the exploration/exploitation tradeoff in a preference-based reinforcement learning setting. The method uses a surrogate reward ensemble variance as intrinsic reward bonus.
main_review: The paper is well written and clear to understand. The method can be considered sound, as it is based on established building-blocks. Some minor issues should be improved, but have no substantial impact:
- Ensemble disagreement (mentioned in abstract) and ensemble variance are not the same thing
- Preference notation is introduced as (1,0),(0,1), but seq_0 > seq_1 is used
- Indifference seems to be not handled, but is introduced. (Which is ok, but should be clarified)
- The agent is not encouraged to visit more "unseen" states, but "uncertain" states (wrt. the true reward)

Additionally, the method has a hidden assumption, which is quite common, but should be made explicit: The preference feedback is assumed to be stationary and acyclic. Without these assumptions, the surrogate reward may not converge towards a low variance solution and the mean reward can not sufficiently reflect the (unknown, human) preference function. In the cyclic preference, case this not possible at all (with a scalar reward) and in the non-stationary case, the (unweighted) mean is not a good estimator. Both assumptions are often violated by real-world human preference feedback.

The evaluation is ok, but more domains should be added for allowing to derive more general conclusions. The ablation study is a good addition, with the exception of the "Quality of learned reward functions" section. As acknowledged by the authors, surrogate rewards may have a different scale than the underlying, true rewards (also when not using tanh). Furthermore, even rewards that differ not only in scale, can induce the same (optimal) policy. Therefore, the comparison should be done in terms of a different metric, like ranking error over states and/or sequences. Even rescaling the learned reward would help, because with the current plot, alignment is not clearly visible.

However, surrogate-reward uncertainty-based exploration is not new and the according advantages have also been shown before (e.g. Akrour 2011 - for more, see "Trajectory Generation" in https://jmlr.org/papers/volume18/16-634/16-634.pdf). None of these methods have been applied to PEEBLE/DeepRL (as far as the reviewer is aware), but this still reduces the validity of the claims. Furthermore, the idea of using ensemble variance as intrinsic reward bonus is also already known from plain RL (as mentioned by the authors). Resultingly, the publication is applying known methods to a different framework (PEEBLE). The impact of this issue can maybe reduced by clearly stating the differences to other preference-based exploration strategies. 

However, even if these issues are resolved, the degree of novelty will likely stay limited. Therefore, the evaluation should be substantially improved (besides adding more domains, potentially adding some results from the appendix). Some suggestions:
- Effect of different beta values
- Comparison to "comparable" formulations, for intrinsic reward, based on preference uncertainty (e.g. ensemble disagreement, instead of variance)
- Optimal/terminal policy in terms of preference queries (instead of only two query counts)
summary_of_the_review: The proposed method is of interest within the domain of preference-based reinforcement learning, as guiding exploration can reduce human operator workload. The novelty of the approach is limited and the claims have been shown before, but in a different framework. This limitation has to be overcome or leveled by a thorough evaluation, allowing generalized conclusions.
review_title: null

Meta-review: This is a borderline paper. The scores were initially below the bar. The novelty of the work is limited and there are strong claims in the paper that should be revised. The authors can also do a better job in positioning their work with respect to the existing results. However, the authors managed to address several questions/concerns of the reviewers and convince them to raise their scores. I would strongly recommend the authors to address the rest of the reviewers' comments, especially those related to strong claims and connection to related work, and further improve their work in preparing its final draft.
Decision: Accept (Poster)