Reward Uncertainty for Exploration in Preference-based Reinforcement Learning

Conveying complex objectives to reinforcement learning ( RL ) agents often requires meticulous reward engineering . Preference-based RL methods are able to learn a more flexible reward model based on human preferences by actively incorporating human feedback , i.e . teacher ’ s preferences between two clips of behaviors . However , poor feedback-efficiency still remains a problem in current preference-based RL algorithms , as tailored human feedback is very expensive . To handle this issue , previous methods have mainly focused on improving query selection and policy initialization . At the same time , recent exploration methods have proven to be a recipe for improving sample-efficiency in RL . We present an exploration method specifically for preference-based RL algorithms . Our main idea is to design an intrinsic reward by measuring the novelty based on learned reward . Specifically , we utilize disagreement across ensemble of learned reward models . Our intuition is that disagreement in learned reward model reflects uncertainty in tailored human feedback and could be useful for exploration . Our experiments show that reward uncertainty exploration improves both feedbackand sample-efficiency of preference-based RL algorithms on complex robot manipulation tasks from Meta-World benchmarks , compared with other existing exploration methods that measure the novelty of state visitation . 1 INTRODUCTION . In reinforcement learning ( RL ) , reward function specifies correct objectives to RL agents . However , it is difficult and time-consuming to carefully design suitable reward functions for a variety of complex behaviors ( e.g. , cooking or book summarization ( Wu et al. , 2021 ) ) . Furthermore , if there are complicated social norms we want RL agents to understand and follow , conveying a reliable reward function to include such information may remain to be an open problem ( Amodei et al. , 2016 ; Hadfield-Menell et al. , 2017 ) . Overall , engineering reward functions purely by human efforts for all tasks remains to be a significant challenge . An alternative to resolve the challenge of reward engineering is preference-based RL ( Christiano et al. , 2017 ; Ibarz et al. , 2018 ; Lee et al. , 2021b ) . Compared to traditional RL setup , preferencebased RL algorithms are able to teach RL agents without the necessity of designing reward functions . Instead , the agent uses feedback , usually in the form of ( human ) teacher preferences between two behaviors , to learn desired behaviors indicated by teacher . Therefore , instead of using carefullydesigned rewards from the environment , the agent is able to learn a more flexible reward function suitably aligned to teacher feedback . However , preference-based RL usually requires a large amount of teacher feedback , which may be timely or sometimes infeasible to collect . To improve feedback-efficiency , prior works have investigated several sampling strategies ( Biyik & Sadigh , 2018 ; Sadigh et al. , 2017 ; Biyik et al. , 2020 ; Lee et al. , 2021c ) . These methods aim to select more informative queries to improve the quality of the learned reward function while asking for fewer feedback from teacher . Another line of works focus on policy initialization . Ibarz et al . ( 2018 ) initialized the agent ’ s policy with imitation learning from the expert demonstrations , and Lee et al . ( 2021b ) utilized unsupervised pre-training of RL agents before collecting for teacher preferences in the hope of learning diverse behaviors in a self-supervised way to reduce total amount of human feedback . Exploration , in the context of standard RL , has addressed the problems of sample-efficiency ( Stadie et al. , 2015 ; Bellemare et al. , 2016 ; Pathak et al. , 2017 ; 2019 ; Liu & Abbeel , 2021 ; Seo et al. , 2021b ) . When extrinsic rewards from the environment is limited , exploration has been demonstrated to allow RL agents to learn diverse behaviors . However , limited previous works have studied the effects of exploration in preference-based RL . Inspired by the impact of exploration , we present RUNE : Reward UNcertainty for Exploration , a simple and efficient exploration method specifically for preference-based RL algorithms . Our main idea is to incorporate uncertainty from learned reward function as an exploration bonus . Specifically , we capture the novelty of human feedback by measuring the reward uncertainty ( e.g. , variance in predictions of ensemble of reward functions ) . Since reward functions is optimized and learned to align to human feedback , exploration based on reward uncertainty may also reflect high uncertainty in information from teacher feedback . We hope that the proposed intrinsic reward contains information from teacher feedback and can guide exploration that better align to human preferences . Our experiment results show that RUNE can improve both sample- and feedback-efficiency of preference-based RL algorithms ( Lee et al. , 2021b ) . We highlight the main contributions of our paper below : • For preference-based RL , we propose a new exploration method based on uncertainty in learned reward functions . • For the first time , we show that exploration can improve the sample- and feedback-efficiency of preference-based RL algorithms . 2 RELATED WORK . Human-in-the-loop reinforcement learning . We mainly focus on one promising direction that utilizes the human preferences ( Akrour et al. , 2011 ; Christiano et al. , 2017 ; Ibarz et al. , 2018 ; Lee et al. , 2021b ; Leike et al. , 2018 ; Pilarski et al. , 2011 ; Wilson et al. , 2012 ) to train RL agents . Christiano et al . ( 2017 ) scaled preference-based learning to utilize modern deep learning techniques , and Ibarz et al . ( 2018 ) improved the efficiency of this method by introducing additional forms of feedback such as demonstrations . Recently , Lee et al . ( 2021b ) proposed a feedback-efficient RL algorithm by utilizing off-policy learning and pre-training . To improve sample- and feedback-efficiency of human-in-the-loop RL , previous works ( Christiano et al. , 2017 ; Ibarz et al. , 2018 ; Lee et al. , 2021b ; Leike et al. , 2018 ) mainly focus on methods such as selecting more informative queries Christiano et al . ( 2017 ) and pre-training of RL agents Ibarz et al . ( 2018 ) ; Lee et al . ( 2021b ) . We further investigate effects of different exploration methods in preference-based RL algorithm . We follow a common approach of exploration methods in RL : generating intrinsic rewards as exploration bonus Pathak et al . ( 2019 ) . Instead of only using learned reward function from human feedback as RL training objective , we alter the reward function to include a combination of the extrinsic reward ( the learned rewards ) and an intrinsic reward ( explo- ration bonus ) . In particular , we present an exploration method with intrinsic reward that measures the disagreement from learned reward models . Exploration in reinforcement learning . The trade off between exploitation and exploration is a critical topic in RL . If agents don ’ t explore enough , then they may learn sub optimal actions . Exploration algorithms aim to encourage the RL agent to visit a wide range of states in the environment . Thrun ( 1992 ) showed that exploration methods that utilize the agent ’ s history has been shown to perform much better than random exploration . Hence , a common setup is to include an intrinsic reward as an exploration bonus . The intrinsic reward can be defined by Count-Based methods which keep count of previously visited states and rewards the agents for visiting new states Bellemare et al . ( 2016 ) ; Tang et al . ( 2017 ) ; Ostrovski et al . ( 2017 ) . Another option is to use a curiosity bonus for the intrinsic reward Houthooft et al . ( 2016 ) ; Pathak et al . ( 2017 ) ; Sekar et al . ( 2020 ) . Curiosity represents how expected and unfamiliar the state is . One way to quantify curiosity is to predict the next state from current state and action Pathak et al . ( 2017 ) , then use prediction error as an estimate of curiosity . If the error is high , that means the next state is unfamiliar and should be explored more . Similarly , instead of predicting the next state , prediction errors from training a neural network to approximate a random function Burda et al . ( 2018 ) can serve as a valid estimate of curiosity . If there are multiple models , then curiosity can also be described as the disagreement between the models Pathak et al . ( 2019 ) . A high disagreement means that the models are unsure about the prediction and need to explore in that direction more . A different approach maximizes the entropy of visited states by incorporating state entropy into the intrinsic reward . State entropy can be estimated by approximating the state density distribution Hazan et al . ( 2019 ) ; Lee et al . ( 2019 ) , approximating the k-nearest neighbor entropy of a randomly initialized encoder Seo et al . ( 2021a ) , or using off-policy RL algorithms to maximize the k-nearest neighbor state entropy estimate in contrastive representation space for unsupervised pre-training Srinivas et al . ( 2020 ) ; Liu & Abbeel ( 2021 ) . These methods all encourage agents to explore diverse states . Our approach adds an intrinsic reward that drives exploration to preference-based RL algorithms . We take advantage of an ensemble of reward models in preference-based RL algorithms , which is not available in other traditional RL settings . To estimate novelty of states and actions , we utilize the disagreement between reward models for our intrinsic reward , in hope of encouraging exploration aligned to directions of human preferences . Trajectory generation in preference-based reinforcement learning . Previous works in preference-based reinforcement learning have investigated several methods to better explore diverse trajectories but close to current optimal policy Wirth et al . ( 2017 ) . One line of works computes agent ’ s stochastic policies that are slightly deviated from optimal policies . Christiano et al . ( 2017 ) uses Trust Region Policy Optimization ( TRPO ) Schulman et al . ( 2015 ) and synchronized A3C Mnih et al . ( 2016 ) . These RL algorithms define stochastic policies to ensure exploration of action space and deviations from optimal policies . However , these exploration methods based on stochastic RL algorithms does not include information from human preferences to drive exploration . Another line of works designs one or multiple criterion to select from multiple possible stochastic policy candidates . Wilson et al . ( 2012 ) proposes to sample several policies from posterior distribution of policy space after updating human preferences . However , such methods come a the cost of requiring many samples collected beforehand . While these methods similarly aims to reduce uncertainty in human preferences , RUNE uses a different metric to estimate such uncertainty through reward functions ensemble . This is different from previous works and is simple , scalable , and easy to implement . A different approach allows human to guide exploration by directly providing additional trajectories . Zucker et al . ( 2010 ) proposes a user-guided exploration method that shows samples of trajectories to human . Human can provide additional feedback to guide exploration . While this method receives exact information from human , it requires additional human labels , which are usually expensive and time-consuming to collect . RUNE however tries to extract information from human feedback revealed in learned reward functions , which doesn ’ t require additional human input .

Summary: This work proposes an ensemble-based intrinsic reward to improve exploration in preference-based RL. The key idea is to incorporate uncertainty in teacher preferences as an intrinsic reward. An ensemble of reward functions is used to capture this uncertainty. The paper discusses experiments on three robot tasks with ablation studies to investigate the method's performance.