Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation

1 Introduction . Deep reinforcement learning ( DRL ) algorithms have achieved many successes in solving problems with high-dimensional state and action spaces [ 27 , 3 ] . However , DRL ’ s performance is limited by its sample ( in ) efficiency . It is often impractical to collect millions of training samples as required by standard DRL algorithms . One way to tame this problem is to leverage additional human guidance by following the general paradigm of Human-in-the-Loop Reinforcement Learning ( HIRL ) [ 51 ] . It often allows the agent to achieve better performance and higher sample efficiency . One popular form of human guidance in HIRL is the binary evaluative feedback [ 15 ] , in which humans provide a `` good '' or `` bad '' judgment for a queried state-action pair . This framework allows 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . non-expert humans to provide feedback , but its sample efficiency could be further improved by asking humans to provide stronger guidance . For example , the binary feedback does not tell the agent why it made a mistake . If humans can explain the `` why '' behind the evaluative feedback , then it is possible to further improve the sample efficiency and performance . Taking the training of an autonomous driving agent as a motivating example , humans can `` explain '' the correct action of `` apply-brake '' by pointing out to the agent that the `` STOP '' sign is an essential signal . One way to convey this information is through saliency information , in which humans highlight the important ( salient ) regions of the visual environment state . The visual explanation will indicate which visual features matter the most for the decision in the current state . Note that requiring human trainers to provide explanations on their evaluations does not necessarily require any more expertise on their part than is needed for providing only binary feedback . In our driving agent example , the human trainers may not know things like the optimal angle of the steering wheel ; however , we can expect them to be able to tell whether an observed action is good or bad , and what visual objects matter for that decision . In fact , the use of human visual explanation has been investigated by recent works in several supervised learning tasks [ 39 , 35 , 34 ] . They show that the generalization of a convolutional neural network can be improved by forcing the model to output the same saliency map as human - in other words , forces to model to make the right prediction for the right reasons . However , it remains unclear how to effectively incorporate domain knowledge in visual explanation in deep reinforcement learning . In this work , we present EXPAND , which aims to leverage - EXPlanation AugmeNted feeDback ( Fig . 1 ) to support efficient human guidance for deep reinforcement learning . This raises two immediate challenges ( i ) how to make it easy for humans to provide the visual explanations and ( ii ) how to amplify the sparse human feedback . EXPAND employs a novel context-aware data augmentation method , which amplifies the difference between relevant and irrelevant regions in human explanation by applying multiple perturbations to irrelevant parts . To reduce the human effort needed to provide visual explanations , EXPAND uses off-the-shelf object detectors and trackers to allow humans to to give explanations in terms of salient objects in the visual state , which are then automatically converted to saliency regions to be used by the RL system . We show that EXPAND agents require fewer interactions with the environment ( environment sample efficiency ) and over 30 % fewer human signals ( human feedback sample efficiency ) by leveraging human visual explanation . We highlight the main contributions of EXPAND below : • This is the first work that leverages human visual explanation in human-in-the-loop reinforcement learning tasks . We show that EXPAND is the state-of-the-art method to learn from human evaluative feedback in terms of environment sample efficiency and human feedback sample efficiency . • We benchmark our context-aware data augmentation against standard context-agnostic data augmentation techniques . Our experiment results help shed light on the limitations of existing data augmentations and can benefit future research in accelerating RL with augmented data . • We show that human visual explanation in a sequential decision-making task can be collected in a low-effort and semi-automated way by using an off-the-shelf object detector and tracker . We note that EXPAND agents have applicability beyond improving the efficiency of human-in-theloop RL with sparse environment rewards . In particular , they can be equally useful in accepting human guidance to align the RL system to human preferences . They can be used to incorporate rewards from human feedback , for example , when environment reward is not defined upfront or to accommodate additional constraints that are not reflected in existing reward signal . EXPAND , thus , can also be useful in aligning objectives of humans in the loop to the agent ’ s [ 15 , 5 ] . Finally , EXPAND ’ s approach of allowing humans to provide their explanations and guidance in terms of objects of relevance to them , and automatically converting those to saliency regions , is an instance of the general approach for explainable and advisable AI systems that use symbols as a lingua franca for communicating with humans , independent of whether they use symbolic reasoning internally [ 13 ] . 2 Related Work . Leveraging human guidance for RL tasks has been extensively studied in the context of imitation learning ( learning from demonstration ) [ 37 ] , inverse reinforcement learning [ 29 , 1 ] , reward shaping [ 28 ] , learning from human preference [ 7 , 22 ] , and learning from saliency information provided by human [ 49 ] . Surveys on these topics are provided by [ 51 , 46 ] . Compared to these approaches , learning from human evaluative feedback has the advantage of placing minimum demand on both the human trainer ’ s expertise and the ability to provide guidance ( e.g . the requirements of complex and expensive equipment setup ) . Representative works include the TAMER framework [ 15 , 45 ] , and the COACH framework [ 25 , 4 ] . The TAMER+RL framework extends TAMER by learning from both human evaluative feedback and environment reward signal [ 16 , 17 ] . DQN-TAMER further augments TAMER+RL by utilizing deep neural networks to learn in high dimensional state space [ 2 ] . Several approaches have been proposed to increase the information gathered from human feedback , which takes into account the complexities in human feedbackproviding behavior . Loftin et al . speed up learning by adapting to different feedback-providing strategies [ 23 , 24 ] ; the Advice framework [ 11 , 6 ] treats human feedback as direct policy labels and uses a probabilistic model to learn from inconsistent feedback . Although these approaches better utilize human feedback with improved modeling of human behaviors , weak supervision is still a fundamental problem in the evaluative feedback approach . Human explanatory information has also been explored previously . The main challenge of using human explanation is to translate human high-level linguistic feedback into representations that can be understood by the agents . As an early attempt , Thomaz et al . allow humans to give anticipatory guidance rewards and point out the object tied to the reward [ 44 ] . However , they assume the availability of an object-oriented representation of the state . Krening et al. , Yeh et al . resort to human explanatory advice in natural language [ 19 , 48 ] . Still , they assume a recognition model that can understand concepts in human explanation and recognize objects . In this work , we bridge the vocabulary gap between the humans and the agent by taking human visual explanations in the form of saliency maps . Other works use human gaze data as human saliency information , collected with sophisticated eye trackers , to help agents with decision making in an imitation learning setting [ 50 , 52 , 14 ] . They require human data to be collected offline in advance . In contrast , this work is closer to Arakawa et al. , Xiao et al. , Cui et al. , where human trainers are required to be more actively involved during training [ 2 , 47 , 8 ] . Recent works in supervised learning also use human visual explanation to extend the human-machine communication channel [ 35 , 34 , 43 ] . The visual explanation is used as domain knowledge to improve the performance of machine learning models . The way we exploit human explanation via state perturbation can be viewed as a novel way of data augmentation . Data augmentation techniques are widely used in computer vision tasks [ 21 , 40 ] and have recently been applied to deep RL tasks [ 18 , 20 , 33 , 26 ] . The framework explanatory interactive learning [ 43 ] also leverages human visual explanation with data-augmentation . However , they only focus on classification tasks with low-dimensional representations . 3 Problem Setup . We intend to verify the hypothesis that the use of visual explanation and binary evaluative feedback can boost an RL agent ’ s performance . We have an agent M that interacts with an environment E through a sequence of actions , states , and rewards . Following standard practice in deep RL to make states Markovian , k ( k = 4 ) preprocessed consecutive image observations are stacked together to form a state st = [ xt− ( k−1 ) , ... , xt−1 , xt ] [ 27 ] . At each time-step t , the agent can select an action from a set of all possible actions A = { 1 , ... , K } for the state st ∈ S. Then the agent receives a reward rt ∈ R from the environment E . This sequence of interactions ends when the agent achieves its goal or when the time-step limit is reached . This formulation follows the standard Markov decision process framework in RL research [ 42 ] . The agent ’ s goal is to learn a policy function π , a mapping from a state to action , that maximizes the expected return . For a deep Q-Learning agent , the policy π is approximated by the Q function Q ( s , a ) , which represents the maximum expected rewards for taking action a at state s. Additionally , we assume a human trainer who provides binary evaluative feedbackH = ( h1 , h2 , ... hn ) that conveys their assessment of the queried state-action pairs given the agent trajectory . We define the feedback as ht = ( xht , b h t , xt , at , st ) , where b h t ∈ { −1 , 1 } is a binary `` bad '' or `` good '' feedback provided for action at , and xht = { Box1 , ... , Boxm } is a saliency map for the image xt in state st. Boxi is a tuple ( x , y , w , h ) for the top left Euclidean coordinates x and y , the width w , and the height h of the bounding box annotated on the observation image xt . 4 Method . EXPAND aims to improve data efficiency and performance with explanation augmented feedback . In the following , we will first describe how EXPAND simultaneously learns from both environment reward and binary evaluative feedback . Then we introduce our novel context-aware data augmentation that utilizes domain knowledge with human explanation . Algorithm 1 in Appendix A presents the train-interaction loop of EXPAND . Within an episode , the agent interacts with the environment and stores its transition experiences . Every few episodes , it collects human feedback queried on a trajectory sampled from the most recent policy . All the human feedback is stored in a feedback buffer similar to the replay buffer in off-policy DRL [ 27 ] . The weights of RL agent are updated twice , first using sampled environment data as in usual RL update , and then using human feedback data , in a single training step . In our experiment , both EXPAND and the baselines follow the same train-interaction loop .

Summary: This paper proposes EXPAND - a method for allowing human input and semantic understanding to guide policy learning for RL agents. Humans are asked to provide binary feedback (good/bad) on actions taken by the agent, as well as highlight visual features which helped guide their decision (e.g. stop sign tells human to stop). The binary feedback is used to train an advantage function by leveraging a large margin classification loss. The visual highlighting is used to supervise feature learning by way of a consistency loss on perturbed data (i.e. blur segments not highlighted by human).