Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
Authors: Lin Guan
Mudit Verma
Sihang Guo
Ruohan Zhang
Subbarao Kambhampati
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: human-in-the-loop reinforcement learning
interactive reinforcement learning
explanations
saliency map
visual explanations
human-agent interaction
multimodal human-agent interaction

Abstract: Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HIRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation
Lin Guan
Mudit Verma
Sihang Guo
School of Computing & AI
School of Computing Al Arizona State University Tempe, AZ 85281 mverma130asu, edu
Department of Computer Science The University of Texas at Austin Austin, TX 78712 sguo19@utexas edu
Arizona State University
Tempe, AZ 85281
1guan90asu.enn edu
Ruohan Zhang
Subbarao Kambhampati School Computing & AI Arizona State University
Department of Computer Science
Stanford University Stanford, CA 94305
Tempe, AZ 85281
zharu@stanford, edu
raošasu. edu
Abstract
Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form explanation. However, this kind of human guidance only investigated in supervised learning tasks, and it remains unclear how best incorporate this type of human knowledge into deep reinforcement learning. this paper, present the first study of using human visual explanations in human- in too--ho-lop reinforcement learning (HIRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" "bad" feedback for queried state-action pairs, but also provides visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through context-awart data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning. and Human-in -amaan-n--heeloop -n-the-lopp RL baselines that only utilize evaluative feedback.
Introduction
Deep reinforcement learning (DRL) algorithms have achieved many successes in solving problems with high-dimensional state and action spaces [27 However, DRL's performance limited by its sample (in)efficiency. It is often impractical to collect millions of training samples required by standard DRL algorithms. One way to tame this problem is to leverage additional human guidance by following the general paradigm of iiuman-in-the-Lone Reinforcement Learning (HIRL) [51]. It often allows the agent to achieve better performance and higher sample efficiency.
One popular form of human guidance in HIRL is the binary evaluative feedback 15 in which humans provide "good" "bad" judgment for a queried state-action pair. This framework allows
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

non-expert humans to provide feedback, but its sample efficiency could be further improved by asking humans to provide stronger guidance. For example, the binary feedback does not tell the agent why made a mistake. If humans can explain the "why" behind the evaluative feedback, then it is possible to further improve the sample efficiency and performance.
Taking the training of an autonomous driving agent as motivating example, humans can "explain" the correct action of "apply-brake" by pointing out to the agent that the "STOP" sign an essential signal. One way to convey this information is through saliency information, in which humans highlight the important (salient) regions of the visual environment state. The visual explanation will indicate which visual features matter the most for the decision in the current state. Note that requiring human trainers to provide explanations on their evaluations does not necessarily require any more expertise on their part than is needed for providing only binary feedback. In our driving agent example, the human trainers may not know things like the optimal angle of the steering wheel; however, we can expect them to be able to tell whether an observed action is good or bad, and what visual objects matter for that decision.
In fact, the use of human visual explanation has been investigated by recent works in several
supervised learning tasks [39] 35 34 They
Query Trajectory
Action
show that the generalization of a convolutional neural network can be improved by forcing the model to output the same saliency map as human in other words, forces to model to make the right prediction for the right reasons. However, it remains unclear how to effectively incorporate domain knowledge in visual explanation in deep reinforcement learning.
Drayy Exacadive
Fecdbacks
Agent
Human
Context-
Anara Data Augmentation
Annotationa
Sotte,
Figure 1:
Overview of EXPAND. The agent
queries the human with a sampled trajectory for bi- nary evaluative feedback on the action and saliency annotation on the state. The Perturbation Module supplements the saliency explanation by perturb- ing irrelevant regions. The agent then consumes the integrated feedback and updates its parameters. This loop continues until the agent is trained with feedback queried every N₂ episodes. The domain shown for "Visual Explanation' is Pixel Taxi.
In this work, we present EXPAND, which aims to leverage- EXPlanation AugmeNted feeDback (Fig. I to support efficient human guidance for deep reinforcement learning. This raises two im- mediate challenges (i) how to make it easy for humans to provide the visual explanations and (ii) how to amplify the sparse human feedback. EXPAND employs a novel context-aware data augmentation method, which amplifies the dif- ference between relevant and irrelevant regions
in human explanation by applying multiple perturbations to irrelevant parts. To reduce the human effort needed to provide visual explanations, EXPAND uses fet-the-shhll object detectors and trackers to allow humans to to give explanations in terms of salient objects in the visual state, which are then automatically converted to saliency regions to be used by the RL system. We show that EXPAND agents require fewer interactions with the environment (environment sample efficiency) and over 30% fewer human signals (human feedback sample efficiency) by leveraging human visual explanation.
We highlight the main contributions of EXPAND below:
This is the first work that leverages human visual explanation in human-int Humman-in-th--lee -lop rein- forcement learning tasks. We show that EXPAND is the state-of-the-arr method to learn from human evaluative feedback in terms of environment sample efficiency and human feedback sample efficiency.
We benchmark our context-aware data augmentation against standard context-agnostic data augmentation techniques. Our experiment results help shed light on the limitations of existing data augmentations and can benefit future research in accelerating RL with augmented data.
We show that human visual explanation in a sequential decision making task can be collected in a low-effort and semi-automatee way by using an off-the-shee object detector and tracker.
We note that EXPAND agents have applicability beyond improving the efficiency of human-in-the- loop RL with sparse environment rewards. In particular, they can be equally useful accepting human guidance to align the RL system to human preferences. They can be used to incorporate

---

rewards from human feedback, for example, when environment reward is not defined upfront or to accommodate additional constraints that are not reflected in existing reward signal. EXPAND. thus, can also be useful in aligning objectives of humans in the loop to the agent's 15 [ Finally, EXPAND's 8 approach of allowing humans to provide their explanations and guidance in terms of objects of relevance to them, and automatically converting those to saliency regions, is an instance of
the general approach for explainable and advisable AI systems that use symbols as a lingua franca for communicating with humans, independent of whether they use symbolic reasoning internally [13].
2 Related Work
Leveraging human guidance for RL tasks has been extensively studied in the context of imitation
learning (learning from demonstration) (37. inverse reinforcement learning (29 [], reward shaping
(28. learning from human preference 7 22 and learning from saliency information provided by
human 491 Surveys on these topics are provided by [51 46).
Compared to these approaches, learning from human evaluative feedback has the advantage of placing minimum demand on both the human trainer's expertise and the ability to provide guidance (e.g. the requirements of complex and expensive equipment setup). Representative works include the TAMER framework [15 45], and the COACH framework [25] The TAMER+RL framework
extends TAMER by learning from both human evaluative feedback and environment reward signal further augments TAMER+RL by utilizing deep neural networks to learn in high dimensional state space [2] Several approaches have been proposed to increase the information gathered from human feedback, which takes into account the complexities in human feedback- providing behavior. Loftin et al.] speed up learning by adapting to different feedback prroviiggg treats human feedback as direct policy labels and uses a probabilistic model to learn from inconsistent feedback. Although these approaches better utilize human feedback with improved modeling of human behaviors, weak supervision is still fundamental problem in the evaluative feedback approach.
[7
strategies 23 24
the Advice framework [11 6
Human explanatory information has also been explored previously. The main challenge of using human explanation is to translate human high-level linguistic feedback into representations that can be understood by the agents. As early attempt, Thomaz et al. allow humans to give anticipatory guidance rewards and point out the object tied to the reward [44]. However, they assume the availability of object-oriented representation of the state. Krening et al.) Yeh et al.) resort to human explanatory advice in natural language 19 48). Still, they assume a recognition model that can understand concepts in human explanation and recognize objects. In this work, we bridge the vocabulary gap between the humans and the agent by taking human visual explanations in the form of saliency maps. Other works use human gaze data as human saliency information, collected with sophisticated eye trackers, to help agents with decision making in an imitation learning setting [50 52, 14]. They require human data to be collected offline in adncece. In contrast, this work is closer to Arakawa et al. Xiao et al. Cui et al.] where human trainers are required to be more actively involved during training
Recent works in supervised learning also use human visual explanation to extend the numan-machine The visual explanation is used as domain knowledge to improve the performance of machine learning models. The way we exploit human explanation via state perturbation can be viewed as a novel way of data augmentation. Data augmentation techniques are widely used in computer vision tasks (21 40 and have recently been applied to deep RL tasks [T8 20 33 26 The framework explanatory interactive learning [43] also leverages human visual explanation with data augmentttmttnnoon.... However, they only focus on classification tasks with low-dimensional representations.
communication channel [35 34 43).
3 Problem Setup
We intend to verify the hypothesis that the use of visual explanation and binary evaluative feedback can boost an RL agent's performance. We have an agent M that interacts with an environment through a sequence of actions, states, and rewards. Following standard practice in deep RI. to make states Markovian, (k: 4) preprocessed consecutive image observations are stacked together to form state st [xt-(k-1) 1): .... 2z
[27]. At each time-step t, the agent can select an action
3

---

(1) Hence, the advantage value quantifies the possible (dis)advantage the agent would have if some other action were chosen instead of the current-best. It can be viewed as the agent's 's judgment on the optimality of an action. Positive feedback means the human trainer expects the advantage value from set of all possible actions A = {1,..., K} for the state E S. Then the agent receives a reward 7y R from the environment& &. This sequence of interactions ends when the agent achieves its goal or when the time-step limit reached. This formulation follows the standard Markov decision process framework in RI. research 42]. The agent's ' goal is to learn a policy function т, mapping from state to action, that maximizes the expected return. For deep Q-Learning agent, the policy ı is approximated by the Q function Q(s, a), which represents the maximum expected rewards for taking action at state
Additionally, we assume human trainer who provides binary evaluative feedback H. (h2, h2, ...h,) that conveys their assessment of the queried state-action pairs given the agent trajectory. We define the feedback as h4 (z)) by. 2t, at, st), where by { 1, 1] a binary "bad" or "good" feedback provided for action de, and xn [Baz1,-.. .... Bor..) } is saliency map for the image Ty in state 8t- Boz; is a tuple (z, y, w, h) for the top left Euclidean coordinates I and y, the width w, and the height h of the bounding box annotated on the observation image Zt.
4 Method
EXPAND aims to improve data efficiency and performance with explanation augmented feedback. In the following, we will first describe how EXPAND simultaneously learns from both environment reward and binary evaluative feedback. Then we introduce our novel context-aware data augmentation that utilizes domain knowledge with human explanation. Algorithm I in Appendix A presents the vrinnintteaccton loop of EXPAND. Within an episode, the agent interacts with the environment and stores its transition experiences. Every few episodes, it collects human feedback queried on a trajectory sampled from the most recent policy. All the human feedback is stored in a feedback buffer similar to the replay buffer in off-policy DRL [27]. The weights of RL agent are updated twice, first using sampled environment data as in usual RL update, and then using human feedback data, in single training step. In our experiment, both EXPAND and the baselines follow the same train-interaction loop.
4.1 Learning from Environment Reward and Binary Feedback
The underlying RL algorithm of EXPAND can be any off-policy Q-learning- -baed algorithm. In this work, use Deep Q-Networks 27) combined with multi-step returns, reward clipping, soft-target network updates and prioritized experience replay (38. We refer to this RL approach as Efficient DQN.
To learn from binary evaluative feedback, we propose new method that doesn't 't require additional parameters to explicitly approximate human feedback. We use the advantage value to formulate the feedback loss function, called advantage loss. Advantage value is the difference between the
Q-value of the action upon which the feedback was given and the Q-value of the current optimal action calculated by the neural network. Given the agent's current policy π,, state s, and action a for which the feedback is given, the advantage value is defined as:
1'(ssa) = Q" (s. - (s) = Q" (s. - (s)))((s))
of the annotated action to be zero. Therefore, we define a loss function, i.e., the advantage loss, which forces the network have the same judgment on the optimality of action as the human trainer. Intuitively, we penalize the policy-approximate when marked "good" action is not chosen as the best action, or when a marked "bad" action is chosen as the best action.
For a feedback h = (Ir*, ae".B' b",,,,,, xr. s), when the label is "good", i.e., 8² = 1, we expect the network to output a target value Á(s, a) 0, so the loss can be defined as |A(s, a) A (s, =2 (s, (s)) Q* a). When the label is "bad", i.e., 5²h = 11, we expect the network to output an advantage value A" a) < O. Since here we do not have a specific target value for A AT a), we resort to the idea of large margin classification loss B1], which forces Q" a) to be at least a margin im lower than the Q-value of the second best action, i.e., max.' Q"((,, d'). One advantage of this interpretation of human feedback is that it directly shapes the Q-values with the feedback information and does not require additional parameters to model human feedback. This kind of large margin loss has been shown to be effective in practice by previous works like DQID [12.

---

Formally, for human feedback h = (x", ,,, a, a, and the corresponding advantage value As,a (s, a), the advantage loss is:
La(s, a,) =L Good (8, (s,0,, + LA Bud (s, (s,a,h)
(2)
where
L (s, =1) f0
: As, 0 b1)== (00= -t)) :otherwise
(3)
and,
LA Bad (s, a, h; = -1)= (() - (maxwsy (s) (m) Assa =0
A.,a < 0
(4)
Note that the COACH framework [2] also interprets human feedback as the advantage function. In COACH's formulation, the advantage value is exactly the advantage term in the policy gradient gquation-human trainers supposed to provide positive/negative policy-dependent feedback only when the agent performs an action better/worse than that in current policy. Thus, such a formula is restricted to on-policy policy-gradies methods. In contrast, the advantage function in EXPAND aims to capture the relative utility of all the actions. Here human feedback is direct policy advice as in the Advice framework [.] indicating whether an action is preferable regardless of the agent's current policy. This property makes the advantage loss in EXPAND a better fit for off-policy value-based methods.
4.2 Leveraging Human Visual Explanation
Human visual explanation informs the agent about which parts of the state matter for making the right decision. These "parts" of the state could be specific regions or objects. In EXPAND, each saliency map consists set bounding boxes over the images, marking a region's 's importance in making the decision. The intuition is that the agent's 's internal representation must correctly capture the relevant features before can make an optimal decision. Based on this, we propose a novel context-awart data augmentation technique, which applies multiple perturbations to irrelevant visual regions in the image and forces the model to be invariant under these transformations. The key idea here is that, the manipulations to irrelevant regions should not affect the agent's policy.
To get more insights on how our data augmenta- tion method benefits policy learning, we follow the mausal-graph interpretation of representation learning with data augmentation visualized in Fig. 2a [66]. In this model, each image obser- vation X can be expressed by content variables and style variables. Content variable contains all the necessary information to make the cor- rect prediction, while style variable contain all other information that doesn't influence cur- rent downstream task. That said, only content causally related to current downstream get of interest Y, and content C' is independent of style S. The goal of representation learning is to accurately approximate the invariant part (content) and ignore the varying parts (style). Based on this formulation, data augmentation is essentially way emulate style variability by performing interventions on the style variables S. However, since the choice of data augmenta- tions implicitly defines which aspects of the data are designated as style and which are content, to make sure the context information C is preserved, existing data augmentation methods in RL are limited to conservative transformations such as random translation or random cropping. Previous works show that transformations altering content can even be detrimental, which referred as aggressive augmentations in 32)
Gone
Representaion Learning
(a)
(b)
Figure 2: (a) Modeling representation learning problem with data augmentation using a causal graph. Figure from (26) (b) Example state of Pixel-Taxi, the top image represents the original state and the bottom represents a sample aug- mented state.
5

---

Different from standard data augmentations in RL, prior domain knowledge about state context is provided by human explanation in EXPAND. Hence, we can resort a wider range of transformations, and thereby more informatively highlight the content variable C' by applying various transformations only to regions marked as irrelevant (style variable S) while keeping the relevant parts (content variable C)) unchanged. Figure 2b shows an example state from Pixel-Taxi domain where the top image is the original state having gray cell as the taxi, red as the passenger, and black as the destination. The bottom image shows example image used by EXPAND which contrastively highlights the taxi and passenger by performing a Gaussian blur over the remainder of the image observation. This illustrates why the proposed context-aware augmentation can be more informative than standard data augmentations.
We choose to use Gaussian blurring to perturb the irrelevant regions, which is also used in a previous Explainable RL work [0]. The reason we use Gaussian blurring that, it effectively perturbs objects in image while does not introduce new information. One counterexample here can be, transformations like cutout might significantly change state content environments where small black blocks have particular semantic meaning.
Formally, consider a feedback = (x", U/ 2;, a, 8), we need to convert state s into augmented states f(s) with perturbations on irrelevant regions. Let M(1, i.j) denote a mask over relevant regions, where ji denotes the pixel at index (i,j) for image æ. We then have:
M(x,i,) = (
-(1
(i. lies Box, Box E 6
(5)
otherwise
6(x, M,i,) = 2
i,,), G(z, oc) O M(a, i,5)
(6)
Then we can perturb pixel (i,j) in image observation x according to mask MI using a function ø defined as above, where O is the Hadamard Product and function G(x, oG) is the Gaussian blur of the observation æ. Hence, we can get an augmented state [()) with perturbed irrelevant regions by applying 6(x M) to each stacked frame with the corresponding mask MI.
We use the context-ave aware data augmentation in the following two ways:
1. For any human feedback h (z bh, I. a, s). we train the model by calculating the advan- tage loss with the original data h as well as the augmented feedback h = (a bA, 2, a, [s)). Note that one human feedback can be augmented to multiple feedbacks by varying the parameters of fs). In EXPAND, we use Gaussian perturbations of various filter sizes and variances (see Appendix for detailed settings).
2. To encourage the RL model's internal representation accurately capture relevant visual features and ignore other irrelevant parts, we enforce an explicit invariance constraints via an auxiliary regularization loss:
L1
(Q(s,,) QU)(s))))l
where A is the action set and g the number of perturbations.
4.3 Combining Feedback and Explanation Losses
We linearly combine all the losses to obtain the overall feedback loss:
LF AALA + XrLs
(8)
where Aa and xy are the weights of advantage loss and invariant loss respectively. In all experiments,
we set A 1.0 without doing byper-parameter tuning. For Ar, we set 0.1 as suggested in (33). which also applies an invariant constraint on the RL model but with standard data augmentations. The agent is trained with usual DQN loss as well as Lp. Note that in EXPAND, the advantage loss is computed with both original human feedback and augmented human feedback. For a baseline agent that doesn't 't utilize human visual explanation (only trained with usual DQN loss and advantage loss with original human feedback), we refer to it as DoN-Feedback.
4.4 Collecting Human Visual Explanation

---

Though human visual explanation offers a strong way to make numan-agent interaction more natural and effective, it might impose an excessive amount of workload if we require the trainer to annotate the image for every single query. To account for possible overheads and reduce hu- man effort, we design an object-oriented interface that enables the human to provide guidance by effortlessly pointing out the labels of salient objects. Note that this design is in line with the idea of using a "symbolic" inter- face in human advisable AI, which argues that humans are more comfortable with symbol-level communication, and the agents are free to learn their own internal representa- tion as long the user interface can ground the human advice (e.g. to pixel space) [3]. Technically, our object- oriented interface contains tracking and detection module, with which the human trainers only need to annotate at the beginning or when the tracker/detector works imperfectly. For example, in the car driving game Enduro (Fig. 3) all the lanes and cars are automatically highlighted and tracked, so the human trainers only need to deselect irrelevant objects in the image.
Figure 3: An example of object detection in driving game Enduro. Users were allowed to remove or add other bounding regions at will.
A screenshot of our video-player like interface can be found in Appendix F. Trainers start/pause the replay of the queried trajectory and provide explanations by directly drawing bounding boxes on the screen. The trainers can also adjust the "video" frame rate based on their needs. Similar to Deep-TAMER [45), we also applied each received feedback and explanation to frames that are displayed between 2 and 0.2 seconds before the feedback occurred we assume that within this 1.8-second window, the salient regions/objects should be the same.
Experimental Evaluation
The experimental evaluation of the work would try to answer the following questions.
1. Whether the use of human explanation improves the environment and feedback sample efficiency?
2. Does EXPAND utilize the human explanation better than other baselines?
3. Is context-aware data augmentation more informative than standard data augmentation?
To answer the first question, we compare EXPAND DON-Feedback and an HIRL algorithm DON-TAMER which combines TAMER with deep RL. (2].
For the second question, we compare EXPAND with two other explanatory interactive learning methods that are adapted from supervised learning, namely Ex-AGIL. and Attention-Align. Ex-AGIL. is adapted from AGIL [50]. which trains separate attention prediction network to generate visual explanation for unseen states. The predicted attention is used as a mask to filter out irrelevant information in the image observation. Then the masked state is fed input to the policy network. Attention-Align uses an auxiliary attention alignment loss that is similar to the loss functions for supervised learning in [36]]]]]]]] 39 34 35). It penalizes if the agent's attention heatmap does not align with human visual explanation. To efficiently obtain the differentiable attention heatmap of the model, we
Explainable RL algorithm FLS 30). The saliency map produced by FLS-DQN is essentially the model's prediction on whether a pixel should be included in a bounding box. Hence the attention alignment loss here is defined be the mean square error between agent's prediction and human visual explanation. The implementation details of the baselines can be found in Appendix C.
Finally, to answer the third question, we replace the context-aware data augmentation in EXPAND with standard augmentations that do not use human saliency information. The context-agnostic augmentations we compare to include Gaussian blurring and random cropping, which result in state-of-the toaftthh-att- art sample efficiency in recent works [20 [8]. Note that EXPAND only augments states which were queried to the human trainer, hence for this comparison, the context-agnostic methods only augment the states for which the system received human feedback.
We conducted experiments on three tasks: Pixel-Taxi (Fig. 2b) and four Atari games. The Pixel-Taxi domain is similar to the Taxi domain which is widely used in RL research [9]. It is grrd-world
7

---

setup in which the taxi agent, occupies one grid cell at a time, and passengers (denoted by different colored dots) occupy some other cells. The taxi agent's goal is to pick up and transport the correct passenger to the destination. To force the agent to learn passenger identities instead of memorizing their "locations", we randomize the passengers' positions at the beginning of each episode. A reward is given only when the taxi drops off the correct passenger at the destination cell. In addition, we choose four Atari games with default settings: Pong, Asterix, Montezuma's Revenge, and Enduro. Original Enduro can be infinitely long and can make human training impractical. Therefore, in Enduro, our goal is to teach the agent to overtake as many cars as possible within 1000 environment steps (an episode); hence we denote this task as Enduro-1000 In Montezuma's 's Revenge, train the agent to solve the first room within 1000 environment steps per episode; hence we denote this task as MR Level 1 or simply MR.
In all experiments, both EXPAND and the baselines use Efficient DQN as the underlying RL. approach. Efficient DQN uses the same DQN network architecture designed by MMihh et al. [2015). Details on the architecture and hyperparameters can be found in Appendix E. Following the standard pre- processing 127]. each frame is converted from RGB format to grayscale and is resized to 84 x 84. The input pixel values are normalized to be in the range of [O, 1J. During training, we start with an e-greedy policy (e 1.0) and reduce by factor of X at the end of each episode until it reaches 0.01. All the reported results are averaged over random seeds.
Algorithm describes the steps for obtaining human feedback: for every N, (N, 4) episode, we sample one trajectory and query the user for binary evaluative feedback as well as a visual explanation. Active querying, although preferable, is left for future experiments since the goal of this work is to demonstrate whether human explanations can effectively augment binary feedback. When collecting feedback, we allow humans to watch the queried trajectories and provide feedback at will (the human can choose not to provide feedback for some queried states).
5.1 Evaluation using Synthetic Feedback and Explanation from Oracle
Figure 4:
The smoothed learning curves of EXPAND and the baselines. The solid lines show the mean score over random seeds. The shaded regions represent the standard error of the mean. In Pixel-Taxi, the score is running average over the last 20 rollouts. Our method (EXPAND, in red) outperforms the baselines in all the tasks.
To perform a systematic analysis, we first conducted experiments that include 5 runs of all algorithms with a synthetic oracle. A synthetic oracle allows us to run a large number of experiments and give consistent feedback across different runs, providing fair and systematic comparisons between different approaches []. The oracle uses trained models from Atari Zoo [4]] for the Atari games, and trained DQN model for Pixel-Taxi. To annotate the "relevant" regions on the image observation, we use hand-coded models to highlight the taxi-cell, the destination-cell, and the target-passenger in Pixel-Taxi; the two paddles and the ball in Atari- Mari-Pong; the player vehicle, lanes, and other vehicles in front of the agent in Enduro- 1000. In both MR and Asterix the oracle highlighted the agent and other regions like, monster, ladder and key in MR, and enemies and target in Asterix, depending upon whether they are spatially close to the agent. (See Fig. and Appendix D for other examples).
Improvements on Environment and Feedback Sample Efficiency : Fig. 4 compares the environ- ment sample efficiency as well as performance between EXPAND (in red) and the baselines. To answer the first question posed in Section [5] the plot clearly shows that EXPAND outperforms the and -QN-TAMER) by a large margin across all the tasks except
HIRL baselines MR, where EXPAND was consistently at par with DQN-Feedback. PAND is able to learn a near optimal policy with 35% less environment samples/ feedback samples. In Enduro- 1000, EXPAND manages to consistently obtain a score over 60 by using 80k samples compared to 120k samples used by DON-TAMER, an over 30% improvement. In Asterix and MR.
Pixel-Taxi and Pong, EX-
8

---

EXPAND achieved a considerably higher score than DQN-TAMER Finally, since human feedback obtained fixed intervals, an improvement in environment sample efficiency (x axis in the figure would imply subsequent improvement in feedback sample efficiency.
EXPAND versus other Explanatory Interactive Learning methods:
rong
1003
MR Lavel1
0.5
0.4
0.2
130k
1006
206
60x
206
1006
2Dk
80k
80x
100K
Figure 5: Comparison of EXPAND with other perturbation techniques and attention-basen Interactive
Learning methods.
As mentioned earlier, to answer the second question posed in Section 5 we compare EXPAND with Ex-AGIL and Attention Align. From Fig. [ we can observe that Ex-AGIL only improves the baseline DQN-Feedback in Pong, while in other more visually-complex environments, the auxiliary attention prediction harms the performance. This highlights two issues with approximating human attention with an additional model, first, the approximation may fail for unseen states and second, the visual explanation is not used as strong signal directly regularize the policy network. On the other hand, the second baseline Attention-Align fails to learn a usable policy in all five tasks. potential reason for its failure objective. This indicates that this type of attention alignment loss might not be
misalignment between the attention prediction objective and
the reward-seekin suitable for a less stable learning system like deep RL. In contrast, data-augmentation methods for RL have been empirically examined over the years, hinting that EXPAND's methodology is more stable.
EXPAND versus Consext-Agnostic Data Augmentations: Additional comparisons between EXPAND and standard data augmentations were made to verify our hypothesis that context-aware data augmentation can be more informative than context-agnostic augmentations. As expected, Gaussian blurring and random cropping fail to help the agent (Fig. (5. Interestingly, the two context- agnostic data augmentations even degrade the performance in some tasks since they add unnecessary complexities when the agent tries to infer human trainer's ' intent behind the binary evaluative feedback. This contradicts EXPAND's methodology, which contrastively highlights relevant regions. This result suggests that standard data augmentation helps RL by obtaining more data to prevent overfitting [88. but its informativeness can be further improved by incorporating domain knowledge as in EXPAND.
5.2 Evaluation with Human in the Loop
So far, we have presented our results using synthetic oracles. In this section, will present results with a human trainer. We run this experiment on Enduro-1000. The objective of this user study is to address the following problems:
1. Can EXPAND perform well with feedback and explanation from human trainer, considering human feedback can be sparser and noisier than synthetic feedback?
2. Is there a low-cost way to collect human visual explanation in sequential decision tasks?
In the experiment, the agent queried the human at an interval of every 10 episodes. We limit the annotation time for each query to be around 5 minutes. Hence the total interaction time is at most 30 minutes. Each algorithm was run 3 times with different random seeds.
Within the 30-minute interaction time, in the baseline DQN- DQN-Feedback, human trainers only need to give binary evaluative feedback, and the trainers provided 2405 binary feedbacks on average. In EXPAND, the human trainers provided 2026 beedbcck-epplanain
pairs on average within the same time limit. The difference in the number of feedbacks is not large so it suggests that the cost of providing visual explanation is low. From Fig. 4 we can observe that EXPAND significantly outperforms DeN-Feedback in the user study, hinting that the use of human explanation can also improve sample complexity given the same wall-clock interaction time.

---

5.3
Ablation Study
18
400
cample
53k
1036
1504
504
1006
100%
200
404 600 60C
Figure 6: Ablation experiments analyzing the effects of each individual loss terms on the performance. The results verify that the combination of both loss terms leads the best performance.
To get more insights into the losses in EXPAND, we conducted an ablation study on two variants of EXPAND, which either only use the augmented data to compute the invariance loss (EXPAND-no- augmented-aadvantage) or only use the augmented data in the advantage loss S-XPAAND-no-ivvarianee) Fig. 6 shows that both variants of EXPAND eventually converged to near-optimal policies. We note (in purple) and perform significantly better than Deeekked highlighting that each saliency loss alone provides significant improvements over the baseline. Finally, combining these losses (EXPAND, in red) boosts this performance even further, indicating that combining invariance constraint with advantage loss with augmented data is a better approach.
that EXPAND-no-invariin
(in light green)
6 Conclusion & Future Work
In this work, we presented novel method to integrate human visual explanation with their binary evaluations of agents' actions in Human-in-the-Loon RL paradigm. We show that our proposed method, EXPAND, outperforms previous methods terms of environment sample efficiency, and shows promising results for human feedback efficiency. Future work can experiment with different types of perturbations beyond Gaussian blurs, including deeainnde perturbations that involve object manipulation. Finally, we mentioned the end of introduction, EXPAND agents an example of AI agents that can take human advice in terms of a symbolic lingua franca [33, and a promising approach for the more general problem of human-advisable RI. systems that can be aligned to human preferences through symbolic advice.
Acknowledgement
This research is supported in part by ONR grants N00014- 16-1-2892, N00014-18-1- 2442. N00014- 18-1-2840. N00014-9- 1-2119, AFOSR grant 1199550-18--00661 DARPA SAIL-ON grant W91 INF-
19- 2-0006 and a JP Morgan AI Faculty Research grant. Ruohan Zhang's 's work on this paper was done when he was PhD student at The University of Texas at Austin.
References
[1] Pieter Abheel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. Proceed- ings of the twenty-fist international conference on Machine learning, page 1, 2004.
[2] Riku Arakawa, Sosuke Kobayashi, Yuya Unno, Yuta Tsuboi, and Shin-ichi Macda. Dqn-tamer: Human-in- thop reinforcement learning with intractable feedback. arXiv preprint arXiv:1810.111 2018.
[3] Kai Arulkumaran, Marc Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath, Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
[4] Dilip Arumugam, Jun Ki Lee, Sophie Saskin, and Michael L Littman. Deep reinforcement leaming from policy-dependent human foedback. arXiv preprint 1rXiv:1922.04257 2019.
[5] Andreca Bobu, Marius Wiggert, Claire Tomlin, and Anca D Dragan. Feature expansive reward learning: Rethinking human input. In Proceedings of the 2021 ACM/IEEE International Conference on Human--Roott Interaction, pages 216 226,224,22 2021.
10

---

[6] Thomas Cederborg, Ishaan Grover. Charles L Isbell, and Andrea L Thomaz. Policy shaping with human teachers. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martie, Shane Legg, and Dario Amodei. Deep reinforce- ment learning from human preferences. In Advances in Neural Information Processing Systems, pages 4299-4307, 2017.
[8] Yuchen Cui, Qiping Zhang, Alessandro Allievi, Peter Stone, Scott Nickum, and W Bradley Knox. The empathic framework for task learning from implicit human feedback. In Conference Robot Learning. PMLR, 2020.
[9] Thomas G Dietterich. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-303, 2000.
[10] Sam Greydanus. Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. arXiv preprint arXiv: 1at1:1111000000 2017.
[11] Shane Griffith, Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L. Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In Advances in neural information processing systems, pages 2625 2633, 2013.
[12] Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-learning from demonstrations. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[13] Subbarao Kambhampati, Sarath Sreedharan, Mudit Verma, Yantian Zha, and Lin Guan. Symbols as lingua franca for bridging human-ai chasm for explainable and advisable ai systems. In AAAI, 2022.
[14] Heecheol Kim, Yoshiyuki Ohmura, and Yasuo Kuniyoshi. Using human gaze to improve robustness against irrelevant objects robot manipulation tasks. IEFE Robotics and Automation Letters, 2020.
[15] Bradicy Knox and Peter Stone. Interactivcly shaping agents via human reinforeement: The tamer framework. In Proceedings of the fifth international conference on Knowledge capture, pages 9-16, 2009.
[16] W Bradley Knox and Peter Stone. Combining manual feedback with subsequent mdp reward signals for reinforcement learning. In Proceedings of the 9th International Conference Autonomous Agents and Multiagent Systems: volume I-Volume I, pages 5-12. Citeseer, 2010.
[17] W Bradley Knox and Peter Stone. Reinforcement learning from simultaneous human and mdp reward. In AAMAS, pages 475-482, 482, 2012.
[18] Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.1366 136499 2020.
[19] Samantha Krening, Brent Harrison, Karen M Feigh, Charles Lee Isbell, Mark Riedl, and Andrea Thomaz. Learning from explanations using sentiment and advice in rl. IEEE Transactions on Cognitive and Developmental Systems, 91):44-55, 2016.
[20] Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abheel, and Aravind Srinivas. Reinforce- ment learning with augmented data. In Advances in Neural Information Processing Systems, volume 33, 2020.
[21] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):22)8-22222 2324, 1998.
[22] Kimin Lee, Laura Smith, and Pieter Abbeel. Pebble: Feedback- interactive reinforcement learning via relabeling experience and unsupervised pre pre-training. In ICML 2021: 38th International Conference on Machine Learning, pages 6155-6163, 2021.
[23] Robert Loftin, Bei Peng. James MacGlashan, Michael L Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning something from nothing: Leveraging implicit human feedback strategies. In The 23rd IEEE International Symposium on Robot and Human Interactive Communication, pages 607-612. IEEE, 2014.
[24] Robert Loftin, Bei Peng. James MacGlashan, Michael Littman, Matthew E Taylor, Jeff Huang, and David L Roberts. Learning behaviors via human delivered discrete feedback: modeling implicit feedback strategies speed up learning. Autonomous agents and multi- agent systems, 30(1):30-5) 59, 2016.
11

---

[25] James MacGlashan. Mark K Ho, Robert Loftin, Bei Peng. Guan Wang, David L Roberts, Matthew E Taylor. and Michael L Littman. Interactive learning from policy-dependent human feedback. In Proceedings the 34th International Conference on Machine Learning- Volume 70. pages 2285-2294, JMLR. org, 2017.
[26] Jovana Mitrovic, Brian McWilliams, Jacob Walker, Lars Buesing, and Charles Blundell. Representation learning via invariant causal mechanisms. arXiv preprint arXiv:2010.07922, 2020.
[27] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Jocl Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Noture, 518(7540):529-533 2015.
[28] Andrew Y Ng. Daishi Harada, and Stuart Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In ICML, volume 99, pages 278-287, 1999.
[29] Andrew Y Ng. Stuart J Russell, et al. Algorithmss for inverse reinforcement learning. In Icml, volume 1, page 2, 2000.
[30] Dmitry Nikulin, Anastasia Ianina, Vladimir Aliev, and Sergey Nikolenko. Free-lunch saliency via attention in atari agents. arXiv preprint (5t1::998000555 2019.
[31] Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted bellman residual minimization handling expert demonstrations. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 549-564. Springer, 2014.
[32] Senthil Purushwalkam and Abhinay Gupta. Demystifying contrastive self- superviidd learning: Invariances, augmentations and datasct biases. arXiv preprint 2rXiv:2007.13916 2020.
[33] Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmen- tation for generalization deep reinforcement learning. arXiv preprint (tti:2000.128222 2020.
[34] Laura Rieger, Chandan Singh, William Murdoch, and Bin Yu. Interpretations are useful: penalizing explanations to align neural networks with prior knowledge. In International Conference on Machine Learning, pages 81 16-8126. PMLR, 2020.
[35] Andrew Slavin Ross, Michael C Hughes, and Finale Doshi Velez. Right for the right reasons: Training differentiable models by constraining their explanations. arXiv preprint arXiv: arXiv: 1703.03717, 2017.
[36] Akanksha Saran, Ruohan Zhang, Elaine Schaertl Short, and Scott Niekum. Efficiently guiding imitation learning algorithms with human gaze. arXiv preprint arx: 12500, 2020.
[37] Stefan Schaal. Learning from demonstration. In Advances in neural information processing systems, pages 1040-1046, 1997.
[38] Tom Schaul, John Quan, Joannis Antonoglou, and David Silver. Prioritized experience replay. In ICLR 2016: International Conference on Learning Representations 20/6, 2016
[39] Patrick Schramowski, Wolfgang Stammer, Stefano Teso, Anna Brugger, Franziska Herbert, Xiaoting Shao, Hans Mans-Georg Luigs, Lnne-Katrin Mahlein, and Kristian Kersting. Making deep neural networks right for the right scientific reasons by interacting with their explanations. Nature Machine Intelligence, 2(8):
476-486, 2020.
[40] Connor Shorten and Taghi M Khoshgoftaar. A survey image data augmentation for deep learning. Journal of Big Data, 6(1 6(1):60, 2019.
[41] Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Jiale Zhi, Ludwig Schubert, Marc G. Bellemare, Jeff Clune, and Joel Lehman. An atari model zoo for analyzing, visualizing, and comparing deep reinforcement learning agents. In Proceedings of the Twenty-Eigtth International Joint Conference on Artificial Intelligence, pages 1260032633 2019.
[42] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
[43] Stefano Teso and Kristian Kersting. Explanatory interactive machine learning. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 239-245,70 2019.
[44] Andrea Lockerd Thomaz, Cynthia Breazeal, et al. Reinforcement learning with human teachers: Evidence of feedback and guidance with implications for learning performance. In Agai, volume 6, pages 1000-1005. Boston, MA, 2006.
12

---

[45] Garrett Warnell, Nicholas Waytowich, Vernon Lawhern, and Peter Stone. Deep tamer: Interactive agent shaping in high-dimensional state spaces. In Thirty-Second AAAI Conference Artificial Intelligence, 2018.
[46] Christian Wirth, Riad Akrour. Gerhard Neumann, and Johannes Fümkranz. survey of preference-base based reinforcement learning methods. The Journal of Machine Learning Research, 18(1):4945 4990, 2017.
[47] Baicen Xiao, Qifan Lu, Bhaskar Ramasubramanian, Andrew Clark, Linda Bushnell, and Radha Poovendran. Fresh: Interactive reward shaping high-dimensee dimensional state spaces using human feedback. arXiv preprint arXiv:2001.06781, 2020.
[48] Eric Yeh, Melinda Gervasio, Daniel Sanchez, Matthew Crossley, and Karen Myers. Bridging the gap: Converting human advice into imagined examples. Advances in Cognitive Systems, 6: 168-1176, 2018.
[49] Zhang, A Saran, B Liu, Zhu, Guo. Niekum, Ballard, and M Hayhoe. Human gaze assisted artificial intelligence: review. In International Joint Conference on Artificial Intelligence, 2020.
[50] Ruohan Zhang, Zhuode Liu, Luxin Zhang, Jake Whritner, Karl Muller, Mary Hayhoe, and Dana Ballard. Agil: Learning attention from human for visuomotor tasks. In Proceedings of the european conference on computer vision (ecov), pages 663-679, 2018.
[51] Ruohan Zhang. Faraz Torabi, Lin Guan, Dana H. Ballard, and Peter Stone. Leveraging human guidance for deep reinforcement learning tasks. In Proceedings of the Twenty-Eighh International Joint Conference Artificial Intelligence, IJCAI-19, pages 1339-6346. International Joint Conferences on Artificial Intelligence Organization, 2019, doi:
[52] Ruchan Zhang. Calen Walshe, Zhuode Liu, Lin Guan, Karl S Muller, Jake A Whritner, Luxin Zhang, Mary M Hayhoe, and Dana Ballard. Atari-head: Atari human eye-tracking and demonstration dataset. arXiv preprint arXiv: 1903.06754, 2019.
13


Reviews:
[{"review_id": "oj-sHXfojFo", "reviewer": "Reviewer_o7pH", "summary": "This paper proposes EXPAND - a method for allowing human input and semantic understanding to guide policy learning for RL agents. Humans are asked to provide binary feedback (good/bad) on actions taken by the agent, as well as highlight visual features which helped guide their decision (e.g. stop sign tells human to stop). The binary feedback is used to train an advantage function by leveraging a large margin classification loss. The visual highlighting is used to supervise feature learning by way of a consistency loss on perturbed data (i.e. blur segments not highlighted by human).", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "It seems the primary difference between EXPAND and previous work DQN-TAMER [1] is that human input is integrated into an Advantage Function (which is also tuned w/ RL), rather than linearly blended with a separate Q function. This change does seem novel and results in a more elegant method with less hyper-parameters. The experimental results show EXPAND significantly outperforms DQN-TANNER in all environments. However, I do wish the authors more precisely described DQN-TAMER, and made these points clearer in the paper.\n\nIt's important to note that most experiments in this paper used an oracle to simulate human input likely to make the study less time consuming. The authors do one experiment using human input on the Enduro-1000 environment, but its difficult to tell if the performance matches the oracle Enduro-1000 runs due to separate y-axis scales. Furthermore, DQN-TAMER is only run with oracle input and not human. I hope the authors fix these bugs, so it becomes more clear if EXPAND can compete with DQN-TAMER when using human input.\n\nFinally, the ablations do a good job justifying visual feature annotation helps with downstream RL task performance. I'm not convinced by the author's claim that Gaussian blurring somehow uniquely doesn't change state content. After all, what if high frequency information is important in the state? However, I do agree that it seems to work fine for the chosen environments.\n\n[1] Arakawa, Riku, et al. \"Dqn-tamer: Human-in-the-loop reinforcement learning with intractable feedback.\" arXiv preprint arXiv:1810.11748 (2018).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would ask the authors to more thoroughly discuss limitations of their method. In particular, if performance is very different between oracle and human input then that is important to discuss. Furthermore, the human input seems far more time consuming to produce. For example, the Enduro-1000 human experiment is only run to 60K steps v.s. 100K+ for oracle. The authors claim this takes total of 30 minutes, but if that's the case why not run for as many steps as the oracle? \n\nRegarding social impacts, the authors claim EXPAND would help humans align the agent's behavior with our own value system. However, this paper does not show experiments where the RL reward is necessarily at odd with human values. Thus, this claim does not seem substantiated, and I would recommend the authors re-write the social impacts section accordingly.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "ClEDlMAR8eP", "reviewer": "Reviewer_1gtV", "summary": "They propose methods to better learn (off-policy) Q functions using binary human feedback, and also using visual explanations from humans.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(disclosure:  I had briefly seen a previous version of this as a poster under a different name.  I do not know the authors and don't think it affects my review)\n\nI like the direction of this work.  It seems good generally for humans to try to indicate subsets of observations that most affect actions as a form of feedback.  Overall, I am somewhat convinced it is a promising method, although I think the presentation of the results could be a lot better.  I would've liked to see experiments in a harder environment where I think the gap between EXPAND and DQN-feedback could've been even larger.  Also would've been nice to see at least a picture of each environment and the oracle behavior, given the nature of the work.\n\nFigure 2: I kinda feel looking at this picture that cutout would be more natural.  would that have been worth trying?\n\nBaselines:\n- DQN-TAMER is never really explained.  It would be nice to explain the difference between it and DQN-feedback, so the reader doesn't need to look at other papers and think about it\n- Not super important, but could context-aware data augmentation be applied to DQN-tamer as another baseline?\n- another baseline to get an understanding of how much augmentation helps, is to run DQN-feedback, but training on an extra frames for each would've-been-augmented frame.  presumably this is a strong baseline, and it would be more impressive (but plausible) to beat it.  if I understand correctly, this would also control better for amount of compute used\n\nImprovements on Environment and Feedback Sample Efficiency\n- it would be good to know what percentage of frames get augmented.  \n\nComparison to explanatory interactive learning methods:  this section overall felt unconvincing to me\n- Attention-Align seems a little complex and the results seem very poor, making me wonder if there were implementation issues or if it was otherwise a poor choice of baseline for some unknown reason.  I'm guessing there's a hyperparameter to tune to balance the attention prediction objective, for example?\n- The comparison of EXPAND with Ex-AGIL seems like it could easily come down to number of oracle/human samples + efficiency of learning the mask, since in one case we have an additional learned model.  If the model is good, then it seems like you get to do a lot more augmentation, right?\n- why were only two environments used in figure 5?  also line 374 says \"all three tasks\" when there are just two\n- might be nice for Figure 5 to have a no-aug baseline\n\nEXPAND versus Context-Agnostic Data Augmentations\n- why not use the augmentations on all states, if there is an oracle?\n- it would be good to know what percentage of frames get augmented.  it would be interesting to see, e.g. 100% gaussian blur vs. gaussian blur except when there's human data\n- overall comparisons here look solid, although again I wonder why there are only 2 environments\n\n5.2 \n- the human trainer seems to do worse than the synthetic oracle.  is this just because the oracle is quite good?  it might be interesting to find a harder task where it's not as easy to write a synthetic oracle - I think results would be more compelling in such a setting.\n- why was DQN-tamer not included in this?\n\n5.3 \n- i assume these were done with the oracle again.  it would be good to explicitly state this\n- again, I wonder why the other environments weren't included here", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "not much here", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "59fTDXV_n3c", "reviewer": "Reviewer_wPs5", "summary": "The paper proposes a way of data augmentation based on human feedback. Namely a human experts is tasked with marking on relevant areas of the state space (visual input, frames) with bounding boxes. The rest of the input is then blurred, which, presumably, should lead to the learning algorithms \"focusing\" on relevant areas and learning to ignore irrelevant (blurred) ones. Experiments on 5 simulated environment show that applying this technique leads to somewhat lower sample complexity and better final performance.\n\nIn my estimation this is an interesting idea on using human-in-the-loop for RL training and would be useful for RL practitioners working on such visual environment where data samples are expensive. Real-world robotics would be one such example.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall I do not have any specific questions or criticism or this work. The idea is valuable, the description is clear and the execution is valid and allows me to trust the results reported in the paper.\n\nOriginality: The idea to use humans to annotate the important regions of the input sample is quite original and in my estimation is well executed in this work, allowing the community to evaluate the benefits of the proposed trick. In general, human in the loop framework offers, in my opinion, a multitude of ways of enhancing the RL process and this work explores an interesting aspect of this area of study.\n\nQuality: The claims are well-formulated and are supported by the empirical results provided. \n\nClarity: The proposed method is presented clearly and with a good level of technical detail allowing to understand the method down to the low-level technical details and mathematical formulation. Experiments and ablations are well-justified and address the right questions.\n\nSignificance: RL is known to be notoriously data hungry which is the main obstacle for its mass adoption in real physical systems where data samples are expensive. The proposed method add a potentially useful tool to the toolbox of RL practitioner working on application where sample complexity is crucially important. I find the contribution of this work to be significant enough to deserve being seen by the RL community. Also, while the result is presented in the context of human-in-the-loop, the Oracle experiments (Section 5.1) demonstrate the benefit of the proposed method also in a fully automated training process. As long as the environment allows to create a useful oracle, this method can be used to speed up learning as an additional module to any vision-based RL process.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations of the method are adequately addressed in the text of the manuscript.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for incorporating richer feedback than a reward signal from humans for training RL agents. Humans provide feedback by (i) indicating which actions are good/bad and (ii) highlighting salient areas in visual observations. All reviewers unanimously agree that the paper should be accepted. The experiments provided by the authors during the rebuttal phase further strengthen the paper. At the same time, I recommend the authors to improve the clarity of the paper, explanation of prior work (e.g., DQN-TAMER) and address other minor concerns raised by the reviewers in the camera-ready version.

If I were to simply go by the technical contributions of the paper, I would recommend it as a Poster. However, the line of research on providing human feedback beyond reward functions is worth highlighting to the research community. I therefore recommend the paper to be presented as a spotlight presentation. 


Decision: Accept (Spotlight)