Fast Adaptation via Human Diagnosis of Task Distribution Shift
Authors: Andi Peng
Mark K Ho
Aviv Netanyahu
Julie Shah
Pulkit Agrawal
Venue: Submitted to ICLR 2023 (2023)
Keywords: human-ai interaction
human-in-the-loop

Abstract: When agents fail in the world, it is important to understand why they failed. These errors could be due to underlying distribution shifts in the goals desired by the end user or to the environment layouts that impact the policy's actions. In the case of multi-task policies conditioned on goals, this problem manifests in difficulty in disambiguating between goal and policy failures: is the agent failing because it can't correctly infer what the desired goal is or because it doesn't know how to take actions toward achieving the goal? We hypothesize that successfully disentangling these two failures modes holds important implications for selecting a finetuning strategy. In this paper, we explore the feasibility of leveraging human feedback to diagnose what vs. how failures for efficient adaptation. We develop an end-to-end policy training framework that uses attention to produce a human-interpretable representation, a visual masked state, to communicate the agent's intermediate task representation. In experiments with human users in both discrete and continuous control domains, we show that our visual attention mask policy can aid participants in successfully inferring the agent's failure mode significantly better than actions alone. Leveraging this feedback, we show subsequent empirical performance gains during finetuning and discuss implications of using humans to diagnose parameter-level failures of distribution shift.

Under review as a conference paper at ICLR 2023
FAST ADAPTATION VIA HUMAN DIAGNOSIS OF TASK DISTRIBUTION SHIFT
Anonymous authors Paper under double-blind review
ABSTRACT
When agents fail in the world, is important to understand why they failed. These errors could be due to underlying distribution shifts in the goals desired by the end user or to the environment layouts that impact the policy's s actions. In the case of multi-task policies conditioned on goals, this problem manifests in difficulty in disambiguating between goal and policy failures: is the agent failing because it can't correctly infer what the desired goal is or because it doesn't know how to take actions toward achieving the goal? We hypothesize that successfully dis- entangling these two failures modes holds important implications for selecting a finetuning strategy. In this paper, we explore the feasibility of leveraging human feedback to diagnose what vs. how failures for efficient adaptation. We develop end-to-end policy training framework that uses attention to produce human- interpretable representation, visual masked state, to communicate the agent's 's in- termediate task representation. In experiments with human users in both discrete and continuous control domains, we show that our visual attention mask policy can aid participants in successfully inferring the agent's 's failure mode significantly better than actions alone. Leveraging this feedback, we show subsequent empiri- cal performance gains during finetuning and discuss implications of using humans to diagnose parameter-level failures of distribution shift.
INTRODUCTION
Humans are remarkably adept at asking for information relevant to learning a task (Ho & Griffiths, 2022]. This is in large part due to their ability to communicate feature-level failures of their internal state via communicative acts to teacher (e.g. expressing confusion, attention, understanding, etc.) (Argyle et al. 1973). Such failures can range from not understanding what the task is, e.g. being asked to go Walgreens when they don't 't know what Walgreens is, to not knowing how to accom- plish the task, e.g. being asked to go to Walgreens and not knowing which direction to walk in. In both cases, human learner would clarify why they are unable complete the task so that they can solicit feedback that is most useful for their downstream learning. This synergistic and tightly coupled interaction loop enables teacher to better estimate the learner's 's knowledge base give feedback that is best tailored to filling their knowledge gap (Rafferty et al. 2016).
Our sequential decision-male making agents face the same challenge when trying to adapt to new scenar- ios. When agents fall in the world due to distribution shifts between their training and test environ- ments (Levine et al. 2020), it would be helpful to understand why they fail that we can provide the right data to adapt the policy. The difficulty today when dealing with systems trained end-to-end that they are inherently incapable of expressing the cause of failure and exhibit behaviours that may be arbitrarily bad, rendering a human user left in the dark with respect to what type of feedback would be most useful for finetuning. Ergo, active learning strategies focus on generating state or action queries that would be maximally informative for the human to label (Akrour et al., 2012) Bobu et al.) 2022, Reddy et al., 2020); Biyık et al., 2011)), but such methods require an unscalable amount of human supervision to cover large task distribution (MacGlashan et al. 2017).
To address the challenge above, we propose a human-in -n-the-loop framework for training an agent end-to-end capable of explicitly communicating information useful for a human to infer the underly- ing cause of failure and provide targeted feedback for finetuming. In the training phase, we leverage attention to train policy capable of producing an intermediate task representation, masked state

---

Under review as a conference paper at ICLR 2023
B
"Go the blue abject
"The agent doean't recognize goal
+
Observed Behaviour
Agent's Task Representation
A
agent to go the goal""
+
Observed Behaviour
Agent's Representationn
Figure 1: A human user trying to diagnose the agent's failure mode. (A) When the human only sees agent behaviour, it is ambiguous why it's failing. (B) the human also has access to the agent's ' intermediate task representation for what it perceives be relevant to the task, they of error and thus parameters of the distribution shift. For example, if the agent is not attending to the target object, it is likely unfamiliar with the user's stated goal-i.e. a what error. (C) Alternatively, if the agent is attending to the target object but generates the wrong behaviour, this can indicate that is unfamiliar with how navigate the object's location-i.e. a how error.
infer the type
that only includes visual information relevant to solving the task. Our key insight is that while vi- sual attention has been studied in the context of visualizing features a deep learning model's black box predictions,
incorrect visual mask also help a human infer the underlying parameters of distribution shift in the event of policy's 's failure. This is done the feedback phase, when we use the masked state help human infer whether the agent is attending to the right features but acting incorrectly (a how error) versus attending to the wrong features (a what error). To close the loop, leverage the identified failure mode in the adaptation phase perform more efficient finetuning via targeted data augmentation of the shifted parameter.
We formalize the problem setting and describe the underlying assumptions. Next, we present our in- teractive learning framework for diagnosing and fixing parameter-level shifts using human feedback. Through human experiments, we verify our hypothesis that visual attention a more informative way for humans to understand agent failures compared to behaviour alone. Finally, we show that this feedback can empirically leveraged to improve policy adaptation via targeted data augmentation. We call the full interactive training protocol the visual attention mask policy (VAMP).
2 RELATED WORK
-onC-Condi Imitation Learning. The learning technique used in our paper is goal- conditioned imitation learning (IM), which seeks learn multi-task policy end-to-end by super- vised learning or "cloning" from expert trajectories (Abbeel & Ng 2004 Ng et al. 2000 Ding et al.) 2019). The learning from demonstrations framework means that we can optimize a policy without the need for a reward function (Pomerleau 1988), albeit we cannot generate new behaviours without feedback. Moreover, unlike standard IM IRL methods, goal-conditioned policies are capable of learning single policy to perform many tasks. Unfortunately, generating enough expert demonstra-
tions to cover large test distribution is difficult, (Ziebart et al. 2008 Finn et al. 2016).
Human-in-the-loop RL. Interactively querying humans for data to aid in downstream task learning belongs to a class of problems referred to as
RL (Abel et al. 2017 Zhang et al, 2019 Existing frameworks like TAMER (Knox & Stone. 2008 and COACH MMacGlashan al.
2017 use human feedback to train policies, but are restricted to binary or scalar labeled rewards. A different line of work seeks to leam tasks using human preferences, oftentimes asking them to com- pare or rank trajectory snippets (Christiano et al. 2017; Brown et al., (020). Yet another direction focuses how to perform active learning from human teachers, where the emphasis is gener-
2

---

Under review as a conference paper at ICLR 2023
Training
Feedback
Adaptation
"What error"
Observed Behaviour
Augment existing new goal
p
"How
Intermediate Task Representatiin
Novel demos
Augment existing goals
Finetuning
The framework overview. In the training phase, a policy learns to generate both actions as well as an intermediate task representation to help the human infer the failure mode in the feedback phase. We then leverage this information to perform targeted augmentation in the adaptation phase.
Figure 2:
ating actions or queries that are maximally informative for the human to label (Bobu et al. 2022 Chao al. 2010). The challenge with these approaches is that the feedback asked of the human is often overfit to specific failures or desired data points, and rarely scale well relative to human effort.
Human-Interpretablle RL Ensuring that deep learning agents are intelligible various concerns encompassing ethical, legal, safety, or usability viewpoints is a key focus of real-world deployments
(Garcia Fernández 2015). Methods range from training a policy generate post-hoc expla- nations (Glanois et al. (021), text-based descriptions of state predicates (Hayes & Shah (017)) and Jacobian saliency maps (Gryydanus et al. (018). However, leveraging interpretable sequential Recision-making systems to extract useful human feedback for adaptation has been under-explored.
3 PROBLEM FORMULATION
Our goal is to develop a framework to help human provide feedback about behaviour-imariin vs. dependent parameters of distribution shift. We formulate the problem as an imitation learn- ing framework. In goal-conditioned setting, goals are represented G. We consider envi- ronments represented by a Markov Decision Process (MDP) (Puterman 2014) defined by tuple M M= AS, P, R), where S is the state space, A the action space, P S A S + [0, 1] the tran- sition probability distribution, and
S G A - R the reward function. parameterized policy denoted Te SSG G A. In imitation learning (IM), rather than seeking to learn TT through in- teracting with the environment as in RL, we instead assume access to a set of expert demonstrations {(s), a si af....//, af) ) from which we "clone" To from (Pomerlcau 1988). This yields dataset n state-action-gon tuples (st, at, g²) for learning. While E in typical settings, need have way of mapping the goal into form that To can process. Motivated with the idea that natural language is a flexible, intuitive interface for humans to communicate, we specify y via natural language instruction, resulting in the following standard GCBC loss:
not be as long
LGCBC (0) = Eee ~ DDraa ( g gg),)))))))
(1)
where f is mean squared enror (MSE) for continuous actions and cross-entropy for discrete actions.
3.1 OUT-OF-DISTRIBUTION FAILURES
Like all supervised learning methods, GCBC suffers from distribution shift (Ross et al. 201T): i.c. no could behave arbitrarily if faced with input at test that it did not at training. Consider the
3

---

Under review as a conference paper at ICLR 2023
simplest case where the training set consists of states concatenated with goals, resulting in Derain: [s.9] E S x g. We can similarly define Diest: [s', E S' x g'. Because we cannot guarantee that either the environment layout or the desired goal will remain unchanged at deployment, i.. [s',g'] & S x G. we face two possible sources of distribution shift: 1) the policy can fail to infer goals that are O0D (what failure) or 2) the policy can fail to produce correct actions for a goal location that is OOD (how failure). Our hypothesis is that in both cases, the erroneous actions generated by the policy are uninformative for human classify the source of failure.
3.2 ADAPTATION FOR SHHAVIOUR-INVARIAN VS. DEPENDENT TASKS
The adaptation challenge we face is how to most efficiently generate data for finctuning policy to perform well on task distribution Prest given knowledge of shifted parameters from Presin In the most naive scenario, methods that assume knowledge of 'rest uniformly sample tasks for querying human demonstrations for. In the case of goal-condi policies for visual navigation, this would include new goals that are both behaviour-invariant, e-g. object colors or types, as well as those that are behaviour-dependent, e.g. locations. However, such a finetuning strategy is practically inscalable when attempting to cover a large Dreat. where all tasks irrespective of which parameter has shifted require novel demonstrations (states and actions) from a human. Our insight is that rather than assuming demonstrations, i.c. new behaviours, are always required, we can disentangle behaviour-dependent from sachaviour-ivvariie tasks and only query for new data when needed.
Assumptions. First, while Dirain # Diest, we assume they are parameterized via a generative model capable of modifying visual parameters, i.e. the state space. These may include features such as object location and color and can be reasonably modified with access to a simulator or scene generator. Although we assume knowledge of the full parameter space, we do not know which parameters are shifting from Purain to Drest and therefore, which tasks require new actions. Thus, we assume the ability to query for this along with novel demonstrations from human. Lastly, in order to "surgically" leverage existing data, we assume access to the original training demonstrations.
Given knowledge of the shifted parameter, we can perform targeted data augmentation. Consider the two failure types in Figure 2. feedback is given that the shifted parameter is gehaviour-invariane e.g. new goal object (A), we can augment our existing demonstrations with the desired goal via our generative model (if we know how to navigate to a blue key in the room we can also navigate to red key in the same room). If instead feedback is given that the shifted parameter is behaviour- dependent, c.g. a new goal location (B), we can query for new actions from our human but then augment existing
blue key in a new room, we now also know how to navigate to red key in that room). To deploy such "surgical" data augmentation techniques, we must develop a framework capable of extracting the shifted parameter from a human in reliable and non-cumbersome manner. We next detail a framework for doing so.
anhaviour-invarian goals (if we are shown how navigate
4 THE VISUAL ATTENTION MASK POLICY LEARNING FRAMEWORK
We propose an interactive learning framework that leverages human feedback to disentangle behaviour-invariant from behaviour-dependent tasks. To do so, we assume conditionally indepen- dent parameters of our training distribution Dursin (tasks that are learned by our policy at training time) and test distribution Drest (tasks that are desired at test time). Parameters range from goal object and color as well as location. Our framework is comprised of three phases: training, feed- back and adaptation. In the training phase, we train an end-to-end policy capable of generating an intermediate task representation, a masked state, an additional inter-bnterpretable output. In the feedback phase, we then use the visual mask to help a human diagnose the shifted parameter. Lastly, leverage that feedback in the adaptation phase perform targeted data augmentation.
4.1 TRAINING PHASE
As described in Section 3 the model for training policies capable of producing goal-conditioned attention masks are conditioned on language instruction g and the current state s. Because our policy combines the standard GCBC loss in conjunction with an intermediate mask loss acting regularizer, we refer to our model as visual attention mask policy (VAMP). As a policy network, the following two modules trained end-to-end: 1) attention module: processes the output of pre-
4

---

Under review as a conference paper at ICLR 2023
to the green goal"
S
Language Model
2D Conv
S
2D Conv
a
MLP
Attention Module
Policy Module
The architecture of our VAMP model. The attention module processes the goal specifi- cation and image state The resulting output is processed by the policy module along with the original state to create masked state 8, which is used for action prediction.
Figure 3:
trained language model that turns natural language instruction into an embedding, concatenated with the output of 2D ConvNet which takes state and produces ID vector; 2) policy module: a 2D ConvNet that takes the output of the attention module and combines it with state s to produce masked state 6, which is used for action prediction. Note, the of visual attention to focus task-relevant features is well studied in the machine learning (Xu et al. 2015 Hayes & Shah 2017)
and cognitive science (Lindsay 2020
Ho & Griffiths 2022) literatures.
Attention module: Our attention module is comprised of three components: 1) a pre-trained gen-
eral purpose language model called SentenceTransformee Ml--MiniLMM--6-22 (Reimers & Gurevych (2019)), ) which we use to process a goal specification g into an embedding, a vector of size 384; 2) a layerr 2D ConvNet with flattened last layer that we process image state s; and 3) 2-layer MLP that we process the concatenated output into a mask.
Policy module: Our policy module processes the mask generated by the attention module with the original state to create a masked state å via 4-layer 2D ConvNet to produce action The objective is:
CVAMP ((s)) Dernin 1e ((TT g'), [11]
As shown by the second term, our attention mask serves as a regularizer reduce the information flow from the full state to the minimal state that is required for action prediction, i.c. we wish find the minimal representation of that is capable of correct action prediction. We additionally introduce weight 3 to control the regularizer's contribution to the total loss so that we can scale the intensity of the masking. Full architecture and training details can found appendix.
4.2 FEEDBACK PHASE
In the feedback phase, we now provide both the policy's output (actions) as well as its intermediate task representation to human user for help in diagnosing specific policy failures. We generate distribution shifts of two types: I) testing on goals outside of the training distribution (what or behaviour-invariant failures) and 2) testing on environment layouts with object locations, i.e. ac- tions, outside of the training distribution (how or behaviour-des dependden failures). For each error type, create trajectories by sampling tasks from the test distribution and generating their correspond- ing intermediate task respresentation to show our human participants. We then query for feedback regarding which failure type is exhibited.
5

---

Under review as a conference paper at ICLR 2023
4.3 ADAPTATION PHASE
As shown in Figure 2, we now leverage the identified failure mode to perform targeted data augmen- tation of training demonstrations for our shifted parameter. For what or behaviour-invare require only access to the generative model for modifying state space parameters, e.g. goal ob- jects or colors. For how or behaviour-dependee failures, we require both the generative model as well as the ability query for novel demonstrations, e-g. navigating
failures,
goal location.
Finetuning a what failure. We augment Dtrain by replacing the original goal specification as well the shifted parameters in the state (for example, if we wish navigate to new keys, augment demonstrations by replacing all existing goal objects with keys). We then finetune our policy using this dataset. This strategy requires novel demonstrations since we are re-using behaviour-invarian demonstrations (i.e. modified states with existing actions).
Finetuning how failure. Because we require new action sequences to navigate location, we are required to collect novel demonstrations, i.c. states and behaviours. While these demonstrations would typically come from the human user, we sample from a simulated expert for ease of exposition. We first sample g ~ 'uest and collect expert demonstrations Duovel for reaching g. We then leverage knowledge of the behaviour-invaree Dirain to create an augmented dataset of these demonstrations for finetuning.
new goal
parameters (e.g. goal object or color)
EXPERIMENTS
In this section, we detail set of empirical results to answer the following questions: Do in- termediate task representations help humans diagnose the underlying parameter shift in the event of a policy failure? 2) Does correctly leveraging this information for data augmentation improve downstream policy performance? 3) Do these gains hold when evaluated on real human subjects?
To answer these questions, we evaluate our proposed framework on two domains: a ingle-goal navigation task and a multi-room compositional task. For each domain, we first detail the task gen- eration process for the training and test distributions. We then conduct experiments with simulated human oracle explore the maximum performance gain that can be achieved by optimal fine- tuning strategy with a fixed data labeling budget. Lastly, we verify these results with real human feedback collected from user studies diagnosing distribution shifts.
5.1 MULTI-OBJECT NAVIGATION
We create an image-based environment where an agent is tasked with navigating to goal object of one color while ignoring dis- tractor object of a different color. States are fully-observable RGB images of dimension 36x36x3 and the action space is continuous and represents the (x,y) distance that the agent can move 2D space. The environment was created test simple object naviga- tion domain while preserving a continuous action space of higher complexity relative discrete gridworld tasks.
Goal Object
Distractor Object
Top locations
Bottom locations
The environment contains 4 colors (red, green, blue, yellow) and 4 starting locations (grid corners). A task distribution D specifies the generative parameters for sampling the goal location/color and can contain any subset of the environment parameters above. To gener- ate tasks, we first place a goal object of color and location uniformly sampled from p( (colors)) })) and p( (locations))). We then place a distractor object and the agent at two randomly sampled remaining locations. Lastly, we assign the distractor object randomly sam- pled unique color (the agent is always white). The task specification defined via a language instruction "go to the <qoal color> object".
Agent
Figure 5: An example multi- object navigation task for "go to the green object".

---

Under review as a conference paper at ICLR 2023
(a) "Go to the green object"
(b) "Go to the red object"
(c) "Go to the green object"
(d) Masked state for (a)
(c) Masked state for (b)
(f) Masked state for (c)
Figure Examples of the multi-object navigation task. An agent (white) trained to navigate to green objects in all locations, when asked to navigate to green object, will produce successful actions (a) as well as a task representation attending to the correct goal (d). If the agent is asked to instead navigate to red object (b), it will exhibit what failure, with the mask attending to the wrong object (e). This contrast to a how failure, where an agent trained to navigate to objects on the top will fail to navigate to objects the bottom (c), even though the mask attended to the correct goal (f).
For each Dtrain, we generate 100 tasks. For each task, we generate an expert demonstration of length 20 by taking continuous actions from the agent's starting location the goal object, yielding 100 demonstrations for our initial policy training. We train and test on 2 possible distribution shifts.
5.2 MULTI-ROOM COMPOSITIONAL TASK
We also design task with multi-room multi-task compositional structure to explore how our method scales with task composition- ality and Long-horizon goals. We adapt the DoorKey environment from Minigrid Seevalier-Boivved et al. (2018)) and create an envi- ronment composed of three sub-tasks (pick up key, the key unlock door, then navigate through the door to goal). The state space fully-observable and consists of RGB images of dimension 36x36x3. The action space is size 6 and allows for cardinal move- ments, picking up/dropping key, and using a key to open a door.
Top locations
Agent
Goal
Lava
Door
Obsttcces
There are possible colors (red, green, blue, yellow), and 10 key, door, and goal locations each. contains parameters for sampling each sub-task color and location. We generate tasks by first uni- formly sampling key color/location, door color/location, and goal color/location from D. We next place three lava objects immov- able obstacles at a randomly sampled location for each sub-task. The agent always begins the top left corner. The task specification defined via a language instruction "go to the <key color> key, <door color> door, <goal color> goal".
Key
Figure 6: An example multi- room compositional task for "go to the yellow key, green door, blue goal".
To explore how task compositionality impacts finetuning efficiency, create distributions for each sub-task by varying the parameters for one object while holding the
7

---

Under review as a conference paper at ICLR 2023
others constant (for example, distribution with randomly sampled red keys in all locations would hold the door and goal colors/locations constant). For what shifts, we train on red colors for each object and test green, blue, and yellow colors in all locations. For how shifts, we train objects in the top half of the room and test on the bottom half. We generate unique shifts while maintaining the compositional structure of the task. We sample 100 tasks from each Drrain and for each generate an expert demonstration of the agent successfully completing all 3 sub-tasks.
5.3 HUMAN EXPERIMENTS
For the finetuning strategies which require human feedback, we conduct user studies at [Anonymous Institution]. We recruit 12 subjects for each domain (71 male, age 18-31). 88% of participants attested to having technical background, although only 17% have worked with machine learning.
The user study is comprised of two phases: a familiarization and feedback phase. In the famil- iarization phase, we introduce the user to the task context, environment, and an example of each failure type. In the feedback phase, we first show the user the agent's behaviour on the test set (5 trajectories for each shift in the multi-object navigation task and 2 trajectories for each shift in the multi-room compositional task, randomized). We then ask for feedback on whether they believed the failure due to what, how, unclear failure. Lastly, trajectories with their corresponding attention mask as additional information, and request a second round of feedback. Altogether, per domain. For each participant, a finetuning strategy was selected based on the average predicted accuracy of each failure type (e.g., participant that predicted 4 of how failures incorrectly as a what failure would deploy the wrong strategy). Unclear responses were treated as incorrect.
then show the users the same
received 120 data points for each control and experimental group
Distribution Shift
Correct Feedback (%)
Domain
Dirain
D Duest
Failure
Naive
Ours
Multi-Obj
RG goals, all loc
BY goals, all loc all goals, top loc
what
6.7
78.3
all goals, bottom loc
how
18.3
95.0
Multi-Room
keys, all loc all keys, top loc
GBY keys, all loc
what
4.2
83.3
all keys, bottom loc
how
8.3
75.0
door, all loc all doors, top loc goals, all loc all goals, top loc
GBY door, all loc
what
16.7
70.8
all doors, bottom loc
how
4.2
54.2
GBY goal. all loc
what
50.0
70.8
all goals, bottom loc
how
0.0
62.5
Table Rate of correct human responses for each distribution shift. We see that our method enables human users to more accurately diagnose the underlying distribution failure type.
5.4
EVALUATION
For each domain, we assess performance of the final policy on 20 sampled tasks from Drest after finetuning with the selected strategy. The metric of evaluation that we seek to minimize is user effort, which we measure by the number of novel demonstrations that are required for any particular strat- egy. While we generate these demonstrations here, this motivated by the idea that in a deployment scenario, this cost is largely incurred by the human teacher who must provide novel demonstrations, and not from data augmentation or finetuning performed in the factory. We make 5 comparisons: 1) no finetuning, 2) finetuning with no human feedback, 3) finetuning with naive human feedback, 4) finetuning with informed human feedback (our method), and 5) oracle finetuning.
No finetuning (None): No finetuning of the policy is permitted.
Finetuning with no human feedback (No Human): Our second baseline describes the scenario where we receive no human feedback and therefore receive information about the desired test distribution. In this case, we allow the generation of 20 novel tasks by randomly sampling from all possible environment parameters, creating their corresponding demonstrations, and finetuning. Finetuning with naive human feedback (Naive Human): Our third comparison the case where receive human feedback without the intermediate task representations from method. This
8

---

Under review as a conference paper at ICLR 2023
Multi-Object Navigation (what failures)
Multi-Object Navigation (how failures)
20
Naive Human
35
30
Distance from goal
Distance from goal
Oracle
15
25
20
10
15
10
10
100
20
40
100
Novel demonstrations generated (a) what failures
Novel demonstrations generated
(b) how failures
Figure 7: Final policy performance as a function of user effort (measured by number of novel demonstrations generated) on the multi-object navigation task. We see that our method requires less user effort to achieve higher policy performance relative to naive human feedback.
represents control group, where human participants are shown agent trajectories only. For every human participant, we finetune a policy according to their selected (potentially erroneous) strategy and average policy performance across participants. If how error selected, we allow the generation of 20 novel demonstrations prior to augmentation (0 are required for what strategy). Finetuning with informed human feedback (Ours): Our fourth comparison is method, where human feedback is informed by both agent trajectories as well as intermediate task representations. We finetune similarly to Naive Human above, but leverage a more accurate augmentation strategy. Finetuning with perfect feedback (Oracle): For completion, we also include comparison to the maximum adaptation gain possible we selected the strategy that perfectly predicted the correct failure mode. The same fixed budget for maximum 20 novel demonstrations are applied here.
Distribution Shift
Distance from Goal (std)
Domain
Deyoin
Drest
Failure
None
No H
Naive H
Ours
Oracle
Multi-Obj
goals, all
goals,
what
(4.6)
(4.0)
8.1 (5.4)
(0.2)
(0.1)
all goals. bottom loc
all goals, loc
how
21.7 (9.1)
16.1 (6.2)
9.3
2.2 (3.2)
0.4 (0.1)
Multi-Rooo Room
keys, all loc
GBY keys, all loe what
(0.4
6.2 (0.7)
(2.7
0.3 (0.6)
0.4 (0.8)
all keys, top loc
all keys, bottom loc how
(2.3
(1.3)
(5.4)
3.6 (2.2)
1.7 (1.4)
doors, all loc all doors, top loc
GBY Y doors, all loc
what
8.0(1.9)
5.2
(..8
2.8 (1.9)
1.0 (0.3)
all doors, bottom loc GBY goals. all loc
how
(.112.1)
(.6(1.1)
6.1
4.0 (0.7)
0.7 (0.9)
goals,
what
6.2(19)
(1.)
8.9.
(0.4)
all goals, loc
all gouls, bottom loc
how
8.19
(.77
(2.3)
0.3 (0.7)
0.3 (0.6)
Table 2: Final policy performance for all tested distribution shifts in both domains. Our method outperforms all baselines, and rivals perfect feedback from an oracle some cases.
Table shows that human feedback using our framework results in more accurate distribution shift diagnosis vis-a-vis those without. Table 2 shows the results of downstream policy performance on each distribution shift after finetuning with a maximum budget of 20 novel demonstrations per strategy. This empirically demonstrates how more accurate feedback results in improved policy performance given a fixed quota of user effort. Note, naive human is sometimes outperformed by no human all. highlighting that an incorrect data augmentation strategy can sometimes be worse than random sampling demonstrations from the test distribution.
DISCUSSION AND CONCLUSION
Summary. We leveraged existing techniques to create a human- human-in-the-loon in-the-loop framework for diagnos- ing and fixing distribution shift in end-to-end sequential systems. We showed that framework

---

Under review as a conference paper at ICLR 2023
effectively utilizes insights from cognitive science produce intermediate task representations pable of aiding humans diagnosing underlying distribution shifts. We also demonstrated the empirical performance benefit of our method in reducing human effort for downstream adaptation.
Limitations. There are parameters that are difficult to practically disentangle through visual atten- tion alone such as object occlusion, shape and texture, and partially observed scenes. Moreover, we assume access to a generative model capable of manipulating those parameters, a challenging task without high-quality scene representation. We remain optimistic that advances in representation learning for feature disentanglement can be easily incorporated into our framework.
Future Impact. As human-in-the-looy systems are increasingly deployed, we must find more effi cient ways of leveraging feedback for improving learning systems if we wish to practically adapt to user preferences. Moreover, we have systems operating around and with humans, we must create more transparent, reliable methods of communicating the underlying parameters of why they fail.
ETHICS STATEMENT
Since our paper relies heavily on human experiments and data for evaluation, we attest the fol- lowing details related to human subject evaluation and data privacy. An institutional IRB was filed prior to subject recruimment. All human subjects were asked voluntarily participate in the exper- iment of their own free will. While we gathered basic demographic information (age, gender, and technical background), participant information was anonymized for analysis and referred to only by ID number. We collected no further information.
REFERENCES
Pieter Abbeel and Andrew Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference Machine learning, pp. 1, 2004.
David Abel, John Salvatier, Andreas Stuhlmüller, and Owain Evans. Agent-agnostic human-in--ie loop reinforcement learning. arXiv preprint 1rXiv::701 04009, 2017.
Riad Akrour, Mare Schoenauer, and Michèle Sebag. April: Active preference learning -baee rein- forcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pp. 116-131. Springer, 2012.
Michael Argyle, Roger Ingham, Florisse Alkema, and Margaret McCallin. The different functions of gaze. 1973.
Erdem Bryik, Kenneth Wang, Nima Anari, and Dorsa Sadigh. Batch active learning using determi- nantal point processes. arXiv preprint arXiv: 190.0099, 2019.
Andreca Bobu, Chris Paxton, Wei Yang. Balakumar Sundaralingam, Yu-Wei Chao. Maya Cakmak, and Dieter Fox. Learning perceptual concepts by bootstrapping from human queries. IEEE Robotics and Automation Letters, ((4):11260-11 1267, 2022.
Daniel S Brown, Wonjoon Goo, and Scott Nickum. Better-than- wwrettthaa--mmmtttttttoo imitation learning via automatically-rankke demonstrations. In Conference robot learning, pp. 330-359, PMLR, 2020.
Crystal Chao, Maya Cakmak, and Andrea L Thomaz. Transparent active learning for robots. In 2010 5th ACM/IEEE International Conference on Human-Robol Interaction (IIRI), pp. 317-324. IEEE, 2010.
Maxime Chevalier-Boisvert, Lucas Willems, and Suman Pal. Minimalistic gridworld environment for gymnasium. https: hitttp://githubss cww/ca//aaaaeeeee http:///tt.....cooooooooooooooo/
2018
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing sys- fems, 30, 2017.
10

---

Under review as a conference paper at ICLR 2023
Yiming Ding, Carlos Florensa, Pieter Abbeel, and Mariano Phielipp. Goal-conditioned imitation learning. Advances in neural information processing systems. 32, 2019.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In Proceedings of the 33rd International Conference on International Conference Machine Learning Volume 48, ICML' 16, pp. 49-58. JMLR.org. 2016.
Javier Garcia and Fernando Fernández. A comprehensive survey on safe reinforcement learning. Journal of Machine Learning Research, 16(1):1437-144)) 2015.
Claire Glanois, Paul Weng, Matthieu Zimmer, Dong Li, Tianpei Yang, Jianye Hao, and Wulong Liu. A survey on interpretable reinforcement learning. arXiv preprint arXiv:2112.13112, 2021.
Samuel Greydanus, Anurag Koul, Jonathan Dodge, and Alan Fern. Visualizing and understanding atari agents. International conference on machine learning, pp. 1792-1801. PMLR, 2018.
Bradley Hayes and Julie A Shah. Improving robot controller transparency through autonomous pol- icy explanation. 2017 12th ACM/IEEE International Conference on Human-Robot Interaction (HRI, pp. 303-312. IEEE, 2017.
Mark K Ho and Thomas I. Griffiths. Cognitive science source of forward and inverse models of human decisions for robotics and control. Annual Review of Control, Robotics, and Autonomous Systems, 5:33-53, 2022.
W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference development and learning, pp. 292-297, IEEE, 2008.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. In arXiv preprint arXiv:2005.06443, 2020.
Grace Lindsay. Attention in psychology, neuroscience, and machine learning. Frontiers in com-
putational neuroscience, 14:29, 2020.
James MacGlashan, Mark K Ho, Robert Loftin, Bei Peng, Guan Wang, David L Roberts, Matthew Taylor, and Michael L Littman. Interactive learning from dolene-de human feedback. In International Conference on Machine Learning, pp. 2285-2294 PMLR, 2017.
Andrew Y Ng. Stuart Russell, et al. Algorithms for inverse reinforcement learning. In Ieml, vol- ume 1, pp. 2, 2000.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1, 1988.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John
Wiley & Sons, 2014.
Anna N Rafferty, Emma Brunskill, Thomas L Griffiths, and Patrick Shafto. Faster teaching pomdp planning. Cognitive science, 1000::199-33333 2016.
Siddharth Reddy, Anca Dragan, Sergey Levine, Shane Legg, and Jan Leike. Learning human objec- tives by evaluating hypothetical behavior. In International Conference on Machine Learning, pp- 8029. PMLR, 2020.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, 11 2019. URL ptctt:///rxivg
abs/ 1908. 1.0.810055 10084
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. reduction of imitation learning and struc- tured prediction no-regret online learning. In Proceedings of the fourteenth international ference on artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.

---

Under review as a conference paper at ICLR 2023
Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference machine learning, pp. 2048-20517 PMLR, 2015.
Ruohan Zhang, Faraz Torabi, Lin Guan, Dana H Ballard, and Peter Stone. Leveraging human guidance for deep reinforcement learning tasks. arXiv preprint MwXiv:/999.09900 2019.
Brian D Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAI, volume 8, pp. 1433-1433 1438. Chicago, II, USA, 2008.
A
APPENDIX
A.I
TRAINING AND ARCHITECTURE DETAILS
Architecture.
The language model processes a goal sentence in natural language into embedding of size 384, which we additionally process through a linear MI.P of output size 100.
Our Conv2D blocks process both the image state as well masked state and consist of 3 stacked Conv2D layers of output channel sizes 32, 64, 32, kernel sizes 8, 4, and strides 4, 2, and 1. Each output layer is processed by BatchNorm2D as well as ReLU activation. After the last Conv2D layer, we flatten the output process through BatchNorm ID, then through a linear layer. The Conv2D block for processing the initial state s generates an output of size 100 while the Conv2D block for processing masked state & directly generates an output of the action size.
Our MLP block consists of two-layer stacked MLP of input size 200, hidden size 100, and output size 1296, which we then reshape into a channel image of dimension 36x36.
Training. We use Adam for optimization with a learning rate of 0.001. All training was performed in PyTorch with SGD. For policy training of tasks from Dtrain we train for 1200 epochs using batch size of 50. For finetuning, we train the policy for an existing 1200 epochs. We performed a parameter sweep over the regularizer term 3 and used 0.005 for our experiments.
A.2 MULTI-OBJECT NAVIGATION
As the multi-object navigation task is one that is created specifically for our project, we detail the full generation details below.
Formally, the environment is characterized by a tuple (2) where A is the agent and Ω is the set of objects in the room. For each task distribution, we define finite set of Colors {red, green, blue, yellow} and two-dimensional Locations := -5), (-5, 5), (5, 55, 5) At reset, A is placed at location A^ U (LLocations). Two objects, w1 and w2, are generated, each with a color en and location An. with C1 ~ U(Colors), A ~ U[{Locations} \ {Aa H and Ag U[(Locations) \ (As, A131- We define an environment configuration by random variable X = (T Tezr 222), where a specific configuration is characterized by the vector (Ax, C1, A1,02, 22).
We sample goal Wg U(wI,w2) and assign the remaining object as distractor was A language
generation function : w/ - L can be thought of as producing the goal specification i.e. language instruction that human would provide at deployment. An image generation function ø Ω x X + renders vector z into an image state s, where the agent is a white circle, the goal a diamond of its assigned color, and the distractor diamond of its assigned color.
A task distribution D contains the parameters of the generation function (colors and locations of the agent, goal, and distractor). For example, one such distribution could be {red, green} and (-,,))) A specific task DD generates an environment as specified above, yielding a tuple (Whg)) (())
(I., so) and describes specific image and its associated goal specification. We sample 100 tasks for each training distribution Derain and generate each an expert trajectory d: (5o, ao, S20. a20) by taking continuous actions toward the goal object, yielding 100 trajectories which we use to train our GCBC policy.
12


Reviews:
[{"review_id": "ybWruBdccE", "reviewer": "Reviewer_sTo9", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles an interesting problem. The basic idea is to use \"privileged\" feedback by the humans in the loop to improve adaptation of RL agents. The paper imposes a \"constraint\" on the agent's representations which helps the human user  to diagnose the agent’s failure mode. The paper tackles two kinds of \"failure model\": how error and what error. Based on the feedback, the human \"parametrically\" augments data from tasks (and thus the paper assumes access to the ground truth generative factors of variation) which may facilate adaptation and generalisation. \n\nCOMMENTS AFTER REBUTTAL:\n\nThe reviewer has read the reviews by other reviewer, as well the author's feedback. I retain my rating/recommendation.", "summary_of_the_review": "The paper tackles an interesting problem: being able to diagnose why the agents are facing and based on the \"surgery\" being able to construct new tasks such that the resulting agents can adapt/transfer efficiently. ", "strength_and_weaknesses": "Strength:\n\n- The paper is well written.\n- The paper tackles an interesting problem i.e., how we can leverage the agent's internal representations to \"do\" interventions at the task level such that after training the agent's on the modified task distribution, the agent adapts efficiently. \n\nWeakness:\n\n- The main weakness is it may not be always feasible to construct \"errors\" based on internal representations on more complicated tasks (still the idea of parametrically augmenting the task distribution is interesting).\n- The paper assumes access to the generative model capable of manipulating the factors of variation (and be able to construct new tasks). This limitation is already discussed in the paper, and the reviewer is hopeful that the future work should be able to leverage advances in generative modelling. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W2owD9SdqT6", "reviewer": "Reviewer_vMKq", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work considers the problem of diagnosing why a policy fails when presented with an out-of-distribution task. Specifically, this work creates a framework to determine if the policy fails for a \"how\" reason (e.g., the policy does not know how to accomplish the provided goal) or a \"what\" reason (e.g., the policy does not know what the goal is at all). Then, this information is used to fine-tune the policy on data augmentations specifically designed to combat the determined failure mode.", "summary_of_the_review": "The idea of diagnosing and fixing policy behavior on out-of-distribution tasks is interesting and practically useful. However, the proposed method has several significant drawbacks, including its inability to generalize to more complex tasks of practical value. Hence, I cannot currently recommend acceptance.", "strength_and_weaknesses": "## Strengths\nThe main strength of this work is that it studies and interesting and practical problem that is relevant to the RL community. It offers the new idea of diagnosing the failure mode of the distribution shift to update the policy via data augmentation.\n\n## Weaknesses\n\nThe main weakness of this work is that it's unclear how general the method is and how well it scales to tasks beyond grid worlds. Specifically, while it seems quite possible that including attention masks as an error analysis tool is widely applicable, failure modes may be less interpretable and fall on axes beyond \"what\" and \"how\" when moving to 3d tasks with many steps. For these sorts of tasks, it may further be unclear what steps to take even after observing the attention mask. Additionally, the current setup requires fairly strong and restrictive assumptions. Specifically, the experiments assume the ability to generate new tasks of the same distribution shift type, which is unlikely to appear in practice. Further discussion of these aspects is warranted.\n\nThe experimental setup could be improved:\n- The results in Table 1 seem somewhat suspicious. The \"Naive H\" feedback appears to be almost always incorrect. However, if you choose the opposite of what \"Naive H\" predicts, then this appears to be more accurate than the mask-augmented feedback much of the time. What's going on here?\n- It would be helpful to tease apart the effects of including more demonstrations vs. including demonstrations of the correct type. For example, does increasing the number of demonstrations with oracle feedback further improve performance? What if you also added some demonstrations of the wrong type?\n- Specifically, how are the demonstrations generated to target \"what\" and \"how\" types? I assume that the \"No human\" baseline uniform task generation is different than just adding half \"what\" and half \"how\" demonstrations. If so, it would be useful to include that as a baseline, which would probably perform better and requires no human intervention?\n\nThis work lacks clarity and precision in many areas. Below are a list of examples:\n- The objective in Equation (2) is somewhat unclear because the parameters are not included in the objective. It's written under the policy module paragraph, which suggests that it's only an objective for the policy module, but the regularization term on $\\hat{s}$ is independent of the policy module. Additionally, the regularization term needs to be inside the expectation, otherwise it's incorrect? It's also unclear how this \"reduce(s) the information flow from the full state s to the minimal state.\" What information is flowing where?\n- How are \"unclear\" responses factored into the accuracy of Table 1? Are they disregarded entirely? If so, what's the point of including such a response?\n- Why are augmentations not included for the \"what\" distribution shifts? e.g., the text states: “If a how error was selected, we allow the generation of 20 novel demonstrations prior to augmentation (0 are required for a what strategy).”\n- It's unclear what it means that the training and test splits are related by a generative model\n- The problem formulation states that it encapsulates both discrete and continuous control, but the objective (1) is written only for continuous control. Specifically, the inside of the expectation should probably just be $\\log \\pi(a \\mid s, g)$. Formally, the goal should be augmented into the state, or otherwise the reward function is undefined on the goal.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "STf6q9LeDfA", "reviewer": "Reviewer_hcEg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of multi-task policies failing in the presence of distribution shifts. Authors disambiguate between goal and policy failures as the agent failing because it can’t correctly infer the desired goal or because it doesn’t know how to take an action toward achieving the goal. This work incorporates human feedback to diagnose the what versus how failure for adapting policies under distribution shift. \n\nThis work proposes a framework to incorporate human feedback by leveraging agents’ intermediate task representation.  Experiments with human users across discrete and continuous control tasks show that (i) intermediate task representations help human users in identifying failure mode; and (ii) subsequent empirical performance gains can be obtained in those tasks by leveraging human feedback. ", "summary_of_the_review": "Overall, while the paper tackles an important problem, the simple baseline of using all the augmentations is missing in the comparison. Moreover, using an attention mask can be unreliable in general and authors should refer to the added references above. \n\nSince the proposed framework is also not general (at least right now it is not clear how to instantiate it under different tasks than the two considered in the work), I am leaning towards rejection. I would be happy to change my score if I share any misunderstanding that authors could clarify during rebuttals. ", "strength_and_weaknesses": "**Strengths** \n- The paper considers a very important problem of incorporating human feedback to improve machine learning systems, in particular, to improve the performance of imitation learning policies.\n- Experimental results are interesting when compared with a naive baseline. However, the paper misses a crucial baseline (see 2nd bullet in weaknesses).  \n\n**Weaknesses** \n- The paper is a bit hard to parse. Mainly the organization in Sections 4 and 5 is a bit confusing. Please refer to the detailed comments below. \n- In both the settings described, since authors are mainly choosing from two kinds of augmentations (depending on what and how failure), what if we could train a model with both the augmentations in the first place (without asking for human feedback)? Did authors try this in their experiments? \n- Do authors use attention to get feedback from humans? While in many tasks attention masks do seem to correlate with models' final output, in very general cases, using attention to get an explanation for the model prediction can be misleading. Refer to [1,2,3]. I encourage authors to add a discussion and contrast their use case with studies in these references. \n- It is unclear how the proposed framework may generalize beyond the two tasks considered in the paper where the disentanglement between how and what is not as easy. A more careful discussion of what kinds of problems the current framework can tackle will strengthen the paper. \n\n\n[1] Pruthi et al. Learning to Deceive with Attention-Based Explanations. ACL 2020. \n\n[2] Jain and Wallace. Attention is not Explanation. NAACLE 2019. \n\n[3] Wiegreffe and Pinter. Attention is not not Explanation. EMNLP 2019. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "2Bdk1pKPm_", "reviewer": "Reviewer_i8yJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to leverage visual attention as a diagnosis tool for reflecting the reason for error during models' task execution. The authors leverage a 2D language navigation task as the test bed and tested mainly using goal-conditioned imitation learning methods. They visualize attention on the 2D map to show whether the model has a wrong understanding of the goal or can not reach the goal. Humans can diagnose the failure of models based on this visualization to augment learning data sources to facilitate task execution. The authors showed that their visualization can help humans diagnose problems of models and further improve the navigation task in cases where train and test distributions shift.", "summary_of_the_review": "This paper proposes to include human diagnosis in 2D navigation by adding additional visualization of goal localizations in the policy module for adaptation to task distribution shifts. However, the benefits brought by the method rely heavily on its design and it's unclear how to generalize the method proposed to more complex tasks. Therefore, I don't think it meets the bar for acceptance for now. I suggest the authors should justify that they are solving a challenging task and the proposed model could be leveraged in more complex scenarios in the following revisions.", "strength_and_weaknesses": "[+] This paper studies the important interactive learning problem and propose a pipeline to include human in the loop for improving model performance under task distributional shift by adding a visualizable intermediate representation that could be used for diagnosis. \n\n[+] In the 2D navigation and goal-conditioned reinforcement learning setting, the proposed baseline is simple and effective where the attention visualization can provide clear clues on what type of errors the model made under a shifted test distribution.\n\n[-] One major concern of this paper lies in the significance and generalizability of the proposed method. Although the 2D navigation task featured in this paper is a good and clear example to illustrate the idea, it is not clear how to add a visualizable or human-interpretable intermediate representation for other general tasks. This makes the current design's contribution somewhat limited as the fast adaptation relies heavily on human diagnosis and efficiently augmenting learning data. Therefore, I do not see its potential as a general plug-and-play module for solving other tasks as well.\n\n[-] As the authors stated, the use of visual attention to focus on task-relevant features is well-studied and widely adopted. However, it also relies on an interpretable canvas for visualization (image, video, or text) and gradually lose its explainability when the model size or depth increases. This adds difficulty to the current design for generating diagnosing visualizations in complex tasks with more computation-heavy (e.g., deep transformers) models.\n\n[-] The current evaluation results are mainly ablative studies of the VAMP model without proper comparison with baselines. Following the previous two points, the authors should consider comparing state-of-the-art models with the proposed VAMP to justify the difficulty and challenge of the proposed 2D navigation problem since if such models can already perform well, the point of using this experimental setting should be questioned.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject