Ask4Help: Learning to Leverage an Expert for Embodied Tasks
Authors: Kunal Pratap Singh
Luca Weihs
Alvaro Herrasti
Jonghyun Choi
Aniruddha Kembhavi
Roozbeh Mottaghi
Venue: NeurIPS 2022 Accept (2022)
Keywords: Expert-Agent Interaction
Embodied AI
Expert-in-the-loop

Abstract: Embodied AI agents continue to become more capable every year with the advent of new models, environments, and benchmarks, but are still far away from being performant and reliable enough to be deployed in real, user-facing, applications. In this paper, we ask: can we bridge this gap by enabling agents to ask for assistance from an expert such as a human being? To this end, we propose the Ask4Help policy that augments agents with the ability to request, and then use expert assistance. Ask4Help policies can be efficiently trained without modifying the original agent's parameters and learn a desirable trade-off between task performance and the amount of requested help, thereby reducing the cost of querying the expert. We evaluate Ask4Help on two different tasks -- object goal navigation and room rearrangement and see substantial improvements in performance using minimal help. On object navigation, an agent that achieves a $52\%$ success rate is raised to $86\%$ with $13\%$ help and for rearrangement, the state-of-the-art model with a $7\%$ success rate is dramatically improved to $90.4\%$ using $39\%$ help. Human trials with Ask4Help demonstrate the efficacy of our approach in practical scenarios.

Ask4Help: Learning to Leverage an Expert for Embodied Tasks
Kunal Pratap Singh PRIOR, Allen Institute for Al
Luca Weihs
PRIOR, Allen Institute for AI
Alvaro Herrasti PRIOR, Allen Institute for Al
Jonghyun Choi Yonsei University
Aniruddha Kembhavi PRIOR, Allen Institute for
Roozbeh Mottaghi PRIOR, Allen Institute for Al
Abstract
Embodied AI agents continue to become more capable every year with the advent of new models, environments, and benchmarks, but are still far away from being performant and reliable enough to be deployed in real, user-facing, applications. In this paper, ask: can bridge this gap by enabling agents to ask for assistance from an expert such as a human being? To this end, we propose the AsK4HELP policy that augments agents with the ability request, and then use expert assistance. AsK4HELP policies can be efficiently trained without modifying the original agent's parameters and learn desirable trade-off between task performance and the amount of requested help, thereby reducing the cost querying the expert. We evaluate Ask4HELP on two different tasks object goal navigation and room rearrangement and see substantial improvements in performance using minimal help. On object navigation, an agent that achieves a 52% success rate is raised to 86% with 13% help and for rearrangement, the state-of-the of-the--rttttttt model with 7% success rate is dramatically improved 90. 4% using 39% help. Human trials with 4sK4HELP demonstrate the efficacy of our approach in practical scenarios.
1 Introduction
The journey toward creating multipurpose household assistants continues to pose many challenges despite recent progress in computer vision and robotics. Open source simulators [27, 32, 44] and benchmarks [45, 2] have enabled advancements in diverse tasks such as navigation [4, 21]. interaction [15, 30], exploration [39, 9]. instruction following [45, 2] and rearrangement [3, 52]. And yet, the best performing models (e.g.. [25, 41J) on most tasks are not capable or reliable enough to be deployed in real-world applications.
For instance, the state-of the-t-rttt (SoTA) navigating towards specified object category hovers around 50%2 in the RoboTHOR [14] environment, and around 30%* in Habitat [32]. Slightly more complex tasks prove more daunting still, with the best models on Room Rearrangement [52] still below 10%4 In addition, these models often tend to get stuck, take repetitive actions and even collide
correspond kunals allenai.org for any queries/comments allenai www./roottorr...ooooooooooooo...
'nttps:/ http:///hhaabaaaa..ooochhaaallla 'nttps://aihaaittat. hhtpchhhhlegge2211/////////////e 'nttpe://Deaderboaar allenai http:///waarrroaaaaaaaaaaaaaaeaa
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

?
?
S/ASK4HELP
F1
t-0
Figure I: Learning to ask for help (ASK4HELP). Left: An agent struggles to find a spray bottle. Middle: The ASK4HELP policy recognizes this confusion and requests help from human. Right: The human provides help at time / = 3, which puts the agent on the right path and it succeeds.
with furniture and walls as they move around in scene. Deploying such agents in the real world can be costly, slow, and frustrating for users. In safety-critical situations such as rescue or reconnaissance operations, such failures can be even more expensive. While future modeling improvements may alleviate these concerns somewhat, we are still a ways away from creating agents that can perform household tasks robustly and safely.
Consider, for example, the setting of Fig. 1. robot is deployed in user's home and is tasked with locating objects, e.g. keys and remote controls, and is equipped with SoTA navigation model such as EmbCLIP [25]. Even if we assume that this performance would seamlessly transfer from simulation to the real world, the agent would only be successful at its task about 50% [25] of the time. This not good enough for deployment: few users would be satisfied with a household assistant which fails as frequently as it succeeds. If we want to deploy these models in practice we must improve their reliability. To this end, in this work, we assume access to an expert that can provide actions to agent upon request but, additionally, number of such queries should be minimized. Hence, we ask: how can we dramatically improve agents' performance and reliability at tasks by using a small amount of expert assistance?
assume that querying this expert is costly and so the
Building embodied agents that can request and use help from humans presents new challenges. Firstly, how does an agent determine when it is most effective to ask for help? A hand-designed heuristic is likely to be sub-optimal from the standpoint of an agent's policy. Secondly, how can we find the right balance between ensuring that we only request the human to intervene when it is really necessary while maximizing the performance at the specified task? Lastly, how can we teach an agent to ask for help, without having retrain the embodied agent that we trying to support? This would avoid the need to have access to the entire model structure and weights, save a significant amount of compute, and enable wider applicability.
To this end, we propose the Ask For Help (ASK4HELP) policy which augments E-AI agents with the ability to request for help when necessary. Our ASK4HELP policy: (a) is minimally invasive one can train this module without modifying the parameters of the underlying E-AI model; (b) can be trained efficiently, with fraction of the training data, compute and time, compared to the original model; and (c) learns a desirable trade-off between task performance and the amount of help, and even presents mechanism to the user to dynamically, at inference time without additional training, specify the cost of asking for help and thereby increase or decrease the amount of help that will be requested in future episodes.
We show the efficacy of our approach on the RoboTHOR object navigation [14] and A12-THOR visual room rearrangement [52] tasks. We learn an ASK4HELP policy support the existing off-the- shelf Embodied CLiP models [25] which have recently achieved SoTA performance on both of these tasks. We demonstrate significant improvements in these tasks using limited help from the expert. In ObjectNav, we improve 52% to 86% using 13% help, and in Rearrangement we go from 7% to 90.4% with 39% expert help. Importantly our ASK4HELP policy is model agnostic and can be easily applied to other F-AI models. Just as has provided improvements over current
it should also provide enhancements to the best models in the future. We also present strong results when

---

using human experts (measured via human trials) and noise-corrupted versions of the algorithmic expert and show that our policy is robust to these variations.
2 Related Work
Embodied AL Recently, various embodied tasks such as navigation [4, 24, 21, 19, 31], instruction following [45, 2, 29, 16], manipulation [15, 55, 58], embodied question answering [13, 20], and rearrangement [52, 3J have witnessed tremendous progress. This can be attributed to the availability
of open-source benchmarks [45, 2, 46, 51, 40, 22] and simulators [27, 32, 38, 44, 18, 55], stronger visual backbone models [25], self-superviser auxiliary tasks [57, 56] and task-specific inductive biases such as semantic mapping [7. 6]. Although, [54] show that the task of Point Navigation [2] in unseen environments can be solved by training 2 billion frames with reinforcement learning, the success rate of the best models for most embodied tasks is still very poor [25, 52, 41].
Complementary to the modeling progress, a line of work has emerged in the past few years that attempts to learn to request assistance in the form of language [35, 49, 37], sub-goals [34], and agent state and goals [33]. However, [34, 35] rely on imitation to learn when the agent should request help. This can be limiting since the human-
criteria to label the help-requesting behavior might not be optimal from the agent's standpoint. [49, 37] collect human-human dialog for task completion to learn communication between the agent and the expert. We try to make requesting expert assistance part of the reinforcement learning problem, and let our ASK4HELP policy infer the trade-off between the amount of assistance and performance. Along similar lines, [33] consider learning an "intention" policy to request varying modalities of information, such object detections, room types, and sub-goals, from the expert. These requested modalities may not, however, be easy or well-suited for actual human user to provide.
With similar motivation to our work, [10] propose two ways of asking for expert help, heuristic criteria based on model confidence and an augmentation of the agent's 's action space. We implement the heuristic criteria and present a comparison using the RoboTHOR [14] object navigation (ObjectNav) task. They also propose augmenting the agent's action space with an extra ask action. However, this involves retraining the underlying embodied model and additional reward shaping to prevent degenerate solutions. In contrast, just train extra policy without modifying the underlying Embodied AI model. We provide assistance in the form of expert actions similar to [10].
Active Learning and Perception. Our method for learning when to provide assistance based on the agent's internal representation is analogous to active learning [43]. In active learning, the agent tries to acquire labeled data to improve its model in the most sample-effcieen manner [17, 42]. However, contrary to the general active learning paradigm, we do not collect additional data to train our agent. Instead, we try to learn a policy with minimal information about the underlying Embodied Al model to answer the question, within an episode, when is the expert intervention most useful? We draw inspiration from active learning in the sense that the model tries pick out points in time for expert intervention as efficiently as possible. Prior works [47, 26, 48] use pre-defined metrics and heuristics to define state discrepancy and ask for demonstrations. However, these heuristics fail to generalize in unseen environments. We show the efficacy of our method in unseen, visually rich 3D environments [14, 52].
Uncertainty estimation using the value function to provide help and improve the sample- of RL algorithms has been explored in the literature before [12]. [50] propose teacher-student framework to study how an RL agent can transfer knowledge to another agent by action-advising. [11] extend this idea to a multi-agent setting where agents learn to teach and advise simultaneously. These works lay down some foundational ideas to involve expert in-the-loop, but they mostly focus on improving the sample efficiency and knowledge transfer of RL algorithms. Contrarily, propose to learn how to utilize human intervention to improve an off-the-shelf embodied agent's performance.
Recently, active perception has also been studied in the Embodied Al community. [8] learn a policy to choose which frames to label for object detection. [5] use perception models learned from internet data to learn an active exploration policy. [28] propose to fine-tune the object detection model at test time while interacting with an environment. [36] learn an exploration policy that balances the trade-off between semantic segmentation performance and the amount of annotation data requested. In contrast, we do not assume access to the underlying perception or planning parameters of the agent, we try to learn when human intervention would be effective in completing the task.
3

---

Frozen Embodied Al Model
Actor 01
Visual Encoder
Recurrent Unit
Critic
Ask4Help Model
b
Agent
Actor
R2
E
MLP
Expert 0z
GRU
Emb
Critic
R
Figure 2: Model overview. The ASK4HELP policy selects if the next action should be taken by the E-AI agent or the expert. b, represents agents "beliefs", i.e. the recurrent unit's hidden state. at denotes the action at time f, the superscript A indicates it was produced by the underlying agent's policy. denotes the expert. The double red lines indicate that gradients do not flow through that part of the model. Emb represents simple look-up embedding layer. RINN represent different reward configurations that we embed (see Sec. 3.3).
3 Learning to Leverage Expert's Help
Today's state-of-the-art E-AI models are not yet performant and reliant enough to be deployed into real applications. We propose ASK4HELP, policy to augment E-AI models with the ability to ping an expert, such as a human being, and leverage their intermittent help for the desired task.
3.1 Problem Definition
Consider typical Embodied AI (E-AI) model as shown in Figure 2. It consists of a visual encoder, a state encoder (typically a recurrent unit), and an actor-critic head, trained using Reinforcement Learning (RL). We wish to augment this model with the ability to query expert for help. Asking for too much help can be costly, or annoying if the expert is human being. Hence, our goal is to achieve good trade-off between asking for minimal amount of help and maximizing the performance of the agent at its task. We propose a framework to learn a policy that decides when it is most effective to ask for help. We refer to this policy the AsK4HELP policy. Importantly, we learn this AsK4HELP policy, without modifying the weights of the underlying E-AI model, allowing us to easily augment any off-the-sheif model with this capability. For this work, we consider that the help is provided in terms of the optimal action at the next time step, denoted by af Figure
3.2 ASK4HELP Policy
The output of the AsK4HELP policy determines if the agent or the expert policy would act at particular time step L. The objective of this policy is to maximize the rate of successful task completion while minimizing the amount of expert intervention. The 4 policy outputs, at every step, one of two actions (Agent or Expert) indicating whether the agent expert should take control at that time step. Hence, when the ASK4HELP policy outputs the Expert action, the action on the next time step is taken by the expert i.c. af.
ASK4HELP model's architecture is depicted Figure 2. and a GRU cell. It takes the underlying E-AI model's GRU hidden state (hereafter referred to as beliefs) b, as an input. The 4SK4HELP policy trained using reinforcement learning, specifically DD-PPO [54]. It receives two negative penalties, relatively large one for task failure, and smaller penalty for requesting expert help. The RL loss tries to balance penalties, thereby attempting avoid failure with minimal expert help. The gradients from this loss are used only to update the 4 policy, recall that the agent's 's policy frozen. We describe the reward structure and training details in Section 4.
It consists of a multi-layer perceptron
trade-off between these two

---

3.3 Adapting to User Preferences at Inference Time
The methodology described in Section 3.2 is an effective way to achieve good trade-off between task success and the amount of help requested. The amount of help requested by the agent is governed by the reward structure used during RI.. Requiring the agent to ask for help less (or more) frequently would require us to retrain the ASK4HEL. policy. This need for retraining is limiting as we may wish to deploy the same system in multiple settings each with very divergent costs associated with querying the expert: for instance, some users might be happy provide feedback during the day while others might find such requests highly disruptive. Ideally, such users should be able to communicate their preferences to the agent and have it query for help at rate that respects those preferences.
naïve way of tackling this would be train multiple ASK4HELP policies with range of potential user preferences and deploy all of them onto the robot. The user could then choose the agent most aligned with their preferences. Unfortunately, the cost of this approach grows linearly in the number of user preferences we wish to capture, fails to share information across ASK4HELP policies, and can only ever represent a discrete collection of preferences.
Therefore, to support a wider range of user preferences while decreasing computational costs, we propose explicitly condition our ASK4HELP policy on user preferences during training. In particular, as shown in Figure 2, we sample range of reward configurations RINN representing potential user preferences with different penalties associated with agent failure. Then, we train our agent by sampling rollouts with each reward configuration R, with uniform probability. We sample R2 for particular episode and keep it fixed throughout that episode. As shown in Figure 2, we embed the randomly sampled reward configuration R; into a 12-dimensional vector and provide that as an additional input the AsK4HELP policy's 's GRU. This allows the ASK4HELP policy to modify its behavior according to user preference. We show in Section 4 that this method allows us to train a single AsK4HEL. policy that can adjust the amount of help based on the reward configuration embedding.
4 Experiments and Analysis
We present empirical results and analyses when using our ASK4HELP policy with agents trained to complete the RoboTHOR Object Navigation (ObjectNav) and Visual Room Rearrangement (RoomR) tasks. We describe our experimental setup and the baselines in Sections 4.1 and 4.2, respectively, and present quantitative evaluations of AsK4HELP and competing baselines in Sections 4.3 and 4.4.
4.1 Experiment Setup
In our experiments, we wish to capture the setting in which practitioner has access to a frozen off-the-shelf Embodied AI model for particular task and wants to extend this model by enabling to ask for help. To be able to add this ask-for- capability, we assume the practitioner has newly created (or held-out) set of training data that was not used to train the off-the-shelf model and that this new training dataset is possibly significantly smaller than the off she-hhh-self model's 's training set. To this end, in what follows we will retrain existing SoTA models for ObjectNav and RoomR on subsets of their original training datasets, freeze these models, and then use the held-out training data to train a lightweight AsK4HEL. policies that works in conjunction with the E-AI models.
Dataset Split. We train the RoboTHOR [14] ObjectNav and iTHOR I-phase RoomR [52] models proposed in EmbCLIP [25], currently the published SoTA models for these two tasks, on 75% of the training scenes for their respective tasks (45 scenes for ObjectNav and 60 for RoomR). We use the publicly available codebase" provided by [25]. With these models fixed and frozen, we use the remaining 25% training scenes to train the the ASK4HELP policy. We evaluate our models on the unseen validation scenes that the agent has not seen before in training.
4.2 Baseline Definitions
We present a comparison of our ASK4HELP framework with various baselines, which are defined as: Naive Helper (NH) The agent receives expert help at a step with probability p. In particular, at
clip
5

---

every step, the NH samples from Bernoulli(p) distribution and takes the agent's is action if it samples a and, otherwise, takes the expert action. We control the amount of help by varying p and produce different variants of this baseline. We denote each variant using NH-p.
Model Confusion (MC): : This baseline is based on heuristic criteria proposed in [10] for requesting expert assistance. Intuitively, MC will request help when the agent is not sufficiently confident that
there is single correct action to take. More specifically, MC will mark agent as confused (thus requiring expert assistance) at time step /. when
Psorted (0) teorted [1] <,;
where Piorteed are the action probabilities at time sorted in descending order. We vary the parameter to produce different variants of this baseline and refer them as MC-c.
These baselines do not assume access to the underlying E-AI model parameters. As mentioned in Section 3.1, our ASK4HELP policy follows the same constraint. However, note that for the Naive Helper and Model Confusion the underlying E-AI model is the SoTA model from [25] trained on 100% of the training scenes. Whereas, for our 4 policy, as mentioned previously, we use the same model architecture trained on 75% training scenes as the underlying E-Al agent. We use the remaining 25% training data for the ASK4HELP policy.
4.3
RoboTHOR Object Navigation
4.3.1 Task Description and Metrics
Task. ObjectNav requires an agent to navigate through an environment and find an object of the specified category. The open-source RoboTHOR [14] environment supports this task with a robotic agent placed into visually rich home environment. The agent starts at a random location and is given a target object category (e-g.. apple) to find. Its action space consists of MoveAhead, RotateRight RotateLeft, LookUp, LookDown and End. An episode is considered successful if an instance of the target object category is visible and within I of the agent.
Metrics, We report the success rate (SR) and SPL [1] for our navigation agents. We also quantify the amount of help with an Expert Proportion (EP) metric, which is defined as
EP Nezpert/NNotaa
(2)
where Nezpert is the number of expert steps, and Ntotal is the total number of steps in an episode.
4.3.2 Training Details
ASK4HELP policy. We train the ASK4HELP policy using DD-PPO [54] for 15 million steps. We use the NavMesh algorithm in RoboTHOR [14] environment as the expert request help from. It provides the optimal action from the shortest path to the agent's 's current location to the target. In practice, one does not have access such an expert during testing/inference. Hence, we not only provide test results with this NavMesh expert but also provide results with noisy version of the NavMesh expert as well as human expert.
We use the AllenAct [53] framework to implement our model and training pipeline. The reward at any time step & is defined as:
F: = failt lask Timit_ask Tatep_asks
where ffait is a negative penalty for failure, Tinit_ask one-time penalty given to the agent when it requests expert help for the first time with Task the indicator function controlling this one- time negative reward, and Tstep_ask ask is a smaller penalty given for every step the expert takes. For ObjectNav, we set ffall = -10, Tamit_ask ask -1, and l'tep_askk -0.01.
The Tinit_ask penalty encourages autonomous operation when possible by discouraging the ASK4HELP policy from requesting help during an episode where the agent may be able to complete the entire episode without any assistance. Additionally, once expert help has been requested during an episode, Tstep ask encourages the policy to minimize the number of requests made to the expert. Here T sstte_ask which made significantly smaller in magnitude than Tinit_ask to reflect the intuitive assumption that the marginal cost of assistance decreases with repeated queries (e.g., if a human has been contacted for assistance, then the cost of giving two expert actions very similar to just giving one).

---

ObjectNav Success Rate (%)
100
100
24% Less Help
80
80-
ObjectNav SPL
70-
70-
50
50
EmbCLIP with Help
30-
Expert upper hound
30-
ASKHHELP
20-
Model Confusion
10-
Naive Helper
10-
35
Expert Proportion (EP %)
Expert Proportion (EP %)
Figure 3: ASK4HELP policies utilize expert queries efficiently for ObjectNav. By varying the parameter of Model Confusion and Naive Helper baselines well as the reward embedding input to the 4 policy (recall Sec. 3.3), we obtain a collection of different trade-offs between the proportion of steps spent querying the expert (EP) and the performance of these models as measured by the success rate and SPL metrics. As these plots show, our proposed 4SK4HELP policies dominate the other methods, achieving better success and SPL values for all EP values. These results shown here are an average of 5 evaluations of each model on the ObjectNav validation set; while very small, we also show error bars corresponding to 1 standard deviation computed across the 5 evaluations.
is also worth noting that we can vary 1 fail to generate multiple reward configurations REN to train a single policy as described in Section 3.3. Specifically, if set ffait 1, which would imply that the cost of failure is not very high for the user, we get a reward configuration R_1 with fast = -1, Tinit_ask 11, and l'ste_ess -0.01. Following the same trend, we vary, Tyail from -1 -30 cover a broad range of user preferences and generate rewards configuration namely R to R-30-
As discussed in Section 3.3, to train agent with all reward configurations {R_1..... R- uniformly sample a reward configuration R, for each episode in the environment. The reward the agent receives during that episode is based on R4
During validation, we not have access to these rewards, therefore we need to learn correspon- dence between user preference and agent behavior. To accomplish that, we associate each reward configuration with an index (e.g. -1 + R_1 - Toal = 1), and embed these configuration indices using a standard lookup in learnable embedding matrix. We provide this embedding input to the agent. This allows the agent to learn correspondence between this embedding input and the cost of failure.
4.3.3 Quantitative Analysis
ASK4HELP results. We present comparison of our ASK4HELP framework with the baselines defined Section 4.2. For these results, we train our ASK4HELP policy just once with multiple reward configurations described in Section 3.3. This allows us produce different variants of our method by simply varying the reward embedding input during inference. Figure 3 shows the and SPL metrics) of our model against competing
performance (as measured by the success baselines when restricted to various EP values. As shown in the figure, our proposed ASK4HELP policy efficiently uses a limited number of expert queries to achieve high success (4% EP results in a 16% absolute gain in success and with just 17% EP the agent achieves >92% success) and strictly dominates the other baselines by obtaining higher performance every EP level. In fact, to achieve greater than 90% success with the Naive Helper strategy, one must increase the EP to %, more than double what is required when using our ASK4HELP policy. Interestingly, the gains in SPL are somewhat less dramatic than for success rate. This is, however, intuitive: a high-qualijg 4SK4HELP policy should only query the expert if has already searched the scene exhaustively (thus attaining low SPL.) and is certain it cannot find the object without assistance. Note that, due to environment noise the NavMesh [23] expert, denoted by the dark blue dashed line, does not achieve 100% success rate. We provide qualitative results in the supplementary materials.
7

---

Algorithmic vs. Human Expert. In the above evaluations and when training our ASK4HELP policy we have used a NavMesh [23] expert which computes expert actions using ground-truth shortest paths from the environment. While such an expert may be available during training, no such expert is likely to be available at inference time as, otherwise, we may as well use such an agent rather than a learned model. This raises an important question: if the expert available at inference time may be different than the expert used at training, is our ASK4HELD policy robust to changes in the expert? To answer this question we conducted a human trial.
Table I: Human Trial Results, SR, SPL and EL represent the Success Rate, Success by path weighted length and episode length metrics respectively. Expert column indi- cates the expert used. EP corresponds to the Expert Proportion metric.
In this trial we selected a set of 131 episodes from the RoboTHOR ObjectNav validation set and, for each of these episodes, evaluated our ASk policy when using as an expert: (a) a set of 11 humans (each human expert was asked to provide assistance for between 10- 15 episodes, details in supplement), (b) the standard Nav agent described above, and (c) a collection of "corrupted experts" CE-e where, on being queried, CE- agent returns either the correct action with probability (I-e) or randomly selected navigation action with probability e. The results of our ASK4HELP policy
Expert
EP (%)
SR (%)
SPL
(%)
EL
Human
12.27
81.6
24.3
211
NavMesh [23]
11.09
87.7
21.9
239
CE-0.1
10.18
82.4
22.7
235
CE-0.2
11.77
84.0
23.0
234
CE-0.4
12.01
80.2
21.5
252
in these trials summarized in Table 1. Note that the performance of the ASK4HELP model when using the human and NavMesh experts is very similar, for approximately the same EP (12.27 v.s. 11.09) the humans appear to result in lower overall success rates (81.6% v.s. 87.7) but appear to guide the agent to using more efficient paths (SPL of 24.3 v.s. 21.9). Upon examining the failure cases of our model when using human experts we found many cases where humans would incorrectly end episodes when in sight of the target object but too far from the object to satisfy the RoboTHOR requirement that agents end their episodes within Im of the target. The results with the CE-e experts are somewhat surprising: even with significant expert corruption our ASK4HELP policy can still make use of the expert's 's feedback and obtains high performance with relatively small EP values. Even when e = 0.8, so that 80% of the expert's ' actions are randomly chosen, the ASK4HELP policy trained without any such noise is able to obtain success rates above the bascline model without help. We present CE-e expert results on the full validation set in the supplement.
CE-0.8
19.88
61.8
18.2
293
None
0
55.7
22.3
184
Data efficiency of ASK4HELP. As discussed in Section 4.1, we train the ASK4HELP policy on 25% of the training scenes. However, in some situations where training data is scarce, sacrificing this fraction might be unfeasible. To ablate the data efficiency of 4 policy, can we train ASK4HELP with 25% and 10% training scenes, with the exact same reward configuration and same underlying pre-trained task agent. We observe only a 2% drop in success rate and 1% in SPL for the same expert proportion (EP%). As the results indicate, ASK4HELP converges to a reasonable expert rele-performacce trade-off despite using just 10% training scenes, which in case of [14] object navigation is just 6 scenes.
Replacing pre-trained agent with a random agent. Having a pre-trained agent is important to ensure that we're not overusing the expert, and the underlying E-AI model is doing a major portion of the task. We train an AsK4HELP policy with a random object navigation policy, and compare with an AsK4HELP policy that works with a pre-trained agent. Both the policies are trained with the same reward configuration and training regime. As shown in Table 2, random agent takes far more expert help than a pre-trained agent, since it lacks to ability to perform object navigation hence is completely reliant on the expert.
Model setting
SR
SPL
EP (%)
EA
ASK4HELP (pre-trained)
86.3
33.2
12.31
17
ASK4HELP (random)
76.44
67.65
98.99
27
Table 2: Random vs Pre-trained task agent. SR denotes success rate, SPL denotes Success by path weighted length. EP denotes the expert proportion metric. EA denotes the number of actions taken by the expert.
Comparing 4 and Model Confusion' baselines under constrained amount of help. We performed an experiment where make the expert available only for 20 steps in an episode
8

---

(both during training and evaluation) and train an Ask policy and present a comparison with the model confusion baseline. On the unseen validation scenes, Ask4Help achieves success rate of 70%, whereas the model confusion baseline (Section 4.2) achieves 61 % success on object navigation.
4.4 Room Rearrangement in ITHOR
4.4.1 Task Description and Metrics
Task. We show results on the recently proposed 1-phase Visual Room Rearrangement (RoomR) [52] task. In RoomR, an agent is placed into a household environment, and given two egocentric images at every time step. One image shows the environment's 's current state, and the other image shows the target state that the agent must rearrange it to. The agent must then navigate around the room and interact with the objects to restore them to the goal state. The agent can take navigation actions MoveAhead, RotateRight etc. and high level interaction actions like PickUpX, where X is object category to be interacted with.
Metrics. We report the Fixed Striet (FS) and Success Rate (SR) metrics on the room rearrangement task. Please refer to the original work [52] for more details on these metrics. We also report the Expert Proportion (EP) metric described in Section 4.3.1.
4.4.2 Training Details
The ASK4HELP policy is trained using DD-PPO [54] for 15 million steps. The reward at any time step / is defined as:
re Lack Tsnit_ask + Tstep_ask ask + Trearrange
The Tinit ask and Tstep_act ask rewards serve the same purpose they did in Object Navigation results presented before. We set Timst_ _ak -0.5, and _step_osk = -.002. Unlike as for ObjectNav which used sparse fail penalty,
found that this sparse reward was insufficient train our ASK4HELP policy (note that SoTA models fail at RoomR in R33% of cases). To this end, we replace the r fail penalty with the reward rrearrange which simply equals the rearrangement specific reward employed in [52]. We use the Heuristic Expert from [52] as the expert to request help from.
4.4.3 Rearrangement Results
Table 3: Room Rearrangement Results. FS and SR represent the Fixed Strict (%) and Success Rate (%) metrics for rearrange- ment respectively. MC-c and NH-p denote the Model Confusion and Naive Helper baseline variants.
We present the performance for our ASK4HELP policy and other baselines in Table 3. We show that using our ASK4HELP Policy, we can boost the performance dra- matically the Fixed Striet (from 18. 13% to 95.32%) and Success Rate (from 6.9% to 90.4%) metrics, with an Expert Proportion of 39%.
Note that the Expert Proportion (EP) metric for rear- rangement is higher than what we observe for Object Navigation. This can be attributed to both the task com-
Model
EP EP
FS (1)
SR (1)
ASK4HELP (ours)
39
95.32
90.4
MC-0.2 [10]
48
88.96
81.4
plexity, and the relatively poor performance of the SoTA
MC-0.3 [10]
63
94.64
89.9
EmbCLIP [25] model. However, since our 4 policy operates independent of the underlying E-AI
NH-0.4
40
85.97
77.5
NH-0.5
50
90.24
83.6
model's 's parameters, our method would still be compat-
NH-0.6
60
92.77
87.6
ible with future models. If more performant models proposed for the task, the AsK4HELP policy will accordingly adjust the amount of expert help requested to support that model effectively.
NH-0.7
70
94.85
89.6
EmbCLIP [25]
0
18.13
6.90
Expert
100
96.53
92.8
We also present a comparison against the Naive Helper (NH) and Model Confusion (MC) baselines described in Section 4.1. As shown in Table 3, we outperform all variants of these baselines with the same or higher Expert Proportion than our method. This highlights the ability of our method to effectively utilize minimal help to boost task performance. Notably, it takes Naive Helper variant 70% Expert Proportion to perform on-par with our 4 policy that requires only 39% EPP Similarly, the Model Confusion baseline requires 63% EP to do the same.

---

5 Conclusion
We propose ASK4HELP policies, a model-agnostic approach to augment a given, pre-trained, E-AI model with the abilitty query expert for help. The goal of ASK4HELP policies is to leverage expert assistance to dramatically improve the reliability of existing embodied Al agents while also minimizing the number of requests made these experts. Our approach is agnostic about the parameters of the underlying Embodied Al and shows that relatively small amount of help results in massive performance improvements in object navigation and room rearrangement, two popular tasks in Embodied AI. By evaluating our ASK4HELP policies using different types of experts (shortest path planner, noisy variation of the planner, and humans), we find that our trained ASK4 policies are robust to expert variation, desirable as it reduces the burden of having to use precisely the same expert during training and inference. In summary, ASK4HELP policies provide a generic and robust mechanism for improving the performance of E-AI models with minimal expert assistance.
Limitations. We highlight two limitations of our approach. First, while it is great advantage that our 4SK4HELP policies are model agnostic and do not require retraining the underlying agent, this has the disadvantage that our agent cannot learn from the expert's feedback. This means that the agent may query the expert multiple times from the same state, a frustrating user experience. Second, we have trained our policies using non-human experts available on- on-policy. While is convenient that these experts exist for ObjectNav and RoomR, some E-AI tasks may only have offline datasets of expert trajectories collected from humans. Extending to this setting is exciting future work.
Acknowledgements We would like to thank members of A12 PRIOR team for participating in the human trials. We would also like to thank Erie Kolve, Jiasen Lu and Kuo-Hao Zeng for helpful discussions and feedback. We would also like to thank the anonymous reviewers for their feedback and discussions. JC was partly supported by the NRF grant (No.2022R 11P22IAACCR A2C4002300), IITP grants AI Graduate School Program (Yonsei University) 5%, No.2021-0-0200 AI Innovation Hub 5% 2022-0-001 13, 5%, 2022-0-00959, 5%, 2022-0-00871, 5%, 2022-0-00951, 5%) funded by the Korea government (MSIT).
20000020-0-01360
References
[1] Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva, and Amir Roshan Zamir. On evaluation of embodied navigation agents. arXiv, 2018. 6
[2] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. visually-groudded navigation instructions in real environments. In CVPR, 2018. 1,
ssiion-and-language navigation: Interpreting
[3] Dhruv Batra, Angel X. Chang, Sonia Chernova, Andrew J. Davison, Jia Deng, Vladlen Koltun, Sergey Levine, Jitendra Malik, Igor Mordatch, Roozbeh Mottaghi, Manolis Savva, and Hao Su. Rearrangement: A challenge for embodied ai. arXiv, 2020. 1,3
[4] Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, and Erik Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects, arXiv, 2020. 1,3
[5] Devendra Singh Chaplot, Murtaza Dalal, Saurabh Gupta, Jitendra Malik, and Ruslan Salakhut- dinov. Seal: Self-supervised embodied active learning. NeurIPS, 2021.
[6] Devendra Singh Chaplot, Dhiraj Gandhi, Abhinav Gupta, and Ruslan Salakhutdinov. Object
goal navigation using goal-oriented semantic exploration. In NeurIPS, 2020. 3
[7] Devendra Singh Chaplot, Dhiraj Gandhi, Saurabh Gupta, Abhinav Gupta, and Ruslan Salakhut- dinov. Learning to explore using active neural slam. In ICLR, 2020. 3
[8] Devendra Singh Chaplot, Helen Jiang, Saurabh Gupta, and Abhinav Gupta. Semantic curiosity for active visual learning. In ECCV, 2020. 3
[9] Tao Chen, Saurabh Gupta, and Abhinav Gupta. Learning exploration policies for navigation. In
ICLR, 2019. 1
[10] Ta- Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-tur, Just ask: An interactive learning framework for vision and language navigation. In AAAI, 2020. 3, 6, 9
[11] Felipe Leno da Silva, Ruben Glatt, and Anna Helena Reali Costa. Simultaneously learning and advising in multiagent reinforcement learning. In AAMAS, 2017.
10

---

[12]
Felipe Leno da Silva, Pablo Hernandez-Leal. Bilal Kartal, and Matthew Taylor. Uncertainty- aware action advising for deep reinforcement learning agents. In AAAI, 2020. Abhishek Das. Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied Question Answering. In CVPR, 2018. 3
[13]
[14] Matt Deitke, Winson Han, Alvaro Herrasti, Aniruddha Kembhavi, Eric Kolve, Roozbeh Mot- taghi, Jordi Salvador, Dustin Schwenk, Eli VanderBilt, Matthew Wallingford, Luca Weihs, Mark Yatskar, and Ali Farhadi. RoboTHOR: An Open Simulation-to- Ro-Real Embodied AI Platform. In CVPR, 2020. 1, 2, 3, 5, 6,
[15] Kiana Ehsani, Winson Han, Alvaro Herrasti, Eli VanderBilt, Luca Weihs, Eric Kolve, Anirud- dha Kembhavi, and Roozbeh Mottaghi. ManipulaTHOR: A Framework for Visual Object Manipulation. In CVPR, 2021. 1,
[16] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency. Taylor Berg-Kirkpatriek Kate Saenko, Dan Klein, and Trevor Darrell. Speaker- follower models for vision-and and-language navigation. In NeurIPS, 2018.
[17] Yarin Gal. Riashat Islam, and Zoubin Ghahramani. Deep bayesian active learning with image data. In /CML, 2017. 3
[18] Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang. Damian Mrowca, Michael Lingelbach, Aidan Curtis, Kevin T. Feigelis, Daniel Bear, Dan Gutfreund, David Cox, James J. DiCarlo, Josh H. McDermott, Joshua B. Tenenbaum, and Daniel L. K. Yamins. Threedworld: A platform for interactive multi-modal physical simulation. In NeurIPS (dataset track), 2021.
[19] Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, and Joshua B. Tenenbaum. Look, listen, and act: Towards audio-visual embodied navigation. ICRA, 2020. 3
[20] Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, and Ali Farhadi. Iqa: Visual question answering in interactive environments. CVPR, 2018. 3
[21] Saurabh Gupta, James Davidson, Sergey Levine, Rahul Sukthankar, and Jitendra Malik. Cogni-
tive mapping and planning for visual navigation. CVPR, 2017. 1,3
[22] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew Davison. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 2020. [23] Arthur Juliani, Vincent-Pierre Berges, Esh Vckay, Yuan Gao, Hunter Henry, Marwan Mattar, and Danny Lange. Unity: A general platform for intelligent agents. arXiv, 2018. 7,8
[24] Peter Karkus, Shaojun Cai,, and David Hsu. Differentiable slam-net: Learning particle slam for visual navigation. CVPR, 2021. 3
[25] Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, and Aniruddha Kembhavi. Simple but
effective: Clip embeddings for embodied ai. In CVPR, 2022. 1, 2, 3, 5, 6, 9
[26] Ross A. Knepper, Stefanie Tellex, Adrian Li, Nicholas Roy, and Daniela Rus. Recovering from failure by asking for help. Autonomous Robots, 2015. 3
[27] Eric Kolve, Roozbeh Mottaghi, Winson Han, Eli VanderBilt, Luca Weihs, Alvaro Herrasti, Daniel Gordon, Yuke Zhu, Abhinav Gupta, and Ali Farhadi. A12-THOR: An Interactive 3D Environment for Visual AI. arXiv, 2017. 3,
[28) Klemen Kotar and Roozbeh Mottaghi. Interactron: Embodied adaptive object detection. In
CVPR, 2022. 3
[29] Jacob Krantz, Erik Wijmans, Arjun Majumdar, Dhruv Batra, and Stefan Lee. Beyond the nav-graph: Iision-and-languaal navigation in continuous environments. In ECCV, 2020. 3 [30] Martin Lohmann, Jordi Salvador, Aniruddha Kembhavi, and Roozbeh Mottaghi. Learning about objects by learning to interact with them. NewrIPS, 2020.
[31] Oleksandr Maksymets, Vincent Cartillier, Aaron Gokaslan, Erik Wijmans, Wojeiech Galuba, Stefan Lee, and Dhruv Batra. Thda: Treasure hunt data augmentation for semantic navigation. ICCV, 2021. 3
[32] Manolis Savva*, Abhishck Kadian Oleksandr Maksymets s Yili Zhao, Erik Wijmans, Bhavana Jain, Julian Straub, Jia Liu, Vladlen Koltun, Jitendra Malik, Devi Parikh, and Dhruy Batra. Habitat: A Platform for Embodied AI Research. In ICCV, 2019. 1,
[33] Khanh Nguyen, Yonatan Bisk, and Hal Daumé III. Learning when and what to ask: a hierarchical reinforcement learning framework. arXiv, 2021.
[34] Khanh Nguyen and Hal Daumé III. Help, anna! visual navigation with natural multimodal assistance via retrospective auriosity-encouraginn imitation learning. In EMNLP, 2019. 3
11

---

[35]
Khanh Nguyen, Debadeepta Dey. Chris Brockett, and Bill Dolan. Vision- bason- navigation with language-based assistance via imitation learning with indirect intervention. In CVPR, 2019.
[36]
David Nilsson, Alcksis Pirinen, Erik Gärtner, and Cristian Sminchisescu. Embodied visual active learning for semantic segmentation. In AAAI, 20
[37]
Aishwarya Padmakumar. Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan- Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani- Tur. Teach:
Task embodied agents that chat. In AAAI, 2022. 3
[38] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li. Tingwu Wang, Sanja Fidler, and Antonio
Torrllb.. Virtualhome: Simulating household activities via programs. In CVPR, 2018.
[39] Santhosh K. Ramakrishnan, Ziad Al-Halah, and Kristen Grauman. Occupancy anticipation for
efficient exploration and navigation. In ECCV, 2020. 1
[40] Santhosh K. Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X. Chang, Manolis Savva, Yili Zhao, and Dhruv Batra, Habitat-matterport 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In NeurlPS (dataset track), 2021.
[41] Ram Ramrakhya, Eric Undersander, Dhruv Batra, and Abhishek Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In CVPR, 2022. 1,3
[42] Ozan Sener and Silvio Savarese. Active learning for convolutional neural networks: A core-set approach. In ICLR, 2018. 3
[43] Burr Settles. Active learning literature survey. 2009. 3
[44] Bokui Shen, Fei Xia, Chengshu Li. Roberto Mart in-Mart'in, Linxi (Jim) Fan, Guanzhi Wang, S. Buch, Claudia. Pérez D' Arpino, Sanjana Srivastava, Lyne P. Tchapmi, Micael Edmond Tchapmi, Kent Vainio, Li Fei-Fei, and Silvio Savarese. igibson, simulation environment for interactive tasks in large realistic scenes. In IROS, 2021. 3,
[45]
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mot- taghi, Luke Zettlemoyer, and Dieter Fox. ALFRED; A Benchmark for Interpreting Grounded
Instructions for Everyday Tasks. In CVPR, 2020. 1,
[46] Sanjana Srivastava, Chengshu Li, Michael Lingelbach, Roberto Mart'in Mia-Mar''in in, Fei Xia, Kent Vainio, Zheng Lian, Cem Gokmen, S. Buch, C. Karen Liu, Silvio Savarese, Hyowon Gweon, Jiajun Wu, and Li Fei-Fei. Behavior: Benchmark for everyday household activities in virtual, interactive, and ecological environments. In CoRL, 2021. 3
[47] Kaushik Subramanian, Charles Lee Isbell, and Andrea Lockerd Thomaz. Exploration from demonstration for interactive reinforcement learning. In AAMAS, 2016. 3
[48] Stefanie Tellex Ross A. Knepper, Adrian Li, Daniela Rus, and Nicholas Roy. Asking for help
using inverse semantics. In RSS, 2014. 3
[49] Jesse Thomason, Michael Murray, Maya Cakmak, and Luke Zettlemoyer. Vision-and-dialog navigation. In CoRL., 2019.
[50] Lisa A. Torrey and Matthew E. Taylor. Teaching on a budget: agents advising agents in reinforcement learning. In AAMAS, 2013. 3
[51] Saim Wani, Shivansh Patel, Unnat Jain, Angel X. Chang, and Manolis Savva. Multion: Benchmarking semantic map memory using multi-object navigation. In NeurlPS, 2020. 3
[52] Luca Weihs, Matt Deitke, Aniruddha Kembhavi, and Roozbeh Mottaghi. Visual room rear- rangement. In CVPR, 2021. 2, 5,
[53] Luca Weihs, Jordi Salvador, Klemen Kotar, Unnat Jain, Kuo-Hao Zeng, Roozbeh Mottaghi, and Aniruddha Kembhavi. Allenact: A framework for embodied ai research. arXiv, 2020. 6 [54] Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. DD-PPO: Learning near near-perfect pointgoal navigators from 2.5 billion
frames. In ICLR, 2020. 3, 4, 6,
[55] Fanbo Xiang, Yuzhe Qin, Kaichun Mo, Yikuan Xia, Hao Zhu, Fangchen Liu, Minghua Liu, Hanxiao Jiang, Yifu Yuan, He Wang, Li Yi, Angel X. X.Chang, Leonidas Guibas, and Hao Su. SAPIEN: A SimulAted bast-based Interactive ENvironment. In CVPR, 2020. 3
[56] Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans. Auxiliary tasks and exploration enable
objectgoal navigation. In ICCV, 2021. 3
[57] Joel Yc, Dhruv Batra, Erik Wijmans. and Abhishek Das. Auxiliary tasks speed up learning pointgoal navigation. In CoRL, 2020.
[58] Yuke Zhu, Daniel Gordon, Eric Kolve, Dieter Fox, Li Fei-Fei, Abhinav Kumar Gupta, Roozbeh Mottaghi, and Ali Farhadi. Visual semantic planning using deep successor representations.
ICCV, 2017. 3
12


Reviews:
[{"review_id": "xcBWyJZdKcT", "reviewer": "Reviewer_uya8", "summary": "This paper proposed a method to decide when to ask for expert’s help to improve the task performance of embodied agents. Instead of using heuristics, e.g. model confusion, or expanding the action space of the embodied agent, this paper learns a separate policy on top of the decisions of off-the-shelf embodied agents using reinforcement learning.   The experiments on RoboTHOR show that the proposed method can achieve higher success rates using fewer expert queries.  ", "questions": "- While the paper claims that it can learn efficiently with a fraction of training data, the experiment setup still suggests that it takes 25% of the training scenes to train the Ask4Help model. This is not a small fraction. It will be helpful if the authors can clarify on data efficiency vs. the performance of the Ask4Help model.\n- How do different reward configurations affect the number of questions asked? If the reward configuration represents user’s preferences for failure, a simple model can be thresholding the predicted success rate based on the selected user preference and have similar results. ", "limitations": "This paper has discussed the main limitation of the proposed method which provides opportunities for future work. Another limitation of the proposed method is the fixed user preference, the method doesn’t adapt the number of questions asked based on the interactions with experts.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The proposed method is generic. Without modifying the trained embodied agent, the method can learn when to perform expert queries.\n- The proposed method reduces the number of expert queries compared to baselines such as model confusion and naive helper.\n- The presentation of the paper is clear. It is easy to follow the paper. \n\nWeakness:\n- Missing ablations on different key components. There are two major inputs for the Ask4Help model, success rate prediction and the embodied agent’s belief, it is unclear the contribution of each component. The success rate is a strong indicator of whether the embodied agent needs help. It is likely that a classifier based on success rate is sufficient.\n- Unclear how to adapt to different reward preferences at inference time. Ask4Help trains the model with a sample of different reward configurations, but the paper doesn’t show how to estimate the expert’s reward configuration at inference time. Do experts select which reward profile they want? It is also possible that an expert adjusts their reward preference while interacting with the system.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "tzew69pMfGP", "reviewer": "Reviewer_gw2r", "summary": "This is a strong paper that examines the question of how to enrich existing policies for embodied tasks (in this case, object navigation and rearrangement) with the ability to “ask for help.” Unlike prior work that defines heuristics for model uncertainty, requires extra supervision in terms of language or subtasks, or requires retraining the base embodied agent policy, the proposed approach — Ask4Help — learns a base policy-agnostic approach for learning when to ask for help, *without the need to retrain the base policy*. Formalized as a separate policy that takes in a set of user preferences, the “belief” (hidden state) of the embodied base policy, and a small number of interactive rollouts in known environments, Ask4Help is able to learn to trade-off task success with “expert load” (amount of queries to the expert) with minimal data, and with remarkable results on two benchmark tasks — RoboTHOR object navigation and AI2-THOR room rearrangement.\n\nThis paper further goes above and beyond to show how the proposed system compares to scenarios where users have different preferences (weightings on success rate vs. desired query load), comparisons to ablations with fixed probabilities of picking “expert” actions, comparison of querying actual (vs. synthetic) humans, as well as the robustness of Ask4Help in the presence of noisy experts.", "questions": "- How are the reward preferences chosen? Is it right now just a weight on the ultimate success rate? \n- Is there a way to regress more versatile/natural preferences from users online?\n", "limitations": "This paper is very transparent about its limitations — namely that this Ask4Help approach only augments an existing policy with the ability to ask for help, not necessarily learn from the provided expert feedback.\n\nI agree that this is probably out of scope for this work, but for future work I’d suggest the authors look into frameworks like Lazy or ThriftyDAgger as ways to bootstrap systems that (1) know when to ask for expert help, and (2) learn/update policies based on that information.\n", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is original and significant, proposing a system that not only makes sense and seems necessary as we build stronger, more powerful embodied systems. The evaluation is incredibly convincing, and the ablations are well chosen and are crucial in showing the efficacy of the proposed approach.\n\nThe clarity of the paper is also an added bonus — the motivating examples were clear and helped ground out the early parts of the paper, the approach is simple yet flexible, and in general, led to a strong paper.\n\nThe sole weakness is that I wish the user preference component had a little bit more discussion; it’s still not clear to me how this component interacts with the rest of the system (and especially the training/reward function), nor how the various “preference embeddings” are specified/chosen.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "kUQGOPIQkV9", "reviewer": "Reviewer_bRWC", "summary": "The paper introduces Ask4Help, a method for augmenting an existing policy with the ability to fall back to an expert policy during an episode. This is achieved without retraining the existing pre-trained agent by introducing a meta-controller that will select whether to follow the agent or the expert at every timestep. The meta-controller does not receive raw observations, but the agent's belief state, a prediction of the agent's success rate, and an embedding informing it about the user's preference (i.e. how costly it is to ask for help). The proposed method is evaluated on two tasks, namely object navigation (RoboTHOR) and room rearrangement (iTHOR), where it greatly boosts the success rate of the pre-trained agent while comparing favorably to other baselines in the amount of expert usage.", "questions": "**Major**\n\nI have described my main questions and concerns in the previous section, namely:\n- Stats about expert usage\n- Additional baselines\n- Importance of the different design choices in the Ask4Help architecture\n- Dataset split\n\n\n**Minor**\n\n- After following the expert, the agent could find itself in states where it is out of distribution. This is a drawback of freezing the pre-trained agent that is not discussed in the manuscript, and a discussion about this would benefit the paper.\n- Section 3.3 describes how Ask4Help can be trained with multiple user preferences. However, Section 4 describes a single reward function. Could you please explain how this is done, and provide examples showing how this affects the success rate and expert usage?\n- Given the connections with Hierarchical RL, I would strongly recommend extending the Related Work section to include an overview of the field.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strenghts**\n\n- The problem of providing agents with the ability to ask for help is an important one.\n- The paper is easy to follow.\n\n\n**Weaknesses**\n\n- The method itself is not novel, as it can be seen as a Hierarchical RL with two low-level policies: the pre-trained agent and the expert.\n- While the quantitative results look strong, the videos in the supplementary material show that the discovered behavior is extremely simple. During a first phase the meta-controller selects the agent, which does not seem to know how to solve the task and simply roams around the room. Then, it selects the expert for a few timesteps -- which finds the object and solves the task. This results in high success rate (thanks to the hard-coded expert) but low expert usage (which is diluded due to the long initial phase where the agent is used). It would be helpful if authors could provide some statistics about expert usage aggregated over all tasks (e.g. detailed stats about the timesteps in which the expert is used within an episode, aggregated over all episodes).\n- In light of the aforementioned qualitative results, I am not convinced that the baselines are adequate. There are some additional baselines that would help understanding whether the Ask4Help policy is learning something trivial or not:\n  - Replacing the pre-trained agent with a random and/or no-op policy. This would provide insight about whether it is really learning to combine the pre-trained agent and the expert, or whether the observed gains just come from the fact that the expert will always solve the task when given enough time.\n  - Replacing the meta-controller with a hard-coded policy that selects the agent for M steps and then runs the expert for N steps (where both M and N should be swept over).\n- The Ask4Help policy has a non-standard architecture, e.g. it takes the predicted success rate as input. There are no ablation studies in the paper (nor the supplementary material), which make it difficult to understand whether this is an important component of the agent or could be removed.\n- The dataset split described in L208 is non-standard in machine learning. My understanding is that the purpose of the split is to deploy the agent on a set of tasks it has never seen before and where it might not perform well. However, why isn't there a third, truly held-out, set ot tasks where one can evaluate whether the Ask4Help policy generalizes? Otherwise, if one is allowed to train and evaluate on the same set of tasks, why can't we re-train the agent on the validation set instead of using Ask4Help?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "ShwB63hMO8L", "reviewer": "Reviewer_rsmr", "summary": "This paper introduces a method, Ask4Help, for incorporating expert knowledge for embodied AI. The authors leveraged a pre-trained embodied AI agent and designed a policy to switch between agents' prediction and expert action as the final acting policy. The switching policy is trained by RL given  the reward from both success rate and the portion of expert knowledge used in the episode. The resulting human-in-the-loop policy achieves performance improvement on several common embodied AI challenges.", "questions": "1. As mentioned previously in the Weakness section, the authors should clarify the motivation of designing a policy that directly adopts expert knowledge without learning from it, especially on how such a policy would benefit us on more embodied AI challenges without that much expert knowledge.\n2. The authors should discuss the design of experimental settings and baseline methods: why not set a constraint of expert knowledge available? why not compare with prior methods that refine models with human expert knowledge under the same constraint? how is the current result significant since it is actually provided with all expert knowledge since the model learns to select between using it and not using it automatically?\n3. Model-wise, the motivation for designing the Success Prediction Model (SPM) is somewhat duplicative to me since it is in-essence doing the same job of predicting which action to use (e.g. the expert's when success prediction is low, the model's when success prediction is high). I hope the authors could discuss if the SPM model is substitutable by the reward received by Ask4Help since it is trained offline with ground truth successful and failure trails.\n4. In figure 3, why is the upper bound of results, not 100%, I guess this is related to the task but hope the authors would clarify.", "limitations": "The authors have partially stated their limitations on continual learning. However, I think there is still more work to do to make the current experimental design solid and sound. I hope the authors would consider the facts stated previously in Weakness and Questions, especially on the significance of the results since it now needs all expert knowledge during training and adopts partial of them during acting. This setting, to me, is not reasonable and might need better adjustment.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] The problem of incorporating expert knowledge and collaborating with humans is becoming increasingly important over the past few years, especially with more agents showing strong potential for indoor tasks. This makes the main topic of this paper important and meaningful.\n\n[+] The overall writing of this paper is clear and illustrative with ideas, methods, and results clearly stated. The authors showed significant performance improvements on several embodied AI challenges by augmenting a pre-trained embodied AI agent with expert knowledge. This shows strong potential for the pipeline of prompting large-scale pre-trained agents with expert knowledge, especially with recent trends on language understanding (e.g. few-shot capabilities of GPT-3).\n\n[-] The major concern of this paper comes with its design and evaluation metrics. The authors stated in the limitation section that the proposed model possesses the disadvantage that it can not learn from the expert's feedback. This, in my opinion, is a critical issue since it always requires expert knowledge to perform well and fails to leverage extra in-context expert knowledge. As the current policy only switches between the pre-trained agent's action and the expert's action, I don't see a way to make the current system a self-adaptive one when put into a new training scene. This makes the whole point of designing such a policy for querying expert knowledge questionable.\n\n[-] Following the previous point, I do think the current experimental settings and results are not convincing or have little impact. The authors used RL to train the Ask4Help policy for deciding which action to use (expert's or the model's) and seem to provide full accessibility of expert knowledge when generating the final scores (since the model can choose the amount of expert knowledge to use), this makes the current results hard to interpret as the model can always learn to use more expert knowledge without a limit. In this case, a proper comparison in my opinion should be constrained under the amount of expert knowledge available and over the performance gain of applying each method. Next, the baseline models are also designed to follow the same pipeline of policy switching without baselines from previous works that leverage the same amount of data for imitation learning or refinement.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IuolXUqfaKp", "reviewer": "Reviewer_HGan", "summary": "This paper is about learning an ask for help (ask4help) policy on top of an already trained embodied agent. Specifically, the ask4help policy first measures the agent's uncertainty at finishing the given task based on its internal belief state using a pretrained Success Prediction Model (SPM). Then, the ask4help policy decides if it should ask for the expert's help (i.e. the next action to do in the environment) or use the pretrained embodied agent's predicted action. In addition, the ask4help can be adapted at inference time to different users' preference regarding the frequency of being \"disturbed\" by the agent to answer it. Experiments were conducted on two different environments: the RoboTHOR Object Navigation and the Room Rearrangement in iTHOR. Empirically, it was shown that ask4help manages to yield a very higher success rate while requesting the least amount of expert feedback compared to different baselines: the Embodied Agent only, a Naive Helper with predefined frequencies for asking for help, and Model Confusion that measures the agent's uncertainty based on the confidence of the predicted action rather than relying on the agent's belief state.\n", "questions": "- In the RoomR environment, did the authors investigate what type of action (navigation vs. interaction) provided by the expert was the most common?\n- What reward configuration is being used during the evaluation?\n- What is the distribution of the reward configurations?\n- Are the loop-up embeddings for the reward configuration learned?\n- Are the authors going to release the code and trained ask4help policy so the community can reproduce the results?", "limitations": "The authors did mention two main limitations of the proposed approach. At the moment, expert feedback is not used to improve the embodied agent which can be annoying to a human user. The second limitation of this work is the authors used proxy experts (available for the tested environments) to train the ask4help policy. Using human-in-the-loop for training is not explored in this work (it is still an active area of research).\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**What I like about this paper**\n- How to reuse existing trained models is an important research direction with positive environmental impact.\n- The authors propose a solution to train a single ask4help policy that can deal with different user preferences via sampling different reward configurations during training.\n\n**Potential weaknesses**\n- It is not clear to me what reward configuration is used at inference time? Section 4.3.2 and 4.4.2 do mention a **single** reward configuration used for training.\n- In 4.4.3, it is said \"If more performance models are proposed for the task, the Ask4Help policy will accordingly adjust the amount of expert help requested...\". That sentence seems to imply that this will happen at inference time. However, it is my understanding that both the ask4help policy and the Success Prediction Model will need to be retrained to accommodate for the likely different embodied agent's belief $b_t$.\n\n\n**Originality, quality, clarity, and significance**\n\nThis work shares many similarities with active learning but takes place in interactive environments. The proposed approach is a novel combination of existing techniques applied together to tackle the important research problem of deciding when to ask for help. I could see future work building on this to integrate better with human workflow (e.g., using language to ask questions and interpret the answer similar to [Asking for Knowledge, Liu et al., ICML2022]). I found the paper well-written and well-organized. Figure 2 helped me understand the model overall but it was not clear how the Success Prediction Model was getting any training signal since the gradients were blocked. I realized later it is pretrained and frozen. The submission seems technically sound to me except for how the different reward configurations were defined during training.\n\nOverall, I tend to recommend this paper for acceptance because of the research problem being addressed and the proposed solution that doesn't require training an embodied agent from scratch. I might have missed some flaws, especially with respect to active learning.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept