Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposed a method to decide when to ask for expert’s help to improve the task performance of embodied agents. Instead of using heuristics, e.g. model confusion, or expanding the action space of the embodied agent, this paper learns a separate policy on top of the decisions of off-the-shelf embodied agents using reinforcement learning.   The experiments on RoboTHOR show that the proposed method can achieve higher success rates using fewer expert queries.  
strengths_and_weaknesses: Strength:
- The proposed method is generic. Without modifying the trained embodied agent, the method can learn when to perform expert queries.
- The proposed method reduces the number of expert queries compared to baselines such as model confusion and naive helper.
- The presentation of the paper is clear. It is easy to follow the paper. 

Weakness:
- Missing ablations on different key components. There are two major inputs for the Ask4Help model, success rate prediction and the embodied agent’s belief, it is unclear the contribution of each component. The success rate is a strong indicator of whether the embodied agent needs help. It is likely that a classifier based on success rate is sufficient.
- Unclear how to adapt to different reward preferences at inference time. Ask4Help trains the model with a sample of different reward configurations, but the paper doesn’t show how to estimate the expert’s reward configuration at inference time. Do experts select which reward profile they want? It is also possible that an expert adjusts their reward preference while interacting with the system.  
questions: - While the paper claims that it can learn efficiently with a fraction of training data, the experiment setup still suggests that it takes 25% of the training scenes to train the Ask4Help model. This is not a small fraction. It will be helpful if the authors can clarify on data efficiency vs. the performance of the Ask4Help model.
- How do different reward configurations affect the number of questions asked? If the reward configuration represents user’s preferences for failure, a simple model can be thresholding the predicted success rate based on the selected user preference and have similar results. 
limitations: This paper has discussed the main limitation of the proposed method which provides opportunities for future work. Another limitation of the proposed method is the fixed user preference, the method doesn’t adapt the number of questions asked based on the interactions with experts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper is about learning an ask for help (ask4help) policy on top of an already trained embodied agent. Specifically, the ask4help policy first measures the agent's uncertainty at finishing the given task based on its internal belief state using a pretrained Success Prediction Model (SPM). Then, the ask4help policy decides if it should ask for the expert's help (i.e. the next action to do in the environment) or use the pretrained embodied agent's predicted action. In addition, the ask4help can be adapted at inference time to different users' preference regarding the frequency of being "disturbed" by the agent to answer it. Experiments were conducted on two different environments: the RoboTHOR Object Navigation and the Room Rearrangement in iTHOR. Empirically, it was shown that ask4help manages to yield a very higher success rate while requesting the least amount of expert feedback compared to different baselines: the Embodied Agent only, a Naive Helper with predefined frequencies for asking for help, and Model Confusion that measures the agent's uncertainty based on the confidence of the predicted action rather than relying on the agent's belief state.

strengths_and_weaknesses: **What I like about this paper**
- How to reuse existing trained models is an important research direction with positive environmental impact.
- The authors propose a solution to train a single ask4help policy that can deal with different user preferences via sampling different reward configurations during training.

**Potential weaknesses**
- It is not clear to me what reward configuration is used at inference time? Section 4.3.2 and 4.4.2 do mention a **single** reward configuration used for training.
- In 4.4.3, it is said "If more performance models are proposed for the task, the Ask4Help policy will accordingly adjust the amount of expert help requested...". That sentence seems to imply that this will happen at inference time. However, it is my understanding that both the ask4help policy and the Success Prediction Model will need to be retrained to accommodate for the likely different embodied agent's belief $b_t$.


**Originality, quality, clarity, and significance**

This work shares many similarities with active learning but takes place in interactive environments. The proposed approach is a novel combination of existing techniques applied together to tackle the important research problem of deciding when to ask for help. I could see future work building on this to integrate better with human workflow (e.g., using language to ask questions and interpret the answer similar to [Asking for Knowledge, Liu et al., ICML2022]). I found the paper well-written and well-organized. Figure 2 helped me understand the model overall but it was not clear how the Success Prediction Model was getting any training signal since the gradients were blocked. I realized later it is pretrained and frozen. The submission seems technically sound to me except for how the different reward configurations were defined during training.

Overall, I tend to recommend this paper for acceptance because of the research problem being addressed and the proposed solution that doesn't require training an embodied agent from scratch. I might have missed some flaws, especially with respect to active learning.
questions: - In the RoomR environment, did the authors investigate what type of action (navigation vs. interaction) provided by the expert was the most common?
- What reward configuration is being used during the evaluation?
- What is the distribution of the reward configurations?
- Are the loop-up embeddings for the reward configuration learned?
- Are the authors going to release the code and trained ask4help policy so the community can reproduce the results?
limitations: The authors did mention two main limitations of the proposed approach. At the moment, expert feedback is not used to improve the embodied agent which can be annoying to a human user. The second limitation of this work is the authors used proxy experts (available for the tested environments) to train the ask4help policy. Using human-in-the-loop for training is not explored in this work (it is still an active area of research).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This is a strong paper that examines the question of how to enrich existing policies for embodied tasks (in this case, object navigation and rearrangement) with the ability to “ask for help.” Unlike prior work that defines heuristics for model uncertainty, requires extra supervision in terms of language or subtasks, or requires retraining the base embodied agent policy, the proposed approach — Ask4Help — learns a base policy-agnostic approach for learning when to ask for help, *without the need to retrain the base policy*. Formalized as a separate policy that takes in a set of user preferences, the “belief” (hidden state) of the embodied base policy, and a small number of interactive rollouts in known environments, Ask4Help is able to learn to trade-off task success with “expert load” (amount of queries to the expert) with minimal data, and with remarkable results on two benchmark tasks — RoboTHOR object navigation and AI2-THOR room rearrangement.

This paper further goes above and beyond to show how the proposed system compares to scenarios where users have different preferences (weightings on success rate vs. desired query load), comparisons to ablations with fixed probabilities of picking “expert” actions, comparison of querying actual (vs. synthetic) humans, as well as the robustness of Ask4Help in the presence of noisy experts.
strengths_and_weaknesses: This paper is original and significant, proposing a system that not only makes sense and seems necessary as we build stronger, more powerful embodied systems. The evaluation is incredibly convincing, and the ablations are well chosen and are crucial in showing the efficacy of the proposed approach.

The clarity of the paper is also an added bonus — the motivating examples were clear and helped ground out the early parts of the paper, the approach is simple yet flexible, and in general, led to a strong paper.

The sole weakness is that I wish the user preference component had a little bit more discussion; it’s still not clear to me how this component interacts with the rest of the system (and especially the training/reward function), nor how the various “preference embeddings” are specified/chosen.

questions: - How are the reward preferences chosen? Is it right now just a weight on the ultimate success rate? 
- Is there a way to regress more versatile/natural preferences from users online?

limitations: This paper is very transparent about its limitations — namely that this Ask4Help approach only augments an existing policy with the ability to ask for help, not necessarily learn from the provided expert feedback.

I agree that this is probably out of scope for this work, but for future work I’d suggest the authors look into frameworks like Lazy or ThriftyDAgger as ways to bootstrap systems that (1) know when to ask for expert help, and (2) learn/update policies based on that information.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper introduces a method, Ask4Help, for incorporating expert knowledge for embodied AI. The authors leveraged a pre-trained embodied AI agent and designed a policy to switch between agents' prediction and expert action as the final acting policy. The switching policy is trained by RL given  the reward from both success rate and the portion of expert knowledge used in the episode. The resulting human-in-the-loop policy achieves performance improvement on several common embodied AI challenges.
strengths_and_weaknesses: [+] The problem of incorporating expert knowledge and collaborating with humans is becoming increasingly important over the past few years, especially with more agents showing strong potential for indoor tasks. This makes the main topic of this paper important and meaningful.

[+] The overall writing of this paper is clear and illustrative with ideas, methods, and results clearly stated. The authors showed significant performance improvements on several embodied AI challenges by augmenting a pre-trained embodied AI agent with expert knowledge. This shows strong potential for the pipeline of prompting large-scale pre-trained agents with expert knowledge, especially with recent trends on language understanding (e.g. few-shot capabilities of GPT-3).

[-] The major concern of this paper comes with its design and evaluation metrics. The authors stated in the limitation section that the proposed model possesses the disadvantage that it can not learn from the expert's feedback. This, in my opinion, is a critical issue since it always requires expert knowledge to perform well and fails to leverage extra in-context expert knowledge. As the current policy only switches between the pre-trained agent's action and the expert's action, I don't see a way to make the current system a self-adaptive one when put into a new training scene. This makes the whole point of designing such a policy for querying expert knowledge questionable.

[-] Following the previous point, I do think the current experimental settings and results are not convincing or have little impact. The authors used RL to train the Ask4Help policy for deciding which action to use (expert's or the model's) and seem to provide full accessibility of expert knowledge when generating the final scores (since the model can choose the amount of expert knowledge to use), this makes the current results hard to interpret as the model can always learn to use more expert knowledge without a limit. In this case, a proper comparison in my opinion should be constrained under the amount of expert knowledge available and over the performance gain of applying each method. Next, the baseline models are also designed to follow the same pipeline of policy switching without baselines from previous works that leverage the same amount of data for imitation learning or refinement.
questions: 1. As mentioned previously in the Weakness section, the authors should clarify the motivation of designing a policy that directly adopts expert knowledge without learning from it, especially on how such a policy would benefit us on more embodied AI challenges without that much expert knowledge.
2. The authors should discuss the design of experimental settings and baseline methods: why not set a constraint of expert knowledge available? why not compare with prior methods that refine models with human expert knowledge under the same constraint? how is the current result significant since it is actually provided with all expert knowledge since the model learns to select between using it and not using it automatically?
3. Model-wise, the motivation for designing the Success Prediction Model (SPM) is somewhat duplicative to me since it is in-essence doing the same job of predicting which action to use (e.g. the expert's when success prediction is low, the model's when success prediction is high). I hope the authors could discuss if the SPM model is substitutable by the reward received by Ask4Help since it is trained offline with ground truth successful and failure trails.
4. In figure 3, why is the upper bound of results, not 100%, I guess this is related to the task but hope the authors would clarify.
limitations: The authors have partially stated their limitations on continual learning. However, I think there is still more work to do to make the current experimental design solid and sound. I hope the authors would consider the facts stated previously in Weakness and Questions, especially on the significance of the results since it now needs all expert knowledge during training and adopts partial of them during acting. This setting, to me, is not reasonable and might need better adjustment.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper introduces Ask4Help, a method for augmenting an existing policy with the ability to fall back to an expert policy during an episode. This is achieved without retraining the existing pre-trained agent by introducing a meta-controller that will select whether to follow the agent or the expert at every timestep. The meta-controller does not receive raw observations, but the agent's belief state, a prediction of the agent's success rate, and an embedding informing it about the user's preference (i.e. how costly it is to ask for help). The proposed method is evaluated on two tasks, namely object navigation (RoboTHOR) and room rearrangement (iTHOR), where it greatly boosts the success rate of the pre-trained agent while comparing favorably to other baselines in the amount of expert usage.
strengths_and_weaknesses: **Strenghts**

- The problem of providing agents with the ability to ask for help is an important one.
- The paper is easy to follow.


**Weaknesses**

- The method itself is not novel, as it can be seen as a Hierarchical RL with two low-level policies: the pre-trained agent and the expert.
- While the quantitative results look strong, the videos in the supplementary material show that the discovered behavior is extremely simple. During a first phase the meta-controller selects the agent, which does not seem to know how to solve the task and simply roams around the room. Then, it selects the expert for a few timesteps -- which finds the object and solves the task. This results in high success rate (thanks to the hard-coded expert) but low expert usage (which is diluded due to the long initial phase where the agent is used). It would be helpful if authors could provide some statistics about expert usage aggregated over all tasks (e.g. detailed stats about the timesteps in which the expert is used within an episode, aggregated over all episodes).
- In light of the aforementioned qualitative results, I am not convinced that the baselines are adequate. There are some additional baselines that would help understanding whether the Ask4Help policy is learning something trivial or not:
  - Replacing the pre-trained agent with a random and/or no-op policy. This would provide insight about whether it is really learning to combine the pre-trained agent and the expert, or whether the observed gains just come from the fact that the expert will always solve the task when given enough time.
  - Replacing the meta-controller with a hard-coded policy that selects the agent for M steps and then runs the expert for N steps (where both M and N should be swept over).
- The Ask4Help policy has a non-standard architecture, e.g. it takes the predicted success rate as input. There are no ablation studies in the paper (nor the supplementary material), which make it difficult to understand whether this is an important component of the agent or could be removed.
- The dataset split described in L208 is non-standard in machine learning. My understanding is that the purpose of the split is to deploy the agent on a set of tasks it has never seen before and where it might not perform well. However, why isn't there a third, truly held-out, set ot tasks where one can evaluate whether the Ask4Help policy generalizes? Otherwise, if one is allowed to train and evaluate on the same set of tasks, why can't we re-train the agent on the validation set instead of using Ask4Help?
questions: **Major**

I have described my main questions and concerns in the previous section, namely:
- Stats about expert usage
- Additional baselines
- Importance of the different design choices in the Ask4Help architecture
- Dataset split


**Minor**

- After following the expert, the agent could find itself in states where it is out of distribution. This is a drawback of freezing the pre-trained agent that is not discussed in the manuscript, and a discussion about this would benefit the paper.
- Section 3.3 describes how Ask4Help can be trained with multiple user preferences. However, Section 4 describes a single reward function. Could you please explain how this is done, and provide examples showing how this affects the success rate and expert usage?
- Given the connections with Hierarchical RL, I would strongly recommend extending the Related Work section to include an overview of the field.
limitations: Yes.
review_title: null

Meta-review: metareview: I thank the authors for their submission and active participation in the discussions. This paper introduces a method for learning a policy that can ask an expert for help, i.e., to obtain the expert action. On the positive side, reviewers found the method to be general [uya8], original and significant [gw2r], intruiging in terms of being able to reuse an existing policy [HGan], and tackling an important problem [rsmr,bRWC], and the paper to be clear [uya8,gw2r,rsmr,bRWC]. In terms of negative points, reviewers were concerned about the novelty [bRWC], unimpressive qualitative results despite strong quantitative results [bRWC], and issues with the range of baselines [bRWC,rsmr] and ablations considered [uya8]. Overall, the paper is borderline. However, bRWC indicated they would raise their score but I don't see this being reflected. Furthermore, in my view reviewer rsmr's concerns regarding baselines and ablations has been addressed by the author rebuttal. Thus, I am siding with reviewers gw2r and HGan, and recommend acceptance. However, I very strongly encourage the authors to further improve their paper based on the reviewer feedback, in particular the points raised by reviewer bRWC regarding the importance of the Success Prediction component of the method.  
confidence: Certain
Decision: Accept