Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation
Authors: Josiah Wong
Albert Tung
Andrey Kurenkov
Ajay Mandlekar
Li Fei-Fei
Silvio Savarese
Roberto Martín-Martín
Venue: CoRL2021 Poster (2021)
Keywords: Mobile Manipulation
Imitation Learning
Error Detection

Abstract: In mobile manipulation (MM), robots can both navigate within and interact with their environment and are thus able to complete many more tasks than robots only capable of navigation or manipulation. In this work, we explore how to apply imitation learning (IL) to learn continuous visuo-motor policies for MM tasks. Much prior work has shown that IL can train visuo-motor policies for either manipulation or navigation domains, but few works have applied IL to the MM domain. Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed. To address these problems, we first propose Mobile Manipulation RoboTurk (MoMaRT), a novel teleoperation framework allowing simultaneous navigation and manipulation of mobile manipulators, and collect a first-of-its-kind large scale dataset in a realistic simulated kitchen setting. We then propose a learned error detection system to address the covariate shift by detecting when an agent is in a potential failure state. We train performant IL policies and error detectors from this data, and achieve over 45% task success rate and 85% error detection success rate across multiple multi-stage tasks when trained on expert data. Additional results and video at https://sites.google.com/view/il-for-mm/home.

Error-Aware Imitation Learning from Teleoperation Data for Mobile Manipulation
Josiah Wong, Albert Tung, Andrey Kurenkov, Ajay Mandlekar, Li Fei-Fei, Silvio Savarese, Roberto Martin-Martin
Stanford University
Abstract: In mobile manipulation (MM), robots can both navigate within and interact with their environment and are thus able to complete many more tasks than robots only capable of navigation or manipulation. In this work, we ex- plore how apply imitation learning (IL) to learn continuous visuo-motor poli- cies for MM tasks. Much prior work has shown that IL can train visuo-motor policies for either manipulation navigation domains, but few works have ap- plied II. to the MM domain. Doing this is challenging for two reasons: on the data side, current interfaces make collecting high-quality human demonstrations difficult, and on the learning side, policies trained on limited data can suffer from covariate shift when deployed. To address these problems, we first pro- pose MOBILE MANIPULATION ROBOTURK (MOMART), a novel teleoperation framework allowing simultaneous navigation and manipulation of mobile ma- nipulators, and collect a first-of-its-kind large scale dataset in a realistic simu- lated kitchen setting. We then propose a learned error detection system to ad- dress covariate shift by detecting when an agent is in a potential failure state. We train performant IL policies and error detectors from this data, and achieve over 45% task success rate and 85% error detection success rate across multiple multi-stage tasks when trained on expert data. Additional results and video http://.ossssssllle..e/ https: /tstte//sites...t google. ggogle.coo/viee//11 il-forrmm/home.
Keywords: Mobile Manipulation, Imitation Learning, Error Detection
1 Introduction
In mobile manipulation (MM), robots combine interaction and locomotion capabilities to sub- stantially extend the depth and breadth of tasks they can perform compared to static manipula- tion [1, 2, 3, 4]. For example, in household environments MM agents can perform tasks such as cleaning up the dining table and bringing the dishes to the dishwasher. Data-driven paradigms such as reinforcement learning are attractive for MM since they enable such agents to learn directly from raw sensory observations, that they can function in wide variety of households. However, due to the many possible interactions that are possible for the agent (looking, navigating, and manipu- lating various parts of the household) the state space for MM tasks is vast, which imposes difficult exploration burden for agents that learn autonomously. Learning from human demonstrations has been extremely effective in addressing the burden of exploration in static manipulation settings. Can we apply the same paradigm for mobile manipulation?
Unfortunately, the vast state space in MM has inhibited data collection from humans due to the difficulty of annotating substantial portions of the state space with good actions. Prior work has avoided this issue by instrumenting the environment to ease the state coverage burden [5, 6, 7, 8). However, even if there were a way to collect full human demonstrations without instrumentation, learning from such datasets would also pose challenge due to covariate shift [9], where trained policies visit states not covered by the demonstrations an outcome that is more likely in MM. In this paper, we address the problem of collecting and learning from human demonstrations in mobile manipulation settings via two key solutions: novel system to collect MM demonstrations, and an algorithmic framework that can detect when an agent is suffering from covariate shift.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Collect Human Demonstrations
Policy Learning and Error Detection
Dud Corted
Are Acdess
Mobile Monipulation
E
2 D
Demonstrations
Φ
Action
error
or detected
π Poliey
Error Detector
Figure Leveraging Human Demonstrations Mobile Manipulation. Humans strong intuitions about how to move and interact based on visual data. We leverage their knowledge by first collecting human demonstrations telcoperating mobile manipulator using MOMART (left), and proceed to train imitation learn- ing policies and error detectors from the resulting dataset (middle). During rollouts, the policy executes actions continuously while the error detector simultaneously checks the current visual state can be reconstructed. sufficiently irregular state provokes the error detector intervene, either resetting the agent well-known configuration or immediately terminating to prevent erratic policy behavior (right).
First, we present MOBILE MANIPULATION ROBOTURK (MOMART), novel teleoperation system that allows humans to remotely control mobile manipulators in a natural and easy manner. Operators control the robot's motion using their smartphone provide real-time navigation and manipulation commands simultaneously. Unlike prior works on IL for MM which leverage privileged information during demonstration collection, our system constrains the user observing what the robot sees from its onboard cameras, resulting in more realistic trajectories and demonstrations that require minimal assumptions about the task hand.
Second, recognizing that human data will be insufficient to cover all relevant states, we propose IL method that augments trained policy with learned error detector that can distinguish between in- and -is-tii-iittitttionn states. When an agent encounters states previously unseen during training,
detector can detect fatal errors and immediately stop the execution. In this way, our error detector can constrain the policy to execute only during states similar to what it has seen before, and prevent potentially unstable behavior from occurring during previously unseen states.
We demonstrate the potential of MOMART for mobile manipulation by generating the most general and largest dataset of mobile manipulation demonstrations publicly available: over 1200 demonstra- tions on five long-horizon tasks spanning expert, suboptimal, and few-shot generalization trajecto- ries, totalling over hours of simulation data. We use the dataset train imitation learning policies that can achieve success rates greater than 45% and distribtiitiiittoonnn detectors that achieve over 85% error detection success rate across all tasks.
In summary, our core contributions are as follows:
We present MOMART, a novel teleoperation system that enables intuitive and expressive tele- operation mobile manipulation robots,
We collect first-of-its kind continuous control dataset in a realistic simulated kitchen domain consisting of over 1200 successful demonstrations across five long-horizon tasks with multi-senso modalities and ablation subsets with domain randomization,
We train performant these policies with a learned error detector model that can accurately detect when the agent is in failure state and immediately terminate, achieving over 85% precision and recall.
task policies that reach over 45% success across all tasks, and augment
Related Work
Robotic Teleoperation for Mobile Manipulation: IL leverages human demonstrations to learn tasks such as stationary manipulation [10, 11, 12, 13] and navigation [14, 15, 16, 17]. Having human operators remotely control an agent, or teleoperation, is common approach for collect- ing demonstrations. Teleoperation for MM is not easy to implement, as requires enabling the user to control both navigation and manipulation, possibly simultaneously. Previous work on this

---

problem has explored using online click-through interfaces [18, 19, 20], muscle signals [21, 22], joysticks [23, 24. 25]. tablets [26], and virtual reality interfaces [27, 28, 29, 30, 31. 32, 33, 34]. These approaches are either scalable or easy to use: web-based tools are widely available but are not well suited demonstrate dexterous continuous control, while VR and other interfaces are intuitive to use but are not widely available. The new system we propose (Sec. 3) is both easy to use and scalable: the interface only requires a web browser for viewing and a phone that combines joystick and motion tracking capabilities for remote control of the agent [35].
Autonomous Mobile Manipulation: Algorithms for autonomous MM have been studied for decades [36, 37], and have primarily been addressed with either control [38, 39, 40, 41] or Task and motion planning (TaMP) [3, 42, 43, 44] approaches. Both are are able to generalize across different robots, environments, and robots, but control approaches are generally limited to short horizon tasks, and TaMP approaches depend on human specification of the symbolic actions and full information about the 3D structure of the environment. To address these limitations, learning- based approaches have recently been applied to MM to learn a direct mapping from raw sensory observations to actions [45, 46, 47]. Due to the lack of explicit planning, these works suffer in long horizon tasks involving heterogeneous types of actions such as grasping and moving objects.
Most related to our work, several works have leveraged IL for MM tasks [5, 6, 7, 8]. [5] presented a web-based tool for crowdsourcing a large scale dataset of MM tasks, and used in combina- tion with motion planning for execution on the robot. [6] and [7] collected RGBD observations of humans performing tasks such as door opening and tabletop object manipulation, and used hyper- graph optimization and search procedure respectively to adapt these trajectories to be executable by robot. Lastly, [8] collected VR demonstrations of pick and place actions and extracted a se- quence of symbolic actions and action parametrizations to adopt them for use on robot. While these works demonstrate using IL for complex MM tasks, their approaches are limited to using the demonstrations to parametrize execution of action primitives, and so do not apply to the problem of learning a general visuo-motor policy of arbitrary manipulation actions as we do in this paper.
Error Detection: A common approach to prevent robotic agents from reaching states that harm the environment or themselves is to detect out-of-distribution (OOD) inputs to the agent [48]. Most relevant to our work, various approaches have been proposed for detecting OOD states using deep neural network-based architectures, such as direct training of an error prediction mode [49], un- certainty estimation for the policy [50, 50, 51, 52], or computing a reconstruction error for the in- put [53, 54, 55, 56). In this work we follow the latter approach by training a conditional autoencoder to predict a future goal state given a current state and using its reconstruction error to determine if the robot is in a bad state. While prior works have similarly utilized such errors for collision avoidance, we go beyond them with multi-modal prior that captures better the multiple solutions that humans demonstrated to the same goal, and apply the concept for the first time in a MM setup.
3 Mobile Manipulation RoboTurk (MOMART)
In this section, we present our approach for collecting demonstrations for MM. We first discuss RoboTuur [35], the precursor to our system, and then present our novel MOMART teleoperation system enabling remote and intuitive control of mobile manipulators.
RoboTurk Overview: RoboTuuk [35, 57] is platform that enables remote teleoperation of real or simulated robot arms. An operator connects to a server, receives live video stream with observa- tions from the robot camera on their web-browser, and controls the robot's end-effector by moving a smartphone. The motion of the phone in Cartesian space (6 DoF, position and orientation) is tracked and mapped directly to the robot's 's end-effector motion, and leverages Web Real-Time Communi- cation (WebRTC) to enable real-time control. RoboTurk has been used to collect large datasets on simulated [35] and real arms [57], accelerating IL for robot manipulation research [58, 59]. How- ever, the platform has been limited to stationary arm manipulation.
3.1 MOMART
Due to the limitations of the original RoboTTurk platform, we design MOMART, an extension ofthe original Robo platform that enables intuitive control all the degrees of freedom of mobile manipulator. In MOMART, we assume that the mobile manipulator consists of single arm, a non-
3

---

Figure 2: Simulated Kitchen Tasks, We present 5 challenging long-horizon mobile manipulation tasks re- quiring varied interactions with multiple household objects. Top: Tasks from left right: Set Table from Dishwasher, Set Table from Dresser. Table Cleanup to Dishwasher, Table Cleanup to Sink, Unload Dishwasher Dresser. Bottom: Our tasks evaluate an agent's ability to execute diverse sets of manipulation and navi- gation skills, including accurate arm positioning and contact-rich, arm-base coordinated interaction of large constrained mechanisms. Teleoperators are limited to this ego-centric view, highlighting partial observability.
holonomic base, and a controllable head, such as the Fetch or PAL Tiago robots. We model our sys- tem close to real-robot hardware and plan to extend to the real-world post-pandemic (Sec. A.1
As evidenced by prior MM teleoperation platforms, designing a well-balanced interface can be challenging for multiple reasons: fully exploring MM requires simultaneous control of base and arm motion, but allowing control of all possible degrees of freedom can overload the operator and degrade the quality of the resulting demonstrations. Moreover, unlike static manipulation with global fixed frame of reference, mobile manipulators often only have access to a local frame of reference, modified only by controlling the head. Finally, the long-horizon nature of MM increases the likelihood of demonstrations diverging over time, resulting in sparsely distributed end states.
We showcase our smartphone interface design in Fig. A.I with all of these challenges in mind. Our platform is easy to use, enabling simultaneous 6DOF arm end-effector control through smartphone motion and base locomotion control through the on-screen joystick. To address the head control problem, we fix the head pan (horizontal) joint and allow the tilt (vertical) joint to automatically maintain the end-effector in the central area of the head camera frame. Lastly, recognizing that telcoperated robots can fall into bad configurations, we include a button-triggere generates trajectory move the arm to a pre-defined stable initial joint configuration. The reset helps regularize the arm state distribution observed during teleoperation and increases the consis- tency between demonstrations, and is leveraged in our final learning system with error detection to try and recover from mistakes (Sec. 4.3). Altogether, MOMART enables collecting of teleoper- ated demonstrations from remote locations with an intuitive interface that allows to simultaneous navigation and manipulation in complex environments. Further details can be found in Appendix A.4, and we also conduct user study to qualitatively evaluate our system Appendix A.5.
arm-reset that
3.2 Simulated Kitchen Dataset
To evaluate the teleoperation capabilities of MOMART for MM and generate data for our novel IL with error detection algorithm (Sec. 4), we create five realistic multi-stage simulated household kitchen tasks and collect large-scale multi-user demonstration dataset. We design the simulated tasks in realistic kitchen environment using PyBullet [60] and the iGibson [61, 62] framework with a Fetch [63] robot that must manipulate a bowl. Across all tasks, the robot's initial pose and bowl location is randomized between episodes. An overview of our tasks can be seen in Fig. 2 and are summarized in the following:
The robot must navigate to the table, pick up the bowl with trash in it, navigate to the trashcan, and empty the trash. Then, it must either (a) take the bowl the dishwasher, open the dishwasher and pull out the tray, and place the bowl into the tray, or (b) take the bowl to the sink and drop it in the basin. In addition to the robot and bowl, the trash can's pose is also randomized between episodes. These tasks evaluate an agent's ability to execute diverse range of manipulation and navigation skills, including accurate arm positioning and contact-rich, base coordinated interaction of large constrained mechanisms.
Table Cleanup to Dishwasher / Sink:

---

Table Setup from Dresser / Dishwasher: The robot must first either (a) navigate to the dresser, and search for the bowl by opening each drawer, or (b) navigate and open the dishwasher, and then pull out the tray. Afterwards, in both cases, it must grab the bowl, and navigate to the table and drop off the bowl. In addition to local randomization, the bowl's location also randomized across the drawers between episodes. Searching the dresser evaluates an agent's 's ability to contextualize its observations on prior actions.
The robot must first navigate to the dishwasher and grab the bowl. Then, it must navigate to the dresser, open the top drawer, and place the bowl inside. This task evaluates an agent's 's ability to avoid obstacles based on estimated visual states.
Unload Dishwasher to Dresser:
For each task, an expert and suboptimal operator, as well as additional few-shot generalization subsets consisting of over 20 demonstrations where key task furniture items have changed significantly their locations, producing a dataset that includes over 1200 successful demonstrations totalling over 11 hours of data. This is first of its kind large-scale human demonstration dataset of continuous control collected on realistic long-horizon tasks in the MM setting, and we hope this dataset facilitates future research in IL for MM.
MOMART to collect over 110 successful demonstrations per task from both
4 Learning Mobile Manipulation from Human Demonstrations
In this section, we present our approach for IL for MM. After some preliminaries, we introduce our modified temporal network architecture with more efficient temporal abstraction for IL of MM, and then propose an error detection model that can distinguish between in- and out-of-distribution states alleviate the challenges of the unbounded state space in MM.
4.1 Preliminaries
Partial Observability. We formalize the problem of solving a robot MM task an infinite-horizon discrete-time Partially Observable Markov Decision Process (POMDP). At every step, an agent in state 8, receives an observation o and uses policy choose an action, ar I (o,, 1..). which moves it to state $1+1 according to the state transition distribution.
Imitation Learning: We train risuo-motor continuous control policy for MM with variant of Behavioral Cloning (BC) [64]. The policy maps observations to base and arm actions for the mobile manipulator. BC trains a policy, (o))), from a set demonstrations, DD by minimizing the objective: arg ming ((o.0)~DD (o) - all We base our policy on a variant of BC that leverages temporal abstraction, BC-RNN, in which the policy is parameterized by recurrent neural network (RNN) that is trained on Teeength temporal observation-action sequences to produce an action sequence, ats -- To account for the possibility of multi-modal possibilities in a given state, actions are parameterized by a Gaussian Mixture Model (GMM) distribution [65], de Σx" w;N( az), where actions are sampled from amongst N weighted individual Gaussian distributions.
4.2 TieredRNN
Prior work on learning from human demonstrations in robotic manipulation domains has shown sub- stantial benefits from models leveraging temporal abstraction [58, 66). Inspired by these works, we extend the RNN model from BC-RNN into a multi-layered variant (TieredRNN). This new variant integrates multiple layers operating at varying timesteps to better streamline information flow from timesteps early in a given sequence to timesteps much further downstream, which can be useful for our long-hone horizon MM tasks.
The TieredRNN consists of N individual RNN layers ("tiers"), with corresponding timestep periods TN,, Ti < ... < TN. For given sequence of length T, t 1..,,, 1.....+++ T and corresponding states St St.1...., St4T, layer i updates its hidden state if T; mod t = O and outputs a Af encoding vector z;, which gets passed in addition to 8; to the immediate proceeding i layer. The output of the final layer i = 1 is the overall output of the TieredRNN. In this way, information at varying levels of temporal abstraction can be preserved across many timesteps and better inform an agent of past context. We refer to BC agents using this temporal structure as BC- GCN agents.
5

---

BC-RNN Suboptimal
BC-TieredRNN Suboptimal
BC-RNN Expert
BC-TieredRNN
Task
ReLMoGen
Expert
Table Cleanup to Dishwasher
0.0 + 0.0
16.0 + 5.5
20.8 + 5.0
44.4 + 4.0
47.8 + 7.3
Table Cleanup Sink
0.0 0.0
10.4 + 5.2
8.9 2.9 2.9
54.4 7.1
61.1 + 3.1
Table Setup from Dresser
20.0 + 0.9
17.8 4.8
61.85.55
68.1 3.4
Table Setup from Dishwasher
0.0
63.7 + 7.7
61.1
68.5 + 7.6
66.2 + 2.3
Unload Dishwasher to Dresser
0.0 0.0
11.9 1 4.5
31.1
56.0
47.8 1 8.1
We include
Appendix
Table I: Simulated Kitchen Tasks Results: IL can solve all of our multi-stage tasks, whereas RL (ReLMo- Gen) cannot solve any task and in all tasks is unable to even grasp the bowl as is detailed in Appendix A.6.
When trained expert data, our BC-TieredRNN model can achieve over 45% success rate over all tasks, and outperforms all other baselines majority of the tasks.
4.3
Error Detection and Intervention Functionality
One of the central challenges of IL for MM is the unbounded state space that the robot can explore, since states that sufficiently differ from collected human demonstrations can cause an II policy to quickly degrade. This common covariate shift problem [9] of II. is thus exacerbated by the MM setup. While is difficult to alleviate this problem without collecting additional online samples, we propose a simple but effective method for detecting errors and improving the overall safety of policy execution by leveraging our demonstration data as a strong prior for distinguishing between in- and out-of-distribution states (Fig. 1).
Detecting Errors. Similar to prior work, our error detector leverages a variational autoencoder (VAE) [67] and its reconstruction error states. In our setting, this means that with sufficient training data and capacity, reconstruct similar states to those observed from the demonstrations with low e. We leverage for distinguishing between in- and out-of- distiibriitttonn states: abnormally high e likely correspond to unseen states and can be interpreted failure modes during rollouts. Implementation details can be found in Appendix A.2.
proxy for distinguishing in- and
should learn to
Intervening During Errors. We implement two discrete intervention actions that can be triggered when an error is detected. recover is triggered the first time an error is encountered, and moves the agent to default pose to allow it to re-attempt execution. If the recover action fails bring the eтror level below the threshold, or if errors have been detected within a given rollout, the error detector executes terminate, which immediately terminates the episode. In this way, error is unrecoverabale, the agent can detect and respond to its circumstance (for example, asking human for help). Crucially, because error detector and policy are not provided any additional online data, we do not expect the recover action to consistently succeed. However, we do expect our terminate intervention be consistent and reliable it relies solely on our error detector's ability to consistently detect out-of-distribttin states; this is essential for mitigating potentially harmful policy behavior resulting from covariate shift. Algorithm formalizes our method.
if an
Experimental Evaluation
In this section, we first evaluate our IL algorithm for MM, BC-TieredRNN, and compare to baselines of and RL, BC-RNN, and ReLMoGen [45]. ReLMoGen that leverages discrete actions with a motion planner. Then, we train and evaluate our error detector and recovery for MM, and show that it can accurately detect oft-of-distribution states and either help the agent recover to in-distribution states know when to terminate due to unrecoverable errors. Videos and other results can be seen at https: t/sssssss.e.///e/eeeee tessites.gooo. google. com/view/1l- ooroom/mommee... Specific training and hyperparameter details can be found in Appendix, Sec. A.1 and A.3.
ofuee-of-the-ar RL-based algorithm
5.1 Simulated Results: Solving Lorg-norizon Mobile Manipulation Tasks
For each task, we train each model for 30 epochs and record the average top three best evaluation success rates from 30 episodes aggregated over 3 seeds. For ReG.M, we use a shaped reward for each task and train the agent for 200K time steps. Our results can be seen in Table 1.

---

Imitation Learning Outperforms Reinforcement Learning. We observe that ReLMoGen cannot solve any task despite its lifted action space and shaped reward. This highlights the exploration burden exacerbated in the MM setup, which can cause current state-of-the-ard RI methods to com- pletely flatline. contrast, our methods can perform well and achieve success learning end-to- end visuo-motor policies on these multi-stage MM tasks.
On the expert dataset, we find that our BC-TieredRNN model outperforms the BC-RNN baseline in the tasks where memory is critical: Table Cleanup to Dishwasher and Table Cleanup to Sink, both of which have randomized locations of the trash bin, and Table Setup from Dresser, where the bowl can be located in multiple drawers. In these tasks where the agent must remember where has been in the past, we hypothesize that the TieredRNN is better equipped to allow long-term information flow through its skip- connecctnn architecture. In contrast, in the tasks that do not require randomization (Table Setup from Dishwasher and Unload Dishwasher to Dresser), the Ni provides no benefit and is instead outperformed by BC-RNN. We further contextualize our usage of the TieredRNN model in Appendix A.7.
Demonstration Quality Dictates Success. With the exception of Table Setup from Dishwasher, observe on average over 200% increase in success rate when training on the same number of expert demonstrations compared to suboptimal demonstrations. In the latter case, the operators were new to using the MOMART system, whereas the expert operator had prior experience. This both showcases the ability of our platform to produce impressive results on multiple MM tasks given experience, and also highlights the importance of the teleoperation interface for collecting high-quality demonstrations that better facilitate learning.
5.2 Simulated Results: Detecting Errors During Rollout
For each task, we train our error detector and set a uniform error threshold ab 0.05 based on analysis from small number of expert rollouts. Because the BC-TieredRNN model performed the best on the majority of tasks, we utilize this model for error detection evaluation. We evaluate our error detector's performance when paired with the trained policy model, and record the precision -inngy Because we are in simulation, we can deterministically
Myous
and recall "goa,
"ppoz +
evaluate the counterfactual between the model augmented with the error detector and the same model without. A true / false positive occurs when the error detector detects an error when the original model fails/ / succeeds, respectively. Likewise, true false negative occurs when the error detector does not trigger during a successful /failed rollout, respectively. Metrics are aggregated over the 3 policy seeds each evaluated on 30 rollouts with mean and standard deviations shown in Table 2.
Errors are Reliably and Robustly Detected. We find that our error detectors can consistently detect errors and achieve over 85% precision and recall rates across all tasks when trained on the expert dataset. The high precision means that our error detectors can accurately distinguish between error and non-error states, and the high recall means that it can also reliably detect true errors when they occur. This is especially important in MM domains where learned agents can casily diverge into unseen states during rollouts and become erratic if left unchecked.
While the error detectors occasionally misfire, and causes a marginal decrease in success, we observe that the success rates remains generally consistent. This highlights that our error detector is not overly conservative and does not prematurely terminate episodes when there is no true error at hand. Interestingly, in some cases, such as in Table Setup from Dresser, the recover action of our error detector is able to result in marginal policy improvement. While unexpected, this suggests that similar methods bringing robot back into well-known states may provide benefits for MM agents.
Data Diversity Impacts Error Detection When trained on suboptimal data, our error detectors surprisingly achieve over 90% precision across all tasks. However, the recall rate substantially suffers. We postulate that this is due to the noisy nature of the suboptimal data encompassing a wider distribution of states, augmenting the resulting error detectors' abilities to reject non-error states while making it more difficult to consistently detect true error states.
Our Error Detector is Interpretable and Easily Tuned. While we deployed all error detectors with the same uniform error threshold value e, each error detector can be individually tuned in an interpretable way. For example, lowering increases the error detector's sensitivity to observation irregularities, thereby improving the likelihood of detecting true errors (increased recall), at the potential cost of additional spurious error triggers (decreased precision). This can be useful in
7

---

SR
SR
ED
ED
Dataset
Task
(no ED)
(ED)
Precision
Recall
Table Cleanup to Dishwasher
34.4 10.3
32.2
96.4
1.0
Table Cleanup Sink
4.2
51.1 4.2
9.9
86.5
Expert
Table Setup from Dresser
63.3
6
1.0
85.9 10.0
Table Setup from Dishwasher
6.81
47.8 11.0
88.2 10.2
100.0 0.0
Unload Dishwasher to Dresser
$6.7--10.9
33.3
85.8
100.0
Table Cleanup Dishwasher
4.77
2.3
93.4+533
76.9 18.8
Table Cleanup Sink
1.1
1.1+1.5
9.24+
66.3+7.1
Suboptimal
Table Setup from Dresser
6.3
15.6
97.6
58.7
Table Setup from Dishwasher
4.2
48.9
9..0
9..2
Unload Dishwasher Dresser
26.7
2.7
9.9
41.2 14.1
Table Cleanup to Dishwasher
4.7
2.7
9.92
9.7
Table Cleanup to Sink
2.8
1..1
3.7
98.7 1.9
Few-Shot Generalize
Table Setup from Dresser
3.8
8.7
2..
64. 6.7
Table Setup from Dishwasher
40.0 + 16.3
24.4 12.9
78.2
100.0 + 0.0
Unload Dishwasher to Dresser
1.2:1.6
2.2 1.6
1.6
100.0 + 0.0
Table 2: Error Detection Results When trained on expert data, our error detectors can accurately detect errors, achieving over 85% precision and recall across all tasks. With suhoptimal data, the error detectors similarly reject non-error states consistently but struggle to detect true error states. Lastly, the eroor detectors can adapt the few-shot generalization setting, maintaining high precision and recall despite the generalized policy's 's significant deterioration.
situations where safety is critical and the additional confidence in reliable error detection is worth the cost of lowered success rate. Further results comparing our method against potential alternatives can be seen in Appendix A.8.
5.3 Simulated Results: Few-Shot Generalization
Lastly, we evaluate the generalizability of our policy and error detector by taking the models trained on the expert data and finetuning them the few-shot demonstrations exhibiting major distribu- tion shift, where the critical furniture objects (dishwasher, dresser, and sink) have swapped places. Similar to Sec.5.2, evaluate the success with and without the error detector, and also report the corresponding detection metrics, shown at the bottom of Table 2.
Imitation Learning Has Potential to Generalize With Little Additional Data. While the success rates drop significantly compared to the original baselines, we find that our model can still solve the task given only a limited number of successful demonstrations in this generalized setting. These results show promise for IL to lower the data burden for learning general visuomotor policies in kitchen environment, where the discrete number of unique object in-
similar MM settings stances finite but the combinatorial possibilities are huge. Appendix A.9 highlights the benefits of finetuning versus training solely the few-shot demonstrations.
Error Detection Can Adapt to Distribution Shifts. Even in this generalized setting, our error detector still performs well, and often achieves more than 80% precision and 90% recall across all tasks. This shows how our detector, once initially trained, can quickly adapt its modeled state distribution to account for new training data and still accurately detect true error states while ignoring non-error states.
Conclusion
We presented three contributions for IL in MM setups. First, introduced MoMART, novel teleoperation platform for MM. Second, we used MOMART collect a first of its kind large-scale MM dataset of continuous control. And third, presented variants of II. and error detection for the MM setup that train performant visuo-motor policies and accurately detect errors. Our next step after the pandemic is to bring our teleoperation interface control real mobile manipulator (Sec. A. 10). We hope our diverse dataset can provide researchers the accessible means to investigate many other important problems in MM from an IL perspective.

---

Acknowledgments
We would like to thank Jiangshan Li for her help in setting up the initial RoboTurk prototype.
References
[1] B. Siciliano and O. Khatib. Springer handbook of robotics. springer, 2016.
[2] O. Khatib. Mobile manipulation: The robotic assistant. Robotics and Autonomous Systems, 26 (2-3):155-183, 1999.
[3] J. Wolfe, B. Marthi, and S. Russell. Combined task and motion planning for mobile manipu- lation. In Twentieth International Conference on Automated Planning and Scheduling, 2010.
[4] S. Tellex, T. Kollar, S. Dickerson, M. Walter, A. Banerjee, S. Teller, and N. Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, 2011.
[5] E. Ratner, B. Cohen, M. Phillips, and M. Likhachev. A web-based infrastructure for recording user demonstrations mobile manipulation tasks. In 2015 IEEE International Conference on Robotics and Aulomation (ICRA), pages 5523-5530, 2015. doi: (000//CRA.211.7139999
[6] T. Welschehold, C. Domnhege and W. Burgard. Learning mobile manipulation actions from human demonstrations. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3196-3201. IEEE, 2017.
[7] T. Welschehold, N. Abdo, C. Dornhege, and W. Burgard. Combined task and action learning from human demonstrations for mobile manipulation applications. In 2019 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS), pages 4317-4324. IEEE, 2019.
[8] G. Kazhoyan, A. Hawkin, S. Koralewski, A. Haidu, and M. Beetz. Learning motion parame- terizations of mobile pick and place actions from observing humans in virtual environments. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9736 9743. IEEE, 2020.
[9] S. Ross, G. Gordon, and D. Bagnell. A reduction of imitation learning and structured predic- tion to no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Pro- ceedings, 2011.
[10] S. Schaal. Is imitation learning the route to humanoid robots? Trends in Cognitive Sciences, 3:233-22 242,, 1999.
[11] A. Billard, S. Calinon, R. Dillmann, and S. Schaal. Robot programming by demonstration. In Springer Handbook of Robotics, 2008
[12] S. Calinon, F. D'halluin, E. L. Sauser, D. G. Caldwell, and A. Billard. Learning and reproduc- tion of gestures by imitation. IEEE Robotics and Automation Magazine, 17:44 54, 2010.
[13] B. D. Argall, S. Chernova, M. Veloso, and B. Browning. A survey of robot learning from demonstration. Robotics and autonomous systems, (155)469-4833 2009.
[14] A. Hussein, E. Flyan, M. M. Gaber, and C. Jayne. Deep imitation learning for 3d navigation tasks. Neural computing and applications, 99(7):389-440) 2018.
[15] P. M. Kebria, A. Khosravi, I. Hossain, N. Mohajer, H. D. Kabir, S. M. J. Jalali, D. Nahavandi, S. M. Salaken, S. Nahavandi, A. Lagrandcourt, et al. Autonomous navigation via deep imita- tion and transfer learning: A comparative study. In 2020 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2907-2912. IEEE, 2020.
[16] J. Xu, Q. Liu, H. Guo, A. Kageza, S. AlQami, and S. Wu. Shared multi-task imitation learning for indoor self-navigation. In 2018 IEEE Global Communications Conference (GLOBECOM)), pages 1-7. IEEE, 2018.

---

[17] H. Karnan, G. Warnell, X. Xiao, and P. Stone. Voila: Visual-observation-ooe only imitation learn- ing for autonomous navigation. arxiv preprint wxXx:22000.0337,, 2021.
[18] D. Wang. J. Yi, D. Zhao, and G. Yang. Telcoperation system of the internet-based omni- directional mobile robot with a mounted manipulator. In 2007 International Conference on Mechatronics and Automation, pages 1799-1806 IEEE, 2007.
[19] V. Annem, P. Rajendran, S. Thakar, and S. K. Gupta. Towards remote teleoperation of a semi- autonomous mobile manipulator system in machine tending tasks. In International Manufac- turing Science and Engineering Conference, volume 58745, page V001T02A027. American Society of Mechanical Engineers, 2019.
[20] J. Yan, J. Zhao, C. Yang, Y. Zhu, L. Ma, and Z. Chen. Design of a coordinated control strategy for cilltii-obbll-mmiiplaaaee cooperative teleoperation system. 2012 IEEE International Conference Mechatronics and Automation, pages 783-788. IEEE, 2012.
[21] B. Wang. Z. Li, W. Ye, and Q. Xie. Development of human comman-macee machine interface for teleoperation of a mobile manipulator. International Journal of Control, Automation and Systems, 10(6): 1225-1231, 2012.
[22] M. Zucker, S. Joo, M. X. Grey, C. Rasmussen, E. Huang, M. Stilman, and A. Bobick. A general-purpose system for telcoperation of the dre-hubo humanoid robot. Journal of Field Robotics, (2(3))336-35,, 2015.
[23] D. Ryu, J.-B. Song, C. Cho. S. Kang, and M. Kim. Development of six dof haptic master for telcoperation of mobile manipulator. Mechatronics, 20(2): 2010.
[24] M. Stilman, K. Nishiwaki, and S. Kagami. Humanoid teleoperation for whole body manipula- tion. In 2008 IEEE International Conference on Robotics and Automation, pages 3175-3 1775-1186 180. IEEE, 2008.
[25] C.-H. King, T. L. Chen, Z. Fan, D. Glass, and C. C. Kemp. Dusty: an assistive mobile manipulator that retrieves dropped objects for people with motor impairments. Disability and Rehabilitation: Assistive Technology, 7(2):168-177) 179, 2012.
[26] D. A. Lopez, J. A. Frank, and V. Kapila. Comparing interface elements tablet for in- tuitive telcoperation of a mobile manipulator. In ASME 2014 12th Biennial Conference on Engineering Systems Design and Analysis. American Society of Mechanical Engineers Digital Collection, 2014.
[27] A. U. Krishnan. Nursing Robot Teleoperation via Motion Mapping Interfaces. PhD thesis, WORCESTER POLYTECHNIC INSTITUTE, 2020.
[28] C. Gan, J. Schwartz, S. Alter, M. Schrimpf, J. Traer, J. De Freitas, Kubilius, A. Bhandwal- dar, N. Haber, M. Sano, et al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint arXiv:2007.04954 2020.
[29] A. Garcia-Gareia, Martinez-Gonzalez, S. Oprea, J. A. Castro-Vargas, Orts-Escollano J. Gardia-Rodriguez, and A. Jover-Alvarez. The robotrix: An extremely photorealistic and very-large-scale indoor dataset of sequences with robot trajectories and interactions. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6790- 6797. IEEE, 2018
[30] P. Martinez-Gonzalez, S. Oprea, A. Garcia-Gareia, A. Jover-Alvarez, S. Orts-Escollano and Garcia-Rodie Rodriguz. Unrealrox: An extremely photorealistic virtual reality environment for robotics simulations and synthetic data generation. arXiv preprint arXiv:1810.06936 2018.
[31] F. Stramandinoli. Robot learning from human demonstration in virtual reality, 03 2018.
[32] X. Gao, R. Gong, T. Shu, X. Xie, S. Wang, and S.-C. Zhu. Vrkitchen: an interactive 3d virtual environment for task-oriented learning. arXiv preprint a7x.v:1903.05557, 2019.
10

---

[33] J. DelPrrto, J. I. Lipton, L. Sanneman, A. J. Fay, C. Fourie, C. Choi, and D. Rus. Helping robots learn: A buman--obot master-apprentice model using demonstrations via virtual reality teleoperation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10226-102333 IEEE, 2020.
[34] T. Zhang. Z. McCarthy, O. Jow, D. Lee, K. Goldberg, and P. Abbeel. Deep imitation learning for complex manipulation tasks from virtual reality teleoperation. arXiv: (0t0.0/7100006) 2017.
arXiv preprint
[35] A. Mandlekar, Y. Zhu, A. Garg, Booher, M. Spero, A. Tung, Gao, Emmons, A. Gupta, E. Orbay, et al. Roboturk: A crowdsourcing platform for robotic skill learning through imita- tion. In Conference on Robot Learning, pages 879-893, PMLR, 2018.
[36] W. F. Carriker, P. K. Khosla, and B. Krogh. Path planning for mobile manipulators for multiple task execution. IEEE Transactions Robotics and Automation, 7(3):403-408, 1991.
[37] K. Nagatani and S. Yuta. Designing strategy and implementation of mobile manipulator control system for opening door. In Proceedings of IEFE International Conference on Robotics and Automation, volume 3. pages 2828-2834. IEEE, 1996.
[38] M. V. Minniti, F. Farshidian, R. Grandia, and M. Hutter. Whole-body mpe for a dynamically stable mobile manipulator. IEEE Robotics and Automation Letters, 444):3687)3694, 2019.
[39] Pankert and M. Hutter. Perceptive model predictive control for continuous mobile manipu- lation. IEEE Robotics and Automation Letters, 5(4):6177-61844 2020.
[40] D. Honerkamp. T. Welschehold, and A. Valada. Learning kinematic feasibility for mobile manipulation through deep reinforcement learning. arXiv preprint arXiv:2101.0552 2021.
[41] M. Mittal, D. Hoeller, F. Farshidian, M. Hutter, and A. Garg. Articulated object interaction in unknown scenes with whole-body mobile manipulation, 2021.
[42] C. R. Garrett, T. Lozano-Perez, and L. P. Kaelbling. Ffrob: Leveraging symbolic planning for efficient task and motion planning. The International Journal of Robotics Research, 37(1): 104-136, 2018.
[43] S. Cambon, F. Gravot, and R. Alami. A robot task planner that merges symbolic and geometric reasoning. In ECAL, volume 16, page 895, 2004.
[44] M. Stilman, J.-U. Schamburek, J. Kuffner, and T. Asfour. Manipulation planning among mov- able obstacles. In Proceedings 2007 IEFE international conference on robotics and automa- tion, pages 3327-3332. IEEE, 2007.
[45] Xia, C. Li, R. Martin-Martin, O. Litany, A. Toshev, and Savarese. Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation, 2021.
[46] C. Li, F. Xia, R. Martún, and S. Savarese. Hrl4in: Hierarchical reinforcement learning for interactive navigation with mobile manipulators. In Conference on Robot Learning, pages 603-616. PMLR, 2020.
[47] C. Wang, Q. Zhang, Q. Tian, S. Li, X. Wang, D. Lane, Y. Petillot, and S. Wang. Learning mobile manipulation through deep reinforcement learning. Sensors, 20(3):939, 2020.
[48] R. Chalapathy and Chawla. Deep learning for anomaly detection: A survey, arXiv preprint arXiv: 1901.03407 2019.
[49] S. Hecker, D. Dai, and L. Van Gool. Failure prediction for autonomous driving. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1792-1799. IEEE, 2018
[50] G. Kahn, A. Villaflor, V. Pong, P. Abbeel, and S. Levine. Uncertainty-awart reinforcement learning for collision avoidance. arXiv preprint arXiv:1702.01188. 2017.
[51] A. Filos, P. Tigas, R. McAllister, N. Rhinehart, S. Levine, and Y. Gal. Can autonomous vehicles identify, recover from, and adapt to distribution shifts? arXiv preprint arXiv:2006. 149/1, 2020.
11

---

[52] K. Menda, K. Triggs-Campbely and M. J. Kochenderfer. Ensembledagger: A bayesian ap- proach safe imitation learning. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5041-5048. IEEE, 2019.
[53] D. A. Pomerleau. Input reconstruction reliability estimation. In Neural Network Perception for Mobile Robot Guidance, pages 133-150. Springer, 1993.
[54] C. Richter and N. Roy. Safe visual navigation via deep learning and novelty detection. In Robotics: Science and Systems, 2017.
[55] N. Hirose, A. Sadeghian, M. Vázquez, P. Goebel, and S. Savarese. Gonet: semi-supervised deep learning approach for traversability estimation. In 2018 IEEE/RSJ International Confer- ence on Intelligent Robots and Systems (IROS), pages 3044-3051. IEEE, 2018
[56] M. Lee, M. Tan, Zhu, and J. Bohg. Detect, reject, correct: Crossmodal compensation of corrupted sensors. arXiv preprint arXiv:2012.00201 2020.
[57] A. Mandlekar, J. Booher, M. Spero, A. Tung, A. Gupta, Y. Zhu, A. Garg, S. Savarese, and
L. Fei-Fei. Scaling robot supervision to hundreds of hours with roboturk: Robotic manipulation dataset through human reasoning and dexterity. arXiv preprint
2019.
[58] A. Mandlekar, D. Xu, R. Martin-Martin, S. Savarese, and L. Fei-Fei. Learning general- ize across long-horizon tasks from human demonstrations. arXiv preprini arXiv:2003. 06085, 2020.
[59] A. Tung. J. Wong, A. Mandlekar, R. Martin-Martin, Y. Zhu, L. Fei-Fei, S. Savarese, D. Xu, R. Martin-Martin A. Kurenkov, et al. Learning multi-arm manipulation through collaborative teleoperation. IEEE International Conference on Robotics and Automation (ICRA), 2020.
[60] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning.
org, 2016-2021.
[61] F. Xia, W. B. Shen, C. Li, P. Kasimbeg, M. E. Tchapmi, A. Toshev, R. S. Savarese. Interactive gibson benchmark: A benchmark for interactive navigation in cluttered environments. IEEE Robotics and Automation Letters, 5(2):713-720, 2020.
and
[62] B. Shen, F. Xia, C. Li, R. Martin-Martin, L. Fan, G. Wang, S. Buch, C. D'Arpino, S. Sri- vastava, L. P. Tchapmi, et al. igibson, a simulation environment for interactive tasks in large realisticscenes. arXiv preprint arXiv:2012.02924 2020.
[63] M. Wise, M. Ferguson, D. King, E. Diehr, and D. Dymesich. Fetch & freight Standard platforms for service robot applications. 2016.
[64] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in neural information processing systems, pages 305-313. 1989.
[65] D. Reynolds. Gaussian Mixture Models, pages 659-663. Springer US, Boston, MA, 2009. ISBN 978-0-387-73005-4 10. 1007000////0
5_19.. URL https://doi. org/
[66] A. Tung, J. Wong, A. Mandlekar, R. Martin-Martin, Y. Zhu, L. Fei-Fei, and S. Savarese. Learn- ing multi-arm manipulation through collaborative teleoperation, 2020.
[67] D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 13/2.111 14, 2013.
12


Reviews:
[{"review_id": "uH1Mk-6SUsh", "reviewer": "Reviewer_56kE", "summary": "This paper describes a new method for collecting demonstrations for long-horizon tasks in mobile manipulation. The approach manages the gap between demonstrator observations and the systems observations. The authors use this to collect a large dataset of long-horizon demonstrations and achieve state-of-the-art results by imitating them. The authors also propose an error detection method to deal with the potentially large shift in states observed between the expert demonstrations and the learned policy. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n--------------\n\nThe paper is well-written and gives a good characterization of related work. \n\nThe demonstration collection method looks novel, non-trivial, and effective. The dataset collected looks like it could be quite useful to the community. \n\nThe policies the authors train go beyond the state-of-the-art in policy learning for long-horizon tasks. This is an important area for the community and the results are impressive. \n\nWeaknesses\n-------------------\n\nWhile the error detection method looks to be useful and clearly helps with imitating the data, the error detection method itself is not really evaluated. In particular, the authors could evaluate their design choices for the error detection. This is important if the authors wish to claim the error detection method as a contribution.\n\nI found the discussion of the few-shot generalization results to be too short. I recommend the authors expand this section of the text (l.307-322). In particular, including some details about the drop in performance to reduce the burden on the reader.\n\nMinor Comments\n------------------------\n[l.173] \"110 successful demonstration per from\" -- missing word\n[l.188] in a POMDP, $\\pi$ is a function of the full history, not just the last observation\n[l.240] \"Algorithm 1 formalizes this method\" --- the paper does not appear to have an algorithm 1\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "As my review indicates, I'm a big fan of this paper. It makes an important step forward and contributes resources that will help the community progress. I will advocate for the paper's acceptance, if only selfishly so that I can build on the dataset and tools they have provided. \n\n\nUpdate after rebuttal\n----------------------------\nI thank the authors for addressing my comments and I am happy they have included more discussion of the error detection method they propose. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Excellent: Expert knowledge on the topic of the paper"}, "extra_text": {"issues": "In my read through the paper, I did not see discussion of publishing the dataset of demonstrations. Can the authors please clarify how they will share this resource (or point me to the place in the paper where I have missed it)?\n\nPlease address the minor comments.\n\nPlease provide more analysis of the few-shot results. \n\nConsider adding evaluations of the design choices in the error detection method."}}, {"review_id": "tloJjQ2o4Xa", "reviewer": "Reviewer_dB9j", "summary": "The paper presents an innovative imitation learning method with error detection in the context of mobile manipulation. It proposes a novel architecture, TieredRNN, which is a multi-layered variant of BC-RNN. It is able to perform significantly better than the current SOTA methods for a series of real-world mobile manipulation tasks. On top of that, it presents a way to detect out-of-distribution (OOD) inputs using VAE reconstruction error and perform intervening error corrections for the agents.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- It is quite impressive that BC based methods perform much better than RL based baselines in a challenging domain like mobile manipulation. It shows the effectiveness of BC + RNN in long sequential tasks and gives room for future exploration into this line of work.\n\n- Successfully evaluated the effect of suboptimal demonstration data\n\n- Presented a novel error detection methods based on VAE construction error\n\nWeaknesses:\n\n- On choice of model architectures: from the experimental results, it is not entirely clear to me how TieredRNN perform better than BC-RNN, and it seems that BC-RNN (which I assume to be a simpler structure) should already give quite impressive results. Is there any reason that the authors choose the more complex model structure of TieredRNN over simplicity? It would be better if there could be more explanations in the paper.\n\n- On experimental results: According to Table 2, the error detection results are quite impressive, but they do not help success rate that much. In fact in many sections, the success rate decreases because of the error detection. How might the authors explain the lack of improvement over task success rates?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is a novel imitation learning method that is applied to mobile manipulation which comes with effective error detection mechanisms for out of distribution inputs. Future improvements could be made from clearer explanations of the significance of experimental results, choice of model architectures, and more insights into error correction strategies.  ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Related to the Weaknesses section above:\n\n- More detailed explanation and justification of presenting the TieredRNN methods\n\n- More justified explanation on how error correction relates to success rate\n\nOn error correction strategies: \n\n- The current interventions when an error is detected are hardcoded programming if I interpret it correctly - only \"recover\" and \"terminate\". I'm curious of whether the authors have considered more versatile intervention methods, e.g. human monitoring and tele-operated corrections?"}}, {"review_id": "hXGoxLOJ5Kj", "reviewer": "Reviewer_3mTs", "summary": "This paper proposed imitation learning (behavioral cloning (BC) more specifically) with an error detector by variational autoencoder (VAE).\nIn order to automate mobile manipulation, the authors first developed a teleoperation framework, MOMART, for collecting huge dataset, which is utilized for imitation learning.\nSince mobile manipulation is a long-term operation, the effect of covariate shift, which is a shortcoming of BC, becomes more pronounced.\nTo prevent malfunctions caused by this, the authors additionally implemented the error detection mechanism based on the recovery accuracy of VAE.\nAlthough it is unclear how high the success rate should be, the authors showed a relatively high percentage of error detection.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors collected a huge dataset of mobile manipulation that had not yet been constructed, although it was simulated.\n- Focusing on that the observation of the dataset is from POMDP, the authors combined BC with TieredRNN to use the observation history appropriately, which makes the task successful with some probability.\n- The VAE-based error detection mechanism achieved the high error detection rate even in the generalized setting.\n\nWeaknesses:\n- The newly introduced error detection mechanism has not improved the success rate of the task itself.\n- Its implementation is also naive, and there is room for improvement.\n- The authors have only verified the developed framework, and have not sufficiently examined and compared other possible implementations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "There is not much novelty in the technology, and it is not a good proposal because various patterns of error detection by VAE have been proposed so far.\nOn the other hand, the construction of a large dataset of long-horizon mobile manipulation tasks, which is closer to real-world problems, is very useful, and I commend the authors for this.\nIf qualitative or experimental comparisons on error detection are added, the paper seems to be worthy enough to be accepted.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- I think the error detection based on the reconstruction accuracy has a disadvantage that the threshold setting can be complicated depending on the observation information.\nThere are several existing methods based on latent variables, and I would like to see a comparison with such methods.\n- The benefits of error detection to the task accomplishment were not sufficiently demonstrated. \nI would like to see a comparison for indicators other than success rate (such as the load caused by the misoperation).\n- The latent prior distribution and the output distribution of BC are assumed to be mixture distributions, but how do the authors give the number of mixtures? Isn't it task-dependent?\n- What kind of observations are included in the constructed dataset?"}}, {"review_id": "-Ga0DpSsLkA", "reviewer": "Reviewer_KoyB", "summary": "The paper addresses learning mobile manipulation tasks by 1) building a smartphone teleoperation interface that enables human operators to demonstrate mobile manipulation tasks in simulation, 2) training a visuomotor policy on such collected demonstrations, and 3) training an error-detection classifier to detect when the learned policy had incurred an unseen state and either attempts to recover or terminates task execution. Experiments were done in a simulated kitchen domain, where a fetch robot was tasked to transport objects by interacting with a variety of articulated objects. A total of 1200 demonstrations were collected, and the learned policies achieved task success rate of 45%.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n-\tThe paper’s motivation is well founded – learning long-horizon mobile manipulation tasks could benefit from more robust human priors.\n-\tThe fact that the demonstrator is not using privileged information is a plus – both the demonstrator and the robot is given the same single-camera view.\n-\tAn easy to use interface for providing mobile manipulation demonstration data is clearly beneficial for the field.\n-\tThe paper is clearly written.\n\nWeaknesses:\n-\tMy main criticism is the paper’s seeming lack of focus. While it builds a system that can collect mobile manipulation demonstrations and train imitation learning policies, it does not study these components in depth. \n-\tOn the teleoperation interface, a more thorough user study would have made the paper more convincing. There are many questions such a study can help answer. What is special about giving demonstrations for mobile manipulation as compared to stationary demonstration? Were task strategies pre-determined or did each demonstrator come up with their own? How varied were the demonstrations, and how noisy were the collected trajectories? Were any data cleaning and filtering done to preprocess demonstration data? What’s the success rate of human demonstrators? Could the camera be moved independently of the robot? How many trials did it take for a new human operator to achieve task success? \n-\tOn imitation learning, while the use of RNN seems appropriate given the partial observability aspect of single-view camera inputs, it would still be helpful to run a baseline comparison against a non-RNN version with a window of observations as input. The claim that the Tiered-RNN architecture performs better than the generic RNN architecture is also not obvious, at least not from Table 2.\n-\tOn covariate shift - the use of error detection to address covariate shift is not entirely satisfying. There are many well established and simple-to-implement methods for reducing covariate shift, from the DAgger algorithm to noise injection during demonstrations, and it is unclear why the authors did not experiment with them. One potential justification may be that the paper is not completely focused on robust imitation learning, but that further highlights the lack of focus in the paper\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I recommend weak reject. In my opinion, the only novel part about the paper is its mobile manipulation teleoperation interface (and perhaps the 1200 collected demonstrations), but the paper does not perform thorough experiments and analyses of this interface. The results from the trained manipulation policies, while interesting, do not bring a new understanding of the problem.\n\n**Update** After the author's response with additional clarifications and the new user study, which helped to address my original concerns and better contextualize the paper, I am changing my recommendation to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please see Strength and Weaknesses for revision suggestions. The website link was not operational at the time of review.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: **After rebuttal**
I thank the authors for their thorough responses, my concerns have been addressed, and the reviewers concerns seems to have been addressed to the reviewers satisfaction.
Thus I recommend accept.

**Initial Meta-review**
**Summary**

This work presents a teleoperation system for long horizon tasks performed by mobile manipulation robots. This teleoperation framework is utilized to collect a continuous control dataset in a simulated kitchen domain consisting of over 1200 successful demonstrations across five long-horizon tasks.  The author then present initial imitation learning results on this dataset. The imitation learning process is augmented with a learned error detector which can detect when the agent is in a failure state.

**Strengths**:
This work contributes a system for mobile manipulation data collection, which is an important and active research are in the robotics community

The manuscript is well written and the work is well motivated 

The work introduces a variant of behavior cloning that leverages hierarchical abstractions, and show that imitation learning with this variant outperforms some relevant baselines 

**Weaknesses**:

This work also introduces an error detection method, however it is barely evaluated. 
All reviewers are missing some more detailed analysis and discussion on various aspects of the evaluation
Decision: Accept (Poster)