Scaling Up Multi-Task Robotic Reinforcement Learning
Authors: Dmitry Kalashnikov
Jake Varley
Yevgen Chebotar
Benjamin Swanson
Rico Jonschkowski
Chelsea Finn
Sergey Levine
Karol Hausman
Venue: CoRL2021 Poster (2021)
Keywords: Multi-Task Reinforcement Learning
Robot Learning

Abstract: General-purpose robotic systems must master a large repertoire of diverse skills. While reinforcement learning provides a powerful framework for acquiring individual behaviors, the time needed to acquire each skill makes the prospect of a generalist robot trained with RL daunting. In this paper, we study how a large-scale collective robotic learning system can acquire a repertoire of behaviors simultaneously, sharing exploration, experience, and representations across tasks. In this framework, new tasks can be continuously instantiated from previously learned tasks improving overall performance and capabilities of the system. To instantiate this system, we develop a scalable and intuitive framework for specifying new tasks through user-provided examples of desired outcomes, devise a multi-robot collective learning system for data collection that simultaneously collects experience for multiple tasks, and develop a scalable and generalizable multi-task deep reinforcement learning method, which we call MT-Opt. We demonstrate how MT-Opt can learn a wide range of skills, including semantic picking (i.e., picking an object from a particular category), placing into various fixtures (e.g., placing a food item onto a plate), covering, aligning, and rearranging.  We train and evaluate our system on a set of 12 real-world tasks with data collected from 7 robots, and demonstrate the performance of our system both in terms of its ability to generalize to structurally similar new tasks, and acquire distinct new tasks more quickly by leveraging past experience.  We recommend viewing the videos at https://karolhausman.github.io/mt-opt/.

Scaling Up Multi-Task Robotic Reinforcement Learning
Dmitry Kalashnikoy
Jake Varley"
Yevgen Chebotar
Benjamin Swanson
Rico Jonschkowskil
Chelsea Finn!
Sergey Levine
Karol Hausman"
General-purpose robotic systems must master large repertoire of di- verse skills. While reinforcement learning provides a powerful framework for acquiring individual behaviors, the time needed to acquire each skill makes the prospect of generalist robot trained with RL daunting. We study how a large- scale collective robotic learning system can acquire repertoire of behaviors si- multaneously, sharing exploration, experience, and representations across tasks. In this framework, new tasks be continuously instantiated from previously learned tasks improving overall performance and capabilities of the system. To this end, we develop scalable and intuitive framework for specifying new tasks through user-provided examples of desired outcomes, devise a multi-robot col- lective learning system for data collection that simultaneously collects experience for multiple tasks, and develop scalable and generalizable multi-tast task deep rein- forcement learning method, which we call MT-Opt. We demonstrate how MT--ppt can learn a wide range of skills, including semantic picking (i.e., picking an object from a particular category), placing into various fixtures (e.g.. placing a food item onto a plate). We train and evaluate our system on a set of 12 real-world tasks with data collected from robots, and demonstrate the performance of our system both in terms of its ability to generalize to structurally similar new tasks, and acquire distinct new tasks more quickly by leveraging past experience. recommend viewing the videos at https: http///aklllaaaaaa gi thub. io/nt-opt/
Abstract:
Keywords: Multi-Task Reinforcement Learning, Robot Learning
Introduction
Today's deep reinforcement learning (RL) methods, when applied to real-world robotic tasks, pro- vide an effective but expensive way learning skills [1. 2]. While existing methods are effective and able to generalize, they require considerable on-robot training time. For example, the QT-Opt [1] system can learn rision-based robotic grasping, but it requires over 500, 000 trials collected across multiple robots. While such sample complexity may be reasonable if the robot needs to perform a single task, it becomes costly if we consider the prospect of training a general-purpose robot with a large repertoire of behaviors, where each behavior is learned in isolation, starting from scratch. Can we instead amortize the cost of learning this repertoire over multiple skills, where the effort needed to learn whole repertoire is reduced compared to learning each skill in isolation?
Prior work suggests that multi-task RL can amortize the cost of single-task learning [3, 4, 5, 6, 7]. Insofar as the tasks share common structure, if that structure is discovered by the learning algorithm, all of the tasks can in principle be learned more efficiently than learning each task individually. In addition. by collecting experience simultancously using controllers for variety of tasks with different difficulty, the easier tasks can "bootstrap" the harder tasks. Finally, by enabling the multi- task RL policy to learn shared representations, learning new tasks can become easier over time as the system acquires more skills and learns more widely-useful aspects of the environment.
However, to realize these benefits for a real-world robotic learning system, we need to overcome number of major challenges [8, 9, 10, 11], which have far made difficult to produce large- scale demonstration of multi-task image-based RL in the real world. First, multi-task reinforcement
Equal contribution
Google Research, Robotics Google Team
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

learning is known to be exceedingly difficult from the optimization standpoint [8, 12]. Second, real-world multi-task learning framework requires the ability to easily and intuitively define rewards for a large number of tasks. Third, while all task-specific data could be shared between all the tasks, has been shown that reusing data from non-correlated tasks can be harmful to the learning process [13]. Lastly, to receive the benefits from shared, multi-task representation, we need to significantly scale up our algorithms, the number of tasks, and the robotic systems themselves. The main contribution of this paper is a general multi-task learning system, which we call MT--ppt. MT-Opt realizes the hypothesized benefits of multi- task RL in a scalable real-world robotic learning sys- tem, addressing the associated challenges through a number of important design decisions, including a carefully designed multi-task data sharing strat- egy and multi-robot parallelized data collection. We train MT-Opt on a set of 12 real-world tasks that include data collected over the course of over 1.5 years, and provide an extensive evaluation (span- ning over 500 real-robot-hours) of our method to- gether with a number of baselines. We show that at such scale, our system can quickly acquire new tasks by taking advantage of prior tasks via shared repre- sentations, new data-sharing strategies and learned policies. Our results indicate that, by learning multi- ple related tasks simultaneously, not only can in- crease the data-efficiency of learning each of them, but also solve more complex tasks than in single- task setup. We present our multi-task system as well as examples of some of the tasks that it is capable of performing in Fig. 1.
A0
Dj
A) Data collection. B) Training ob- jects. C) Sample of training tasks. D) Sample of behaviorally and visually distinct tasks such as covering, chasing, alignment, which we show our method can adapt to. MT-Opt learns new tasks faster (potentially zero-shot if there sufficient overlap with existing tasks), and with less data compared to learning the new task in isolation.
Figure 1:
2 Related Work
Multi-task learning, inspired by the ability of humans to transfer knowledge between different tasks [14], is a promising approach for sharing struc- ture and data between tasks to improve overall effi ciency. Multi-tare Malki-task architectures have been successful across multiple domains; natural language processing [15, 16, 17, 18] and computer vision [19, 20, 21, 22, 23, 24). In this work, we apply multi-task learning concept in RL setting to real robotic tasks.
Combining multiple task policies has been explored in RL by using gating networks [25, 26], condi- tioning policies on tasks [27], mapping tasks to parameters of policy [28, 29, 30J, distilling separate task policies into shared multi-task policy [31, 32. 33, 34, 35, 36). Advantages of multi-task learn- ing for visual representations has been explored in [37]. Pinto and Gupta [4] use a shared multi-task neural network architecture, which is trained with separate task-speeific losses. In contrast, in our work, we concentrate on tasks with a common loss structure within a Q-learning framework. Sev- eral works explore how to mitigate multi-task interference and conflicting objectives [38, 39]. In our experiments, we find that better data routing helps with not only better mitigating conflicting objectives but also iesomvinoiin improving learning efficiency through data reuse.
Learning complex skills has been addressed through hierarchical reinforcement learning with op- tions [40, 41, 42], combining multiple sub-tasks [43, 44, 45, 46, 48]. reusing samples between tasks [49], relabeling experience in hindsight [50]. introducing demonstrations [51, 52, 53, 54, 55, 56, 57]. A range of works employ autonomous supervision learn diverse skills, e.g. by scaling up data collection [58), sampling suitable tasks [59] or goals [60] to practice, learning task em- bedding space amenable to sampling [7]. or learning a dynamics model and using model-predictee control to achieve goals [61, 62, 63, 64]. Riedmiller et al. [5] learn sparse-reward tasks by solving easier auxiliary tasks and reusing that experience for offline learning of more complex tasks. In Cabi et al. [65], previously collected experience is relabeled with new reward functions in order to solve new tasks without re-collecting the data. our work, we similarly design techniques for reusing experience between related tasks and apply them to large-scale data collection on real robots.

---

3 MT-Opt: a Scalable Multi-Task RL System
Fig. 2 overviews our multi-task learning sys- tem. We devise distributed, off-po multi- task RL algorithm together with a multi-task visual success detector (SD) to learn multi- ple robotic manipulation tasks simultaneously. The SD, trained from video examples of de- sired outcomes, (Fig. 2A) in part determines how episodes are leveraged to train an RL pol- icy (Fig. 2C). During evaluation and fine-tuning (Fig. 2B), at each time step, a policy takes as input a camera image and one-hot encoding of the task, and sends a motor command to the robot. At episode end, the outcome image of this process is graded by the SD that deter- mines which tasks were accomplished success- fully and assigns sparse reward O or I for each task. The system then decides whether an- other task should be attempted or if the environ- ment should be reset. We train this system with multiple robots, where each robot concurrently collects data for a different, randomly selected task. The generated episodes are used as offline data for training future policies (Fig. 2C) and are available to improve SDs.
Task Difliittinn
success
detector
episade
8) Data Collection
cpisode
episode
Data Sharing and RL Training
policy
Ti
Tj
T
MI-Opt training
Figure 2: MT-Opt overview. A) The user defines success detector (SD) for tasks through examples of desired outcomes, and relabeling outcomes of prior episodes. B) Utilizing the SD and the MT-Opt pol- icy, new episodes are collected for multiple tasks. C) Offline episodes enter the data-sharing pipeline that pands and re-balances the data used to train MT- Opt policy, while optionally more on-policy data is col- lected, esp. for new tasks. This is iterative pro- cess, which results in additional experiences that can be leveraged define new tasks and train future policies.
In order enable our system take full advan- tage of multi-task training, we develop a num- ber of components and evaluate their impor- lance for overall performance. First, we use single, multi-task deep neural network learn policy for all the tasks simultaneously, which enables parameter sharing between tasks. Second, we devise data management strategies that share and re-balance data across certain tasks. Third, since all tasks share data and parameters, we use some tasks as exploration policies for others, which aids exploration. Although the individual components are based on previously studied concepts in multi-task learning, we show through our experiments that the particular choice and design of these components is essential for our large- scale multi-task learning system to effectively leverage data from all tasks to improve performance, particularly for "underrepresented"" tasks that have the least available training data.
3.1 Multi-Task Reinforcement Learning Algorithm
We denote the multi-task RL policy (alss T:), where a E A denotes the action, which in our case is the desired change in end-effector position, a change in yaw angle, binary gripper open and close commands and termination command, from the robot's cameras, and Ti denotes an encoding of the 7th task drawn from a categorical task distribution T ~ pTT), with possible categories, each corresponding to a different task. At each time step, the policy selects an action a given the current state s and the current task T, set at the beginning of the episode, and receives a task-dependent reward ryy ra S, T:). The goal of the multi- task RL policy is to maximize the expected sum of rewards for all tasks drawn from the distribution P(T). The episode finishes when the policy selects TERMINATE action or reaches a step limit.
denotes the state, which corresponds to images
We build on the single-task QT-Opt algorithm [1], which itself is a variant of Q-learning [66], and learns single-task optimal Q-Function by minimizing the Bellman error: Le(()
((s..)))))(ss.s')) [D(Qv(s, a), Qr(s, s')] where Qr(s, a,s) r(s. + (V(s') is a target Q-value and D is divergence metric, such as cross-entropy, is discount factor, V(s') is the target value function of the next state computed using stochastic optimization of the form V(s')= max Q(s', a'), and the expectation is taken w.r.t. previously seen transitions a,s). Similarly to [1], we use the cross-entropy method (CEM) to perform the stochastic optimization to compute the target value function.
3

---

To extend this approach to the multi-task setting, let s(t), a denote a transition that was generated by an episode e! for the ith task Tj Each transition could be used for multiple tasks, In the task case, the loss becomes (with (s()) alt), s(())) being transitions generated by tasks T:):
Cmmil(() BT [c((0]] = ET-D((T E Eppctt ait al0) (nt)) D(Qo(s(), ,,
(1)
Ti), (r(())))))))))))))))))))))))QQ a s TDi]]
3.2 Sharing Data Between Tasks
uu Inservontte
One advantage of an off-policy RL algorithm is that collected experience can be used to update the policy for other tasks, not just the task for which it was originally collected. This section describes how we effectively train with multi- task data through task impersonation and data re- balancing, as summarized in Fig. 3.
Te
To
Tork imperssation
Data ne-oskanoity
fI
Office Data
Repanded
Bskasseed Date
We leverage experience sharing at the episode level rather than the individual transition level. The goal is use all transitions of an episode e(f) generated by task Ti to aid in training pol- icy for a set of k; tasks T(k. We refer to this process as task impersonation (see Algorithm 1), where the impersonation function fI transforms episode data collected for one task into a set of episodes that can be used to also train other tasks, iee: e(A}} = ()((())) Note that in general case (k:) is subset of all tasks {n}, and it depends on the original task T; that the episode e(f) was collected for.
Figure Task impersonation: episodes are routed to train relevant tasks, and data re-balancing where the ratio of success (S) and failure (F) episodes and proportion of data per task is controlled. Pale blue and pale red show additional training data coming from other tasks. The bar height indicates different amount of data across tasks and successful outcomes.
Algorithm Task Impersonation
procedure fi(e", original episode)
.xpanded.episodes [
(Dkk <- set of SDs relevant to task T;
for S SDx in (D(k: do
I| e+ e but rewards for task Tl not * = (D)(ee)
We consider two ends of the data-sharing spectrum: an identity impersonation function St.n (el+)- = c(9) where no task impersonation takes place, i.e. an episode e( generated by task T, is used to train the policy only for that task; and fr. = e (n), where each task shares data with all n - tasks (maximal data sharing). While flay fails to leverage the reusable nature of multi-task data, fta results in many unrelated episodes used as negative examples for the target task.
expanded expanded_epissodessuuu apissodss..ppeedde"" return expanded episodees
For MT-Opt, we devise a new task impersonation strategy fram that uses more fine- the-grained similari- ties between tasks. We refer to it as a skill-based taskiimpersonation strategy, where we overload the term "skill" as a set of tasks that share semantics and dynamics, yet can start from different initial conditions or operate on different objects. this work we manually assign each task to a particular skill based on the subjective similarity of the required manipulation primitive. For example, tasks such as sllace-ojject-o on platta and place-object-ig siice-object-n-bbwl bowl belong to the same skill. Our impersonation function flan allows us to impersonate an episode el only as the tasks belonging to the same skill as T. Namely, given an episode e; generated by task Tr skill Sj that task belongs to is detected. The e, will be impersonated only for the tasks T(S,} belonging to that particular skill. In order to avoid impersonating too many episodes and overloading the replay buffer, we introduce stochas- tie impersonation function. An impersonated episode candidate will be routed to training with the probability pe if success, or with probability p, if it's a failure. We provide more details on the specific parameters in the Appendix 7.3. In our experiments, we conduct ablation studies comparing fraa (ours) with other task impersonation strategies.
While training, due to the design of our task impersonation mechanism, as well as the variability in difficulty between tasks, the resulting training data stream often becomes highly imbalanced, see Fig. 3B. To address this, we re-balance each batch both between tasks, such that the relative propor- tion of training data for each task is equal, and within each task, such that the relative proportion of successful and unsuccessful examples is kept constant.

---

Evaluation Tasks:
Object Acquisition Skill
Object Manipulation Skill
lift-any
lift sau-sae
mift-milk
Aft-bottle
Aft--con
place top-left
place ony
lift vit-banona
lift corrot
lft=food-box
place top right
place bottom
Figure 5: 12 evaluation tasks, giving rise to Object Acquisition and Object Manipulation skills.
4 SD-Based Rewards and Continuous Data Collection
We learn tasks evaluated on the final state of an episode. This sparse-reward assumption allows us to train a multi-view multi-task CNN success detector model (SD), conditioned on task ID, which given 3 final images from 3 different camera angles, infers task success (see Appendix 7.1 for details).
To generate training data for the SD, we ask users to collect positive and negative examples of pos- itive and negative outcomes of task. These examples are not demonstrations - just examples of what successful completion (i.e., the final state) looks like. We use this data to train the initial ver- sion of the multi-task SD, which we then continuously fine-tune given the images gathered during autonomous data collection to avoid out of distribution images that might be caused by various real- world factors such as different lighting conditions, and novel states which the robot discovers. We continue to manually label such images and incrementally retrain D to obtain the most up-to-date SD. result, we label 5, 000 images per task and provide more details in the Appendix 8.
Our main observation w.r.t. the multi-task data collection is the use of easier tasks to boot- strap learning of more complex tasks. In par- ticular, an average MT-Optt policy for simple tasks might occasionally yield episodes suc- cessful for harder tasks. Over time, this allows us to start training an MT-Optt policy now for the harder tasks, and consequently, to collect better data for those tasks. To kick-start this process and bootstrap our two simplest tasks, we use two crude scripted policies for picking and placing (see Sec. 7.2 for details) follow- ing prior work [1]. To simplify exploration for longer-horizon tasks, we allow the individual tasks to be ordered sequentially, where one task is exccuted after another using simple heuris- tics such as executing placing tasks after lifting tasks. Our dataset grows over time w.r.t. the amount of per-task data well as percentage of successful episodes for all the tasks. All data from variety of different policies is accumulated over time: different versions of trained policies, their ppsilon-greedy counterparts (e = 20%), and scripted policies. Importantly, this data collec- tion process results in imbalanced dataset, as shown Fig. 4. Our data impersonation and re-balancing methods address this imbalance by efficiently expanding and normalizing data.
#Episodes by Task by Policy Type
ank
Ecredy
500- 50K
Sokdee
T00K
20K
DOOK
500K
400%
300x
200x
Figure 4: Offline dataset properties. Our data collection strategy collects data for multiple tasks, where it uses casier, more general tasks (e.g. lift-any) to bootstrap learning of more specialized tasks (c.g. lift-carrot). The resulting dataset is imbalanced across the distribution of exploration policies per task and success rate per task.
5 Experiments
The goal of our real-world experiments is to answer the following questions: (1) How does MT-Opt perform, quantitatively and qualitatively, on a large set of vision-based robotic manipulation tasks? (2) Does training a shared model on many tasks improve MT-Opt's performance? (3) Does data sharing improve performance of the system? (4) Can we use easier tasks to bootstrap learning of more difficult tasks? (5) Can MT-Opt quickly learn distinct new tasks by adapting learned skills?
5

---

5.1 Experimental Setup
MT-Opt provides a general robotic skill learning framework that we use to learn multiple tasks, including semantic picking (i.e., picking an object from a particular category), placing into various fixtures (e.g., placing a food item onto a plate). We focus on basic manipulation tasks that require repositioning objects relative each other. A wide range of manipulation behaviors fall into this category, from simple bin-picking to more complex behaviors, such as covering items with a cloth. As described in Sec. 4, our task set grows over time allowing for more and more tasks to be similar. To analyze and ablate various properties of MT-Opt, we focus on set of different but similar tasks, where multi-task learning shows maximal efféctiivness. In the following experiments, we use set of 12 tasks for quantitative evaluation of our algorithm. These 12 tasks include a set of plastic food objects and divided plate fixtures and they can be split into "object acquisition" and "object manipulation" skills. Since the criteria of what constitutes a task are not well defined, in our system, different tasks correspond to different success conditions as defined by our SD. Our most general object acquisition task is lift-any, where the goal is to singulate and lift any object. We also define 7 semantic lifting tasks, where the goal is to search for and lift a particular object (ex: carrot). Importantly, these semantic tasks require search in clutter and singulation before the actual picking can be performed. The placing tasks utilize a divided plate where the simplest task is to place the previously lifted object anywhere on the plate (place-any). Harder tasks require placing the object into a particular section of a divided plate, oriented arbitrarily. Fig. 5 visualizes the tasks.
All polices are trained with offline RL from large dataset summarized in Fig. 4. The resulting policy is deployed on 7 robots attempting each task 100 times for evaluation. To reduce the variance of the evaluation, we shuffle the bins after each episode and use a standard evaluation scene (see Appendix, Fig. 17), where all of objects are present, hence all 12 evaluation tasks are feasible.
5.2 Quantitative and Qualitative Evaluation of MT-Opt
Fig. 6 shows MT--ptt success rates on the 12 evaluation tasks.
Parameter Sharing (Success Rate)
We compare to three baselines: (i) single-task QT-Opt [1], where each per-task policy is trained separately using only
Model: 2-Task
12- Tassk
lift-any 0.82
0.89
data collected specifically for that task, (ii) an enhanced QT-
place-any 0.63
0.85
Opt baseline, which we call QT-Opt Multi-Task, where we train shared policy for all the tasks but there is no data impersonation or re-balancing between the tasks, and (iii) a Data-Sharing Multi-Taa Task bascline that is based on the data- sharing strategy presented in [65]. where we also train a sin- gle Q-Function but the data is shared across all tasks. From the average performance across all task, we observe that MT- Opt outperforms the baselines, in some cases with R 3x average improvement. While the single task QT-Opt baseline performs similarly to MT-Opt for the task where we have the most data (see data statistics in Fig. 4). lift-any, its performance drastically drops (to a 1%) for underrepresented tasks, such as lift-can. Note, that we are not able run this baseline for the placing tasks, since they require a separate task to lift the object, which is not present the single-task baseline. A similar observation applies to QT-Opt Multi-Task where the performance of rare tasks increases compared to QT-Opt, but is still 4x worse on average than MT-Opt. Sharing data across all tasks also re- sults in a low performance for semantic lifting and placing tasks and, additionally, it appears to harm the performance of the indiseriminate lifting and placing tasks. The MT-Opt policy, besides attain- ing the 80% success rate on (lift-ary,, also performs the 7 semantic lifting tasks and the 4 placing and rearrangement tasks at significantly higher success rate than all baselines. Since all ablated experiments are trained on the same exact offline dataset (except for the case where the multiple policies are trained on the per-task shard of the dataset) and have the same computation budget, we explain these performance gaps by the way MT-Optt shares the representations and data, and provide a more comprehensive analysis of these factors in the following experiments. To highlight learned behaviors, Fig. 7 shows the policy work the carrot out of the corner before picking it up.
Parameter sharing: the pol- icy that learns two tasks (Lift-ay,, place any) in addition to 10 other tasks out- performs a policy trained only for the two target tasks. Both policies are trained on the same offline datasct.
Table 1:
Fig. 4, shows amounts and type of data collected for various tasks. Tasks such as lift-carrot and lift-bottle, which have more data, especially on-policy data, have higher success rates than under- represented tasks, such as lift-box. The performance of these underrepresented tasks could be further improved by focusing the data collection on these tasks.

---

5.3 Sharing Representations Between Tasks
To explore the benefits of training a single policy on multiple tasks, we compare the 12-task OTt policy with a 2-task policy that learns lift-any and place-any. Both of these policies are evaluated on (lift-any and place-any). We use the same ftmm task impersonation strategy, and the same offline dataset (i.e. both policies use the data from the extra 10 narrower tasks, which is impersonated as lift- any and place-any data) without on-policy fine-tuning, so data-wise the experiments are identical.
The 12-task policy outperforms the 2-task policy even on the two tasks that the 2-task policy is trained on (Table 1), suggesting that training multiple tasks not only enables the 12-task policy to perform more tasks, but also improves its performance on the tasks through sharing representations. This may suggest that the additional supervision provided by training on more tasks has beneficial effect on the shared representations.
5.4 Data Sharing Between Tasks
To test the influence of data-sharing and re-balancing on the multi-task policy's performance, we compare our task impersonation strategy fram introduced in Sec. 3.2 to baseline impersonation function that does not share the data between the tasks fi... and a baseline where each task is impersonated for all other tasks fig- In our skill-based task impersonation strategy Stam the data is expanded only for the class of tasks having similar visuals, dynamics and goals. We also test data re- re-balancing strategy, where we re-balance each training batch between the tasks as well as within each task keep the relative proportion of successful and unsuccessful trials the same.
The results of this experiment are in Table 2, with the full results reported in the Appendix, Ta- ble 4. Sharing data among tasks using our method of task imper- sonation and re-balancing pro- vides significant improvement across all the tasks, with im- provements of up to 10z for some tasks. The full data-sharing strategy performs worse than both the no-data-sharing baseline and our method, suggesting that naively sharing all data across all tasks is not effective. Because of our data-collection strategy, the resulting multi-task dataset contains much more data for broader tasks (e.g.. lift-any) than for more narrow, harder tasks (e-g.. lift-hox), as shown in Fig. 4. Without any additional data-sharing and --ballncing, this data imbalance causes the bascline strategy fime to attain good performance for the easier, overrepresented tasks, but poor performance on the harder, underrepresented tasks (see Table 2, first row), whereas our method performs substantially better on these tasks.
Data Strategies (min, mean, max, mean of low data tasks)
Imp. Fn
Data Re- BaBalaciing Strategy
uniform sampling
task re- re-balanced sampling
1.0/0.32/055 0.18
0.16 0.55 / 0.85/0.5550 0,42
fia
0.07 /0.21 /0.62 0.13
0.02 0.35 0.95/0.22
Flam
0.17 / 0.46 / 0.88/ 0.32
0.29/ / 0.58 / 0.89 / 0,50 Ours
Table Min, average and max task performance across 12 tasks, as well as average performance across 6 tasks having least data (s 6K cpisodes) for different data-sharing strategics. flam impersonation and data re-balancing
complimentary: they both improve over the base- lines, while the best effect is achieved by combining both. The effect especially pronounced for the underrepresented tasks.
5.5 Using Easier Tasks to Bootstrap Harder Tasks
To explore question (4), we study whether learning an easier, broader task (lift-any) helps with a structurally related task that is harder, more specific (lift-sausage). We separate out data for lift- sausage which consists of 5400 (nearest hundred) episodes collected for that task (i.c. 4600 failures and 800 successes). In addition, there are 11200 episodes of successful sausage lifting and as many as 740K failures that were collected during the lift-any task. Combining the lift-sausage data and the extra successes from lift-any yields 16600 episodes (12000 successes and 4600 failures). To investigate the influence of MT-Opt and task impersonation on the bootstrap problem. we compare our 12-task MT-Opt policy to a single-task policy trained on these 16600 episodes, including the same set of successful lift-sausage episodes as MT-Opt, but not including failures from other tasks.
The single-task policy learned from the 16600 episodes yields performance of 3%. MT-Opt, using impersonated successes and failures, achieves 39% success for the same task, a RJ 10x improve- ment. Both experiments use identical data representing successful episodes. The benefits of MT-Opt are twofold. First, we leverage an easier lifi-any task to collect data for the harder lift-sausage task. Second, MT-Opt benefits from the additional failures impersonated from other tasks. These fail- ures, which often include successful grasps of non-target objects, when re- e-baalnced as described in Sec. 3.2, results in the significant task performance boost. This demonstrates the value of both successful and unsuccessful data collected by other tasks for learning new tasks.
7

---

5.6 Learning New Tasks with MT-Opt
In addition to retroactive relabelling of prior data, MT-Opt can learn new tasks via proactive adap- tation of known tasks, even ones that are visually and behaviorally different than the initial training set. To accommodate future tasks, we start learning with larger one-hot vector. Once a new task is defined and its success detector is trained, we allocate a next available one-hot task ID to this task. To demonstrate this, we perform a fine-tuning experiment, bootstrapping from the MT-Opt 12-task policy (Sec. 5.2). We use the MT-Opt policy to collect data for unseen tasks: lift-cloth and cover- object ( Fig. 7 bottom row). Unlike lift-sausage, prior to starting collection of these new tasks, no episodes in our offline dataset can be relabelled as successes for these tasks.
We follow the continuous data collection pro- cess (Sec. 4): we define and train the suc- cess detector for the new tasks, collect initial data using our lift-any and a place-any poli- cies. and fine-tune a 14-task MT-Opt model that includes all prior as well as the newly de- fined tasks. While the new tasks are visually and semantically different, in practice the above mentioned policies give reasonable success rate necessary to start the fine-tuning. We switch to running the new policies on the robots once they are at parity with the lift-any and place- any policies. After 11K pick-cloth attempts and 3K cover-object attempts (requiring < day of data collection on 7 robots), we obtain an extended 14-task MT policy that performs cloth picking at 70% success and object cov- ering at 44% success. The policy trained only for these two tasks, without support of our of- fline dataset, yields performance of 33% and 5% respectively, confirming the hypothesis that MTTOpt method is beneficial even if the tar- get tasks are sufficiently different, and the tar- get data is scarce. By collecting additional 10K pick-cloth episodes and 6K cover-object episodes, we further increase the performance of 14-task MT-Opt to 92% and 79%, for cloth picking and object covering respectively. We perform this fine-tuning procedure with other novel tasks such as previously unseen trans- parent bottle grasping, which reaches a perfor- mance of 60% after 4 days of collection.
MT. Opt quantitative performance comparison
Ni--aat (ours
O7-Cye to Taal
Data Staring Mi Tark-Tack
0.8
07-Opt
3.6
04
aay
onfosttee
Reccomo
alernayy
Figure 6: Quantitative evaluation of MT-Opt across 12 tasks. QT- [1] trains each task individually us- ing only data collected for that task. QT-Opt Multi- Task [1] trains a single network for all tasks but does not share the data between them. Data- Data- Sharing Multi-Tase also trains a single network for all tasks and shares the data across all tasks without further re-balancing. MT- Opt (ours) provides significant improvement over the basclines, especially for the harder tasks with less data.
Figure 7: Top: Example of pick-carrot. The robot repo- sitions the carrot out of the corner to pick it. Bottom: cover-object. The deformable covers the object.
5.7 Limitations of MT-Opt
While we show successful execution of MT-Opt on a number of tasks and its potential scale, is important to list limitations of this system. i) The shown tasks are short- thort-horizon ( < 20 steps). Long- horizon real-worl. robotic tasks are still hard for RL methods; ii) The task outcome is determined from the final episode frame. In theory, this could be extended to learned intermediate rewards; iii) Our tasks leverage an automated reset mechanism. While this is not an assumption of the method, we have not demonstrated learning any reset-free behaviors; iv) Learning drastically different skills challenging, as our method works best when adapting more similar tasks. remains an open question to how far MT-Opt can be scaled to very diverse set of skills; v) MT-Opt is 4DoF with gripper and terminate actions. Learning higher DoF tasks is challenging yet principally possible.
Conclusion
We presented a general multi-task learning framework, MT-Opt, that encompasses a multi-task data collection system, a scalable success detector framework, and a deep RL method that is able to effectively utilize the multi-task data. Our experiments show that by sharing data and parameters across tasks we significantly increase the data efficiency of learning individual tasks and that MT-Optt quickly acquires new tasks via the shared multi-task representations and exploration strategies.
8

---

References
[1] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, et al. Qt-opt: Scalable deep reinforcement learning for vision- based robotic manipulation. arXiv preprint arXiv: 10293, 2018.
[2] I. Akkaya, M. Andrychowicz. M. Chociej, M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, R. Ribas, et al. Solving rubik's 's cube with a robot hand. arXiv preprint arXiv:1910.07113, 2019.
[3] L. Espeholt, H. Soyer, R. Munos, Simonyan, V. Mnih, T. Ward, Doron, V. Firoiu, T. Harley, I. Dunning, et al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. arXiv preprint a /802.01661, 2018,
[4] L. Pinto and A. Gupta. Learning to push by grasping: Using multiple tasks for effective learning. In ICRA, pages
IEEE, 2017. ISBN 978-1-5090-4633-1.
[5] M. Riedmiller, R. Hafner, T. Lampe, M. Neunert, J. Degrave, T. Van de Wiele, V. Mnih, N. Heess, and T. Springenberg. Learning by playing-solving sparse reward tasks from scratch. arXiv preprint arXiv:1802.00 10567, 2018.
[6] A. Wilson, A. Fern, S. Ray, and P. Tadepalli. Multi-task reinforcement learning: hierarchical bayesian approach. In Proceedings of the 24th international conference on Machine learning, pages 2015-1022.0 ACM, 2007.
[7] K. Hausman, J. T. Springenberg, Z. Wang. N. Heess, and M. Riedmiller. Learning an embed- ding space for transferable robot skills. In International Conference Learning Representa- tions, 2018.
[8] T. Schaul, D. Borsa, J. Modayil, and R. Pascanu. Ray interference: source of plateaus in deep reinforcement learning. 1rXiv:1904.11440 2019.
[9] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforcement learning with popart. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages
2019.
[10] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. arXiv preprint arXiv:171.002257, 2017.
[11] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning, arXiv preprint arXi::2001.0678/ 2020.
[12] Yu, D. Quillen, Z. Hc, R. Julian, K. Hausman, C. Finn, and S. Levine, Meta-world: A benchmark and evaluation for multi-task and meta reinforcement learning. In Conference Robot Learning, pages 1094-1100, 2020.
[13] B. Eysenbach, X. Geng, S. Levine, and R. Salakhutdinov. Rewriting history with inverse ri: Hindsight inference for policy improvement. arXiv preprint arXiv:2002. 2020.
[14] R. Caruana. Multitask learning. Mach. Learn., 28:41-75, 1997.
[15] S. Subramanian, A. Trischler, Y. Bengio, and C. J. Pal. Learning general purpose distributed sentence representations via large scale multi-task learning. arXiv preprint arXiv: 2018.
[16] M. Johnson, M. Schuster, Q. V. Le, M. Krikun, Y. Wu, Z. Chen, N. Thorat, F. Viégas, M. Wat- tenberg, G. Corrado, et al. Google's multilingual neural machine translation system: Enabling zero-shot translation. Transactions of the Association for Computational Linguistics, 5:339- 351, 2017.
[17] B. McCann, N. S. Keskar, C. Xiong, and R. Socher. The natural language decathlon: Multitask learning question answering. arXiv preprint arXiv:1806.0888 2018.

---

[18] X. Liu, P. He, W. Chen, and J. Gao. Multi-task deep neural networks for natural language understanding. In A. Korhonen, D. R. Traum, and L. Märquez, editors. ACL (1), pages 4487- 4496. Association for Computational Linguistics, 2019. ISBN 978-1-950737-448
[19] Z. Chen, V. Badrinarayanan, C.-Y. Lee, and A. Rabinovich. Gradnorm: Gradient normaliza- tion for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning, pages 114-803, PMLR, 2018.
[20] I. Misra, A. Shrivastava, A. Gupta, and M. Hebert. Cross-stitch networks for multi-task learn- ing. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3994-4003, 2016.
[21] O. Sener and V. Koltun. Multi-task learning multi-objective optimization. In Advances in Neural Information Processing Systems, pages 527-538, 2018.
[22] E. Perez, F. Strub, H. De Vries, V. Dumoulin, and A. Courville, Film: Visual reasoning with general conditioning layer. arXiv preprint arXiv: 1709.77871 2017.
[23] A. R. Zamir, A. Sax, W. B. Shen, L. J. Guibas, J. Malik, and S. Savarese. Taskonomy: Disen- tangling task transfer learning. In IEEE Conference on Computer Vision and Pattern Recogni- lion (CVPR). IEEE, 2018.
[24] T. Standley, A. R. Zamir, D. Chen, L. Guibas, J. Malik, and S. Savarese. Which tasks should be learned together in multi-task learning?, 2020.
[25] M. E. Taylor and P. Stone. Cross-domain transfer for reinforcement learning. In Z. Ghahra- mani, editor, ICML, volume 227 of ACM International Conference Proceeding Series, pages 879-886. ACM, 2007. ISBN 978-1-59593--93-3.
[26] K. Mülling, J. Kober, O. Kroemer, and J. Peters. Learning to select and generalize striking movements in robot table tennis. Int. J. Robotics Res., 32(3):263-21) 2013.
[27] M. P. Deisenroth, P. Englert, J. Peters, and D. Fox. ICRA, pages 3876-3881. IEEE, 2014.
Multi-task policy search for robotics. In
[28] B. C. da Silva, G. D. Konidaris, and A. G. Barto. Learning parameterized skills. ICML. ieml.ce / Omnipress, 2012.
[29] Kober, A. Wilhelm, E. Öztop, and J. Peters. Reinforcement learning to adjust parametrized motor primitives to new situations. Auton. Robots, 33(4):361-379, 2012.
[30] R. Yang, H. Xu, Y. Wu, and X. Wang. Multi-task reinforcement learning with soft modular- ization. CoRR, abs/2003. 13661, 2020.
[31] S. Levine, C. Finn, T. Darrell, and P. Abbeel. End-to-end training of deep visuomotor policies. The Journal of Machine Learning Research, 17(1):1334-137,, 2016.
[32] Y. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick, R. Hadsell, N. Heess, and R. Pas- canu. Distral: Robust multitask reinforcement learning. In Advances in Neural Information Processing Systems, pages 4406 2017.
[33] A. A. Rusu, S. G. Colmenarejo, C. Gulcehre, G. Desjardins, Kirkpatrick, Pas- canu, V. Mnih, K. Kavukcuoglu, and R. Hadsell. Policy distillation. arXiv preprint 1511:1511.00299 2015.
[34] E. Parisotto, J. I.. Ba, and R. Salakhutdinov. Actor-mimic: Deep multitask and transfer rein- forcement learning, arXiv preprint arXiv:1511.066: 2015.
[35] D. Ghosh, A. Singh, A. Rajeswaran, Kumar, and S. Levine. Divide-and-conquee reinforce- ment learning. arXiv preprint 171 2017.
[36] H. Arora, Kumar, Krone, and C. Li. Multi-task learning for continuous control. arXiv preprint arXiv: 1802,01034, 2018.
10

---

[37] L. Pinto, D. Gandhi, Y. Han, Y.-L. Park, and A. Gupta. The curious robot: Learning visual rep- resentations via physical interactions. In B. Leibe, J. Matas, N. Sebe, and M. Welling, editors, Computer Vision ECCV 2016, pages 3-18, Cham, 2016. Springer International Publishing. ISBN 978-3-319-46475-6
[38] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. van Hasselt. Multi-task deep reinforeement learning with popart. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3796-3803, 2019.
[39] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. CoRR, 0als/2001.667822 2020.
[40] R. S. Sutton, D. Precup, and S. Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial Intelligence, 112(1):181 211, 1999. ISSN 0004-3702.
[41] A. G. Barto and S. Mahadevan. Recent advances in hierarchical reinforcement learning, Dis- crete Event Dynamic Systems, (61-2)):1-77, 2003.
[42] C. Daniel, G. Neumann, and J. Peters. Hierarchical relative entropy policy search. In N. D. Lawrence and M. A. Girolami, editors, AISTATS, volume 22 of JMLR Proceedings, pages 273-281. JMLR. .org. 2012.
[43] A. G. Barto, S. Singh, and N. Chentanez. Intrinsically motivated learning of hierarchical collections of skills. In Proceedings of International Conference on Developmental Learning (ICDL). MIT Press, Cambridge, MA, 2004.
[44] T. G. Dietterich. Hierarchical reinforcement learning with the maxq value function decompo- sition. Artif. Intell. Res., 13:227-303, 2000.
[45] J. Morimoto and K. Doya. Acquisition of stand-up behavior by a real robot using hierarchical reinforcement learning. Robotics Auton. Syst., 36(1):37-51, 2001.
[46] M. Ghavamzadeh and S. Mahadevan. Hierarchical policy gradient algorithms. In T. Fawcett AAAI Press, 2003. ISBN 1-57735-189-4,
and N. Mishra, editors, ICML, pages
[47] Shu, C. Xiong, and R. Socher. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning, arXiv preprint arXiv:1712.07299. 2017.
[48] A. Zeng, Song, S. Welker, J. Lee, A. Rodriguez, and T. Funkhouser. Learning synergies between pushing and grasping with self-supervised deep reinforcement learning, 2018.
[49] A. Lazaric, M. Restelli, and A. Bonarini. Transfer of samples in batch reinforcement learning- In W. W. Cohen, A. McCallum, and S. T. Roweis, editors, ICML. volume 307 of ACM Interna- tional Conference Proceeding Series, pages 551. ACM, 2008 ISBN 978- 11881-60558-205-44
[50] M. Andrychowicz, D. Crow, A. Ray, J. Schneider, R. Fong. P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, pages 5048-5058 2017.
[51] R. Rahmatizadeh, P. Abolghasemi, I.. Bölöni, and S. Levine. Vision-based multi-task manip- ulation for inexpensive robots using end-to-end learning from demonstration. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 3758-3765 IEEE, 2018.
[52] A. Xie, Ebert, S. Levine, and C. Finn. Improvisation through physical understanding: Using novel objects as tools with visual foresight. Robotics: Science and Systems (RSS), 2019.
[53] K. Hausman. Y. Chebotar, S. Schaal, G. Sukhatme, and J. J. Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems, pages 1235-1245, 2017.
[54] Y. L.i J. Song, and S. Ermon. Infogail: Interpretable imitation learning from visual demonstra- tions. In Advances in Neural Information Processing Systems, pages 3812-3822, 2017.
11

---

[55] A. Singh, E. Jang, A. Irpan, D. Kappler, M. Dalal, S. Levine, M. Khansari, and C. Finn. Scalable multi-task imitation learning with autonomous improvement. arXiv preprint MoXiv:2003302630 2020.
[56] T. Shankar, S. Tulsiani, L. Pinto, and A. Gupta. Discovering motor programs by recomposing demonstrations. In International Conference on Learning Representations, 2020.
2020..
[57] L. Shao, T. Migimatsu, Q. Zhang, K. Yang, and J. Bohg. Concept2robot: Learning manipu- lation concepts from instructions and human demonstrations. Roboties: Science and Systems (RSS), 2020.
[58] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In ICRA, pages 3406-3413. IEEE, 2016. ISBN 978-1-4673-8026-3.
[59] Sharma, Jha, Hegde, and B. Ravindran. Learning to multi-task by active sampling- arXiv preprint N.N.:00772.000555 2017.
[60] A. Nair, V. Pong, M. Dalal, S. Bahl, S. Lin, and S. Levine. Visual reinforcement learning with imagined goals. In S. Bengio, H. M. Wallach, H. Larochelle, K. Grauman, N. Cesa- cesa-Bianchhi, and R. Garnett, editors, NeurIPS, pages 9209-9220, 2018.
[61] C. Finn and S. Levine. abs/1610.00696, 2016.
Deep visual foresight for planning robot motion.
CoRR,
[62] F. Ebert, C. Finn, S. Dasari, A. Xie, A. X. Lee, and S. Levine. Visual foresight: Model-based deep reinforcement learning for vision-based robotic control. CoRR, fbss/882.0055 2018.
[63] Y.-C. Lin, M. Bauzá, and P. Isola. Experience-embedded visual foresight. In P. Kaelbling, D. Kragic, and K. Sugiura, editors, CoRL, volume 100 of Proceedings of Machine Learning Research, pages 1015-1024. PMLR, 2019.
[64] H. J. T. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. CoRR, abs/2002.09093, 2020.
[65] S. Cabi, S. Gómez Colmenarejo, A. Novikov, K. Konyushkova, S. Reed, R. Jeong, K. Zolna, Y. Aytar, D. Budden, M. Vecerik, et al. Scaling data-driven robotics with reward sketching and batch reinforcement learning. arXiv, pages arXiv-1909, 2019.
[66] R. S. Sutton and G. Barto. Introduction Reinforcement Learning. MIT Press, Cambridge, MA, USA, 1st edition, 1998. ISBN 0262193981.
[67] I. Akinola, J. Varley, and D. Kalashnikov. Learning precise 3d manipulation from multiple uncalibrated cameras. arXiv preprint arXiv:2002.09110,, 2020.
12

---

7 Appendix
7.1 Neural Network Architecture
Repeat x
Repeat
Repeat
Repeat
Conv/64, 2)
Conv/64, 1)
Conv/64, 1)
Conv(64, 1)
MaxPool(3)
3axPool(3)
MaxPool(2)
MaxPool(3)
FC/64)
Q a, T.
Log
1,(422, 472,
Target Valne Function arg max 1 8.
Cartesian
Gripper Rocation
Reshape(1. 1, 64)
a Open Gripper
9aper
Close Gripper
FC,258)
Sum
FC(64)
Terminate
5
{
Task ID
Gripper open?
Gripper height
The architecture of MT-Opt Q Q-function. The input image is processed by a stack convolutional layers. Action vector, state vector and one-hot vector Ti representing the task of interest are processed by several fully connected layers, tiled over the width and height dimension of the convolutional map, and added to it. The resulting convolutional map is further processed by numher of convolutional layers and fully connected layers. The output gated through sigmoid, such that Q-values are always in range [0. 1].
Figure 8:
Compariuan of Mosel Archibectures
We model the Q-function for multiple tasks as large deep neural network whose architecture is shown in Fig. 8. This network resembles one from [1]. The network takes the monocular RGB image part of the state 8 as input, and processes with convolutional layers. The actions a and additional state features (gatatues (height) and task ID are transformed with funnecconnected lay- ers, then merged with visual features by broad- casted element wise addition. After fusing state and action representations, the Q-value Qo(s.a) modeled by more convolutional layers fol- lowed by two fully-connected layers. In our sys- tem the robot can execute multiple tasks from in the given environment. Hence the input image not sufficient to deduce which task the robot commanded to execute. To address that, feed one-hot vector representing task ID into the network to condition Q-Function to learn task- specific control.
Mi-del Singte Noazes Weee
un aacaaee
Figure 9: Comparison single-headed and multi- headed neural networks approximating the Q- function. In both cased task ID was fed as the in- put the network. Multi-headed architecture of the Q-function under- performms on a wide range of tasks, winning only on lift-any tasks which has most of the data.
In addition to feeding task ID we have experimented with multi-headee architecture, where n sep- arate heads each having 3 fully connected layers representing n tasks were formed at the output of the network. Fig.9 shows that performance of the system with the multi-headed Q-function architecture is worse almost for all tasks. We hypothesize that dedicated per task heads "over- compartmentalizes" task policy, making it harder to leverage shared cross-task representations.
13

---

7.2 Description of Scripted Policies
As discussed in Section4 we use two crude scripted policies to bootstrap easy generic tasks.
To create successful picking episodes, the arm would begin the episode in random location above the right bin containing objects. Executing crude, scripted policy, the arm programmed to move down to the bottom of the bin, close the gripper, and lift. While the success rate of this policy is very low (+e 10%), especially with the additional random noise injected into actions, this enough to bootstrap our learning process.
Scripted Picking Policy:
Scripted Placing Policy: The scripted policy programmed to perform placing would move the arm to a random location above the left bin that contains fixture. The arm is then programmed to descend, open the gripper to release the object and retract. This crude policy yields success rate of (47%) at the task of placing on fixture (plate), as the initial fixture is rather large. Data collected by such simplistic policy is sufficient to bootstrap learning.
7.3 flava impersonation strategy details
Task impersonation is an important component of the MT-Opt method. Given episode and task definition, the SD classifies if that episode is an example of a successful task execution according to that particular goal definition. Importantly, both the success and the failure examples utilized by our algorithm. The success example determines what the task is, while the failure exam- ple determines what the task is not (thus still implicitly providing the boundary of the task), even it's an example of a success for some other task. Fig. shows offline success rates and Fig. 12 shows by how much the per task data is expanded using the Su impersonation function.
efficiently
In Section 3.2 we discuss a problem arising when using a naive frau episodes impersonation func- tion, and suggest a solution to impersonate data only within the boundaries of a skill. Namely, given an episode e; generated by task Tr a skill S, that task belongs to is detected. The e, will be impersonated only for the tasks T(s,) belonging to that particular skill.
Replay Buffer
Read
Log Replay
Label with SD [1) (uuel impentorate
I1 auooses
fallure
Trainer
JUOOWSS
Feluue
-40k
Re balanced Sampling
Ueteler ond litne taaks
Real Robots running MDP
Bellmen Update
O,s5 parameters
(Recompules O-Taagels per look)
Pizl modill
Figure 10: System overview: Task episodes from disk are continuously loaded by LogReplay job into task replay buffers. LogReplay process assigns binary reward signal to episodes using available Success Detectors and impersonates episodes using fi (or other strategy). Impersonated episodes are compartmentalized into dedicated per task buffers, further split into successful and failure groups. Bellman Update process samples tasks using re-balancing strategy to ensure per task training data balancing and computes Q-targets for individ- ual transitions, which are placed into train buffer. These tramsitions (s, a, Ti) are sampled by the train workers to update the model weights. The robot flect and Bellman Update jobs are reloading the most to date model weights frequently.
Note, that sometimes impersonation for all T(s.), tasks within a skill could result in too excessive data sharing. For example, the bulk of the data for object-acquisition skill represents variants of tasks involving foods objects. If we want to learn a new task within the same skill using visually significantly different objects, e.g. transparent bottles, all offline episodes involving the plastic ob- jects will be (correctly) impersonated as failures for the liff-rransparent-blit task. That is, a few intrinsic failures for that task will be diluted in large set of artificially created negatives.
14

---

Primary SD
Total
Success
Failure
Success Rate
F. Neg.
F. Pos.
Other F. Neg. Rate
Other
Name
Count
Count
Count
Rate
Rate
Pos. Rate
lift-any
16064
7395
8672
46%
1%
2%
0%
0%
lift-banana
6255
510
5745
8%
2%
1%
0%
1%
lift-bottle
6472
430
6042
7%
5%
1%
1%
lift-sausage
6472
461
6011
7%
3%
0%
0%
1%
lift-milk
6472
158
6314
2%
7%
0%
3%
9%
lift-box
6467
487
5980
8%
1%
1%
2%
lift-can
6467
270
6197
4%
2%
0%
3%
3%
lift-carrot
6481
911
5570
14%
0%
1%
0%
0%
place-any
3087
1363
1724
44%
1%
2%
0%
place-bottom
2893
693
2200
24%
2%
1%
1%
3%
place-top left
2895
346
2549
12%
10%
0%
3%
8%
place-top-right 2897
312
2585
11%
4%
0%
5%
Table 3: Success detection holdout data statistics. Table shows success detector error rate for held out labelled success detector data. We split out the evaluation dataset based the robot, c.g. all data generated by Robot #1 used for evaluations and not for training. This strategy results in a much better test of generalization power of the success detector, compared to the conventional way to split out 20% of the data randomly for evaluation. The Other Task False [Positive/egaaive]] Rates columns indicates how well the success detector for task A classifies outcomes for all other tasks. For example want ensure that successful lift-carrot episode does not trigger lift- lift-banana success, i.e. not only success detector should manifest dedicated task success, but also reliably reason about other related tasks. This "contrastiveness" property of the success detectors is of great importance in our system. As success detectors determine tasks data routing and experience sharing, error in this tasks data assignment would drive anti anti-correlated examples for each task, resulting in poor performance of the system.
#Offline Dataset: Success By
To solve this issue we introduce a stochastic im- personation function. An impersonated episode candidate will be routed to training with the probability ps if it's a success, or with proba- bility Pf if it's failure. We experiment with Ps 1.0, and Py <= 1.0. The reasoning is that it's always desirable to utilize surplus im- personated examples of a successful task execu- tion, but it could be better to utilize only frac- tion of the surplus failures to balance intrinsic artificial failures for that task.
0.5
Success
0.2
0.1
0.0
Iif any
Iin banana
ii--bottte
lift int-susaage
lifomiik
litebox
lif-can
Int-carrt
place-
bottom
piece- tog-lort
place-
night
place
This gives rise to the frxu (p..P)) imperson- ation function which is suitable in some situa- tions explained above.
Figure 1I: Effective success rate each task our offline dataset. This plot represents the distribution of successes within the entirety of our offline dataset col- lected over time from many policies, not the perfor- mance of any particular policy. #Successtul Sploeces Imporsonated Eptcodes
7.4 Distributed Asynchronous System
Original saccesins
imsprrsraain wwdddrrrrdddddoooooooooooo BACCEODRE
Fig.10 10 provides overview of our large scale distributed Multi-Task Reinforcement Learning system.
205k
8 Reward Specification with Multi-Task Success Detector
20k
Training a visual success detector is iterative process, as new task initially has no data to train from. We have two strategies to efficiently create an initial SD training dataset for a new task. 1) We collect 5Hz videos from different camera angles where every frame of the video human is demonstrating task success, and then short video demonstrating failure (sce exam-
Practical effect of task impersonation for successful outcomes. Dark blue indicates data specif- ically collected for a task; light blue indicates episodes impersonated from some other tasks which happen to be success for the target task.
Figure 12:
15

---

Overhood Camera
Rith Camera
Loft Camera
place- wnw.aaoryrheerre SUCCRIS essompless
place onywihrre fullure stamples:
Figure 13: SD training images. Each row represents a set images captured at the same time that are fed into the SD model. These images demonstrate our train-time SD data augmentation process they have been distorted via cropping. brightening, rotat- ing, and superimposing shadows.
Video frames for the place-anywherr task. Success and failure videos are iteratively cap- tured in pairs to mitigate correlations with spurious workspace features such hands of the user, back- grounds, bins, and distractor objects.
Figure 14:
ples in Fig. 14. Note that the user shows the de- sired and non-desired outcome of the task, not to be confused with demonstrations of how the task needs to be done.
The intention here is to de-correlate spurious parts of the scene from task-specifics. This process is repeated for approximately 30 minutes per task. 2) We relabel data from a policy that occasionally generated success for the new task (c.g., relabel lift-any data for lift-carrot task.).
The user would then change the lighting. switch out the objects and background, and then collect another pair of example videos (see Fig. 14 for example one video where there always something on plate being moved around paired with another video where there is never anything on plate).
Once the initial SD is trained, we can train RL policy, and begin on-policy collection. We continue to label on-policy data which keeps coming for the new task until SD is reliable. Table shows false positive and false negative error rates holdout data for the SD model used in our ablations. Our holdout data consisted of all images from particular robot.
During the SD training process, the data is artificially augmented to improve generalization, which involves cropping, brightening, rotating, and superimposing random shadows onto the images.
Fig. 13 shows training images after these distortions have been applied. Our success detector model trained using supervised learning, where we balance the data between success and failures as well as tasks. We use the architecture that is based on that from [67] with the exception of the action conditioning as not needed for this classification task. For each task the network outputs the probability representing whether given state was a success or failure for the corresponding task. The model receives three images as an input that come from an over-the-shoulder camera (same image RL policy), and two additional side cameras. These side camera images only used by the SD model, not the RL model. The additional cameras ensured that the task goals would be unambiguous, with single camera, it was often difficult for human to discern from an image whether or not the task had succeeded.
A breakdown of the labelled SD training data is provided in Fig. 15. While training SD, we incor- porated data sharing logic based on task feasibility. For example any success for lift-carrot would also be marked as failure for all other instance lifting tasks, and as a success for lift-any. In this manner, the original set of labelled data shown in Fig. 15 could act effectively as a much larger dataset for all tasks, where successes of one task often worked an interesting negatives for other
16

---

tasks. Additionally we balanced the proportion of success and failure examples per task seen by the model during training.
9 Robot Setup
In order for our system to be able to learn a vision- based RL policy that can accomplish multiple tasks, we need to collect a large, diverse, real-robot dataset that represents data for various tasks.
Lebdled Succem Champ
To achieve this goal, we set up an automated, multi- robot data collection system where each robot picks a task Ti to collect the data for. Collected episode stored on disk along with the T; bit of informa- tion. Our learning system can then use this episode collected Ti for train ing MT-Opt data impersonation algorithm. Once the episode is finished, our data collection system de- cides whether to continue with another task or per- form an automated reset of the workspace.
of other tasks utiliz-
200
In particular, we utilize 7 KUKA IIWA arms with two-finger grippers and 3 RGB cameras (left, right, and over the shoulder). In order to be able to au- tomatically reset the environment, we create an ac- tuated resettable bin, which further allows us to au- tomate the data collection process. More precisely, the enviromment consists of two bins (with the right bin containing all the source objects and the left bin containing a plate fixture magnetically attached anywhere on the workbench) that are connected via a motorized hinge so that after an episode ends, the contents of the workbench can be automatically shuffled and then dumped back into the right bin to start the next episode. Fig. 16 depicts the physi- cal setup for data collection and evaluation. This data collection process allows us to collect diverse data at scale: 24 hours per day, 7 days a week across multiple robots.
Counts of labelled SD training data by task and outcome. This data was generated ei- ther from human video demonstration, or by la- belling terminal images from episodes produced by robot. Note, that not all of the negatives were hand-labelled. As we may know dependencics be- tween the tasks, c.g. that a success for lift-carrot is always failure for lift- tift-banana, we can auto- matically generate negative examples. Similarly, all successes for the semantic lifting tasks are also successes for the lift-any task.
Figure 15:
One episode has 10 steps on average, taking 25 seconds to be generated on a robot, including environment reset time. This accounts to Re 3300 episodes/day collected on single robot, or s 23K episodes/day collected across our flect of 7 robots.
9.1 Details of Data Collection to bootstrap a Multi-Task System
This section contains more details on the data collection process introduced in Section 9.1. Real world robot data is noisy. For this project nearly 800,000 episodes were collected through the course of 16 months. The data was collected over different:
1. Locations: Three different physical lab locations.
2. Time of day: Robots ran as close to 24x7 as we could enable.
3. Robots: 6-7 KUKAs with variations in background, lighting, and slight variation in camera pose.
4. Success Detectors: We iteratively improved our success detectors.
5. RL training regimes: We developed better training loops parameeasseeters and architec- tures time went on.
6. Policies: Varied distribution of scripted, epsilon greedy, and on-policy data collection over time.
Our data collection started in an original physical lab location, was paused due to COVID-19, and the robots were later setup at different physical lab location affecting lighting and backgrounds.
17

---

Task Name
#Eps.
QT-Opt
flae rand QT-Optt MultiTask
flom rebal
fia rand
DataShare
MultiTaak
lift-any
635K
0.88
0.94
0.85
0.62
lift iat-banana
9K
0.04
0.13
0.38
0.09
lift-botte
11K
0.02
0.16
0.66
0.15
lift-sausage
5K
0.02
0.10
0.38
0.15
lift-milk
6K
0.01
0.13
0.42
0.13
lift-box
6K
0.00
0.12
0.16
0.08
lift-can
6K
0.01
0.16
0,46
0.07
laft-carrot
80K
0.71
0.41
0.72
0.37
place-any
30K
N/A
0.86
0.74
0.30
place-bottom
5K
N/A
0.43
0.57
0.30
place-top rigr-tt
4K
N/A
0.16
0.55
0.08
place-top-left
4K
N/A
0.23
0.75
0.19
Min
0.00
0.10
0.16
0.07
25-th percentile
0.00
0.13
0,41
0.09
Median
0.01
0.16
0.56
0.15
Mean
0.14
0.32
0.55
0,21
75-th percentile
0.03
0.42
0.73
0.30
Max
0.88
0.94
0.85
0.62
Mean (low data)
0.01
0.18
0.42
0.13
Task Name
Fm rebal
Taan 0.15 rebal
Taan, rand
/aann. rebal (ours)
lift-any
0.95
0.80
0.88
0.89
lift- lift-banana
0.30
0.58
0.62
0.33
lift-bottle
0.48
0.68
0.55
0.69
tift-sausage
0.39
0.42
0.28
0.38
lift-milk
0.27
0.27
0,52
0.51
lift-box
0.22
0.12
0.28
0.29
lift-can
0.28
0.47
0.43
0.43
lift-carrot
0.75
0.52
0.71
0.70
place-any
0.24
0.83
0.57
0.85
blace-bottom
0.02
0.62
0.17
0.87
place-top-right
0.10
0.26
0.27
0.54
place-top-left
0.16
0.39
0.22
0.53
Min
0.02
0.12
0.17
0.29
25-th percentile
0.20
0.36
0.28
0.42
Median
0.28
0.50
0.48
0.54
Mean
0.35
0.49
0.46
0.58
75-th percentile
0.41
0.64
0.58
0.74
Max
0.95
0.83
0.88
0.89
Mean (low data)
0.21
0.36
0.32
0.5
Table Quantitative evaluation of MT-Opt with different data impersonation and re-halancing strategies. This table reports performance of different models on the ablation tasks, trained on identical offline dataset, with identical computation budget, and evaluated executing 100 attempts for each task for each strategy on the real robots (totaling to 12* 10007=8400 evaluations). In all cases shared policy for all 12 tasks is learned. The difference across the strategies is in the way the data impersonated (expanded), and in the way the imperson- ated data is further re-balanced. The last column is best strategy featuring skill-level data impersonation further data re-balancing. This strategy outperforms other strategics on many different percentiles across all 12 tasks; however the effect of that strategy is even more pronounced for the tasks having scarce data, e.g. lift-can, lift-box. place-top-right see Mean (low data) statistic. The column #2 indicates the number of episodes which were collected for each task.
18

---

Initially scripted policies were run collecting data for the lift-anything and place-anywher tasks. Once performance of our learned policy for these tasks out-performed the scripted policy we shifted to mix of epsilon greedy and pure on-policy data collection. The majority of our episodes were collected for the lift-anything and place-anywhere tasks with learned policies. is worth mentioning that over the course of data collection many good and bad ideas where tried and evaluated via on-policy collection. All of these episodes are included in our dataset. Additional tasks being incorporated over time.
After we had policy capable of the lift-anything and specific variations of pick and place tasks where either a specific object needed to be picked, or an object needed be placed specific location on the plate. this point, our data collection process consisted of executing a randomly selected pick task followed by randomly selected place task.
tasks we introduced more
As a result of the collection process described above, we were left with a 800,000+ episode offline dataset, very diverse along tasks, policies, success rate dimensions.
10 Details for real world experiments
The robot workspace setup for the 12 task ablations shown in Fig 17. Table 4 summarizes studies of 7 different data impersonation and re- re-balancing strategies for 12 tasks. The last column features the model which average outperforms other strate- gies. Note that this strategy not the best across the board. For example, due to big imbalance of our offline dataset, the native data management strat- egy (column #3) yields best performance for the over represented tasks, but very bad performance for un- derrepresented tasks.
Shouder Coneras
Overvaal RON
Actatedd Bis Ragnnk Fidunas
Robot workspace consisting of an overhead camera (red), two over the shoulder cameras (brown), and a pair articulated reset- table bins with a plate fixture that can be magnet- ically attached to the bin (blue).
Figure 16:
Figure 17: Representative evaluation scenes used for ablation experiments. Contains one of three different color plates. And nine graspable objects: One of each object from our seven object cate- gories with two extra toy food objects sometimes from the seven object categories, sometimes not.
19


Reviews:
[{"review_id": "vAiSoqxYZYS", "reviewer": "Reviewer_UVT2", "summary": "\nThe authors propose MT-Opt, a method for scaling multi-task robot learning trained on an \"arm farm\" of real robots. This has two steps: (1) end user specification of the goal state of a task (via demonstrations), and (2) multi-task reinforcement learning.\n\nThey train a success detector detector based on these human user demonstrations, then describe a curriculum-like system for collecting additional data, using data from easy tasks to bootstrap harder ones, and rebalancing data to make sure it can learn even these difficult problems. Very importantly, they also use a relabeling strategy so that examples can be used to train multiple tasks simultaneously, which allows them to collect data using these simple policies (\"grasp anything on the table\", for example), and label it for more specific tasks (grasping a specific object, for example).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "On the positive side, the system is able to perform a many tasks in a 2d pick and place environment, and shows both training and task specification that can be scaled on a real robot. Using simple tasks (\"pick up anything\") to collect data that can then be used to learn a hard task (\"pick up the sausage\") is a good, simple idea, that I think could be useful for a lot of people.\n\nOn the negative side, though, I have a few serious issues with their method. It does not really seem to scale all that well, taking a very large amount of data and being quite restricted in terms of what sorts of tasks it can represent. Basically everything is a 2D pick and place task. And it seems like the tasks are very similar -- \"lift banana\" vs. \"lift sausage,\" for example. It is not demonstrated on a really wide range of scenarios.\n\nIt would be great to compare to transporter nets, another paper that works on a wide range of 2d object pick and place tasks, but which seems to need much less hardware. The high level concept of providing task supervision from human demonstrations has also appeared before, e.g. in Concept2Robot [57]. The authors show doing so in a continuous way, which is interesting.\n\nOn the positive side, MT-Opt allows for sharing data between tasks, and they show an improvement by doing so. I thought this was the most interesting part of the paper.\n\nHowever, this part of the paper is not made very clear. For example, the explanation of \"task impersonation\" seems very important to what the paper's unique contributions are over the state of the art. However, the explanation is delegated to the appendix. I think it's fine to include lots of details in the appendix, but the paper should still stand on its own. Task impersonation appears to be used to create essentially hard negative examples, which is interesting.\n\nI also like the fact that the success detector can label multiple successes for different tasks at the end of a trial; it seems like they do not need to limit use of the success detector (SD) to episode end, so I wonder if it would change much. Maybe this limitation is due to the training method for the SD? How could the SD generalize to tasks not obvious from a single top-down image? Would this affect the training process?\n\nI also thought this had its weaknesses, though. If the SD is trained online,  couldn't it end up being biased towards a subset of the original data?\n\nFinally, a more philosophical concern: if this approach only works on strongly goal-directed tasks -- those where success or failure is determined by a single image -- why should we use this method, over a method like [48] or transporter nets, which takes advantage of this fact to learn much more efficiently?\n\nFinally, I think the \"learn new skill\" example was interesting, although the approach seems fairly straightforward: hope that you have a good enough policy that you can collect enough positive examples to start fine-tuning. This again seems not very scalable, since with a higher DOF task space (say, a kitchen), would this even be feasible?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think while the goal of this paper is admirable, we've seen a lot of very high quality results in 2d pick and place tasks, and this paper has enough gaps that I am leaning against publication. Some of the details are still unclear to me, especially what advantages this system has compared to related work. A lot of their contributions (such as using simple tasks to bootstrap harder ones) seem well explored in the literature, and all the tasks are very simple.\n\nIn the end, the paper is attempting to solve an important problem, but it shows results using a tremendous amount of time, data, and engineering resources on what seems to be a fairly straightforward suite of tasks, which somewhat undermines their goals.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "\n- It seems like the tasks are all very similar, and I'm not sure what the advantage is over existing work (or just using grasp prediction)\n- In addition, the method only really applies to RL with clear goals - no explanation of how it could work on dynamic tasks, for example.\n- It would be great to explain what the advantages of this method are and how it would scale past the cases show. \n- Information about the robot's action space is missing; this makes it hard to tell how well the system actually scales.\n- Discussing the applicability of the algorithms to more complex and longer horizon examples would be nice. Could the authors do some simulation experiments on a more complex environment? If the environment is so simple that random pick-and-place can collect positive examples, I think it's hard to make any strong claims about the method."}}, {"review_id": "r8jPTleQaDm", "reviewer": "Reviewer_vENi", "summary": "This paper proposes a multi-task reinforcement learning framework, which can be considered as a multi-task expansion of QT-Opt. It features an continuous, iterative process of task definition, data collection and RL training. The key components that enable the multi-task data collection and RL training are the success detector, task impersonation and data rebalancing. The success detector along with task impersonation determine what task to collect data for in the next iteration and also whether to include a particular episode from other tasks for training the given task. Data rebalancing makes sure that the number of data samples are balanced across tasks, and the successful/unsuccessful data are balanced within each task.\nThrough extensive experiments on multiple real world robotic manipulation tasks, the paper showed that the proposed framework significantly improved the success rates across many tasks when compared with baselines that either learn from each individual task or do not use all components in the proposed framework. This framework allows the agent to learn to more efficiently solve harder tasks by bootstrapping from easier tasks. It also makes it possible to define new tasks that are not in the original offline set and leverage data from other tasks to solve the new task. Ablation studies showed that both the data sharing and representation sharing are beneficial. \n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. This paper is well written and pleasant to read\n2. Novel data management strategy that allows scalable data collection and sharing\n3. Extensive real-world experiments\n4. The accompanying video nicely articulates the key ingredients in the paper\n\nWeaknesses/Suggestions\n\nI find it to be a strong paper with enough details. I don’t have many suggestions to make. Just some clarification questions:  \n1. Task ID is a one-hot vector. Is it of fixed length? Is it a problem when we are in a continual setting where new tasks are being added?  \n2. Line 226-227, “allow the individual tasks to be ordered sequentially, where one task is executed after another ...” I’m a bit confused here. Is there some fixed ordering of the tasks? Or is the ordering random?  \n3. Line 390, “pre-trained MT-Opt policy for collecting the data for the new task”. Is it pre-trained on all 12 tasks, or on the lift-any and place-any tasks as Line 368 seems to suggest?  \n4. Line 624 in appendix, how is a skill defined and detected?  \n5. Figure 8, is the input image a concatenation of three grayscale images from the three RGB cameras? If so, it would improve clarity if it’s stated or illustrated.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I find it to be a strong system paper that’s relevant to the robot learning community at large and hence I recommend acceptance. The proposed multi-task RL framework is shown to be more effective than the baselines, backed up by large-scale real-world experiments. Although the multi-robots hardware setup might be too expensive for the framework to be widely adopted, it shows the great potential of scaling up existing RL algorithms for multi-task robot manipulations. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please clarify the items discussed in the weaknesses/suggestions section. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "i2kzGo3RNVQ", "reviewer": "Reviewer_xsVH", "summary": "This submission desribes MT-Opt, a multi-task robot-learning framework whic\ntrains a single model to solve all tasks, using semantically driven data\nsharing and re-balancing strategies.  The entire framework bootstraps from an\ninitial collection of labeled human demonstrations, but proceeds by using the\nrobot attempts as novel demonstrations and training data.\n\nThe system is roughly divided into 3 components:  Task definition, which\nconsists in the labeling of both human and robot demonstrations.  Data\nCollection, in which a series of robots is assigned tasks to perform, whose\nexecutions are re-used as new training data.  Data Sharing and RL Training,\nwhich determines how the trajectories which are labeles for a specific task can\nbe used as training data for similar tasks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n  * Addresses a complex problem, very relevant to the robot-learning community.\n  * Evaluation is performed on multiple aspects of the system, rather than\n    exclusively focusing on the overall performance as a whole.\n  * Very well written and pleasant to read.\n\nWeaknesses:\n  * While the general outline of the system is clear, a number of key details\n    are missing, which makes the submission seem incomplete and to some degree\n    non-reproducible.\n  * Only suitable for tasks whose success can be assessed by a final\n    observation, i.e., cannot handle \"execution modalities\" or specific ways of\n    performing the task (e.g. quickly, or carefully).\n  * The work is a bit incremental, as none of the components of MT-Opt seem to\n    be particularly original.\n  * Some claims seem to be partially embellished or not very close to the\n    actual implementation / evaluation details, e.g.:\n    * Although the system is described as sharing data belonging to semantically\n      similar tasks, in practice the tasks chosen are primarily limited to the\n      format lift-X and place-Y, and the semantic sharing seems to be very\n      limited, e.g. only between lift-any and lift-X.\n    * Although one of the main parts of the motivation was the ability to\n      define novel types of tasks, in practice some implementation details made\n      use of the pre-known 12 tasks, e.g., using one-hot encodings, even though\n      the tasks were not all introduced at the same time.\n    * In Fig.1, claims of potential zero-shot learning capabilities are\n      unsubstantiated, and the main motivation for the claim is very weak\n      (i.e., a human was able to reinterpret a robot demonstration as a novel\n      task).\n\nExtensions:\n* Tasks being represented as simple prepositions makes me wonder how easy it\n  really is to introduce and generalize to new tasks.  A straightforward\n  extension would be to consider tasks as predicates which accept arguments,\n  e.g., lift(any), lift(carrot), etc.  This way, a new lift-object task would\n  not be defined as a completely new task label, but simply as a new object\n  representation, paired with an already trained lift() task predicate\n  representation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My general impression of the work is positive, however my current\nrecommendation is a weak reject, in light of certain limitations and issue, not\nonly with the work itself, but also with its presentation.  Primarily, these\nconcern missing details and certain embellished claims which make the actual\nsystem seem fancier than it really is (e.g., when it comes to the types of\ntasks that it is able to solve, and how tasks can be grouped into semantic\ngroups, which in practice seems very limited).  To be clear, I think the system\nitself is a reasonable (incremental) contribution, in particular given the\ngreat overall difficulty of building such a system to begin with; my main\nconcern is not the presence of these limitations, but the fact that the text\nseems to be trying to sweep these under the rug, in a manner of speaking.  I\nthink the text should be more honest and open to discussing its own\nlimitations.\n\nMy current recommendation is for the submission as it stands;  I will be happy\nto modify it in light of a author response and/or updated manuscript.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Poor: Limited knowledge of the area"}, "extra_text": {"issues": "Major:\n\n* \"a policy takes a camera image and a one-hot encoding of the task ...\"  How\n  can one-hot encodings of non-predefined tasks?  If you extend you task space,\n  you have to extend the encoding space.  How does this work?  (I would find no\n  issue if a task embedding were to be used, rather than one-hot encoding).\n  Similar issue with the predefined probability over tasks in eq.(1).\n\n* What are the details of the data rebalancing?  This seems to be a major part\n  of the submission which seems to be not explained.\n\n* The main text mentioned semantic similarity between tasks, but the experiment\n  seems lacking.  Aside from the similarity between lift-any and lift-X (for\n  any given X), are there other examples?  is there any semantic similarity\n  being explored, e.g., between lift-X and lift-Y?  or between lift-X and\n  place-X or place-Y?  Which lift- and place- tasks are semantically similar?\n\n* In Fig.7 (and other parts of the text), there is a claim that the system is\n  able to perform other novel tasks.  In practice, it seems that these novel\n  tasks are simply alternative labels made up by humans, who are allowed to be\n  as fancyful as they please since most of these new tasks are never evaluated\n  in the real system.\n\n  In fact, one can notice a very distinct difference between the real\n  evaluation tasks (only composed of lift-X and place-Y), and the made up\n  \"novel\" tasks (which include very sophisticated tasks such as cover-X,\n  serve-y, chase-Z, etc).\n\n  It seems excessive (and mildly misleading) in fig.7 to take this much space\n  to show so many pictures of novel tasks when the main take-away is simply\n  that robot demonstrations may be reinterpreted as new tasks, despite those\n  specific novel tasks not being put to the test.\n\n* Lines 310-312 imply that data re-balancing is only applied to f_I_skill, and\n  not f_I_all.  If my understanding is correct, the comparison between\n  f_I_skill and f_I_all is extremely unfair, as the re-balancing (rather than\n  the data-sharing strategy) may be the important component.\n\nMid:\n\n* In fig 4, why make such a distinction between pure on-policy (which I assume\n  is greedy) and e-greedy policies?  What value epsilon is used?\n\n* In fig.6, are all methods given the same amount of data / experience and/or\n  training time?\n\n* In fig.7, top row, how do we know this is actually an intentional\n  repositioning action, rather than a failed first grasp which bumps and moves\n  the carrot?\n\nMinor:\n\n* In line 31, the sentence \"Can we instead amortize the cost of learning this\n  repertoire over multiple skills, where the effort needed to learn whole\n  repertoire is reduced, easier skills serve to facilitate the acquisition of\n  more complex ones, and data requirements, though still high overall, become\n  low for each individual behavior?\" is a mouthful and hard to parse/read.\n\n* In fig.4, it would help if the order in the legend matches the order in the\n  bar plots.\n\n* In the legend of fig.6, what does [36] mean?\n\n* I assume the numbers of episodes in section 5.5 are approximations, since I\n  find it hard to believe that the number of successes and failures for any\n  given task happens to be clean multiples of 100.  Since all numerical figures\n  are provided, I think you should somehow indicate that this is an\n  approximation, or simply provide the exact numbers.\n\n* Fig.5 mentions \"ablation tasks\", which is an unclear description;  as I\n  understand, these are the main 12 tasks which the system is primarily trained\n  on, so something like \"evaluation tasks\" would be more suitable.\n\n* \"In ... f_I_all, the data is expanded only for the class of tasks having\n  similar visuals, dynamics and goals\".  How do the tasks differ in visuals and\n  dynamics?  Seems like they all have the same visuals and dynamics.  In\n  practice, the semantics are only based on the goal, i.e. whether both task\n  are lift-X or place-Y."}}, {"review_id": "Lves3EPLBJ8", "reviewer": "Reviewer_k7mA", "summary": "The paper describes a learning framework Mt-Opt to scale up learning of object arrangement tasks on physical robots in a table top setup. \nThe paper presents learning and generalization results from data collected over 1.5 years on 7 robots. There are several techniques used to scale up the data collection, from automatic resets, to starting with scripted policies, to redistributing data using known relationships between tasks. The results demonstrate that the framework allows robots to share data and representations, allowing them to learn novel tasks faster.   ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The success and failure task annotation methods using images is much easier than annotating policies.\n2) The length of the robot experiment is impressive, and demonstrates the challenges in learning general purpose policies and representations. \n3) The data rebalancing techniques showing that we cannot just dump all available data at the same time.\n4) The experiments are very through. \n\nWeakness:\n1) This is a lot of data over 7 robots, and 1.5 years. The paper demonstrates sample complexity issues with robot learning. One would expect more zero shot results. \n2)  There is significant semantic information present in the data collected. From labelling general and specialized tasks to teaching tasks in the order of first bootstrapping pick tasks from a scripted policy and then learning to place. \n3) I am not sure if the learning is faster or more robust when compared to an engineer scripting these pick and place behaviors with conventional vision pipelines such as those available with Cognex.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I appreciate the length of the experiment and thoroughness of the results of this paper. The work demonstrates that more fundamental advances in sample complexity and the explore-exploit problems are needed.   ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "1) Results of trying zero shot: What percentage of novel tasks were solved zero-shot if any. \n2) Would the data be released?\n3) A through list of assumptions about the object arrangement problem being solved. For example resets are available here, which are not a requirement in general object arrangement scenarios.  "}}]

Author rebuttal:

Meta-review: Summary: This paper presents a system for multi-task robot learning at scale that combines several existing ideas such as success classifiers etc. Although the individual components and insights are not novel, the overall system has clearly involved significant thought and work.

However, AC and Reviewers find several issues with the paper:
1. Multi-task: Is it really?  In the experimental setup, the tasks seem quite arbitrary. What makes a task different from a different task? It is unclear why lift-banana needs to be a different task than lift-carrot. Shouldn't 'lift' be the task and different objects are just different instances of lifting. Overall, the definition has been chosen just to show lots of different tasks. AC finds this nomenclature misleading and agrees with reviewers there is lot of embellishment and projection of fanciness.
2. Furthermore since both picking and lifting require grasping, it would be nice to comparisons against state of art grasping methods. This well help address if multi-task robot learning is really necessary to learn how to perform such simplistic grasping behaviors compared to off the shelf controller+planners.
3. There is no takeaway/singular insight in this work. As acknowledged in the introduction and related works section, most of the components have been studied in detail before. Success classifiers have been used in several prior work, most notably in work by Ebert and Finn [1], where success classifiers were used in combination with model-based RL. Unfortunately, the paper fails to provide its technical contribution at all.
4. Applicability to dynamic tasks
5. Reproducibility: Given the scale of data used for the project and the arbitrary amount of data collected + mode of collection, I wonder if this experiment can be ever reproduced. Will the authors release all the data?


===

Post Rebuttal:

There was substantial discussion between reviewers and authors.  AC still finds issue with the task definition. Lifting a carrot and banana might be different but they are just different instance of same category (Note two chairs also look different but they are still part of same category called chair). AC believes the paper claims much more than it actually achieves.

AC is convinced that there is definitely value in building systems even though individual ideas might have existed. AC is highly appreciative of the hardware experiments while lot of people only focus on simulation. Overall, AC is inclined to accept but would highly recommend authors to include discussion on tasks, have more baselines and even boil down some of the claims [some of them seem over the board frankly].

In terms of reproducibility: the author promise release of data...the AC would suggest releasing all code as well so people with same robot can reproduce the results [including low-level code if possible].


Decision: Accept (Poster)