Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: 
The authors propose MT-Opt, a method for scaling multi-task robot learning trained on an "arm farm" of real robots. This has two steps: (1) end user specification of the goal state of a task (via demonstrations), and (2) multi-task reinforcement learning.

They train a success detector detector based on these human user demonstrations, then describe a curriculum-like system for collecting additional data, using data from easy tasks to bootstrap harder ones, and rebalancing data to make sure it can learn even these difficult problems. Very importantly, they also use a relabeling strategy so that examples can be used to train multiple tasks simultaneously, which allows them to collect data using these simple policies ("grasp anything on the table", for example), and label it for more specific tasks (grasping a specific object, for example).
strengths_and_weaknesses: On the positive side, the system is able to perform a many tasks in a 2d pick and place environment, and shows both training and task specification that can be scaled on a real robot. Using simple tasks ("pick up anything") to collect data that can then be used to learn a hard task ("pick up the sausage") is a good, simple idea, that I think could be useful for a lot of people.

On the negative side, though, I have a few serious issues with their method. It does not really seem to scale all that well, taking a very large amount of data and being quite restricted in terms of what sorts of tasks it can represent. Basically everything is a 2D pick and place task. And it seems like the tasks are very similar -- "lift banana" vs. "lift sausage," for example. It is not demonstrated on a really wide range of scenarios.

It would be great to compare to transporter nets, another paper that works on a wide range of 2d object pick and place tasks, but which seems to need much less hardware. The high level concept of providing task supervision from human demonstrations has also appeared before, e.g. in Concept2Robot [57]. The authors show doing so in a continuous way, which is interesting.

On the positive side, MT-Opt allows for sharing data between tasks, and they show an improvement by doing so. I thought this was the most interesting part of the paper.

However, this part of the paper is not made very clear. For example, the explanation of "task impersonation" seems very important to what the paper's unique contributions are over the state of the art. However, the explanation is delegated to the appendix. I think it's fine to include lots of details in the appendix, but the paper should still stand on its own. Task impersonation appears to be used to create essentially hard negative examples, which is interesting.

I also like the fact that the success detector can label multiple successes for different tasks at the end of a trial; it seems like they do not need to limit use of the success detector (SD) to episode end, so I wonder if it would change much. Maybe this limitation is due to the training method for the SD? How could the SD generalize to tasks not obvious from a single top-down image? Would this affect the training process?

I also thought this had its weaknesses, though. If the SD is trained online,  couldn't it end up being biased towards a subset of the original data?

Finally, a more philosophical concern: if this approach only works on strongly goal-directed tasks -- those where success or failure is determined by a single image -- why should we use this method, over a method like [48] or transporter nets, which takes advantage of this fact to learn much more efficiently?

Finally, I think the "learn new skill" example was interesting, although the approach seems fairly straightforward: hope that you have a good enough policy that you can collect enough positive examples to start fine-tuning. This again seems not very scalable, since with a higher DOF task space (say, a kitchen), would this even be feasible?

summary_of_recommendation: I think while the goal of this paper is admirable, we've seen a lot of very high quality results in 2d pick and place tasks, and this paper has enough gaps that I am leaning against publication. Some of the details are still unclear to me, especially what advantages this system has compared to related work. A lot of their contributions (such as using simple tasks to bootstrap harder ones) seem well explored in the literature, and all the tasks are very simple.

In the end, the paper is attempting to solve an important problem, but it shows results using a tremendous amount of time, data, and engineering resources on what seems to be a fairly straightforward suite of tasks, which somewhat undermines their goals.
issues: 
- It seems like the tasks are all very similar, and I'm not sure what the advantage is over existing work (or just using grasp prediction)
- In addition, the method only really applies to RL with clear goals - no explanation of how it could work on dynamic tasks, for example.
- It would be great to explain what the advantages of this method are and how it would scale past the cases show. 
- Information about the robot's action space is missing; this makes it hard to tell how well the system actually scales.
- Discussing the applicability of the algorithms to more complex and longer horizon examples would be nice. Could the authors do some simulation experiments on a more complex environment? If the environment is so simple that random pick-and-place can collect positive examples, I think it's hard to make any strong claims about the method.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This submission desribes MT-Opt, a multi-task robot-learning framework whic
trains a single model to solve all tasks, using semantically driven data
sharing and re-balancing strategies.  The entire framework bootstraps from an
initial collection of labeled human demonstrations, but proceeds by using the
robot attempts as novel demonstrations and training data.

The system is roughly divided into 3 components:  Task definition, which
consists in the labeling of both human and robot demonstrations.  Data
Collection, in which a series of robots is assigned tasks to perform, whose
executions are re-used as new training data.  Data Sharing and RL Training,
which determines how the trajectories which are labeles for a specific task can
be used as training data for similar tasks.
strengths_and_weaknesses: Strengths:
  * Addresses a complex problem, very relevant to the robot-learning community.
  * Evaluation is performed on multiple aspects of the system, rather than
    exclusively focusing on the overall performance as a whole.
  * Very well written and pleasant to read.

Weaknesses:
  * While the general outline of the system is clear, a number of key details
    are missing, which makes the submission seem incomplete and to some degree
    non-reproducible.
  * Only suitable for tasks whose success can be assessed by a final
    observation, i.e., cannot handle "execution modalities" or specific ways of
    performing the task (e.g. quickly, or carefully).
  * The work is a bit incremental, as none of the components of MT-Opt seem to
    be particularly original.
  * Some claims seem to be partially embellished or not very close to the
    actual implementation / evaluation details, e.g.:
    * Although the system is described as sharing data belonging to semantically
      similar tasks, in practice the tasks chosen are primarily limited to the
      format lift-X and place-Y, and the semantic sharing seems to be very
      limited, e.g. only between lift-any and lift-X.
    * Although one of the main parts of the motivation was the ability to
      define novel types of tasks, in practice some implementation details made
      use of the pre-known 12 tasks, e.g., using one-hot encodings, even though
      the tasks were not all introduced at the same time.
    * In Fig.1, claims of potential zero-shot learning capabilities are
      unsubstantiated, and the main motivation for the claim is very weak
      (i.e., a human was able to reinterpret a robot demonstration as a novel
      task).

Extensions:
* Tasks being represented as simple prepositions makes me wonder how easy it
  really is to introduce and generalize to new tasks.  A straightforward
  extension would be to consider tasks as predicates which accept arguments,
  e.g., lift(any), lift(carrot), etc.  This way, a new lift-object task would
  not be defined as a completely new task label, but simply as a new object
  representation, paired with an already trained lift() task predicate
  representation.
summary_of_recommendation: My general impression of the work is positive, however my current
recommendation is a weak reject, in light of certain limitations and issue, not
only with the work itself, but also with its presentation.  Primarily, these
concern missing details and certain embellished claims which make the actual
system seem fancier than it really is (e.g., when it comes to the types of
tasks that it is able to solve, and how tasks can be grouped into semantic
groups, which in practice seems very limited).  To be clear, I think the system
itself is a reasonable (incremental) contribution, in particular given the
great overall difficulty of building such a system to begin with; my main
concern is not the presence of these limitations, but the fact that the text
seems to be trying to sweep these under the rug, in a manner of speaking.  I
think the text should be more honest and open to discussing its own
limitations.

My current recommendation is for the submission as it stands;  I will be happy
to modify it in light of a author response and/or updated manuscript.
issues: Major:

* "a policy takes a camera image and a one-hot encoding of the task ..."  How
  can one-hot encodings of non-predefined tasks?  If you extend you task space,
  you have to extend the encoding space.  How does this work?  (I would find no
  issue if a task embedding were to be used, rather than one-hot encoding).
  Similar issue with the predefined probability over tasks in eq.(1).

* What are the details of the data rebalancing?  This seems to be a major part
  of the submission which seems to be not explained.

* The main text mentioned semantic similarity between tasks, but the experiment
  seems lacking.  Aside from the similarity between lift-any and lift-X (for
  any given X), are there other examples?  is there any semantic similarity
  being explored, e.g., between lift-X and lift-Y?  or between lift-X and
  place-X or place-Y?  Which lift- and place- tasks are semantically similar?

* In Fig.7 (and other parts of the text), there is a claim that the system is
  able to perform other novel tasks.  In practice, it seems that these novel
  tasks are simply alternative labels made up by humans, who are allowed to be
  as fancyful as they please since most of these new tasks are never evaluated
  in the real system.

  In fact, one can notice a very distinct difference between the real
  evaluation tasks (only composed of lift-X and place-Y), and the made up
  "novel" tasks (which include very sophisticated tasks such as cover-X,
  serve-y, chase-Z, etc).

  It seems excessive (and mildly misleading) in fig.7 to take this much space
  to show so many pictures of novel tasks when the main take-away is simply
  that robot demonstrations may be reinterpreted as new tasks, despite those
  specific novel tasks not being put to the test.

* Lines 310-312 imply that data re-balancing is only applied to f_I_skill, and
  not f_I_all.  If my understanding is correct, the comparison between
  f_I_skill and f_I_all is extremely unfair, as the re-balancing (rather than
  the data-sharing strategy) may be the important component.

Mid:

* In fig 4, why make such a distinction between pure on-policy (which I assume
  is greedy) and e-greedy policies?  What value epsilon is used?

* In fig.6, are all methods given the same amount of data / experience and/or
  training time?

* In fig.7, top row, how do we know this is actually an intentional
  repositioning action, rather than a failed first grasp which bumps and moves
  the carrot?

Minor:

* In line 31, the sentence "Can we instead amortize the cost of learning this
  repertoire over multiple skills, where the effort needed to learn whole
  repertoire is reduced, easier skills serve to facilitate the acquisition of
  more complex ones, and data requirements, though still high overall, become
  low for each individual behavior?" is a mouthful and hard to parse/read.

* In fig.4, it would help if the order in the legend matches the order in the
  bar plots.

* In the legend of fig.6, what does [36] mean?

* I assume the numbers of episodes in section 5.5 are approximations, since I
  find it hard to believe that the number of successes and failures for any
  given task happens to be clean multiples of 100.  Since all numerical figures
  are provided, I think you should somehow indicate that this is an
  approximation, or simply provide the exact numbers.

* Fig.5 mentions "ablation tasks", which is an unclear description;  as I
  understand, these are the main 12 tasks which the system is primarily trained
  on, so something like "evaluation tasks" would be more suitable.

* "In ... f_I_all, the data is expanded only for the class of tasks having
  similar visuals, dynamics and goals".  How do the tasks differ in visuals and
  dynamics?  Seems like they all have the same visuals and dynamics.  In
  practice, the semantics are only based on the goal, i.e. whether both task
  are lift-X or place-Y.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a multi-task reinforcement learning framework, which can be considered as a multi-task expansion of QT-Opt. It features an continuous, iterative process of task definition, data collection and RL training. The key components that enable the multi-task data collection and RL training are the success detector, task impersonation and data rebalancing. The success detector along with task impersonation determine what task to collect data for in the next iteration and also whether to include a particular episode from other tasks for training the given task. Data rebalancing makes sure that the number of data samples are balanced across tasks, and the successful/unsuccessful data are balanced within each task.
Through extensive experiments on multiple real world robotic manipulation tasks, the paper showed that the proposed framework significantly improved the success rates across many tasks when compared with baselines that either learn from each individual task or do not use all components in the proposed framework. This framework allows the agent to learn to more efficiently solve harder tasks by bootstrapping from easier tasks. It also makes it possible to define new tasks that are not in the original offline set and leverage data from other tasks to solve the new task. Ablation studies showed that both the data sharing and representation sharing are beneficial. 

strengths_and_weaknesses: Strengths
1. This paper is well written and pleasant to read
2. Novel data management strategy that allows scalable data collection and sharing
3. Extensive real-world experiments
4. The accompanying video nicely articulates the key ingredients in the paper

Weaknesses/Suggestions

I find it to be a strong paper with enough details. I don’t have many suggestions to make. Just some clarification questions:  
1. Task ID is a one-hot vector. Is it of fixed length? Is it a problem when we are in a continual setting where new tasks are being added?  
2. Line 226-227, “allow the individual tasks to be ordered sequentially, where one task is executed after another ...” I’m a bit confused here. Is there some fixed ordering of the tasks? Or is the ordering random?  
3. Line 390, “pre-trained MT-Opt policy for collecting the data for the new task”. Is it pre-trained on all 12 tasks, or on the lift-any and place-any tasks as Line 368 seems to suggest?  
4. Line 624 in appendix, how is a skill defined and detected?  
5. Figure 8, is the input image a concatenation of three grayscale images from the three RGB cameras? If so, it would improve clarity if it’s stated or illustrated.  

summary_of_recommendation: I find it to be a strong system paper that’s relevant to the robot learning community at large and hence I recommend acceptance. The proposed multi-task RL framework is shown to be more effective than the baselines, backed up by large-scale real-world experiments. Although the multi-robots hardware setup might be too expensive for the framework to be widely adopted, it shows the great potential of scaling up existing RL algorithms for multi-task robot manipulations. 
issues: Please clarify the items discussed in the weaknesses/suggestions section.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper describes a learning framework Mt-Opt to scale up learning of object arrangement tasks on physical robots in a table top setup. 
The paper presents learning and generalization results from data collected over 1.5 years on 7 robots. There are several techniques used to scale up the data collection, from automatic resets, to starting with scripted policies, to redistributing data using known relationships between tasks. The results demonstrate that the framework allows robots to share data and representations, allowing them to learn novel tasks faster.   
strengths_and_weaknesses: Strengths:
1) The success and failure task annotation methods using images is much easier than annotating policies.
2) The length of the robot experiment is impressive, and demonstrates the challenges in learning general purpose policies and representations. 
3) The data rebalancing techniques showing that we cannot just dump all available data at the same time.
4) The experiments are very through. 

Weakness:
1) This is a lot of data over 7 robots, and 1.5 years. The paper demonstrates sample complexity issues with robot learning. One would expect more zero shot results. 
2)  There is significant semantic information present in the data collected. From labelling general and specialized tasks to teaching tasks in the order of first bootstrapping pick tasks from a scripted policy and then learning to place. 
3) I am not sure if the learning is faster or more robust when compared to an engineer scripting these pick and place behaviors with conventional vision pipelines such as those available with Cognex.
summary_of_recommendation: I appreciate the length of the experiment and thoroughness of the results of this paper. The work demonstrates that more fundamental advances in sample complexity and the explore-exploit problems are needed.   
issues: 1) Results of trying zero shot: What percentage of novel tasks were solved zero-shot if any. 
2) Would the data be released?
3) A through list of assumptions about the object arrangement problem being solved. For example resets are available here, which are not a requirement in general object arrangement scenarios.
review_title: null

Meta-review: Summary: This paper presents a system for multi-task robot learning at scale that combines several existing ideas such as success classifiers etc. Although the individual components and insights are not novel, the overall system has clearly involved significant thought and work.

However, AC and Reviewers find several issues with the paper:
1. Multi-task: Is it really?  In the experimental setup, the tasks seem quite arbitrary. What makes a task different from a different task? It is unclear why lift-banana needs to be a different task than lift-carrot. Shouldn't 'lift' be the task and different objects are just different instances of lifting. Overall, the definition has been chosen just to show lots of different tasks. AC finds this nomenclature misleading and agrees with reviewers there is lot of embellishment and projection of fanciness.
2. Furthermore since both picking and lifting require grasping, it would be nice to comparisons against state of art grasping methods. This well help address if multi-task robot learning is really necessary to learn how to perform such simplistic grasping behaviors compared to off the shelf controller+planners.
3. There is no takeaway/singular insight in this work. As acknowledged in the introduction and related works section, most of the components have been studied in detail before. Success classifiers have been used in several prior work, most notably in work by Ebert and Finn [1], where success classifiers were used in combination with model-based RL. Unfortunately, the paper fails to provide its technical contribution at all.
4. Applicability to dynamic tasks
5. Reproducibility: Given the scale of data used for the project and the arbitrary amount of data collected + mode of collection, I wonder if this experiment can be ever reproduced. Will the authors release all the data?


===

Post Rebuttal:

There was substantial discussion between reviewers and authors.  AC still finds issue with the task definition. Lifting a carrot and banana might be different but they are just different instance of same category (Note two chairs also look different but they are still part of same category called chair). AC believes the paper claims much more than it actually achieves.

AC is convinced that there is definitely value in building systems even though individual ideas might have existed. AC is highly appreciative of the hardware experiments while lot of people only focus on simulation. Overall, AC is inclined to accept but would highly recommend authors to include discussion on tasks, have more baselines and even boil down some of the claims [some of them seem over the board frankly].

In terms of reproducibility: the author promise release of data...the AC would suggest releasing all code as well so people with same robot can reproduce the results [including low-level code if possible].


Decision: Accept (Poster)