Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper proposes a framework for learning robotic tasks from multiple datasets of varying quantity and quality. An agent architecture is presented that consists of a planner mapping (embedded) observations onto desired future (embedded) observations and an executor that serves as an inverse model and maps observed and desired embedded states onto actions. 
The transformer based architecture uses embeddings that are pre-trained on ImageNet and some of the high-quantity-low-quality datasets. Robotic data comprising actions is then used to further fine-tune the agent.
Experiments demonstrate effectiveness of the agent on a set of simulated benchmark tasks. Further, the paper demonstrates that relative position encodings appear to be superior to commonly used absolute position embeddings.

review: Overall, I think the paper does a good job motivating the problem of learning from a lot of lower quality and little high quality data. This is a relevant problem for the robot learning community. The targeted data categories and learning system are well described as are the experiments.
I have some concerns about the design of datasets in the experiments and some of the explanations of the approach.

line 105: ‘many of these trajectories are generated by activities that most people will not find meaningful’ - In the presented experiments D_vmt and D_ttd are generated from the same agent and tasks with action noise added in D_vmt. It would be interesting to study the hypothesis where D_vmt is in fact a set of tasks that are not goal-oriented and ‘meaningless’ or at least the goals do not coincide precisely with D_ttd.

line 181 ‘When a modality is missing, it is replaced by trainable placeholder vectors during embedding’ - Is there an intuition as to why you use trainable placeholders? How does it compare, for example, to feeding randomised inputs or using dropout? 

Equation (2) and surrounding text: I might be misreading this, but should the summation on the right-hand side start at t=T+1 instead of t=1+L ?

line 229: ‘[..], this involves deciding which part of PLEX to adapt’. - In the experiments, it seems that the last layers comprising 5% of the parameters are fine-tuned. Have you compared fine tuning to other different parts of the network or how did you ‘decide’ what to fine-tune in your system? 

line 236: ‘behaviour cloning [..] is equivalent to a stopgrad-free version of L_EX’ - Could you elaborate on this? It seems L_EX is taking as input the past and a future state and regresses on an action, while BC usually takes in only past states and a task conditioning. Do you suggest the latter and remove the planner entirely?

line 254: ‘standard deviation of 0.5’ - What are the units here and how does 0.5 relate to the distribution of actions taken by the scripted policy, i.e. your D_ttd? This relation seems to be important for interpreting how close D_ttd is to D_vmt or how much D_vmt indeed represents play-like data without goals.

line 291: In the experiments D_vmt is a noisy version of D_ttd generated on the exact same tasks and with the same agent (the scripted policy). One explanation as to why adding actions does not help, is that PLEX might be able to connect image and action from D_vmt and image to goal from D_ttd. It might be an artefact of your choice of D_ttd and D_vmt. It would be interesting to conduct this study with a D_vmt that does not resemble the exact same tasks from D_ttd.

As noted in the limitations, the presented approach currently uses data from a single robot whereas the approach is to leverage large amounts of data that potentially stem from other robots. Different robots may have different proprioception and action representations. How would your approach extend from a single action / proprioception representation?

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I recommend the paper for submission, because it is well motivated, the approach is explained clearly and the experiments examine the approach mostly well. My only concern is the choice of datasets in the experiments that might be overly simplistic. I would like to hear from the authors on this topic in the rebuttal.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper introduces PLEX (planning-execution), a transformer architecture with a task-specific planner and a subsequent executor that outputs actions. Each component is trained on different data: the planner is pretrained on a larger video dataset without access to actions and outputs a plan in a learned observational embedding space. The executor is pretrained on a smaller visuomotor dataset with observations and actions. MetaWorld experiments show that PLEX can complete some tasks zero-shot (without finetuning on demonstrations). Robosuite experiments show that PLEX's relative positional encoding enables better performance than the DT baseline.
review: Strengths:
* The PLEX architecture enables pretraining on different kinds of data, e.g. a large-scale video dataset and small-scale demonstrations.
* The experiments show strong performance for PLEX in zero-shot, finetuning, and training from scratch settings.

Weaknesses:
* The motivation for using multiple data sources (access to large-scale internet/human videos) does not apply well to most of the experiments. The experiments use pretraining data from the same simulation environment as the evaluation. PLEX+R3M does use internet data, but it is not being presented as the primary method. Further experiments using diverse, out-of-distribution pretraining data are needed to determine the benefits of this architecture.
* Some hyperparameters are different between MetaWorld and RoboSuite and between PLEX and DT. Is there motivation for the differences in number of layers (2 for PLEX Meta-World, 3 for PLEX Robosuite, and 4 for DT Robosuite)? The training steps per epoch is also different. 

Minor suggestions:
* The introduction is quite long and overlapping with some sections. The second paragraph (L27-39) is very similar to section 2.2 and could be shortened/removed from the intro. L61-75 feels like it belongs in related work.
* Figure 1 is hard to follow. The font size on the diagram is too small to read easily. The caption is perhaps too detailed, in addition to its small font size and lack of margins.
* Typo: L286 samll -> small
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The idea of enabling multiple forms of pretraining/data is well-motivated. The experiments for the most part don't actually use different data sources (just the task/presence of actions is changed). Experiments with real-world pretraining data would strengthen the paper.

I'm not so familiar with the related work on transformers for manipulation, so I cannot determine the novelty/impact of this particular architecture. If most of the benefit is from relative positional encodings rather than the planner-executor split, the paper should be reframed.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: The paper proposes an architecture for robot-learning for manipulation from readily available data. The paper conducts an analysis of the variety of data available and the volume of each type, identifying that unsupervised video demonstrations are easier to collect than sensorimotor data followed by task-specific data. Motivated by this, the paper presents a novel modular architecture and training paradigm.

The architecture is composed of a planner and an executor. First, the planner is trained using unsupervised video-demonstrations and then the executor is trained using sensorimotor data. Since observation (image) encoders are shared by the planner and the executor, the executor can be trained with fewer demonstrations (on top of frozen image encoders) after the planner is trained.  The architecture can also be fine-tuned using task-specific demonstrations. 

Experiments provide good empirical demonstration of the efficacy and data-efficiency of the approach.
review: Strengths

1. The paper provides an insightful analysis of groups of datasets relevant to robot learning based on quality, modality and volume. Thereafter the paper proposes an architecture and training methodology motivated by the above analysis.

2. Empirical results demonstrate that the proposed architecture is effective and data-efficient.

3. The approach is fairly novel and the writing is clear.

Weaknesses

1. I encourage the authors to add a baseline similar to 'Learning from play' but trained on sufficiently many sensorimotor demonstrations. 
     1.1 In my understanding, the reason the 'Learning from play' baseline performs poorly is because of fewer sensorimotor demonstrations     available. While this setting is valid given the hypothesis presented in the dataset analysis, the results of the proposed baseline will further establish the data-efficiency of the approach. 

2. I encourage the authors to perform experiments on demonstrations of long-horizon robot manipulation tasks. Will just fine-tuning the planner on long-horizon video demonstrations lead to good performance on such tasks? The authors should investigate this.

Minor revisions

1. Correct 'samll' to 'small' in line 286.

robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I did not give a lower rating becasue:

1. The dataset analysis presented by the authors is insightful. The architecture is devised keeping in mind the nature of available datasets.

I did not give a higher rating because:

1. Experimental results can be richer, solving more complex long-horizon tasks. 

2. Hardware experiments are unavailable.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work proposes a modularized method that attempts to learn: a) visuomotor actions through task-agnostic videos of a robot (e.g. play data) and b) planning via predicting embeddings of future states in image space through task-conditioned object manipulation videos of tasks other than those used for evaluation. The two parts are trained on disjoint data. The goal is to move a step forward towards building a foundation model for robotic manipulation that can achieve zero-shot generalization at test time and can learn from arbitrary video demonstrations including human. 
review: Strengths
1. Very well written paper with regards to clarity and organization, easy to follow and understand. 
2. Interesting idea that positively adds to the community's aspiration of building a foundation model for robotics. 
3. Limitations are clearly stated. 
4. Positional encoding ablation was an interesting finding.

Weaknesses
1. Baselines lacking: the method is not really compared to any baselines that also claim zero-shot generalization.  
2. State-of-the-art claim not fully standing.
3. Mismatch of expectations: Zero-shot generalization can be of variant degrees of difficulty. The paper initially prepares the reader for learning from videos in-the-wild which didn't end up happening. In fact, the experiments that relied on encoders pretrained on human videos demonstrated worse performance which implies that the domain gap was left unaddressed. Ultimately, the method shows good results when video demonstrations and target tasks are in the same environment with the same embodiment which is a strictly easier type of zero-shot generalization to tackle.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The method proposed was genuinely interesting, thank you to the authors for presenting. A direction like this could be promising towards eventually building a foundation model for robotics manipulation. My recommendation is weak reject at the moment primarily due to: 

a) the paper is framing the work more about learning from videos in-the-wild where experiments stay in the same domain for both executor and planner models. Please rework the introduction to set expectations for the body of the paper accordingly. 

b) the method is not being compared with other baselines that actually claim zero-shot generalization. Please add at least one baseline of a method that claims to be doing zero-shot generalization. 

c) SOTA seems like too strong of a claim. Please do not claim SOTA since technically on one task, the method is not doing better and/or add numbers for the remaining 6 tasks.

I'm willing to increase my score if the suggested resolutions are addressed.
review_title: null

Meta-review: The authors propose an architecture for learning robot policies from a mix of three types of data sources: (i) video-only data, (ii) non-task-specific sensorimotor data, and (iii) task-specific sensorimotor data.  The architecture has a few components which are trained in a prescribed fashion: a latent-space planner which can be trained from (i), and a goal-conditioned / inverse-model they call an executor.  Experiments show they are able to use this combination of different data sources with some success, for example in a way they outperform the "Learning from Play" baseline which is not designed to use (i).  While there are many other prior works that can also use (i), the work is perhaps relatively unique in its combination of all (i), (ii), (iii).

The reviewers had a variety of opinions.  Common strengths pointed to include that it is nice the method can do training on these different types of data, has strong performance in some settings, and provides useful empirical analysis.  The weaknesses mentioned mostly center around what I would characterize as an opinion by the reviewers that the authors have overclaimed their results, and/or that the results don't match the claims.  As reviewer yTsm puts it (paraphrased) "the motivation does not well apply to most of the experiments, pretraining from the same env as the eval."  Reviewer aYEz also had several critiques on the claims using the term "zero-shot", which were in part addressed in the rebuttal, but remains in their final determination still offput by the inaccuracy of the "in-the-wild" claim.  Although the other 2 reviewers suggested Strong Accept, these two reviewers (yTsm, aYEz) stood by their Weak Reject suggestions, with valid and well-argued concerns, despite the rebuttal attempts by the authors.

After careful consideration I would like to recommend the following:

- First, the reviewers should see the authors' revised manuscript which removes any mention of the term "in-the-wild" (https://openreview.net/attachment?id=iJx-Ksnsz4&name=file_upload -- I think that hyperlink will still work).  I think this partially relieves the dissenting opinion of aYEz in particular, and in part yTsm.
- However, even though the revised manuscript is a step in a good direction, the authors should amend their statements *further*... specifically, although there is no "in-the-wild" term used, the authors cite "[4-8]", which are in-the-wild video datasets.  This implies that their method will use these types of data, particularly since paragraph 2 says, "PLEX achieves exactly that".
- The authors should amend the introduction so that it does not imply this.  A suggested workaround is rather than to characterize (1), (2), (3) different types of "datasets" in the introduction paragraph, the authors can characterize three different "classes of data", and simply state that it is interesting to study methods that can use video-only data, even if the dataset comprised of such data is taken from the identical environment from which the robot actions could have simply been used -- which is the case in this paper's experiments.  The authors should only refer, even implicitly through citation, to in-the-wild video datasets in the context of related or future work.
- This is important because words matter!  There are many, for example, PhD students trying to work towards the vision of actually using in-the-wild video data in a way that is broadly useful for robotics.  This paper does not accomplish that goal, and should not imply that it does.  It can be viewed as perhaps a step towards that goal, but it does not arrive there.
- Similarly, also important, the authors should remove the notion of "embodiments" plural from the Section B, Problem formalization, because there are no experiments that address the multi-embodiment case.  This is classic over-generalization in a problem formalization.
- Regardless, if we re-examine both reviewers yTsm's and aYEz's dissenting opinions with the writing modifications requested as above, I think they would feel much more comfortable with accepting the paper.  The results are still interesting as they are truthfully, and don't need to be oversold.
Decision: Accept (Poster)