Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes to use a deep neural network model trained on Ego4D human videos to generate latent representations (visual embeddings) with the hypothesis that it will speed up learning as compared to training from scratch or using other visual embedding generation methods.
strengths_and_weaknesses: Merits and Strengths:

The paper does indeed address an important challenge in manipulation: what should be the backbone model/representation for the robot, given that it is so expensive to collect data? The contrast between robotics and CV/NLP is also drawn where the latter fields have benefited greatly from LLM and VLMs trained on large corpi of data – can we benefit from that in robotics?

The paper does a good job of evaluating and benchmarking the proposed approach on a number of simulated environments. There are also a number of robot experiments implemented with a Franka Emika Panda in the context of in-home robotic applications. The benchmarks mostly seem appropriate.

Weaknesses and Limitations:

The biggest concern with the paper is: What is the core technical contribution/novelty? The paper uses an existing data-set, together with a well-established learning procedure (the losses, data preparation, and model are fairly standard) to generate an encoder that is then used in a fairly standard imitation learning framework. It seems that the essence of the paper is “using a model trained on Ego4D data set may be better than using one trained on CLIP or other similar data-sets”. I expected to see some contribution along some axes such as: insight for general principles of what makes a data-set particularly useful for robotic application, or some novelty in training, data-augmentation, or generalization. For example, what specifically about the R3M data-set makes it more effective than CLIP , ImNet, or MoCo? What types of data should we be collecting based on the preliminary results of this manuscript? The experiments should be more specific to tease out these differences and highlight why R3M is able to outperform, not just that it does.

Beyond the above concern, the performance improvement is quite modest. The success rate in the Franka Kitchen environment is low for all models, and there is a modest (about 15% based on the graph) improvement but nothing to suggest that R3M is the solution. The Meta Worlds results are inconclusive, the performances are quite similar. Adroit is between the two, R3M does better but it's within 10% of the next baseline. We cannot conclude that R3M is a significant advance on the state-of-the-art based on these results.

The limitations discussed are okay. I agree with the first one, it is not obvious whether a representation used for imitation learning is also good for RL. The second limitation discussed is more of a future work rather than an issue with the approach. The model provides a functionality (language module) that was not used in this study but could potentially be useful for other tasks. I find the last limitation the least satisfying, it is basically a statement of “more data is probably better”. That statement is likely true, but this is a limitation of all learning methods, the more data the better they are likely to perform.

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The manuscript provides some encouraging preliminary results; however, it needs to provide deeper insights into why the particular model works (what is specifically useful about the data-set) and what general principles can we garner/look into studying to improve our visual encoders for robotics applications (e.g., what types of data should we be collecting and using?). There are a wealth of interesting questions to ask and experiments to run to evaluate hypotheses that I believe this work can, in future submissions, highlight. Please see previous section for more details.
issues: The manuscript requires a significant re-write with an emphasis on analysis and a deeper dive on the fundamentals of why R3M works (slightly) better than existing methods with concrete statements and experiments to verify. It may be ambitious to attempt this during the short few weeks of rebuttal. The issues are listed in sections above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes reusable representation for robotic manipulation (R3M), a framework that leverages time-contrastive learning, video-language alignment, and an L1 penalty to encourage sparse and compact representations, learning reusable representations at a pre-training stage using Ego4D human video dataset. Experiments are ran on the MetaWorld, the Franka Kitchen, and Adroit environments. This is an immitation learning method, requiring annotatated videos with sentences describing tasks.

strengths_and_weaknesses: ## Strengths
- The usage of large-scale datasets to build reusable representations in RL and robotics in particular, as opposed to training tabula-rasa, is a well-motivated research problem.
- I agree with the authors to certain  degree that one cause for the representation component of RL behind behind of other ML problems such as NLP can be due to our lack of pre-training strategy agreements and access to appropriate robotics datasets.
- Compelling results and real world robot results in general.


## Weaknesses
- The reliance on annotated dataset, where each video is paired with a natural language description, that describes what task is being completed in the video. 
- No other baseline presented, other than using their framework emebedded with representation learning techniques such as CLIP and MoCo. This makes the claims weak, since there are other frameworks focused on representation learning for immitation learning/RL. Other baselines to consider could be direct comparison to PVR, or CURL (contrastive-based learning) / other immitation learning frameworks etc.
- Weak contribution/novelty since the method is combining a few existing pieces under an immitation learning framework where in general its less of a challenging setting compared to RL. Also experimenting with regularization methods such as $L1$ is a common practice, for me it doesn't add any value in terms of contribution (its more like an implementation detail).
- Language embedding for representations is not new, for instance CARE (https://arxiv.org/pdf/2102.06177.pdf) is one example.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The main components of this method are as follows:

Starting with dataset $D$ of $N$ videos, where each video consists of a sequence of RGB frames. Ego4D dataset (tasks such as cooking to socializing to assembling objects from more than 70 locations across the globe).


### Time Contrastive Learning.
Given a batch of videos, they train the encoder to produce a representation such that the distance between images closer in time is smaller than for images farther in time or from different videos.

### Video-Language Alignment.
In order to capture semantically relevant features, they train a language prediction module from the embedding. That is, capturing features
predictive of language, like “putting the apple on the plate”.

### Regularization
L1 regularization to encourage learning sparse representations.

### Experiments:
In the MetaWorld environment the tasks are assembling a ring onto a peg, picking and placing a block between bins, pushing a button, opening a drawer, and hammering a nail.
In Franka Kitchen, sliding the right door open, opening the left door, turning on the light, turning the stove top knob, and opening the microwave. In Adroit, reorienting the pen to the specified position, and picking and moving the ball to specified position.


Overall this paper demonstrates interesting arguments and a variety of interesting manipulation tasks - But the core contributions are not compelling. I am happy to discuss and change my score if provided with a strong argument for the lack of external baseline methods. More in particular, I'd like to see comparisons with other SOT of immitation learning methods.
issues: I have the following questions and concerns:

- Word-to-word repeat of the following paragraph : `We hypothesize that training a good representation for robotic manipulation from human video data should capture three key components ...` In section 1 Intro and 3.3.
- When using contrastive learning, for negative samples, you sample a negative example sampled from a different video in the batch.
Does different video mean a completely different task? how do you ensure the tasks are non-related so the contrastive learning objective can work best?

- The tasks attempted are slightly non-standard, especially going from the tasks tackled in simulation vs real world. I am wondering how the authors picked these tasks?

- Where is the balance between needing variety of annotations vs tasks to learn meaningful representations? Since Ego4D dataset already contains these, but when you want to learn the tasks on the real system? For instance do you need one sentence per task, or variety of tasks? This was briefly discussed in the CARE paper (referenced above) where there is heavy reliance on the exitence of language annotations.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a novel visual representation learning approach for robotic manipulation.
The algorithm uses language annotated human video sets (Ego4D) and contrastive learning to learn a consistent visual representation.
The resulting pre-trained visual representation can then be deployed to downstream robot manipulation tasks.
The promise of R3M is to improve data-efficiency of policy learning by providing a pretrained visual module as opposed to learning it from scratch.
Extensive simulated and real world results show that R3M is able to outperform other methods in learning robot policies from demonstrations via behavior cloning in previously unseen environments.
strengths_and_weaknesses: Strengths

The paper proposes the original idea of using video recordings of human manipulation and interaction tasks together with language annotation to learn a visual representation network.
Due to the diversity of robotic manipulation pre-training datasets are not, or only sparsely available.
However, relying on real-world video recording of human manipulation can form a basis for vision-guided robot manipulation tasks as well.
The paper proposes an intuitive self-supervised learning method that is able to exploit the sequential information in videos, language annotations to capture semantic meaning and enforces sparsity of the image embeddings to avoid overfitting.
The overall method is well motivated and the paper is easy to follow.
Despite the limited page number the paper provides simulated and real robot results in the context of imitation learning with BC using the pre-trained visual representation.

Weaknesses

While the idea is original and promising, a significantly extended evaluation will be needed to fully support the claims of the paper.
The paper also discusses limitations on this aspect.
It would be also interesting to see a deeper analysis on what knowledge from the Ego4D dataset is relevant for the (simulated) robotic tasks.
Overall, a more in-depth qualitative analysis of the approach would be welcome.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper proposes an original and potentially impactful idea for visual representation in robot manipulation. 
While this work could use a significantly extended page number to fully support its claims, the paper at its present state does a good job in conveying the main messages.
issues: Could you put as much results as possible into the main paper from the supplementary material? The figures showing the success rates take up quite some space, I'd recommend using a table instead and also presenting Fig. 8 and 9 from the Supplementary material.

Question: why does the paper emphasize R3M to be 'frozen'. Couldn't we just update it in downstream tasks to fit the task a bit better?
review_title: null

Meta-review: metareview: This paper proposed a pre-training method on the Ego4D dataset and used it as visual embedding for robot manipulation. This paper receives diverging reviews strong reject, weak reject, and strong accept. 

Reviewers appreciate the evaluation on different sim and real-world environments. However, at the same time, all reviewers also agree that a more extended evaluation is necessary to support the paper's claims.
The major concerns are summarized below: 

1. Contribution (Reviewer QF3c QzCR). What is the paper's main contribution and then to the performance improvement 1) the usage of the "Ego4D" dataset or 2) the proposed representation learning method? What are the insights a reader can learn from this paper? 

2. Modest performance improvement and lack of in-depth analysis (Reviewer QF3c, JMfs). This is also related to the next point (baseline), making it hard to evaluate whether such representation can indeed advance the SOTA performance. 

3. Other representation learning baselines in robotics (Reviewer QF3c,QzCR). Direct comparison to PVR or CURL (contrastive-based learning) / other imitation learning frameworks will be helpful to demonstrate whether such representation can indeed advance the SOTA performance. While CLIP is a popular visual representation for computer vision tasks, it is expected to perform poorly since it is trained on static image captioning.  


During rebuttal and later discussion phase, the reviewers have a long and productive discussion. 
While all reviewers agree on this paper's major strengths and weaknesses, however, different reviewer tends to weigh them differently. 
- all reviewers agree this paper presents the limited novelty "uses and existing data-set with an existing architecture, with existing losses and training/algorithms"
- however, reviewers also all agree that the  "the pre-trained vision model"  does bring positive value to the robotics community. 
- In terms of performance improvement, reviewers have different opinions on whether the current improvement is significant, while AC agrees with reviewer QzCR that considering the number of tasks tested, the performance improvement is convincing. 

Weighting both strengths and weaknesses, AC would like to recommend accepting this paper since the pretrained weight can be a valuable resource to the community. 

Author are encouraged to restructure the experiment section to include more insights and discussions. For example, "insights into why the particular model works (what is specifically useful about the data set) and what general principles we can garner/look into studying to improve our visual encoders for robotics applications (e.g., what types of data should we be collecting and using?)."

confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)