Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper benchmarks the effectiveness of visual-pretraining across several well-known datasets. The finding is interesting: pre-training on certain datasets like Ego4D/RoboNet actually can hurt the downstream task performance. Also, blindly increasing the number of frames while not paying attention to the underlying distribution can result in ineffective upscaling.

review: ## Strengths

### Clarity
The paper is easy to follow and well-organized. The contribution is clearly stated. Comprehensive background information (current status of visual pre-training for robotics, how different datasets look like, etc.) are provided. Enough experimental details are presented for reproducing results.

### Quality
The paper is technically sound to me. Claims are supported with experiments. Methods are used appropriately.

### Significance
I think this paper can help the community explore the correct pre-training recipe for various robotics tasks.

## Weaknesses

### Typos
- Figure 2's caption: "from the both the" -> "from both the"
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend weak accept as I think the paper presents enough interesting findings for the community.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper investigates the impact of dataset biases and implementation choices on visual pre-training for robotics. The authors conducted experiments using MAE representations trained on 15 different data distributions obtained from 5 commonly used datasets. The representations are fine-tuned to solve manipulation tasks (both simulated and real) via behavior cloning. 
review: The conclusion of the submitted paper is that 

- Standard computer vision datasets, such as ImageNet, surprisingly yield strong performance for robotic pre-training. The simple ImageNet representations outperformed both Ego4D representations and representative baselines in the field. The key insight is that the diversity of the image distribution plays a more significant role than the raw number of images during pre-training.

- Building on the insight above, the authors explored strategies to scale the data distribution by carefully mixing data from different sources. As part of their investigation, they trained a final model called Soup-1M + 1M DoH, which demonstrated a 30% improvement over the baselines on real-world robotic tasks.

- The paper demonstrates how simple regularization techniques like Dropout can enhance real-world performance. 

- Results obtained in simulation environments do not necessarily transfer directly to real-world tasks. 

Strengths:

- The authors conducted various experiments supporting their claims.

- The paper is clearly presented and well-organized.

Weaknesses:

- There are remaining questions the paper needs to discuss about. Please check below.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I would like to give a borderline accept for now, but it will be much better if the authors can address at least 1 or 2 concerns listed above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors seek to verify if the common wisdom of using out-of-domain dataset Ego4D as pretraining of visual stacks for robotics manipulation holds. They compare 5 different datasets using MAE self-supervision pretraining. Then finetuning each pretraining with BC on small in-domain dataset (<= demonstrations).
They find that traditional vision datasets like imagenet yield more general representations than ego4d with less data. The insight is that diversity in image distribution is more important than the size of the dataset. A soup of datasets with 1M examples works as well as with 2M examples. The DoH dataset added to the soup-1M yields a big boost in perf. Authors hypothesize that despite having bigger domain gap with robotics, datasets sampled from the internet (as opposed to actively collected in robotics settings) have more diversity which explains the higher performance.
Authors provide some recipes leading to improved performance, with code and models.

review: Originality: The paper provides an unbiased fresh perspective on what datasets to use for pretraining in robotics manipulation, evidence that sim results do not correlate with real results, as well as some insights into improving performance. This is somewhat original in that the findings are somewhat invalidating common wisdom that ego4d is a better choice for pretraining than imagenet for example.

Quality: method is sound. Same SOTA self-supervision MAE is used for all datasets. Then 2-layer policy is trained on top of pretrained encoder. Entire model is finetuned end to end. This is similar to prior work with the addition of using dropout and data augmentation.
I appreciate the investigation into dataset balancing and finding that training on the balanced dataset soup leads to more robustness and performance.

Clarity: paper is well written and very clear. Good amount of details in appendix for reproducibility.

Significance: The number of evaluated tasks remains somewhat small, one can still wonder how general the findings are. It could however inform future data collections efforts to focus on diversity rather than volume for example, which could lead to significantly more general representations and higher performance and robustness.

Relevance: highly relevant to learned robotic manipulation.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: A good well-written pape with an interesting meta and unbiased analysis questioning common wisdom around how to do pretraining for robotics. Some good experimental recipes such as insights into how to train dataset soups. It is a welcomed paper with some impact, however it does not seem groundbreaking either.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper studies the problem of what qualities in datasets are important for effective representation learning for vision-based robot learning systems. Recent representation learning methods have trained on the large Ego4D dataset focusing on human-object interaction, but this paper finds that when used in MAE pretraining, standard image datasets such as ImageNet can perform better than Ego4D even with much fewer frames since the image distribution is diverse. The authors study the effects of using dropout and find that in their evaluations, higher performance in simulation does not always transfer to real world performance.
review: Strengths:
- This is an important problem for real world robot learning. 
- The paper does provide a nice suite of experiments in simulation and on a real Franka robot. 

Weaknesses:
- The conclusions in the paper are a bit misleading. The evaluation is specific to the representation learning method (MAE) and architecture used. Papers like R3M (Nair et al. 2022) showed that training using time-constrastive learning and other losses significantly outperform vanilla MAE approaches. They also compare pretraining with Ego4D vs ImageNet in their work and find that training with Ego4D is better. The authors should include a much more thorough discussion of these past analyses in prior work in the related works section.
- In order to make the claims it does about ImageNet vs Ego4D, the paper should do a more thorough evaluation using different representation learning losses and architectures, especially since there have been many more new papers on pretraining, and with tricks beyond dropout. 
- Would be nice to have more conclusions and analyses on the properties of these datasets that make them more suitable for robot training beyond just "increased visual diversity".

Other:
- Would be good to see standard error on the results. 
- Bold important numbers in the tables.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Due to the weaknesses in the evaluation, I think this paper needs more thorough experimentation and analyses before publication at a conference.
review_title: null

Meta-review: The authors present a study of a wide variety of visual pre-training choices for subsequently training visuomotor policies.  The paper submitted first draft focused on the use of vanilla MAE self-supervision, but swapping out the choice of dataset, then subsequently training a 2-layer policy on top.  In the rebuttal the authors have also added the case of using SimCLR self-supervision instead of MAE self-supervision.  The results show that there is an interesting spread of outcomes form choosing different datasets for pre-training.  For one, they emphasize the importance of the nature of the data distribution, not only its size.  Importantly, as the authors put it well in their rebuttal, they evaluate whether we should "uncritically prefer Ego4D for robot pre-training".

The reviewers had a variety of opinions on the work, but the majority of the reviewers responded positively to the findings, the clear writing, the overall technical soundness, and significance.  Reviewers had several good questions which were engaged with robustly by the authors.  The authors were able to help clarify some aspects of the comparison with the settings studied in some prior work, for example R3M. As the authors confirmed, in R3M the use of ImageNet as pretraining was evaluated in the form of using ImageNet's supervised labels, but the authors here investigate using self-supervised MAE on ImageNet, shown by He at al to produce stronger downstream representations.  They also show that the prior works of R3M and MVP come to opposite conclusions on which pretraining is better -- I don't think we can consider this debate settled at all, but the authors of the present work add additional data, as well as a broader set of data, to this comparison.  They also compare with other useful baselines, like training from scratch, and VC-1.  Overall it is compelling to see how such a simple MAE pretraining option, on a variety of datasets, can perform quite well compared to all of these baselines.  Reviewer 35RN remained suggesting a weak reject, but did agree that the rebuttal updates helped improve the paper.  Their remaining suggestions seem within scope for revising into a final camera-ready submission.

Due to the overall value of the work, I'm recommending acceptance for the paper, with the 4 following required TODOs for the authors:

1. This was not mentioned by any of the reviewers, but I think it is critical. The authors should **remove almost all mentions of the word "diversity"** in the paper, other than as mentioning it as a hypothetical and unevaluated hypothesis. The reason is that diversity is not defined, in any precise sense.  It is true that the nature of the data distribution seems to be having an effect which may be more impactful than purely the dataset size, but it is *not* evaluated whether or not this related to any technical, quantitative definition of diversity.  I would like to point out that Ego4D may actually be "more diverse", in some technical sense, than ImageNet -- I'm not saying it is, but it's not evaluated.  Perhaps really the reason that ImageNet is performing better in pre-training than Ego4D is that ImageNet images have the right level of structure in the image distributions to make the MAE task doable, yet still hard.  If the conditional distribution of pixels modeled in MAE (i.e. p(rest-of-pixels|some-pixels) is too close to a uniform distribution, there is nothing to be gained from MAE pre-training. Ego4D uses ego-motion cameras, which produce a challenging image distribution in which it is hard to predict some pixels from the others -- meanwhile, ImageNet images are from much cleaner "internet image" distributions.  This I think is both incredibly important but easy to fix in about 3 minutes, just ctrl+f for all 'diversity' and adjust for each mention of diversity, for example as follows in the abstract: "the pre-training dataset’s *diversity* matters more than its sheer size" --> "the pre-training dataset’s *distribution* matters more than its sheer size".  Further more insight could be gained by actually analyzing at the MAE loss on the different datasets.
2. The authors should clarify what exactly is the difference between the set of MAEs they are studying, and the "MVP" baseline.  It's my impression that intentionally these are exactly the same, other than "MVP" refers to a specific set of weights which was trained by the authors of that work, on a particular mixture of images.
3. The authors should clarify which images used for the real-robot experiments! In the videos I see there is a wrist-mounted camera, but also a fixed-mount camera, and I don't see where it specifies which (or both) is used.  Critical! Especially since one of these is ego-motion, one is not.
4. The authors should, as reviewer 35RN requests, "provide more qualifying statements throughout the paper, so that it is clearer what specifically is being compared and how that relates to other existing work."
Decision: Accept (Poster)