Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes to use diffusion models to represent the different possible multimodal skills in the robotic setting. To improve the diversity of the diffusion model, the paper proposes to use a latent codebook in the denoising function of the diffusion model
review: I found the idea of using diffusion models to represent language anotated skills from learning to play to be natural, albeit a bit incremental (see some of the omitted related works below)

The paper demonstrates the efficacy of the approach across a wide set of different tasks and has several interesting design choices over prior work.

While I understand the motivation of using a discrete bottleneck to enable combinatorial generation, I think this makes a lot more sense in the language encoder as opposed to the denoising function. The denoising function needs to predict the noise added to noisy trajectories, which can be a uniform Gaussian. It seems like quantizing its prediction would be actively harmful as it prevent accurate denoising as opposed to specializing skills.

The paper is missing a couple relevant other papers which also use language conditioned diffusion models to represent policies. Some examples include [1][2][3]. 

[1] Learning Universal Policies through Text-Conditioned Video Generation
[2] LAD: Language Augmented Diffusion for Reinforcement Learning
[3] StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects

L127 skillsm -> skills
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: For the rebuttal, please address my concerns above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper proposes collecting play data, doing hindsight language annotation, and using a diffusion policy conditioned on RGB and language to perform robotics tasks. The method is evaluated on a real robot as well as three different simulation setups.
review: Strengths
* The paper is very well presented and convincing.
* Strong experimental section with proper ablations.
* Results on multiple environments like CALVIN, Language conditioned Ravens, and Franka
* Testing on long horizon tasks on the CALVIN evaluation suite make results more trustworthy with 128 total rollouts.
* Very nice video results showing the PlayFusion policy running on the robot

Weaknesses
* Lacking some info on datasets and data collection process.
* Other works have shown good results for language conditioned imitation learning. While results in this paper seem stronger than baselines, I think the result in the paper could be stronger if it can be shown that the diffusion + discretization improves sample efficiency or provides some other strong advantage. See section below requesting more information on dataset sizes/numbers.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend a **Weak Accept**. I think this is a strong paper overall that could be improved with some more info and/or experiments as suggested above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a diffusion model to generate natural-language-conditional actions. It introduces a discrete bottleneck in the diffusion model and uses discrete language conditional embedding with a VQVAE training objective. On CALVIN, Kitchen, and a real-world task, the proposed model outperforms baselines based on VAE, transformers, and behavior cloning. An ablation study on the introduced changes is included.

review: I am not totally familiar with the task setting and baseline models. My review below assumes the tasks are standard and challenging to existing models and the baseline models are state-of-the-art.

Applying diffusion models to this task seems like a sane choice. From my understanding, the contribution of this paper is threefold:
- Usage of a diffusion model as an action generator
- The discrete bottleneck in the diffusion model
- Discrete language embedding

The authors have given a good amount of evidence to support these methods:
- The proposed model outperforms baseline models.
- Through ablation, discrete bottleneck and discrete language embedding both contributed to improved performance.

I hope the authors can kindly answer my questions below. I also hope the reviewers could fix the typos, as there are quite a few of them (such as line 174 and 255).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe this contribution is interesting to the community and can potentially inspire future work, while the significance might be limited.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work extends language conditioned learning from play, proposing that diffusion policies with a discrete bottleneck are well suited to addressing inherent multimodality in noisy human-collected data.

review: ## Strengths
* Well motivated: play data is typically multimodal, which motivates using richer classes of generative models to capture multiple modes in the action label distribution.
* Clear description of methodology and setting.
* Strong initial signs of life on recognized manipulation environments.
* Clean presentation and writing.
* Compelling results on real robots.

## Weaknesses
* Limited novelty. As this work points out in the related work section, Diffusion Policies (Chi et al 2023) showed the benefits of diffusion models as policies in imitation learning scenarios. This work seems to take this insight and apply it to the hindsight language annotated learning from play setting, with no modification to the LfP setting and weakly justified modification to the diffusion policy setting (see "Weak experimental evidence"). However, while not necessarily novel algorithmically, this may provide useful further evidence to the community that diffusion policies are a sensible model class, especially when action label distributions are multimodal.
* Simulation results presented without confidence intervals over multiple trials. This makes it difficult to assess whether the gains over prior work are statistically significant, or if we are simply observing sampling error. Either confidence intervals should be added or claims like “PlayFusion also achieves significantly higher success rate than the baselines on Ravens” should be toned down.
* Weak experimental evidence. Lack of confidence intervals makes interpreting important ablations difficult. L247: “The success rate (averaged over CALVIN A and B) is worsened with the removal of either the U-Net discretization and the language embedding discretization.” These ablations are what justifies the main method over continuous alternatives, but the results are presented without any confidence intervals making it hard to judge how significantly the proposed modifications are over baseline. Particularly because the success rates seem rather close (53.1 vs 51.1 vs 45.1), it is hard to say currently if the main proposed method is actually better than simpler alternatives or if we are just observing sampling error. I.e. is a 1.9% difference between discrete and continuous diffusion policy representations significant? This strikes me as a pretty major current weakness, as it seems to be justifying the only novel algorithmic components of the work.
* Strong claims: Multiple times throughout the paper (e.g. L160), the claim is made that discrete skill representations generalize well in new settings without citation or further justification. The work would benefit from bolstering this claim somehow. What is it about discrete representations that should lead to better generalization, either theoretically or empirically? As a counterexample, BC-Z showed generalization to new combinations of skills not present in training data with continuous internal learned representations (Resnet + MLP).
Generalizing beyond the training distribution is presented as a major aim of the paper and discrete diffusion policies are presented as an answer. Why are there no experiments in simulation that test for generalization beyond the training distribution? This would presumably be straightforward to setup and would be far more replicable than the real world experiments.

## Nit
* Multiple references to “language-annotated play data” or “language conditioned play data” (section 4.1) without citing prior work “Language Conditioned Imitation Learning Over Unstructured Data” which focuses specifically on this setting.
Does Diffuser (Chi 2023) use discrete bottlenecks? If not, this seems like an algorithmic change worth mentioning as a key difference in the related section.
* L53 “However, these diffusion model-based approaches are still limited to learning from demonstrations or reward-labeled datasets, expensive data collection methods that may introduce undesirable assumptions to real-world robotics.” Rather than saying Diffusion Policies are limited, it seems more accurate to say that Diffusion Policies have not been empirically shown yet to work on unstructured demonstration data. This seems especially true because the current work doesn’t seem to modify Diffusion Policies algorithmically, but rather applies them to a new setting.
L59: “We believe that learning discrete skills will not only make the whole process more efficient but will also allow interpolation between skills and generalizations to new tasks.” Why would learning discrete skills make interpolation easier? Why would discrete skills generalize better? The manuscript could benefit from supporting these motivations with evidence or providing some further justification.
L127: typo, “skillsm”
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The presented work feels well executed, but with questionable research novelty. That which is claimed to be novel in the current work is weakly justified as important to success in the experiments. Even if not novel, the empirical results may provide useful signal to the community of the continued success of diffusion policies in new, less structured imitation learning scenarios.
review_title: null

Meta-review: The authors propose using diffusion models to model multimodal skills in a robotic manipulation setting, where a policy is trained on hindsight labeled play data. They propose a novel latent codebook in the diffusion denoising step in order to improve the diversity of the model. They evaluate the resultant policy on both real and simulated environments and show that their method outperforms related work.

The authors presented a thorough and detailed set of experiments during the rebuttal phase and addressed a number of reviewer concerns (e.g. missing references, additional ablations, explanations for various design choices, notation clarifications, etc). As a result reviewers did not increase their score, but the overall recommendation was still (slightly) leaning towards acceptance (and was so initially). It also should be noted that the reviewer proposing a weak reject score, did not engage in discussion after the author's rebuttal, yet the authors were quite convincing in their responses.

One potential concern not raised by reviewers (but as a suggestion from the AC) is that while the fractional improvement of their method over baselines is impressive, the final real-world success rates are never-the-less very low (e.g. 20% success on the Sink task). For future work I would consider verifying that the “improved policies” are actually qualitatively performing the tasks and not just “getting lucky” and triggering success thresholds more often. Perhaps a fractional reward metric (i.e. a dense reward) might better highlight the policy performance over baselines.
Decision: Accept (Poster)