Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a unified architecture for conditioning on task specifications in multiple modalities. 
The modalities covered are video, audio, text, and images. 
Broadly, the method involves learning a joint representation space for the multiple modalities that can be used for guiding manipulation tasks. 
The method also suggests matching less information-dense modalities to video demonstrations, which are more information dense. 
The paper also introduces a real-world dataset with multiple input modalities. 
The method is evaluated in simulation and on hardware.
The authors find that multimodal pre-training improves their performance on instructions given in a single modality, and that both the multimodal pre-training and matching to video data result in performance increases across almost all modalities.
review: ## Strengths

1. **Expanding beyond two modalities**: Most multimodal work focuses on images and text. By expanding out to video and audio and supporting instructions in *any* modality, the proposed method is much more flexible than most multimodal specifications. The authors devote a lot of effort to describing and exploiting the ways in which modalities can be complementary to each other. 

2. **Hardware Evaluation**: The hardware evaluation is not just a proof-of-concept demonstration, and the paper includes quantitative results on how the method performed on the hardware. 

3. **Dataset**: The real-world data the authors introduce will likely be useful for future work on multimodal task-specifications. 

4. **Relevant Ablations**: The authors ablate out the relevant components of their model to support their claim that multimodal training improves performance on the tasks tested. 



## Weaknesses 
1. **Lack of baselines**: the model is evaluated against an ablated version of itself, but there are no external baselines. While this does not detract from the pitch of the paper as a multimodal training method (since the relative changes are important, not the absolute performance) it would be helpful to contextualize the results with other unimodal baselines, at least for the text goals/instructions and image goals, for which baseline systems exist. 
Along the same lines, it wasn't clear to me how the unimodal baselines were trained. 

2. **Lack of details**: There are a number of details which are omitted from the paper, without which I think reproducing the results would be difficult. These include hyperparameter settings like the masking ratio for different modalities, the loss mixture ratios, the specifications of the different encoders, the projection layers, policy encoder, decoder, etc.. This information would be fine to include as supplementary material, but should be provided with the paper somehow. 

3. **Framing issues**: There is a bit of divergence between the framing of the paper and the experiments. One goal stated on L131-132 is to use multimodal training to generalize to novel tasks, but there are no experiments to this end. 


## Comments 
- The loss equation on L158-159 seems unneccessary, since it doesn't actually tell me what the loss between $y_i$ and $\hat{y}_i$ is. The equation also uses overloaded notation. All this equation tells me is that you averaged the loss across modalities, and you could save a lot of space by just writing that out. 

- L226 has a typo or missing sentence in "...whereas a Kinect Azaure workspace camera."
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Based on the weakness section in my review, I'm recommending Weak Accept. I think the technical and data contributions of the paper are very strong, but I have qualms about the current presentation, especially about the lack of details which I think would hinder the reproducibility of the method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This work presents an approach for learning a unified representation from a multimodal task representation and a collection of expert demonstrations for each task. The main premise of the this work is to allow cross-modal information to further enhance the representation required for learning robust policies to solve the downstream tasks from expert demonstrations. The approach, consists of two stages: (1) A mask modeling approach is used to allow for cross-modal information to be encoded in the learned representation; (2) A policy is learned using this representation to solve a task from any or a combination of task representation modalities. The paper then presents experimental results in both real world simulation and real world domains (100 tasks 14 in simulation and 40 tasks in the real world) and demonstrates improved performance when such representations are used.
review: This is an ambitious effort to bring in multiple modalities of task description for both representation learning and control which is an important idea in Robotics. I also like the fact that it has been utilized in real world robotics experiments. However I have the following issues with this wok: 

Technical: While the concept behind the work is important, it lacks sufficient supporting evidence in the form of experiments and studies to substantiate its claims and they are tested at the surface:
-- I have some issues with the main premise of the paper. It feels like the paper aims both perception and control which requires a lot more experiments and evidence to back these claims: which one of these is the man claim (it feels like both)?: (1) The unified representation is more rich compared to learning representations based on individual modalities. If this is the claim, then this work lacks ablation experiments and results that demonstrates for example masked training performance in each modality is improved when combined with the other modalities independent of the second stage (control); (2) If the claim is that the unified representation is superior for control, my primary concern is that this work fails to adequately address crucial control issues, such as the compounding error factor present in vanilla behavior cloning.

-- "novel architecture based on transformers": the architecture is not that novel, adding and moving around self-attention and cross attention blocks do not necessarily makes it novel. It is essentially a variant of transformer encoder-decoder architecture.

-- Lack of other baselines in experiments: the authors only compare to a baseline of using individual modalities. They could have at least compared it to approaches such as VIMA although different.

Paper organization and presentation: 
(1) The paper's readability and coherence are compromised as it becomes challenging to maintain a consistent context throughout. The inclusion of plain text (e.g., sections 3.1 and 3.2) without accompanying figures, algorithmic view, or visualizations makes it more difficult to comprehend and follow the research. 
(2) Additionally, there are a few readability and grammatical issues which need to be fixed.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I understand the technical material well however I do not have a broad knowledge of the relevant work except for a few. Independent of that, I think the claims are broad and the experiments do not provide enough evidence to back the claims. I recommend weak reject.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a novel policy learning framework that allows flexible task specifications, which is an important problem for learning from different modalities. The method proposes a two-stage training method that provide supervision on cross-modality masking and distillations to enable better generalization performance. The method is evaluated on both simulation and real world experiments and demonstrated better performance compared to modality-specific training.
review: Strength:

1. The motivation is sound that as humans, we can specify tasks with different methods and can generalize well. Recent vision results such as segment-anything also show the importance of considering multiple ways of prompting to learn a generalizable representation space.
2. The evaluation is thorough, 100s of tasks with 1000s of trials each.
3. The cross-masked modeling procedure and the transformer architecture (with pretrained encoders for speech and language instructions) makes sense and seemed to work well.

Weakness:
1. The current method relies on paired labels for each modality, which is expensive and prohibitive. Would be interested to see if the framework can still work with in-the-wild human demonstration data, or larger train-test gaps.
2. The overall results seem to indicate a small improvement over only video-conditioned policy, which is, from my understanding, trained from scratch and less practical for test-time.
3. The overall writing and figures an be improved.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper presents a well-thought and clearly written method for solving policy learning with different modalitiy. The protocol and code (if released) are also very useful contribution for the community. There are still several weaknesses regarding experiments and algorithms and would love to hear about the response from the authors. Overall I recommend acceptance of this paper.
review_title: null

Meta-review: The reviewers unanimously vote for accepting the paper and I concur. Some reviewers raised concerns about writing and presentation. G6C8 remarks, "but I have qualms about the current presentation". A closer look at the draft reveals that even the first sentence in the abstract might be incomplete. There are also concerns about reproducibility and comparisons to the baselines that authors presented in the rebuttal. I encourage the authors to address the presentation concerns and include the baseline comparisons in the final version of their paper.
Decision: Accept (Poster)