Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a new way of pretraining RL models such that they can be directly applied to Goal Reaching and Skill Prompting tasks, or finetuned for Offline RL. In contrast to previous works that pre-train these models using an autoregressive objective, the authors propose using the masked autoencoder (MAE) objective.  The bidirectional transformer model is pre-trained through reconstructing masked state and action tokens.  The authors demonstrate how such a model can be deployed to the three types of downstream tasks using distinct masking strategies.  Although the method is a simple extension of MAE pre-training, which has recently shown state-of-the-art performance in certain CV and NLP tasks, this paper shows extensive empirical comparisons against baselines.  In particular, offline RL results on Walker are comparable to SoTA in ExoRL [31].  Ablation study is also provided to justify some of their design choices such as the mixed mask ratio strategy.

[31] D. Yarats, D. Brandfonbrener, H. Liu, M. Laskin, P. Abbeel, A. Lazaric, and L. Pinto. Don’t
380 change the algorithm, change the data: Exploratory data for offline reinforcement learning.
strengths_and_weaknesses: Strengths:  The biggest strength of this paper is to investigate the use of MAE for unsupervised pretraining in RL.  Although I might have missed related works, this seems to be the first work that provides practical implementations of the method and comparisons against relevant GPT baselines that use the next state/action prediction loss.  Extensive experiments do show MaskDP outperform autoregressive baselines. 
Overall, the paper is clearly written.  It is interesting to see how MAE performs in unsupervised RL given its success in CV and NLP domains.  

Weaknesses: As acknowledged by the authors themselves, MaskDP still uses some expert data, which are not “unlabeled” as in CV and NLP settings.   This is okay as the method can be compared against other methods using demonstrations.  Unfortunately, I’m not very aware of the state-of-the-art results on these tasks using demonstrations.  Although the walker result is compared against SoTA, I cannot comment on the significance of this offline RL result section.  Is there a reason why only the Walker domain is compared against SoTA results?

-----
As the authors addressed my concerns, I've updated my score.
questions: I wonder if you could replace the word “zero-shot” performance with another word since goal reaching tasks are instances of the pre-training masked autoencoding task.

limitations: Yes, the authors have addressed the potential broader impact.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes to use masked autoencoding to learn a self-supervised encoder-decoder transformer model from sampled trajectories. The tokens that are masked and then predicted are a sequence of (interleaved) states and actions. The trained model is then used for goal reaching (both in an open- and closed-loop setting), “skill prompting” and offline RL. Experiments on DM control suite tasks show promising results vs. imitation learning, and an autoregressive baseline. 
strengths_and_weaknesses: ### Positive

- The method is straightforward, and the downstream use of the pre-trained model is crafty. Skill-prompting appears to solve an interesting inference problem. Insights on the appropriate masking strategy in this domain appears useful.
- The experiments are well designed, and the performance is measured online (as opposed to just offline). The baselines are appropriate, and highlight the masking aspect while controlling for confounders.
- The presentation is nice, and the paper is generally well written.

### Not Positive

- The application appears to be somewhat limited in scope (only robotics environments), and it is not clear if and how it might extend to pixels (though a VQGAN like approach can be promising).
- The influence of the policy used to generate pre-training data is not studied but left to future work.
questions: - For the goal reaching task, the trajectory length must be fixed in advanced. How is this dealt with in practice? Is there a “skip this step” token?
- How does trajectory length interact with training? Does the model generalize to trajectory lengths it has not seen during training? If not can that be fixed with an appropriate positional encoding, e.g. ALiBi (Press et al., “Train Short, Test Long …”)?
- In the multi-goal setting, how does MaskDP perform if future goals are not available, e.g. if they are unmasked gradually? This baseline should help confirm that foresight is being leveraged to “produce a better plan”.
- For the offline RL task, is it only the encoder that is being finetuned (as typical in vision tasks) or is it both encoder and decoder?
limitations: I'm concerned that the effect of expert (data-generating) policy on downstream tasks seems a bit too important to be left to future work. If it is left to future work, the authors should include at least some preliminary results in the next revision.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes to adapt the Masked Autoencoder (MAE) from computer vision into decision transformer for RL demonstration modeling. The paper propose several downstream application after the unsupervised pre training: Goal reaching, prompting, and fine tuning. The experiments are done in the deep mind suit with Mojuco. The authors also provides analysis on masking ratio and scalability.
strengths_and_weaknesses: Strengths:
This paper is well written and easy to follow. While the idea of MAE is not novel, the application of MAE into RL is novel and the proposed downstream prompting and goal reaching are novel, which I think will have large impact. The experiment results are convincing and the choice of baselines are reasonable, showing the effectiveness of mixed masking ratio and versatilty of the large pretrained models.

Weakness:
I couldn’t find any major flaw, but one thing I would love to see is how is this work compared with PromptDT [1]. While I understand these can be concurrent work, but it could be better if there is discussion / comparison between these two very similar papers.

[1] https://arxiv.org/abs/2206.13499
questions: Can you also add experiment on robotics domain like Metaworld?
limitations: Not that I can see
review_title: null

Meta-review: metareview: The paper investigates the use of Masked Auto Encoders (MAE) for unsupervised pretraining in RL. Reviewer ufWH summarizes well, "this paper shows extensive empirical comparisons against baselines. In particular, offline RL results on Walker are comparable to SoTA in ExoRL [31]". cGtJ says, "The paper propose several downstream application after the unsupervised pre training: Goal reaching, prompting, and fine tuning. The experiments are done in the deep mind suit with Mojuco. The authors also provides analysis on masking ratio and scalability."

Overall, reviewers find this to be a simple but strong method for unsupervised training for RL. I agree. 
confidence: Certain
Decision: Accept