Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, authors present a method for pretraining a generalisable and resilient SSL model with control transformer for multi-task sequential decision making. They propose a self-supervised and control-centric objective that encourages the transformer-based model to capture control-relevant representation. The evaluation is performed on multiple domains and tasks. The results show model’s effectiveness and robustness to distribution shift. 


strength_and_weaknesses: Strength: This paper proposes an effective approach for pertaining SSL model for multi-task sequential decision making. This paper clearly presents the proposed approach, with carefully designed extensive experiments on DeepMind Control Suite and comparison with a set of baselines.

Weakness: Overall, I find this paper has limited contribution. Also, the section 6.3 (Ablation and Discussion) is brief towards discussion and ablation studies can be extended.


clarity,_quality,_novelty_and_reproducibility: This paper clearly presents the proposed methodology and the authors mention to release the code for reproducible research with camera ready version. 


summary_of_the_review: The proposed SMART method for training SSL model with control transformer for multi-task sequential decision making is shown as effective and robust. This paper clearly presents the proposed approach and conducts extensive experiments on multiple tasks and comparison with multiple baselines. 

I think authors should consider improving section 6.3 with more discussion on the results of ablation experiments.

Additonally, I find this paper has limited contribution with control transformers.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The manuscript proposes a new method which learns essential representations in a pretraining phase, including both long-term and short-term dependency information, in a sequential decision making process using reinforcement learning. For this, authors have proposed a novel control-centric objective which contains three components: forward dynamic prediction, inverse dynamics prediction, and random-masked hindsight control. First two components exploit local dependency information and the third component learns long-term dependency. The proposed method is first pretrained on multi-task dataset to learn reward agnostic representations which are utilized on downstream tasks. The method is evaluated on DeepMind Control (DMC) suite, demonstrating superior performance compared to scratch, single task pretrained, and ACL method reported by Yang et.al. 2021.
strength_and_weaknesses: Strengths:
+ The proposed method learns reward-agnostic representations in pre-training phase to improve generalizability
+ The proposed method effectively captures long-term dependency information using random-masked hindsight control mechanism.
+ Ablation studies demonstrates effectiveness of each component of control-centric objective function.
+ The paper is organized well and explained every aspect in clear terms.
 
Comments/Questions:
- Figure 1 finetuning phase shows that R_t is shown as input at all different time points: t, t+1 and t+L. Are there typos? Should it be R_{t+1} at t+1 and R_{t+L} at t+L time stamps?
- Proposed method was evaluated on DeepMind Control (DMC) suite tasks cheetah: run, cart-pole: swingup, cart-pole: balance, hopper: hop, hopper: stand, walker: stand, walker: run, walker: walk, pendulum: swingup, finger: spin. This dataset also contains other tasks such as ball-in-cup: catch, reacher: hard, finger: turn, manipulator: bring ball, swimmer: 6 links, swimmer: 15 links, fish: swim, cheetah: random action, hopper: random actions, walker: random actions, humanoid: random actions, humanoid: stand, humanoid: walk, humanoid: run, CMU motion capture. It would be interesting to see how the proposed method would perform on these slightly more challenging tasks. 

clarity,_quality,_novelty_and_reproducibility: Good
summary_of_the_review: The proposed method is innovative and effective as demonstrated by their empirical study. The presentation of the paper is clear and code will be available to the research community to facilitate the reproduction of reported results and applications on other learning tasks.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presented a new self-supervised pre-training method for sequential decision-making tasks. Specifically, a new general pre-training-finetuning pipeline named SMART (Self-supervised multi-task pertaining with control transformer) was proposed with a Control Transformer (CT) and a control-centric pre-training objective for the model supervision. Experiments in DeepMind Control Suite show the effectiveness of the proposed method over several tasks. The main contributions of this paper are the proposed new self-supervised pre-training pipeline with the corresponding objective and also the carefully designed experimental analysis.
strength_and_weaknesses: **Strengths**

\+ The paper well summarised the challenges unique to sequential decision-making scenarios and accordingly proposed a solution to address the issues.

\+ The idea of the proposed control transformer (CT) is interesting and well-motivated, especially the purposefully designed exclusion of the reward signal for pre-training.

\+ The experiments were well designed to validate the proposed method from several different perspectives in response to the questions mentioned at the beginning of Sec. 6.

\+ The proposed method was shown to have better resilience when facing pretraining data with different qualities, as shown in Fig. 5. 

\+ The paper is well-written and easy to follow.


**Weaknesses**

\- The novelty of the proposed random masked hindsight control is a bit limited. Although the authors claimed the difference to BERT-like models, the general idea of mask-and-predict is the same and the only difference is the items to mask. This idea is also similar to those visual pre-training models like ViT etc.

\- It is unclear why only masking the actions while leaving out the observations. The authors claimed the reason to be "force the model to learn global temporal relations", but there is no clear evidence to support this.
What if including the observations as well, or partially? or what if similar to the random scheme as proposed, randomly predict the observations as well?

\- From the description, the overall objective used all three mentioned terms as mentioned in Sec. 5.2 / Fig. 2. The description also seems to suggest the effectiveness of the 3rd term of random masked hindsight control, but it is unclear how would the model performs if only using this term.

\- For the experiment of ‘versatility’ as shown in Fig. 3, there are some cases where the CT-single performs on par with or even better than the proposed SMART, i.e. single-task performs better. It would be better if some explanation or insights could have been provided.
clarity,_quality,_novelty_and_reproducibility: The clarity and quality of this work are good. The technical novelty is okay though a bit limited if considering the high-level idea similarity to those mask-based transformer approaches and the related control- or decision-based methods. As the code was uploaded, there should be no issue with reproducibility.
summary_of_the_review: The studied problem is clearly stated with convincing motivations, the paper is well-written and the idea is mostly clearly presented. The experiments also well validate the claims with shown effectiveness of the proposed method over other solutions. Although there are a few concerns as mentioned in the Weaknesses section above, they could be addressed in the rebuttal phase, and overall this paper would be a good contribution to the community. As a result, I would like to recommend 'accept' for this paper.

-----------------After rebuttal---------------------

Thanks to the authors' responses, which well addressed my concerns. The additional included experiments and the clarification also make the paper stronger.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors consider the problem of unsupervised pretraining representation for  high-dimensional sequential control in various downstream tasks. The authors propose a framework as a pretraining then finetuning pipeline for sequential decision making, it consists of a Control Transformer (CT) which is coupled with a novel control-centric pretraining objective in a self-supervised manner. The authors show that the proposed method, SMART, significantly improves the learning efficiency in both seen and unseen downstream tasks and domains. 
strength_and_weaknesses: Strength
- This work proposes a simple and effective method, for unsupervised pretraining and adaptation to downstream imitation learning and RL tasks. 
- The experiments are extensive and show the strength of SMART, and I appreciate the authors conduct pretraining experiments on random and exploratory datasets since otherwise simple behavior cloning would be sufficient.  

Weakness
- How does SMART perform in goal conditioned tasks and model-based tasks? Showing the results could help have a better understanding of what SMART captures from pretraining, especially since it’s trained with forward, inverse dynamic prediction and random hindsight prediction. 
- The Figure 11 scalability experiment is interesting because it shows after a certain amount of model capacity, there is no further relative improvement. I wonder if the authors tried smaller model sizes since the current smallest model is arguably quite large for simpler tasks. 



clarity,_quality,_novelty_and_reproducibility: Quality
- Paper proposes a simple and effective method for unsupervised pretraining using Transformer.
- The idea is novel and simple to implement. The idea is to pretrain a Transformer in unlabelled datasets by predicting the masked tokens. 
- The experimental results show the effectiveness, SMART outperforms baselines in multiple settings. The evaluations are extensive and convincing. 

Clarity
- Mostly very clear. The key concepts are well explained and main claims are well supported. 
- How did the authors decide to choose which tasks and domains for pretraining and which for finetuning? I appreciate the authors present the Figure 8 to illustrate the relationship between pretraining and finetuning tasks but it’s unclear the reason behind the decision. 

summary_of_the_review: This paper tackles the problem of unsupervised pretraining for sequential decision making. The proposed algorithm is simple and effective, with extensive experiments to show its effectiveness. There are some minor limitations of this paper that I would like the authors to address.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper presents a new method for pretraining essential representations for fine tuning on downstream sequential decision-making tasks using a Control Transformer and self-supervised learning.

All reviewers agreed on the importance of this topic, the clarity of presentation, and the thoroughness and quality of the experiments. A few clarification issues were raised and some further control experiments were suggested in the reviews. Authors were very responsive during the rebuttal period, including adding results on 5 more tasks, adding an experiment with observation prediction and various other variants of their model, comprehensive ablations. Their results show that it’s important to learn both short-term and long-term dependencies for useful representations, which should be of interest to all ICLR attendees interested in deep RL and effectiveness of pretraining/fine tuning.

Although 3 out of 4 reviewers failed to respond to author rebuttals, after reading through everything, I'm satisfied that authors have adequately addressed their critiques. I’m happy to recommend acceptance, and look forward to seeing the updates and the release of the code with the camera ready.

justification_for_why_not_higher_score: This is a very well-done paper, but experiments were limited to continuous control tasks, which hinders its broad applicability.
justification_for_why_not_lower_score: All reviewer concerns were addressed as far as I can see, and I’m confident there will be a lot of interest in this paper at the conference. I believe it warrants a spotlight for that reason.
Decision: Accept: notable-top-25%