Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work introduces a contrastive loss for decision transformers, this is motivated by the need for better state, action representations for different returns. A new loss is introduced to separate such embeddings and is compared to a reasonable baseline. Policy performance using such a embedding+loss is shown to be improved on several OpenAI and Atari benchmarks. Ablations show which components of this algorithm are useful. They include studies of the embeddings and their separation during training.
strengths_and_weaknesses: Strengths:
* The paper is very clearly written: the algorithm is clear, repeatable and compared to strong baselines on convincing benchmarks.
* Additional ablations and studies of the embeddings confirm the intuitive motivation for such an alteration to DTs
* The improvements are significant.

Weakness:
* It would be a much more convincing results with some real robot experiments, it seems like it works in simulated setups though.
* One reason for using sequence models is to model non-markovian systems (quite common in robotics): returns cannot be determined by the current (s, a) alone and require historical (s, a). However, with the state-action embedding clustering this property is lost. It would interesting to see if there are cases of such setups where this markov-assumption clustering actually hurts.
* Some details of the transformer architecture would be useful: in the comparisons of DT and ConDT are they controlled for the same number of parameters? What would happen if the transformers for both models were much bigger, surely a big enough transformer can model this separation if it useful to the task.
* I assume there is one embedding for the state and another for the actions, but this was not clear. There are mentions of state-action embedding which makes it seem like they are embedded to a single space?
* The results between ConDT w/o Prod and ConDT are mixed: it seems like it is not clear which method to use.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Decision transformers are a promising sequence modeling approach to reinforcement learning and this work suggests an improved version with a contrastive loss to separate state, actions for different returns. It would be useful for the community to reproduce this work on additional benchmarks, especially real robot setups. The contrastive loss introduced is not revolutionary and likely a large transformer has the modeling capacity to accomplish, however, in simulated benchmarks shown the results are convincing improvement over DT.
issues: * Repeat some of the comparison for different # of params of transformer models - fixed for all models within the comparison.
* Some comments in the limitations about a (s, a) embedding without historical context when the transformer itself has that context.
* Comment of limitations in different reward settings, a lot of settings use binary rewards would you still recommend this method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a new decision transformer method that combines decision transformers with contrastive learning for better representation learning, The authors perform contrastive learning on the embedding after the return-dependent transformation based on prior works RCRL and show that such an addition to decision transformer can lead to performance boost on tasks in D4RL and Atari games.
strengths_and_weaknesses: **Strengths:**

1. The idea of the method intuitively makes sense and is easy to understand. It is neat to see that decision transformers can leverage contrastive learning for better performance.

2. The results of the proposed method seem to improve over vanilla decision transformer on standard benchmarks.

**Weaknesses**:

1. The paper appears to be a bit incremental in terms of novelty. The method is heavily built upon prior works decision transformer and RCRL. It seems to me that the only difference is that the authors use SimCLR rather than the simple regression loss in RCRL as the contrastive loss.

2. The improvement over the decision transformer is rather marginal. In particular, the simple DT+Prod method can perform quite comparably to the proposed method, questioning the effectiveness of the contrastive loss. 

3. Real-world experiments are regrettably missing.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Based on my comments in the Strengths And Weaknesses, I think the novelty and the results of the paper are not convincing enough. I would vote for a weak reject.

--------------------
post-rebuttal updates: thank you for the response! I appreciate that the authors added real-world experiments and offered clarifications. Therefore, I've updated my score to a weak accept.
issues: See the Strengths And Weaknesses section.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes ConDT, which applies contrastive learning to decision transformer. Specifically, the author utilize return-based contrastive learning to optimize input embeddings such that state-action pairs with similar returns are clustered in the embedding spaces. The authors showed that ConDT improves the performance of DT by a large margin on standard offline RL benchmark.
strengths_and_weaknesses: * Strengths

1. Simple yet effective method: given fixed dataset, contrastive learning can provide additional signals for NNs and the authors showed that it is useful for offline RL using decision transformer. The idea is simple and main components (decision transformer & return-based contrastive learning) are from existing works. However, its novel combination makes a big difference. 

2. Large gains on standard benchmark: the authors showed that ConDT outperforms DT on standard offline RL benchmarks. However, it would be also nice if the authors could compare to other state-of-the-art offline RL algorithms

* Weaknesses

1. Computational overhead: it would be nice if the authors also can point out some drawback from the proposed method. For example, additional auxiliary loss can increase the training time. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I'd like to recommend "weak accept" because (1) the proposed idea is well-motivated and (2) ConDT is very effective even though core idea is simple. 
issues: * Comparing training time. 

* Comparison with other state-of-the-art offline RL algorithms

* Evaluation on more hard domains such as ant-maze
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a Contrastive Decision Transformer (ConDT) that improves DT by treating the bin of target returns as each task and separating the representation of state and action embeddings into clusters labeled by each return bin. Specifically, the paper introduces a transformation layer for each bin and introduces a contrastive loss that makes embeddings from each transformation be apart from each other. The proposed method is evaluated on OpenAI Gym locomotion tasks and Atari games.
strengths_and_weaknesses: Strengths
- Idea is intuitive and interesting. Especially the intuitive explanation on how the proposed idea of separating the representations can effectively reduce the search space that would enable DT to perform better is interesting.
- Experimental results are interesting with supporting ablations and analysis. Trends in ablations are not perfect, but it can be understood.

Weaknesses
- The main weakness I see from the paper is that the paper is somehow submitted to not-relevant conference. In CoRL guideline, it is stated that ```All CoRL submissions must demonstrate the relevance to Robot Learning through Intent—explicitly address a learning question for physical robots, or Outcome—test the proposed learning solution on physical robots.``` This concern is intensified when I see the experimental results on Atari games -- how this could be related to robot learning? Last year's CoRL guideline also states that experiments on OpenAI gym locomotion tasks (e.g., Cheetah) should be supported with **intent** for transferring to real robots but I couldn't see any argument on it. How does the method solve challenges that would arise when learning robots in real-world? Moreover, as far as I know, there's still no successful application of DT on real robots. 
- Presentation can be improved -- Please do not assume that readers would know the details from the introduction. For instance, in Contribution point 2, readers would now know what *discriminator approach* and definitely would not know $\mathcal{L}_{\text{SimRCRL}}$ loss. It would be nice to provide high-level idea of your method in contributions without assuming the knowledge of your method that can be known after reading the method section.
- Eq 3: This is a question, how could the length of input and target sequence could be different in causal transformer architecture of GPT?

Minor comments
- line 26: \citet -> \cite
- line 30: DT's -> DT
- line 34: Reference is wrong. Why Multi-Game DT [12] is referred in the success of contrastive learning? Citing representative works like CPC, SimCLR, MoCo would be desirable. 
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I would like to suggest to reject that paper, because it seems that the paper is not relevant to the robot learning conference. The paper did not mention or show the intend to improve robot learning, which makes it difficult to see the paper is of interest to the CoRL community. Of course the paper could be augmented with additional robotics experiments, I think it needs a significant revision with excluding Atari experimental results and including new main results, ablations, and analysis, which makes it difficult to recommend the acceptance of the paper.

However, experimental results are still interesting, so I would like to recommend the authors to improve the presentation of the draft and then submit the paper to other machine learning conference whose topic includes generic offline RL method.
issues: - Not enough, or entirely no relevance to robot learning. Motivation should be more clear in the context of robotics, and the experiments should more focus on robotics tasks. Analysis and ablations on Atari domain is very awkward to be seen in a CoRL submission.
- Presentation can be improved
review_title: null

Meta-review: metareview: Phase 1:

Strengths:
The submission is intuitively and clearly written. The application of contrastive losses to decision transformers is clear, well presented and provides convincing improvements on some established benchmarks. Provided ablations are helpful.

Weaknesses:
The combination of DT and SIMCLR loss can be seen as straightforward, which would be fine with strong empirical improvements. While these are convincing on Atari, improvements to the simpler DT+Prod baseline in Gym are limited. Further, common offline RL baselines are missing and the comparison focuses on variations of DT as baseline. 
No real robot experiments are performed and missing clarity regarding the path to application was emphasized (maybe a discussion of computational feasibility would be helpful here). Additional details are missing to improve the presentation (e.g. embeddings, size comparisons between DT and ConDT, increased computational costs).

Phase 2:

The feedback has been a borderline case with positive and negative feedback. While it is seen as well written and intuitive, the contributions are marked as straightforward while improvements are limited. During the rebuttal, the authors have been able to convince all reviewers to change their ratings to weak accepts. Main criticisms towards comparing against offline RL state-of-the-art methods remain but the connection towards robotics has been strengthened with additional empirical results. 

The paper remains a very close borderline case which should be extended with additional baselines instead of purely focusing on the decision transformer. In the end, my recommendation follows the reviewer's suggestion of acceptance because of the added simulated robot experiments. However, I strongly suggest the addition of further baselines for a camera-ready version if accepted.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)