Contrastive Decision Transformers
Authors: Sachin G Konan
Esmaeil Seraj
Matthew Gombolay
Venue: CoRL 2022 Poster (2022)
Keywords: Reinforcement Learning
Decision Transformers
Contrastive Learning

Abstract: Decision Transformers (DT) have drawn upon the success of Transformers by abstracting Reinforcement Learning as a target-return-conditioned, sequence modeling problem. In our work, we claim that the distribution of DT's target-returns represents a series of different tasks that agents must learn to handle. Work in multi-task learning has shown that separating the representations of input data belonging to different tasks can improve performance. We draw from this approach to construct ConDT (Contrastive Decision Transformer). ConDT leverages an enhanced contrastive loss to train a return-dependent transformation of the input embeddings, which we empirically show clusters these embeddings by their return. We find that ConDT significantly outperforms DT in Open-AI Gym domains by 10% and 39% in visually challenging Atari domains.

Contrastive Decision Transformers
Sachin Konan"
Esmacil Seraj
Computer Science
Interactive Computing
Two Sigma Investments, LP sachin@twosigma.c com
Georgia Institute of Technology e edu
Matthew Gombolay
Interactive Computing
Georgia Institute of Technology matthew gombo gombolay@cc. ayocc. gatech. edu
Abstract: Decision Transformers (DT) have drawn upon the success of Trans- formers by abstracting Reinforcement Learning as penget-retunncoondiionnn quence modeling problem. In our work, we claim that the distribution of DT's target-returns represents a series of different tasks that agents must learn to han- dle. Work in multi-task learning has shown that separating the representations of input data belonging to different tasks can improve performance. We draw from this approach to construct ConDT (Contrastive Decision Transformer). ConDT leverages an enhanced contrastive loss to train return-dependent transformation of the input embeddings, which we empirically show clusters these embeddings by their return. We find that ConDT significantly outperforms DT in Open-Al Gym domains by 10% and 39% in visually challenging Atari domains. Addition- ally, ConDT shows promising application to robot learning by outperforming DT by 20% in the Adroit Robotic HandGrip Experiments.
se-
Keywords: Reinforcement Learning, Decision Transformers, Contrastive Loss
Introduction
Decision-making reasoning process resulting in the selection of belief course of actions that leads to desired outcome [1]. peeiiion-mmainn has been conventionally approached by Rein- forcement Learning (RL) methods [2] in robotics [3. 4, 5], game-playing [6. 8], and multi-agent collaboration [9, 10, 11, 12, 13. 7, 14, 15]. as well as by Learning from Demonstration (LiD) meth- framed as learning mapping from state observation to an action that maximizes the cumulative discounted environment rewards (i.e., return), and has seen success with traditional methods that value functions [18] or compute policy gradients [19]. Re- cently, Chen et al. [20] introduced the Decision Transformer (DT) to the realm of RL to draw from the simplicity and scalability of transformer architectures [21]. DT abstracts the decision-making process in RL as sequence modeling problem and attempts to learn celurn-conditione state- action mapping [20]. The return-condin condiioondiionnllityy means that given a history of return-state-an state-action tokens, such that the last token represents the desired return at the current-state. the DT predicts the action required to achieve this desired return. DT achieved promising performance in many decision- making problems [20, 22], but there still exists sizable margin for transformers to improve in the realm of RL as shown in the results of Chen et al. [20] where DT matched or performed worse than existing offline RL baselines in bevy of tasks.
ods [16, 17]. In RL, dacision-makingg
In this work, we hypothesize that integrating Contrastive Representation Learning (CRL) into the DT architecture will help DT learn a higher quality state-to-action mapping (i.e., policy) and achieve better convergence results. The objective CRL is cluster data representations with respect to their respective classes in order to strengthen the discriminability of the representations generated by Neural Networks (NN) [23, 24]. The power of CRL can be seen from its success in a large
Research was conducted while undergraduate at Georgia Tech.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

range of learning problems [23, 25J. Therefore, we investigate decision transformers trained with a contrastive loss, since DT an inherently eeturnconniitioed system (i.e., the predicted optimal action depends on the magnitude of user-defined returns at the current time-step).
The distribution of the target returns with which the DT is conditioned upon represents distribu- tion of sub-tasks with which the DT must learn to handle. To simultaneously handle the conflicting gradients of differing tasks, we posit that modifying the DT architecture to include a contrastive loss and an input transformation layer to transform the input data into mutually separated sub-spaces of the input-ding siput-dimension. We posit that this will implicitly help with the performance of DTs. We accomplish this process by introducing Contrastive Decision Transformers (ConDT), which enables learning discriminable aaput-space embeddings for DT by categorizing the embeddings and the sub- task to which they belong. Our new modified architecture allows DT to more easily choose the optimal action corresponding to the current state, desired return, and history of information. Our re- sults in Atari and Open-Al-Gyn environments show consistent improvements over the conventional DT by achieving significantly higher return- perun--erfoomncce. Our contributions are as follows:
1. We propose ConDT, modified, contrastive DT architecture and training procedure that uti- lizes CRL and return-dependent embedding -speee transformations to effectively discrimi- nate state and action embeddings belonging to different return classes.
2. We propose an enhanced CRL objective, which we call the LSimReRI. loss, for optimiz- ing representational distance of state and action embeddings. We show that our LSimRCRI. results in better separation of these embeddings in representation space.
3. We demonstrate that ConDTT improves return performance by average of 10% in Open- Al Gym, 39% in Atari, and 20% in the Adroit Robotic Handgrip experiments. These notable gains indicate the strength of applying CRL to the transformers' input embeddings.
2 Related Work
RL has seen great success in decision -aaoinn problems in robotics [3. 4], game-playing [6. 7], and multi-agent collaboration [15, 10, 11, 26]. For improved sample-efficieney, researchers have studied offline RL [4]. in which the agent learns from pre-recorded dataset rather than environment interactions. Offline RL methods are highly applicable to robot lcarning tasks with effective perfor- mance and has been leveraged in robotic hand and object manipulation tasks in prior work [27, 28). Q-learning is one of the most successful offline methods [18, 29], and with other methods, has benefited from observing history of states and actions, rather than just the last state [30]. We refer to this "history" as temporal context. Recurrent- architectures [31] were generally used with RL to solve domains (i.e., games) with temporal context, and with the introduction of transformers [21, 32), researchers have investigated replacing RNNs with transformers offline RI. [33, 34, 20). In the following, we discuss the latest applications of transformers to RL with respect to ConDT.
Transformers in RL Chen al. [20] introduced Decision Transformer (DT) as a replacement to recurrent-based RL architectures for cases when temporal context is known. Additionally, Jan- ner et al. [35] introduced Trajectory Transformer as a long-term planning framework for RL. In this work, we propose a modular input-embedding space transformation and contrastive procedure (ConDT), which was tested on DT, but applicable to other transformers. ConDT achieves success because its methods effectively categorize the distribution of returns as distinguishable sub-tasks. This is notably distinctive from Multi Game DT (MGDT), which augment the input sequence of DT with observations from multiple- gamlls showing that a single DT can generalize to multiple different games. MGDT trains a DT to solve multiple games, while ConDT trains a DT to better solve the sub-tasks within a single game.
Contrastive Learning for Transformers Contrastive Learning has been applied RL, contrast- ing representations in the state space [24, 36, 37] and the action space [38]. Liu et al. [39] recently introduced Return-Basss Based Contrastive Learning (RCRL) which applies a contrastive loss across the return-space in RL. Contrastive methods for transformers generally take advantage of the InfoNCE loss [24], which has seen success auto-regressive tasks. The InfoNCE loss has been success- fully applied to multi-modal video Transformers [40] and bi-directional language Transformers [23]. These methods are relevant to non-RL. domains, but our method, ConDT, attempts to apply RCRI. to the input embedding layers of DT, because DT's architecture is inherently return-conditioned.

---

Pre-Training Objectives for Transformers Transformers, in both vision and NLP applications, have benefited from pre-training [41, 42]. These works benefit from either introducing pre-training tasks related to the original objective [42], whereas other works perform pre-training
different training set and then fine-tune on the main set [43, 44]. ConDT builds off the intuition of the former works, but utilizes return-based contrastive objective to train the input-embedding space, unlike the pretraining techniques introduced by Devlin et al. [42].
3 Preliminaries
Problem Formulation - We consider the problem of learning in a Markov Decision Process (MDP) defined by the four-tuple: AS P, R), where S represents the state-space and A represents the action space. particular state and action at a time-step, is represented by 8t, Ot. respectively. P represents the transition function 0 < P(s++1| [sy, ae) S and R represents the reward function and R(sr, at). In this paper, we follow convention of the original DT
the reward is defined as: and define return, (gt), as the non-discounted rewards-to-go: Jt = >, rz. As stated earlier, DT is an offline RL method where the learner only from some fixed limited dataset (i.e., environment trajectories) instead of obtaining data via environment interactions. Offline RL is known to more challenging as it removes an agent's is ability to explore [20].
Decision Transformer (DT) Objective DT [20] takes input a sequence of three-tokens: Gt-K), K). (gt, St, (4)), where K is the game-specific context length. DT encodes
((gt- K. 8t-K; K; Gt- each token into embedding and adds a positional encoding to each embedding. The embeddings are then fed into the GPT-2 Causal Transformer [45] where an attention mechanism is applied to predict a left-shifted version of the input: ((3--K, B K. -t)K+1);
($,, âz 9c+1)). During in- ference, Gr is unknown and therefore, a₃ is null-token. For training, and for inference, the only relevant output token is âg which represents the model's predicted action that will maintain the re- turn in the current-state, Considering the true-optimal action as a and predicted action as â,, the training objective of DT in discrete and continuous environments can be shown in Eq.
|| arl1² aa (ag() if discrete, ag is an integer and å E A
if continuous,
LDr
Return-Based Contrastive Learning (RCRL) Objective - RCRL performs contrastive represen- tation learning state-action embeddings [39]. In any offline dataset, return-state-action tuples, (gt, St, dr). Returns will range from [Amin Imaz], which we will split into L distinct return buckets, 4xEL [by (lower), by (upper)). where by refers to the k-th return bucket that ranges from by (lower) to by (upper). RCRL adds discriminator, D(x, y). to the network architecture, which returns scalar indicating the degree of correlation between embeddings x and y. The RCRL loss (Eq. 2) samples a batch of B anchor state-action embeddings, Eah, whom each have some bin designation, bah. as well as B positive embeddings, Zp. whom have bin designation bp s.t. bp = bar, and also B embeddings, Zn, whom each have bin designation bn s.l. bn of bah-
given series of
LRCRL. 2B
Σ Zyo
((D(zah) =p) -p) -1)² (D(Eah;zz)) =x))²²
Methodology
Overview The full architecture of the proposed Contrastive Decision Transformer, ConDT, is shown in Fig. Contrary to the original DT [20], we create a sub-space transformation layer that represents return-dependent transformation of the state and action embedding spaces. The transformation layer is trained by our modified contrastive objective, the SimRCRL loss introduced in Eq. 4, discriminate state-action embeddings depending on target return.
Next, we present a theoretical motivation for our following sections on return- based sub-space trans- formations and our contrastive learning objective, SimRCRL.
Motivation and Insight - Consider a simplistic reduction of the problem formulation, wherein we are given memory bank BB containing set of three-tuples (9',,6; a() where the distribution of returns approximately uniform. Now, given some gt and 8t, what is the optimal o, that could achieve this ? A simple search-based solution could use k-nearest neighbors, whose complexity
3

---

a
by
5,4
D2:
Causal Transformer: Lr
5.0
44:
S.0
by
Embedding sub-space transformation: T ge-1
Embedding sub-space transformation:
5,0
SimRCRL Loss:
SImBCRL
Decision-boundaries represent RCRL buckets
8t-1
a--1
Et
Figure 1: The architecture of the proposed Contrastive Decision Transformer. We create sub- space transformation layer that represents a retur-dependent transformation of the state and action embedding spaces. The transformation layer is trained by our modified contrastive objective, the SimRCRL loss in Eq. 4, to discriminate state-action embeddings depending on target return, as shown in the box on right. The purple box state and action embedding compression layer.
([ +SS). where [9] the size of the return-space and S) the size of state-space. Upon finding subset of tuples that match the return, 9t. and state, their associated af can be chosen as the optimal action. However, imagine if the memory bank was organized by g' before-hand, Le {of [(5], a.),... .....: }. We index the memory bank by gr in O(1), resulting in [B//|/] state-action 2-tuples, and apply the k-NN across the state-space of the 2-tuples to find the optimal a,. Therefore, restructuring the memory bank reduces complexity plistic reduction of the problem formulation, because transformers effectively search through many combinatorial permutations of (g's s/c at). dependent on the context-length, however, this analysis motivates the benefit of pre-indexing the data by return. Since our data is high gigh-dimensional, we turn the transformations and contrastive methods in the following sections to effectively "index" the state and action embeddings by return.
This is sim-
Return-Based Embedding Sub-Space Transformation As stated in Section 1, the distribution of potential, conditionable returns represents a distribution of sub-tasks which DT must solve. Cheung al. [46] proposed several transformations that could be applied to the input data, such that data corresponding to different tasks existed in orthogonal sub-spaces of R-V, where N is the dimension of the input, x. They considered a task-dependent context transformation, Ck, where k E L and is the distribution of possible tasks. For single-layer, linear network (W), the output corresponding the k-th task can be written as yk W(Cxz).
In our formulation, we denote the transformation Cs. as Tg. which indicates a return-dependent, 9t. transformation applied to each embedding space. We note that, here remove gi from DT's input set of tokens, since the Tg. should effectively encode into the representations of & and a,, the 2f and 2f, respectively. Therefore, modify the input to the GPT [45] portion of DT as follows:
GPT (TT -0--,-2-K0 Ty. 2t- K. T 2, T gg. (St-K, K, at-Ka Kt-Ktt+
K+1," . St, Gt, Ic+1]
Now, we need to choose an optimal transformation function, Tg. To this end, we follow the static rotation-based approach proposed by Cheung et al. [46]. Following the return-bin notation of the RCRL objective in Section 3, we can generate the relevant rotation matrix. Rig, associated with some given return bucket-label, bg. by sampling some orthogonal matrix from the Haar Distribu- tion [46] for each possible bucket. Thus each sequence of (gt, St, a,) tokens can appropriately be transformed by defining Tg. Rog- s.t. & is the return bucket index that Je belongs to. Addition- ally, we design T to be learnable transformation, by considering Tg diagonal matrix whose entries are populated by the embedding generated by ft. this case, T₁ simply equivalent to the vector-product of the embedding generated by 9: and the embeddings generated by and However, with a learnable Tg there no guarantee that the transformed embeddings be distant in representation space. Accordingly, we develop a contrastive objective for T

---

Contrastive Learning of the Embedding Transformation We explore contrastive learning as direct optimization objective to train Tg push state and action embeddings belonging to differ- ent by to be more distant in representation space. The was also the motivation behind the RCRI. loss [39], introduced in Section which claims that training discriminator on pairs of state-action embeddings will cause the embeddings to grow distant/together in representation space. This claim was supported through an analysis the cosine-similarity change over time [39], although, in our experiments, we found that optimization of the discriminator required by RCRL did not always strictly achieve this result.
Here, we improve upon the RCRL objective, in Eq. 2, by first sampling a batch, B, of bins from L. Then, for each bin we sample two sets of (gt, St. 0z) tuples, where the first is considered the the anchor and the second the positive sample associated with the anchor. Therefore the loss con- siders a total of 2B samples. For given anchor and its associated positive, the rest of the other anchors/positives serve negative samples (i.e., 2N 2), with which we may contrast the embed- dings upon. We define the strict-RCRL. objective as the of the state-action embeddings, whose construction is drawn from SimCLR [47]. We call objec- tive SimLCRL, which defined as in Eq. 4, where the compressed representation of the i-th anchor, which is state-action embedding, is indicated by 2y and its associated positive pair embedding as =p. 1 (i, is the indicator function that is O when i = j and if not, and as the temperature hyper-parameter. Be the batch-size used in LSiRCRRL
entropy loss
Be
exp((='n ( 2/,//T)
LIIRRRI
log
BB
=//h/T) ((x,))(exp((Ha)
+ exp((=)n =b)/T]]
Now, putting everything together, our total contrastive loss for training ConDT, LCONDT, is the sum of DT's objective and our improved contrastive objective, weighted by parameter that weighs how much the contrastive objective should affect the DT during training. LCOnDT is shown in Eq. 5. consider pre-training the embeddings layers with LsimRCRL, followed by training the entire DT with slowed learning rate on the embedding layers and LDr loss.
During experimentation,
CCODD
Cer 3 LSimRCRL.
(5)
5 Empirical Evaluation
Baselines We tested our improvements upon the original DT architecture Chen et al. [20] by testing the below five methods. For each experiment, examine the following methods where (1) is the original baseline DT [20] architecture, (2-3) are ablations that study the impact of learnable vs fixed returntransforrmations of the state and action embeddings, and (4-5) study the combination of (1) and (3) with the LimmRRRL contrastive objective:
1. -ascline DT proposed by Chen et al. [20]
2. DT+Rot Baseline DT with fixed-rotation embedding boo-spacc-transformatto
3. DT+Prod - Baseline DT with learnable input-embedding transformation
4. ConDT w/o Prod (Ours) - Baseline DT' trained with our LCODDT loss
5. ConDT (Ours) PTod trained with our LCOLDT loss
Evaluation Environments - We empirically validate the enhanced performance of ConDT against baselines several decision-making domains, including Atari 2600, Open-Al Gym, and the Adroit Handgrip Environments. For environment descriptions and details, please refer to the pro- vided supplementary material. Next, we present our evaluation results and investigative studies of the learned representations. We note that we publicly provide our code-base (including ConDT implementation and the baselines) at pnttps://gitte com/CORE- /O////CRRRROOOO/ hete--ooootiii-ee Cerooiicc-aabCon11
5.1 Results, Ablation Studies, and Discussion
Open-AI Gym - We first measure return performance across three standard Open-Al Gym locomo- tion domains: hopper, halfcheetah, and walker2d. We did not test the Reacher domain, as done in DT [20], since the dataset for this domain not publicly available. Agents tasked with
5

---

achieving human- return performance (i.e., 3600 in hopper, 12000 in halfcheetah, and 5000 in walker2d for results in Table 1). In Gym and Atari, we report the mean, f4, and standard error, SE, of the return achieved in three random seeds.
We train the agents using the Medium (M.), Medium-Replay (M.R.), and Medium-Experr (M.E.) datasets provided by Fu et al. [48]. These datasets differ in size and the policies used to generate them. Thus, we tested on all three variations to study ConDT's robustness. Details of the distinction between these datasets can be found in the supplementary and in the original DT paper [20].
Dataset
Env.
DT [20]
DT+Rot
DT+Prod
ConDT w/o Prod (Ours)
ConDT (Ours)
hopper
2097::38.9
1397: 25.8
2330+:33.2
2293
2403+:30.6
M.
halfcheetah
1981128.1
50351:36.8
14.9:114.
5020:111.7
5060:11.7
walker2d
3466:+56.3
3453+555.9
3521+30.5
3609:-44.88
3545-+49.44
hopper
2393::43.4
13.6=0.07
1959=16.3
2869::29.2 2
3076:+27.7
M.R.
halfcheetah
4070:152.4
703.4
47911:30.6
44911
48031136.2
walker2d
2988:1114
968.3 966.3
65.7
3322:177.4
3332-176.6
hopper
3133:+6100
65.8
3354+556.9
3505+26.3
3871=8,49
M.E.
halfcheetah
10779=29.2
8956+1189
11021+62.1
10821+49.5
11218=14.5
walker2d
4941112.9
1195173.4
495112.61
4850135.6
501111.44
Table I: Baseline Gym test results for medium (M), medium-replay (M.R.), and medium-expert (M.E.) datasets (as described in [20]) of hopper, half-cheetah, and walker2d, shown as μ + SE
As shown in Table 1, DT+Prod and ConDT w/o Prod exceed the performance of DT, while ConDT exceeds or matches DT+Prod and ConDT w/o Prod in 8/9 tasks. This shows that learnable, return- based transformations benefit return performance, which can be further enhanced with the con- trastive training objective. RT+ either matches or performs slightly worse than DT, although this trend dramatically falters across the datasets in Medium-Replay. We hypothesize the reason for the significant performance dip is because the return distribution in medium-replay is much more leet-skewed than medium medium- eecitut.ee.pr..... This would cause the inputs to DT+Rot to only be transformed with Ra, where by corresponded to bins in the lower range of the eeturn-distribttion and thus harm inference when larger returns were provided to RTo.
Atari 2600 We also tested on Atari 2600 environments [49], which contains of suite of classic Atari Games that are generally considered challenging due to the high dimensionality of the obser- vation space ((210 x 160) RGB image) and time-delayed credit assignment rewards. We train each baseline on 1% of all samples in the DQN-replay dataset provided by Agarwal et al. [29], representing 500K out of the 50 million transitions observed by an online DQN. We tested all the baselines across four domains, Breakout, Qbert, Seaquest, and Pong, and results are in Table 2.
Env.
Target g:
DT' [20]
Do
DTo+Prod
ConDT w/o Prod (Ours)
ConDT (Ours)
Breakout
90
48.8:16669
42.7-22.08
70.4:1:.73
67.8:14.73
71.1-
Qbert
14000
3763+348
6877+3302
3677+300
10735-334
6432+314
Pong
20
16.8++0.77
11+:0.59
16.7+0029
13.2++0.59
07.9
Seaquest
1150
9487
880::32.1
1018:+32.7
1364:1:9.0
1250:+27..3
Table 2: Baseline Atari test results for Breakout, Qbert, Pong and Seaquest, shown as + SE.
Causal attention allows GPT [45] to understand the relation between each token and its preceding sub-sequence. Therefore, the removal of the return tokens from the input GPT, as is the case in DT+Rot, D'T+Pood and ConDT, gives rise to the concern of whether the removal will have a negative effect on the performance of the overall DT. As shown in Table 2, even without learn- able return- esturn-trane transforrmaiinn DT+Rot performs only slightly worse than DT across 3/4 of the Atari domains and doubles the performance of DT in Qbert, indicating return has successfully been en- coded into the state and action embeddings. With learned transformations, DT+prod and ConDT w/o Prod outperform DT in 3/4 of the experiments, except in Pong, where DT+Prod matches DT's performance, but most notably, in Qbert, ConDT w/o Prod nearly triples the performance of DT. We hypothesize that Co w/o Prod outperforms ConDT 2/4 tasks, because its causal attention allows it to relate return embeddings from any preceding time-step to specific state, action embed- ding, whereas in ConDT, return embeddings are only related to the state, action embeddings at the same specific timestep.

---

(a) Pen: orientate pen
(b) Hammer: hammer a nail
(c) Relocate: move blue ball
Figure 2: The Adroit Robotic HandGrip, 24 degree-of -freedom simulated robot hand domain, including three difficult manipulation tasks (i.e, Pen, Hammer, and Relocate).
Robotic HandGrip Manipulation - Previous works have studied the application of offline RL to robotics [27, 28). In our work, to study the applicability of ConDT to robot learning, we investigate the Adroit Robotic HandGrip Environments [27], wherein a degree-of-freede is trained to hammer nail (i.e., Hammer), orient pen (i.e., Pen). and relocate object (i.e., Relocate). Fig. shows a sample demonstration of these complex manipulation tasks. We detail ConDT's performance gains in Table 3, and provide visualizations of the trained policies in the supplementary material. We also provide a video demonstration of the simulated Adroit Robotic HandGrip executing the learned policies by DT and ConDT for each task supplementary material. The demo showcases the superior policy learned through our ConDT over the base DT..
robotic hand
Env.
Target 9r
DT [20]
DT+Prod
ConDT w/o Prod (Ours) ConDT (Ours)
Pen
4000
1565=159
1883+192
1560:1162
2016:119
Hammer
12000
44036=146
160
16078-198
26364=210
Relocate
4500
4331
4362:1447
4483:1402
4526:1:10
Table 3: The Adroit Robotic HandGrip test results for ConDT and baselines, shown as μ + SE.
5.2 Investigating the Learned Representations
We hypothesize that return-based transformation, Tg. of the input embedding spacee with the LCOND objective helps distance the state and action embeddings based on the return-class they be- long to. To support this hypothesis, study how the distance in the state and action embeddings change over time. Additionally, we visually investigate clustering behavior in the embedding-spaal
RCRL
SimRCRL (Ours)
0.8
Similarty
Similarity
0.5
0.5
0.4
0.3
300
500
200
Training Epochs
Training toochs
(a) Positive Similarity during Pre- Pre-Training
(b) Negative Similarity during Pre- pe-Training
Figure 3: Comparing the loss convergence between LRCRL. and LSiRRCRL
Cosine Similarity Analysis For this analysis, we seek to understand how the representation dis- tance of state-action embeddings from ConDT change over time w.r.t. two loss functions: LRCRL and LSIRRCRL For this purpose, we utilize the cosine similarity as the distance metric [47]. Here a value of represents parallel embeddings, and 0 represents orthogonal embeddings. Therefore, state-action embeddings of the same bins should desirably have similarity value close to (i.e., high positive similarity or low intra-class distance) while representations belonging to different bins should have similarity value close to 0 (i.e., low negative similarity or high inter-class distance).

---

During pre- traiiiin of ConDT with our modified LimmRRI objective, we sampled fixed number of data samples across different return bins. Then we measure the average cosine similarity to samples within the same return bin and different return bins, which we call positive and negative similarity respectively in accordance with [47]. In Fig. 3a and Fig. 3b, we show the positive and negative similarities from the contrastive training of ConDT for the Atari-Breakout experiment, compar- ing LIRRRRL with LRCRL. LSimRCRL results in more stable convergence of the positive-similarity and negative-cosine similarity, whereas CRCRL loss does not result in monotonic increase of the Ssstive-similarityy nor strict convergence to either or O for positive and negative similarity. There- fore, our enhanced LiibRRL objective is more effective in achieving the transformation-basse ob- jective we formulated for application to DTs.
Embedding Space Visualization - Here, we intend to visualize the state-action embedding- space for a comparison between DT and ConDT. The goal is to observe a clustering behavior as result of our embedding- space transformation with contrastive learning, such that the between-class distances would increase and the within-class distances would decrease between the embeddings. To visualize the joint state-action embedding space, we sampled a batch of ten return bins, with 30 samples per bin. Then, we generated the state-action embeddings of this input data for DT and ConDT, and created 2D TSNE visualization of the embeddings, shown in Fig. 4. As shown, DT demonstrates no clustering behavior of the state-action embeddings based on returm; points of the same color scattered amongst points of different colors (i.e., colors indicate the return bins). However, for ConDT, points of the same color are clearly clustered together and have distance to points of different color, indicating that the contrastive embedding effectively pushes state-action embeddings of different returns apart, and similar returns together.
Bin
Bin
Bin
Bin
Bin
Bn
Bm 7
Bn
Bin
Bin 10
DT
ConDT
20
20
Feature Dim 2
Feature Dim 2
10-
10-
0-
-10
-10
-20
-20
-30
-10
10
20
-20
20
Feature Dim 1
Feature Dim 1
Figure 4: TSNE 2D visualization of state-action embedding-spaces for DT and ConDT.
5.3 Limitations
limitation DT [20], which is shared in ConDT, is the size and distribution of returns in the offline data. As mentioned in the Open-Al Gym analysis, the left-skewed nature in the return distribution of medium-replay dataset harmed the performance of DT+ The original DT [20] addresses this concern by sampling trajectories according to the magnitude of their cumulative return; thus, causing higher return trajectories to be sampled more often. Therefore, sampling techniques and the return distribution of the dataset can become bottleneck in decision transformer methods.
6 Conclusion
Decision Transformer (DT) was one of the first introductions of transformers to RL, and while experimentation showed promise, there still lies performance gap between DT and existing offline RL methods [20]. In this work, we hypothesized that performing a return-dependent transformation of the input embeddings to the DT can help enhance return performance. We proposed Contrastive Decision Transformer (ConDT). ConDT performs an embedding-space transformation of the state and action embeddings, where the transformation is trained using our enhanced CsimRCRL objective, which we empirically show can better maximize distance in state-action embedding space. We verify the applicability to robot learning and strength of ConDT's naturn-based transformation and contrastive learning objective by testing across several Atari and Open-Al AI Gym domains, as well as multiple 24-DoF robotics handgrip tasks, in which we show significant performance gains.
8

---

Acknowledgments
This work is supported by the Naval Research Lab under grant 100173-21-1--G00 We also thank the Two Sigma for sponsoring travel to/from the conference for the first author. However we note that: "the views expressed herein are solely the views of the author(s) and are not necessarily the views of Two Sigma Investments, LP or any of its affiliates. They are not intended to provide, and should not be relied upon for, investment advice."
References
[1] W. Edwards. The theory of decision making. Psychological bulletin, 51(4):380, 1954.
[2] R. Sutton and A. Barto. Reinforcement learning: An introduction. MIT press, 2018.
[3] M. M. Botvinick. Hierarchical reinforcement learning and decision making. Current opinion in neurobiology, 22(6):956-966 2012.
[4] Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01644, 2020.
[5] E. Seraj, X. Wu, and M. Gombolay. Firecommander: An interactive, probabilistic multi-agent environment for heterogeneous robot teams. arXiv preprint arXiv:2011.01155, 2020.
[6] A. Rajeswaran, I. Mordatch, and V. Kumar. A game theoretic framework for model based reinforcement learning. In ICML, pages 7953-7963. PMLR, 2020.
[7] L. Pimentel, R. Paleja, Z. Wang, E. Seraj, J. E. Pagan, and M. Gombolay. Scaling multi-agent reinforcement learning via state upsampling.
[8] E. Seraj. Embodied team intelligence in multi-robot systems. AAMAS, pages 1869-1871, 2022.
[9] Sukhbaatar, R. Fergus, et al. Learning multiagent communication with backpropagation. NeurlPS, 29, 2016.
[10] S. Konan, E. Seraj, and M. Gombolay. Iterated reasoning with mutual information in coopera- tive and byzantine decentralized teaming. arXiv preprint 27X1v22201.08484, 2022.
[11] E. Seraj, Z. Wang. R. Paleja, D. Martin, M. Sklar, A. Patel, and M. Gombolay. Learning efficient diverse communication for cooperative heterogeneous teaming. In Proceedings of the 21 International Conference Autonomous Agents and Multiagent Systems, pages 1173- 1182, 2022.
[12] J. Foerster, I. A. Assael, N. De Freitas, and S. Whiteson. Learning to communicate with deep multi-agent reinforcement learning. NeurlPS, 29, 2016
[13] E. Seraj, Z. Wang, R. Paleja, M. Sklar, A. Patel, and M. Gombolay. Heterogeneous graph at- tention networks for learning diverse communication. arXiv preprint arXiv: 108.09568 2021.
[14] E. Seraj and M. Gombolay. Coordinated control of uavs for human-centered active sensing of wildfires. In 2020 American Control Conference (ACC), pages 1845-1852. IEEE, 2020.
[15] E. Seraj, A. Silva, and M. Gombolay. Multi-uav planning for cooperative wildfire coverage and tracking with -uality-of-servic guarantees. Autonomous Agents and Multi-Agent Systems, 36(2):1-39, 2022.
[16] L. Chen, R. Paleja, M. Ghuy, and M. Gombolay. Joint goal and strategy inference across heterogeneous demonstrators via reward network distillation. In Proceedings of the 2020 ACM/IEEE International Conference on Humon-Robor Interaction, pages 559-668, 2020.
[17] I. Chen, S. Jayanthi, R. R. Paleja, D. Martin, V. Zakharov, and M. Gombolay. Fast lifelong adaptive inverse reinforcement learning from crowdsourced demonstrations. In 6th Annual Conference Robot Learning.

---

[18] M. Hessel, J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
[19] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.:0:7777.63777 2017.
[20] L. Chen, K. Lu, A. Rajeswaran, K. Lee, A. Grover, M. Laskin, P. Abbeel, A. Srinivas, and
I. Mordatch, Decision transformer: Reinforcement learning via sequence modeling. NeurIPS, 34, 2021.
[21] A. Vaswani, N. Shazeer, N. Parmar. J. Uszkoreit, Jones, A. N. Gomez, E. Kaiser, and I. Polo- sukhin. Attention is all you need. NeurIPS, 30, 2017.
[22] K.-H. Lee, O. Nachum, M. Yang, L. Lee, D. Freeman, W. Xu, S. Guadarrama, I. Fis- cher, E. Jang, H. Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. 15241, 2022.
[23] C. Sun, F. Baradel, K. Murphy, and C. Schmid. Contrastive bidirectional transformer for temporal representation learning. arXive preprint (TM:::906.055330 1906.
[24] A. v. d. Oord, Li, and O. Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018
[25] K. He, H. Fan, Y. Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738, 2020.
[26] E. Seraj, L. Chen, and M. C. Gombolay, A hierarchical coordination framework for joint perception-action tasks in composite robot teams. IEEE Transactions on Robotics, 38(1):139- 158, 2021.
[27] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. arXiv preprint arXiv: 10087, 2017.
[28] S. Sinha, A. Mandlekar, and A. Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. Conference on Robot Learning, pages 907-917, PMLR, 2022.
[29] R. Agarwal, D. Schuurmans, and M. Norouzi. An optimistic perspective on offline reinforce- ment learning. In ICML, pages 104-114. PMLR, 2020.
[30] B. Bakker. Reinforcement learning with long short-term memory. NeurIPS, 14, 2001.
[31] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 173: 1780, 1997.
[32] A. Dosovitskiy, L. Beyer, A. Kolesmikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghani, M. Minderer. G. Heigold, S. Gelly, al. An image is worth 16x16 words: Transform- ers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[33] J. Q. Davis, A. Gu, K. Choromanski, T. Dao, C. Re, C. Finn, and P. Liang. Catformer: Design- ing stable transformers via sensitivity analysis. In ICML, pages 2489 2499. PMLR, 2021.
[34] E. Parisotto, F. Song, J. Rae, R. Pascanu, C. Gulcehre, S. Jayakumar, M. Jaderberg, R. L. Kaufman, A. Clark, S. Noury, et al. Stabilizing transformers for reinforcement learning. In ICML, pages 7487-7498. PMLR, 2020.
[35] M. Janner, Q. Li. and S. Levine. Offline reinforcement learning as one big sequence modeling problem. NeurIPS, 34, 2021.
[36] A. Srinivas, M. Laskin, and P. Abbeel. Curl: Contrastive unsupervised representations for reinforcement learning. arXiv preprint arXiv:2004.04 2020.
10

---

[37] M. Schwarzer, A. Anand, R. Goel, R. D. Hjelm, A. Courville, and P. Bachman. Data-efficient reinforcement learning with self self-predietive representations. arXiv preprint (xXiv:2007.05929 2020.
[38] M. Kim, K. Rho, Y.d. Kim, and K. Jung. Action-driven contrastive representation for rein- forcement learning. Plos one, 17(3)x0025456, 2022.
[39] G. Liu, C. Zhang. L. Zhao, T. Qin, J. Zhu, Li, representation learning for reinforcement learning, arXiv preprint 0rXiv22122.1096 2021.
N. Yu, and T.-Y. Liu. Return-based contrastive
[40] H. Akbari, L. Yuan, R. Qian, W.-H. Chuang, S.-F Chang, Y. Cui, and B. Gong. Vatt: Trans- formers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 2021.
[41] I.. Wang, F. Liang, Y. Li, W. Ouyang, H. Zhang, and J. Shao. Repre: Improving -up-ssuee vision transformer with reconstructive pre-training. arXiv preprint (txXiv:2201008855 2022.
[42] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.0488 04000, 2018
[43] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. arXiv preprint arXiv:1910. 10683, 2019.
[44] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, Z. Jiang, Q. Hou, and J. Feng. Deepvit: Towards deeper vision transformer. arXiv preprint arXiv:2103. 1886, 2021.
[45] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188:9, 2019.
[46] B. Cheung, A. Terekhov, Y. Chen, P. Agrawal, and B. Olshausen. Superposition of many models into one. NeurIPS, 32, 2019.
[47] K. Sohn. Improved deep metric learning with multi-class n-pair loss objective. NeurIPS, 2016.
[48] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven 2020.
reinforcement learning, arXiv preprint arXiv:2004.0722
[49] M. Bellemare, Y. Naddaf, Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. JAIR, 47:253-279, 2013.


Reviews:
[{"review_id": "i6i4Gwsiew", "reviewer": "Reviewer_hRuX", "summary": "This paper proposes ConDT, which applies contrastive learning to decision transformer. Specifically, the author utilize return-based contrastive learning to optimize input embeddings such that state-action pairs with similar returns are clustered in the embedding spaces. The authors showed that ConDT improves the performance of DT by a large margin on standard offline RL benchmark.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Strengths\n\n1. Simple yet effective method: given fixed dataset, contrastive learning can provide additional signals for NNs and the authors showed that it is useful for offline RL using decision transformer. The idea is simple and main components (decision transformer & return-based contrastive learning) are from existing works. However, its novel combination makes a big difference. \n\n2. Large gains on standard benchmark: the authors showed that ConDT outperforms DT on standard offline RL benchmarks. However, it would be also nice if the authors could compare to other state-of-the-art offline RL algorithms\n\n* Weaknesses\n\n1. Computational overhead: it would be nice if the authors also can point out some drawback from the proposed method. For example, additional auxiliary loss can increase the training time. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I'd like to recommend \"weak accept\" because (1) the proposed idea is well-motivated and (2) ConDT is very effective even though core idea is simple. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "* Comparing training time. \n\n* Comparison with other state-of-the-art offline RL algorithms\n\n* Evaluation on more hard domains such as ant-maze", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "fMjE5OPvgK", "reviewer": "Reviewer_werf", "summary": "The paper presents a new decision transformer method that combines decision transformers with contrastive learning for better representation learning, The authors perform contrastive learning on the embedding after the return-dependent transformation based on prior works RCRL and show that such an addition to decision transformer can lead to performance boost on tasks in D4RL and Atari games.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The idea of the method intuitively makes sense and is easy to understand. It is neat to see that decision transformers can leverage contrastive learning for better performance.\n\n2. The results of the proposed method seem to improve over vanilla decision transformer on standard benchmarks.\n\n**Weaknesses**:\n\n1. The paper appears to be a bit incremental in terms of novelty. The method is heavily built upon prior works decision transformer and RCRL. It seems to me that the only difference is that the authors use SimCLR rather than the simple regression loss in RCRL as the contrastive loss.\n\n2. The improvement over the decision transformer is rather marginal. In particular, the simple DT+Prod method can perform quite comparably to the proposed method, questioning the effectiveness of the contrastive loss. \n\n3. Real-world experiments are regrettably missing.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Based on my comments in the Strengths And Weaknesses, I think the novelty and the results of the paper are not convincing enough. I would vote for a weak reject.\n\n--------------------\npost-rebuttal updates: thank you for the response! I appreciate that the authors added real-world experiments and offered clarifications. Therefore, I've updated my score to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See the Strengths And Weaknesses section.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "Rs1GwB-_rcw", "reviewer": "Reviewer_JfJf", "summary": "This work introduces a contrastive loss for decision transformers, this is motivated by the need for better state, action representations for different returns. A new loss is introduced to separate such embeddings and is compared to a reasonable baseline. Policy performance using such a embedding+loss is shown to be improved on several OpenAI and Atari benchmarks. Ablations show which components of this algorithm are useful. They include studies of the embeddings and their separation during training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is very clearly written: the algorithm is clear, repeatable and compared to strong baselines on convincing benchmarks.\n* Additional ablations and studies of the embeddings confirm the intuitive motivation for such an alteration to DTs\n* The improvements are significant.\n\nWeakness:\n* It would be a much more convincing results with some real robot experiments, it seems like it works in simulated setups though.\n* One reason for using sequence models is to model non-markovian systems (quite common in robotics): returns cannot be determined by the current (s, a) alone and require historical (s, a). However, with the state-action embedding clustering this property is lost. It would interesting to see if there are cases of such setups where this markov-assumption clustering actually hurts.\n* Some details of the transformer architecture would be useful: in the comparisons of DT and ConDT are they controlled for the same number of parameters? What would happen if the transformers for both models were much bigger, surely a big enough transformer can model this separation if it useful to the task.\n* I assume there is one embedding for the state and another for the actions, but this was not clear. There are mentions of state-action embedding which makes it seem like they are embedded to a single space?\n* The results between ConDT w/o Prod and ConDT are mixed: it seems like it is not clear which method to use.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Decision transformers are a promising sequence modeling approach to reinforcement learning and this work suggests an improved version with a contrastive loss to separate state, actions for different returns. It would be useful for the community to reproduce this work on additional benchmarks, especially real robot setups. The contrastive loss introduced is not revolutionary and likely a large transformer has the modeling capacity to accomplish, however, in simulated benchmarks shown the results are convincing improvement over DT.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "* Repeat some of the comparison for different # of params of transformer models - fixed for all models within the comparison.\n* Some comments in the limitations about a (s, a) embedding without historical context when the transformer itself has that context.\n* Comment of limitations in different reward settings, a lot of settings use binary rewards would you still recommend this method."}}, {"review_id": "EbuwFzsEoK", "reviewer": "Reviewer_bC6T", "summary": "This paper proposes a Contrastive Decision Transformer (ConDT) that improves DT by treating the bin of target returns as each task and separating the representation of state and action embeddings into clusters labeled by each return bin. Specifically, the paper introduces a transformation layer for each bin and introduces a contrastive loss that makes embeddings from each transformation be apart from each other. The proposed method is evaluated on OpenAI Gym locomotion tasks and Atari games.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Idea is intuitive and interesting. Especially the intuitive explanation on how the proposed idea of separating the representations can effectively reduce the search space that would enable DT to perform better is interesting.\n- Experimental results are interesting with supporting ablations and analysis. Trends in ablations are not perfect, but it can be understood.\n\nWeaknesses\n- The main weakness I see from the paper is that the paper is somehow submitted to not-relevant conference. In CoRL guideline, it is stated that ```All CoRL submissions must demonstrate the relevance to Robot Learning through Intent—explicitly address a learning question for physical robots, or Outcome—test the proposed learning solution on physical robots.``` This concern is intensified when I see the experimental results on Atari games -- how this could be related to robot learning? Last year's CoRL guideline also states that experiments on OpenAI gym locomotion tasks (e.g., Cheetah) should be supported with **intent** for transferring to real robots but I couldn't see any argument on it. How does the method solve challenges that would arise when learning robots in real-world? Moreover, as far as I know, there's still no successful application of DT on real robots. \n- Presentation can be improved -- Please do not assume that readers would know the details from the introduction. For instance, in Contribution point 2, readers would now know what *discriminator approach* and definitely would not know $\\mathcal{L}_{\\text{SimRCRL}}$ loss. It would be nice to provide high-level idea of your method in contributions without assuming the knowledge of your method that can be known after reading the method section.\n- Eq 3: This is a question, how could the length of input and target sequence could be different in causal transformer architecture of GPT?\n\nMinor comments\n- line 26: \\citet -> \\cite\n- line 30: DT's -> DT\n- line 34: Reference is wrong. Why Multi-Game DT [12] is referred in the success of contrastive learning? Citing representative works like CPC, SimCLR, MoCo would be desirable. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would like to suggest to reject that paper, because it seems that the paper is not relevant to the robot learning conference. The paper did not mention or show the intend to improve robot learning, which makes it difficult to see the paper is of interest to the CoRL community. Of course the paper could be augmented with additional robotics experiments, I think it needs a significant revision with excluding Atari experimental results and including new main results, ablations, and analysis, which makes it difficult to recommend the acceptance of the paper.\n\nHowever, experimental results are still interesting, so I would like to recommend the authors to improve the presentation of the draft and then submit the paper to other machine learning conference whose topic includes generic offline RL method.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Not enough, or entirely no relevance to robot learning. Motivation should be more clear in the context of robotics, and the experiments should more focus on robotics tasks. Analysis and ablations on Atari domain is very awkward to be seen in a CoRL submission.\n- Presentation can be improved"}}]

Author rebuttal:

Decision: Accept (poster)