Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectories
Authors: Qinqing Zheng
Mikael Henaff
Brandon Amos
Aditya Grover
Venue: Submitted to ICLR 2023 (2023)

Abstract: Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements. We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting. Here, an agent has access to two sets of trajectories: labelled trajectories containing state, action, reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information.  For this setting, we develop a simple meta-algorithmic pipeline that learns an inverse-dynamics model on the labelled data to obtain proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy-labelled trajectories. Empirically, we find this simple pipeline to be highly successful --- on several D4RL benchmarks~\cite{fu2020d4rl}, certain offline RL algorithms can match the performance of variants trained on a fully labeled dataset even when we label only 10\% trajectories from the low return regime. Finally, we perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e.g., inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.

Under review as a conference paper at ICLR 2023
SEMI-SUPERVISED OFFLINE REINFORCEMENT LEARN- ING WITH ACTION-FREE TRAJECTORIES
Anonymous authors Paper under double-blind review
ABSTRACT
Natural agents can effectively learn from multiple data sources that differ in size, quality, and types of measurements. We study this heterogeneity in the context of offline reinforcement learning (RL) by introducing a new, practically motivated semi-supervised setting. Here, an agent has access two sets of trajectories: labelled trajectories containing state, action, reward triplets at every timestep, along with unlabelled trajectories that contain only state and reward information. For this setting, we develop a simple meta- mhta-algorithmic pipeline that learns an inverse- dynamics model on the labelled data to obtaín proxy-labels for the unlabelled data, followed by the use of any offline RL algorithm on the true and proxy- labellade trajectories. Empirically, we find this simple pipeline to be highly successful - several D4RL benchmarks (Fu et al., 2020), certain offline RL algorithms can match the performance of variants trained on a fully labelled dataset even when we label only 10% trajectories from the low return regime. Finally, perform a large-scale controlled empirical study investigating the interplay of data-centric properties of the labelled and unlabelled datasets, with algorithmic design choices (e.g.. inverse dynamics, offline RL algorithm) to identify general trends and best practices for training RL agents on semi-supervised offline datasets.
INTRODUCTION
One of the key challenges with deploying reinforcement learning (RL) agents is its prohibitive sample complexity for real-world applications. Offline reinforcement learning (RL) can significantly reduce the sample complexity by exploiting logged demonstrations from auxiliary data sources (Levine et al., 2020). However, contrary to curated benchmarks in use today, the nature of offline demonstrations in the real world can be highly varied. For example, the demonstrations could be misaligned due to frequency mismatch (Burns et al., 2022), use different sensors, actuators, or dynamics (Reed et al., 2022; Lee al., 2022), or lacking partial state (Ghosh et al., 2022; Rafailov et al., 2021; Mazoure et al., 2021), ) or reward information (Yu et al., 2022). Successful offline RL in the real world requires embracing these heterogeneous aspects for maximal data efficiency, similar to learning in humans.
In this work, we propose a new semi- supervieer setup for offline RL. Standard offline RI. assumes trajectories be sequences of obser- vations, actions, and rewards. However, many data sources, such as videos or third-person demonstrations lack direct access actions. Hence, we propose a semi-supervised setup, where an agent's 's offline dataset also consists of uution-unlabelled trajectories in addition the aforementioned
nopper-medium-experte
10
complete trajectories action-tree trajectories
B
density
trajectories. Standard offline RL algorithms, such as Conservative Q Learning (CQL; Kumar al. (2020)) or Decision Transformer (DT; Chen et al. (2021)), cannot directly operate on such unlabelled trajectories. At the same time, naively throwing out the unlabelled trajectories can be wasteful, espe- cially when they have high returns. Our goal in this work to enable compute and data efficient learning with additional action-unlabelled trajectory logs.
2
0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 normalized return
Figure 1.1: An example of the la- belled and unlabelled data distri- butions.
Unlike traditional semi- supervisd learning, our setup has few key differences. First, we do not assume that the distribution of the labelled and unlabelled trajectories necessarily identical. In realistic scenarios, expect these be different with unlabelled data

---

Under review as a conference paper at ICLR 2023
having higher returns than labelled data e.g., videos of human professional are easier to obtain than installing actuators for continuous control tasks. We replicate such varied data quality setups in some of our experiments; Figure 1.1 shows illustration of the difference in returns between the labelled and unlabelled dataset splits for the hopper-med um-expert D4RI, dataset. Second, end goal goes beyond laheling the actions in the unlabelled trajectories, but rather we intend to use the unlabelled data for learning a downstream policy that is better than the behavioral policies used for generating the offline datasets. Hence, there are two kinds of generalization challenges: generalizing from the labelled to the unlabelled data distribution and then going beyond the offline data distributions to get closer the expert distribution. Regular offline RL is concerned only with the latter. Finally, we are mainly interested in the case where a significant majority of the trajectories the offline dataset are unlabelled. One motivating example for this setup is learning from videos or third-person demos. There are tremendous amounts of internet videos that can be potentially used to train RL agents, yet they are without action labels and are of varying quality.
Our paper seeks to answer the following questions:
1. How can we utilize the unlabelled data for improving the performance of offline RL algorithms? How does our performance vary as a function of data-centric properties, such as the size and return distributions of labelled and unlabelled datasets?
3. How do offline RL algorithms compare in this setup?
To answer these questions, we propose a meta-algorithmic pipeline to train policies in the semi- supervised setup deseribed above. We call our pipeline Semi-Supervised Offlin Reinforcement Learning (SS-ORI). SS-CRF EC-L contains three simple and scalable steps: (1) train a multi-transition inverse dynamics model on labelled data, which predicts actions based on transition sequences, (2) fill in proxy-actions for unlabelled data, and finally (3) train an offline RL agent on the combined dataset. Empirically, we instantiate SS SS-ORL with CQL (Kumar et al., 2020), DT (Chen et al., 2021), and TD3BC (Fujimoto& & Gu, 2021) as the underlying offline RL algorithms respectively, and conduct experiments on the D4RL datasets (Fu et al., 2020). We highlight a few predominant trends from our experimental findings below:
Given low-quality labelled data, SS-ORT agents can exploit unlabelled data that contains high- quality trajectories and thus improve performance. The absolute performance of SS-ORL is close to or even matches that of the oracle agents, which have access to complete action information.
2. When the labelled data quality is high, utilizing unlabelled data does not bring significant benefits.
3. The choice of value vs. behavior cloning based methods can significantly affect performance in the semi-supervised setup. our experiments, CQL and TD3BC are less sensitive to the missing actions compared to DT. They enjoy better absolute performance when the labelled data is of low quality, and their performance gap relative to the oracle agent is also smaller. See Appendix H for more details.
2 RELATED WORK
Offline RL The goal of offline RL is learn effective policies from fixed datasets which generated by unknown behavior policies. There are two main categories of model-frec offline RL methods: value-based methods and behavior eloning (BC) based methods.
Value-based methods attempt to learn the value functions based on temporal difference (TD) updates. There is line of work that aims to port existing off-policy value-based online RL methods the offline setting, with various types of additional regularization components that encourage the learned policy to stay close to the behavior policy. Several representive techniques include specifically tailored policy parameterizations (Fujimoto et al., 2019; Ghasemipour et al., 2021), divergence- regularization on the learned policy (Wu et al., 2019; Jaques al., 2019; Kumar al., 2019), and regularized value function estimation (Nachum et al., 2019; Kumar et al., 2020; Kostrikov et al., 202 la; Fujimoto& & Gu, 2021; Kostrikov et al., 2021b)
Recently, growing body of work has tried to formulate offline RL as supervised learning prob- lem (Chen et al., 2021; Janner et al., 2021; Emmons et al., 2021). Compared with the value- methods, these methods enjoy several appealing properties including algorithmic simplicity and training stability. Generally speaking. these approaches can be viewed as conditional behavior cloning methods (Bain & Sammut, 1995), where the conditioning parameters related information such
2

---

Under review as a conference paper at ICLR 2023
as goals rewards. Similar to value-baseds methods, these can be extended to the online setup as well (Zheng et al., 2022) and demonstrate excellent performance in hybrid setups involving both offline data and online interactions.
Semi-supervised Learning Semi-supervised learning (SSL) is a sub-area of machine learning that studies approaches to train predictors from a small amount of labelled data combined with large amount of unlabelled data. In supervised learning, predictors only learn from labelled data. However, labelled training examples often require human annotation efforts and are thus hard to obtain, whereas unlabelled data can be comparatively easy to collect. The research on seni-supervised learning spans several decades. One of the oldest SSL techniques, self-training, was originally proposed in the 1960s (Fralick, 1967). There, a predictor is first trained on the labelled data. Then, at each training round, according to certain selection criteria such as model uncertainty, a portion of the unlabelled data is annotated by the predictor and added into the training set for the next round. We refer the readers to Zhu (2005); Chapelle et al. (2006); Ouali et (2020); Van Engelen & Hoos (2020) for comprehensive literature surveys.
Imitation Learning from Observations There have been several works in imitation learning (IL) which do not assume access to the full set of actions, such as BCO (Torabi et al., 201 8a), MoBILE (Kidambi et al., 2021). GAIFO (Torabi et al., 2018b) or personn IL approaches (Stadie et al., 2017; Sharma et al., 2019). The recent work of Baker et al. (2022) also considered setup where a small number of labelled actions are available in addition to a large unlabelled dataset. A key difference between our work and these is that the IL setup typically assumes that all trajectories are generated by an expert, unlike our offline setup. Further, some of these methods even permit reward-free interactions with the environment which is not possible in the offline setup.
Learning from Videos Closely related to IL from observations, several works (Schmeckpeper et al., 2020b;a) consider training agents with human video demonstrations, which are without action annotations. Distinct from out setup, in those works, the offline observational data (videos) are from different embodiment. Moreover, the agents can interact with the environment, and can even collect reward information sometimes.
3 SEMI-SUPERVISED OFFLINE REINFORCEMENT LEARNING
Preliminaries We model our environment as Markov decision process (MDP) (Bellman, 1957) denoted by (S, A. P. R, 7). where S is the state space, AA the action space, p(x1) is the distri- bution of the initial state, P(s++1 8t, at) is the transition probability distribution, R(s,, 04) is the deterministic reward function, and is the discount factor. At each timestep I, the agent observes a state s, and executes an action a E A.. As response, the environment moves the agent the next state Sr+1- P(-[st, ar), and also returns the agent a reward rt = R(s;,a;).
3.1 PROPOSED SETUP
We assume the agent has access to a static offline dataset Tomlne- The dataset consists of trajectories collected by certain unknown policies, which are not necessarily optimal. Let denote trajectory and |T| denote its length. We assume that all the trajectories in Toftine contain complete rewards and states. However, only a small subset of them contain action labels, while most of the trajectories are missing actions.
We are interested in learning policy by leveraging the offline dataset without interacting with the environment. This setup is analogous to semi semi- supervipee learning, where actions serve the role of labels. Hence, we also refer to the complete trajectories as labelled data (denoted by T Jubelled) and the action-free trajectories as wnlabelled data (denoted by Tunlabelled). Further, we assume the labelled data are sampled from distribution P Plabelled and the unlabelled data are sampled from P uullabelled- In general, the two distributions can be different. Practically, one case are particularly interested in is when Plabelled generates low-to-moderate quality trajectories, whereas P Punlabelled generates trajectories of diverse qualities including ones with high returns.
Our setup shares some similarities with state-only imitation learning (ljspeert et al., 2002; Bentivegna et al., 2002; Torabi et al., 2019) in the use of action- untioonllleeeleed trajectories. However, there also some key differences. In state-only II., the unlabelled demonstrations are from the same distribution as the labelled demonstrations and correspond to near-optimal expert policy. In our setting, both
3

---

llheeleed and P unllahelled can be different from each other and also from the expert policy. Further, many state-only imitation learning algorithms (e.g., Gupta et al. (2017); Torabi et al. (2018a;b); Liu et al. (2018); Sermanet et al. (2018)), similar to their original counterparts (c.g., Ho & Ermon (2016); Kim Under review as a conference paper at ICLR 2023
Algorithm 1: Semi-supervised offline RI. (SS-ORL)
Input: trajectories Tunaka and Timishellat IDM transition size k offline RL method ORL,
// train a stochastic multi-traan -ransitionn IDM using the labelled data
0 argming ket. log (t
// fill in the proxy actions for the unlabelled data 3 Uproay + 2
for each trajectory r E Tuntabetied do à, mean of N (ug(st- [g(+- t= 1, 1,....171
Tprox with proxy actions (āe) [ filled in
T proxy - T presy U (Teaay)
// train an offline RL agent us using the combined data π - policy obtained by training ORL using dataset Tunelled Uprnyy
Output: π
et al. (2020)), permit (reward-fire) (fre)) interactions with the environments. This is not possible in our proposed offline supervupe setup where the agents are only provided with Tabelled and Jnnlahelled
3.2 TRAINING PIPELINE
RL policies trained on low to moderate quality offline trajectories are often sub-optimal, as many of the trajectories might not have high return and only cover a limited part of the state space. Our goal is to find a way to combine the action labelled trajectories and the unlabelled action-free trajectories, so that the offline agent can exploit structures in the unlabelled data to improve performance.
One natural strategy is to fill in proxy actions for those unlabelled trajectories, and use the annotated data together with the labelled data as whole to train an offline RL agent. Since assume both the labelled and unlabelled trajectories contain the states, we can train an inverse dynamics model (IDM) that predicts actions using the states. Once we obtain the IDM, use it to generate the proxy actions for the unlabelled trajectories. Finally, we combine those proxy-labelled trajectories with the labelled trajectories, and train an agent using the offline RL algorithm of choice. In particular, we propose stochastic multi-transition IDM (see Section 3.3), which is favored by our experiments. Our meta-algorithmit pipeline summarized in Algorithm 1.
Remarks. The annotation process, which involves training an IDM on the labelled data and generat- ing proxy actions for the unlabelled trajectories, is similar to one step of self-training (Fralick, 1967). A key difference is that in self- eeff-traniing the predictor is trained in multiple rounds. Once an initial predictor is trained, annotated data is selected according to certain criteria, and added into the training set for the next round. As opposed to self training we do not retrain the IDM but directly move to the next stage, where we train the agent using the combined data.
used for obtaining annotations on the unlabelled dataset. Then, a subset of
There are a few reasons that we do not employ self-training for IDM. First, it is computationally expensive to execute multiple rounds of training. More importantly, end goal is to obtain downstream policy with improved performance via utilizing the proxy-labelled data. One commonly used data selection criterion for self-training is based on the model uncertainty. There, one adds the proxy-labelled data with sufficiently low predictive uncertainty into the training set for the next round. However, we empirically found that such an uncertainty based augmentation strategy did not improve the performance of SS-SSS0 SOOL agents. See Section 4.3 and Appendix F for the experiment details.
3.3 STOCHASTIC MULTI-TRANSITION INVERSE DYNAMIC MODEL
In past work (Pathak et al., 2017), the IDM typically maps two subsequent states (st, 8t,,8)+)) to az We introduce multi-transition IDM that predicts a, using both transitions before and after timestep t, which we found works better empirically. More precisely, our inverse dynamic model which predicts a, using 2k + 1 transitions, including the current transition (st, St4,), the previous & transitions that leads to 8,, and the next k transitions starting from 91+1- We call k the transition size parameter.
4

---

Under review as a conference paper at ICLR 2023
Let St-A:t+k+1 denote the sequence (min(0,t- R.). St, 8t-1..... (max(1)))++++)) Specifically, we model the distribution of a, as a multivariate Gaussian distribution with a diagonal covariance matrix:
Eo(st-R:++++))
(1)
Let (a k:+4++++)) ) be the probability density function of N (uo (1o(s)-- (s =
Given the labelled trajectories Thahelled. we minimize the negative log-likelihood loss Note that the standard IDM which predicts de from
Bei -Tade [- log (9((u)/st-::+++111 (31, Sc-1) under the lg loss, is a special case subsumed by our model: it is equivalent to the case == O and the diagonal entries of Eg (i.c., the variances of each action dimension) are all the same. Choosing k > O allows us to account for nan-Markovian behaviour policies and partially observable MDP (POMDP), see Appendix E. For all the experiments in this paper, use ie = 1. We ablate this design choice in Section 4.3.
4 EXPERIMENTS
Our experiments aim to answer three primary questions:
1. Can SS-ORL closely track or even match the performance for fully supervised offline reinforce- ment learning, when only a small subset of trajectories are labelled?
2. How does the performance of SS-ORL vary a function of the size and qualities of the labelled and unlabelled datasets?
3. Do different offline RL methods respond differently under varying setups of data size and qualities?
To answer these questions, we focus on three Gym locomotion tasks hopper, wa ker, and hal fcheetah, and we use the v2 med i um-expert, med i um and med i imm-repla aa datasets from the D4RL benchmark (Fu et al., 2020). We address the first question in Section 4.1 and the other two in Section 4.2, respectively. Finally, we discuss the design choices for SS- SORL in Section 4.3.
4.1 BENCHMARKING
Data Setup For given offline dataset, we subsample 10% of the total trajectories from the dataset, whose returns are from the bottom 9%, & q & 100. We keep the actions for those trajectories, and discard the actions for the rest. We call this setup the coupled setup, since P labellldd and P unlabblled will change simultancously when we vary the value of q. When = 100, we are uniformly sampling the trajectories and we have Plabelled = P unlaabelled-- Under this setup, we always have 10% trajectories labelled and 90% unlabelled, and the total amount of data used later for training the offline RL agent is the original offline dataset size. This allows to easily compare
results with results under the standard, fully labelled setup. In Section 4.2, we shall decouple the distributions P Plabelled and P Punlabelled for thorough understanding of their individual influences.
Inverse Dynamic Model We train an IDM as described in Section with parameter k = 1. In other words, the IDM predicts a using consecutive transitions: ($t-1, St, 8141, S1+2). The mean and the covariance matrix are predicted by two independent multilayer perceptrons (MLPs), each contains two hidden layers and 1024 hidden units per layer. To prevent overfitting, we randomly sample 10% of the labelled trajectories as the validation set, and use the IDM that yields the best validation error within 100k training iterations.
Offline RL Methods We instantiate Algorithm with DT, CQL and TD3BC (Fujimoto & Gu, 2021) and test their performances. Among these methods, DT is recently proposed conditional behavior cloning (BC) method that uses sequence modeling tools to model the trajectories; CQL is a representative value-based offline RL method; and TD3BC is a hybrid method which adds BC term to regularize the Q-learning updates. We refer to those instantiations as SS-DT, SS- CQLL and SS- SS-TD33C, respectively. We defer the implementation details to Appendix A.
We compare the performance of those SS-ORL agents with corresponding baseline and oracle agents. The bascline agents are trained on the labelled trajectories only, and the oracle agents are trained on the full offline dataset with action labels. Intuitively, the performances of the baseline and the oracle agents can be considered as the (estimated) lower and upper bounds for the performance of the OR- agents. For each method, we train 5 instances under different seeds, and
Results
Due the space limit, the results on medi i um and m um-repl ay datasets deferred to Appendix C.
5

---

Under review as a conference paper at ICLR 2023
hopper medpirr
nopper-mediue
TD3BC
normalized retum
normalized retumm
0.8
00.0
2.8
0.6
0.6
0.6
CS
0.4
0.4
Dabaseline orade
0.4
baseline
oracle
TD39C
100
100
50
100
(labelled data quality param)
0 (labelled data quality param)
(labelled data quality param)
walter2dmmeium-spptt .OLL
walber2d-medd weller22-meiuu-eppprr DT
wwlker2d-mediumeeseee TD3BC
7.0
1.0
3.0
nom 3ized rotur
nomallzen altzed return
0.8
0.8
Nommlltadd..o
0.6
66
0.6
-
55 CQL baveline
S5 DT
+ 55 TDBBC
DT baveline
1039C baselne
oracle
DT Grade
TD39C oracle
20
100
20
100
20
200
(labelled data quality param!
(labelled data quality param))
(labelled data quality param)
malthectah-mddiumm expertt COL
mediesa expert
halfcheetah- cettiiaa-mmmummmpeeeee
-X--
55-COL
55 OT
1.0
CQL baseline
DT basaline
1.0
normatzos aedatoodd return
E2t oracle
normellzd return
0T onarle
0.8
0.
0.8
narmal
0.6
0.6
5S-TDIB
0.4
0.4
0.4
TD39C baseline
Tes9c oracle
20
40 60 80
100
20
40
80
100
20
40
60
80
100
9 (labeleed data quality param)
4 (labelled data quality param)
0 @Jabelled data qua param)
Figure 4.1: The return (average and standard deviation) of SS-ORL agents trained on the D4RL medi um-eperrt datasets. The SS-ORL agents able to make use of the unlabelled data to improve their performance, and can even match the performance of oracle agents trained on the fully labelled offline datasets.
for each instance deviation after 200k iterations. Figure 4.1 plots the results on medium-expe the three environments and all the three offline RI. methods, the 5s SS-ORL agents improve upon the baselines. Remarkably, even when the labelled data quality is low, the SS-ORL agents are able to obtain decent returns. For example, when q = 10, i.c., the labelled trajectories are the bottom 10% trajectories, the average return obtained by SS DSSTDBBC is 0.93, 0.91 and 0.79 for hopper, walker and ha wwlfeeheetahhee On average, this is 87.4% relative the oracle performance (1.02, 1.1 and 0.89). As the value q increases, the labelled data quality increases and the distributions P labelled and unlabelled are getting closer. The performance of the SS-ORL agents also keeps increasing and finally matches the performance of the oracle agents. Similar observations can be found in the results of medi um and med i um- replae ay datasets, see Figure C. and C.2. We found relatively suboptimal results for DT on hal walffcheettto aa in all cases, consistent with prior results in Zheng et al. (2022).
30 evaluation trajectories. We report the average return and the standard
datasets. For all
4.2 ABLATION STUDY
We conduct experiments to understand the semi- supervised approach from the perspective of both datasets and learning algorithms. For a systematic study, we depart from the coupled setup in Section 4.1 and consider a decoupling of the labelled data distributions P Plabelled and the unlabelled data distribution P Ponlabelled- We first vary the quality of the labelled and unlabelled trajectories, and examine how the final performance of those SS -RRL agents changes. Next, we vary the size of the labelled and unlabelled trajectories and investigate their influences. To understand how the value-based methods and the BC methods will potentially react differently under these data setups. we report the results of SS CCQL and 5S-DT for the aforementioned setups. Last, we ablate the design choice of the transition size k for the proposed IDM. For all the experiments we present, the results are aggregated over instances with different seeds.
Quality of Unlabelled Data We divide the trajectories from the hopper-med um-expert rt dataset into 3 groups, which consist of trajectories whose returns are the bottom 0% to 33%, 33% to 67%, and

---

Under review as a conference paper at ICLR 2023
SS-DT| High
55-DT Iabelled-Ned
5S-DT labelled-Low
1.0
1.0
normalizos return
1.0
normalzoo
0.8
Un bow
Daseehee (sbeled dats mny!
0.8
Stadle
Medl-----e Med
0.6
stadie
obdlledd--to High
otadle
105
35508
2x18
55
2x10P
55.10"
105
1
28100
Iteratio
Reration
(teration
SS-COL HabelledHHgh
55-C10 labe -ee-
55-0OL labelled-LLo
porm normatized return.
besalin Oabaled data only)
rormllbed
unabblled-
hommalioed
crade
0.6
0.6
cradie
0.4
uhibblledd High
0.4
24
crade
Iteration
Reration
(a) We fix the labelled data quality and vary the unlabelled data quality. When the labelled data quality is low or moderate, SS-ORL can significantly improve the performance upon the baselines by utilizing high quality unlabelled data.
S5-DT unlaballed-Hopt
SS-DT unlabellled-ee Med
S5:DT uriabelled-Low
nomalter return
3.0
1.0
nomalized return
1.0
0.0
labe sod-Low
0.8
oack
nommllzee
labe lod Hol
pon
otscle
labe tigh
0.4
Sx10f
10
1.5100
2x30
5x10
10
1.5
2x10
5x'30=
10
2x10
Itration
Reration
ation
SS-COL unlabelled-Hggh
SS-COL| uniabelled-Med
55 SS-COL | uniabelled--ae LLow
Formalized retum
Formolizzed retum
carmalzeed return
eade
0.6
0.4
0.1
5x10
iteration
Reration
(b) We fix the unlabelled data quality and vary the labelled data quality. The performance of ss SRL improves as the labelled data quality increases.
Figure 4.2: The return (average and standard deviation) of SS-DT DT and SS-CQL agents trained on the hopper-medi um expert dataset, when the qualities of the labelled and unlabelled data vary, Both the sizes of the labelled and unlabelled data are 10% of the offline dataset size.
67% to 100%, respectively. We refer to them Low, Medi um, and Hi high quality groups. In particular, the High group contains trajectories generated by the expert agents (Fu et al., 2020). As before, we report the performance of DT and CQL agents trained on the labelled data only as the baselines. We also report the results under the oracle mode, where we fill in the unlabelled trajectories with the true actions, and combine them with the labelled trajectories to train offline RL agents.
We first report the performance of SS-DT when the labelled data is sampled from the Hig high group, and the unlabelled data are sampled from Low, Med, and High groups, respectively. Both the size of the labelled and unlabelled trajectories are 10% of the total offline dataset size. The top left panel of Figure 4.2a plots the results. Clearly, when the labelled data quality is high, training on the labelled data only is sufficient to achieve the expert performance, and adding unlabelled data does not bring extra benefits. We repeat the same experiment when the labelled data is sampled from Medi um and Low, see the top middle and top right panels of Figure 4.2a. For those cases, adding unlabelled data with the higher or the same quality* improves the performance, whereas the lower quality unlabelled
When the labelled and unlabelled data are sampled from the same quality group, are simply adding data from the same distribution.
7

---

Under review as a conference paper at ICLR 2023
55-DT
Cases
Labelled Quality L
Labelled Quality
labeled labeled 10% labeled
0.60
0.75
0.90
1.05
0.60
0.75
0.90
1:05
0.60
0.90
1.05
Normalized Retur
S5 COL
All Cases
Labelled Quality L
Labelled Qualit -
50% labeled
labeled
0.60
0.75
0.90
1.05
0.60
0.90
1.05
0.75
0.90
1.05
Normallzzd Retur
We fix the unlabelled data size
10% of the offline dataset size, and vary the labelled data size.
S5-DT
Cases
Labelled Quality
Labelled Quality T
50% unlabeled
25% unlabeled
10% unlabeled
0.60
0.75
0.90
1.05
0.60
0.75
0.50
1.05
0.60
0.75
0.90
1.05
Normalized Return
S5-COL
Cases
Labelled Quality
Labelled Quality
50% unlabeled
25% unlabeled
10% unlabeled
0.60
0.75
0.90
1.05
0.60
0.75
0.50
1.05
0.60
0.75
0.90
1.05
Normallzzd Return
(b) We fix the labelled data size to be 10% of the offline dataset size, and vary the unlabelled data size.
Figure 4.3: The 95% stratified bootstrap CIs of the interquartile mean returns of SS-DT and S5-CQL when the sizes of the labelled and unlabelled data change. SS-CQL is almost insensitive to the changes, whereas ST-DT is more sensitive when the labelled data quality is low.
data is not significantly helpful. The performance of SS-COL follows the same trends, see the bottom panels of Figure 4.2a
To summarize, the experiments provide strong evidence that when the labelled data is of low or moderate quality, SS-ORL is capable exploit the high quality unlabelled data and remarkably boosts the performance compared with the baselines. The resulting performance is close to that of the oracle agent, and is often optimal (at least 1) or near-optimal (close to 1).
Quality of Labelled Data Similarly, we fixed the unlabelled data quality and vary the quality of the labelled data. Figure 4.2b shows the results. For both S5-DT and ss SS-CQU, increasing the labelled data quality raises the performance for all the cases.
Size of Labelled Data We train ORL agents where we fix the number of unlabelled trajectories to be 10% of the total number of offline trajectories, and vary the number of labelled trajectories as 10%, 25%, and 50% of the total size. Similar to the above experiments, we consider four data quality setups, where the labelled and unlabelled trajectories are sampled from the bottom half (denoted by L) and top half (denoted by T) trajectories, respectively. We consider both hopper-med um um-expe expert rt and wal 1 ker-med i um-expert datasets. To take account of different environments and data setups, we report the 95% stratified bootstrap confidence intervals (CIs) of the interquartile mean of the return for all these cases and training instances (Agarwal al., 2021). We use 50000 bootstrap replications to generate the Cls. Compared with some other statistics like the mean or the median, the IQM is robust to outliers and also a good representative of the overall performance. The stratified bootstrapping is handy tool to obtain CIs with descent coverage rate, even if one only have a small number of training instances per setup. We refer the readers Agarwal al. (2021) for the complete introduction.
Figure 4.3a plots the confidence intervals when we consider all four quality setups, or when the labelled data quality is low or high, respectively. We found that SS-DT and SS-CQL respond slightly differently. Overall, SS-CQL is almost immune to changes in the size of the labelled data, as is SS-DT when the labelled data quality high. However, SS SS-DT's performance moderately increases as the labelled size grows when the labelled data quality is low. More detailed results, including the plots of the evaluation curves, CIs of the mean and the median, can be found in Appendix D.
Size of Unlabelled Data As before, we vary the number of labelled data size with the unlabelled data size fixed, and report the 95% stratified bootstrap CIs in Figure 4.3b. Similarly, Ss-COL is almost insensitive whereas SS-DT is sensitive when the labelled quality is low.
"The interquartile mean of a list of sorted numbers is the mean of the middle 50% numbers.
8

---

Under review as a conference paper at ICLR 2023
Value-based vs. Conditional BC As discussed above, SS-COL is insensitive to the data size changes, whereas SS-DT is more responsive when the labelled data quality is low. Regarding the data quality, we are mostly interested in the scenarios where the labelled data quality is low or moderate, see the red and blue curves of Figure 4.2b. In that regime, if the unlabelled data quality is high (the left column), the distribution shift from the labelled data the unlabelled data is challenging to handle, and the proxy-actions predicted by the IDM will be less accurate. There, the absolute performance of SS-CQL is slightly better than SS-DT, with smaller performance gaps compared to the oracle agents. If the unlabelled data quality is moderate or low (the middle and right columns), SS-CQL clearly outperforms SS-DT. Both observations suggest that SS-COL is less sensitive the action quality.
4.3 DESIGN CHOICES
Transition Size k of the IDM We train SS S--SSS---A TD3BC, SS SS-CQL and
SS-DT agents on the hopper-medi um-expert dataset, under the coupled setup as in Section 4.1. We consider 6 different values of 10, 30, 50, 70, 90 and 100. For all the 18 setups (3 SS-ORL agents and different values), we train the agents using the multi-transition IDM with & 0, 1, respectively. As in the previous section, Figure 1 plots the 95% stratified bootstrapped CIs for the IQM return across all the setups and training instances, which are generated by 50000 bootstrap replications. The results favors the choice & = 1. See Appendix E for more experiment details, such as the average return for each setup and the CIs for the return mean.
0.96 Normalized Return
0.99 1.02
Figure 4.4: The 95% strat- ified bootstrap CIs of the IQM returns obtained by SS-ORL agents using dif- ferent values of k.
Data Augmentation Strategy As discussed in Section 3.2, we consider variants of SS- TDDBCC and SS-DT using uncertainty based data augmen- tation. Following Lakshminarayanan al. (2017), we train an ensemble of 3 independent IDMs on Jiahelled. We generate the proxy actions for the unla- belled trajectories using the combined model, and also estimate the predie- tive uncertainties. We then only add proxy-labelled data whose uncertainties are below p% to the final RL training dataset. Specifically, we test 4 values of p: 25, 50, 75 and 95, We compare the results with standard S3-TD3BC and SS-DT where all the proxy- proxy-labelled data are added into the final RL training dataset. Again, we consider both the hoppe r-medi um-expert rt and walker-medd aum-expert datasets and use the coupled setup under with 4 differrnnt q values: 10, 30, 68, 75 for hopper-medi un-expert and 10,30, 54, 60 for wa walkers malker-medd r- um-expert. Figure 4.5 plots the 95% stratified bootstrap Cls of the IQM return across all the setups. Adding all the proxy-labelled data without filtering outperforms uncertainty based data augmentation; see Appendix F for more details. Intuitively, to make use of the unlabelled data, most of the SSL pipelines would assume Piabelled and Punlabelled are similar or even the same (Chapelle et al., 2006). This is not the case in our setup, where P Plabelied only generates low return trajectories, and all the high return ones come from P Punlabelled- It remains an open question if self- training with the uncertainty based selection rule can help us generalize to high return trajectories.
no fittering
below 75%
below 50%
below 25%
3.75 0.80 a.ss
Normalzed Retur
Figure 4.5: The 95% stratified bootstrap CIs of the IQM returns obtained by SS-ORL agents with different data augmenta- tion strategies.
DISCUSSION
We proposed a novel setup for offline RL where the trajectories do not have all of the action information, for which have introduced a semi-supervised mlgrrttgo pipeline. Our experiments identified key properties that enable the agents to learn from unlabelled data and show that near-optimal learning can be done with only 10% of the actions labelled for low-to-moderate quality trajectories. It would be interesting to study other heterogeneous data setups for offline RI. in the future, including reward-free or pure state-only settings.
This work is step towards a broader goal of empowering robotic systems with the ability to extract meaningful knowledge from copious and ever-growing amounts of unlabelled demonstration data. Beyond simply not having the action labels, many trajectories may be from different robotic systems or tasks and therefore are not directly transferable the system and task hand. As we continue building robotic systems to leverage these forms of auxiliary knowledge, we expect that weakly-supervised learning paradigms such as the one explored in this work will be useful.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances neural information processing systems, 34:29304-293331 2021.
Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Bowen Baker, lige Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune. Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022. URL https:tpp//.cmm / /ttpavaxivvvv..////v//aaa//aaava org/ abs //1///22.................
Richard Bellman. markovian decision process. Indiana Univ. Math. J., 1957.
Darrin C Bentivegna, Ales Ude, Christopher G Atkeson, and Gordon Cheng. Humanoid robot learning and game playing using pe-based vision. In JEEE/RSJ international conference on intelligent robots and systems, volume 3, pp. 2449-2454 IEEE, 2002.
Kaylee Burns, Tianhe Yu, Chelsea Finn, and Karol Hausman. Offline reinforcement learning at multiple frequencies. arXiv preprint arXiv:2207. 13082, 2022.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning. 2006. Cam- bridge, Massachusettes: The MIT' Press View Article, 2, 2006.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https: htmpp://oppee http///ppeeeeeeeeeeeeeeeeeeeeeee net. forum? a=
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.10751: 2021.
S Fralick. Learning recognize patterns without teacher. IEEE Transactions on Information Theory, 13(1):57-64, 1967.
Justin Fu. Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Gu. A minimalist approach to offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https:// openreview. http:/iwwwww.eeeeeeeeeeeeeeeeeee wwt.:eet/ooora
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International Conference Machine Learning, pp. 2052-2062 PMLR, 2019.
Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: Expected-man q-learning operator for simple yet effective offline and online rl. In International Conference on Machine Learning, pp. 3682-3691. PMLR, 2021.
Dibya Ghosh, Anurag Ajay, Pulkit Agrawal, and Sergey Levine. Offline rl policies should be trained to adaptive. In International Conference on Machine Learning, pp. 7513-7530. PMLR, 2022.
Abhishek Gupta, Coline Devin, YuXuan Liu, Pieter Abbeel, and Sergey Levine. Learning invariant feature spaces transfer skills with reinforcement learning. arXiv preprint arXiv:1703.0204 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
Auke Jan Ijspeert, Jun Nakanishi, and Stefan Schaal. Movement imitation with nonlinear dynamical systems in humanoid robots. In Proceedings 2002 IEEE International Conference on Robotics and Automation (Cat. No. 02CH37292). volume 2, pp. 1398-1403, IEFE, 2002.
10

---

Under review as a conference paper at ICLR 2023
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. In Thirty- Conference on Neural Information Processing Systems, 2021.
URL https::t///.mm / wwwpp/eepeeeeeeeeeeeeeeeeeeeeeep net httprr/wwwweeeetttooteeeeeeeeee. forum?iddwwer 630gSw.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.004555 2019.
Rahul Kidambi, Jonathan Daniel Chang, and Wen Sun. MobILE: Model-based imitation learning from observation alone. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.),
Advances Neural Information Processing Systems, 2021. URL
net/
htte//ooummmdd_eeeee -ed_dd___ma__------e------- 10temArYnII IIIIII
Kuno Kim, Yihong Gu, Jiaming Song, Shengjia Zhao, and Stefano Ermon. Domain adaptive imitation learning. International Conference on Machine Learning, pp. 5286-5295. PMLR, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.6988 69800, 2014.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Hya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning, 2021b.
Aviral Kumar, Justin Fu, George Tucker, and Sergey Levine. Stabilizing off- off-policy q-learning via bootstrapping error reduction. arXiv preprint arXiv: 1906.09990 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. arXiv preprint arXiv:2006.0477 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Kuang-Hus Huei Lee, Ofir Nachum, Mengjiao Yang. Lisa Lee, Daniel Freeman, Winnie Xu, Sergio Guadarrama, lan Fischer, Eric Jang, Henryk Michalewski, et al. Multi-game decision transformers. arXiv preprint arXiv:2205. 15241, 2022.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pp. 1118-1125. IEEE, 2018
Bogdan Mazoure, Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Improving zero-shot generalization in offline reinforcement learning using generalized similarity functions. arXiv preprint arXiv:2111. 11. 14629, 2021.
Ofir Nachum, Bo Dai, Ilya Kostrikov, Yinlam Chow, Lihong Li, and Dale Schuurmans. Algaedice: Policy gradient from arbitrary experience. arXiv preprint arXiv:1912.02077. 2019.
Yassine Ouali, Céline Hudelot, and Myriam Tami, An overview of deep semi-supervised learning. arXiv preprint arXiv:2006.0522 2020.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2778 2787. PMLR, 2017.
Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control, pp. 1154-1168. PMLR, 2021.
11

---

Under review as a conference paper at ICLR 2023
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez. Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint (rxXix:2005006155 2022.
Karl Schmeckpeper, Oleh Rybkin, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Reinforce- ment learning with videos: Combining offline observations with interaction. arXiv preprint arXiv:2011.06507, 2020a.
Karl Schmeckpeper, Annie Xie, Oleh Rybkin, Stephen Tian, Kostas Daniilidis, Sergey Levine, and Chelsea Finn. Learning predictive models from observation and interaction. In European Conference on Computer Vision, pp. 708-725. Springer, 2020b.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. contaccote networks: Self-supervised learning from video. In 2018 IEEE international conference on robolics and automation (ICRA), pp. 1134-1141. IEEE, 2018.
Pratyusha Sharma, Deepak Pathak, and Abhinav Gupta. Third-person visual imitation learning via decoupled hierarchical controller. In NeurIPS, 2019.
Bradly C. Stadie, Pieter Abbeel, and Ilya Sutskever. Third-person imitation learning. CoRR, abs/1703.01703, 2017. URL
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. CoRR, abs/1805.01955 2018a. URL httphhtttpp//rxxi..rrg////// ab000/// 91954.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. CoRR, hew/1807.01555 2018b. URL htttt::/arxxv..oo/ abs 011880006611
Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observa- tion. arXiv preprint arXiv: 1905: 13566, 2019.
Jesper E Van Engelen and Holger Hoos. A survey on semi-supervised learning. Machine Learning, 109(2):373 440, 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.1136 2019.
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint 1rXiv:1904.00663 2019.
Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. arXiv preprint arXiv:2202.01741, 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. arXiv preprint wwXiv:2202.05607 2022.
Xiaojin Zhu. Semi-supervised learning literature survey, Technical Report 1530, Computer Sciences, University of Wisconsin-Maddsonn 2005.
12

---

Under review as a conference paper at ICLR 2023
A EXPERIMENT DETAILS
In this section, we provide more details about our experiments. For all the offline RL methods we consider, we use our own implementations adopted from the following codebases:
DT https://gittiei
TD3BC https/////gatuu..ooooooaoaaaeaee CQL https://githht: com http.//.ccc//ccccteeooooososseee
We use the stochastic DT proposed by Zheng et al. (2022). For offline RI., its performance is similar to the deterministic DT (Chen et al., 2021). The policy parameter is optimized by the LAMB optimizer (You et al., 2019) with & = 10 The log-temperature parameter is optimized by the Adam optimzier (Kingma & Ba, 2014). The architecture and other hyperparameters are listed in Tabel A.1. For TD3BC, we optimize both the critic and actor parameters by the Adam optimizer. The complete hyperparameters to optimize the critic, actor and the log-temperature parameters. The architecture of critic and actor networks and the other hyperparameters listed in Table A.3.
listed in Table A.2. For CQL, also the Adam optimizer
We use batch size 256 and context length 20 for DT, where each batch contains 5120 states. Corre- spondingly, we use batch size 5120 for CQL and TD3BC.
Hyperparameter
Value
number of layers
4
number of attention heads
embedding dimension
512
context length dropout
20
0.1
activation function batch size
relu
256
learning rate for policy
0.0001
weight decay for policy
0.001
learning rate for logeterper
0.0001
gradient norm clip learning rate warmup target entropy evaluation return-to-gu
0.25
linear warmup for 10 steps
ddm((A)
3600 Hopper
5000 Walker
6000 HalfCheetah
Table A.1: : The hyperparameters used for DT.
Hyperparameter
Value
discount factor
0.99
target update rate
0.005
policy noise
0.2
policy noise clipping
(-0.5, 0.5)
policy update frequency
2
critic learning rate
0.0003
critic hidden dim
256
critic hidden layers
2
actor learning rate
0.0003
actor hidden dim
256
actor hidden layers
2
activation function regularization parameter a
ReLU
2.5
Table A.2: The hyperparameters used for TD3BC.
13

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Value
discount factor
0.99
target update rate
0.005
critic learning rate
0.0003
critic hidden dim
256
critic hidden layers
3
actor learning rate
0.0001
actor hidden dim
256
actor hidden layers
log-temperature learning rate
0.0003
activation function
ReL.U
number of sampled actions
10
target entropy
dim (A)
minimum Q weight value
Lagrange
False
Importance Sampling
True
Table A.3: The hyperparameters used for CQL.
B THE RETURN DISTRIBUTIONS OF THE D4RL DATASETS
hopper-medium-replay--22
hopper-medium-v2
hopper-medium-expertV
density
density
2.5
0.0
0.0
0.5
1.0
8.00
0.75
1.00
normalized return
normalized return
normalized return
walker2d walkerrzdmmdduumm-repaayyw
walker2d-medium--2
wa wallerrZ--mddim--exprrr...
density
density
density
normalized return
normalized return
normalized return
halfcheetah medium repala---
halfcheetah-medium-22
haffcheetah-meiiuu--xeeervvw
density
density
20
density
10
10
0.9
0.0
0.0
normalized return
normalized return
normalized return
Figure B.1: The distributions of the normalized returns of the D4RL datasets.
C ADDITIONAL EXPERIMENTS UNDER THE COUPLED SETUP
conduct experiments on the medi um and mediumm-rent the same setup as in Section 4.1. Figure C.I and C.2 reports the results. The general trend is as the same as that in Figure 4.1. We note that the results on the hal falfcheetaie ah-medi um dataset are less informative than the others. This is because the data distributions of hal halfcheetah--i aeethhheedd um is very concentrated, similar to a Gaussian distribution with small variance, see Figure B. In such a case, varying the value of & does not drastically change the labelled data distribution.
datasets D4RL benchmark, using
14

---

Under review as a conference paper at ICLR 2023
COL
terallized rdtur
0.5
nomalliad
0.4
24
S5
oracle
crack
100
100
100
(labelled data quality param)
(labelled data quality param)
(labelled data quality param)
walee/dmmedum- COL
i walke/2-mmedum DT
#alker2d-medimm TD3BC
0.9
0.0
0.8
nom rOUUT
0.7
0.7
0.6
0.6
0.5
0.5
0.5
0.4
0.4
0.4
+ 55 COL
sS DT
*- 55 T5-DCBC
baselire
baveline
T039C baselnw
T939C
20
100
20
100
20
200
(labelled data quality
(labelled ditt quality param))
(labelled data quality param)
haltcheetah-mediuee
haltthedta-mmedim
halfcheetah
0.9
0.9
2.9
- 55-COL
55-01 DT
- 55-TDBBC
0.5
COL basdlne
0.8
DT basaline
0.8
TDS9C basoline
permalaao reture
0.7
CTC
orecle
eemaloged reture
0.7
BT onarle
3.7
TD30C onacle
0.4
06
2.6
rermalaed
0.5
0.5
0.5
0.4
0.4
0.4
0.3
0.3
0.3
0.2
0.2
02
0.1
20
40
60
80
100
0.1
20
40
30
100
0.1
20
40
60
80
100
9
(labeleed data quality param)
9
(labelled data quality param)
0
@Jabelled data quality param)
Figure C.1: The return (average and standard deviation) of SS - ORL agents trained on the D4RL medium um datasets.
hopper-medium replay COL
hopper-raddum-eepkyy
hopper-medium-rrplay
S5-COL
pemma 1ommllzzd retum
normalzeed wizedd returm
3.8
0.5
0.6
0.4
0.4
2.4
0.2
0.2
0.2
qualit param)
quality param)
quality param)
walke/2d-medium-oolaay- COL
walker2d mediumrred TD3BC
1.0
1.0
35-D7
0.5
0.8
OT basaline
2.8
normalzedretuun
onacle
rormalze-
0.6
-----
SS-QQL
0.6
0.6
55- 1BC
COL baseline
TD3BC baseline
CQL oracle
0.4
TD38C crack
20
100
20
100
20
200
4 (labelled data quality param)
(labelled data quality param)
0 (labllled data quality param)
replayy
halichetal--meduree mellume-tttmeeeuuu-eeeelaa
0.5
0.5
0.5
Dormal food retur
normallzed return
TD39C 1D30C
82
TD36C oracle
55 baseline
0.1
0.1
baselie
0.1
COL oradle
OT orade
0.0
20
40
80
80
100
0.
20
5G
20
100
3.0
20
40
60
80
100
9.
(labelled data quality param)
0
(labelled data quality param)
0
(labelled data quality param)
Figure C.2: The return (average and standard deviation) SS SS-ORL agents on the D4RL medi um- cum-replay datasets.
15

---

Under review as a conference paper at ICLR 2023
dataset
q=10
q=30
q=50
q=70
q=90
q=100
Supper-medium-reee
0.007
0.022
0.05
0.074
0.109
0.149
walker2d--edium--rppla
-0.002
0.005
0.023
0.048
0.087
0.156
malicheetah-meeduu--rpaaaa
0.001
0.092
0.179
0.202
0.269
0.275
hopper-medium
0.231
0.310
0.355
0.388
0.418
0,443
walker2d-medive
0.135
0.287
0,44
0.557
0.599
0,618
halfcheetah-meedee
0.361
0.383
0.397
0.396
0.406
0,405
medium-
0.252
0.341
0.394
0.451
0.594
0,645
mallker2d-mddiu--expee
0.201
0.469
0.605
0.732
0.791
0.827
halfcheetah- malfchhetaa-mmedium-xxpeeeee
0.377
0.397
0.405
0.537
0.604
0.638
Table C.I: The average return of the labelled trajectories increases in our experiments. Results
aggreagted over seeds.
Walker Subsampled
7.5
densiby
5.0
2.5
0.0
0.0
0.5
1.0
normalized return
Figure C.3: The density randomly subsampled dataset of the wa 1 environment.
mwlite-2d-Sulsamplle
walkter
soler24-ubbämmsled TD3BC
0.2
norma loed recurn
norma Foed reture
0.4
2.8
0.6
0.
62
nurma
0.4
0.1
basaline
oracle
0.2
oracle
0.2
20
40
60
80
100
20
50
30
100
20
40
60
80
100
9
(labelled data quality param]
9
(labellied data quality parien)
9
(labelled data quality param)
Figure C.4: The return (average and standard deviation) of SS-ORL agents on the subsampled dataset.
One may notice that for the hopper repoerr--eiiiee um uemreplay and walker-medi um wumuueee replay datasets, SS-ORL does not catch up with the oracle as quickly as on the other datasets q increases. Our intuition is that the return distributions of these two datasets concentrate on extremely low values, as shown Figure B.I. In our experiments, the labelled trajectories for those two datasets have average return small than 0.1 even when 70. In contrast, the return distributions of the other datasets concentrate on larger values. For the hal f aheetah-meddi um wn-replall replay and all the medium and medi wwwlummeeepeee datasets, increasing the value of g will greatly change the returns of labelled trajectories, see Table C.I.
To demonstrate the performance of SS- on dataset with more wide return distribution, consider a subsampled dataset for the walkes ker environment generated as follows.
1. Combine the wa walker-meeiimm-repl and wa malker-medd um datasets.
2. Let Rmin and Rmax denote the minimum and maximum return in the dataset. We divide the trajectories into 40 bins, where the maximum returns within each bin are linear spaced between Ruin and Rmaxx- Let be the number trajectories in bin i.
3. We randomly sample 1000 trajectories. To sample trajectory, we first first sample bin e [l...., 40] with weights proportional to 1/n,, then sample a trajectory uniformly at random from the sampled bin.
16

---

Under review as a conference paper at ICLR 2023
Figure C.3 plots the return distribution of the subsampled dataset. It is wide and has 3 modes. We run the same experiments as before on this subsampled dataset, and Figure C.4 plots the results. We can see that SS-ORL methods can catch up with the oracle agents even when q is small.
D INFLUENCES OF THE LABELLED AND UNLABELLED DATA SIZE
Figure D.1 plots the average return of SS-DT and SS-CQL when we fix the number of unlabelled trajectories and vary the number of labelled trajectories. We found that there is a bad seed for SS-CQL when both labelled and unlabelled trajectories are sampled from L and the size of labelled data is 10%, so that the result there the bottom right panel) exhibits large variance. Correspondingly, Figure D.2 plots the 95% stratified bootstrap CIs for the median, mean, interquartile mean, and the optimality gap of the return of 55-DT and S$ SS-CQL.
Similarly, Figure D.3 and D.4 plots the results when we vary the number of unlabelled trajectories, while the number of unlabelled ones is fixed.
SS-DT LL
S5-DT LLXT
55-DT LTXL
55-DT T.X.T
normalbed retu-m
normallzedd retumm
1.0
normalzedd alazdd retum
0.00
normaiized
300
lebelled sizem 109
sizc-25%s
Inbelled 50%%
Lavis 2038
28
2x12
34277
2410
3232
12
1001
Iteration
iteration
Iteration
SS-CQL LLLL
SS-COL L
SS.COLI TTXL
SS.COL T
nomalizze seturn
normalizes return
normalized neturn
nomalized
A0
6000
Inbelled 10z==10%%
lnbelled -5%-
labelled 5.%
Liia 2:30
5K25e
15:20
512
25.00
5520
18
15:20 1228
iteration
iteration
Nteration
iteration
(a) Results the hoppe r-med um comm-experrt rt dataset.
S5-DT LxL
S5-DT Lxx
SS-DT [TxL
55-DT T
1.
permaized reture
pormalzed retur
1.70
8m
pormalized
0.20
nommalized
0.80
0.00
Jabelled 8lze-2%%
sixe-10%
0.00
Inbelled 50%)
0.30
5=107
15x30!
3x3
3K297
32
35x3el
3x277
32
35x10
3010
5x30
32 35xtcl
Nteratio
iterationn
iteration
iteration
S5-COL |L
SS-COL T
SS-CQL x
S5-COL TT
normalized return
normalized retum
wwrmalzzed.n.. retum
1.77
normalizes caraalzedd
labelled size-10%
labelled 51z0-25%
Ibbelled 50ze
M4
2238
Iteration
iteration
Iteration
Iteration
(b) Results on the wa 1 ker-kedi wwt.meeeddueeeeeee.em dataset.
Figure D.I: The return (average and standard deviation) of SS-DI and SS-COL agents trained on the medium um- uatuummepppm expprt datasets with different sizes of labelled data. The unlabelled data size is fixed to be 10% of the offline dataset size. Results aggregated over instances with different seeds.
17

---

Under review as a conference paper at ICLR 2023
SS-DT All
Median
IQM
Mean
Optimality
50% labeled
25% labeled
10% labeled
0.60
0.75
0.90
1.05
0.60
0.75
0.90 1.05
0.60
0.75 0.90
1.05
0.00
0.15
0.30
Normalized Return
SS-DT Labelled Quality -L IQM
Median
Mean
Optimality Gap
labeled
labeled
10% labeled
0.60 0.75
0.90
1.05
0.60 0.75
0.90
1.05
0.60
0.75 0.90 1.05
0.00
0.15
0.30
Normalized Return
S5-DT Labelled Quality T IQM
Median
Mean
Optimality Gap
50% labeled
25% labeled
10% labeled
0.60 0.75
0.90
1.05
0.60
0.75
0.90
0.60 0.75 0.90 1.05
0.00
0.15
0.30
Normalized Return
(a)
SS-CQL All Cases
Median
IQM
Mean
Optimality Gap
50% labeled
25% labeled
10% labeled
0.60 0.75
0.90
1.05
0.60
0.75
0.90
1.05
0.60
0.75 0.90
1.05
0.00
0.15
0.30
Normalized Return
SS-CQL Labelled Quality IQM
L
Median
Mean
Optimality Gap
50% labeled
25% labeled
10% labeled
1.05
0.60 Normalized Return
SS-CQL Labelled Quality TT
Median
Mean
Optimality Gap
50% labeled
25% labeled
labeled
0.60
0.75
0.90
1.05
0.60
0.75
0.90
1.05
0.60 0.75
0.90
1.05
0.00
0.15
0.30
Normalized Return
(b)
Figure D.2: The 95% stratified bootstrap CIs of four statistics (the median, mean, interquartile mean, and the optimality gap) over the returns of SS-DT and sS- CS-CQL across all the setups, when trained with different sizes of labelled trajectories. The number of unlabelled trajectories 10% of the total number of offline trajectories of corresponding dataset. We use 50000 bootstrap replications to generate the Cls.
18

---

Under review as a conference paper at ICLR 2023
SS-D7
0.70
norme lodd reture
normak 20d return
normall 20d return
1.0
2.0
0.00
0.0
0.55
unabeled site 10%
0.45
unabeled 22%-25%
0.4
unaabeled 50%
5K10
10
15x26P
2x16
SXI0F
15H10P 2x10
5x1M
1.511
2x10-
15510 2x10
iteration
Iteration
SS-CQL Lx
SS-CQL L
SS-COL T
55 3S-CQL
1.1
normaiized return
normalized return
normalized return
2.0
0.05
0.
8.0
normalized
0.9
0.00
9.7
urinbelled streel 0%
0.55
libeelled sizund 0%
0.6
5x10 10 15x10" 2x10
SxI0f
10t
15x10"
2x10
5x10
10
155x0
2x30'
5x
10a
1,5x10
2x10"
iteration
lerraion
iteration
Reretion
(a) Results on the hopper-med um comperepprtt dataset.
LxL
0.8
1.0
normalizd return
0.7
nermalizedd return
normalized return
0.7
0.8
0.8
normalizes lieedd
0.6
0..6
0.4
untabellod stc-10%
2.4
unlobelled stee-25
0.2
0.3
Iteration
Iteration
SS-CQL L
S5-COL XXXT
SS-COL Txx
SS-CQL TXT
12
normalized retures
0.80
nerma nermalzzed eeturn
hormalizedd neturn
2.0
2.2
0.75
8.8
3.0
0.7
0.66
crlbbclldd sic--10
3-6
0.6
0.8
abcllcd 3tc-- 5%
lbblledd stzedd 5%
0.4
5x10
19
L5x10
2x12
SHIOFF
105
1.5+10
2x20>
5X324
10t
1.5510F 2x30
54101 10t 1.510 2x10
iteration
Iteration
iteration
iteration
(b) Results on the wa www.-keeeddumeeepeeeeee...eeeeee dataset.
Figure D.3: The return (average and standard deviation) of SS- DT and SS-CQL agents trained on the medium um expert datasets with different sizes of unlabelled data. The labelled data size is fixed to be 10% of the offline dataset size. Results aggregated over instances with different seeds.
19

---

Under review as a conference paper at ICLR 2023
SS-DT All Cases
Median
IQM
Mean
Optimality
50% unlabeled
unlabeled
10% unlabeled
0.75
0.90
1.05
0.60 6.75
0.90 1.65
0.60
0.75 0.90
1.05
0.00
0.15
0.30
Normalized Return
Labelled Quality IQM
Mean
Optimality Gap
50% unlabeled
25% unlabeled
10% unlabeled
0.60 0.75
0.90
1.05
0.60 0.75
0.90 1.05
0.60
0.75 0.90 1.05
0.00
0.15
0.30
Normalized Return
SS-DT Labelled Quality T IQM
Median
Mean
Optimality Gap
50% unlabeled
25% unlabeled
10% unlabeled
0.60
0.75
0.90
1.05
0.60
0.75
0.90 1.05
0.60
0.75
0.90 1.05
0.00
0.15
0.30
Normalized Return
(a) Results of SS-DT.
SS-CQL All Cases
Median
IQM
Mean
Optimality
50% unlabeled
unlabeled 10% unlabeled
0.60
0.75
0.90
1.05
0.60
0.75
0.90
1.65
0.60
0.75
0.90
1.05
0.00
0.15
0.30
Normalized Return
Ss-COL Labelled Quality 1QM
Optimality Gap
50% unlabeled
25% unlabeled
10% unlabeled
0.60
0.75
6.90
1.05
0.60
0.75
0.90 1.05
0.60
0.75 0.90
1.05
0.00
0.15
0.30
Normalized Return
S5-COL Labelled Quality T IQM
Median
Mean
Optimality Gap
50% unlabeled
25% unlabeled
10% unlabeled
0.60
0.75
0.90
1.05
0.60 0.75 0.90 1.05
0.60 0.75
0.90 1.05
0.00
0.15
0.30
Normalized Return
(b) Results of SS-COL.
Figure D.4: The 95% stratified bootstrap CIs of four statistics (the median, mean, interquartile mean, and the optimality gap) of the returns of SS-D7 and 5$ SS-CQL across all the setups, when trained with different sizes of unlabelled trajectories. The number of labelled trajectories 10% of total number of offline trajectories of corresponding dataset. We use 50000 bootstrap replications to generate the Cls.
20

---

Under review as a conference paper at ICLR 2023
E TRANSITION SIZE k FOR THE MULTI-TRANSITION INVERSE DYNAMIC
MODEL
E THEORY
Let B denote the behaviour policy. When h = 0, the IDM is modeling
P(ar|$141;55) 8c) P(a St+1 [[ [dc, Sta3(a))s)) 1(())118))
(2)
P( (s))
For the cases where l > O, w.l.o.g, we assume k = 1. The IDM is modeling
P(ar|S142, 3 Sc, St-1) = S12.....
P(s St, 86+2, St- P(ar P St, P(11)/( Sts 81-1) P(s St)S(a St s1-1)/ P(s (st, st-1),
where in the last line we used the fact that the policy 8 can only generate actions based on previous states, the Markovian transition property P(( lat, 8t; 8842, 8t-1) P(st-11 8t), and also the induced property P(s++1| 5t-1) P(s++1 [5142, Se, 8t-1).
If the behaviour policy 3 is Markovian, we have that 3(a, (sx) = (ar 8t- 1), and as a consequence
P(a:|se41, 8t) P(ar|S142 8 S8, 8t-1) 1) C(4441 St, St-1);
where C = 2( is independent of the action ae. Therefore, the probabilities that the IDMs with k = 0 and k = 1 are modeling, are equivalent up state-only dependent scaling. The cases where & > 2 can be derived analogously.
In practice, the offline dataset might contain trajectories generated by multiple behaviour policies and it is unknown any of them is Markovian. Therefore, choosing l > 0 allows us to take into account past information before timestep t. For future, we do not need anything beyond 81+1 for MDP, but our formulation is general purpose to account for POMDP as well, where both past and future partial observations might be needed to infer the action dg. To summarize, choosing k > 0 is more general and have been shown to be favorable in the empirical experiments presented in next section.
E.2 EMPIRICAL EXPERIMENTS
We train SS-TD3BC, SS-CQL and SS-DT with 3 IDM transition size: k = 0, and 2 on the hepper-medi um expert dataset. We the coupled setup described in Section 4.1, with 6 different values of q. Table E. reports the performance of those agents for each case.
In addition the interquartile mean considered in Section 4.3, also consider 3 other statistics of the return across all the setups: the mean, the median and the optimality gap. Figure 4.3 plots the 95% stratified bootstrap confidence intervals for all the four statistics, genereated by 50000 bootstrap replications.
q 10
30
q- 50
70
90
100
Average
k = 0
0.81 + 0.12
0.89 + 0.05
0.93 + 0.05
1.05 1 0.04
1.03 - 0.06
1.01 : 0.04
0.95
SS-T SS-T33BC
==
0.93
= 0.07
1.01
+ 0.05
0.86 : 0.06
0.98
+ 0.06
1.03 2 0.06
1.03
- 0.04
0.98
0.80 + 0.12
0.91
+ 0.03
0.93 : 0.05
0.95 0.08
1.01
0.06
1.04
0.02
0.94
0.69 + 0.17
0.69 4: 0.15
0.88 4 0.15
1.04 0.04
1.11
1.10 4 0.03
0.92
S8-CQL
0.69 + 0.15
0.90 4: 0.05
0.89 0.13
1.03 0.07
1.07
1.11 0.01
0.95
2
0.90 = 0.11
0.90 + 0.09
0.86 - 0.11
1.08 0.05
0.00
1.11 0.01
0.99
= 0
0.72 = 0.17
0.75 + 0.20
0.90 0.14
1.06 0.04
1.11
1.11 0.01
0.94
SS-DT
k = 1
0.69
+: 0.20
0.94 + 0.07
0.99 + 0.05
1.05 + 0.04
1.11
1:00
1.11 : 0.00
0.98
2
0.78
+ 0.07
0.89 + 0.08
0.85 4 0.15
1.05 0.02
1.11
+ 0.00
1.11 1 0.00
0.97
Table E.1: The return (average and standard deviation) of SS -ORL agents trained on the hopper- medi wwuuum--eppce expert dataset under the coupled setup, where the IDM is trained with 2 different values of h: 0, and 2. Results aggregated over training instances.
21

---

Under review as a conference paper at ICLR 2023
Median
IQM
Mean
Optimality Gap
0.92
0.96
1.00
1.04
0.92
0.96
1,00 1.04
0.92 0.96
1.00 1.04
0.06 0.09 0.12 0.15
Normalized Return
Figure E.I: The 95% stratified bootstrap CIs of four statistics (the median, mean, interquartile mean, and the optimality gap) of the returns obtained by SS-ORL agents, with different values of k.
F DATA AUGMENTATION STRATEGY
Following Lakshminarayanan et al. (2017), we train an ensemble of independent IDMs on Tiabelled- Each individual IDM models the action N(,, E,), 3. The ensemble models the action using equally weighted Gaussian mixture of these three distributions. We predict the action by the mixture's mean and predict the uncertainty by the mixture's 's variance; both can be written in close form.
diagonal Gaussian distribution (see Equation (I))
We conduct experiments for SS-DT and SS- SS-TD3BC, where we only add proxy-labelled data whose uncertainties are below p% to the final RL training dataset. Specifically, we test 4 val- ues of p: 25, 50, and 95.:w We compare the results with standard SS-DT and SS-TD3BC where all the proxy-labelled data are added into the final RL training dataset. We consider both the r-pedi eeeeediu--exprree and wa ker-medi comedium-exxeee um-expert datasets. use the coupled setup described in Section 4.1, where consider 4 different values of q: 10, 30, 68, 75 for hopper- medi ddiummesee expeeetert and 10, 30, 54, 60 for wwi.-k--medium--epppeee
Table F.1 reports the average return and standard deviation obtained by SS-DI and SS- TD3BC under different data augmentation strategies, when trained on the hopper-meeseeeee.. r- um-expert dataset. The results the walker cotee-mett r-medi um-expert dataset reported in Table F.2. It is easy to that uncertainty based data augmentation degrades the performance, compared with adding all the proxy- labelled data without filtering. Overall, the latter performs consistently well across different setups. Figure F. plots the 95% stratified bootstrap CIs for this experiments. All the statistics favor the no filtering strategy.
hopper-medi wwweeeddum-xxeeeeee
==
q= 30
q 68
q= 75
Average
below 25%
0.60 0.03
0.62
0.02
0.71
0.04
0.86 0.02
0.70
below 50%
0.62 0.02
0.66 + 0.06
0.76 0.04
0.86
0.72
SB-TD3BC
below 75%
0.70 = 0.06
0.74
0.04
0.94
0.80
below 95%
0.82 0.05
0.82 0.09
0.90 1.09 0.09
0.96
0.88
filtering
0.80 0.07
0.92 + 0.04
0.91 0.06
0.94
0.89
below 25%
0.61 0.12
0.62 0.05
0.70
0.95 0.13
0.72
below 50%
0.60 0.14
0.65 0.04
0.69 0.02
+
0.75
SS-DT
below 75%
0.42 0.04
0.63 0.15
0.75
1.04
0.71
below 95%
0.51 0.16
0.82 0.12
0.85 0.05
0.05
1.06
0.03
0.03
0.81
filtering
0.47 0.14
0.71 1 0.14
0.83 0.07
1.06 + 0.03
0.77
Table F.1: The return (average and standard deviation) of SS-ORL agents trained on the hopper- medi wwtuum-eeepee expert dataset under the coupled setup, using different data augmentation strategics. Results aggregated over 5 training instances.
22

---

Under review as a conference paper at ICLR 2023
walKer-nedium-exprrr
10
= 30
q= 54
60
Average
below 25%
0.82 0.02
0.82 + 0.01
0.80 + 0.06
1.04 + 0.06
0.87
below 50%
0.83 0.03
0.84 0.02
0.01
1.02 0.09
0.88
SS-TD3BC
below 75%
0.74 0.11
0.00
1
0.87
below 95%
0.86 0.04
0.88
1.10 0.07
0.93
no filtering
0.86 0.05
0.01
1.10
0.92
below 25%
0.69 0.04
0.74
0.70 0.03
0.84
0.74
below 50%
0.67 = 0.03
0.72 0.02
0.03
0.15
0.77
SS GT-DT
below 75%
0.71 0.03
0.60 0.13
0.73
0.14
0.74
below 95%
0.73 0.08
0.52 0.11
0.58
0.10
0.70
no filtering
0.79 0.05
0.55 + 0.13
0.08
1.15
0.74
Table F.2: The return (average and standard deviation) of SS SS-ORL agents trained on the w medi um-expert. dataset under the coupled setup, using different data augmentation strategies. Results aggregated over 5 training instances.
Median
Mean
Optimality
filtering
bellow 95%
below 50%
below 25%
0.75 0.80
0.85
0.75
0,80
0.85
0.76
0.80
0.84
0.88
0.16 0.20 0.24
Normalized Return
Figure F.I: The 95% stratified bootstrap CIs of four statistics (the median, mean, interquartile mean, and the optimality gap) of the returns obtained by SS-ORL agents, when combined with different data augmentation strategies.
G COMPARISON WITH GATO UNDER THE COUPLED SETUP
Inspired the multi-task and multi-modal generalist agent proposed by Reed et al. (2022), we consider a GATO type of variant of DT that can incorporate the unlabelled data into policy training.
GATO is trained on the labelled and unlabelled data simultancously. The implementation details are:
We form the same input sequence as DT, where we fill in zeros for the missing actions for unlabelled trajectories.
For the labelled trajectories, GATO predicts the actions, states and rewards; for the unlabelled ones, GATO only predicts the states and rewards.
We use the stochastic policy as in online decision transformer (Zheng et al., 2022) to predict
the actions.
We use deterministic predictors for the states and rewards, which are single linear layers built on top of the Transformer outputs.
Let 9t =iilee be the return-to-go of trajectory at timestep t. Let H. Pusette denotes the policy entropy included on the labelled data distribution. For simplicity, we assume the context length of GATO is 1. We refer the readers Zheng al. (2022) for the formulation with general context length and more details. Equation (5) shows the training objective of GATO.
min
E B(ar,))tt - Prittled {- log (a St. 0) + Aalls - (,) + A -(())
B(ssorigi) {d.1s, 6,(0) + Ar (.())
(5)
s.t. 11 Punate o|8,9] 2
23

---

Under review as a conference paper at ICLR 2023
hopper-medium--epprt
hopper-medium-expert
1.0
1.0
normaliza aallzezd return
norma return
0.8
0.8
0.6
0.6
SS-DT
*
SS-CQL
DT baseline
SS-DT
0.4
0.4
SS-TD3BC
oracle
GATO
20
40
60
80
100
20
40
60
80
100
q (labelled quality param)
9 (labelled quality param)
Figure G.I: (The performance of sS ORL and GATO on the hopper-medi eeppreeemmmiiieeeeeeeeeeee expert dataset. For GATO, we use A 0.01 and X 1.0. (L) SS-DT significantly outperforms GATO, where GATO only slightly improves upon the baseline. (R) SS-COL, SS-DT and SS-TD3BC all outperform GATO.
I0M
A,- 000001
0.001
4- 0.001. A.-
A A.-
0.01.A
4- 0.00 0.01,
Ay- 0.1.A-00.000. 0.000
Ag 0.1.
A.1.
A.1.A. m... 0.000
1.0.0.0000..
0.50
0.54
3.68
Figure G.2: The 95% stratified bootstrap CIs of four statistics (the median, mean, interquartile mean, and the optimality gap) of the returns obtained by GATO agents, with different combinations of regularization parameters.
The constant Å₃ and X prefixed papper-prammters, where v is the target policy entropy, and A and Xy are regularization parameters used to balance the losses for actions, states, and rewards. We dim (A) for DT (see Appendix A). To choose the regularization parameters A and X for GATO, we test 16 combinations where X₂ and xy are 1.0, 0.1, 0.01 and 0.001 respectively. We run experiments in Section 4.1 for q = 10, 30, 50, 70, 90, 100, and compute the confidence intervals for the aggregated results. Figure G.2 shows that X = 0.01 and A, 0.1 yield the best performance.
Figure G.1 compares the performance of CATO (with A 0.01 and A, 0.1) and SS-ORL agents. It is clear that SS-ORL agents outperform GATO.
H PERFORMANCE GAP OF SS-ORL AGENTS
For chosen offline RL method, the relative performance gap between the corresponding SS-ORL and oracle agents illustrates how sensitive this offline RL is to missing actions:
Oracle-ORL - STRRL-SS--0 SS-ORL
6
Oracle-O8
We consider the coupled setup as in Section 4.1. For each of the datasets (hopper,walkez)) fahhefahcheahhh with medium-experts medium, and medi medium-replay datasets), we compute the relative performance gap for SS-COL, SS-DT and SS-TD3BC, trained with different values of q: 10, 30, 50, 70, 90 and 100. Table H.1 reports the aggregate results over
24

---

Under review as a conference paper at ICLR 2023
seeds. On average, SS-COL and SS-TD 3BC have smaller relative performance gap, suggesting that CQL and TD3BC are less sensitive to the missing actions.
method
hopper-me
walker2d-me
bo-me
hopper-m
walker2d-m
be-m
hopper-mr
walker2d-me
Average
SS-COL
0.147
0.114
0.062
0.078
0.077
0,003
0.388
0.379
0.106
0.1.0
SS-ID3aC
0.994
3S-DT
0.119
0.167
0.0002
0.016
0.039
0.003
0.399
0.554
0.109
0.156
Table H.1: The relative performance gap of SS-CQL, 3D3CCB and SS ST-DT.
25


Reviews:
[{"review_id": "xD5FiPIwT9", "reviewer": "Reviewer_az7q", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a method for semi-supervised learning in the offline RL setting where the unlabelled part of the dataset consists of action-free state trajectories and the labelled part consists of the full trajectories. They use an inverse dynamics model to learn actions that give rise to state transitions and use the learned model to inject labels for unlabelled data and perform offline learning using classic model-free offline RL algorithms such as CQL.", "summary_of_the_review": "Overall, I lean towards rejecting this paper. My reasoning is that while it is tackling a notable practical problem in the RL setting, there is not enough technical contribution nor is it polished enough in its exposition of the design choices for me to recommend acceptance. ", "strength_and_weaknesses": "Weaknesses:\n- My main concern is with the technical contribution of this paper. Considering the inverse dynamics model as the crucial technical novelty, I am not sure the method given in Eq. 1 is the best, and at the very least, some further evaluations and arguments for its design choice is needed. For one, the covariate matrix with k>0 would seem to break the markov property of the RL setting. Furthermore, it is not justified why this choice was made beyond the empirical validation, which is not enough either in my opinion. I am not convinced that this extension of IDM significantly contributes to the overall goal of taking advantage of semi-supervision, as compared to the fully unsupervised case. In particular, it would have been interesting to see whether semi-supervision in this regime would help when the data split is within trajectories and not between trajectories. \n- Along the lines of the above, I think an analysis of why only a single labelling round was used instead of the conventional self-training paradigm of retraining per round could have been included to make the paper stronger. In particular, I assume it could be possible to provide a more detailed analysis of how well the inverse dynamics model learns when compared to the ground truth data on the log-likelihood of the multivariate Gaussians used to estimate them since you have access to those data labels. \n- As it stands, this paper's main contributions are a careful study of the different considerations one should take when doing semi-supervised offline RL. It has a good experimental validation of how performant value-based and BC methods. However, it does not make enough technical contribution to take advantage of the specific literature in the semi-supervised learning setting or does not justify its design choices well. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "utyHuT20BA", "reviewer": "Reviewer_omLk", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper provides an empirical study about a setting where a portion of the offline RL dataset doesn't include actions. To exploit action-missing data, this work proposes to learn an inverse dynamics model on data with actions to generate proxy actions from state transitions. A set of empirical studies on d4rl gym-locomotion control are provided to give insights into how such semi-supervised learning will be helpful for the performance of the final policy. The ablation studies show the proposed semi-supervised method is particularly helpful when the action-missing data is of high quality and labelled data is of lower quality.", "summary_of_the_review": "The paper studies an interesting setting for offline RL which has significant practical value. The study of different scenarios of data availability and the quality of the data is a timely topic for offline RL research.\nHowever, my main concern about this paper is it's not very informative. The whole paper looks more like a technical report about applying the inverse dynamics model to the d4rl dataset and I'm struggling to figure out what's the key takeaway. \nThe authors summarised three of their key findings in the introduction:\n1. One interesting claim of the paper is SS-ORL agent works well in the setting with lower-quality labelled and high-quality unlabelled data. But I'm not sure if such a claim can generalize to the setting when the policies are more diverse than a single or a mixture of two unimodal policies, as I argued in the second point of the weakness. \n2. \"When the labelled data quality is high, utilizing unlabelled data does not bring significant benefits.\" This might just reveal that most of the data from the d4rl gym-locomotion are redundant because of the low diversity of the policy?\n3. \"CQL and TD3BC are less sensitive to the action labels compared to DT.\" This is not very obvious to me by just looking at Figure 4.1 and Figure 4.2. It would be better to have quantitative results if that's true.\n\nGiven the reasons above, I cannot recommend acceptance at the current stage. But I may change my mind if the authors or other reviewers can convince me about point 1, or remind me why point 2 and 3 are significant enough to accept the paper.", "strength_and_weaknesses": "Strength:\n1. The setting is well-motivated and has decent potential for real-world applications. \n2. The empirical studies are thorough and in particular, ablation over the quality of data is well-done.\n\nWeaknesses:\n1. The proposed method is quite standard in online settings. The claimed novelty (multiple transitions as input to the inverse model) is more of technical detail and the reason why it is helpful for the Markovian setting is not explained properly.\n2. Most of the experiment results in the paper are based on the medium-expert dataset of d4rl gym-locomotion. This raises questions about whether the conclusion can be generalised to the setting with more diverse data. For medium-expert, the behaviour policy is basically a mixture of two policies (medium level and expert level). If the trajectories are split into multiple groups according to the returns, it's likely that the root cause of the varied returns is initial states rather than the quality of the policy. One clue is the experiments on medium-replay data in Figure C.2 show for hopper and walker2d, the quality of the unlabelled data plays a much more important role than the case in the medium and medium expert.\n\nMinor issues\n1. Is \"label\" a proper word to replace action? My impression is unlabelled data in offline RL is more about reward-missing data. The intuition behind that is humans can label rewards to trajectories easily because it's a scale but \"labelling\" actions seems like very difficult.\n2. There are some obvious grammatical errors in the paper: e.g. \"we are mainly interested in the case **only** a **significant majority** of the trajectories in the offline trajectories are **unlabeled**.\" \"How can we utilize the unlabelled data for improving **the performance offline RL algorithms**?\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "bdlDjxuX2eP", "reviewer": "Reviewer_Mz9R", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new, practically motivated semi-supervised setting, where the agent can access both labeled trajectories and unlabelled trajectories that do not include the actions of the trajectories. A model is trained to give actions for unlabeled data and then the offline RL method is trained by the whole dataset. Experiments based on D4RL dataset show the good performance of the proposed method.", "summary_of_the_review": "See the above-mentioned questions.", "strength_and_weaknesses": "The paper is well-written and easy to follow. The setting is novel and meaningful since there are a lot of unlabeled data like videos in real-world scenarios.  The experiments are designed carefully to illustrate the influence of data with different qualities. For the weakness, Iwould like to ask some questions:\n1. For IDM, the length of the input, k, could be changed, and k=1 in the paper. So the input of the model is four states (s\n_t-1,..., s_t+1). How does the model encode these states?\n2.  The experiments only use the expert dataset, which means most trajectories are good. D4RL also provides random and medium-level dataset. As your claim in the paper, the quality of the data has a huge influence on the performance. Is there any analysis based on these data? Could we say that the method also needs high-quality data for both labeled and unlabeled data to achieve good performance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject