CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning
Authors: Jinxin Liu
Lipeng Zu
Li He
Donglin Wang
Venue: CoRL 2023 Poster (2023)
Keywords: Offline Reinforcement Learning
Intrinsic Rewards
Learning Skills

Abstract: Offline reinforcement learning (RL) aims to learn an optimal policy from pre-collected and labeled datasets, which eliminates the time-consuming data collection in online RL. However, offline RL still bears a large burden of specifying/handcrafting extrinsic rewards for each transition in the offline data. As a remedy for the labor-intensive labeling, we propose to endow offline RL tasks with a few expert data and utilize the limited expert data to drive intrinsic rewards, thus eliminating the need for extrinsic rewards. To achieve that, we introduce Calibrated Latent gUidancE (CLUE), which utilizes a conditional variational auto-encoder to learn a latent space such that intrinsic rewards can be directly qualified over the latent space. CLUE's key idea is to align the intrinsic rewards consistent with the expert intention via enforcing the embeddings of expert data to a calibrated contextual representation. We instantiate the expert-driven intrinsic rewards in sparse-reward offline RL tasks, offline imitation learning (IL) tasks, and unsupervised offline RL tasks.  Empirically, we find that CLUE can effectively improve the sparse-reward offline RL performance, outperform the state-of-the-art offline IL baselines, and discover diverse skills from static reward-free offline data.

CLUE: Calibrated Latent Guidance for Offline Reinforcement Learning
Jinxin Liu' Zhejiang University
Lipeng Zu*
Westlake University zulp@mail ustcc.cc...o edu.
Westlake University liujinxin inujnssssstlaeee edu. cn
Li He
Donglin Wang Westlake University
Westlake University
heli. celi.ccoperffoxma. com
edu.
Abstract: Offline reinforcement learning (RL) aims to learn an optimal policy from pre-collected and labeled datasets, which eliminates the time-consuming data collection in online RL. However, offline RL still bears a large burden specifying/handrrattingg extrinsic rewards for each transition the offline data. As remedy for the labor-intensive labeling, we propose endow offline RL tasks with a few expert data and utilize the limited expert data to drive intrinsic rewards, thus eliminating the need for extrinsic rewards. To achieve that, introduce Calibrated Latent g U (CLUE), which utilizes a conditional variational auto-encoder to learn a latent space such that intrinsic rewards can be directly qualified over the latent space. CLUE's key idea to align the intrinsic rewards consistent with the expert intention via enforcing the embeddings expert data calibrated contextual representation. We instantiate the expert-driven intrinsic rewards in sparse-reward offline RL tasks, offline imitation learning (IL.) tasks, and unsupervised offline RL tasks. Empirically, we find that CLUE can effectively improve the rewad offline RL performance, outperform the state-of-the offline IL baselines, and discover diverse skills from static reward-free offline data.
Keywords: Offline Reinforcement Learning, Intrinsic Rewards, Learning Skills
1 Introduction
Recent advances in reinforcement learning (RL) have shown great success in decision-making domains ranging from robot manipulation [1, 2] to navigation [3, 4] and large-language models [5]. Generally, an RL agent receives two sources supervisory signals associated with the learning progress: 1) environment transition dynamics and 2) task-specifying rewards, where 1) the transition dynamics coordinate the agent's behaviors toward the environment affordances and 2) the task- specifying rewards capture the designer's preferences supervised signals themselves also limit the applicability of RL methods, since in many tasks, especially in real-world domains, either collecting online environmental transitions or labeling complex task-specifying rewards is time-consuming and laborious.
agent behaviors. However, the two
To tackle the above challenges, two separate RL branches have been proposed: 1) offline RL [6]. also known as batch RL, which promises learn effective policies from previously-collected static datasets without further online interaction, and 2) intrinsic rewards [7], which aim to capture rich form of task knowledge (such as long-term exploration or exploitation) that provides additional guidance on how an agent should behave. Aligning with the task-specifying rewards, such intrinsic
"iixiin Liu and Lipeng Zu contribute equally to this work. !Coresponding author: Donglin Wang.
7th Conference Robot Learning 2023), Atlanta, USA.

---

sarrescil trajectories
Offiiee RL
solect
Offline
clustering
class
CLUE
class 3
spacse-ewwrd
rewaad-rraabeed oftiine data
:
offiine
expert trajectories
reward-frre
Offlie RL
CLUE
CLUE
resccad-fiiee offiine data
reward-labeled ofline dala
reworddlabbded
oflinedata
Figure 1: Three instantiations for the assumed "expert" data in offline RL settings: 1) sparse-reward offline RL, 2) offline imitation learning (IL) setting, and 3) unsupervised offline RL setting (aiming to learn diverse skills/policies from static reward-free offline data).
rewards promise to accelerate online RL by augmenting or replacing the manual task-specifying rewards (hereafter extrinsic rewards). In fact, prior offline RL methods [6] typically introduce policy/value regularization and operate in a form of reward augmentation, which thus can be seen as a special kind of intrinsic motivation. However, such intrinsic motivation is only designed eliminate the potential out-of-distribution (OOD) issues in offline RL and does not account for representing task-specifying behaviors (extrinsic rewards). In this work, we aim to design offline RL intrinsic reward that promotes offline RL performance while representing task-specifying behaviors.
worth noting that adapting the online intrinsic rewards to offline RL problems is non-trivial. In online RL, intrinsic rewards often capture the long-term temporal dependencies of interaction trajectories [7. 8]. For example, Badia al. [9] capture the novelty of states across multiple episodes; Eysenbach et [10] quantify the discriminability between skills represented by latent variables. However, such temporal dependencies rely on online interaction transitions, which thus cannot be straightly captured in offline settings. In this work, we thus propose to discard the above temporal dependence scheme and use an "expert" to facilitate labeling intrinsic rewards and guiding the offline agent. To do so, we identify three scenarios for the expert instantiations offline RL settings:
For sparse-reward offline RL tasks, we filter out the trajectories that do accomplish the tasks and take the success trajectories the expert behaviors. By relabeling continuous intrinsic rewards for those failed trajectories, we expect such reward relabelling procedure can promote offline learning.
2) For reward-free offline RI. tasks, we assume that the agent has access to additional (limited) expert data generated by an expert policy. We expect that such limited expert data can provide useful intrinsic rewards for unlabeled transitions and then bias the learning policy toward expert behaviors.
3) Also considering the reward-free offline RL setting, we do not assume any additional expert data. Instead, we choose to cluster the offline transitions into a number of classes and take each class as separate "expert". Then, we encourage offline agents to produce diverse behaviors when conditioned on different classes, in a similar spirit to the unsupervised skill learning in online RL.
We see that in all three settings (Figure 1), we assume the existence of an expert (limited expert data), either from trajectory filtering, from external expert, through clustering. To instantiate the above intrinsic rewards, we propose Calibrated Latent gUidanc (CL.U),, which aims to label intrinsic rewards for unlabeled (or spare-reward) transitions in the offline RL setting. Specifically, CLUE uses a conditional variational auto-encoder to learn a latent space for both expert data and unlabeled data, then labels intrinsic rewards by computing the distance between the latent embeddings of expert and unlabeled data. CLUE's key idea to explicitly bind together all the embeddings of expert data, thus learning a calibrated embedding for all expert behaviors. Intuitively, this binding procedure encourages the latent space to capture task-oriented behaviors such that latent space can produce task-oriented intrinsic guidance when computing distance over the latent space.

---

In summary, we make the following contributions in this paper: 1) We propose CLUE, which can provide pluggable intrinsic rewards for offlin RI. methods. 2) We demonstrate CLUE can effectively improve the spare-reward offline RL performance. 3) Considering offline imitation learning (IL) settings, CLUE can achieve better or comparable results compared to both the reward-labeled offline RL methods and the state-of-the-ar offline IL methods. 4) We find that CLUE is able discover diverse behaviors/skills in the unsupervised (reward-free) offline RL setting.
2 Related Works
The goal of our work is to learn task-oriented intrinsic rewards for sparse-reward or reward-free offline data. While there is a large body research on learning rewards for RL tasks [11, 12, 13, 14, 15], most work assumes online RL settings, while we consider the offline RL setting. Additionally, little work has yet verify intrinsic rewards across sparse-reward. IL, and unsupervised RL tasks together.
Typically, many intrinsic rewards have been proposed to encourage exploration in sparse-reward (online) RL tasks. In this case, intrinsic rewards are often formulated as state visitation counts [16, 17, 18], prediction error [19. 20), prediction uncertainty [21, 22], information gain [23], state entropy [24, 25, 26), and deviation from a default policy [27, 28]. However, these intrinsic rewards are often not well aligned with the task that the agent solving. In contrast, the goal of our work is learn task- oriented intrinsic reward such that it promotes the policy learning progress for sparse-reward tasks.
Beyond the standard offline RL setup, learning from (static) reward-labeled offline data [6, 29, 30, 31, 32], offiin imitation learning (IL) considers learning from expert trajectories and (reward-free) sub- optimal offline data, which can be generally folded into two paradigms [33]: behavior cloning (BC) and offline inverse RL (IRL). BC directly learns a policy from expert trajectories using supervised learning [34]. Due compounding errors induced by covariate shift [35], BC methods require large amount of expert data, thus hindering the application data-scarce scenarios. To overcome such limitations, offline IRL methods consider matching the state-action distributions induced by the expert [36, 37, 38, 39, 40, 41]. Typically, they formulate the expert matching objective by introducing a discriminator and trying to find the saddle point of a min-max optimization, which tends to be brittle and sensitive to the training (offline) data. However, our CLUE does not introduce any adversarial objective, thus exhibiting more robust performance on a wide variety of tasks.
The idea of unsupervised RL is to learn diverse behaviors/skills in open-ended environment without access to extrinsic rewards [42, 43, 44]. Previous unsupervised RL methods are often formulated through the lens of empowerment [45]. Central to this formulation is the information-theeortiic skill discovery approach, where diverse skills can be discovered by optimizing the long-term temporal dependencies of interaction trajectories, e.g., maximizing the mutual information between induced trajectories and some latent/context variables [10, 46, 47, 48, 49, 50, 51]. this work, we propose to discard this online temporal dependence scheme and use clustering methods formulate such diversity and use CLUE to label intrinsic rewards to guide offline agents.
3 Preliminary
Offline RL. We consider RL in a Markov Decision Process (MDP) M (S, po. 7). where the state space, A is the action space, the environment transition dynamics, r is the task- oriented extrinsic reward function, Po is the initial state distribution, and y the discount factor, The goal of RL is to find optimal policy To (a(a)) that maximizes the expected return Exo(r) 1 rrr, when interacting with the environment M, where trajectory := (So. ao, To, 51, denotes the generated trajectory, Su ~ (o(),), extrinsic reward r(s,, at) at time step f. In offline RL, the agent can not interact with the environment and only receives a static dataset of trajectories 'D == [T; ) pre-collected by one or a mixture of (unknown) behavior policies. Then, the goal of offline RL is to find the best policy from offline data.
~ To(a, (s)), S1+1 ~ T(s a.), and denotes the
Conditional variational auto-encoders (CVAE). Given offline data x, the variational auto-encoder (VAE) [52] proposes to maximize the variational lower bound,
3

---

log Pa (x) = (z](1s(z))) (22g(())) + B B (- log Uo (2|x) + log Pe (x,
(1)
2 (x]x))( + (x/x)) [log pe (x z)],
(2)
where pz(z) the prior distribution, go (z(xx) denotes the encoder model, and po(x(z) denotes decoder
model. Considering the structured output prediction settings, conditional VAE (CVAE) maximizes the variational lower bound of the conditional log- ligliikel:oood:
log (xly) 2 - KLL(as (
(z(())) + Bxl(a)x))) (x
(3)
4 CLUE: Calibrated Latent Guidance
In this section, we introduce our method CLUE (Calibrated Latent g gUidancE) that learns calibrated latent space such that intrinsic rewards can be directly gauged over the latent space. We begin by assuming access to limited expert offiin data and describe how we can use it to label intrinsic rewards for reward-free offline data in Section 4.1. Next in Section 4.2, we describe three offline RL instantiations, including one sparse-reward and two reward-free (offline imitation learning and unsupervised offline RL) settings, each corresponding to a scenario discussed previously (Figure 1).
4.1 Calibrated Intrinsic Rewards
Assuming access to limited expert offline data D : [ss a. ))) and a large number of reward-free offline data D= == {(s, a, } our goal is to use DD to learn an intrinsic reward function f(s, a) for the reward-free transitions in D, such that we can recover expert behaviors using the relabeled offline data D, == {ss, a, f, s') With a slight abuse of notation, here we write in transitions ((s, a, f()) denote the relabeled intrinsic reward f(s, a).
We first use CVAE to model the mixed offline behaviors in De D.D. Specifically, we take state as the input/conditional variable and take action a as the prediction variable. For each behavior samples (s, a) in mixed data De U DD we maximize the following variational lower bound:
log po (a(s) 2 - (L a) a)lppe (z 8)) + Egg (2la.s) [log (o(a,,,))
1
~ kL (2lp +
T logpo(alz s) as; (5)
where (1)
Nz(lpp a), of (s. a)) 2, L the number of samples, and LCVAE(s, a 0; の) is the corresponding empirical lower bound. For simplicity, set the prior distribution the standard Gaussian distribution, i.e., pz(z|s) 10.
For a query sample a), we label its intrin- sic reward f(s, a) by computing the negative distance between the latent embeddings of the expert data and the query sample,
CVAE
CLUE((urs)
sumiard desiation
2.00
epport
stand and devintionn
eper
8.56
(.44
6:52
f(s, a) exp (-c Iize z(s, a).
(6)
where 2, = (s.a)-D (z)s a)). 2(s, a) 9s (z]s, a), and e > 0 is temperature factor.
Figure 2: Latent embeddings of expert and non- expert offline data on D4RI. animaze-medium diverse-v2 dataset, where embeddings are learned by the naive CVAE (left) and our CLUE (right).
However, naively maximizing LEVaE Equa- tion 5 may lead to undesirable embeddings with varying scales that do not capture task-relevant behaviors when computing the latent distance. For example, in Figure left, visualize the embeddings of the expert data De and the unlabeled (non-expert) offline data 'D. We can see that the embeddings for the expert and non-expert data are generally mixed together without clear separation. There is large variance in expert data embeddings,
"e define (s, a) and (s, a) to be feed- forward networks with parameters Φ. taking concatenated s and and outputting the parameters (mean and std) of Gaussian distribution in the latent space, respectively.

---

and directly estimating the mean of the expert embeddings (i.e., Ze = ((s.a)-D= (z)s, a)]) cannot effectively represent task-oriented behaviors, causing the labeled intrinsic reward f(ss, a) to be biased.
To guarantee the intrinsic reward formulation f(s. a) in Equation propose learn calibrated embeddings. To do so, explicitly bind together the expert embeddings, expecting the expert embeddings "collapse" into single embedding. Thus, we introduce the following calibration regularization over expert embeddings:
be task task-oriented, we thus
min Cealier == (s..)-- [1] (s. a)] + Io a)]?
(7)
Due to the standard Gaussian prior for po(z)s) in Equation 5, constrain not only the variance of the expert embeddings but also the mean of the expert in Equation 7. Intuitively, Cealibr unifies expert embeddings (" ("collapsed" to single point), therefore providing effective Ze when computing intrinsic rewards f(s, a) in Equation 6. As shown in Figure 2 right, the expert embeddings and their mean are almost bound to a single point, so we can directly measure intrinsic rewards in latent space.
4.2 Intrinsic Reward Instantiations
Here we describe three offline instantiations, one spare-reward, and two reward-free offline settings, that permit us to meet the previous expert data assumption (in Section 4.1) and label intrinsic rewards.
Spare-reward offiine RI.. Considering the challenging spare-reward offline data {(s, a, r, s')} we can filter out the unsuccessful trajectories and take the finished trajectories the expert data. Then, can replace the original spare rewards with the learned continuous intrinsic rewards.
Offline H.. Considering the reward-free offline RL data ((s, s')) }. we can assume the agent has access to additional expert data (as few as only one trajectory). Then we can use the learned intrinsic rewards to relabel the reward-free transitions, obtaining labeled offline data {(s, a, f, s')}.
Unsupervised offline RL. Given the reward-free offline data {(s, s'),, algorithms to cluster the data into multiple classes and then treat each class separate "expert" data. In this way, we expect to learn different skills/policies when conditioned in different classes.
clustering
5 Experiments
In this section, we first empirically demonstrate the advantages our pluggable intrinsic rewards in sparse-reward offline RL tasks. Second, we then evaluate CLUE in offline II. tasks, studying how effective our intrinsic reward is in contrast broad range of state-of-the-an offline imitation learning methods. Third, we study CLUE in unsupervised offline RL settings, expecting to discover diverse behaviors from static offline data. Finally, we conduct ablation studies on the calibration regularization and the amount of unlabeled offline data. All of our results are evaluated over 10 random seeds and 10 episodes for each seed.
Implementation. Note that intrinsic rewards are pluggable and can be combined with any offline RL algorithms. In our implementation, we combine CLUE with the Implicit Q-Learning (IQL) algorithm [53] which is one of the state-of of-the-art offline algorithms and can solve most of the (reward-labeled) offline tasks with competitive performance. Our base IQL implementation is adapted from IQL', and we set all hyperparameters to the ones recommended the original IQL paper.
5.1 Rewase Offfine RL Tasks
Here we evaluate CLUE on spares-reward tasks (including AntMaze and Adroit domains) from the D4RL benchmark. In this setting, we use the inherent sparse rewards in the dataset to select expert data, i.e., selecting the task-completer or geal-reached trajectories from the sparse-reward dataset and treating them as the expert data. In Tables and 2, we compare our method with baseline methods (IQL [53] and OTR [54)) when only one expert trajectory* is selected. For comparison, we train
hutpss//giiibb.om//kottkooooiii con//oostikkoviiiiiiitt... learning.
'In the appendix, we also compare our method with baseline methods when at most 10 completed trajectories selected, where "at most 10" refers that there may be less than 10 successful trajectories in D4RL dataset.
5

---

Table I: Normalized scores (mean and standard deviation) of CLUE and baselines on sparse- reward AntMaze tasks, where both OTR and CLUE use IQL as the base offline RL algorithm. The highest scores between our CLUE and base- line OTR are highlighted.
Table 2: Normalized scores (mean and standard deviation) of CLUE and baselines on sparse- reward Adroit tasks, where the highest scores be- tween CLUE and OTR are highlighted.
Dataset
IQL
OTR
CLUE
door-cloned
1.6
0.01
0.02
Dataset
IQL
OTR
CLUE
door-human
4.3
5.9
7.7
umaze
88.7
81.6 7.3
92.1 3.9
hammer-cloned
2.1
0.9
1.4
umaze-diverse
67.5
70.4 +
61.0
hammer-human
1.4
18
1.9
medium-pla play
72.9
73.9
63
pen-cloned
37.3
20.9
59.4
medium-diverse
72.1
6.9
74.6
pen-human
71.5
66.8 21.2
82.9
large-play
43.2
49.7
55.8
relocate-cloned
-0.2
-0.24 a 0.03
-0,23
large-divents
46.9
48
4.9
relocate-human
0.1
0.1
0.2
AntMaze-v2 total
391.3
396.2
415.7
Adroit-vO total
118.1
122.2
153.3
Table 3: Normalized scores (mean and standard deviation) of CLUE and baselines on locomotion tasks using one (K=1), five (K=5), and ten (K=10) expert demonstrations, Both CLUE and OTR uses IQL as the base offline RL algorithm, and we highlight the highest score in each setting.
Dataset
IQL
OTR (K=1)
CLUE (K=1)
OIR (K=5)
CLUE (K=5)
OTR (K=10)
CLUE (K=10)
halfchoctal-medimm feffhhetthh-mediu--rce
27.4
432
45.6
232
4522
43.1
45.7
44.2
41.3
43
41.9
43.2
41.6203
43.2
halfcheetah- medium--xppee
86.7 5.A
89.6 1.0
90.0-
89.9 19
91
87.9
91.0
hopper medium
6..7
78.
78.3
79.5
19.1
80.0
29.
hopper-men saerr--eddimmmpllleeee
9..
84.8
94.3
8..4
93.3
84.4
93.7
hopper medium expert
91.5 14.3
23.2
14.5
21.4
104.0.54
96.6
102.3
walker2d-mede 2d-mudi
7.7
7..4
80.7
19
79.6
79.2
81.7
walker2d-me 24-mediun--epple
73.8
66.0
76.3=28
71.0200
13.
71,8+38
75.3
walker2d medium--xxerr
109.3
109.4.0.4
10..9.0..
109.6+0.5
110.7
locomation- v2 total
69244
685.6
714.5
690.6
731.3
694.2
723.5
IQL over the naive sparse-reward D4RL data and train OTR over the relabeled D4RL dataset (using optimal transport to compute intrinsic rewards and employing IQL to learn offline RL policy). We can find that 13 out of 14 tasks across AntMaze and Adroit domains, our CLUE outperforms the baseline OTR. Meanwhile, compared to naive IQL (with sparse rewards). our CLUE implementation obtains total score of 106.2% AntMaze tasks and 129.8% Adroit tasks. This means that with only a single expert trajectory, reward in offline RI. tasks, which can even achieve higher performance.
completely replace the sparse rewards with our intrinsic
5.2 Offline Imitation Learning Tasks
Then, we evaluate CLUE on offline II. tasks. We continue to use the D4RL, data as offline data, but here we explicitly discard the reward signal. Then, we use SAC to train an online expert policy to collect expert demonstrations in each environment. We first compare CLUE to 1) naive IQL with the (ground-truth) reward-labeles offline data and 2) OTR under our offline ILL setting. In Table 3, we provide the comparison results with 1, 5, and 10 expert trajectories. We can see that in 22 out of 27 offline IL settings, CLUE outperforms (or performs equally well) the most related baseline OTR, demonstrating that CLUE can produce effective intrinsic rewards. Meanwhile, with only a single expert trajectory, our CLUE implementation can achieve 103.2% of the total scores of naive IQL. in locomotion tasks. This means that with only one expert trajectory, our intrinsic rewards can even replace the continuous ground-truth rewards in offline RL tasks and enable better performance.
Next, we compare CLUE to representative set of offline IL baselines: SQIL[55] with TD3+BC [56] implementation, ORIL [57], IQ-Learn [58], ValueDICE [37], DemoDICE [59], and SMODICE [60]. Note that the original SQIL is an online IL method, here we replace its (online) base RL algorithm with TD3+BC, thus making it applicable to offline tasks. Table 4, we provide the comparison results over D4RL locomotion tasks. We can see that, overall, our CLUE performs better than most offline IL baselines, showing that intrinsic reward is well capable of capturing expert behaviors. Further, we point out that CLUE is also robust to offline data with different qualities: most previous

---

Table 4: Normalized scores (mean and standard deviation) of CLUE and offline IL baselines on MuloCo locomotion tasks using one expert trajectory (K=l) and ten expert trajectories (K=10). We highlight the scores that are within two points of the highest score.
Dataset
SQIL
IQ
IQ-Leam
ORIL
ValueDICE
DemoDICE
SMODICE
CLUE
halfeheetah-medding
24.
21.7
16.8
36,4
42.0a08
42.4
45.6
1.
1.
1.12
39
3.
38.3
43.5
halfcheetah medium expert
6.7+12
2.
48.7+22
1,0+24
66.2+43
80,9
90.0+24
66.9
29.6
96.3
16
78.3
hopper medium replay
98.6=0.7
23.0=94
16.7
52.5
70.7
30.4
94.3
hoppe--medium-cpper
13.6
9.1
22.
2
103.7
82.4
96.5 14.7
walker 2d medium
11.9
4.7
136
13.9
74
67.8
80.7
walker 2d- ddmmedium-rrplay
42.
17.0=76
12.7
57.2
4..
23
walker2d- 2d-mmdiumm-xpeeret
18.8 13.1
7.7
11.6e 14.7
3 13.7
87.3 105
94,8
109.3 2.1
hallcheetah mediiam
29.2+
5.97
40.0+1.
41,9.05
41.6
45.7 +.2.
halfcheetah medium replay
a0.
29.6
46.2=06
39,6
26.5
39.3
43.2
halfcheetah medium expert
114+43
2.9
46.6=60
25.2+
67.1
84.4
91.0
hopper-mediun
65.8
31.6262
101.5006
37.6295
$7.4
$5.9
29.9
nopper-medium-ree
96.6
38.0=70
29.0
83,6
56.9
32.6
93.7
hopper-medium--r
19.6
19.3
18.9
28.4
96.9
89.3
102.3
walker2d-medium
22.4
46.4
8..
54.3
71.3
67.9
81.7
walker 2d medium replay
82.4+47
9..6
70.0
54.6+96
58.1+79
52.4+68
75.3 4.6
12.5
24
6.5 882
40.1 194
103.5
107.5 1.0
110.7 8.2
Ant-v2
HalfChectab-w2
Walker2d-22
medium (ampa
Invddaun (100 steps)
medimma (100 steps)
6.87
8.81
0.67
random (20 stpps)
Fandom (60 steps)
random (100 steps)
79 717
Figure 3: Qualitative visualizations of the learned skills in Ant, HalfCheetah, and Walker domains. We can see that the ant learns to move in different directions, the half-cheetah learns to flip upright and run at different speeds, and the walker learns to walk at different speeds.
adversarial baasd methods deliberately depict unlabeled offline data as sub-optimal and tag expert data optimal, which easily lead to biased policyldiseriminator. For example, see that ORIL's performance deteriorates severely all medium-exper tasks. On the contrary, CLUE does not bring in any adversarial objectives and is therefore much more robust.
5.3 Unsupervised Offline RL Tasks
Considering the reward-free offline RL settings, here we expect to learn diverse skills/policies from the static offline data. so, we first use K-means clustering to cluster similar transitions and treat the transitions in each of the clustered classes as separate expert data. For each class, we then use CLUE to learn the corresponding intrinsic reward function and label intrinsic rewards for the of the unlabelled data to learn corresponding skills. We visualize the learned diverse behaviors in Figure (see videos in the supplementary material). We see that our CLUE+K-means implementation provides a promising solution to unsupervised offline RL, which successfully produces diverse set of skills and thus illustrates a huge potential for skill discovery from static offline data.
5.4 Ablation Studies
Ablating the calibration regularization. The key idea of CLUE encourage learning calibrated expert embeddings, thus providing effective embedding space when computing intrinsic rewards.

---

Here we verify this intuition by ablating the calibration regularization Cealibr in Equation 7, and directly using CVAE to learn the embed- ding space. We show ablation results in Fig- ure 4. We can observe that the naive CVAE implementation (ablating the calibration regular- ization L suffers from significant perfor- mance drop compared with our CLUE,, indicat- ing that our calibration regularization is effective in promoting embedding alignment and produc- ing task-oriented intrinsic rewards.
K-10
pertemmance dlffomence
u-d mop m-d Up 1- Figure 4: Ablating the effect of the calibration reg- ularization. We can that ablating the regular- ization generally causes performance degradation, implying that calibration regularization deed encourage task-oriented intrinsic rewards. u: umaze. medium. I: large. d: diverse. p: play.
u-d map m- kp 141
in-
Varying the amount of unlabeled offiine data. To assess the effectiveness of our intrinsic re-
wards under data-scarce scenarios, here we vary the amount of unlabeled offline data available for offline II. settings (see results for sparse-reward settings in the appendix). In Figure 5, we show the normalized results with a small number of D4RL data ranging from 5% ~ 25%. We can see that across a range of dataset sizes, CLUE can perform well and achieve competitive performance compared the state-of-le of-thh-art offline baselines.
halfchectah
hulfcheetah
Issper-modien--2
marmalted scoet
Marmallee scorE
Intmallee scurE
15
19
15
18
15
26
18
15
the amount offline data 1% (4RL)
the amount of offiine data 1% D4RL)
the smount of offfiie data (% D4RL.)
the smount of offline data (% D4RL.)
CLUE
OTR
SQIL
1Q-Learn
DemoDICE
SMODICE
Figure 5: Ablating the number of unlabeled offline data. To compare CLUE with the baseline that CLUE's performance
methods, shade the area below the scores of our CLUE. We generally outperforms offline baselines across a range of domains and dataset sizes in each domain.
6 Discussion and Limitations
In this paper, we propose calibrated latent guidance (CLUE) algorithm which labels intrinsic rewards for unlabeled offline data. CLUE is an effective method and can provide pluggable intrinsic rewards compatibility with any offline RL algorithms that require We have demonstrated that CLUE can effectively improve the spare-reward offline RL performance, achieve competitive performance compared with the state-of-the-an baselines in offline IL tasks, and learn diverse skills in unsupervised offline RL. settings.
data for offline learning.
Future work and limitations. Our CLUE formulation assumes the presence large batch of offline data and expert trajectories. This setting is common in many robotic domains. However, we also point out that in some tasks, expert trajectories may be state-only and not contain actions. Also, there may be transition dynamics shifts between the expert data and the unlabelled offline data in some cross-domain problem settings. Thus, future directions could investigate the state-only expert data and the cross-domain intrinsic rewards. view of the fact that our intrinsic rewards are measured in the latent space, it is feasible to apply our CLUE approach in both the state-only and cross-domain scenarios, as long as we impose the corresponding regularization on the learned latent embeddings. For example, regularization, making CLUE suitable for cross-domain tasks. In summary, we believe future work to this work will contribute to general reward-relabelin framework capable of labeling effective intrinsic rewards and addressing more realistic robotic tasks e.g., discovering diverse robot behaviors from static manipulation data and transferring offline cross-domain behaviors in sim simareal tasks.
directly add cross-domain constraints [61, 62)
our calibration

---

Acknowledgments
This work was supported by the National Science and Technology Innovation 2030 Major Project (Grant No. 2022ZD0208800), and NSFC General Program (Grant No. 62176215).
References
[1] D. Kalashnikov, A. Irpan, Pastor, Ibarz, Herzog, E. Jang, D. Quillen, Holly, M. Kalakr- ishnan, V. Vanhoucke, et al. Scalable deep reinforcement learning for vision-based robotic manipulation. In Conference on Robot Learning, pages 651-673, PMLR, 2018.
[2] A. Singh, L. Yang, K. Hartikainen, C. Finn, and S. Levine. End-to-end robotic reinforcement learning without reward engineering, arXiv preprint arXiv:1904.07854 2019.
[3] K. Zhu and T. Zhang. Deep reinforcement learning based mobile robot navigation: A review. Tsinghua Science and Technology, 26(5):674-691, 2021.
[4] D. Shah, B. Eysenbach, N. Rhinchart, and S. Levine. Rapid exploration for open-world navigation with latent goal models. arXiv preprint arXiv:2 10.005859 2021.
[5] L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[6] S. Levine, A. Kumar, G. Tucker, and J. Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[7] Z. Zheng, J. Oh, M. Hessel, Z. Xu, M. Kroiss, H. Van Hasselt, D. Silver, and S. Singh. What can learned intrinsic rewards capture? In International Conference on Machine Learning, pages 11436- 1446. PMLR, 2020.
[8] S. Hansen, G. Desjardins, K. Baumli, D. Warde-Farley, N. Heess, S. Osindero, and V. Mnih. Entropic desired dynamics for intrinsic control. Advances in Neural Information Processing Systems, 34:1 1436-1144444 1448, 2021.
[9] A. P. Badia, P. Sprechmann, A. Vitvitskyi, D. Guo, B. Piot, S. Kapturowski, O. Tieleman, M. Arjovsky, A. Pritzel, A. Bolt, et al. Never give up: Learning directed exploration strategies. arXiv preprini arXi::2002.060338 2020.
[10] B. Eysenbach, A. Gupta, Ibarz, and S. Levine. Diversity is all you need: Learning skills without reward function. arXiv preprint 17Xiv:1802.00070, 2018
[11] B. Ibarz, J. Leike, T. Pohlen, G. Irving, S. Legg, and D. Amodei. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31, 2018.
[12] D. S. Brown and S. Niekum. Deep bayesian reward learning from preferences. arXiv preprint 1rXiv:1912.0447777 2019.
[13] D. Lindner, M. Turchetta, S. Tschiatschek, K. Ciosek, and A. Krause. Information directed reward learning for reinforcement learning. Advances Neural Information Processing Systems, 34:3850-3862. 2021.
[14) X. Yu, Y. Lyu, and I. Tsang. Intrinsic reward driven imitation learning via generative model. In International conference on machine learning, pages 10925-10935. PMLR, 2020.
[15] E. Biyik, D. P. Losey, M. Palan, N. C. Landolfi, Shevchuk, and D. Sadigh. Learning reward functions from diverse sources of human feedback: Optimally integrating demonstrations and preferences. The International Journal of Robotics Research, 41(1):45-67, 2022.

---

[16] M. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton, and R. Munos. Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016.
[17] H. Tang, Houthooft, Foote, A. Stooke, O. Xi Chen, Y. Duan, J. Schulman, F. DeTurck, and Abbeel. exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
[18] G. Ostrovski, M. G. Bellemare, A. Oord, and R. Munos. Count-based exploration with neural density models. In International conference on machine learning, pages 2721-2730. PMLR, 2017.
[19] D. Pathak, P. Agrawal, A. Efros, and Darrell. Curiosity-driven exploration by self- supervised prediction. In International conference on machine learning, pages 2778-2787. PMLR, 2017.
[20] Y. Burda, H. Edwards, A. Storkey, and O. Klimov. Exploration by random network distillation. arXiv preprint arXiv:18100 12894, 2018.
[21] D. Pathak, D. Gandhi, and A. Gupta. Self-super exploration via disagreement. In International conference on machine learning, pages 5062-5071. PMLR, 2019.
[22] R. Sekar, O. Rybkin, K. Danillidis, P. Abbeel, Hafner, and D. Pathak. Planning explore via supf-supe world models. In International Conference Machine Learning, pages 3583-8592. PMLR, 2020.
[23] R. Houthooft, X. Chen, Y. Duan, J. Schulman, De Turck, and P. Abbeel. Vime: Variational information maximizing exploration. Advances in neural information processing systems, 29, 2016.
[24] L. Lee, B. Eysenbach, E. Parisotto, E. Xing, S. Levine, and R. Salakhutdinov. Efficient exploration via state marginal matching. arXiv preprint arXi::1906.05274 2019.
[25] Y. Seo, L.. Chen, J. Shin, H. Lee, P. Abbeel, and K. Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pages 9443-9454, PMLR, 2021.
[26] H. Liu and P. Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34:18459-1841 2021.
[27] D. Strouse, M. Kleiman- Weiner, J. Tenenbaum, M. Botvinick, and D. J. Schwab. Learning to share and hide intentions using information regularization. Advances in neural information processing systems, 31, 2018.
[28] A. Goyal, R. Islam, D. Strouse, Z. Ahmed, M. Botvinick, H. Larochelle, Bengio, and Levine. Infobot: Transfer and exploration via the information bottleneck. arXiv preprint 1rXiv:1901.10900 2019.
[29] Z. Zhuang, K. Lei, J. Liu, D. Wang, and Y. Guo. Behavior proximal policy optimization. arXiv preprint arXiv:2302.11312. 2023.
[30] Y. Lai, J. Liu, Z. Tang, B. Wang, H. Jianye, and P. Luo. Chipformer: Transferable chip placement via offline decision transformer. ICML, 2023. URL j 1wiiW7
http:perrrrieeeeteteeeeeeeeeeeee hwtp:neettppf/// iid=
[31] Liu, H. Zhang, Z. Zhuang, Y. Kang, D. Wang, and B. Wang. Design from policies: Conser- vative test-time adaptation for offline policy optimization. arXiv preprint arXiv:2306.144. 2023.
10

---

[32] J. Liu, Z. Zhang, Z. Wei, Z. Zhuang. Y. Kang, S. Gai, and D. Wang, Beyond ood state actions: Supported cross-domain offine reinforcement learning. arXiv preprint (rXiv:2306.12777 2023.
[33] H. Xu, X. Zhan, H. Yin, and H. Qin. Discriminnatrr-weighted offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, pages 24725- 24742. PMLR, 2022.
[34] D. A. Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural compulation, 3(1):88-97, 1991.
[35] S. Ross, G. Gordon, and D. Bagnell. reduction imitation learning and structured prediction to no-regret online learning. Proceedings of the fourteenth international conference on artifi- cial intelligence and statistics, pages 627-635, JMLR Workshop and Conference Proceedings, 2011.
[36] J. Liu, He, Y. Kang, 2. Zhuang. D. Wang, and H. Xu. Ceil: Generalized contextual imitation learning. arXiv preprint arXiv:2306. 2023.
[37] I. Kostrikov, O. Nachum, and Tompson. Imitation learning via off-policy distribution matching. arXiv preprint arXiv:
2019.
[38] M. Sun, A. Mahajan, K. Hofmaan, and Whiteson. Softdice for imitation learning: Rethinking off-policy distribution matching. arXiv preprint wrXiv:2106.03/55 2021.
[39] Jarboui and V. Perchet. Offline inverse reinforcement learning. arXiv preprint urXiv:2106.05006 2021.
[40] G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: A game- theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032. PMLR, 2021.
[41] B. He, Z. Sun, J. Liu, S. Zhang, X. Chen, and C. Ma. Offline imitation learning with variational counterfactual reasoning. arXiv preprint 2rXiv:2310.04700 2023.
[42] M. Laskin, D. Yarats, H. Liu, Lee, A. Zhan, K. Lu, C. Cang. Pinto, and P. Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 10. 15/91, 2021.
[43] V. Campos, A. Trott, C. Xiong, R. Socher, X. Giró-i Nieto, and J. Torres. Explore, discover and learn: Unsupervised discovery of state-covering skills. In International Conference on Machine Learning, pages 1317-1327. PMLR, 2020.
[44] Q. Tian, G. Wang, J. Liu, D. Wang, and Y. Kang. Independent skill transfer for deep reinforce- ment learning. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 2901-2907, 2021.
[45] C. Salge, C. Glackin, and D. Polani. Empowerment introduction. Guided self-Organization: Inception, pages 67-114, 2014.
[46] A. Sharma, S. Gu, Levine, V. Kumar, and K. Hausman. Dynamics-aware unsupervised discovery of skills. arXiv preprint 190..0997.001655 2019.
[47] J. Liu, H. Shen, D. Wang, Y. Kang, and Q. Tian. Unsupervised domain adaptation with dynamics-aware rewards reinforcement learning. Advances in Neural Information Processing Systems, 34:28784 28797, 2021.
[48] Y. Kang, D. Shi, J. Liu, L. He, and D. Wang. Beyond reward: Offline preference-guided policy optimization. arXiv preprint arXiv:2305.1 00022355.622111 162/7, 2023.
[49] G. Berseth, D. Geng, C. Devin, C. Finn, D. Jayaraman, and S. Levine. Smirl: Surprise minimizing rl in dynamic environments. arXiv preprint 1rX 2019.
11

---

[50] Q. Tian, J. Liu, G. Wang, and D. Wang. Unsupervised discovery of transitional skills for deep reinforcement learning. In International Joint Conference on Neural Networks (IJCNN), pages 1-8. IEEE, 2021.
[51] J. Liu, D. Wang, Q. Tian, and Z. Chen. Learn goal-conditioned policy with intrinsic motiva- tion for deep reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7558-7566, 2022.
[52] D. Kingma and M. Welling. Auto-encoding variational bayes, 2022.
[53] I. Kostrikov, A. Nair, and Levine. Offline reinforcement learning with implicit q-learning,
2021.
[54] Y. Luo, Z. Jiang, S. Cohen, E. Grefenstette, and M. P. Deisenroth. Optimal transport for offline imitation learning. arXiv preprint arXiv:2303. 13977, 2023.
[55] S. Reddy, A. D. Dragan, and S. Levine. Sqil: Imitation learning via reinforcement learning with sparse rewards. arXiv preprint arXiv:1905.11108, 2011..
[56] S. Fujimoto and S. Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132 14:20132-20115, 2021.
[57] K. Zolna, A. Novikov, K. Konyushkova, C. Gulcehre, Z. Wang. Y. Aytar, M. Denil, N. de Freitas, and S. Reed. Offlin learning from demonstrations and unlabeled experience. arXiv preprint arXiv:2011.13882 2020.
[58] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028-400 -01228 2021.
[59] G.-H. Kim, Sco, J. Lee, W. Jeon, H. Hwang, Yang, and K.-E. Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In International Conference on Learning Representations, 2022.
[60] Y. J. Ma, A. Shen, D. Jayaraman, and Bastani. Smodice: Versatile offline imitation learning state occupancy matching. arXiv e-prints, pages arXiv-2202, 2022.
[61] T. Franzmeyer, P. Torr, and J. F. Henriques. Learn what matters: cross-domain imitation learning with task-relevant embeddings. Advances in Neural Information Processing Systems, 35:26283-2666 26294, 2022.
[62] J. Liu, H. Zhang, and D). Wang. Dara: Dynamics-aware reward augmentation in offline reinforcement learning, arXiv preprint arXiv:2203.06662 2022.
[63] Wu, H. Wu, 7. Qiu, J. Wang, and M. Long. Supported policy optimization for offline reinforcement learning, 2022.
12

---

7
Additional Results
balichrtall-mddimee
bappe-mmlien-/
CLuE
CLEE
cltE
cltl
Dermatteed soar
Darmattioo scare
tarmalled soaee
tarnalled sorr
19
10
10
15
18
15
30
the amount of cfffiiie data 1% (HRR.)
the amoonn of dfffiine data (% (4RE.)
the smeunt of offlino data (% (4R1))
the smoonnt of offinn dataa (% (ARRE)
Figure 6: Ablating the number of unlabeled trajectories. We investigate the effect of unlabeled trajectories on the performance. CLUE's performance generally outperforms OTR. Further, we can that CLUE approximates the vanilla IQL method (with D4RL rewards) more closely and can even outperform IQL given such a lack of offline data (S 25%).
Varying the amount of unlabeled offiine data. Here we vary the amount unlabeled offline data available for sparse-reward settings. Figure shows that adding more unlabeled data improves the performance of both CLUE and OTR. However, across range of offline imitation tasks, CLUE shows better performance compared to OTR. We also plot the performance curve of naive IQL with (reward-labeled) offline data in Figure 6. We can see that with extremely limited offline data (5 25%), CLUE approaches IQL's performance more closely on the salfcheetah-mediinee task, and can even outperform IQL on the remaining three tasks.
Table 5: Using 10% D4RL data, normalized scores (mean and standard deviation) of CLUE and baselines antmaze tasks using one (K=1) and ten (K=10) expert demonstrations. The expert trajectories are picked from the chosen 10% dataset. The highest score each setting is highlighted.
Dataset
IQL
OTR (K=1)
CLUE (K=1)
OTR (K=10)
CLUE (K=10)
umaze
73.7
75.4 6.1
75.1 8.3
82.5 2.1
umaze-diverse
21.6
33.0
45.4 10,4
30.8 13.5'
58.6
medium-play
23.0
38.7
33.9
37.3 10.0
36.6 12.7
rediiue-dveeree
5.89
60.9 8.7
6..4
59.2
57.8 87.8
large-play
5.8
8.40
12.0
13.9
29.4
large-diverse
3.6
3.3 3.6
a 1.5
9,0
9.7 4.5
antmaze-v2 total
186.0
222.3
228.6
225.3
274.6
Only two successful trajectories are the chosen sub-dataset and the results belong K=2.
Varying the number of expert trajectories. Using 10% of D4RL data, we vary the number of expert trajectories for sparse-reward reward offline RL settings in Table 5. We compare method with baseline methods (IQL and OTR) when only expert trajectory is selected. For comparison, train IQL over the naive rpware D4RL data and train OTR over the relabeled D4RL dataset (using optimal transport to compute intrinsic rewards and employing IQL to learn offline RL policy). We can find that in out of 12 AntMaze tasks across, our CLUE outperforms the baseline OTR. Meanwhile, compared to naive IQL. (with sparse rewards), our CLUE implementation generally outperforms better than IQL. This means that with only single expert trajectory, we can completely replace the sparse rewards with intrinsic reward in offline RL tasks, which can even achieve higher performance in such data-scarce scenario (10% of D4RL data).
Varying the value of the temperature factor in intrinsic rewards. In Tables 6 and 7, we present the results on AntMaze tasks when we vary the value the temperature factor e in intrinsic rewards. We can find that CLUE can generally achieve robust performance across range of temperature factors. In Figure that CLUE's 's reward prediction shows stronger correlation with the ground-truth rewards from the dataset, which can be served good reward proxy for downstream offline RL algorithms.
further analyze our intrinsic reward distribution following OTR. We can find
13

---

Table 6: Normalized scores (mean) when varying the temperature factor e with single expert
trajectory (K=1).
6
9
10
umaze
89.4
89.96
91.84
90.88
91.96
92.12
91.68
90.72
90.92
91.2
umaze-diverse
43.08
46.76
43.16
43.76
42.36
56.72
52.6
59.04
66.48
68
medium-play play
60.4
63.2
65.2
68.92
68.04
75.32
71.76
74.12
72.2
73.64
medium-diverss
57.8
63.28
63.24
62.04
66.04
70.12
74.56
69.4
72.92
large-play
34.16
44.84
46.88
50.68
52.72
$3.08
53.64
55.2
53.52
55.8
large-diverss
27.04
33.96
43.16
46.8
44.88
47.44
47.44
49.92
47.28
47.11
Table 7: Normalized scores (mean) when varying the temperature factor with 10 expert trajectories (K=10).
c
c
c
c=
c=
6
c= 7
c=
C 9
c= 10
87.88
91.08
90.96
91.16
91
89.92
89.44
90.72
91.92
umaze-diverse
45.64
40.32
41.04
39.52
51.64
57.11
71.68
medium-plas 58.72
64.2
68.24
71.44
69.92
75.56
74.12
76.2
75.8
76.48
medium-diverse
60.36
57.04
62.12
64.24
63.56
61.44
62.36
64.64
65.47
69.2
large-play
48.24
45.8
51.56
48.2
48.4
52.36
49.91
50.58
52.28
51.87
large-diverse
36.32
46.08
48.64
50.84
51.16
52.44
53.6
50.92
51.4
53.68
8 Experimental Details
8.1 Hyperparameters for CVAE Implementation
We list the hyperparameters used for training CVAE models in MuJoCO locomotion, AntMaze, and Adroit tasks. The other CVAE hyperparameters are kept the same as those used Wu et [63].
Table 8: Hyperparameters for training CVAE.
MuJoCo Locomotion
Antmaze
Adroit
full-data
partial-data
full-data
partial-data
full-data
Hidden dim
128
128
512
512
128
Batch size
128
128
256
256
128
Numbers iterations
10a
10a
10a
10a
10
Learning rate
10
10-4
10-3
10
10-4
Weight for Ceatity
0.1
0.1
0.8
0.8
0.1
Spare-reward setting:
Number of expert trajectories
3
3
8.2 Hyperparameters for our IQL Implementation
The IQL hyperparameters employed in this paper are consistent with those utilized by Kostrikov et al. [53] in their offline implementation. It is important to note that IQL incorporates a procedure for rescaling rewards within the dataset, which allows for the use of the same hyperparameters across datasets that differ in quality. As CLUE generates rewards offline, we similarly apply reward scaling following the IQL methodology. For the locomotion, adroit, and ant tasks, we rescale rewards To regularize the policy network for the chosen sub-dataset, we similarly introduce Dropout with rate of 0.2.
with max return min return
1000
MuJoCo locomotion and Adroit tasks. We set the learning rate 10-3 for hopper-medium-exper dataset (K=10) and 3 x 10-4 for the rest of tasks. We run IQL for 1M gradient steps and average mean returns over 10 random seeds and 10 evaluation trajectories for each seed.
Antmaze tasks. We set the learning rate 5 10 for umaze-din diverre dataset (K=1 and K=10) and 3 x 1 for the rest of tasks. For medium- play dataset (K=1 and K=10), medium ciddiii-diverse dataset
14

---

halfchetall-mediinn-222
trajectory return after labeling
hopper-mediem--2
trajectory return after luhheling
aatunazelargg-ddeere--2
32-
550
counts (x 10000)
28-
450
-200
24-
20-
350
-400
16-
250
150
-500
-1000
0.0
0.2
0.4
0.6
0.8
1.0
900
1900
1500
2009 1500 3000
distribution lubeled scures
trajectory return D4RI. data
trajectory return D4RL duta
Figure 7: Qualitative comparison of learned intrinsic rewards with different temperature factors.
(K=1), and large-play dataset (K=10), we set the dropout rate 0.2 to gain a better performance. We rum IQL for 1M gradient steps for the full dataset and 0.3M for the partial dataset, respectively.
8.3 Hyperparameters in K-means
We use CLUE to learn diversity skills on Anr-v2, HalfCheetah--2,, and Walker2d-v2. The K-means, an unsupervised learning method, is employed to cluster the offline transitions {(s, 0, 6')} from each dataset into 100 classes and take each class separate "expert". Specifically, use KMEANS method exacted from sklearn.cluster API. The hyperparameters are set as follows: n_clusters 100, random state
1,n n init 1, max iter 300.
8.4 Offline IL Baselines
SQIL, proposes learn soft Q-function where the reward labels for the expert transitions are one and the reward labels for the non-expert transitions are zero. The offline implementation of SQIL is adapted from the online SAC agent provided by Garg et al. [58]. and combine it with TD3+BC.
IQ-Learn advocates for directly learning a Q-function by contrasting the expert data with the data collected in the replay buffer, thus avoiding the intermediate step of reward learning. In our experiments, we used the official Py PyTorc implementation with the recommended configuration by Garg et al. [58].
ORIL assumes the offline dataset is mixture of both optimal and suboptimal data and learns discriminator to distinguish between them. Then, the output of the discriminator is used as the reward label to optimize the offline policy toward expert behaviors. We borrowed the TD3+BC implementation reproduced by Ma et al. [60] in our experiments.
VIIE is the earliest DICE-based IL algorithm that minimizes the divergence of the state-action distribution between the learning policy and the expert data. The code used in the experiments is the official TensorFlow implementation' released by Kostrikov et al. [37].
DemoDICE proposes to optimize the policy via a state-action distribution matching objective with an extra offline regularization term. We report the performance of DemoDICE using the TensorFlow implementation" by Kim al. [59], while the hyperparameters are same as the ones in the paper.
SMODICE aims to solve the problem of learning from observation and thus proposes to minimize the divergence of state distribution. Besides, Ma et al. [60] extends the choice of divergence so that
"hapss//gttubb.cm////99999000000
dice
15

---

the agent is more generalized. The code and configuration used in our experiments are from the official repository*.
In What Cases Should We Expect CLUE to Help vs to Hurt?
the distribution the expert data is unimodal, our method can learn effectively while providing effective intrinsic rewards. On the contrary, if the distribution the expert data explicitly binding the embeddings of the expert data together would instead affect the learning of z, thus resulting in ineffective intrinsic reward for policy learning.
10 Learned Diverse Skills
To encourage diverse skills from reward-free offline data, we cluster the offline transitions into 100 classes using K-means and take each class separate "expert". Then, these expert data from different classes to label the original reward-free data and train IQL. policy to learn the corresponding skills. In this section, we illustrate all the learned skills by CLUE.
16

---

10.1 Learned Diverse Skills from Ant-Medium Dataset
ant-medium-v2
PCAS
PCA1
13.74
Figure 8: Visualization of unsupervised skills learned from the ant- ant-medium dataset.
17

---

10.2 Learned Diverse Skills from Ant-Randon Dataset
ant-random-v2
PCA2
PCA1
nm
M0-20 eXa
46.25
Figure 9: Visualization of unsupervised skills learned from the ant- ant-random dataset.
18

---

10.3 Learned Diverse Skills from Halfcheetah-Meddium Dataset
halfchectal-mediim-22
PCA2
PCA1
11.75
Figure 10: Visualization of unsupervised skills learned from the halfcheetah-medine dataset.
19

---

10.4 Learned Diverse Skills from Saalicheetta-Ranndon Dataset
halfcheetab-random-22
PEPOME
PCA2
PO
PCA1
TRR
FORE
RC
PIRE
PROOE
PC PTONT
PORE
PORRPOTR
PC HARE
PT
PORK
POIE
POWERY
PT HABY
HIRE
PC FOME
PEPORPRERICCK
PENNNY
Figure 11: Visualization of unsupervised skills learned from the salfchhetah--anndeee dataset.
20

---

10.5 Learned Diverse Skills from Walker2d-Medium Dataset
walker22dmedium-22
PCA
PCA1
0.89
Figure 12: Visualization of unsupervised skills learned from the walker2d-mediun dataset.
21

---

10.6 Learned Diverse Skills from Walker2d-Random Dataset
walker2d-random-2
PCA2
PCA1
56.25
Figure 13: Visualization of unsupervised skills learned from the walker2d-randon dataset.
22


Reviews:
[{"review_id": "qOvEjzp02wh", "reviewer": "Reviewer_pVh3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes CLUE, a learning framework for labeling intrinsic reward functions to offline transition data. The key idea is to perform an auxiliary task of learning conditional action distributions on all the offline data, and leverage the learned latent embedding space to compute the intrinsic reward. Specifically, the reward is computed as the negative distance between the latent embeddings of the expert data and the query sample. It assigns a larger reward value to a state-action pair that is more similar to the expert data, and vice versa. The paper applies CLUE with an offline RL algorithm to perform sparse-reward offline RL, offline IL, and unsupervised offline RL. The learned policies outperform state-of-the-art baselines in sparse-reward offline RL and offline IL tasks.", "summary_of_recommendation": "Despite some weaknesses, the paper makes a major novel contribution to automatic reward design for offline RL and offline IL. The potential impact of the work is quite high, given the relevance of those problems to robotics as well as the strong empirical results presented in the paper. Therefore, acceptance is recommended.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "* The practical use of unsupervised offline RL remains a bit unclear. How would you determine which specific skill to use, given a task specification?\n* In addition to performing calibration on the expert data, the authors could have also added another loss term on the non-expert data to further separate those embeddings from the expert ones (for instance by encouraging larger stdev for non-expert data). Did you consider doing that?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, CLUE is a strong method for labeling intrinsic rewards, which improves offline RL performance while being able to specify tasks in the form of expert demonstrations. The major strengths are three fold. 1) It nicely leverages the auxiliary task of predicting conditional action distributions, so that the embedding space distance naturally provides a dense intrinsic reward based on similarity to expert demonstrations. The use of CVAE makes the learning easier than adversarial training. 2) Because CLUE can take any data as expert data, the user can repurpose it for different problems, including sparse reward offline RL and offline IL. 3) CLUE can be plugged in to any offline RL algorithm that can take in data with dense reward signals. These strengths are well-demonstrated throughout the paper. The figures and the tables are well-presented and make the paper easy to follow.\n\nOn the other hand, the paper has some weaknesses. 1) Unsupervised offline RL is based on clustering at the transition-level, which leaves out the possibility to cluster long-term behaviors that take more than one transition to complete (for instance, a cyclic behavior that can take 5 transitions to form a gait for the walker). 2) In the experiments section, the OTR baseline is introduced without enough details. I was unable to find the citation that corresponds to it. The authors should clarify the differences between OTR and CLUE, preferably in the Related Work section.", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "YnniYnRQjL", "reviewer": "Reviewer_CYF2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper studies a modification of the offline RL problem statement in which there is a small amount of expert data that can be used to create an intrinsic reward function. The intrinsic reward function is the distance of non-expert data from expert data in the embedding space of a cVAE. The authors apply this approach to improve performance on sparse-reward tasks, offline imitation learning tasks with some expert trajectories labeled, and offline RL assuming no expert trajectories. The datasets for each environment come from D4RL.", "summary_of_recommendation": "I recommend accepting this paper because the idea of deriving methods for multiple offline RL problem statements by aligning latents with expert trajectories is powerful and the claims are well evaluated.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "In equation (1) why is the KL term negative. Following from equation (1) in Kingma et. al. (2022), shouldn't it be: $\\log p_\\theta(x) = KL(q_\\phi(z|x)||p_\\theta(z|x)) + \\mathbb{E}_{q_\\phi(z|x)}[- \\log_q(z|x)+ \\log_p(x,z)]$?\n\nI find the presentation of the 3 problem statements confusing. E.g., this may be a bit pedantic but isn't offline-RL without labels just an IL problem with messy data? Also in most of these cases it seems reasonable to assume that we have access to some offline reward function (even if it's quite naive, such as labeling each trajectory with a sparse reward of 1 at the terminal state). Could simple reward functions such as that be a strong baseline for these tasks?\n\nD. P. Kingma and M. Welling. Auto-encoding variational bayes, 2022.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Clarity: The method is described clearly in section 4. I found the introduction to be too all-encompassing, which made it difficult to understand what the paper actually does. Specifically, paragraphs 2 and 3 cover everything from intrinsic rewards to augmented reward terms in offline RL. Although it was nice to see the connection to multiple problem statements within RL, I think it would make the paper stronger if the motivation for CLUE was described more succinctly and directly.\n\nOriginality: The method is logical and novel. I appreciated how the authors demonstrated how the method could be applied in multiple problem statements (e.g., by clustering the trajectories when no expert data is available).\n\nSignificance: Leveraging a cVAE could provide a new unifying framework for multiple offline sequential learning problem statements, which feels significant.\n\nQuality: Claims are verified against a reasonable collection of relevant baselines (IQL, OTR).", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "JdxcAhRRI8", "reviewer": "Reviewer_GEyR", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes to use a latent variable model to provide auxiliary rewards for offline RL. The method trains a latent variable model on both expert and non-expert data, computes the average latent variable value for expert data, and assigns rewards for other transitions based on the similarity of the inferred latent variable to this average value from expert data. The method is then applied to sparse reward tasks, imitation-style tasks (with demonstration and non-demonstration data), and clustering-based unsupervised RL, with comparisons to other offline RL and offline imitation learning methods.", "summary_of_recommendation": "The paper has an interesting idea, and I would strongly encourage the authors to develop this idea into a great paper.\n\nAs is however, the technical motivation for this idea is lacking, and the method comes across as ad hoc.\n\nThe relevance to robotics is also somewhat weak, as this is purely an RL algorithms paper. The algorithms in question *are* highly relevant for robotics, but the paper could well be submitted to a machine learning venue instead.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "The main weakness with the paper is the lack of any theoretical motivation for the proposed reward term. I would consider raising my score if the authors either provide a more complete justification of their approach, or else rederive the approach to be more principled (which may require rerunning the experiments).\n\nQuestions that would be good to answer with any such derivation include:\n\n1. Does the method converge to the optimal policy with these reward terms under at least strong assumptions (e.g., tabular, perfect LVM, etc.)? Such proofs are easy to derive for conventional intrinsic motivation in online RL for example, or unbiased reward shaping (e.g., as in the Ng '00 paper). Could something like this be shown for the proposed method or a more principled variant of it?\n\n2. Why is Eq (6) or Eq (7) reasonable?\n\n3. In what cases should we expect this method to help vs to hurt?\n\nI would raise my score if the authors convincingly address these issues.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overall, I think the core idea in the paper is neat. I think there is definitely something to the core concept of using density estimation and latent variable models to provide auxiliary rewards or reward shaping for unlabeled data or sparse reward tasks.\n\nOn the other hand, the paper is not well motivated from a technical perspective (see more detailed comments below), and the theoretical treatment of the core concept in the work could be improved significantly.\n\nThe experimental results are thorough. While the results are not overwhelmingly strong, they do suggest that the method works, and the method is compared to a significant number of logical baselines.\n\nIt's not completely clear if the work is a good fit for CoRL, in that it's a core RL algorithms paper without robotic validation.\n\nSpecific comments:\n\nFirst, regarding the technical approach:\n\nEquation (6) is a bit puzzling for me. It states that the intrinsic reward is the exponentiated squared distance (looks like an RBF kernel) to z_e, which is defined as the expected *probability* q(z|s,a) (so the \"type\" of z_e is a probability?). I assume this is a typo, and in fact the authors meant to define z_e as the expected value of z sampled from the posterior q(z|s,a), averaged over the dataset. But this is still a perplexing choice: there is no reason why the average value of z should have any particular meaning -- for example, the expert data could fall into two clusters on opposite sides of the latent space, in which case the average z might be close to zero, not reflecting much information about where the expert behavior tends to lie in the latent space. Perhaps the authors meant to use a reward that roughly estimates how close z(s,a) is to expert data (e.g., something like log E_{(s',a') ~ D^e)}[q(z(s,a)|s',a')]), but the current definition doesn't really make much sense to me.\n\nThe authors instead deal with this issue by adding the somewhat ad-hoc Eq (7) objective term to cause expert embeddings to clump together. Maybe this corresponds to some well-defined probabilistic model (and I can imagine such an explanation might exist and might even provide a better motivation for Eq (6)), but as stated this seems like a rather mysterious and ad hoc decision.\n\nI do think there is probably a much nicer way to motivate the method from the perspective of variational inference, which would provide some better justification for why we should reward the agent for transitions that resemble expert transitions, but as written now the method comes across as mostly a heuristic.\n\nI think this is the biggest issue with the work right now.\n\nOther (more minor) issues:\n\n- The sparse reward offline RL results (which are closest to widely studied benchmark tasks in prior work) show some improvement, but not a very large amount. I don't think this is a major weakness, as the results are good enough, but not overwhelmingly so. I don't think this is a big point against the paper, since it still validates the core scientific idea, but I do wonder if a more principled treatment of the technical method could lead to better results.\n\n- The discussion of intrinsic motivation in the introduction and related work is kind of misleading in my opinion. Intrinsic motivation and intrinsic rewards in online RL are typically used to drive exploration, whereas the use of the reward in this work more closely resembles reward shaping.\n\n- There are some obvious ways of using the expert data that are not compared to, such as naively adding a BC term on the expert data, or something like this paper: https://arxiv.org/abs/2110.14770\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "DjB7A7O0DS", "reviewer": "Reviewer_ashV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a new approach to learning from very few expert demonstrations. Their proposed method estimates intrinsic rewards that are more in line with extrinsic task-specific rewards obtained from expert data. The key idea is to add a calibration regularisation term to CVAE to estimate the intrinsic rewards. This calibration reduces the variances of estimated mean and std in CVAE for expert data. The authors demonstrate that the proposed calibration yields good results in 3 settings of expert data usage in offline RL. ", "summary_of_recommendation": "This paper proposes an effective and intuitive approach to tackle the problem of learning from very few expert demonstrations. Overall the writing is clear and demonstrates strong empirical results. I am leaning towards accepting this paper, as long as the concerns listed above are addressed appropriately. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. In Appendix, it seems that the choice of temperature hyperparameter $c$ could impact the final performance in a non-insignificant way (Table 1 in the main text; Table 2 and Table 3 in Appendix). On a closer look at the 3 tables, it seems that the result from Table 1 in the main text is an ensemble of results trained under different hyperparameters $c$ (i.e., *medium-play* is from $c=6$ while *medium-diverse* is from $c=8$, etc, from table 2 in Appendix). Could the authors clarify? Is this also the case for other results presented?\n\n2. For the 2nd instantiation – The result in Tables 3 and 4 for locomotion tasks hopper/walker2d/half-cheetah with K=1 seems impressive. How about more complicated tasks such as Adroit or Antmaze to further understand the results compared to OTR? Additionally, OTR hyperparameters should be included for completeness. \n\n3. For the 3rd instantiation - What are the baselines for this instantiation of expert data usage? Is there a quantitative way to better understand the results?\n\n4. It is unclear to me how to interpret Figure 2 correctly. Is it generated from CVAE/CLUE trained on the whole expert data $\\mathcal{D^e}$ vs trained on whole non-expert data $\\mathcal{D}$?\n    1. What do the authors think about the result where the CVAE std centre seems to be around $\\sigma = 0.9$ and, after calibration, the std centre seems to shrink towards $\\sigma = 0.96$? This is also the case for non-expert data.\n    2. How good must the expert demonstration be so that the latent embeddings will shrink towards $\\sigma = 1$? \n\n5. Related to question 4, can the method still perform well if, instead of using the \"optimal\" expert demonstration in the dataset, let's say uses the demonstrations in the 90th percentile? How sensible is the proposed method to expert demonstrations compared to OTR? (i.e., how much does a chosen demonstration affect the method?) ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Strength:**\n\n- The paper is generally well written and has a good amount of experiments.\n- The proposed calibration regularisation term is intuitive, novel, easy to implement by itself and should introduce minimal training time overhead.\n- Strong empirical results on the 3 instantiations of expert data in offline RL settings.\n- Experiment results on offline IL tasks versus baseline algorithms (Table 4) are particularly impressive.\n- Supplemented code is appreciated.\n\n\n**Weaknesses:**\n\n- The justification for using CVAE is not well motivated.\n- The proposed calibration regularisation needs to be applied with CVAE together.\n- Since the paper’s claim is that the proposed regularisation term can be combined with any offline RL algorithms, comparisons using other base algorithms would be preferable. Is there a particular reason that IQL was chosen as the base algorithm?\n- The temperature hyperparameter $c$ used in various experiments cannot be found in the texts.\n\n\n **Minor notes:**\n\n- How to apply the proposed calibration regularisation term is not clearly written (i.e., where the loss should be applied). It can be inferred from the texts but can be written more clearly.\n- typo in line 105, I think $\\mu_0$ should be $p_0$?\n- I’m a little bit puzzled by the supplemented code; for CVAE, it seems the code is implemented as a beta-CVAE (with beta=0.5) whereas there is no mention of this in the text.\n- The hyperparameters of OTR and some other baselines (SQIL and ORIL) were not mentioned.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a framework that leverages a few expert data and obtains intrinsic rewards that guide an offline RL agent. The effectiveness of the proposed method was evaluated with sparse-reward offline RL tasks, offline imitation learning tasks, and unsupervised offline RL tasks.

The initial evaluation was mostly positive, and the authors addressed most concerns raised by reviewers in the initial review.
While Reviewer GEyR pointed out that the derivation of the proposed algorithm could be more formal, all reviewers agree that the idea in this paper is interesting and novel.

Based on the assessment from reviewers, AE recommends the acceptance of the paper.
AE strongly encourages the authors to carefully incorporate the points raised by reviewers before submitting the final version. Especially, please check the notation and equations carefully once again.
Decision: Accept (Poster)