Curiosity-Driven Unsupervised Data Collection for Offline Reinforcement Learning
Authors: Chenyu Sun
Hangwei Qian
Chunyan Miao
Venue: Submitted to ICLR 2023 (2023)
Keywords: Offline Reinforcement Learning
Data Collection
Reachability
Unsupervised Learning
Curiosity-Driven Learning

Abstract: In offline reinforcement learning (RL), while the majority of efforts are focusing on engineering sophisticated learning algorithms given a fixed dataset, very few works have been carried out to improve the dataset quality itself. More importantly, it is even challenging to collect a task-agnostic dataset such that the offline RL agent can learn multiple skills from it. In this paper, we propose a Curiosity-driven Unsupervised Data Collection (CUDC) method to improve the data collection process. Specifically, we quantify the agent's internal belief to estimate the probability of the k-step future states being reachable from the current states. Different from existing approaches that implicitly assume limited feature space with fixed environment steps, CUDC is capable of adapting the number of environment steps to explore. Thus, the feature representation can be substantially diversified with the dynamics information. With this adaptive reachability mechanism in place, the agent can navigate itself to collect higher-quality data with curiosity. Empirically, CUDC surpasses existing unsupervised methods in sample efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.

Under review as a conference paper at ICLR 2023
CURIOSITY-DRIVEN UNSUPERVISED DATA COLLEC- TION FOR OFFLINE REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In offline reinforcement learning (RL), while the majority of efforts are focusing on engineering sophisticated learning algorithms given fixed dataset, very few works have been carried out to improve the dataset quality itself. More impor- tantly, it even challenging to collect task-agnostic dataset such that the offline RL agent can learn multiple skills from it. In this paper, propose a Curiosity- driven Unsupervised Data Collection (CUDC) method to improve the data collec- tion process. Specifically, probability of the k-step future states being reachable from the current states. Dif- ferent from existing approaches that implicitly assume limited feature space with a fixed temporal distance between current and next states, CUDC is capable of adapting how many steps into the future that the dynamics model should predict. Thus, the feature representation can be diversified with the dynamics information. With this adaptive reachability mechanism in place, the agent can navigate itself collect higher-qualijy data with curiosity. Empirically, CUDC surpasses existing unsupervised methods in sample efficiency and learning performance in various downstream offline RL tasks of the DeepMind control suite.
quantify the agent's internal belief to estimate the
INTRODUCTION
Deep reinforcement learning has demonstrated remarkable breakthroughs in games, robotics, and navigation in complex environments (Kiran et al.) [202]] [Singh et al.) [202)] (Snn et al. 2022a). For
online RI., agents constantly update the policy to acquire different skills through active interactions with the environments. However, online RI. impractical in many real-world environments as direct interactions with the environments might be expensive or dangerous (Kiran et al.) [021] [Singh cct al. 2022). In recent years, offline RL has become a promising research area to cope with limited interactions, where agents learn policy exclusively from previously-collected experiences stored
in fixed dataset (Levine et al. 2020 Kostrikov et al. 2021 Fujimoto & Gu (021).
In view of the growing popularity of offline RL, the majority of current research focuses model- centric practices by successively developing new algorithms (Kumar et al. 2020 |020:Danner Janner et al. [02T]] Matsushima al., 2022 Emmons al., 2022; Kumar et al.) 2022). Despite the rapid progress in these algorithmic advances, their performances are inevitably limited by the quality of the pre- collected dataset itself. Recently, the concept of data-centric approaches has become critical in the machine learning community, emphasizing the importance of improving the training data quality over algorithmic advances (gg 202T, Motamedi al. 202T Patel al., (022). Motivated by this, the offline RL research community attempts to eye on ways of engineering the training data (Pru- dencio et al., (022). To focus on more useful data, one solution is to exploit the sample importance with sampling (Zhang et al. (2200) re- re-weighting (Wu et al. 202T). Different from this approach, we aim to collect a higher-quality dataset that can be directly used for offline RI. agents.
More importantly, it is even more desirable yet challenging to collect a task-agnostic dataset such that offline RL agents are able to extract effective policies for multiple downstream tasks. To analyze and understand these challenges, ExORL (Yarats et al.) 2022) empirically shows that unsupervised RL methods are superior to supervised methods to collect the exploratory data that allows even vanilla off- policy RL algorithms effectively learn offline and acquire different skills. Nevertheless, these existing methods pre-define a fixed temporal distance between current states and future states to train the models, which implicitly limits the diversity in the learned feature representation.

---

Under review as a conference paper at ICLR 2023
As we observe that fixing the temporal distance between current and future states may limit the feature space and result in low-quality dataset, it is desired to enhance feature representation by ex- ploiting the reachability from current state to more distant future states. Although existing works of reachability analysis have been introduced in RI. (Savinov et al., 2019, Péré et al. 2018: Ivanovic et al. 2019 Yu et al. 2022), these approaches are however not directly applicable. For example, Savinov et al. (2019) only considers the reachability in a binary case, and it requires extensive com- parisons to the stored embeddings in memory. Moreover, the reachability in goal space exploration requires kernel density estimation, increasing the computational cost substantially. Different from these approaches, we propose Curiosity-driver Unsupervised Data Collection (CUDC) method with a novel reachability module. Inspired by the fact that human curiosity can enhance the learn- ing process by motivating human beings on the novel knowledge that is beyond ones' perception (Rossing & Long 1981, Markey & Loewenstein] 2014] [Sun et al., 2022b). CUDC facilitates the agent to collect dataset curiously without any task-specific reward. In particular, we define the reachability module to characterize the probability of k-step future state being reachable from the current state, with no episodic memory or feature space density modeling required. This module allows the agent to automatically determine how many steps into the future that the dynamics model should predict, where the learned feature representation can be incorporated with the information of dynamics. Compared with the existing unsupervised methods, it avoids relying on the fixed feature space by gradually expanding to more distant future states. With the enhanced representation learn- ing, a mixed intrinsic reward encourages curious exploration towards more meaningful state-action space as well as the under-learned states. As a result, the collected dataset can lead to improved sample efficiency and better performances in downstream offline RL. tasks.
Our contributions can be summarized as follows. 1) We are the first to introduce reachability for improving data collection in offline RL, which is defined in a more efficient way and can enable the agent to navigate curiosity-driven learning coherently. 2) We empirically show that adapting the number of steps between current and future states to perform increasingly challenging prediction can enhance feature representation with information of the dynamics, thereby improving the collected dataset quality. 3) With the learned state and action representations, CUDC additionally incentivizes the agent to explore diverse state-action space well as the under-learned states with high prediction errors through a mixed intrinsic reward and regularization. 4) Under the EXORL. (Yarats et al.) 2022) setting, CUDC outperforms the other unsupervised methods to collect the dataset that can be learned offline in multiple downstream tasks of the DeepMind control suite
(Tasaa et al. 2018).
2 RELATED WORKS
Reachability in RL Savinov et al. (0199) constructed a reachability network estimate how many environment steps to take for reaching a particular state. It intrinsically rewards the agent to explore the state that is unreachable i.e., takes more than a fixed threshold step, from other states in memory. However, it only takes the binary case of reachability into consideration and is quite inefficient when comparing the similarity with all stored states in memory. In the goal exploration tasks, Péré et al. (2018) defined the reachability of goal with an estimated density and proposed to sample increasingly difficult goals to reach during exploration. Although the goal space can be learned in unsupervised manner other than in specifically engineered way, its sampling process requires a kernel density estimator, increasing the computational cost substantially. Following the similar idea, BARC (Ivanovic et al.] 2019) adapts the initial state distribution gradually from easy- to-reach to challenging-to-rean
goals. As a result, agents can perform well even in a hard robotic control task. Recently, RCRL (Yu et al. (002) has shown that leveraging reachability analysis (Hsu let al. 2021) can help learn an optimal safe policy by expanding the limited conservative feasible set to the largest feasible set of the state space.
-uriee RL Curiosity-driven RL. intrinsically encourages agents to explore the task en- vironment in a human-like way, which is of vital importance when the task-specific rewards are sparse or absent Aubret et al. 2019 Sun et al.) 2022b). The main type of curiosi RL is to incorporate an intrinsic reward that self motivatts agents to explore based on various aspects of the state, such novelty (Bellemare et al. 2016), entropy (Seo et al. 2021 Liu & Abbeel 20211), reachability (Savinov et al. 2211)), prediction errors (Pathak et al. [017]] Berseth et al. (2020), com-
plexity (Campero al. 2020), and uncertainty (Pathak et al. 2019 Sekar al. 2020 Li et al.
2

---

Under review as a conference paper at ICLR 2023
2021].
Another type of curiosity-driven RL is to prioritize the experience replay towards under- explored states (Schaul et al. 2016: Zhao & Tresp 2019: Brittain et al. 2019 Jiang et al.) 2021). However, the curiosity mechanism can be further exploited and introduced into other components of RL for exploration, as shown in CCLF (Sun et al. 2022a). Therefore, we propose to curiously adapt the temporal distance to explore more distant future states, enhancing the learned representation space with dynamics information. Mcanwhile, our method regularizes the Q-learning by assigning the importance weights through a curiosity weight to focus more on under-learned tuples.
Unsupervised Data Collection EXORL Yaratt et al. 2022) evaluates 9 unsupervised data col- lection algorithms, demonstrating a superior capabilitt over supervised methods for multi-task of- fline learning. In particular, knowledge-drive models of ICM (Pathak et al.) 2017). Disagreement (Pathak al. 2019). and RND (Burda et al.) 2019) encourage agents explore by maximizing the prediction errors of the states. Data-driven models of APT (Liu & Abbeel 2021b) and ProtoRL. (aarats al. 202T) incentivize to uniformly explore the entire state space. By leveraging some prior information, competence- based models of DIAYN (Eysenbach et al. 2019), SMM (Lee et al.) 2019) and APS (Liu & Abbeel, (02 encourage agents to learn diverse skills. However, all these meth- ods are not tailored for data collection; instead, they were originally proposed for online pretraining and fine-turme tine-tunning at task learning process as evaluated in the URLB benchmark (Laskin et al. 2021).
Moreover, their feature space limited by relying rent state and future state for model training. Concurrently, Explore20ffline (Lambert et al.] 2022) leverages the intrinsic model predictive control to simulate trajectories and their resulting predicted intrinsic rewards, but it does not consider the sample importance while updating the policy.
fixed temporal distance i between the cur-
3
CURIOSITY-DRIVEN UNSUPERVISED DATA COLLECTION (CUDC)
3.1 PROBLEM SETTING
We consider the problem of multi-task offline learning with three main steps of data collection, reward relabeling, and downstream offline learning as described in both ExORL[(Yarats et al., 2022)
and Explore20ffline (Lambert et al., 2022). In the data collection phase, the exploratory agent (data collector) has the access to Markov Decision Process (MDP) environment with a state of the environment 8 E S, an action E AA based on policy T(s). transition probability p(s'| [is"|ss,0 a) mapping from the current state 8 and action a to the next state s', a reward r, and discount factor y E 00, 1) weighting future rewards. In particular, the exploratory agent online interacts with the environment and stores the unlabeled tuples (s, o, s') in the dataset D. The second phase is to relabel the collected dataset DD using the given reward function about the downstream task r(s, a) for each tuple. It transfers information from task-agnostic exploration to downstream tasks. The last step to perform multiple downstream tasks with an offline RL agent on the labeled dataset. In this paper, we focus on the most challenging part of this problem setting, which is the task-agnostic data collection and we evaluate the quality of the collected dataset in multiple downstream tasks.
Co lect New Date
Anter's Inter nal Edief (feaetsttilll Morkiw))
Atapi Dimppns
Dirinta We
Dument Statei
Cooder
Iepurrzton
Feruerd Deariaa,,
Fedladd f...
Mird aetsi
Cartorty Daage
Fasctrale Vatsoe
lepeoratin...
Explore
Regulavios
Translion Tasie
Figure 1: Curiosity-driven Unsupervised Data Collection (CUDC): The diagram on the left illustrates the framework of CUDC. A batch of transition tuples is sampled from the accumulated dataset, and then the reachability between the k-step future and current states is measured by agent's internal belief. The reachability module outputs the mixed intrinsic reward to encourage diverse exploration. Meanwhile, it also outputs a curiosity weight to adapt the temporal distance and regularize the backbone RL. Therefore, the agent can collect new data until the capacity is reached. The diagram on the right shows how the agent measures and updates its internal belief on the probability of the k-step future states being reachable from the current states.
3

---

Under review as a conference paper at ICLR 2023
3.2 FRAMEWORK OVERVIEW
In this section, we propose Curiosity-driver Unsupervised Data Collection (CUDC) method, which employs DDPG (Liliicrap et al. 2015) as the base RL algorithm for the exploratory agent. As shown in Figure (right), a novel reachability module is constructed measure the probability of k-step future state being reachable from the current state. With this module in place, the exploratory agent is encouraged to diversely explore by a mixed intrinsic reward, and meanwhile regularize the critic-actor update to focus more on under-learned tuples. Most importantly, the temporal distance of step between current and future states is adaptively increased to incorporate the dynamies infor- mation in the learned feature representation. As a result, the exploration and data collection become more diverse. More details are listed in Algorithm
3.3 THE REACHABILITY MODULE
In EXORL (arats et al. 2022). the existing unsupervised methods assume the limited feature space by implicitly fixing the temporal distance k between current and future states to train the model. To diversity the feature representation, a natural choice is to employ the reachability analysis that can adjust & during exploration. However, the existing implementations of reachability are not desired due to the limited capability by binary classification of reachable states (Savinov et al. 2019). or costly density estimation of goal space (Péré et al. 2018). As these approaches are not practical in the offline RI. setting, we define the reachability in CUDC by the self-supervised estimation on the probability of k-step future state St,+k being reachable from the current state St,, with no expensive density estimation or manual labelling required. As result, the proposed method can enhance feature representation when expanding the feature space through an adaptive k-step. In fact, this motivation has been shown effective in the other works of reachability, such as constrained RL (Yu et al., (022) and robotics (Ivanovic et al. 2019).
Given a batch of unlabeled tuples (st,, at,, St. )) we first encode the state features 2s,, = (st.). Z0e,+* = (s(++t))) and action feature 2as; = Φa (at,)) with a state encoder (-) and an action encoder (.), and perform the one-hot encoding for the temporal distance h. To facilitate the reachability analysis, a forward dynamic network e = fe(Ze., Za., , k; 0,) ). is constructed, which takes the inputs of Ext,, .aa,. , and the encoded k to predict the future state feature Exc,tu fully exploiting the information of dynamics. can be updated by an lg norm loss of ||= In 1
In addition, we want to enforce Ex,+, to match with its own Σs, as much as possible, while keep- ing apart from the other future states within the same batch. This contrastive intuition is that each future state should be most reachable from its own current state, which can quantify the reachabil- ity in a simple and efficient way. In fact, the self-supervised contrastive learning has been recently introduced RL and is shown to be capable of learning rich representation with more semantic estimate the latents (Srinivas et al. 2020: Liu & Abbeel 2021b). Following this intuition, probability of st,+k being reachable from St, by
exp(h((ee ) Whh(Ze
4 exp(h(Ea,,,a) ) (W + Σj exp(h((an, STWT (E
where is the batch size, h((-) is deterministic projection function, W is a hidden weight to compute the similarity between the two projections, and h(()) as well as 2(-) are respectively the momentum soanentum-based moving average of the projection and state feature to ensure consistency and sta- bility (He et al. 2020). In this way, this reachability network can be updated by the contrastive loss of Creach
Di= log 4 in self-supervised way without maunal labeling.
To further improve the representation learning, the reachability module also includes two inverse models for predicting action feature E.,, and temporal distance k. Similar to ICM (Pathak et al. 2017] and Disagreement (Pathak et al. 2019), we define بي, = fa(zs,, Res.te k; 0.) with a back- ward loss of ||zar, - La,, l|,, which ensures the encoded features are robust to environment variations that are uncontrollable by the agent. For the inverse model of the k-step, & fx fx + 0k) characterizes the prediction with a distribution P(k). It is updated through a cross-entropy loss, which enables the encoders to capture the dynamics information in the encoded features.
4

---

Under review as a conference paper at ICLR 2023
The agent can therefore update its internal belief in self-superviser manner without any expensive labeling process compared to supervised learning. More importantly, we can allow the temporal distance of k-step to adapt during learning, rather than relying existing unsupervised methods. It is of vital importance as the feature representations of both states and actions become more informative and robust when adjusting the temporal distance of k-step. Meanwhile, this module is also capable of computing a curiosity weight wy for each tuple i as
fixed hyperparameter in many
w; = 1, E [0, 1].
(2)
Intuitively, a large value of w, means that the agent mistakenly believes the true future state is unreachable from the current state, which induces high curiosity due to the conflict with agent's current internal belief. It further indicates that this under-learned transition tuple contains novel information, and the encoders are not capable of extracting meaningful features yet. With this reachability module in place, we can enable the agent to perform the task-agnostic dataset collection in curious manner. Detailed curiosity-driven learning is illustrated in the next subsection.
3.4 CURIOSITY- DRIVEN LEARNING
It should be noted that reachability in previous works is only incorporated encourage diverse exploration. contrast, our proposed CUDC extends the reachability to more stages of learning. Firstly, it adapts the temporal distance, i.e. k-step to explore, enhancing feature representation with the dynamics information of more distant future states. Secondly, it incorporates mixed intrinsic reward encourage effective exploration with the expanded feature space. Lastly, it regularizes the critic-actor update for the DDPG algorithm by utilizing the curiosity weights to focus more on under-learned tuples. Different from the existing methods evaluated in ExORL that only utilize intrinsic rewards as curiosity, our CUDC extends curiosity driven learning to different RL components, improving the data collection process coherently.
intrinsic reward to
3.4.1 ENHANCE FEATURE REPRESENTATION WITH ADAPTIVE S-STEP
worth noting that all methods discussed in EXORI. limit the autonomy of feature space by expecting the agent to reach the future states in exactly 3 steps. Recent works on online pre-training RL such as SPR (Schwarzer et al. 2020) and SGI (Schwarzer et al.] (022I) predict agent's ' own latent state representations multiple steps into the future, addressing the challenge of sample efficiency. However, this approach requires iterative predictions by calling the forward dynamic network for times. In contrast, CUDC can automatically adjust the temporal distance k to directly perform the k-step future state estimation in the proposed reachability module, without increasing the computa- tional complexity substantially. To expand the feature space by leveraging the dynamics information, our main idea is to keep the reachability estimation increasingly challenging with an adaptive k-step-
In particular, agent is not curious enough for the current reachability analysis. For this purpose, we set a threshold C., for low curiosity and a threshold Cj for the proportion of tuples with low curiosity. Thus, the agent will adapt i when Σ. [[w;<C=]
adaptively adjust the h by encouraging curiosity, i.e., & is increased by if the
> Ck- The intuition is that the agent should be encouraged to explore further based on the reachability, when they are no longer curious about the majority of tuples within the same batch. As a result, the feature representation can be enhanced by learning the dynamics of more distant future states. With the expanded feature space, the data collection performed towards more informative and diverse features. It should be noted that there are many possible ways to vary the k-step such as by sampling from a probabilistic distribution. For this reason, an ablation study is also conducted to validate the effectiveness of the proposed curiosity- driven method, compared to other sampling- based methods.
3.4.2 INCORPORATE MIXED INTRINSIC REWARD
CUDC employs a mixed intrinsic reward of state-action entropy and prediction error of future states. On one hand, the existing methods of APT (Liu & Abbeel bb2 and RE3 (Seo et al. 2021) have shown that agents can be encouraged to explore the state space more uniformly, by utilizing the particle-based k-nearest neighbors state entropy as intrinsic reward. This is consistent with the entropy maximization principle (Beirlant et al. 1997, [Singh et al. 2003). However, we believe that the agent should diversely explore not only the state space, but also the action space. Therefore,
5

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Implementation of the proposed CUDC
Initialize parameters of encoders @₂ and das forward dynamic network fa. inverse models fa and fa. projection ht critic Q, policy и, hidden weight W, and the temporal distance k
Initialize the task-agnostic dataset D
for each time step t do // COLLECT TRANSITIONS
Interact with the environment using the policy at ~ (8t) and observe 81+1
DU 0t, 8141) D
// UPDATE INTERNAL BELIEF OF REACHABILITY
Sample a minibatch {sL,, dt,, &t, k) D
for each tuple in the minibatch do
Encode the state and action, and predict the t + k's future state feature Bea Evaluate the curiosity weight w; according to Equation (2)
Compute the intrinsic reward ry using w; by Equation
end for
Update the internal belief of reachability module WADAPT THE K-STEP TO PREDICT
if 1 E. [[w < Cw. > Cj then
Increase the temporal distance k = le end if
/RRGULLAIIEE CRITIC-ACTOR UPDATE Update the critic Q with regularization by Equation (4)
Update the actor и with regularization Perform the momentum update for h and z
end for
CUDC extends state embedding to state-action embedding and we prove that the k-nearest neighbor entropy estimation in the state-action representation space can be used for entropy maximization.
Lemma 3.1. Let u = (Ze, zu) represent the state-action representation. The particle-baser entropy H((u) proportional K-nearest neighbor (K-NN) distance,
H(u) log |mu us
K-NN
22
Following the same idea to treat each tuple as particle (Liu & Abbeel 20216
Sco et al.
2021 we formulate an intrinsic reward to estimate the particle-based entropy, I'H (8 de, logl NK - -uK-NN +1 where = (*,,) (0r()) and Nx is the number of K-NN. As the encoded features are constantly updated to capture the dynamics of more distant future states in the reachability module, the proposed n'H can lead more diverse acte-action space exploration, where the state-of-the-art off-policy RL algorithm SAC (Haarnoja et al. 2018) has also shown that exploration through discounted maximum-poli (actor)-entropy term is more effective.
On the other hand, the prediction errors of future states are often employed as the intrinsic re- ward to incentivize the agent to explore surprising states that are beyond their exceptions (Pathak) ct al. 2017: Burda et al.) 2018). CUDC additionally integrates the prediction eиror (st ar,) Pss,- 2s,,+ull Il in the intrinsic reward as well, where the reachability module is conveniently re-used without additional networks. Finally, the mixed intrinsic reward in CUDC is given by
ri(St, ar,) = rH ar.)) + or e dc,) + 3,
where a controls the prioritization of under-learned state exploration and / is constant for numer- ical stability.
3.4.3 REGULARIZE THE CRITIC-ACTOR UPDATE
Furthermore, CUDC adaptively regularizes the backbone DDPG algorithm to focus more under- learned tuples, by utilizing the curiosity weight w,. In particular, w = (w,, Wy. w,,) quan- titatively describes the curiosity weight each transition tuple, which can be seamlessly used to

---

Under review as a conference paper at ICLR 2023
characterize the sample importance and regularize both critic and actor updates. Therefore, we can perform the Q-learning in DDPG by minimizing the following objective,
(4) Meanwhile, the policy can be updated by maximizing E "') [ 7 [s))]. In this way, CUDC facilitates the agent to adapt the learning process in self supervid manner in the sense that the regularization is controlled by the conceptualized curiosity to exploit sample importance.
E )( w (Q(s, a) (r.(s.a) + Vurgal (s', n <
EXPERIMENTAL RESULTS
Environments We evaluate the proposed CUDC on continuous control tasks with state observa- tions in the DeepMind control suite (Tassa et al.] 2018), which consists of 12 downstream tasks across 3 main challenging domains: Walker, Quadruped, and Jaco Arm. Walker is controllable entity with locomotion-related balancing controls, where it can learn walking, running, flipping and standing. Quadruped is a passively stable body in more challenging 3D environment to learn vari- ous locomotion skills of walking, running, standing, and jumping. Jaco Arm is 6 degree-of- degrec-of-freedom freedom robotic arm with frree-fingrr gripper for object manipulation, where the downstream tasks re- quire it to reach different positions. Note that PointMass Maze is not included since most baseline methods in EXORL have already demonstrated excellent performances.
Baseline Models We compare CUDC against state-of-the the- unsupervised methods across all three categories as benchmarked in ExORL, i.c., a knowledge-driven baseline of ICM (Pathak et al. 2017], data-driven baselines of APT (Liu & Abbeel, 202 and ProtoRI. (Yarats et al. (021), )) and a competence-drrvve baseline of APS (Liu & Abbeel 2021a). Meanwhile, random data collector is also included, which collects the data by performing randomly sampled actions. The other four methods discussed in EXORL are excluded since their performance are less competitive. We set the same hyperparameters and model architecture as reported in ExORL. To demonstrate that all proposed components play important roles in the performance, we also compare four versions of CUDC as follows. CUDC!C vary and CUDCAPT: trinsic rewards based on the original ICM and APT methods. CUDCreward: extending state-action entropy with mixed intrinsic reward based on CUDCAPT CUDCreachi adding the full reachability module without regularization based on CUDC The detailed implementation and differences from the full model are summarized in Appendix A
vary." adapting the temporal distance of k-step by the in-
Model Training and Evaluation To impose model stability during learning, we restrict the k- step to be increased from 3 to 6 and set the upper and lower bounds for the regularization weights to ensure stability. For further details regarding the network implementation and hyperparameter setting of the proposed CUDC, readers can refer to Appendix A During data collection, all methods are trained using a DDPG (Lillicrap et al. 2015) agent as the backbone for fairness. They interact with 3 domain environments in the absence of extrinsic rewards for 1M steps. For the main results, a total of 90 datasets (6 algorithms x 3 main tasks x 5 seeds) are collected. After that, relabeling is performed for each downstream task the collected dataset. During the evaluation, TT3 (Fujimoto et al. 2018) agent learns offline from each relabeled dataset for 500K steps. We report the performance score at 100K step for sample efficiency and at 500K step for learning performance.
Main Results on 12 Downstream Tasks As shown in Figure 2 no single baseline method can greatly improve the dataset quality for all domains. In contrast, the dataset collected by CUDC is capable of boosting the offline agent's learning performances at 500K step in all 12 downstream tasks across challenging domains, as highlighted in Table Specifically, CUDC outperforms the competence-basee based method (APS) in the Walker domain by 6%, outperforms the data-basee method (APT) in the Quadruped domain by 31%, and outperforms the knowledge-band based method (ICM) in the Jaco Arm domain by 10%. As for the sample efficiency, it can be observed in Figure CUDC leads to the best sample efficiency with a significant margin on 3 downstream tasks of Quadruped. In the easy domain of Walker, CUDC can help the offline agent converge faster in 3 downstream tasks. However, the sample efficiency in the hardest domain of Jaco Arm is not desirable. It might be caused by too much complexity in this most challenging environment, increasing the difficulty of reachablility analysis. More results are included in Appendix [C. [..] and evaluation using another offline RI. algorithm of CQL (Kumar al. 2020) is conducted in Appendix C.2
7

---

Under review as a conference paper at ICLR 2023
Table 1: Main results of the offline RI. agent 12 downstream tasks main domains. The proposed
CUDC collects a more useful dataset such offline RI. agents can improve sample efficiency (100K) in out of 12 downstrcam tasks and achieve better learning performance (500K) in all 12 downstream tasks.
100K Step Score
Random
APS
ProtoRL
ICM
APT
CUDC
Walker, Walk
190 153
652:1:227
532-332
5034:2588
548-1338
827
Walker, Flip
136
590:1:7
610=95
530::59
5791:51
615-166
Walker, Run
53::25
368177
332=81
2331:111
372:1400
381:137
Walker, Stand Quadruped. Walk
401=295
923::76
831=318
7971:112
8781101
984:111
135=101
2061:25
153=119
338::211
248
338::147
Quadruped, Run
145+96
183+17
121=31
210+663
249+24
256+1133
Quadruped, Stand
271=105
334+148
193=78
585+300
524+153
618+3111
Quadruped, Jump Jaco Arm, Reach Top Left
223=600
237
150-89
466++208
391
483--222
514
631140
68.149
88178
42.177
$4.160
Jaco Arm, Reach Top Right
491137
119166
76.139
99173
51167
32.159
Jaco Arm, Reach Bottom Left
35.131
87174
76.175
101:446
30.143
74.189
Jaco Arm, Reach Bottom Right
49+46
100:75
85-66
113#89
92:442
121
500K Step Score
Random
APS
ProtoRL
ICM
APT
CUDC
Walker, Walk
198::266
8454:38
826=67
802-167
799-173
893-134
Walker, Flip
303-
561
645=150
5344:218
591
717-+44
Walker, Run
624:24
369+-33
386-38
261++142
3841-20
393-:11
Walker, Stand Quadruped. Walk Quadruped, Run
519=3112
949::24
954=17
868173
830
971:7
77::53
1691:38
177=181
201
363
425176
102:
1791:52
77:136
165118
198157
349:800
Quadruped, Stand
162--96
335++800
127=111
343+133
464+144
637+1188
Quadruped, Jump
152=74
261+711
994:51
242++008
3294-85
574+74
Jaco Arm, Reach Top Left
59:755
129+266
138=52
166+:54
41+29
212++11
Jaco Arm, Reach Top Right
81:154
152.182
166-19
195133
951132
214.117
Jaco Arm, Reach Bottom Left
91-168
1031174
100-68
216.116
101.56
218.17
Jaco Arm, Reach Bottom Right
107-156
197-13
149-69
229.18
131
229.16
Walee
Wetro
Wilke
1300
200
Envionmentl
Entromeraa
Doviommet
$40
Teo
Figure 2:
Learning curves of the offline RL agent on the task-agnostic dataset collected by different methods. The proposed CUDC demonstrates superior capability of improving the sample efficiency and learning perfor- mances of the offline agent.
Effects of Adapting the k-Step We empirically show that adapting the temporal distance to plore more distant future states can certainly help to enhance the feature representation, and thereby improve the data collection process. By comparing the results in Figure CUDC formed ICM significantly, with average 1.25 x sample efficiency 100K step and f.16 x offline learning performance 500K step. Similarly, CUDCAPT obtains respectively 1.12 x and 1.04 x scores at 100K and 500K steps across 4 downstream tasks, compared with APT. We note that the standard deviation increases a bit, which might be caused by the introduced complexity of more dis- tant future state when improving the learned representation. Thus, is desirable to find an adaptive way smooth this process, e.g., by incorporating the other proposed components inherently.
vary has outper-
8

---

Under review as a conference paper at ICLR 2023
Walter Wal
Fly
Fan
Water Clard
1044
The performance score evaluated at 100K and 500K steps in 4 downstream tasks of Walker. All four versions of CUDC perform better than ICM and APT.
Figure 3:
Effectiveness of the Other Proposed Components We additionally incorporate the mixed intrin- sic reward to CUDCary vary as CUDC and it further improves both sample-efficiency and learning capability of the offline RL. agent, shown in Table 4 and Figure 3 As the new intrinsic reward en- courages more uniform exploration states, the performance at 100K is inevitably unstable, with a 67% increase of standard deviation. Therefore, it is necessary to leverage the proposed reachability module as agent's internal belief and facilitate the data collection. By comparing CUDCreach with CUDC the dataset collected by CUDCreach stabilizes the offline learning process by reducing 48% and 25% of standard deviation at 100K and 500K steps. However, its performance scores slightly decrease in 2 tasks. To address this problem, the full model further regularizes the critic-actor update with a curiosity weight, compared with CUDCreach It achieves the best performances with minimum standard deviation at 500K step- More experiments are conducted in Appendix [C.4 by removing each proposed components. It can be concluded that all components play important roles to collect useful dataset for improving the downstream task learning and resolving instability on offline RL agents.
the state-action space and focuses more on under learned
Waler
Walee
100
300
300
Envionmental
Distommrral
Dolionmett
Learning curves of the offiine RL agent on 4 downstream tasks of Walker. The k-step adaptation proposed in CUDC performs better than the other two sampling methods in out of 4 downstream tasks.
Figure 4:
Adjusting the k-Step in Different Ways One may wonder how the feature representation im- proves with different ways of adjusting the temporal distance in the reachability module. Thus, explicitly conduct an ablation study on the Walker domain by sampling & uniformly (Uniform) from to 6 and normally (Normal) with an increasing mean. can be observed in Figure that Uniform performs the worst in all 4 tasks it cannot adapt the temporal distance in curious way to enhance the representation learning. At 500K step, it only achieves 85% overall learning capa- bility with 300% increase in the standard deviation, compared to CUDC. Normal some extent adapts k through an increasing mean, and it even outperforms CUDC in the Flip task. However, its overall performance is still 4.5% weaker than CUDC and its standard deviation is 128% higher than CUDC, indicating the instability issue. Overall, the curious adaptation method proposed in CUDC the best and is interesting to investigate more adaptive ways in the future.
CONCLUSION
We propose CUDC, a curiosity-driven unsupervised data collection method to improve the dataset quality for offline RL agents in the multi-task setting. A reachability module is introduced, quan- tifying agent's internal belief of estimating the probability of a k-step future state being reachable form the current state. Substantially, CUDC can enhance feature representation by adaptively allow- ing the agent to explore more distant future states in the reachability module. Empirically, CUDC demonstrates superior data collection capability with improved sample efficiency and better per- formances in downstream multi-task offline learning. We also present this work with insightful empirical evidence of effective data collection methods for future research.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Arthur Aubret, Laetitia Matignon, and Salima Hassas. A survey on intrinsic motivation in reinforce- ment learning. arXiv preprint arXiv: 1908. 069766 2019.
Jan Beirlant, Edward J Dudewicz, László Györfi, Edward C Van der Meulen, al. Nonparametric entropy estimation: An overview. International Journal of Mathematical and Statistical Sciences, 6(1):17-39, 1997.
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. NeurIPS, 29:1471-1479, 2016.
Glen Berseth, Daniel Geng, Coline Manon Devin, Nicholas Rhinchart, Chelsea Finn, Dinesh Ja- yaraman, and Sergey Levine. Smirl: Surprise minimizing reinforcement learning in unstable environments. In International Conference on Learning Representations, 2020.
Marc Brittain, Josh Bertram, Xuxi Yang. and Peng Wei. Prioritized sequence experience replay.
arXiv preprint 1rXiv:1905.120 2019.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros. Large-scale study of curiosity-driven learning. In International Conference on Learning Repre- sentations, 2018.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Seventh International Conference on Learning Representations, pp. 1-17, 2019.
Andres Campero, Roberta Raileanu, Heinrich Kuttler, Joshua B Tenenbaum, Tim Rocktäschel, and Edward Grefenstette. Learning with amigo: Adversarially motivated intrinsic goals. Interna- tional Conference on Learning Representations, 2020.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline via supervised learning? International Conference on Learning Representations, 2022.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without a reward function. International Conference on Learning Representa- tions, 2019.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbecl, et al. Soft actor-critic algorithms and appli- cations. arXiv preprint
2018.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Kai Chieh Hsu, Vicenç Rubies-Royo, Claire J Tomlin, and Jaime F Fisac. Safety and liveness guarantees through reach-avoid reinforcement learning. 17th Robotics: Science and Systems, RSS 2021. MIT Press Journals, 2021.
Boris Ivanovic, James Harrison, Apoorva Sharma, Mo Chen, and Marco Pavone. Bare: Backward reachability curriculum for robotic reinforcement learning. In 2019 International Conference on Robotics and Automation (ICRA), pp. 15-21. IEEE, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. Prioritized level replay. In ICML, pp- 4940 4950. PMLR, 2021.
10

---

Under review as a conference paper at ICLR 2023
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions Intelligent Transportation Systems, 2021.
Iya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-77777. PMLR, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 1991199-199, 2020.
Aviral Kumar, Joey Hong, Anikait Singh, and Sergey Levine. Should i run offline reinforcement learning or behavioral cloning? In International Conference on Learning Representations, 2022.
Nathan Lambert, Markus Wulfmeier, William Whimey, Arunkumar Byravan, Michael Bloesch, Vib- havari Dasagi, Tim Hertweck, and Martin Riedmiller. The challenges of exploration for offline reinforcement learning. arXiv preprint arXiv:2201. ,
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint 1r0iv:2/10.15191 2021.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint 2Xiv:1906.05274, 2019.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. ArXiv e-prints, pp- arXiv-1607, 2016.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.06643 2020.
Kevin Li, Abhishek Gupta, Ashwin Reddy, Vitchyr H Pong, Aurick Zhou, Justin Yu, and Sergey Levine. Mural: Meta-learning uncertainty-awarr rewards for outcome-driven reinforcement learning. ICML, pp. 6346-6356, 2021.
Timothy P Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint I50.:
2015.
Hao Liu and Pieter Abbeel. Aps: Active pretraining with successor features. In International Conference on Machine Learning, pp. 6736-6747. PMLR, 2021a.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training Advances in Neural Information Processing Systems, 34: 144991948,,77, 2021b. 2021b..
Amanda Markey and George Loewenstein. Curiosity. International handbook of emotions in edu- cation, pp. 238-255 2014.
Tatsuya Matsushima, Hiroki Furuta, Yutaka Matsuo, Ofir Nachum, and Shixiang Gu. Deployment- efficient reinforcement learning via model-based offline optimization. In ICLR, 2021.
Mohammad Motamedi, Nikolay Sakharnykh, and Tim Kaldewey. data-centric approach for train- ing deep neural networks with less data. arXiv preprint arXiv:21/0.036/1 2021.
Andrew Ng. chat with andrew on mlops: From model-centric to data-centric ai, 2021.
Hima Patel. Shammukha Guttula, Ruhi Sharma Mittal, Naresh Manwani, Laure Berti Berti-Equille and Abhijit Manatkar. Advances in exploratory data analysis. visualisation and quality for data centric ai systems. Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 481 4815- 2022.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2787. PMLR, 2017.
11

---

Under review as a conference paper at ICLR 2023
Deepak Pathak, Dhiraj Gandhi, and Abhinav Gupta. Self-supervised exploration via disagreement. In Interaational conference on machine learning, pp. 5062- 5071. PMLR, 2019.
Alexandre Péré, Sébastien Forestier, Olivier Sigaud, and Pierre-Y Oudeyer. Unsupervised learn- ing of goal spaces for intrinsically motivated goal exploration. In International Conference on Learning Representations, 2018.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey offline reinforcement learning: Taxonomy, review, and open problems. arXiv preprint arXiv:2203.01387, 2022.
Boyd E Rossing and Huey B Long. Contributions of curiosity and relevance adult learning motivation. Adult Education, 32(1):25-36, 1981.
Nikolay Savinov, Anton Raichuk, Damien Vincent, Raphael Marinier, Mare Pollefeys, Timothy Lillicrap, and Sylvain Gelly. Episodic curiosity through reachability. In International Conference Learning Representations, 2019.
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. In
ICLR (Poster), 2016.
Max Schwarzer, Ankesh Anand, Rishab Goel, Devon Hjelm, Aaron Courville, and Philip Bach- man. Data-efficient reinforcement learning with self-predictive representations. In International Conference on Learning Representations, 2020.
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch. Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34: 14:6666--22999 2021.
Ramanan Sekar, Oleh Rybkin, Kostas Daniiliiis, Pieter Abbeel, Danijar Hafner, and Deepak Pathak. Planning to explore via self-supervised world models. In ICML, pp. 8583-8592. PMLR, 2020.
Younggyo Seo, Lili Chen, Jinwoo Shin, Honglak Lee, Pieter Abbeel, and Kimin Lee. State entropy maximization with random encoders for efficient exploration. In International Conference on Machine Learning, pp. 9443 9454. PMLR, 2021.
Bharat Singh, Rajesh Kumar, and Vinay Pratap Singh. Reinforcement learning in robotic applica- tions: comprehensive survey. Artificial Intelligence Review, 55(2):945-99)) 2022.
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates entropy. American journal of mathematical and management sciences, 23(3-4):301-321, 2003.
Aravind Srinivas, Michael Laskin, and Pieter Abbeel. Curl: Contrastive unsupervised representa- tions for reinforcement learning, arXiv preprint arXiv:2004.04136, 2020.
Chenyu Sun, Hangwei Qian, and Chunyan Miao. Celf: A contrastive-curiostty-iii learning framework for sample-efficient reinforcement learning. In Proceedings of the Thirty-First Inter- national Joint Conference on Artificial Intelligence, IJCAI-22, pp. 3444-3450, 7 2022a. Main
Track.
Chenyu Sun, Hangwei Qian, and Chunyan Miao. From psychological curiosity to artificial curiosity: Curiosity-driver learning in artificial intelligence tasks. arXiv preprint arXiv:2201.08.0 2022b.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Bud- den, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancy, al. Deepmind control suite. arXiv preprint 1rXiv:1801.00690 2018.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M Susskind, Jian Zhang, Ruslan Salakhutdi- nov, and Hamlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference Machine Learning, pp. 11319-11333 11328. PMLR, 2021.
12

---

Under review as a conference paper at ICLR 2023
Denis Yarats, Rob Fergus, Alessandro Lazarie, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference on Machine Learning, pp. (1920-11933 PMLR. 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint arXiv:2201. 13425, 2022.
Dongjie Yu, Haitong Ma, Shengbo Li, and Jianyu Chen. Reachability constrained reinforcement learning. In International Conference Machine Learning, pp. 25636-25655. PMLR, 2022.
Ruiyi Zhang, Bo Dai, Lihong Li, and Dale Schuurmans. Gendice: Generalized offiine estimation of stationary values. arXiv preprint 0e::20002.0907.. 2020.
Rui Zhao and Volker Tresp. Curiosity-drin driven experience prioritization via density estimation. arXiv preprint arXiv:1902.000 08039, 2019.
13

---

Under review as a conference paper at ICLR 2023
A
CUDC IMPLEMENTATION AND HYPERPARAMETER SETTING
A.1 IMPLEMENTATION DETAILS
A.1.1 CUDC
In the reachability module of CUDC shown in Figure [ the state encoder d, is a 1-layer MLP with the ReLU activation. Subsequently, the output is passed to single normalized fully-connected layer by LayerNorm Lei Ba et al. 2016) with the tanh nonlinearity applied at the end. The target encoder momentum updated from the state encoder obtain The action encoder @a a 3-layer MLP with the ReLU activation. For the forward network fa and the backward networks fa and fk. they are 2-layer MLP with the ReLU activation. The projection network h a 2-layer MLP with hidden size of 128 and output size of 64, followed by LayerNorm. h is also momentum updated from h.
A.1.2 CUDCICM
VARY AND CUDCAPT VARY
CUDC and CUDC
vary are two variants of the proposed CUDC. Given the implementation of ICM and APT in EXORL (Yarats et al., 2022). we added the forward network f. and backward network Sa in order to let these models learn more feature representations with the dynamics infor- mation. Therefore, the temporal distance of k-step can adapted similar way as in CUDC. increased by if the proportion of tuples with low intrinsic rewards is greater than threshold, i.c.
>7 [[,) Ck. this way, we can investigate the effects of adapting environment step on two n unsupervised baseline methods.
A.1.3 CUDC
CUDCreward is variant of the proposed CUDC. Given the implementation of CUDCAPT in the previous subsection, we extended the state entropy to state-action entropy and meanwhile add a prediction error of the k-step future state formulated in Equation 3 Therefore, agent can be encouraged to explore more diverse state-action space while focusing on the under states with high prediction errors. In this way, we can investigate the effects of the proposed mixed intrinsic reward.
vary
A.1.4 CUDCREACE
CUDCreach is variant of the proposed CUDC. Given the implementation of CUDC in the previous subsection, we incorporated the full reachability module with the reachable network. As result, the adaptive update of the environment step can be facilitated by the curiosity weight w outputted by the reachability module and the enhanced representation learning can be carried out in a sulf-supervised manner. Compared with the full CUDC model, CUDCrach has disabled the regularization of the critic-actor update. In this way, we can investigate the effects of the proposed reachability module.
A.2 HYPERPARAMETER SETTING
A.2.1 DATA COLLECTION
We provide a full set of common hyperparameters used in baselines as welll as CUDC Table which closely follows the same settings from ExORL Yaraas et al. 2022) and URLB (Laskin al. 2021I.
For the other hyperparameter used in CUDC, they are listed in Table 3
should be noted that Explore20ffline (Lambert et al. 2022) is a concurrent work for data collec- tion, but its source code is not available at the moment.
A.2.2 OFFLINE RL
For the offline RL agent, follow the findings reported in ExORL that the vanilla off-policy RL algorithm of TD3 (Fujimoto et al., 2018) can outperform the carefully designed offline RL algorithms when the collected dataset is of high quality. Thus, implement offline RL. of TD3
14

---

Under review as a conference paper at ICLR 2023
Table 2: Common hyperparameter setting for the unsupervised data collection methods
Hyperparameter
Value
Observation type
states
Replay buffer Size
10
Action repetitions
Seed frames
4000
Batch size
1024
Discount factor
0.99
Optimizer
Adam
Learning rate
10
Non-linearity
ReLU
Agent update frequency
2
Critic target EMA rate
0.01
Hidden dimension
1024
Exploration stddev clip
0.3
Exploration stddev value
0.2
Table 3: Hyperparameter setting for the proposed CUDC
Hyperparameter
Value
k-step range
5,
State representation dimension
512
Actor representation dimension
64
MLP hidden dimension for action encoder
64 for action encoder
MLP hidden dimension
128 for projection
Projection dimension
64
Regularization clip
[0.2, 2] for Walker
[0.2, 1] for Quadruped
[0.9, 1] for Jaco Arm
Intrinsic reward weights (a, B)
(10-3 10020) 2) for Walker
(10-4, 10-10-1 for Quadruped
(100, 0) for Jaco Arm
K-NN Threshold C.,
12
0.02 for Walker and Quadruped
0.01 for Jaco Arm
Threshold Ck
512
evaluate the quality of the collected dataset. The detailed implementation and hyperparamter settings can be found in EXORL.
15

---

Under review as a conference paper at ICLR 2023
B PROOF OF LEMMA 3.1
Proof. It has been shown in APT Liu & Abbeel (0021b) that the particle-based entropy estimator of the state representation be derived as
1 Σ log no + b(K)
H(22) =
log e
(5)
1=1
where b( K)) represents bias correlation and oz indicates the volume of the hypersphere with radius of || 2, K-NN between the 2; and its K -th nearest neighbor 2; ,K-NN
By substituting uzK
- - K ///a/2
where I is gamma function and n, is the dimension of 2,
1mm.2+41
we can obtain
H(() x log ||
K-NN
(6)
Let u = (2., 2o) represent the state-action representation and we can further substitute : = u into
Equation 66 obtain
H(u) x log 14 un uK-NN113
16

---

Under review as a conference paper at ICLR 2023
C ADDITIONAL RESULTS AND DISCUSSIONS
C.I FULL RESULTS OF MAIN EXPERIMENTS
Figure S and Figure show the full results on the 12 downstream tasks across 3 domains. Figure] summarizes the overall performances in domains. They demonstrate that our proposed methods significantly outperform the baseline methods in 3 downstream tasks of Walker and 3 downstream tasks of Quadruped. However, in the hardest domain of Jaco Arm, the learning sample efficiency 500K step catches up
becomes poor in downstream tasks although its learning performance with the best baseline method of ICM. We explain this by the fact that our method can introduce learn
more complexity and challenges for the agent whenever adapting the temporal distance better representation. However, as Jaco Arm is indeed the most challenging domain, the introduced complexity cannot fully coped with this environment. Thus, the poor performance in sample effi ciency can occur, and it is interesting to further study how to cope with hard domain environments in the future works.
SULR
Suat Wab
Oa art
Dusdhnenk mr Chane
Guathaane Aume
1004
Peed
0.00
Preck
0.00
0.000
Figure 5: Full results average performance score evaluated at 100K and 500K steps the offline agent on each downstream task. CUDC significantly improves the sample-efficiency in 9 out 12 tasks at 100K step and learning performances in all 12 tasks at 500K step.
Waterr
Walter
Walhe
Wathrr Sharet
120
Quaaliggd Jury
Reach Top
Janc Sauch Tra
JacC Arts Reach Batar
Jaco Beson Righ
Bam
Crotoreenta CROC
C C2000
Figure 6: Learning curves of the offline RL agent full 12 downstream tasks with the task-agnostic dataset collected by different methods. The proposed CUDC demonstrates superior capability of improving the sample- efficiency and learning performances of the offline RL agent in the domains of Walker and Quadruped.
17

---

Under review as a conference paper at ICLR 2023
Waker
An
Psces
KM
C000
A7E 0.00
APS
Prack
0.00
Figure 7: The overall offline learning performance across 3 domains. CUDC consistently leads to significant improvement in offline agent's 's performance at 500K step in all 3 domains on average.
In Figure we plot how our proposed method adapts the k-step during data collection the Walker domain. It be observed that agents take around 450K step to learn and adjust the temporal distance from A
to k: = 4. Then, it takes around just 200K steps to increase from k 4 k 5. Finally, it takes more than 400K steps to reach implies that at the early phase of training, we should not inject a too challenging k value. Once the agent has learned enough dynamics information, they are able learn quickly on more challenging reachability analysis. However, after a certain stage (k = 5 in this domain), the learning becomes difficult for the agent. Therefore, interesting further adapt this over-difficllt knowledge in the future works as well.
6. This increasing behavior
Figure 8: The adaptive increase how many steps into the future that the dynamics model should predict. is averaged by random runs in the Walker domain, where the k-step is limited to change from 3 to
In Figure 9 we visualize how the loss of dynamics model (forward and inverse networks) and the contrastive loss of the reachable networks change w.r.t. training steps. Both losses decrease and converge during learning. In particular, they only increase when the step is adapted to increase. After that, both losses decrease quickly. The agent can well predict each sampled k-step future state being most reachable from its own current state rather than those from other transition tuples. Even without inputting the sequence of intermediate actions states, the agent can predict the k-step future state accurately. During the model training, the feature representations are enhanced as well. Therefore, it can be validated that the learned representation contains more semantic latents with dynamics information, with the self-supervised learning of agent's internal belief.
Lok Fovare settoore Nevving
Oata
Stop
Data Collectltooon
Figure 9: The visualization of the dynamics model loss as well reachability loss (contrastive loss) during learning in the Walker domain.
In addition to the offline multi-task learning performance, we compare the quality of the collected datasets by plotting the normalized density of the true reward for the downstream task Stand in Walker environment. The visualization is shown in Figure 10 It can be seen that the dataset collected by our proposed CUDC of higher quality, with larger density for high rewards and larger proportion at the low reward part. Specifically, the mean trajectory reward of CUDC is
18

---

Under review as a conference paper at ICLR 2023
0.312, which is 69% higher than APS, 21% higher than ProtoRL, 16% higher than ICM, and 10% higher than APT. Moreover, the 75% quartile of the trajectory reward for CUDC is 0.498, which is 143% larger than APS, 42% larger than ProtoRL, 19% larger than ICM, 16% larger than APT. Therefore, it indicates the effectiveness of our proposed method to collect higher-quality dataset, where increasingly more rewarding states have been visited.
Wedee
Mtaloy Stadd
Walee Stone
Welkey Stand
Stard
Figure 10: The visualization of the true reward density Walker, Stand task for the task-agnostic dataset collected by the proposed method and baseline models.
C.2 EVALUATION BY ANOTHER OFFLINE RL ALGORITHM OF CQL
In this work, we consider the problem setting for offline RL into three main steps: data collection, reward relabeling, and downstream offlin learning. Our work focuses only on the first step of collecting high-quality dataset and thereby is agnostic to the offline RL algorithms. For the main experiments, we chose TD3 (Fujimoto et al. (018)) to evaluate the multi-task downstream learning, since it was concluded in EXORL that the vanilla TD3 algorithm can effectively learn offline and outperform carefully designed offline RL algorithms by improving the dataset quality.
To demonstrate that the dataset collected by our proposed method is of high quality than the other methods, conduct additional experiments using another offline RL algorithm of CQL (Kumar eet al.] 2020) that regularizes the Q-values during training. The results are shown in Figure IT and our proposed CUDC has also demonstrated improved sample efficiency 100K learning steps and learning capability at 500K learning steps in all 4 downstream tasks against baseline methods. By comparing the performance score at 100K and 500K steps of CUDC against the best baseline method, CUDC has achieved on average 18.2% and 12.3% improvement at respectively 100K and 500K CQL agent learning steps, across 4 downstream tasks. Specifically, CUDC 21 % higher than the best baseline of APT at 100K steps on Run task while its learning performance is 15% higher than the best baseline of APS at 500K steps.
Waller Walk
Welve Ra
Walke Stard
1C:4
Pi
100
Prenett
Frovit.
CUDC
CUDD
Envionmenal Depps 30
Disfommeral Shers
Cerormete SHeR
B00
B004
17/4
JPE
5.00
0.02 AP5
Figure 11: Learning curves (top) and performance scores at 100k and 500K learning steps (bottom) for the CQL agent on downstream tasks of Walker. The proposed CUDC demonstrates superior capability of improving the sample-efficiency and learning performances of the CQL agent in all 4 tasks.
19

---

Under review as a conference paper at ICLR 2023
C.3 RESULTS OF ADDING EACH PROPOSED COMPONENT ON TOP EACH OTHER
We present the performance scores of the four variants of CUDC at 100K and 500K steps in Table Moreover, the detailed learning curves the offline RL agent on downstream tasks are shown in Figure It can be concluded that adapting the temporal distance to reach more distant k-step future can substantially improve both sample efficiency and learning capabilities. Moreover, all proposed components facilitated by the proposed reachability module are necessarily important to yield the improvement. As result, the full CUDC model further addressed the instability issue to obtain the minimum standard deviation among all methods.
Table 4: Performance scores (mean & standard deviation) of four versions of CUDC 100K and 500K environment steps. The full model outperforms other versions on out of 4 tasks in sample efficiency (100K) and all four tasks in learning performance (500K) regimes, across random seeds.
100K Step Score
ICM
CUDC
APT
CUDC
CUDC
CUDC
CUDC
Walker, Walk
5031:258
6861144
5481.338
7851:56
796.185
811:699
827
Walker, Flip
530:159
589-182
5791:51
542::173
601::156
$7011
615:166
Walker, Run
233:1111
320:::66
372:1:00
387::19
3851:69
370-1400
381::37
Walker, Stand
797=312
909-888
878=101
936:1:9
950:140
983=11
984=11
500K Step Score
ICM
CUDCIGM
APT
CUDC
CUDC
CUDC
CUDC
vary
Walker, Walk
802:67
822:552
799173
820::11
824::52
849136
893::34
Walker, Flip
5341218
665-132
591
674
6661:53
679==1
717=41
Walker, Run
261+1142
336+1115
384+-20
370+338
394+-43
375+27
393--11
Walker, Stand
868+
927++74
890++139
916:
970++12
969-111
971
130
800m
Z200
30
20
CLD
XPT
CJ0C2
CUDC
CJECau
CUTCC
Figure 12: Full results of learning curves on 4 downstream tasks of Walker environment with the task-agnostic dataset collected by different versions of the proposed CUDC. They are capable of improving both sample efficiency at 100K step and learning capabilities at 500K steps, compared to the baselines of ICM and APT. All proposed components work inherently to collect the high quality dataset for offline learning.
C.4 RESULTS OF REMOVING EACH PROPOSED COMPONENT FROM FULL MODEL
To further investigate the effectiveness of each proposed component and quantify the importance of them, we carry out additional experiments by respectively removing each proposed component from the full model. particular, the following models are used to collect the task-agnostic dataset for the Walker environment with 5 random seeds.
CUDC -Enroon The proposed intrinsic reward of KNN-based particle entropy of state and action ( 0t) is removed, and only the prediction error based reward (e 3 is used.
CUDC Pp: The proposed intrinsic reward of prediction emoo re(se, ar) is removed, and only the KNN-based particle entropy reward state and action I'd (St, 0z) used.
CUDC R the mechanism of regularizing the backbone DDPG algorithm is re- moved.
CUDC_vary
k-step is fixed be 3 throughout learning.
20

---

Under review as a conference paper at ICLR 2023
CUDC Reach- The proposed reachability module is removed while k-step is still adapted through the loss of the dynamics model.
CUDC Inverse:
The inverse networks of predicting the action and step & removed.
CJDC
0100-m
CJCC
CU0C
CUDC
CIDC
CUDD
ane
cloc
Figure 13: Learning curves (top) and performance scores 100K and 500K learning steps (bottom) on 4 downstream tasks of Walker environment with the task-agnostic dataset collected by different models of removing each proposed component from CUDC. Removing any proposed component will cause less desirable performance throughout the offline multi-task task learning process.
Table 5: Relative performance of the average scores for the ablation models (removing each pro- posed component from the full CUDC model) at 100K and 500K learning steps. Adapting k-step is the most effective component the improved sample efficiency and learning capabilities, while the proposed mixed intrinsic reward is the second.
100K Relative Performance
CUDC Emey
CUDC
CUDC Regulvite
CUDC Vay
CUDC Rexh
CUDC Inrene
CUDC
Walker. Walk
0.954
0.768
0,849
0.963
0.946
Walker, Flip
0.979
1.028
0.909
0,819
0.976
1.074
1.000
Walker, Run
0,851
0.827
0,916
0,857
0.986
0.974
1,000
Walker, Stand
1.000
0.995
0.994
0.977
0.965
0.954
1,000
Mcan
0.946
0.904
0.930
0.875
0.973
0.987
500K Relative Performance
CUDC Eatroy
CUDC ME
CUDC
Rogukaiiz
CUDC Vary
CUDC Ieah
CUDC Insene
CUDC
Walker, Walk
0.935
0.934
0.952
0.900
0.922
0.837
1,000
Walker, Flip
0.843
0.894
0.937
0.809
0.929
0.929
1,000
Walker, Run
0.856
0.899
0.914
0.902
0.942
0.853
1,000
Walker.
0.979
0.999
0.962
0.981
Mean
0.903
0.931
0.948
0.893
0.948
0.900
1,000
Figure 13 shows the overall offline learning performances. We can observe that limiting k = 3 and removing the inverse networks will cause the most significant performance decreases among the evaluated models. It implies that adapting the step between current and future states can help learn rich representation, which plays the most important role in our proposed CUDC collect high-quality dataset for offline multi-task learning. Meanwhile, the inverse networks are necessary learn the representation that is robust the uncontrollable features by the agent's actions and enables the encoders to capture the dynamics information in the learned representation.
To quantitatively measure the benefits brought by each proposed component, we compute the relative performance scores at 100K for sample efficiency and 500K for learning capabilities based on the full model. The results are summarized in Table For the sample efficiency at 100K steps, removing the k-step adaptation has caused the worst performance with 18.1 1 decrease in Flip task and an overall 12.5% decrease across all 4 downstream tasks, followed by removing prediction-error based reward (9.6%), removing regularization (7.0%) and removing entropy-basee reward (5.4%).
21

---

Under review as a conference paper at ICLR 2023
For the learning capabilities at 500K steps, removing the k-step adaptation has also resulted in the worst performance with a 19.1% decrease in Flip task and an overall 10.7% decrease across all downstream tasks, followed by removing inverse networks (10.0%), removing entropy- based reward (9.7%) and removing prediction-errorf many steps into the future that the dynamics model should predict is most effective the improved sample efficiency and learning capabilities, while the proposed mixed intrinsic reward is the second most effective.
reward (6.9%). Thus, we conclude that adapting how
C.5 ABLATION STUDIES ON THE RANGE OF VARYING K-STEP
In our work, set the range of varying k-step from to 6. It starts from 3 as we follow the same setting as the EXORL benchmark, where all 8 data collection baselines strictly limit k=3 for the future state in each transition tuple. As for the threshold of C. and Ck. did not specifically hypertune these values and they were just set to ensure that can be varied from to adaptively during the 1M dataset collection process. As for the upper bound of 6, it is optimal end value to obtain the desired performances. To support this finding, we carry out a sweep of the upper bound of l from 3 to 8, and present the results in Figure 14] Firstly, it can be observed that the learning capabilities at 500K learning steps first increase and then decrease with the increase of the upper bound of k. across all 4 downstream tasks. Setting the upper bound of achieves the highest in Walk and Run tasks while it is the second highest in Flip and Stand tasks. Secondly, there is no clear trend of performance at 100K steps by setting different values of the upper bound for k. It can be explained by the complexity caused by varying the k-step for the future states. However, we can still observe that setting the upper bound of 6 achieves the highest tasks. Therefore, we believe k should vary from 3 to 6 to learn rich representation with more semantic latents. Wakee Stard
Waleor
Walee
Wakor
Exc
104
2305-
1arr 31x
Figure 14: Learning curves (top) and performance scores at 100K and 500K steps (bottom) on 4 downstream tasks of Walker with the task-agnostic dataset collected by different range of k-step. Overall, setting the range from 3 to 6 performs relatively the best.
22


Reviews:
[{"review_id": "k8nQ7WYbT5", "reviewer": "Reviewer_F8ph", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of dataset collection in offline RL, while most of the prior work focuses on building more efficient algorithms given a fixed dataset. To be specific, the paper proposes curiosity-driven unsupervised data collection, that helps the agent to collect data in a self-supervised fashion. The paper proposes to collect task-agnostic data independently, then label the dataset with task-specific rewards. The paper conducts experiments on the DeepMind control suite including 12 different downstream tasks, and the proposed method achieves good performance. ", "summary_of_the_review": "Please see the strengths and weaknesses above. I am overall positive about this paper. But I hope the authors can address the points I raised above.", "strength_and_weaknesses": "Strength: \n1. The paper tackles an important problem of data collection in the offline RL setting. \n2. The proposed self-supervised algorithm is novel and demonstrates good downstream performance. \n3. The paper is clearly written. \n4. The experiments conducted are solid. \n\nWeakness: \n1. My major concern is about how different offline-RL algorithms can benefit from data collection. The paper uses TD3 as the evaluation protocol algorithm. However, I think evaluating at least some more offline RL algorithms and comparing their performance will make the paper more convincing. \n2. Another concern is that I feel the ablation in Table. 4 is not super conclusive. I especially feel the performance difference for different ablations is small. I am not sure what conclusion we can draw from the table, about the effectiveness of each component. \n3. I am wondering why the authors don't report any results in some D4RL benchmarks (e.g., some mujoco tasks, ant maze tasks or robotics tasks). \n4. The Mixed Intrinsic Reward module is not super novel, given that the entropy and inverse dynamics have been well-known. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "dIJO0i9Q8L", "reviewer": "Reviewer_LA4m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the problem of unsupervised exploration in RL, such that the resulting dataset is good for downstream learning (specifically offline RL). The majority of the technical contribution of the paper is in a method for unsupervised exploration. \n\nThe proposed method has quite a few moving parts. \n- First, it learns a state and action representation with a contrastive dynamics modeling loss, where conditioned on timesteps k, a model predicts the state reached k steps into the future after applying k actions. It trains such that the predicted state is closer to the embedding of the true s_{t+k} than negatives. \n- It also trains inverse dynamics models and temporal distance predictors from these low-dimensional embeddings. \n- It simultaneously estimates the particle-based entropy of the state-action distribution using K nearest neighbors distance.\n- The intrinsic exploration reward is formulated as a weighted sum of the particle-based entropy, and the prediction error of the dynamics models (forward, inverse, temporal distance)\n- Finally, when training DDPG with intrinsic reward, during policy optimisation the Q values are weighted inversely proportionally to the \"reachability\" as measured by the k-step dynamics model, up weighting the reward on less reachable states.\nThe method also has a curriculum which updates the choice of k over the course of training to be further away.\n\nIn experiments, they run the proposed approach and other ablations and baseline exploration algorithms for 1M steps, then train offline RL agents on the collected data and evaluate their performance. The proposed approach appears to improve over methods like ICM and APT.", "summary_of_the_review": "Overall, the paper proposes an interesting exploration algorithm. However the method has many moving parts that are not clearly explained or motivated, and overall the presentation of the paper and clarity of the writing needs significant improvement. The method itself seems to work decently, but its a bit hard to evaluate the experiments/ablations with the current presentation. ", "strength_and_weaknesses": "*Strengths*\n- How to design unsupervised exploration/data collection methods to facilitate offline RL on that data is an important problem. \n- The proposed method does seem to produce better datasets for offline RL on Mujoco tasks than the baselines.\n\n*Weaknesses*\n\nClarity/Method: The main flaw in the paper is the clarity of presentation. The paper proposes a complicated method with many moving parts but fails to motivate or clearly explain them. \n- A large part of the introduction and abstract proposes the \"k-step\" exploration as the key advancement proposed in this work. However it is not clear from much of the paper what exactly is meant by this (e.g. action repeats for k steps, planning length k sequences at a time, extending the horizon of each episode, etc.) Ultimately, it seems like the only impact the choice of k has is in choosing how many steps into the future the dynamics model should predict. The paper could do a much better job of clearly explaining this from the beginning, and probably should choose some different term than \"environment step\". \n- Its also not very convincing that the scheme for adapting this environment step is necessary or effective. The paper proposes to start the value of k at 3 and gradually increase it to 6, with some thresholds on the intrinsic reward to determine when to increase it. This feels a bit handcrafted, and looking at FIgure 4 it doesn't seem convincingly better than the simple uniform or gaussian baseline. \n- The dynamics model takes as input a sequence of k actions correct? The notation seems to suggest its just one action, but predicting s_{t+k} seems ill posed without a sequence of k actions.\n- Overall the method seems to be taking 3 established types of exploration objectives (reachability, entropy, and model error), and combining them by summing the entropy and model error rewards, then scaling the Q function loss by the reachability reward. This way of combining things feels a bit arbitrary. Why is this the right way to combine these parts of the objective? And why should all these be combined, it seems there could be redundancy between then? The paper should better motivate these design choices. \n- Figure 1 is very difficult to parse. \n\nExperiments:\n- Are there error bars for figure 3?\n- I additionally found the ablation naming quite confusing. I would suggest renaming them based on what exactly is being removed from the method in each. \n- Overall the experiments seem to show that the proposed method outperforms the baselines and ablations, though I'm not 100% sold on these experiments due to (1) the ablations seem incomplete (don't include removing each of the 3 parts of the exploration reward) and (2) the minimal gains/no error bars in the ablation experiments.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "GyxXzQkabj", "reviewer": "Reviewer_bF1X", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an approach for self-supervised learning for agents to collect data, which is then relabelled with task reward and used for downstream tasks with offline RL. The approach estimates reachability of states using a contrastive loss where predicted state features and corresponding ground truth features comprise positive pairs (negatives are randomly sampled from the batch). Transitions with higher reachability estimates are prioritized during offline RL training. The intrinsic reward consists of particle-based kNN entropy and prediction error. The paper includes experiments on Walker, Quadruped and Jaco arm-reaching. ", "summary_of_the_review": "While this paper seeks to tackle an important problem, there are key issues that prevent me from recommending acceptance, including justification of the reachability metric, missing discussion/comparison of model disagreement (for both intrinsic reward and reachability), missing analysis of alternate schemes for offline RL and how they compare to the reweighting approach proposed (given the familiar intrinsic reward that combines known terms). ", "strength_and_weaknesses": "\nWeaknesses \n\n1. Reachability objective justification - The paper motivates its approach by emphasizing the importance of reachability of future states for offline RL, and proceeds to provide an estimate for this using a contrastive loss where predicted features and corresponding ground truth features are positive pairs, with negatives randomly sampled. There is no explanation for this particular choice beyond the intuition that 'each future state should be most reachable from its own current state'. If there is a lot of overlap of states across trajectories, this statement will likely not hold. Further, even if the states are different, they might need to have similar representations for control (eg: objects might be in similar positions). \n\n2. Model disagreement missing - The paper uses ICM as the comparison for knowledge-based exploration. However, ensemble disagreement of predictive models has shown better results [1,2,3]. This approach is prospective as opposed to retrospective, since it does not rely on prediction error but rather on the variance of the prediction across models, and can solve some problems that prediction error based methods struggle on like the noisy-TV problem. The paper does not compare to this, and also does not discuss why they use prediction error in the intrinsic reward term instead of this. Furthermore, related to 1), an ensemble of predictive models provides an estimate of reachability of states (states on trajectories with lower disagreement are likely to be more reachable). \n\n3. Familiar Intrinsic reward formulation - The paper defines the intrinsic reward to be the sum of terms introduced in previous work. The KNN-based particle entropy of state [4] is modified to include state and action representations, but maximizing entropy of action for exploration has also been previously explored [5]. The other part of the intrinsic reward is just the prediction error of the predicted state feature [6]. These have been studied and examined independently in the past, and the paper does not contribute any new insight into how agents should define rewards for themselves. \n\n4. Experiments/Contribution questions - The paper does include extensive comparisons to a set of prior approaches. However what part of the proposed method leads to the stated benefit? Is it mostly because of the combination of prediction error and particle estimation? Or is it because of the reachability estimate used in offline learning? If it's the latter, how does using better offline RL algorithms such as AWAC/IQL compare (since they also reweight transitions)? If the main contribution is a reweighting scheme for offline RL, then can that be studied independently of the self-supervised exploration setting?\n\nStrengths \n\n1. The paper considers an important problem - that of agents that can direct themselves to collect rich data. Agents that can autonomously act in this manner can keep collecting large datasets, since the bottleneck of task/reward specification is removed. \n\n\n\n\n[1] - Pathak, Deepak, Dhiraj Gandhi, and Abhinav Gupta. \"Self-supervised exploration via disagreement.\n[2] - Sekar, Ramanan, et al. \"Planning to explore via self-supervised world models.\" \n[3] - Mendonca, Russell, et al. \"Discovering and achieving goals via world models.\"\n[4] Liu, Hao, and Pieter Abbeel. \"Behavior from the void: Unsupervised active pre-training.\n[5] Haarnoja, Tuomas, et al. \"Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor.\"\n[6] Pathak, Deepak, et al. \"Curiosity-driven exploration by self-supervised prediction.\"\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "9sls-N7FYq", "reviewer": "Reviewer_S2Ud", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a curiosity driven unsupervised data collection method for collecting task-agnostic datasets for offline reinforcement learning. The approach proposes a learnable state reachability module that provides an intrinsic reward for exploration by combining a state-action entropy bonus with a forward dynamics prediction error based reward; the \"horizon\" or \"step\" of the forward dynamics prediction is adapted over time to enable the method to focus on more challenging dynamics over time which serves to better condition the learned representations and aid exploration. Additionally, the learned reachability module provides an importance weight based on the estimated reachability; this is used to weight both the actor and critic updates and enables the agent to focus on less-explored parts of the state space. The approach is used together with DDPG for collecting data, the resulting dataset is relabeled and used for offline RL using TD3. The performance of these learned policies are compared with several baselines and ablations on three domains with four tasks each; results show significant improvement compared to baselines.", "summary_of_the_review": "Overall, this approach presents an interesting application of curiosity driven exploration for data collection for offline RL. The results are promising and the evaluation is well done. I would suggest a weak accept.", "strength_and_weaknesses": "Strengths:\n1. The approach is well motivated and different parts of the proposed approach are analyzed reasonably well in the ablation experiments. Particularly, the adaptation of the environment step and its application to data-collection for offline RL is quite interesting and seems to aid downstream performance.\n2. The evaluation is thorough (albeit on a simple set of tasks) and the learned policies show significant improvements in performance compared to prior work. Ablations clearly demonstrate the importance of the different choices made. \n\nWeaknesses:\n1. The approach is a bit complex and has several moving parts. In particular it is not clear if the inverse networks are necessary as no explicit ablation is done removing them, and the intrinsic bonus and importance weights do not directly depend on these predictions.\n2. While most results show downstream agent performance after offline RL using TD3, it would have been useful to see some quantification of the (downstream task) reward within the collected datasets. This can be a good way to directly compare how useful the dataset is for each of the downstream tasks; plotting this against data collection progress on the x-axis can also help quantify if the change in environment step leads to increasingly more rewarding states being seen.\n3. The adaptation of the environment step is a key novelty in the proposed approach and while it is ablated against different schemes (and different intrinsic rewards) it would be helpful if results are provided for a sweep over hypers for the specific formulation chosen in the paper (specifically, the start/end values of k, and threshold C_w and C_k). \n4. The intrinsic reward weights \\alpha and \\beta for the Jaco task are orders of magnitude different from the others, is there a specific reason for this choice?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject