SlowGuess/NeurIPS_2023_OCR huggingface dataset:
plot exploration trajectories in maze_large in Figure S15. We note that POLO is a privileged baseline, as it requires goals to be specified during training, and its value function is trained online by optimizing against its uncertainty. Our method, on the other hand, learns its value function offline from unlabeled data, and is therefore more widely applicable.
While the closer implementation (POLO*) is less performant, the improved version (POLO**) performs comparably or better than our method in three out of four environment. Despite successfully leveraging online value learning, we find its potential for long-horizon exploration to be limited, as shown by a large drop in performance in maze_large. We confirm this by visualizing trajectories gathered by POLO** (value-driven), and by the unsupervised exploration phase used in our work (dynamics-driven) in Figure S15.
H Compact Algorithmic Description
We report the full method (TD3 + MDP + Aggregation) in a compact algorithmic description in box S1, as well as in a short verbal summary below.
Our work deals with extracting a goal-reaching controller from the outcomes of curious exploration. Exploration data is collected while learning a dynamic model ensemble by maximizing its disagreement through model-based planning with zero-order trajectory optimization. This constitutes the intrinsic phase, which returns an exploration dataset, and a trained dynamics model. We then train a value function offline on the exploration dataset with TD3. During inference, we are given a goal. We select actions through model-based planning with the learned dynamics model. This time, actions
Figure S15: Random uniform, value-driven (as in POLO) and dynamics-driven (default in our method) exploration in maze_large. Lighter colors represent high exploration incentive. Uniform exploration is insufficient, and dynamics-driven exploration achieves dense coverage of the state space while being task-agnostic.
sequences are selected by maximizing the aggregated value of the last state in imagined trajectories, where aggregation is performed with graph search over the value estimates from TD3.
Algorithm S1 TD3 + MDP + Aggregation
1: for each exploration episode do Intrinsic Phase  
2: Collect a trajectory by maximizing the intrinsic reward  $r_{\mathrm{int}}$  (S17, S18) with iCEM and the dynamics model  $\hat{\mathcal{P}}$ .  
3: Add the trajectory to the replay buffer  $\mathcal{D}$ .  
4: Train the dynamics model  $\hat{\mathcal{P}}$  via MLE over the replay buffer  $\mathcal{D}$ .  
5: end for  
6: for each offline training epoch do  
7: Train a goal-conditioned value function  $\hat{V}$  with TD3 on the replay buffer  $\mathcal{D}$ , relabeled as described in Subsection 4.1.  
8: end for  
9: for each evaluation goal do Extrinsic Phase  
10: Build graph  $\mathbf{G}$  as described in Subsection 6.2.  
11: Act by maximizing the aggregated value  $\bar{V}$  (5, S19) with iCEM and the dynamics model  $\hat{\mathcal{P}}$ .  
12: end for
I Implementation Details
This section provides an in-depth description of methods and hyperparameters. Our codebase builds upon mbrl-lib [34], and adapts it to implement unsupervised exploration and goal-conditioned value-learning algorithms. To ensure reproducibility, we make it publicly available<sup>6</sup>.
Figure S16: Environments used for evaluation: from left to right, maze2d_large, pinpad, fetch.push and kitchen.
I.1 Environments
Empirical evaluations rely on four simulated environments. maze_large is adapted from D4RL [14], with the sole modification of fixing the initial position of the agent in a corner of the maze, in order to make exploration more challenging. The goal distribution includes 3 goals, placed on far-away corners of the maze, as displayed in Figure 6 (main paper). pinpad is implemented on top of D4RL [14] as a continuous, state-based version of the environment introduced in Hafner et al. [18]. The agent is initialized in a room with 4 buttons at its corners, and is tasked with pressing 3 of them in the correct order. Observations are thus 7 dimensional, and include 2D position and velocity of the agent, as well as a 3-dimensional history of the last buttons pressed. The agent is controlled through acceleration, as in maze_large.kitchen is adapted from gymnasium-robotics [13] in order to allow goal-conditioning. The goal distribution includes controlling each of the seven objects individually: a sliding door, a cabinet, a light switch, two burners, a teapot and a microwave oven. We use a frame skip value of 50 to encourage exploration. fetch.push is one of the benchmarks presented in Andrychowicz et al. [1], and we use it in its unmodified form, as provided in gymnasium [13]. A summary of further environment parameters can be found in Table S1.
$^{6}$ Code available at sites.google.com/view/gcopfce
|S|	|G|	|A|	Episode Length	γ
maze_large	4	2	2	600	0.99
pinpad	7	3	2	300	0.99
fetch.push	25	3	4	50	0.95
kitchen	30	17	8	280	0.99
I.2 Unsupervised Exploration
Unsupervised exploration of the environment relies on curious model-based planning [38, 43]. We will thus describe the exploration algorithm, followed by the architecture of the dynamics model  $\hat{\mathcal{P}}$  and the computation of the intrinsic cost signal.
Unsupervised exploration alternates environment interaction with model training, until sufficient experience is collected. We warm-start exploration by executing a uniformly random policy for 3000 steps, and saving its trajectories in the replay buffer  $\mathcal{D}$ . Then, before each episode, the dynamics model  $\hat{\mathcal{P}}$  is trained on data from the replay buffer  $\mathcal{D}$  via supervised learning for 12 epochs. Adam [21] is used as an optimizer with batch size  $B = 512$ , learning rate  $\gamma_{\mathrm{lr}} = 0.00028$  and weight decay parameter  $\lambda = 0.0001$ .
The dynamics model  $\hat{\mathcal{P}}$  is learned as an ensemble of Gaussian neural networks, as suggested in Chua et al. [8]. Given a state  $s\in S$  and an action  $a\in S$ , each of the  $M$  ensemble member outputs a Gaussian distribution parameterized by its mean and its variance:  $\{f_m(s,a) = [\mu_m(s,a),\mathrm{Var}_m(s,a)]\}_{m = 1}^M$ . In practice, inputs and outputs are normalized, and the model predicts state differences. By default, the ensemble contains 7 members; during inference, only the best 5 members according to log-likelihood on a validation set are queried. Implementation and hyperparameters, reported in Table S2, are the default from mbrl-lib [34].
Exploration episodes are controlled through finite-horizon model-based planning with zero-order trajectory optimization (i.e., iCEM with the learned model  $\hat{\mathcal{P}}$ ). Given a state-action pair  $(s, a)$ , an intrinsic reward signal can be computed as the trace of the covariance matrix over the ensemble's predictions [38, 49]
$$
r _ {\text {i n t}} (s, a) = \operatorname {T r} \left(\operatorname {C o v} \left(\left\{f _ {m} (s, a) \right\} _ {m = 1} ^ {M}\right)\right). \tag {S17}
$$
This reward function can be aggregated into a cost function for trajectory optimization, as described in Section I.4. Overall, our implementation of unsupervised exploration runs at 5K steps per hour on a single Nvidia RTX3060 GPU or equivalent.
Table S1: Environment parameters.
Hyperparameter	Value
# of ensemble members	7
# of elites	5
# of layers	4
# of units per layer	200
Activation function	SiLU
Propagation method	TS1 [8]
# of particles	20
Table S2: Hyperparameters for the dynamics model  $\hat{\mathcal{P}}$
I.3 Value Learning
The relabeling procedure for exploration data is adapted from Tian et al. [46], and described in Section 4. We implement TD3 [14] and CRR [50] for value learning, and build upon TD3 for two model-based algorithms: MBPO [19] and MOPO [53]. We found DDPG [26] and SAC [17] to be slightly less performant in comparison to TD3. Each value learning algorithm is trained for 200 epochs, which, for a buffer size of 200K, corresponds to a wall-clock time less than 3 hours on a single NVIDIA RTX3060 GPU. Hyperparameters are listed in Table S3. Those tagged as (MBPO) are adopted by both MBPO and MOPO, while those tagged as (MOPO) or (CRR) are algorithm-specific. Remaining hyperparameters are shared by all algorithms. All hyperparameters are shared across environments, and they were tuned through a grid search across maze_large and fetch.Push.
I.4 Model-based Planning
Model-based planning is used both during the exploration phase, and during goal-conditioned evaluation. During exploration, the cost of each trajectory  $(s_0, a_0, \ldots, s_{H-1}, a_{H-1})$  is computed as:
$$
c _ {\text {i n t}} \left(\left(s _ {0}, a _ {0}, \dots , s _ {H - 1}, a _ {H - 1}\right)\right) = - \sum_ {t = 0} ^ {H - 1} r _ {\text {i n t}} \left(s _ {t}, a _ {t}\right) \tag {S18}
$$
During goal-conditioned evaluation, a value function  $V(s; g)$  is optimized, and the cost of each trajectory is directly adapted from the reinforcement learning objective:
$$
\begin{array}{l} c _ {\text {e x t}} \left(\left(s _ {0}, a _ {0}, \dots , s _ {H - 1}, a _ {H - 1}\right)\right) = - \left(\sum_ {t = 0} ^ {H - 2} \gamma^ {t} R \left(s _ {t}; a _ {t}\right) + \gamma^ {H - 1} V \left(s _ {H - 1}; g\right)\right) \tag {S19} \\ = - \gamma^ {H - 1} V (s _ {H - 1}; g), \\ \end{array}
$$
where the equality holds due to the extreme sparsity of  $R(s;g) = \mathbf{1}_{s = g}$  in continuous spaces. Hyperparameters are fixed across environments, except for the planning horizon  $H$ . We find exploration to benefit from a longer horizon, which is thus set to  $H = 30$ . On the other hand, we found goal-conditioned evaluation to be more prone to model exploitation and hallucination. For this reason, during the extrinsic phase, we set  $H = 10$  for pinpad and  $H = 15$  for the remaining environments. For the same reason, we also compute the variance of particles for each trajectory, and discard those exceeding a set threshold.
While vanilla CEM [37] also performs well as zero-order trajectory optimizer, we found iCEM [35] to reach similar performances when using a reduced number of samples. Hyperparameters are reported in Table S4.
I.5 Metrics
Unless specified, all plots show the mean estimate from 9 seeds, which are the combination of 3 seeds for collecting exploration data, and 3 seeds for value learning and planning for each dataset. Each mean estimate is accompanied by a  $90\%$  simple bootstrap confidence interval.
I.6 Computational Costs
Our method's training costs (TD3 + model-based planning + aggregation) are those of TD3 ( $\sim 70$  minutes on a single NVIDIA RTX3060 GPU), comparable to other value-learning baselines ( $\sim 50-90$  minutes). Inference costs differ greatly: actor networks can be queried at  $>100\mathrm{Hz}$ ; model-based planning with and without aggregation at  $3\mathrm{Hz}$  and  $0.3\mathrm{Hz}$  respectively. An optimized implementation would bring significant speedups, e.g. through precomputation and parallelization of graph search, which could address the significant slowdown.
J Open Challenges and Negative Results
Our method provides insights on how to mitigate the occurrence of estimation artifacts in learned value functions. However, it does not correct them by finetuning the neural value estimator. This possibility is interesting, as it would bring no computational overhead, and was broadly explored. TD finetuning with CQL-like penalties on the value of  $\mathcal{T}$ -local optima was attempted, but unsuccessful. In this context, we found such correction strategies to be sensible to hyperparameters, and often resulting in overly pessimistic estimates. We, however, stress the significance of direct value correction as an exciting direction for future research.
K Numerical Results
Table S5 reports success rates from Figures 2 and 7 (main paper) in numerical form.
Hyperparameter	Value
# of critic layers	2
# of units per critic layer	512
# of actor layers	2
# of units per actor layer	512
Batch size	512
Critic learning rate	1e-5
Actor learning rate	1e-5
Activation function	ReLU
Polyak coefficient	0.995
Target noise	0.2
Target noise clipping threshold	0.5
Policy delay	2
Policy Squashing	True
pg	0.75
pgeo	0.2
(MBPO) # updates before rollouts	500
(MBPO) # of rollouts	5000
(MBPO) Rollout horizon	5
(MBPO) Buffer capacity	1M
(MBPO) Real data ratio	0.0
(MOPO) Penalty coefficient λ	0.1
(CRR) β	1.0
(CRR) # of action samples	4
(CRR) Advantage type	mean
(CRR) Weight type	exponential
(CRR) Maximum weight	20.0
Table S3: Hyperparameters for value learning.
Hyperparameter	Value
# of iterations	3
Population size	400
Elite ratio	0.01
α	0.1
Population decay	False
Colored noise exponent	3.0
Fraction of kept elites	0.3
Table S4: Hyperparameters for model-based planning.
maze_large	pinpad	fetchishly	kitchen
CRR	0.095 < 0.173 < 0.253	0.027 < 0.055 < 0.088	0.083 < 0.083 < 0.083	0.253 < 0.331 < 0.412
MBPO	0.000 < 0.000 < 0.000	0.017 < 0.028 < 0.040	0.083 < 0.083 < 0.083	0.111 < 0.143 < 0.174
MOPO	0.023 < 0.049 < 0.079	0.033 < 0.050 < 0.072	0.083 < 0.083 < 0.083	0.015 < 0.046 < 0.079
TD3	0.079 < 0.144 < 0.206	0.244 < 0.267 < 0.291	0.083 < 0.083 < 0.083	0.079 < 0.110 < 0.142
CRR+MBP	0.000 < 0.031 < 0.063	0.283 < 0.321 < 0.361	0.152 < 0.190 < 0.231	0.174 < 0.206 < 0.238
MBPO+MBP	0.024 < 0.060 < 0.098	0.043 < 0.068 < 0.091	0.638 < 0.719 < 0.791	0.174 < 0.222 < 0.285
MOPO+MBP	0.000 < 0.025 < 0.098	0.013 < 0.025 < 0.038	0.824 < 0.884 < 0.935	0.365 < 0.444 < 0.523
TD3+MBP	0.206 < 0.301 < 0.412	0.190 < 0.232 < 0.274	0.638 < 0.693 < 0.745	0.333 < 0.379 < 0.428
TD3+MBP+Aggregation	0.627 < 0.705 < 0.814	0.314 < 0.410 < 0.500	0.587 < 0.647 < 0.708	0.301 < 0.365 < 0.428
MBP+Sparse	0.030	0.100	0.730	0.430
Table S5: Average success rates with  $90\%$  bootstrap confidence intervals from Figures 2 and 7 (main paper).