Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning
Authors: Wesley A. Suttle
Aamodh Suresh
Carlos Nieto-Granda
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning
offline reinforcement learning
exploration
entropy
Primary area: reinforcement learning
TL;DR: We explore a new exploration objective for RL and show that it generates superior datasets for subsequent offline RL.

Abstract: Entropy-based objectives are widely used to perform state space exploration in reinforcement learning (RL) and dataset generation for offline RL. Behavioral entropy (BE), a rigorous generalization of classical entropies that incorporates cognitive and perceptual biases of agents, was recently proposed for discrete settings and shown to be a promising metric for robotic exploration problems. In this work, we propose using BE as a principled exploration objective for systematically generating datasets that provide diverse state space coverage in complex, continuous, potentially high-dimensional domains. To achieve this, we extend the notion of BE to continuous settings, derive tractable $k$-nearest neighbor estimators, provide theoretical guarantees for these estimators, and develop practical reward functions that can be used with standard RL methods to learn BE-maximizing policies. Using standard MuJoCo environments, we experimentally compare the performance of offline RL algorithms for a variety of downstream tasks on datasets generated using BE, R\'{e}nyi, and Shannon entropy-maximizing policies, as well as the SMM and RND algorithms. We find that offline RL algorithms trained on datasets collected using BE outperform those trained on datasets collected using Shannon entropy, SMM, and RND on all tasks considered, and on 80\% of the tasks compared to datasets collected using Renyi entropy.

Published as a conference paper at ICLR 2025
BEHAVIORAL ENTROPY-GUIDED DATASET GENERA- TION FOR OFFLINE REINFORCEMENT LEARNING
Wesley A. Suttle", Aamodh Suresh', Carlos Nieto-Granda U.S. Army Research Laboratory Adelphi, MD 20783, USA wesley. www.ss.tttee..oo ct suttee. army mil,
aamodnégmail. com,
carlos, w.llos.p.nietee www.nniet.......m wwt.:e...vvveee cive.riiiaaaag mil
ABSTRACT
Entropy-basee objectives are widely used to perform state space exploration in reinforcement learning (RI. and dataset generation for offline RI.. Behavioral en- tropy (BE), a rigorous generalization of classical entropies that incorporates cog- nitive and perceptual biases of agents, and shown to be a promising metric for robotic exploration problems. In this work, we propose using BE principled exploration objective for systematically gen- erating datasets that provide diverse state space coverage in complex, continuous, potentially dighddimensional domains. To achieve this, extend the notion of BE to continuous settings, derive tractable k-nearest neighbor estimators, provide theoretical guarantees for these estimators, and develop practical reward functions that can be used with standard RI. methods to learn BBB-maximizing policies. Us- ing standard MuJoCo environments, offline RI. algorithms for a variety of downstream tasks on datasets generated using BE, Rényi, and Shannon maaropy-maximiiing policies, well the SMM and RND algorithms. We find that offine RL algorithms trained on datasets col- lected using BE outperform those trained on datasets collected using Shannon tropy, SMM, and RND on all tasks considered, and on 80% of the tasks compared datasets collected using Rényi entropy.
recently proposed for discrete settings
experimentally compare the performance
INTRODUCTION
Reinforcement learning (RL) methods can successfully solve challenging tasks in complex envi- ronments, even outperforming humans variety of cases (Mnih al. 2015 Silver et al. 2018). However, due to the online nature of standard RL algorithms and their reliance on informative, often hand-engineered reward signals, RL methods are typically sample-inefficie
and lack generalizabil- ity to new tasks. Offline RL (Levine et al.) [2020] Prudencio et al., (023) is an alternative approach that applies RL-based techniques to train policies entirely offline using static datasets of trajectories collected from the target domain. The key innovation is that a single offline dataset can be relabeled with a variety of different reward functions, enabling reuse of datasets learn a variety of down- stream tasks. This paradigm magnifies the importance of learning to generate datasets with diverse coverage of the state space hopes of covering regions that correspond to a wide a variety of down- stream tasks Yarats et al. (2022). The design of existing algorithms for dataset generation (Pathak)
al. 2017 Eysenbach et 2018 Lee al. 2019 Burda et 2019 Liu Abbeel 2021; Yarats 2021 relies on uncertainty metrics, such as entropy, to quantify the quality and control the variety state space coverage. This renders the choice of uncertainty metrics critical to the diversity of datasets that can be achieved. While Shannon entropy (SE) and Rényi entropy (RE) have been widely used as exploration objectives in RL. Hazan et al. (2019): Liu & Abbeel (202T): Yarats et al. 2021); :ohang et al. (202T): Yuan et al. (2022). the variety of datasets that can be achieved using them is limited. In the SE case this due the fact that SE provides just single objective and thus single notion of optimal coverage (see Fig In the RE case, though its parametric form
et al.
provides a variety of notions of coverage, this coverage remains partial (see Figure 2b) and comes
"Equal contribution.

---

Published as a conference paper at ICLR 2025
Agent
Probasdlty Perception
Range of Driropies
Detanet
Offine (CQL) Performance
1000
StreronAgent
Wandard Seeeit
Mtrnbard Marnterd pniwaay
HpMO portal oageofort
1.1 Un19000 coNtaao
Penglagent
#amnland
1al lange orgegias
diie coverage
Shannon
Remyt
Figure (Left) Comparison of Shannon entropy, Rényi entropy, and behavioral entropy (ours) and their effects on datasct generation, shown in PHATE plots, when used as an exploration objective. (Right) Perfor- mance comparison of an offline RL algorithm (CQL) for three downstream tasks datasets generated using Shannon, behavioral entropy (ours), and Rényi entropy for the parameter 1.1 shown in the left-hand figure.
the cost of instability arising due to its inherent discontinuity in parameter space (Suresh et al. 2024) and poor coverage for many parameter values (see Figure Tp. Developing a family of exploration objectives overcoming these issues remains an open question.
Recently, Suresh al. (2024) proposed behavioral entropy (BE), novel generalization of classi- cal entropies that composes SE with the probability weighting functions widely used in behavioral economics to model human decision making (Dhami 2016). In Suresh et al. (2024), the authors rig- orously established two key facts: (i) BE is a valid generalization of the classical notion of entropy, and (ii) BE provides the most general notion of entropy to date in the sense that its parametric form captures a wider range of valid generalized entropies than existing parametric families of entropies, such RE. The first property guarantees BE is smooth in its parameter and attains its maximum the uniform distribution, which is critical for exploration and data generation applications aiming for uniform coverage. The second property stems from BE's definition in terms of probability weight- ing functions (Prelec, 1998), the use which allows it to achieve a broader range of entropies than comparable methods (see Figure T. As Suresh et al. (024) experimentally demonstrated on robotic exploration problems, when BE is used as an objective this flexibility induces diverse exploration policies ranging from those providing coarse and widespread coverage of the environment to dense and focused coverage, achieving the best performance and the widest variety of coverages in discrete probability spaces compared with SE and RE. Together, these results establish BE as principled al- ternative to existing exploration objectives that provides a rich variety of exploration behaviors and state space coverage. However, the current formulation of BE restricted to discrete probability distributions and the experimental evaluation provided in Suresh et al. (2024) limited to classical, Glanning-based solutions to discretized, two-dimensional robotic exploration tasks. This impedes the applicability of BE to more complex problems.
In this paper we propose using BE principled exploration objective for systematically gener- ating datasets that provide diverse state space coverage in complex, continuous, potentially high- dimensional domains. We hypothesize that using BE in this way will lead to superior offline RL performance on BE-generated datasets compared with objectives providing an inferior diversity of coverage, particularly SE and RE, but also the widely used Random Network Distillation (RND) (Burda et al. 2019) and State Marginal Matching (SMM) (Lee et al., 2019) algorithms. There are three primary challenges to using BE in this way: (i) continuous-space version of BE must be for- mulated; (ii) principled estimators for BE that enjoy theoretical guarantees while remaining tractable in continuous, potentially nigh-dimensional settings must be developed; (iii) practical RL methods for training BE-maximizing policies must be derived. We address all three of these challenges in this work, then the resulting RL method to generate datasets providing a rich variety of state space coverage for subsequent offline RL. We experimentally confirm our hypothesis, demonstrat- ing that BE-generated datasets lead to superior offline RL performance over SE, RE, RND, and SMM datasets, and that offline RL methods enjoy better data- and sample-efficiency when applied to BE- and RE-generated datasets compared with existing benchmarks.
2

---

Published as a conference paper at ICLR 2025
H
H
HB
0.5
H
ml fx))
0,3
-]
-1
6000
0.3
10
0.0
P
(a) Perception of normal distribution (O 1)) un-
(b) Diversity of Shannon, Rényi and BE values as fumction of Bernoulli trial parameter p. Behav- ioral entropy H captures entire behavior spectrum from overvaluing uncertainty (light blue, a = 0) to highly undervaluing uncertainty (dark blue, o >> 0). Rényi. H² captures the former (light red, = 0), but cannot capture the latter. Dotted red curve shows H as q + oc. Shannon, H is dotted black curve.
der Prelec weighting function. Black dotted line
shows standard normal density, f. For 0 the per- ceived density is uniform, indicating over-weighting of uncertainty over entire support of f- For a > 0 the perceived density approaches a step function around the mean, indicating under-weighting of certainty near f's tails.
recovers original f.
Figure 2: Visualizations of probability weightings (left) and superior expressiveness of BE (right).
Our main contributions are:
Behavioral entropy estimation in continuous spaces. We propose a version of BE applicable to continuous probability distributions, derive k-nearest neighbor (k-NN) estimators for BE with general probability weighting functions, and provide convergence guarantees and probabilistic bounds characterizing the bias and variance of these estimators.
Exploration and data generation via RL-based BE maximization. We derive practical BE- maximizing exploration objectives and experimentally illustrate their effectiveness to generate datasets with diverse levels of state space coverage unsupervised RL settings.
Offline RL performance on BE-generated data. We experimentally evaluate the performance of offline RL algorithms for a variety of downstream tasks BE, RE, SE, RND, and SMM datasets. We find that BE datasets lead to superior offline RI performance over SE, RE, RND, and SMM, and that offline RI. methods enjoy better data- and sample-efficiency when applied to BE- and RE-generated datasets compared with existing benchmarks.
BEHAVIORAL ENTROPY IN CONTINUOUS SPACES
Behavioral Entropy. The various notions of entropy that have been studied in the literature since the initial work by [Shanoon 1948) quantify the uncertainty inherent in a random variable by measuring how evenly distributed its associated probability density is over its support. Let X be discrete random variable over a finite set of M elements, and let denote its probability mass function (p.m.f.). Two classical and widely used entropies are the Shannon and Rényi entropies (Shannon, 1948 Rényi 1961), given respectively by
1 (x)
> log(p:)p..
(L)
H q
HR
log
P
q
0. 1. (2)
These entropy functionals, along with others such Tsallis entropy T'slllis 1988). belong the class of admissible generalized entroiies satisfying the first three Shannon-Khinchin axioms (see (Amigó al., 2018) for details). These axioms ensure that entropy functional is well-behaved by ensuring their continuity, stability with respect to addition or removal of known outcomes, and maximality of the uniform distribution.
As mentioned in the introduction, the recent work (Suresh et al.) 2024) proposed composing Shan- non's entropy with the probability weighting functions widely used in behavioral economics code human cognitive and perceptual biases. The composition of Shannon's 's entropy with Prelec's probability weighting function Prelec (1998) yielded BE, an admissible generalized entropy that
3

---

Published as a conference paper at ICLR 2025
provides tractable mathematical formalism for incorporating helpful human biases into uncer- tainty quantification via entropy. The probability weighting functions used to encode human biases defined as follows (see (Dhami 2016 Ch. 2.2)).
Definition 1. A function w [0. + [0, 1] said to be a probability weighting function if is
continuous, strictly increasing, satisfies w()) 0 and w(1) = 1, and has a unique, continuous, and strictly increasing inverse.
The most widely used probability weighting function, and that which was the focus of Suresh et al. 2024), is Prelec's probability weighting, given by
w<xx) e -=g0(oo a, > 0.
Prelec's ' function is smooth in both the probability and parameter space and has the ability to control the fixed point and shape of the weighting function (see (Prelec, 1998) Dhami (216) for details). Figure 2alistussss its effect on Gaussian density. Equipped with equation proposed and studied the following.
3 Suresh et al. (2024)
Definition 2. Letting w be as in equation behavioral entropy is given by
H (x) Σ w(p)) log(w(p:)).
(4)
The parameter a controls the shape of the perceived probability curve, enabling a wide range of
probability perceptions (see Figure 2ap and the resulting perceived entropies, as illustrated in Figure
2b
Under the condition that 3 el- TTaa) ai log(log(M)), equation 4 was shown in Suresh et al. (2024)
to belong to the class of admissible generalized entropies. With this conditioning, equation 3 3 alllww
control of the third fixed point of which is critical for ensuring BE remains an admissible entropy, whereas other probability weighting functions lack this property (Dhami 20T6) and do not generate meaningful, admissible generalized entropies.
Interestingly, it was shown (Suresh et al.) 2024) that using BE over other approaches led to sig- nificant acceleration of robotic exploration tasks as well as emergent search behaviors similar to breadth-fire first and depth-first search, depending on choice of a. These results indicate that BE holds promise as objective for a broad range of exploration tasks in complex environments, yet (urresh jet al. (024) only applied BE to discrete, binary random variables. pave the way for the applica- tion of BE to more complex problems, we now extend it to continuous settings.
Differential Behavioral Entropy.
this subsection we extend the definition of BE to that of differential BE, providing a novel entropy functional that is applicable in continuous, potentially high-dimensional spaces. Let f E A(X) be p.d.f. over X' C Rd, where N+ We first recall the differential versions of Shannon's 's entropy and Rényi entropy of order g, where > which are defined, respectively, by
0, # 1,
H - log L ("()dd
H - -000--x (o)()
(5)
(6)
1-9
We will use these expressions in our experimental comparisons below and thus include them here for easy reference. We next state the definitions of our continuous-spaces analogues of equation 4 Definition 3. For an arbitrary probability frigghiingfunctiin w. differential generalized behavioral entropy is given by
H B.B. BB, - - (o)
In particular, substituting Prelec's probability weighting from equation into equation differential behavioral entropy, given by
yields
B.o., (f) 33 - (- fx))
(8)
is important to note that, unlike Definition I for probability weightings in the discrete setting,
where [0. 1] + [0, 1] in the continuous setting w must be generalized to [0, - [0, oo) to accommodate arbitrary densities, f. Desirable structural properties of w are described in the detailed statement of Theorem in the appendix. We will henceforth abuse both terminology and notation by omitting "differential" when referring to differential entropies and by suppressing the dependence of equation 8 on a, 3 when these are clear from context.
4

---

Published as a conference paper at ICLR 2025
3 K-NEAREST NEIGHBOR BEHAVIORAL ENTROPY ESTIMATION
We next turn to the problem of BE estimation in continuous, potentially high- high-dimensional spaces. To accomplish this, derive k-nearest neighbor (k-NN) estimates of BE along the lines of the estimates studied in (Kozachenko & Leonenko) 1987, Singh et al., 2003 Leonenko et al., 2008 Sricharan et al. 2012 |ingh & Póczos ( and others for Shannon and Rényi entropy. The K-NN family of nonparametrie estimators enables estimation of arbitrary densities from a finite number of ii.d. samples in continuous and potentially high- sigh-dimensional settings, making them particularly well-suited to the RL context considered in the following section. Let f E A ((R") be a probability IR+ Let X1, X2...., Xn
density function (p.d.f.) over RRd where samples drawn from f- In (Loftsgaarden & Quesenberry 1965 Devroye & Wagner 1977) and subsequent works was established that, for suitably chosen and k, reasonable approximation of f is provided by the k-NN density estimator
f()) be n N+ i.i.d.
kT (d/22-11 1)
f(z)
nz Rd ( (x))
where (x) || (N)(2 is the Euclidean distance between and its kth nearest neighbor among (X1,..,, X,, and [(x) _oe fd is the gamma function. When X,, will write Rn, Rg.n (X,) for simplicity. A natural first approximation Shannon entropy of f is given by the plug-in estimator
(
- n
1
Σ log f(x.) Ri Ex,r() [- log f(X.)) - -- log f(xx) f(x)dr.
a
(10)
With this in mind, the naïve approach to estimating equation foor general w via
B. knn (ʃ) = -----11
71 Σ w w(((x)) log w (((())).
(11)
Since X; f(-), however, the estimator of equation
I| is biased, since
B (J) ~ Ex [-w(](X)) log w/ w(f(x.)) = J w(f(x)) log w/ w(f(x)) f(x)da. (12)
Dividing by the approximation f yields the alternative, importance sampling-correcd corrected estimator
B, k.n
(f) =
1
wl v((()))) log w (f(.))
(13)
f(X,)
f(x.) ( ((((x)))
w/ (f(x,)) - w( w(f(x)) log w w(((x))
f(x) dx.
f(x)
Though this importance sampling-corrected plug-in estimator will be biased for the same reasons detailed in (Singh et al. 2003 Thm. 8) for U (f), for large n and suitable & equation 13 provides reasonable estimator of equation 8 as characterized by the following results.
kn Theorem 1. Suppose that k = kn + oc, kn + O, and logn + as + co. Assume that w is Lipschitz, that I is absolutely continuous, and that there exist C > O such that c1 f(x) C2 for all x E X. Then BB (f) H B. (f) both uniformly and in probability.
The result follows from the strong uniform consistency of f Devroye & Wagner 1977).
Though asymptotic guarantees like Theorem are somewhat reassuring, in practice we will use finite k in our k-NN estimators and therefore need more fine- grained characterization of the bias and variance. Unfortunately, for finite the approximator H (f) remains biased n + oo due to the biasedness of for fixed k and the lack of known bias correction procedure for BE approximator f...B. (f). This contrasts with the situation for simpler estimators like those for Shannon and Rényi entropies, for which explicit bias correction terms are known (see (Singh al. (203); Leonenko et al.] (2008); [Singh & Póczos (2016)). Nonetheless, we establish probabilistic guarantees the bias and variance of our proposed BE estimator in the following main result.
5

---

Published as a conference paper at ICLR 2025
Theorem 2. Suppose and w satisfy suitable differentiability and continuity conditions. Then, for E >> 0, it holds with probability 1 that
(1) +0
(og(m/e) (+)
log n
ifd > 2,
E
B (5) 11 H.
k
(14)
O (
+0
log
log(n/e) b
=
Var (ft
(15)
The proof and a precise statement are provided in the appendix. Equipped with the N--N estimators and theoretical guarantees derived above, we next turn their application the RL setting.
BEHAVIORAL ENTROPY IN THE REINFORCEMENT LEARNING CONTEXT
In this section we leverage the k-nearest neighbor generalized behavioral entropy estimates devel- oped in the previous section for general probability weightings to derive a practical reward function that can be used in conjunction with standard RL methods to maximize behavioral entropy of an RL agent's 's state occupancy measure.
Behavioral Entropy as RI. Objective. State occupancy measure entropy has been used as exploration objective for RL in a wide range of previous works (Hazan et al. 2019 Liu & Abbeel 2021 Yarats al. 2021 /Zhang al. 2021; Yuan et. (022). In order to formally define behavioral entropy for state occupancy measures in this context, we first provide preliminary background on
Markov decision processes (MDPs). Let an average-reward MDP .M- (S, p,r) ) be given, where S is the state space, A is the action space, mapping state-action pairs (s, a) S A probability distributions p(-1s, a) E A(S) over the state space, and r : S x A + IR is the reward function. Given a policy S + A(A) mapping states to probability distributions over the action space, .M evolves as follows: timestep E N. the system in state 8t, action 0₃ ~ 7-(18)) is selected and executed, a reward r, = r(sr, az) is received, the state transitions according to 8141 p(-1st, at), and the process repeats. To each
S x - (SS) is the transition probability kernel
ri]. and the goal policy π is associated the long-run average reward (TT) = limm E D determine an optimal policy x² arg max, J(z).
Under mild conditions on the transition kernel and policy (see (Puterman 2014)), each policy π induces a state occupancy measure (+) ASS) capturing the long-run state visitation behavior induced by π over S. When S is continuous, for a measurable subset B C S we have dæ (B) = lim, Pst B). We will henceforth assume that each measure d_ has corresponding p.d.f. and abuse notation by denoting the value of this p.d.f at by ds).. [] State occupancy measure entropies can be obtained by directly substituting / = da and X' = S in the entropy definitions in equation 5 equation 6 and equation g In particular, the behavioral entropy induced by resulting from this substitution in equation g is given by
B.a.3 (d=) - e -- (- du (s)
(16)
We propose to use equation 16 as an exploration objective. To achieve this, we leverage the k-NN estimator of equation 13 developed in the preceding section to derive reward function r such that J(x) R H B,0 (dz) in the following subsection.
Behavioral Entropy Reward Derivation. We next build on the k-NN estimator of equation I3 to derive practical reward function such that ((T) ~ H B. (dz). Our derivation is similar to the reward derivations followed in (Liu & Abbeel, 202T;: Yarats et al. 202T) for Shannon entropy and (Yuan et al. 2022) for Rényi entropy. Once we are equipped with this reward, we can leverage existing RI. methods to learn behavioral mntrooy-maxxmiziin exploration policies. Let 81, 8n ~ d.)... Substituting f = dz and 8; = X, into equation for i = 1,..., n, we have that (st) =
'Similarly, и induces a state-actio action occupancy measure X₂ (s a) = d, (s) (als). In this work we focus on state occupancy measures, but all results and methods can be extended to apply to state-action occupancy measures in straightforward manner.

---

Published as a conference paper at ICLR 2025
kT(d/2 1)/nx//2 k.m. where Ri,,,n |s, (s:))l and (s) denotes the k-NN of s;
within { Recalling that w(x)) = -[- lug(xx))" we can write
(()r = = -8( (00-o((T((d/22)))-ooo
)
(17)
= e B(dlop Re, Da,n)"
(18)
where Di.,. log(A[(d]) (d/2 2 + 1)) + log(nnd/22 = log ((nx)22)
Substituting
equation 18 into equation gives
1 B.o.B (da)
1 Σ (st)
w dx (8:)) log w(da
(19)
k,n.
3
nTT 2
R k, B(dlog nt (dlog Ra + Dk
(20)
n
kl'dd2 1
1
Rd kn e -B(d D (dlog Ri,k, + Dkm))
(21)
11
28
R R A(dlog Reaen (dlog R,
(22)
12
where the approximate proportionality in equation 22 follows from the fact that, under suitable
conditions n, (see, e.g., Theorem T the contribution of Dr., the value of equation 21 is
negligible. Since E, k.n
L (dz)) H HB.a. (dzz by Theorems and 2 and since equation 22 is
approximately proportional to B. (dz), equation 22 suggests
a(s,a) (laa)==|s->> NNN -Nd (+)11)) (d || (s)()))
3(dlog||=-
(23)
as a suitable proxy reward for maximizing behavioral entropy in an RL context. For numerical stability, we follow Yaartts al.) 2021 Liu Abbeel, (021) by making the additional simplification of setting d = 1 and adding constant > 0 inside the logarithms to obtain
r(s, a) = ||s [s) -(lo0(000 (00xdlll-- +o)) (logíIls NN&((s)12 + tc))
(24)
A visualization of equation 24 with comparison to the SE reward function is provided in Fig. 13 in the appendix. Armed with this reward, any standard RL method can be applied to learn explo- ration policies approximately maximizing BE using equation [6 We illustrate its application in data generation for offline RL in the next section. We note that implementing the k-NN estimator in equation 24 can be computationally challenging in high dimensions for large & values due the well-known curse of dimensionality of suffered by k-NN estimators Beyer et al. (1999). To address this, in practice < 15 is selected and dimension reduction to a feature space of manageable di- mensions is performed before K-NN estimation is carried out, thereby limiting computational costs Liu & Abbeel (202T) Yarats al. (0221).
EXPERIMENTAL RESULTS
The experiments pre- sented in this section (i) provide qualita- tive insights into the state space coverage achieved by policies that maximize BE, RE, and SE, and (ii) examine the utility of BE-maximizing policies for performing offline dataset generation for subsequent offline RL compared with datasets generated using the SE and RE objectives and datasets generated using the RND and SMM algorithms. The state space coverage visualizations that present suggest that BE-generated
Environment
Task
BE
RE
SE
RND
SMM
Walker
Stand
990.38
988.93
954.93
947.89
496.09
Walk
904.66
878.20 288
895.89
735.77
409.46
Run
385.07
440.53
360.64
341.03
140.29
Quadruped
Walk
845.31
776.64
755.79
699.22
425.11
Run
522.32
490.75
490.46
490.66
275.38
Comparison of best offline RL performance across all datasets, training Table 1:
seeds, and offline RI. RI. algorithms.

---

Published as a conference paper at ICLR 2025
datasets achieve wider variety of coverage than RE- and SE-generated datasets, and that the RE objective is unstable as a function of 9 and provides poor coverage for > 1. We provide coverage visualizations for RND and SMM in the appendix. In our offline RL experiments, we demonstrate that offline learning on BE datasets leads to superior performance over SE, RND, and SMM datasets on all five tasks, and superior performance to RE datasets on four out of five tasks (see Table !
Experimental Sctup. For our experiments, we generated BE, RE, SE. RND, and SMM datasets for the Walker and Quadruped environments using the Unsupervised Reinforcement Learning Bench- mark (URLB) framework (Laskin et al. 2021 [ We subsequently generated I-SNE plots (Hinton & Roweis 2002) and PHATE plots (Moon et al. 2019) from the BE, RE, SE, RND, and SMM datasets to visualize their varying state space coverage. Finally, we performed offline RL. training on all datasets using the Exploratory Data for Offline RL (ExORL) framework YYarats et al. 2022 We emphasize that the datasets we generated contained just 500K elements, only 5% as many as the 10M-element datasets considered in the ExORL framework (Yarats et al. 2022). and that per- formed just 100K offline training steps, only 20% of the 500K performed in ExORL Despite these limitations, achieved comparable performance to that achieved in (Yarats et al. 22022,, indicating that using BE-generated data for subsequent offline RL leads to significant improvements in both data- and sample-efficiency.
Dataset Generation and Visualization. For dataset generation, we used the Active Pre-Training (APT) algorithm (Liu & Abbeel, (02T) implemented in the URLB framework to maximize BE using the reward proposed in equation 24 for various val- ues of a, RE using the reward proposed in Zhang jet al.] (202I) for various values of q. and SE using the default reward from Liu & Abbeel (202)) Specifically, for behavioral and RE we considered a E 00.2, and q E {0.2, 0.5, 1.1, 3.0, 5.0). To ensure admissibility of the behavioral entropies we considered, we used the conditioning 3 e(l-a) (Mg()((M)) from Suresh al. (2024), where
0.2
3.0, 3.0,5.00
0.9
11
M is the dimensions of the representation space. See the discussion following equation f Section 2 for details. For each of the and values, as well as for SE, we trained APT on the correspond- ing reward for 500K pretraining steps on both the Walker and Quadruped environments, collecting the resulting trajectories to form our datasets. This re- sulted in 17 datasets for each environment, for total of 34 datasets. To ensure a fair comparison across entropies, for the APT hyperparameters we used the default URLB pretraining hyperparameter val- ues across all datasets (see Table in the appendix). For the datasets generated using the Random Net- work Distillation (RND) (Burda et al.) 2019) and State Marginal Matching (SMM) Lee et al. (2019) algorithms,, we similarly trained for 500K pretrain- ing steps and for consistency we used the same RND and SMM hyperparameters considered in URLB.
3.0
(a) PHATE plots for BE for Walker.
0.2
0.5
9
9
2.0
(b) PHATE plots for Renyi for Walker. Coverage for q 3.0, 5.0 similar q 2.0.
Figure 3: PHATE plots for Walker tasks.
To provide qualitative insight into the state space coverage of the SE, RE, and BE datasets, we generated two-dimensional t-SNE (Hinton & Roweis 2002) and PHATE (Moon et al., (019) plots of the trajectories they contain We also generated t-SNE and PHATE plots for the RND and SMM
httte://daaaab..co//lreeeerooo http:///ww...oorarrooooooooeeeer benchnark
"OOK representative samples from each dataset were gathered uniformly. totaling 170K samples for each domain from 17 datasets. t-SNE and PHATE were implemented on this aggregated eleK-element dataset to ensure uniformity in projections for each dataset and then correspondingly represented individually for clarity.
8

---

Published as a conference paper at ICLR 2025
datasets, pictured in Figure 12 in the appendix. While t-SNE has been previously used to visualize 2021), to our knowledge this is the first time PHATE plots have
RL trajectory data (Zhang et al. been used to visualize such data. PHATE tends to better retain global structure such as the temporal nature of trajectory data, while t-SNE obscures it (Moon et al. 2019). Figure indicates that while both BE and RE- RE-generated datasets provide more flexible levels of state space coverage than SE, BE-generated datasets achieve a wider variety of coverage than RE. See the appendix for t-SNE and PHATE plots for the remaining datasets. Importantly, these plots indicate that the RE objective unstable as a function of & and provides poor coverage for q > 1, while the level coverage provided by BE varies smoothly in a. We provide experimental support for this in the following section, where highly unstable offline RL performance on RE datasets and the contrasting stability on BE datasets is illustrated in Figure
Offi Danieet
3M
C-33
22.1+32
18.0-12
Othe Ageritve
L5
1-
(60
1-11
Waber Stend
Datase
leyio
Ortine Aegorttir
Detavt
600
1-
Figure 4:
Comparison of offline RL performance over the entropy objectives used in dataset generation. Plots show mean and standard deviation over five seeds. Dotted line shows performance of RL policy trained online until approximate optimality.
Offline RL Experiments. We compared the TD3, CQL, and CRR offline RL algorithms (Fujimoto et al. 2018 Kumar et al., 2020 Wang et al. 2020) implemented in the EXORL framework on the datasets generated described above for all eight a values and for g {0.2, 0.7, 0.9. 1.1 We omitted offline RL training for RE datasets with E (2.0, 3.0, 5.0} after observing in initial trials that performance
better than for q 1.1 and typically worse (see poor performance

---

Published as a conference paper at ICLR 2025
on q = 1.1 datasets in Figure To gain insight into the effect of using larger datasets, also considered a 3M-element SE-generated dataset. Altogether we considered 17 datasets: eight BE, five BE, two SE, and one cach for RND and SMM. On the Walker datasets Walk, and Run tasks, while on Quadruped we considered the Walk and Run tasks. For each of the 5 x 17 85 task-dataset combinations, training steps, evaluating performance every 10K training steps. For each of the 255 task-dataset algorithm combinations, we repeated this training process for a total of 5 different seeds, resulting in our training 1275 offiie RL policies altogether. To ensure a fair comparison across all entropies, used default EXORL hyperparameter values across all datasets (see Table 3 in the appendix).
considered the Stand,
trained each of TD3, CQL, and CRR for 100K offline
As summarized in Table T offline RL training on BE-generated datasets leads to superior perfor- mance over SE-, RND- and -MM--grrrrtte datasets all five tasks we considered, and superior performance to generrtee datasets on four out of five tasks. Figure / a detailed overview of the experimental results for a {0.2, 0.5, 0.7, 0.9, 1.5) and q E (0.2, 0.5, 0.7, 0.9. 1.1} (com- plete results for all a values are shown in the appendix). This figure illustrates that Be-gnnerated datasets lead to significantly better performance over the other methods on Quadruped Walk and Walker Walk for the best-performing a values, while offline RL performance on RE datasets for the best-performing values of q is only slightly below that of BE datasets in Quadruped Run and Walker Stand. These trends hold across all algorithms for each of the tasks. On Walker Run, the rest-performing RE parameter clearly leads to superior offline RL performance over both BE and SE datasets in the TD3 and CQL trials, but performance on BE datasets is again better than on RE datasets in the CRR trials. Performance on SMM datasets is clearly inferior across all tasks consid- ered. Performance on RND datasets is inferior on Walker tasks, but is almost competitive with BE on Quadruped tasks. Interestingly, the 3M-element SE datasets lead to strong downstream performance on Walker Stand and improved downstream performance over the 500K -elemeeeeent SE datasets on the Walker Stand and Run tasks, but the 3M-element SE datasets actually lead to worse performance compared with the 500K-element SE datasets on both Quadruped tasks and TD3 performance on Walker Walk. Overall, best offline RL performance on BE-generated datasets clearly exceeds best performance on RE datasets on 13 out of 15 alsk-algori combinations and best performance on SE, RND, and SMM datasets 15 out of 15 task-algorithm combinations.
We observed sensitivity of performance to parameters a and q as well as choice of offline RL algo- rithm. Regarding the latter, notice in Figure 7 in the appendix that on Walker Walk the SE-generated datasets are competitive with the average and best-performing BE datasets in the TD3 trials, while BE datasets significantly outperform SE ones in both the CQL and CRR trials. On Quadruped Run, on the other hand, performance on BE and SE datasets remains roughly the same across all algo- rithms, while average RE performance is significantly worse (see appendix for average performance plots for all tasks). These results suggest that offline RL algorithm performance depends in a com- plex way on the choice of exploration objective used in dataset generation. Werf-perf,,,,ng flexible objectives such as BE- and to a lesser but still significant extent, RE therefore merit additional study tools for dataset generation for offline RL.
CONCLUSION
In this work we developed the theory and practice of behavioral entropy in continuous spaces, en- abling the incorporation of human cognitive and perceptual biases into uncertainty perception in complex, real-world scenarios. We first developed and analyzed k-nearest neighbor estimators for BE with general probability weightings. We subsequently derived practical reinforcement learning- based methods for maximizing BE under Prelec's probability weighting in sequential decision- making problems, enabling the use of BE as a state space exploration objective in the RL context. Leveraging these algorithmic developments, we experimentally investigated the utility of BE for dataset generation for offline RL. Our experiments demonstrate that BE- BE-generated datasets lead to superior offline RL performance over both Shannon and Rényi entropy generatgeee datasets, that BE is stabler and therefore easier to use an exploration objective compared with Rényi entropy, and that BE-generated datasets lead to improved data- and sample-efficient ing methods. As limitation, due to the computational burden of dataset generation and offline RL training for nontrivial variety of a and g values additional environments and offline RI. algorithms were not considered. These limitations are important directions for future work.
for offline RL over exist-
10

---

Published as a conference paper at ICLR 2025
REFERENCES
José M. Amigó, Sámuel G. Balogh, and Sergio Hernández. A brief review of generalized entropies. Entropy, 20(1 (1),1 2018.
Kevin Beyer, Jonathan Goldstein, Raghu Ramakrishnan, and Uri Shaft. When is "nearest neighbor" meaningful? In Database Theory-ICDD 7th International Conference Jerusalem, Israel, January 10-12, 1999 Proceedings 7, pp. 217-235. Springer, 1999.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In Seventh International Conference on Learning Representations, pp. 1-17, 2019.
Luc Devroye and Terry J Wagner. The strong uniform consistency of nearest neighbor density estimates. The Annals of Statistics, pp. 536-540, 1977.
Sanjit S Dhami. The foundations of behavioral economic analysis. Oxford University Press, 2016.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Divensity all you need: Learning skills without reward function. arXiv preprint
2018.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor- critic methods. In International conference machine learning, pp. 1587-1596. PMLR, 2018.
Elad Hazan, Sham Kakade, Karan Singh, and Abby Van Soest. Provably efficient maximum entropy exploration. In International Conference Machine Learning, pp. 2681-2691. PMLR, 2019.
Geoffrey E Hinton and Sam Roweis. Stochastic neighbor embedding. Advances in neural informa- tion processing systems, 15, 2002.
Lyudmyla Kozachenko and Nikolai N Leonenko. Sample estimate of the entropy of a random vector. Problemy Peredachi Informatsii, 23(2):9-16, 1987.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Michael Laskin, Denis Yarats, Hao Liu, Kimin Lee, Albert Zhan, Kevin Lu, Catherine Cang, Lerrel Pinto, and Pieter Abbeel. Urlb: Unsupervised reinforcement learning benchmark. arXiv preprint arXiv:21 10.::2/00011 1911, 2021.
Lisa Lee, Benjamin Eysenbach, Emilio Parisotto, Eric Xing, Sergey Levine, and Ruslan Salakhutdi- nov. Efficient exploration via state marginal matching. arXiv preprint 1xXiv:/906.05:00 2019.
Nikolai N Leonenko, Lue Pronzato, and Vippal Savani. class of rényi information estimators for multidimensional densities. Annals of Statistics, 36(5):2153-2) 2182, 2008.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto-
rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643, 2020.
Hao Liu and Pieter Abbeel. Behavior from the void: Unsupervised active pre-training. Advances in Neural Information Processing Systems, 34: 14:19999-18473, 2021.
Don O Loftsgaarden and Charles P Quesenberry A nonparametric estimate of multivariate density function. The Annals of Mathematical Statistics, 36(3): 1965.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 51 5177554))559-5333 2015.
Kevin R Moon, David Van Dijk, Zheng Wang, Scott Gigante, Daniel Burkhardt, William S Chen, Kristina Yim, Antonia den Elzen, Matthew J Hir, Ronald Coifman, et al. Visualizing structure and transitions in high-dimensional biological data. Nature biotechnology. (7(1)) 1492, 2019.
11

---

Published as a conference paper at ICLR 2025
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity- exploration by self-supervised prediction. In International conference on machine learning, pp. 2778-2777 2787. PMLR. 2017.
Drazen Prelec. The probability weighting function. Econometrica, pp. 497-527, 1998.
Rafael Figueiredo Prudencio, Marcos ROA Maximo, and Esther Luna Colombini. A survey offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Alfréd Rényi On measures of entropy and information. In Proceedings of the fourth Berkeley symposiun on mathematical statistics and probability, volume I: contributions the theory of statistics, volume 4, pp. 547-562. University of California Press, 1961.
Claude Elwood Shannon. A mathematical theory of communication. The Bell system technical journal, 27(3):379-422 1948.
David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Mare Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, et al. A general reinforcement learning algorithm that masters chess, shogi, and go through self-play Science, 362(6419)::140) 1144, 2018
Harshinder Singh, Neeraj Misra, Vladimir Hnizdo, Adam Fedorowicz, and Eugene Demchuk. Near- est neighbor estimates of entropy. American journal of mathematical and management sciences, 23(3-4):3013211 2003.
Shashank Singh and Barnabás Póczos. Finite-sample analysis of fixed-k nearest neighbor density functional estimators. Advances in neural information processing systems, 29, 2016.
Kumar Sricharan, Raviv Raich, and Alfred O Hero. Estimation of nonlinear functionals of densities with confidence. IEEE Transactions on Information Theory, 58(7):1135-4153 2012.
Aamodh Suresh, Carlos Nieto-Granda, and Sonia Martínez. Robotic exploration using generalized behavioral entropy. IEEE Robotics and Automation Letters, 90))8011-8008 2024.
Constantino Tsallis. Possible generalization of boltzmann-gibbs statistics. Journal of statistical physics, 52:479-487, 1988.
Ziyu Wang, Alexander Novikov, Konrad ZoIna, Josh S Merel, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768- -778, 2020.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Reinforcement learning with pro- totypical representations. In International Conference Machine Learning, pp. 11920-11931. PMLR, 2021.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforeement learning. In ICLR 2022 Workshop on Generalizable Policy Learning in Physical World, 2022.
Mingqi Yuan, Man-On Pun, and Dong Wang. Rényi state entropy maximization for exploration acceleration in reinforcement leamiin. IEEE Transactions on Artificial Intelligence, 4(5): 1164, 2022.
Chuheng Zhang, Yuanying Cai, Longbo Huang, and Jian Li. Exploration by maximizing rényi entropy for reward-free rl framework. In Proceedings of the AAAI Conference Artificial Intel- ligence, volume 35, pp. 10859-10867,, 2021.
Puning Zhao and Lifeng Lai. Analysis knn density estimation. IEEE Transactions on Information Theory, (9911)))771177995 2022.
12

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 PROOFS
Fix probability weighting function and let g(v) log log(w())))(()) Fix p.d.f. f A(X),
where X' R is compact. Fix n,k N. and let Xi..... Xn f()). Recall the definition of f from equation Define
Let μ denote the Lebesgue measure and (x) = fx' E Rd || - æ|12 < r).
BB)(() = /
1 ((f(=))
(25)
n
n
H²)(() Σ
(Fg(
log(w
Σ s(f(xi)),
(26)
f(x)
71
(() = Σ
log(( (og
1
log(
Σ g(f(x.))
(27)
==1 f(x)
Our goal is to establish bound on the error
E
HB. HB:
(28)
In general, for finite k, even as n + oc the approximator B,0 (f) will remain biased due to the biasedness of f for fixed lo and the lack of a known bias correction procedure for our BE approx-
imator F R. (/). This contrasts with the situation for simpler estimators like Shannon and Rényi entropies, for which explicit bias correction terms are known (see Singh et al.] (2003); Leonenko et al. SSngh & Póczos (2016)). Nonetheless, in Theorem 2 are able build existing
results to establish a probabilistic bound on equation 28 We first recall the following result.
Lemma ((Singh & Póczos 2016)). Suppose that, for some & (0, 2). is E-Hölder continuous and strictly positive on X. Suppose furthermore that there exists function fo X + ER and
constant f* such that fx(x) S <Sn.(z)) fy)dd//ll (Bt/:/((BBBrr (x)) I f co, for all E X.r
(0, vdl, and assume that So e ===x* ==)x*((x)dx 00. Then
R. (()) -HB.. (f) =
k
(29)
Var (H (H* = (
(30)
The proof of this result follows directly from that of (ingh & Póczos 2016 Thm. 5) due to the fact that HB.w() is an unbiased estimator of BB.(()). Also note that the variance bound can be trivially strengthened to apply to B. (f) due to the fact that the latter is simply the sample average of n. i.i.d., bounded random variables:
Corollary 1. Under the conditions of Lemmal
Var
( -
It remains to characterize equation 28
We first recall another useful result from the literature. For O, let A (8,r) denote the covering number, the minimum
given set C X, radius r, and number of balls of radius r needed to cover S. Let I1-Ilopp denote the operator norm.
Lemma 2 (Zhao & Lai 2022 Suppose there exist C1, C2, C3. > 0 and B E (0, 1] such that the following conditions hold:
[V
(i)
f(x) & <G:
(i1)
SG; (iti)
VI > 0, P(F(2) < S C31
f(z)
No
(iv) N ((xff(()) m),r) s mdrdd for some y > 0 and all m 0.
Then, for e > 0, it holds with probability (w.p.) 1 - that
1 1 log v + log(n/s)
ifd 2,
sup f(x) f(2) =
(31)
o log n. + Ing(n/c)
ifd= 1,2.
13

---

Published as a conference paper at ICLR 2025
We now in position to prove main result.
Theorem 2. Suppose f satisfies the conditions of Lemmas 7 and 2 ous. Then, for E > O, holds w.p.
Assume w is Lipschitz continu-
E that
0
1 +0
( log
log log(n/e) Ing(n/e) b
idd 2,
E H (f)
HH (f)
k
(32)
O (4))
+0
l: log
ifd=
Proof. First notice that
E A [ (5) &
k. (1) A. (1) + TE [H ( (B) (33)
The second term can be bounded using Lemma I so it just remains to bound the first term. Recall that X' is compact, is bounded strictly away from 0 on X. and w is Lipschitz. We therefore have that g is the product of Lipschitz, bounded functions and therefore itself Lipschitz on its domain. Let K denote the minimal Lipschitz parameter of g. Rewriting equation 33 in terms of g, obtain
E Pan (f) HBB." (
E
[o(f(x))) [(((x))]
(34)
E - -((((x))
(35)
[[o(f(xx)) o(f(x)))
(36)
KI
[(xx)) s(xi)
(37)
S KI sup [(x) s(0)
(38)
where equation a follows from the fact that the XI,..., are iid. An application of the law of total probability and Lemma 2 to the last term completes the proof.
14

---

Published as a conference paper at ICLR 2025
A.2 HYPERPARAMETERS
Optimization hyperparameter
Value
replay buffer capacity mini-batch size agent update frequency discount factor (7)
10s
1024
2
0.99
optimizer
Adam 10-1
learning rate critic target rate (7)
0.01
exploration stddev clip exploration stddev value
0.3
0.2
APT hyperparameter
forward net architecture inverse net architecture representation dimension
(512 + dim( - 1024 + 512 ReLU MLP
x2 512) - 1024 - dim(AA ReLU MLP
512
NN approximator
average top A in NN
True
RND hyperparameter
representation dimension
512
predictor, target network architecture
dim(S) + 1024 + 1024 - 512 ReLU MLP 5
normalized observation clipping
SMM hyperparameter
skill dimension
4
skill discriminator learning rate
10-3
VAE learning rate
10-2
Table 2: Data generation hyperparameters
Shared hyperparameter
Value
replay buffer capacity
10
mini-batch size agent update frequency discount factor (y) optimizer number of hidden layers
1024
2
0.99
Adam
hidden dimension
1024
learning rate critic target rate (r)
10
0.01
training steps
105
TD3 hyperparameter stddev clip
0.3
CQL hyperparameter
COL-specific a
0.01
Lagrange
False
number of sample actions
CRR hyperparameter
number samples to estimate transformation
10
indicator
Table 3: Offline RL hyperparameters
15

---

Published as a conference paper at ICLR 2025
A.3 ADDITIONAL OFFLINE RL EXPERIMENTS
Offi Calent
Key d-
100
Ofiiw 14 Aggriti
office Datoect
Shamma
82
66.e-50
Fen de
Akgothee
Cffics fonsoe
Staran
E: 2-2
eath
cx00
Offine fLA gartiom
Cftics Danser
Staran30X
22 2-
bey, c-03
Offine ALA gertamn
Offire Dateet
800
Offine Akgorther
Figure 5: Offline RL results for all a and values evaluated. Initial trials showed E (.00, led performance no better (and usually worse) than g 1.1, offline RL training for these values was not performed.
16

---

Published as a conference paper at ICLR 2025
Gardrgged
Qurdroped
Number of Offi ne Trainin Steps
Number of Offinn
800
803
Trairing Shaps
Shannon 1O0K
Sharmon 10CK
Eptooda Romord
Sharmen
Epuodl Rewerd
Svernen
mis
800
200
200
Offine AL Alcortther
Offtine Algorithm
Wallier
Walker
Offiine Trennng Steps.
Offine
Twinin Stegs Shannon
Shannon
Eptodk Rowrrd
Stursar 200c
Tppoodde Bowand
Sturnar 2000
criine
crline
400
400
200
200
M5
Offline RL gorlthim
Number
000
400
100
Offine RL Alporithm
Figure 6: Ablation result comparing the effect of performing 100K vs. 200K offline RL training steps on a 3M-element dataset generated using Shannon entropy as exploration objective. These results suggest that performing additional offline RL training has only marginal effect down- stream task performance.
17

---

Published as a conference paper at ICLR 2025
Shar
Yalee Surd
Shannor
Stoneer
30
45
Offire Celend
Offic Aggntte
Figure 7: Offline results averaged over all a, q values.
A.4 QUANTITATIVE COVERAGE EXPERIMENTS
18

---

Published as a conference paper at ICLR 2025
lypersspeee Radii
0.8
BE,
0.6
BE, d
BE, d= 5.0
Renyl, 0.2
r
Ranyl, 000.
0.4
Renyl. q 007
Renyl, 0.9
Renyl, Q m.1
Renyl. 2.0
Shannon
SMM
100000
200000
300000
400000
500000
T
(a) Volumetric coverage for data generation on Walker
Hypersphere Radii
0.8
BE, 1.5
BE, 2.0
0.6
BE, a
BE, a= 5.0
Ranyi, =.2
r
Renyi,
0.4
Renyi, q
Renyl, -
Ranyl, -
Renyl 22.0
Shannon
SMM
100000
200000
300000
400000
500000
T
(b) Volumetric coverage for data generation on Quadruped
Figure 8: Visualization of evolution of smallest hypersphere radius (normalized by the maximum radius achieved over all datasets) over the course of data generation training step for the Walker and Quadruped domains. We refer to this coverage metric volumetric coverage. Welzl's algorithm was used to determine the radius r. 10K data points were sampled uniformly from every 50K iteration increment and cumulatively added get total of 100K samples for 500K iterations. Volumetric coverage varies considerably with the choice of parameters and data generation methods. For the range of parameters and g that considered, BE exhibits higher volumetric coverage than RE on average on the problems under consideration. SE volumetric coverage was about average, while RND and SMM volumetric coverage differed sharply across domains: RND outperformed all other data generation methods on Walker, while SMM was not far behind; on Quadruped, on the other hand, both underperformed. Values of q 1 enjoy higher volumetric coverage for RE both tasks, while values of a < enjoy higher volumetric coverage for BE on Quadruped; since q 1,a 1 tend correspond superior downstream offline RI performance suggests volumetric may be positively correlated with performance on downstream tasks. We also note note that RE with
Fig. (5). this
in general shows both poor volumetric coverage and poor qualitative coverage (PHATE and (-SNE), which might correspond to its poor performance in all tasks. These relationships conclusive, however, and further investigation is needed.
19

---

Published as a conference paper at ICLR 2025
A.5
ADDITIONAL QUALITATIVE VISCALIZATIONS
C
0.2
= 0.5
0 0.7
0.9
HS
01 1.5
2.0
Q 3.0
= 5.0
(a) TSNE plots for GBE for Walker
0.2
0.5
0.7
4
0.9
q = 1.1
q
2.0
(b) TSNE plots for Renyi for Walker
Figure 9: TSNE plots for Walker
20

---

Published as a conference paper at ICLR 2025
a = 0.2
a = 0.5
a 0.7
a = 0.9
HS
Q = 1.5
Q 3.0
a = 5.0
Q = 2.0
(a) PHATE plots for GBE for Quadruped
= 0.2
q = 0.5
q -=0.7
q = 0.9
q = 1.1
q = 2.0
(b) PHATE plots for Renyi for Quadruped
Figure 10: PHATE plots for Quadruped
21

---

Published as a conference paper at ICLR 2025
0.2
0.5
-= 0.7
0.9 0.9
1.111
2.0
(a) TSNE plots for GBE for Quadruped
0.2
0.5
0.7
q 1.1
2.0
TSNE plots for Renyi for Quadruped
Figure 11: TSNE plots for Quadruped
22

---

Published as a conference paper at ICLR 2025
SIM
RND
SMM.
RND
(a) t-SNE plots for RND and SMM for Walker
(b) t-SNE plots for RND and SMM for Quadruped
SMM
BND
SMM
RND
(c) PHATE plots for RND and SMM for Walker
(d) PHATE plots for RND and SMM for Quadruped
Figure 12: Qualitative visualization of SMM and RND for data generation
A.6 BE REWARD FUNCTION VISUALIZATION
5
1.00
0.75
4-
0.50
0.25
3-
r(s,a) a)
0.00
2
0..5
0.50
0.75
1.00
2
4
8
10
ss - NN (s)ll2
10"
Figure 13: Visualization of the BE reward function equation 24 by varying the parameter with conditioned according to (4) from (Suresh et al. 2024) with M 512, denoting the representation dimensions. These visualizations highlight the diversity and variety of rewards that be obtained by a BE--maximizing reward function (blue region) as compared to the single SE objective (dotted black line).
23


Reviews:
[{"review_id": "MAcfriye5a", "reviewer": "Reviewer_esfg", "summary": "This paper studies dataset generation for offline RL. More specifically, this paper extends recently proposed behavioral entropy (BE) for discrete settings to continuous tasks, and designs estimator and RL algorithms for exploration and data generation. The paper compares offline RL performance on BE-generated data with other entropy-based data generation and shows that the proposed method outperforms other baselines in the Walker and Quadruped environments.", "questions": "There are some questions and concerns, which I have outlined in the previous section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is well-motivated and theoretically grounded.\n2. The writing is clear and easy to follow.", "weaknesses": "While well-motivated, I have some questions about this work:\n\n1. The authors propose a KNN-based estimator with convergence guarantee. I wonder in practice how $k$ is chosen and if KNN introduces huge computation overhead compared to other baselines. For instance, if SE and RE have the same computation budget as BE, will they collect more data and hence perform better?\n2. The experiments only generate 500K elements for each task, which might be not enough for the offline RL to learn a good policy. Can we test if the performance of BE, SE, and RE will scale with more data?\n\nMinor:\n\n1. While this paper focuses on the offline RL setting, I am curious about whether this entropy-based exploration can be applied in the online RL setting and how it performs compared with other online exploration methods.\n\nOverall, while this paper presents some interesting ideas, I am unable to recommend acceptance at this stage given the questions mentioned above. However, I would consider raising my score if the authors could address my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BWDkiIEL4m", "reviewer": "Reviewer_UCex", "summary": "This paper explores how to extend Behavior Entropy to high-dimensional continuous settings and leverages this as a metric for exploration to elicit high state space coverage. This exploration objective can be systematically applied to dataset generation for offline RL algorithms. The authors additionally derive tractable KNN estimators and practical reward functions that can be used in tandem with RL to learn an effective policy. They empirically show that generated datasets with Behavior Entropy outperform datasets generated using Shannon entropy and Rényi entropy with offline RL algorithms such as CQL on Mujoco tasks.", "questions": "- For BE, is there a clear selection procedure for $q$ empirically that would transfer across algorithms/tasks?\n- Currently, the approach relies on a k-NN estimator, which has limitations as the state dimensionality increases or the size of the dataset is large. As offline RL scales to be used with more realistic domains, these concerns will be more common place. Are there tradeoffs with using this as an estimator?\n- Along with the qualitative visualizations you have provided (e.g PHATE/tsne), could a quantitative analysis of the dataset state/action coverage be done to compare the different dataset generation approaches?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors have a novel contribution with the extension of Behavior Entropy to continuous domains as well as a tractable formulation using KNNs (with convergence guarantees) to define an reward function for RL. They show initial empirical validation on a subset of the Mujoco domains in the Walker and Quadraped tasks with three offline RL algorithms: CRR, CQL and TD3. Additionally, the visualizations of the dataset generation provide insights into what differentiates the different exploration approaches.", "weaknesses": "Some of the empirical results aren’t compelling such as the $\\alpha$/$q$ sweep done in Figure 4, where the gap from BE and Shannon/Renyi Entropy is minimal or within the margin of error for domains, which questions the advantages of the dataset generated. Additionally, it would be good to check the consistency of the dataset generation, where the dataset generation process is repeated for each $\\alpha$ or $q$, $N$ times and performance is aggregated across these generation attempts. Additionally, the domains evaluated are state based and a limited subset of the Mujoco benchmark. It would be good to see the clear benefits of BE-generated datasets across multiple tasks and algorithms. Additionally, there are a myriad of other exploration objectives people consider such as RND, Count Based Exploration, etc. It would be good to empirically compare against these approaches for the synthetic dataset generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kwmFVJmard", "reviewer": "Reviewer_B32C", "summary": "Entropy-based objectives are widely used to perform state space exploration in reinforcement learning (RL) and dataset generation for offline RL. In this paper, the authors borrow the idea of behavioral entropy (BE) from a prior robotics exploration paper and extend it to continuous space. This paper argues that, compared to prior entropy metrics, using BE as the RL objective enables more complete state coverage.", "questions": "What is admissible generalized entropies and why \"admissible\"  and \"generalized\" are important?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "State coverage is an important direction of unsupervised RL, which could benefit downstream task solving by learning useful skills, or, as in this paper, by generating a dataset for offline RL.\n\nRegarding originality, even though the idea of applying BE to exploration originates from a cited prior work, to the best of my knowledge, the idea of applying it to state coverage is novel. Further, the authors extend BE from discrete to continuous space and propose a knn-based measurement.\n\nRegarding clarity, the figures in the paper are well designed: Fig 2 clearly illustrates the property of BE under different values of $\\alpha$ (one of its parameter) and the differences between BE and prior entropy metrics. Meanwhile, the paper uses the same color for each entropy metric across the paper, which further eases the reading.", "weaknesses": "My main questions are about the evalution on state coverage and offline RL performance. Specifically:\n1. Could authors explain, in Fig3, why visualizing each experiment using separate plots is a valid metric of state coverage? IIUC, PHATE, t-SNE, and methods does some non-linear projection to project data to 2D, and this brings two problems when comparing these figures:\n* As state trajectories go through non-linear projection, the distance between points more reflect their similarity rather their true distance, thus making it hard to say more spanned points covers large state space (I may be wrong on this). Regarding this, one way may be to visualize trajectories of some points and see if they are indeed very distinct states. Another way I would like to see is to run experiments on some 2d domains so we can visualize without PHATE.\n* As each experiment is visualized separately, the PHATE may use different projections in different subfigures, making it hard to say points indeed spread out or the projection makes them look spread out. Regarding this, one more fair way is to draw PHATE plots on trajectories across all experiments to ensure they share the same projection.\n2. If we assume PHATE plot is a valid, I still have some questions:\n* I do not see singnificant difference between SE and BE for Quadruped coverage, is there any? If not, why that's the case?\n* In offline RL experiments, do you train each method until it converges? Authors mentions \"we performed just 100K offline training steps\", but I thought it's more fair to compare performance under convergence.\n* The authors motivates in the intro that better state coverage should lead to better offline RL performance, but in Fig 6, the $\\alpha$ leading to best coverage ($\\alpha$=5) typically has the worst performance across all alpha, why that's the case? Is the state coverage metric wrong or the assumption of better state coverage leading to better performance wrong or becase of other reasons?\n\nMinor suggestions on writing:\n1. In intro, authors motivates the usage of BE by only saying \"it's widely used in behavioral economics to model human probability perception\" and the results from Suresh et al. (2024). This feels void and doesn't explain why it could helps. There is no explanation why reweighting the entropy (making it more uniform or sharper) could lead to better coverage.\n2. L100, \"We derive practical RL methods\", it's more of a different reward/objective, as you still use an existing RL algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GTHAw2yuWl", "reviewer": "Reviewer_T8Fn", "summary": "The paper proposes to use behavioral entropy (BE), a generalized version of Shannon's entropy using a reweighting function, as reward to generate datasets and tests whether these datasets would be better suited for offline RL. \nThe behavioral entropy allows reweighting low-probable events and high-probable events, and it was used in Economics to explain human behavior. \nThe paper shows that data generated with maximizing BE can lead to higher downstream task performance in offline RL.", "questions": "- The intuition behind BE did not came across well. Also, why would one want a different weighting of high and low-prob. values?\n- The graphics look nice, but in the context of beh. generation and exploration, there is no parametric underlying distribution (like Bernoulli) but essentially, one is trying to achieve uniform distribution over the state-space. An illustration on how this exploration strategy really performs differently and how would be interesting.\n\n- how often does the $k$-NN tree for fast retrieval be rebuild? After every episode? \n- Since alpha<1 and q<1 work best, one could say that seldomly visited states (low probability) should give a higher weighting in the reward / entropy\n- How was alpha / q selected? \n- How does the method scale to high-dimensional input?\n\n## Post rebuttal update\nthe authors have improved the paper significantly and addressed most of my concerns. I am increasing my score to 6.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- analysis of behavior entropy on exploration\n- nice visualizations\n- continuous space estimators for BE", "weaknesses": "- another hyperparemeter introduced, not unsurprising that better performance can be achieved by tuning this parameter per environment\n- computationally heavy (because of k-NN that needs to be rebuild every time) and no mentioning of that\n- no comparison to simple baselines like RND\n- no improvements for strong offline methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)