Behavioral Entropy-Guided Dataset Generation for Offline Reinforcement Learning

Authors: Wesley A. Suttle, Aamodh Suresh, Carlos Nieto-Granda
Primary area: reinforcement learning
Keywords: reinforcement learning, offline reinforcement learning, exploration, entropy

Abstract:
Entropy-based objectives are widely used to perform state space exploration in reinforcement learning (RL) and dataset generation for offline RL. Behavioral entropy (BE), a rigorous generalization of classical entropies that incorporates cognitive and perceptual biases of agents, was recently proposed for discrete settings and shown to be a promising metric for robotic exploration problems. In this work, we propose using BE as a principled exploration objective for systematically generating datasets that provide diverse state space coverage in complex, continuous, potentially high-dimensional domains. To achieve this, we extend the notion of BE to continuous settings, derive tractable $k$-nearest neighbor estimators, provide theoretical guarantees for these estimators, and develop practical reward functions that can be used with standard RL methods to learn BE-maximizing policies. Using standard MuJoCo environments, we experimentally compare the performance of offline RL algorithms for a variety of downstream tasks on datasets generated using BE, R\'{e}nyi, and Shannon entropy-maximizing policies, as well as the SMM and RND algorithms. We find that offline RL algorithms trained on datasets collected using BE outperform those trained on datasets collected using Shannon entropy, SMM, and RND on all tasks considered, and on 80\% of the tasks compared to datasets collected using Renyi entropy.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730547908239

thread_type:
official_review

<thread_object>
creator: reviewer_t8fn

<content>
summary:
The paper proposes to use behavioral entropy (BE), a generalized version of Shannon's entropy using a reweighting function, as reward to generate datasets and tests whether these datasets would be better suited for offline RL. 
The behavioral entropy allows reweighting low-probable events and high-probable events, and it was used in Economics to explain human behavior. 
The paper shows that data generated with maximizing BE can lead to higher downstream task performance in offline RL.

strengths:
- analysis of behavior entropy on exploration
- nice visualizations
- continuous space estimators for BE

weaknesses:
- another hyperparemeter introduced, not unsurprising that better performance can be achieved by tuning this parameter per environment
- computationally heavy (because of k-NN that needs to be rebuild every time) and no mentioning of that
- no comparison to simple baselines like RND
- no improvements for strong offline methods

questions:
- The intuition behind BE did not came across well. Also, why would one want a different weighting of high and low-prob. values?
- The graphics look nice, but in the context of beh. generation and exploration, there is no parametric underlying distribution (like Bernoulli) but essentially, one is trying to achieve uniform distribution over the state-space. An illustration on how this exploration strategy really performs differently and how would be interesting.

- how often does the $k$-NN tree for fast retrieval be rebuild? After every episode? 
- Since alpha<1 and q<1 work best, one could say that seldomly visited states (low probability) should give a higher weighting in the reward / entropy
- How was alpha / q selected? 
- How does the method scale to high-dimensional input?

## Post rebuttal update
the authors have improved the paper significantly and addressed most of my concerns. I am increasing my score to 6.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730586821925

thread_type:
official_review

<thread_object>
creator: reviewer_b32c

<content>
summary:
Entropy-based objectives are widely used to perform state space exploration in reinforcement learning (RL) and dataset generation for offline RL. In this paper, the authors borrow the idea of behavioral entropy (BE) from a prior robotics exploration paper and extend it to continuous space. This paper argues that, compared to prior entropy metrics, using BE as the RL objective enables more complete state coverage.

strengths:
State coverage is an important direction of unsupervised RL, which could benefit downstream task solving by learning useful skills, or, as in this paper, by generating a dataset for offline RL.

Regarding originality, even though the idea of applying BE to exploration originates from a cited prior work, to the best of my knowledge, the idea of applying it to state coverage is novel. Further, the authors extend BE from discrete to continuous space and propose a knn-based measurement.

Regarding clarity, the figures in the paper are well designed: Fig 2 clearly illustrates the property of BE under different values of $\alpha$ (one of its parameter) and the differences between BE and prior entropy metrics. Meanwhile, the paper uses the same color for each entropy metric across the paper, which further eases the reading.

weaknesses:
My main questions are about the evalution on state coverage and offline RL performance. Specifically:
1. Could authors explain, in Fig3, why visualizing each experiment using separate plots is a valid metric of state coverage? IIUC, PHATE, t-SNE, and methods does some non-linear projection to project data to 2D, and this brings two problems when comparing these figures:
* As state trajectories go through non-linear projection, the distance between points more reflect their similarity rather their true distance, thus making it hard to say more spanned points covers large state space (I may be wrong on this). Regarding this, one way may be to visualize trajectories of some points and see if they are indeed very distinct states. Another way I would like to see is to run experiments on some 2d domains so we can visualize without PHATE.
* As each experiment is visualized separately, the PHATE may use different projections in different subfigures, making it hard to say points indeed spread out or the projection makes them look spread out. Regarding this, one more fair way is to draw PHATE plots on trajectories across all experiments to ensure they share the same projection.
2. If we assume PHATE plot is a valid, I still have some questions:
* I do not see singnificant difference between SE and BE for Quadruped coverage, is there any? If not, why that's the case?
* In offline RL experiments, do you train each method until it converges? Authors mentions "we performed just 100K offline training steps", but I thought it's more fair to compare performance under convergence.
* The authors motivates in the intro that better state coverage should lead to better offline RL performance, but in Fig 6, the $\alpha$ leading to best coverage ($\alpha$=5) typically has the worst performance across all alpha, why that's the case? Is the state coverage metric wrong or the assumption of better state coverage leading to better performance wrong or becase of other reasons?

Minor suggestions on writing:
1. In intro, authors motivates the usage of BE by only saying "it's widely used in behavioral economics to model human probability perception" and the results from Suresh et al. (2024). This feels void and doesn't explain why it could helps. There is no explanation why reweighting the entropy (making it more uniform or sharper) could lead to better coverage.
2. L100, "We derive practical RL methods", it's more of a different reward/objective, as you still use an existing RL algorithm.

questions:
What is admissible generalized entropies and why "admissible"  and "generalized" are important?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665083708

thread_type:
official_review

<thread_object>
creator: reviewer_ucex

<content>
summary:
This paper explores how to extend Behavior Entropy to high-dimensional continuous settings and leverages this as a metric for exploration to elicit high state space coverage. This exploration objective can be systematically applied to dataset generation for offline RL algorithms. The authors additionally derive tractable KNN estimators and practical reward functions that can be used in tandem with RL to learn an effective policy. They empirically show that generated datasets with Behavior Entropy outperform datasets generated using Shannon entropy and Rényi entropy with offline RL algorithms such as CQL on Mujoco tasks.

strengths:
The authors have a novel contribution with the extension of Behavior Entropy to continuous domains as well as a tractable formulation using KNNs (with convergence guarantees) to define an reward function for RL. They show initial empirical validation on a subset of the Mujoco domains in the Walker and Quadraped tasks with three offline RL algorithms: CRR, CQL and TD3. Additionally, the visualizations of the dataset generation provide insights into what differentiates the different exploration approaches.

weaknesses:
Some of the empirical results aren’t compelling such as the $\alpha$/$q$ sweep done in Figure 4, where the gap from BE and Shannon/Renyi Entropy is minimal or within the margin of error for domains, which questions the advantages of the dataset generated. Additionally, it would be good to check the consistency of the dataset generation, where the dataset generation process is repeated for each $\alpha$ or $q$, $N$ times and performance is aggregated across these generation attempts. Additionally, the domains evaluated are state based and a limited subset of the Mujoco benchmark. It would be good to see the clear benefits of BE-generated datasets across multiple tasks and algorithms. Additionally, there are a myriad of other exploration objectives people consider such as RND, Count Based Exploration, etc. It would be good to empirically compare against these approaches for the synthetic dataset generation.

questions:
- For BE, is there a clear selection procedure for $q$ empirically that would transfer across algorithms/tasks?
- Currently, the approach relies on a k-NN estimator, which has limitations as the state dimensionality increases or the size of the dataset is large. As offline RL scales to be used with more realistic domains, these concerns will be more common place. Are there tradeoffs with using this as an estimator?
- Along with the qualitative visualizations you have provided (e.g PHATE/tsne), could a quantitative analysis of the dataset state/action coverage be done to compare the different dataset generation approaches?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730691478949

thread_type:
official_review

<thread_object>
creator: reviewer_esfg

<content>
summary:
This paper studies dataset generation for offline RL. More specifically, this paper extends recently proposed behavioral entropy (BE) for discrete settings to continuous tasks, and designs estimator and RL algorithms for exploration and data generation. The paper compares offline RL performance on BE-generated data with other entropy-based data generation and shows that the proposed method outperforms other baselines in the Walker and Quadruped environments.

strengths:
1. The proposed method is well-motivated and theoretically grounded.
2. The writing is clear and easy to follow.

weaknesses:
While well-motivated, I have some questions about this work:

1. The authors propose a KNN-based estimator with convergence guarantee. I wonder in practice how $k$ is chosen and if KNN introduces huge computation overhead compared to other baselines. For instance, if SE and RE have the same computation budget as BE, will they collect more data and hence perform better?
2. The experiments only generate 500K elements for each task, which might be not enough for the offline RL to learn a good policy. Can we test if the performance of BE, SE, and RE will scale with more data?

Minor:

1. While this paper focuses on the offline RL setting, I am curious about whether this entropy-based exploration can be applied in the online RL setting and how it performs compared with other online exploration methods.

Overall, while this paper presents some interesting ideas, I am unable to recommend acceptance at this stage given the questions mentioned above. However, I would consider raising my score if the authors could address my concerns.

questions:
There are some questions and concerns, which I have outlined in the previous section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732390491343

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We thank all the reviewers for their valuable feedback and insightful questions. We are encouraged that they consider our work well-motivated and novel (esfg, UCex, B32C), and appreciated our theoretical contributions (esfg, UCex) and the clarity of the writing and figures (esfg, UCex, B32C, T8Fn).

**Revision and new experiments:**

We have uploaded a revised version of our paper based on reviewer suggestions. Our main changes are highlighted in blue. The primary additions are as follows:
* new **offline RL experiments on datasets generated using RND, SMM,** and a 3M-element SE dataset, including $t$-SNE and PHATE visualizations (esfg, UCex, T8Fn) -- see Table 1 and Figs. 4, 5, 6, 12
* new **quantitative comparisons of dataset coverage** (UCex, B32C, T8Fn) -- see Fig. 8
* **clarification of the motivation for using BE for exploration** (B32C, T8Fn) -- see intro, Figs. 8, 13

**Summary of core contributions:**

1. we extend the recently proposed BE to continuous spaces **(theoretical)**
2. we derive $k$-NN estimators for BE, rigorously establish convergence guarantees for these estimators, and derive practical RL rewards based on these $k$-NN estimators for use in maximizing BE **(theoretical)**
3. we provide qualitative and quantitative comparisons of coverage provided by BE and other methods, illustrate superior variety of coverage provided by BE **(experimental)**
4. we provide quantitative comparison of offline RL performance on datasets generated using BE, RE, SE, RND, and SMM, and find that BE datasets generally lead to superior (or comparable in the worst case) performance on downstream tasks **(experimental)**

**Final remarks:**

The reviewers focused heavily on our experimental results in their reviews, yet key parts of our contribution are theoretical (see contributions above). **We believe the reviewers may have undervalued our theoretical contributions,** and we hope the reviewers will take this into consideration during the discussion period and revise their scores where appropriate.

We address specific reviewer concerns in individual responses below. We hope that our revisions, additional experiments, and individual responses have addressed all reviewer questions and concerns. Please let us know if there are any further questions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a novel extension of behavioral entropy (BE) to continuous state spaces for exploration and dataset generation in offline reinforcement learning (RL). By proposing a KNN-based estimator, the authors show that BE-generated datasets outperform those based on Shannon and Rényi entropy, leading to improved offline RL performance on Mujoco tasks.

While the work is generally seen as a valuable theoretical contribution, there are concerns about its practical applicability. Introducing environment-specific hyperparameters (e.g., α and q) requires tuning for each task, which could be cumbersome in offline RL settings. The computational complexity of the KNN estimator, especially in high-dimensional spaces, was also noted. Despite these concerns, the authors clarified that the method works well in practice, though further explanation of computational costs could be useful. Some reviewers also pointed out that BE improved state coverage but did not consistently lead to better performance compared to simpler methods like Shannon entropy or strong offline methods like TD3.

Overall, the rebuttal addressed many of the reviewers' concerns, improving the clarity of the paper. Given the theoretical contributions and promising results, I recommend acceptance. However, the authors must clarify practical aspects such as hyperparameter tuning and computational cost in the final version.