Fewer May Be Better: Enhancing Offline Reinforcement Learning with Reduced Dataset
Authors: Yiqin Yang
Quanwei Wang
Chenghao Li
Hao Hu
Chengjie Wu
Yuhua Jiang
Dianyu Zhong
Ziyou Zhang
Qianchuan Zhao
Chongjie Zhang
Bo XU
Venue: ICLR 2025 Poster (2025)
Keywords: Offline Reinforcement Learning; Data Selection; Grad Match
Primary area: reinforcement learning
TL;DR: Contruct the reduced dataset to improve algorithm performance while accelerating algorithm training.

Abstract: Research in offline reinforcement learning (RL) marks a paradigm shift in RL. However, a critical yet under-investigated aspect of offline RL is determining the subset of the offline dataset, which is used to improve algorithm performance while accelerating algorithm training. Moreover, the size of reduced datasets can uncover the requisite offline data volume essential for addressing analogous challenges. Based on the above considerations, we propose identifying Reduced Datasets for Offline RL (ReDOR) by formulating it as a gradient approximation optimization problem.  We prove that the common actor-critic framework in reinforcement learning can be transformed into a submodular objective. This insight enables us to construct a subset by adopting the orthogonal matching pursuit (OMP). Specifically, we have made several critical modifications to OMP to enable successful adaptation with Offline RL algorithms. The experimental results indicate that the data subsets constructed by the ReDOR can significantly improve algorithm performance with low computational complexity.

Published as a conference paper at ICLR 2025
FEWER MAY BE BETTER: ENHANCING OFFLINE REIN- FORCEMENT LEARNING WITH REDUCED DATASET
Yiqin Yang', Quanwei Wang", Chenghao Li,, Hao Hu², Chengjie Wu?, Yuhua Jiang', Dianyu Zhong", Ziyou Zhang", Qianchuan Zhao?, Chongjie Zhang', Bo Xu If
'The Key Laboratory of Cognition and Decision Intelligence for Complex Systems,
Institute of Automation, Chinese Academy of Sciences
*Tsinghua University
3 WWahington University St. Louis yig yigin. yang@ ia ac.cn
ABSTRACT
Offline reinforcement learning (RL) represents significant shift in RL research, allowing agents to learn from pre-collected datasets without further interaction with the environment. A key, yet underexplored, challenge in offline RL is selecting an optimal subset of the offline dataset that enhances both algorithm performance and training efficiency. Reducing dataset size can also reveal the minimal data requirements necessary for solving similar problems. In response to this challenge, we introduce ReDOR (Reduced Datasets for Offline RL), method that frames dataset selection as a gradient approximation optimization problem. We demon- strate that the widely used actor-critic framework in RL can be reformulated as a submodular optimization objective, enabling efficient subset selection. To achieve this, we adapt orthogonal matching pursuit (OMP), incorporating several novel modifications tailored for offline RL. Our experimental results show that the data subsets identified by ReDOR not only boost algorithm performance but also so with significantly lower computational complexity.
INTRODUCTION
Offline reinforcement learning (RI. (Levine al. 2020) has marked paradigm shift in artificial intelligence. Unlike traditional RL (Sutton & Barto 2018) that relies on real-time interaction with the environment, offline RL utilizes pre-collected datasets learn prccision-maknng policies (Yang et al., 2021 Janner et al.) 202T). This approach is increasingly favored for its practicality in scenarios where real- real-time data acquisition is impractical or could damage physical assets. Moreover, offline learning can avoid the significant time and complexity involved in online sampling and environment construction. This streamlines the learning process and expands the potential for deploying RL across
more comprehensive array of applications (Yuan et al. 2022 Zhou et al. 2023 Nambiar et al. 2023).
However, offline reinforcement learning relies large pre-collected datasets, which result in substantial computational costs during policy learning (Lu et al.) 2022), especially when the algorithm model requires extensive parameter tuning (Sharir et al. 2020). Moreover, additional data may not always improve performance, as suboptimal data can exacerbate the distribution shift problem, potentially degrading the policy (Hu et al. 2022). In this work, attempt explore effective offline reinforcement learning methods through a data subset selection mechanism and address the following question:
How do determine the subset of the offline dataset improve algorithum performance and accelerate algorithum training?
In this paper, we formulate the dataset selection challenge as a gradient approximation optimization problem. The underlying rationale is that if the weighted gradients of the TD loss the reduced
Corresponding Author

---

Published as a conference paper at ICLR 2025
dataset can closely approximate those the original dataset, the dataset reduction process should not lead to significant performance degradation. However, directly solving this data selection problem is NP-Hard (Killamsetty et al. 2021bbc) c). To this end, we first prove that the common actor-crite framework can be transformed into a submodular optimization problem (Mirzasoleiman et al. (020).
Based on this insight, we adopt the Orthogonal Matching Pursuit (OMP) (Elenberg | 2018) to solve the data selection problem. On the other hand, different from supervised learning, target values in offline RL evolve with policy updates, resulting in unstable gradients that affect the quality of the selected data subset. To solve this issue, we stabilize the learning process by making several essential modifications to the OMP.
Theoretically, we provide a comprehensive unalysis of the convergence properties of our algorithm and establish an approximation bound for its solutions. We then prove the objective function can be upper-bounded if the selected data is sufficiently diverse. Empirically, we evaluate REDOR on the D4RL benchmark (Fu et al. 2020). Comparison against various baselines and ablations shows that the data subsets constructed by the REDOR can significantly improve algorithm performance with low computationally expensive. To the best of our knowledge, our work the first study analyzing the reduced dataset in offline reinforcement learning.
2 RELATED WORKS
Offline Reinforcement Learning. Current offline RL methods attempted to constrain the learned
policy and behavior policy by limiting the action difference (Fujimoto al. 2019), adding KL-
divergence (Yang et al. Nair et al. [020 Peng et al. 2019 Wu et al. [2019] regularization (Ku
mar et al. 2019 conservative estimates (Yang et al. 2023; Ma et al. [202 Kumar et al. 2020
Ma et al. b0 penalizing uncertain actions (Janner et al. 2019: Yu et al.) [022]] Kidambi et al.
2020 These studies provide a solid foundation for implementing and transferring reinforcement
learning to worldw tasks.
Offline Dataset. Some works attempted to explore which dataset characteristics dominate in offline RL algorithms (Schweighofer al. 2021 Swazinna al., 2021 Chen et al.. 2020 Yue al. 2022) or investigate the data generation Yarats et al. 2022). Recently, some researchers attempted to solve the sub-optimal trajectories issue by constraining policy to good data rather than all actions in the dataset (Hong et al. 2023b) or re-weighting policy (Hong et al. (202a)). However, limited research has addressed considerations related the reduced dataset in offline RL.
Data Subset Selection. The research on identifying crucial samples within datasets is concentrated on supervised learning. Some prior works use uncertainty of samples (Coleman al., 2019: Paul et al. T02T) or the frequency of being forgotten (Toneva et al.) (018)) as the proxy function to prune the dataset. Another research line focuses on constructing weighted data subsets to approximate the full dataset (Feldman 2020). which often transforms the subset selecting to the submodular set cover problem (Wei et al. 2015 Kaushal et al., 2019). These studies establish the importance of selecting critical samples from datasets for practical training. However, unlike supervised learning, target values in offline RL evolve as policies update, leading to unstable gradients that significantly complicate the learning process.
3 BACKGROUND
Reinforcement Learning (RL) deals with Markov Decision Processes (MDPs). A MDP can be mod- eled by tuple (S, A.,. y), with the state space S, the action space A, the reward function r(s, a). the transition function p(s'|s, a). and the discount factor 7. We follow the common assumption that the reward function is positive and bounded: Va S, a E A, S r(s,a) S Rmax. where Rmax the maximum possible reward. RL aims to find policy x(a s) that maximizes the cumulative discounted return:
π arg max ((T) = arg max E, #
Σ y (rr(s;, at),
where H. is the horizon length. For any policy the action value function is Q (sc, ar) Fx E= * (0,)))s=, The state value function is Vd (*()))
H-t
2

---

Published as a conference paper at ICLR 2025
E. 18 IE H- x ( a=+))ss==s. It follows from the Bellman equation that (*(ss) EoEA EOEAT(als)) T[als)Q" (3,, ar).
Offline RL learns policy without interacting with environment. Rather, the learning is based on a dataset D generated by behavior policy Tg. One of the major challenges in offline RL the issue of distributional shift (Fujimoto et al. 2019), where the learned policy is different from the behavioral policy. Existing offline RL methods apply various forms of regularization to limit the deviation of the current learned policy:
arg max (JD(z) a (
where (TT) is the cumulative discounted return of policy the empirical MDP induced by the dataset D, and (T,, π.) a divergence measure between т and x3. In this paper, we base our study on TD3+BC (Fujimoto & Gu 2021), which follows this regularized learning scheme.
We introduce the concept of offline data subset selection. Specifically, let D {(, a,, rs. } i=1 denote the complete offline dataset, and let S _ D, indexed by j. represent the reduced dataset. We formulate the subset selection as:
S arg min S)
s.t. J(ms) 2 ((T)) +
DCD
where Tp and as are the policy trained using Eq. with dataset D and S, respectively. c 2 0 the policy performance gain.
Compact Subset Selection for offline reinforcement learning remains largely under-explored in existing literature. However, research efforts have been directed toward reducing the size of training
samples in other deep learning fields like supervised learning (Killamsetty et al. 2021a soleiman et al. 2020).
Mirza
Specifically, there are some research explorations on transforming the subset selection problem into the submodular set cover problem (Mirzasoleiman et al. (2020). The submodular set cover problem is defined as finding the smallest set that achieves utility p:
S arg min (S), SCD
s.t. F(S) P.
where we slightly abuse the notation and use to denote complete supervised learning dataset. We require F to be submodular function like set cover and concave cover modular (Iyer et al. 2021). function F is submodular if it satisfies the diminishing returns property: for subsets C D
and j E \ T, F(j S) & F(SUj)-FFU) 2 F(j IT) and the monotone property: F(j () 2 for any j E D S and S D.
4
METHOD
For the data subset selection problem, RL and supervised learning are significantly different in two aspects: (I) In supervised learning, the loss value the primary criterion for selecting data. However, the loss value in RL is unrelated to the policy performance. Therefore, need to consider new criteria for selecting data in RL. (2) Compared with the fixed learning objective supervised learning, the learning objective in offline RL evolves policies update, significantly complicating the data selection process. To solve these issues, we first formulate the data selection problem in offline RL the constrained optimization problem in Sec. 4.1 Then, present how to effectively solve the optimization problem in Sec. 4.2 Finally, we balance the data quantity with performance in Sec. 4.3 The algorithm framework is shown Algorithm
4.1 GRADIENT APPROXIMATION OPTIMIZATION
We first approximate the optimization problem 3 using the Q-function (* a) as the performance measure (TT) = QT ao) and requiring that Q and QT to approximately equal for any action-state pair (s, a):
S* = arg min (S).
s.t. (8 - (s, (s,a) S 6.
(5)
SCD
3

---

Published as a conference paper at ICLR 2025
We gradient approximation optimization deal with the constraint in the optimization problem S Suppose that Q-functions are represented by networks with learnable parameters 9 and updated by gradients of loss function [()). e.g., the TD loss (Mnih et al. 2015). If identify reduced training set S such that the weighted sum of the gradients of its elements closely approximates the full gradient over the complete dataset D, then we train S and converge to Q-function that is nearly identical to the one trained on D.
Formally,
((0) = (0 Em
0z, Tr s4,0)
(6)
IED
is the standard Q-learning TD loss, and
Lzde (w,, 0) = = (0)
(7)
the loss on the reduced subset C 'D. In order to better approximate the gradient for the full dataset, we use the weighted data subset. Specifically, w; is the per-element weight in coreset S. During the learning process, we approximate the entire dataset's s gradient by multiplying the samples' gradient in coreset by their weights. We define the following error term:
Err (w, S, C,
Σ wi VoL (0) VoL (0)
Minimizing Eq. ensures the dataset selection procedure can maintain or even improve the policy performance. Similarly, define the regularized version of Err (w, S,C, as
EXTa (w, S, () 0) = Err (w, 6. + Al| A||w0|
(9)
Then, the optimization problem
3 is transformed into:
wr, S arg min Errx (w,, C,
(10)
4.2 ORTHOGONAL MATCHING PURSUIT FOR OFFLINE RL
Directly solving problem is NP-hard (Killamsetty al. 2021b/c) and computationally intractable. To solve the issue, we consider using the iterative approach, which selects data one by one to reduce Errx (ur, S, f). To ensure newly selected data are informative, we prove the optimized problem 10 can be transformed into the submodular function.
Specifically, we introduce constant Lmax and define (X( Lunax min ErTA (w, S, 0). Then, we consider the common actor-critic framework in data subset selection, which has an actor- network (s) and a critic network Qa(s, a) that influence the TD loss and thus the function (S). Therefore, the submodularity analysis of (S) involves two components: (S) that depends on the critic loss (0)), and F (S) that depends on the actor loss (Ø) The following theorem shows that both P (S) and (S)) are weakly submodular.
Theorem 4.1 (Submodular Objective). For |S] < N and sample (s;, (s,, E D, sup- pose that the TD loss and gradients are bounded: [C)( S UTD (VoQo(s, a:)le UvQ IIV Qo To(si))ll2 I Uva. ([Tg(s;) - aill2 I U.. (0 I Un. and (8)a(s Up then F (S) is 8-weakly submodular, with
X
82 X+ (N(TUUU)))
(11)
and (S) is 6-weakly submodular, with
X A N 2U.UR\UU
8 82
(12)
4

---

Published as a conference paper at ICLR 2025
Please refer Appendix A.2 for detailed proof.
Based on the above theoretical analysis, let ErTy (w, Sj-1, L. C) represent the residual error at iteration j. Then, adopt the Orthogonal Matching Pursuit (OMP) algorithm (Elenberg ct al. 2018). which selects a new data sample and takes its gradient VoC* (0) as the new basis vector to minimize this residual error. In this way, update the residual to Erra (w, L, f). where Sj S U {i). However, the dynamic nature offline RL. poses a challenge when using OMP, leading to unstable learning. To address the unique challenges of offline RL, we propose the following novel techniques to enhance gradient matching:
(I) Stabilizing Learning with Changing Targets. In supervised learning, the stability of training targets leads to stable gradients. However, in offline RL. target values evolve with policy updates, resulting in unstable gradients in Eq. [8 that affect the quality of the selected data subset. To address this issue, we will stabilize the learning process by using empirical returns from trajectories to smooth the gradient updates. This provides more consistent learning signal and mitigates instability caused by changing target values. Specifically, rather than adopt the gradient of the TD loss, we calculate gradient VoL (0) from the following equation
H-
VeL = VolElly Qo(s), y y=
Σ Fr(stt)
(13)
Furthermore, we will adopt multi-round selection strategy where data selection occurs multiple rounds T. In each round, a portion of the data is selected based on the updated Q-values, reducing variance and ensuring that the subset captures the most critical information. This multi-round approach allows for dynamic adjustment of the selected subset as learning progresses, improving stability and reducing the risk of overfitting
specific trajectories. Specifically, we calculate C C (6.) at each round based on Eq. 13 where Og is the parameter updated in the f-round. In practice, we pre-store parameters 0, with various rounds and load them during training.
(II) Trajectory-base Selection. In offline RL, collected data is often stored in trajectories, which are coherent and more valuable than individual data points. For this reason, we modify OMP to trajectory-based gradient matching. Specifically, we select new trajectory i of length K and take (0.) Σx= Vo, (0:) K as the basis vector minimize the residual error. Then, we update the residual Errx (w,SS, L, 0), where Sj Sj-1 U U (Trajectory )
the mean of gradients Vo, Vo, Linj
4.3 BALANCING DATA QUANTITY WITH PERFORMANCE
In offline RL, while additional data can help generalization, suboptimal data may lead to significant performance degradation due to distribution shifts. To address this, we will introduce a constraint term that biases the TD- DD-gradient matching method toward selecting data with higher return estimates. Then, based on the design in the Sec. 4.2 (D) the Equation 13 is transformed into
H-t
VeL = VOR 2o(s),a)))
y
7 (8 (8++))
(14)
k=
s.t. y Top m 7% ((Reuut(((Traaccco)) (21)
This regularized constraint selection approach ensures that the selected subset not only reduces computational costs but also focuses on data points that are aligned with the learned policy, avoiding performance degradation caused by suboptimal trajectories.
THEORETICAL ANALYSIS
In this section, study the convergence property of our method and the error bounds of the solutions it finds. We work with mild assumptions that the gradient of the TD loss is Lipschitz smooth with constant L: [V())) [())] < L| ØlI. and that the gradient is bounded by ||V [(O]] I
Firstly, we show that the TD loss of the offline Q function Q trained on the reduced dataset S can converge.
5

---

Published as a conference paper at ICLR 2025
Algorithm Reduce Dataset for Offline RL (REDOR)
Require: Complete offline dataset D
2: Initialize parameters of the offline agent for data selection Qa, To
3: for 1,... ,T do
4: Load parameter 0g for Qe.
5: Calculate Ve, ((),), Vo, ( (8.) based on Equation
6: S,, w. = OMP V. ((O,), Vo, (Tn)((0),, 0z)
7: end for
8: Reduced offline dataset S Lie ST
9: Initialize parameters of the offline agent for training on the reduced offline dataset Qa Ty
10: Train Qo me based S and
Algorithm 2 OMP algorithm
I: Require: Vo. (0,,) Vo, Lia (0z), lr, regularization coefficient X
2: t ErTa (w), St, L, 01)
3: while r2c do
4: e = arg max (V St. finj (0),,)) (0n)
5: St - St U (Trajectory,)
6: wi arg minw, EITa (w,, St, L.).)
7: r + ERTA (we, St, L. 0c)
8: end while
9: Return S, and wt
Theorem 5.1. Let 6* denote the optimal QTh parameters, 0 the parameters after t training steps. We have
Do E t=l
min L(0.) S C(00) 1:G
(15)
VG
Here L(0) = EiED (8s, Tb 8/ 0) is the TD loss, G is the number of total training sleps,
D = -|0- -b and e = Err (w, S, L. 02) is the gradient approximation errors.
Proof. Please refer to Appendix A.4 for detailed proof.
We assume the gradients of selected data are diverse and they can be divided into K clusters {C..... Cx) } with the cluster centers set = ErT (w, S,C,8) can be upper bounded:
Then, we prove the residual error
Theorem 5.2. The residual error Err (w,, S, C, is upper bounded according to the sample's 's gradient of TD loss:
min
Σ min V L (0) voL (0) 1|2.
(16)
CEC
IED
Proof. Please refer to Appendix A.3 for detailed proof.
We then prove that the reduced dataset selected by our method can achieve a good approximation for the gradient calculated the complete dataset, which also means = Err (w, S, C, in Theorem i. bounded.
Corollary 5.3 (Approximation Error Bound of the Reduced Dataset), The expected gradient approxi- mation error achieved by our method is at most 5(In K + 2) times the error of the optimal solution S*:
Err (w, S, C. < 5(In K + 2) Err (w,S", S.,,
(17)
Proof. The proof is derived by applying Theorem 5.2 along with Theorem 4.3 from (Makarychey al. 2020). by observing that cluster centers included in the reduced dataset.

---

Published as a conference paper at ICLR 2025
weller2b-mmdium--0 (Hard)
wxcee222-eeppr---0 (Hard)
wwlleer2--mmiium-neee (Hand)
Re30W (Olen)
Epkorde Raurn
RebOit [0un))
Epecrde Remuen
-5
Ryniiile (Ores)
Raifor
Progica
20-
Resdom
Exkande
Complets Dotaxet
Carrplee Datyort
20-
Compatt Dotswer
20
0,20
01.33
0.75
1,00
0000
10:30 Millon Stepe
1.00
Millio Stops
Million Staps
hopper-medum-xO (Hard)
hopper-expert-00 (Hard)
hopper-msdiu-reppayy-- (Hard)
1004
100
(Ours)
Eptcde Recure
abx
Priottored
100
ReDOR (Dus)
Epiadd
Fandom
Complet
Prectradd
Randors
Rand
Complete Datawe
Complete Dotooe
0.00
0.50
0.75
LE0
0.10
0.20
13.00
0.50
1.00
0.00
01
0.30
0.20
1.00
Millier Steps
Millin Staps
Stare
halfchoctah-meediiee -otirtttttt-ddii---oo------- (Hard)
ha hallchhetaah--ppe--d (Hard)
halfchestah- reppaaee-- (Hard)
100
Epoode Ream
Raboh (Oan)
20-
ReDOR (Dues)
Expoode
Randon
Epeode
WaDOR WapoR.jOaad
26-
Compleee Dataset
Comples Dataee:
Complete Dasaset
0.00
0.20
0,50
0.75
1.50
0.25
0,50
c
1.0
0.25
não
0.50
1.00
Million Soaps
Million
Million Steps
Figure 1: Experimental results the D4RL (Hard) offline datasets. All experiment results were averaged over five random seeds. Our method achieves better or comparable results than the baselines with lower computational complexity.
The aforementioned theoretical analysis has the following limitations: First, we assume that the gradients are uniformly bounded. Therefore. if the gradients of the algorithm diverge in practice, it would contradict our assumptions, and the selected data subset would no longer be valuable. Current offline RI. methods can only ensure that the Q-values do not diverge Kumar et al. ( Fujimoto & Gu (2021). Although this can, to some extent, reflect the gradients of the Q-network that have not diverged, there is no rigorous proof that the bounds of the gradients can be guaranteed. Second, the above theoretical analysis is based on the classic TD loss. However, to provide a consistent learning signal and mitigate instability caused by changing target value, the techniques in Section 4.3 adopt fixed target rather than TD loss.
Discussion
EXPERIMENT
In this section, we assess the efficacy of our algorithm by addressing the following key questions. (1) Can offline RL algorithms achieve stronger performance on the reduced datasets selected by REDOR? (2) How does REDOR perform compare to other offline data selection methods? (3) What the factors that contribute to REDOR's ' effectiveness?
6.1
SETUP
We evaluate algorithms on the offline RL benchmark D4RL (Fu et al. 2020) to answer the afore- mentioned questions. In addition, we consider a more challenging scenario where we add additional low-quality data the dataset simulate noise real-world tasks, named D4RL. (hard). The evaluation process commences with the selection of offline data, followed by the training of widely recognized offline RL algorithm, TD3+BC (Fujimoto Gu (222I) on this reduced dataset for million time steps. To ensure a fair comparison, we apply the same offline RL algorithm to data subsets obtained by different algorithms. Evaluation points are set at every 5,000 training time steps and involve calculating the return of 10 episodes per point. The results, comprising averages and standard deviations, are computed with five independent random seeds. On the other hand, we can also incorporate method into offline model-based approaches, such as MOPO Yuu et al. 2020
7

---

Published as a conference paper at ICLR 2025
KRLS
Log-Det
BlockGreeddy
REDOR
Hopper-medium-v0
69.412.5
58.413.6
83.7=2.2
94.3:14.6
Hopper-expert-vo)
91.0++1.1
90.7++1.3
98.7+0.5
110.0++0.5
Hoppe--nediun--eplayyvd
28.5+:3.2
29.4+:1.2
30.5+2.4
35.3+:3.2
Walker2d-medium-v00
49.1
47.513.4
53.313.6
80.5+2.9
Walker2d-expert-v00
68.413.2
67.5155.6
74.813.44
104.6:12.5
Walker2d-meeium-rrep
14.3#1.2
15.2+2.2
16.7=113
21.1=11.8
Halfcheetah mediumm-v
23.4=0.5
21.910.9
27.5+0.7
41.0=10.2
Halichestah--xperr--d
73.9+114
72.142.2
79.2+
88.5++2.44
Haffchertah-mciiu--eelaay
39.5+0.3
39.9+0.5
4.0
41.1++0.4
Table I: Experimental results on the D4RL (Hard) offline datasets. All experiment results were
averaged over five random seeds. Our method performs better than the dataset reduction baselines.
buffehoctah-rrcee wew.derrrreireeeeeeeeeee -ra cepert-00
hopper 1aapa---aaaar-100
hogpet mediam cepert-00
wallee22-redaar expet-9
Epeode team
Epeook besam
HebOO (0mm)
NatOH (Uum)
Heboet (Dam))
HeDoo (Dam))
Piaciix
Raador
Ban Steps
Villor Steps
14 Bon Steps
Millon Steps
Figure 2: Experimental results on the D4RL offline datasets. All experiment results were averaged five random seeds. Our method achieves better comparable results than the baselines consistently.
and MoERL (Kdambi et al., 2020). Similarly, we only need to replace the current offline loss with the corresponding policy and model loss.
Baselines. We compare REDOR with data selection methods RI. Specifically, previous work on prioritized experience replay for online RL (Schaul et al.) 2015) aligns closely with our objective. make this a baseline Prioritized where samples with the highest TD losses form the reduced dataset. Baseline Complete Dataset presents the performance by training TD3+BC with the original, complete dataset. Baseline Random randomly selects subsets from the D4RL dataset that are of the same size REDOR. We also compare from supervised learning. Specifically, we adopt the coherence criterion from Kernel recursive least squares (KRLS) (Engel al. 2004),, the log det criterion by forward selection in informative vector machines (LogDet) (Seeger 2004) and the adapting kernel representation (BlockGreedy) (Schlegel al. 2017) as our baselines.
method with general dataset reduction techniques
6.2 EXPERIMENTAL. RESULTS
Answer of Question To show that REDOR can improve offline RL algorithms, compare REDOR with Complete Dataset, Prioritized, and Random in the Mujoco domain. The experimental results in Figure that our method achieves superior performance than baselines. By leveraging the reduced dataset generated from REDOR, the agent can learn much faster than learning from the complete dataset. Further, the results in Figure 2 show that REDOR also performs better than the complete dataset and data selection RL baselines in the standard D4RI. datasets. This is because prior methods select data in random or loss-priority manner, which lacks guidance for subset selection and leads degraded performance for downstream tasks.
addition, to test REDOR's generality across various offline RL algorithms on various domains, we also conduct experiments on Antmaze tasks. We use IQL (Kostrikov et al. (222 as the backbone of offline RL algorithms. The experimental results 2 show that our method achieves stronger performance than baselines. In the antmaze tasks, the agent is required to stitch together various trajectories to reach the target location. In this scenario, randomly removing data could result the loss of critical data, thereby preventing complete the task. Differently, REDOR extracts valuable
8

---

Published as a conference paper at ICLR 2025
2
s
17777 S 1A 3
3
5
4
Figure 3: Visualization of the complete dataset and the reduced dataset in hal fcheetah task. The higher opacity of point represents large time step towards the end of episode. The dataset embedding is characterized by its division into different components. Samples selected by REDOR connect different components by focusing on the data related the task.
Env
Random
Prioritized
Complete Dataset
REDOR
Antmaze-umaze--V
75.1
7..2
1..
90.7=3.3
Antmaze-umaze-diig
4..3
44.7+2.7
62.2:2.00
76.7=2.2
antmaze-medium-poe
59.3+1.6
60.3+2.9
71.2+
80.3=2.9
Antmaze- -eneiiu--iiver---u v0
43.6++2.7
46.9+3.8
7..0+1.6
84.9=338
Antmaze-large-play-W
3.740.77
15.0+3.5
39.6:13.6
46.0
Antmaze-large-divvess--v
16.0433.6
20.5+3..7
47.5
3..0
Table 2: Experimental results on the Antmaze offline datasets. All experiment results were averaged over five random seeds. Our method performs better than baselines.
subset by balancing data quantity with performance, achieving stronger performance than the complete dataset.
To test whether REDOR can select more valuable data than the data selection algorithms in supervised learning, we compare our method with KRLS (Engel et al. 2004), Log-Det (Seeger 2004) and BlockGreedy (Schlegel et al. 2017) in the D4RL (Hard) datasets. The experimental results in Tsble that our method generally outperforms baselines. We hypothesize that supervised learning is static with fixed learning objectives, while offline RL's dynamic nature makes the target values evolve with policy updates, complicating the data selection process. Therefore, the data selection methods in supervised learning cannot be directly applied to offline RL scenarios.
Answer of Question 2:
Answer of Question 3: To study the contribution each component in our learning framework, conduct the following ablation study. Q Target: We replace the empirical returns used to update Q functions with the standard target Q function in the TD loss function. Single Round: We set the number of data selection rounds to 1 and study the function of multi-round data selection. The experimental results in Figure 4 in Appendix shooow that removing any of these two modules will worsen the performance of REDOR. In case like wa wawker2ee -eeerdddmddddd um, ablation Single Round even decrease the performance by over 80%, and ablation Q Target results in a 95% performance drop in walker2d expert. Furthermore, removing the two modules is relatively small. This result can be attributable to the fact that this task has limited state space, and we can directly apply OMP the entire dataset and identify important and diverse data.
also find that the hal fcheetah tasks, the impact of
We visualize the selected data by REDOR to better understand how it works. Figure 3 displays the t-SNE low-dimensional embeddings, with the complete dataset in blue and the selected data in orange. The higher opacity of a point indicates larger time step. The dataset's structure revealed by its segmentation into diverse components: the agent. For example, from to 7, they represent falling, leg lifting, jumping, landing, leg swapping. stepping, and starting, respectively. We can observe that the selected samples by REDOR not only cover each component of the dataset but also effectively bridge the gaps between them, enhancing the dataset's versatility and coherence. Moreover, we find that REDOR is less concerned with the falling
halfcheetah, each component reflects distinct skill of

---

Published as a conference paper at ICLR 2025
data and instead focuses the data related to the task. This observation explain the improved performance of REDOR. For additional visualizations, please refer to Appendix C.I
6.3 COMPUTATIONAL COMPLEXITY
We report the computational overhead REDOR on various datasets. All experiments are conducted on the same computational device (GeForce RTX 3090 GPU). The results in Appendix e indicate that even on datasets containing millions data points, the computational overhead our method remains low (e.g., several minutes). This low computational complexity can be attributed to the trajectory-based selection technique in Sec. 4.2 (II) and the regularized constraint technique in Sec. 4.3 making method easily scalable to large-scale datasets.
CONCLUSION
In this work, demonstrate critical problem in offline RL identifying the reduced dataset to improve offline algorithm performance with low computational complexity. We cast the issue as the gradient approximation problem. By transforming the common actor-critic framework into the submodular objective, we apply the orthogonal matching pursuit method to construct the reduced dataset. Further, we propose multiple key modifications to stabilize the learning process. We validate the effectiveness of our proposed data selection method through theoretical analysis and extensive experiments. For future work, we attempt to apply our method to robot tasks in the real world.
ACKNOWLEDGMENTS
This work was supported by Strategic Priority Research Program of the Chinese Academy of Sciences under Grant No.XDA27040200, in part by the National Key R&D Program of China under Grant No.2022ZD0116400
REFERENCES
Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang, Yanqiu Wu, and Keith Ross. Bail: Best- action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353-18366 2020.
Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec, and Matei Zaharia. Selection via proxy: Efficient data selection for deep learning. arXiv preprint arXiv: /829, 2019.
Ethan R Elenberg, Rajiv Khanna, Alexandros G Dimakis, and Sahand Negahban. Restricted strong convexity implies weak submodularity. The Annals of Statistics, 46(6B):3539 2018.
Yaakov Engel, Shie Mannor, and Ron Meir. The kernel recursive least-squares algorithm. IEEE Transactions on signal processing, 52(8):2275 2004.
Dan Feldman. Core-sets: Updated survey. Sampling techniques for supervised or unsupervised tasks, pp. 23-44, 2020.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforeement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 2052-2062. PMLR, 2019,
Zhang- Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Hamessing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306.1300 13085, 2023a.
10

---

Published as a conference paper at ICLR 2025
Zhang- Wei Hong, Aviral Kumar, Sathwik Karnik, Abhishek Bhandwaldar, Akash Srivastava, Joni Pajarinen, Romain Laroche, Abhishek Gupta, and Pulkit Agrawal. Beyond uniform sampling: Of- fline reinforcement learning with imbalanced datasets. Advances in Neural Information Processing Systems, 36:4985-5009, 2023b.
Hao Hu, Yiqin Yang, Qianchuan Zhao, and Chongjie Zhang. On the role of discount factor in offline reinforcement learning. International Conference on Machine Learning, pp. 9072-9098. PMIR, 2022.
Rishabh Iyer, Ninad Khargoankar, Jeff Bilmes, and Himanshu Asanani. Submodular combinatorial information measures with applications in machine learning. In Algorithmic Learning Theory, pp. 722-754. PMLR, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-123 1286, 2021.
Vishal Kaushal, Rishabh Iyer, Suraj Kothawade, Rohan Mahadev, Khoshrav Doctor, and Ganesh Ramakrishnan. Learning from less data: unified data subset selection and active learning framework for computer vision. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 1289-1299. IEEE, 2019.
Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model- based offline reinforcement learning. Advances in neural information processing systems, 33:
11810-21822 2020.
Krishnateja Killamsetty, Sivasubramanian Durga, Ganesh Ramakrishnan, Abir De, and Rishabh Iyer. Grad-match: Gradient matching based data subset selection for efficient deep model training. In International Conference on Machine Learning, pp. 1464-54475 PMLR, 2021a.
Krishnateja Killamsetty, Durga Sivasubramanian, Ganesh Ramakrishnan, and Rishabh lyer. Glister: Generalization based data subset selection for efficient and robust learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 81 (0-8118, 2021b.
Krishnateja Killamsetty, Xujiang Zhao, Feng Chen, and Rishabh Iyer. Retrieve: Coreset selection for efficient and robust smi-suuerriieed learning. Advances in Neural Information Processing Systems, 2021c.
Ily Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.06/60 2021.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy g-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 199-1199, 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint urXiv:2005.01643 2020.
Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker-Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206. 220.0229000.779 2022.
Xiaoteng Ma, Yigin Yang. Hao Hu, Qihan Liu, Jun Yang. Chongjie Zhang, Qianchuan Zhao, and Bin Liang. Offline reinforcement learning with value-based episodic memory. arXiv preprint 1rXiv:2110.09770 2021a.
Yecheng Ma, Dinesh Jayaraman, and Osbert Bastani. Conservative offline distributional reinforcement learning. Advances in Neural Information Processing Systems, 34: 1eti1-9229999 2021b.
11

---

Published as a conference paper at ICLR 2025
Konstantin Makarychev, Aravind Reddy, and Liren Shan. Improved guarantees for k-means++ and k-means++ parallel. Advances in Neural Information Processing Systems, 33:16142-1611 161522 2020.
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. In International Conference on Machine Learning, pp. 6950-6960.
PMLR, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-3 533, 2015.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprini arXiv:2006.09359, 2020.
Mila Nambiar, Supriyo Ghosh, Priscilla Ong, Yu En Chan, Yong Mong Bee, and Pavitra Krish- naswamy. Deep offline reinforcement learning for real-world treatment optimization applications. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp. 4673-4684, 2023.
Mansheej Paul, Surya Ganguli, and Gintare Karolina Dziugaite. Deep learning on data diet: Finding important examples early in training. Advances in Neural Information Processing Systems, 34: 20596-20607, 2021.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-wrightcdd regresso:::: Simple and scalable off- policy reinforcement learning. arXiv preprint arXiv:1911000117,2201 2019.
Tom Schaul, John Quan, loannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint ar arXiv:1511.088
2015.
Matthew Schlegel, Yangchen Pan, Jiecao Chen, and Martha White. Adapting kernel representations online using submodular maximization. In International Conference Machine Learning, pp. 3037-3046. PMLR, 2017.
Kajetan Schweighofer, Markus Hofmarcher, Marius-Constantin Dinu, Philipp Renz, Angela Bitto- Nemling, Vihang Prakash Patil, and Sepp Hochreiter. Understanding the effects of dataset charac- teristics on offline reinforcement learning. arXiv preprint 11.02111,,4414, 2021.
Matthias Seeger. Greedy forward selection in the informative vector machine. In Technical report, Technical report. Citeseer, 2004.
Or Sharir, Barak Peleg. and Yoav Shoham. The cost of training nlp models: A concise overview. arXiv preprint arXiv:2004.08900 2020.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Phillip Swazinna. Steffen Udluft, and Thomas Runkler. Measuring data quality for dataset selection in offline reinforcement learning. 202/ IEEE Symposium Series on Computational Intelligence (SSCI), pp. 1-8. IEEE, 2021.
Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes, Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint 1812.0551 2018
Kai Wei, Rishabh Iyer, and Jeff Bilmes. Submodularity in data subset selection and active learning. In International conference on machine learning, pp. 1954-1963. PMLR, 2015.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprini arXiv: 19.1. 2019.
Yiqin Yang, Xiaoteng Ma. Chenghao Li, Zewu Zheng, Qiyuan Zhang, Gao Huang, Jun Yang, and Qianchuan Zhao. Believe what you see: Implicit constraint approach for offline multi-agent reinforcement learning. Advances in Neural Information Processing Systems, 34: 14:10299-10312, 2021.
12

---

Published as a conference paper at ICLR 2025
Yiqin Yang, Hao Hu, Wenzhe Li, Siyuan Li, Jun Yang, Qianchuan Zhao, and Chongjie Zhang. Flow to control: Offline reinforcement learning with lossless primitive discovery. Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp. 10843-10851, 2023.
Denis Yarats, David Brandfonbrener, Hao Liu, Michael Laskin, Pieter Abbeel, Alessandro Lazaric, and Lerrel Pinto. Don't change the algorithm, change the data: Exploratory data for offline reinforcement learning. arXiv preprint vrXi:2221.134222 2022.
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33: 14121144444, 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954-28967, 2021.
Yiping Yuan, Ajith Muralidharan, Preetam Nandy, Miao Cheng, and Prakruthi Prabhakar. Offline reinforcement learning for mobile notifications. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pp. 3614-3623, 2022.
Yang Yue, Bingyi Kang, Xiao Ma, Zhongwen Xu, Gao Huang, and Shuicheng Yan. Boosting offline reinforcement learning via data rebalancing. arXiv preprint arXiv:22100092411 2022.
Gaoyue Zhou, Liyiming Ke, Siddhartha Srinivasa, Abhinav Gupta, Aravind Rajeswaran, and Vikash Kumar. Real world offline reinforcement learning with realistic data source. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 7176-7183, IEEE, 2023.
13

---

Published as a conference paper at ICLR 2025
A PROOFS OF THEORETICAL ANALYSIS
A.1 NOTATIONS
Notation
Explanation
UTD
Bound of TD Loss
UvQ
Bound of Gradient
Uva
Bound of Gradient
Un
Bound of Action Difference
Ux
Bound of Action
D
Complete Dataset
S
Reduced Dataset
N
Size of Reduced Dataset
À
Minimum Eigenvalues
C
Cluster
Total Training Steps Gradient Approximation Errors
0,
Updated parameter at the tt epoch
07
Optimal model parameter
Table 3: Organization of the notations used througout this paper
A.2 SUBMODULAR
Theorem 4.1 (Submodular Objective). For |S| < N and sample (s) E D. sup- pose that the TD loss and gradients are bounded: [C' s UTD a < Uvq. |/V vole)) Qo (8i, [o(s)))ll2 S Uva ([ aill2 S U.. [T S Un. and ( U then F (S) is 8-weakly submodular, with
A 82 A+ (N(DUUV)))
(11)
and F7 is d-weakly submodular, with
X
8 82 X N(Uvaa 2U.
(12)
Proof. As mentioned in Section 3] we use the TD3+BC algorithm as the basic offline RL algorithm. TD3+BC follows the actor-critic framework, which trains policy and value networks separately. For single sample (si, as, Ti, s)), the loss of the value network is also named as TD error, which is defined by:
(0) = (y: - Qo 0z))2
(18)
where = ri + 7Q
7 ($,, To (5:) +
(19)
(20)
The gradient is:
V (0) = (y: Qo(si (o) 2
(21)
Offline RL algorithms attempt to minimize the TD error and compute the Q-value through neural network. Therefore, we assume the upper bound of the TD enrrr is max; ||y - Qo(s,, a;) & UTD The upper bound of the gradient of the value network is max; V/V a;)ll2 Uvq Then, Equation 21 can be transformed into:
V e (0)112 2UToUve
(22)
14

---

Published as a conference paper at ICLR 2025
Similarly, for single sample(st, dt, Ti, st)., the loss of the policy network is
(x) - ---oolos TT + (s:) a.112
(23)
(24)
The gradient is:
V (0) (d) ÖTg(s))
(25)
Ôn (s:)
db
(el))) (e)))) ( 2(Tx(s:) - a:) T [T((s)s]] VaTo(s))
(26)
Here is used to balance the conservatism and generalization in Offline RL, which is defined by:
E [IQ a,)
(27)
where x is hyper-parametee in TD3+BC. Note that although includes Q, it is not differentiated over.
Offline RL algorithms attempt to limit the deviation of the current learned policy from the behavior policy while maximizing the Q-value of the optimized policy. Therefore, we assume the upper bound of the gradient of the value network is max; | (s.) Qo (s(s))) I Uv.- The up- bound of the action error is max; [[+) a:l12 < U.. The upper bound of the output of the policy is max, (s;)l22 I Un. The upper bound of the gradient the policy network is max; V TT (s.)ll2 I Uvn-
Then, Equation 26 can be bound:
( (0)1l2 S (Uva/a + 2UAU, Uv
(28)
We can define two functions (o(3),44 P +
1
10( = -1 Σ
3 V OoL (0) voC(O)12 - A[ll1a
==1
(29)
4(8)= = Σ
3, C (0) voc(o)ll2 2lsllk
We assume iis N N-sparse vector that 0 on all but N indices. Then we can transform maximizing (S),,, (S) into maximizing ((3) 1 1(0):
max P SISISN
max (o(3) (0) 8:8ge-0 I
(30)
max SISKNN
(S)
+ max
(3(3) - (4)
8:9ge_o
N
where S" means the complementary set of S. and 8g- = means 3 is 0 on all but indices i that e S. 1(0) means the value of I(-) when input is vector 0. it serves basic value. Since (g) < 0, 1, (8) I O, we can easily find that the minimum eigenvalues (1) and (3) are both at least X.
15

---

Published as a conference paper at ICLR 2025
Next, the maximum eigenvalues of (lQ(3) and (3) are
31 V (0)
B1 VoCo (0)
B2 v Q (0)
32 VoVoL (0)
Amax (-12(5)) = X + Trace
B DI V
CIDIT (0,)
1TD VoLa Dia (0)
P
(31)
X+
8 V C (0)
X U=Q))
Amax ( (-4(8)) & À NNUVa 2,, 2
Following the Theorem in Elenberg et al. 2018), we can derive that (S) 6-weakly submodular with 8 2 X44N(UDUea!? And FX(() is d- w-weklly submodular with 62 X+NUVs/at22.0,,400
A.3 UPPER BOUND OF RESIDUAL, ERROR
Theorem 5.2. The residual error Err (w, S, C, is upper bounded according to the sample's s gradient of TD loss:
min
min
VoL (0)
VoL (of (0)
(16)
oEC
ID
Proof. The residual error is larger than the special case where all wy are [D|/|S]:
D
Err (w, S, 0) <
Σ
Vol' (0)
Σ
VoL (0) 1|2
S
iES
IED
Using Jensen's inequality, have
Err (w, S,C,0) <
Σ VoL (0)
1 S
VoL 10)
According to the monotone property of submodular functions. adding more samples to S reduces the residual error. We assume 5 starts with the cluster center {cu it follows that
Err (,,,,,L0)) & Σ
VoL (0) VoL* (0)
Σ min
|v (0) cEc
voL (0)
(32)
Eq. 32 is exactly the optimization objective typical of the clustering problem.
A.4 CONVERGENCE ANALYSIS
Theorem 5.1. Let O* denote the optimal QT parameters, O, the parameters after i training steps. We have
G-1 Σ
Do
min 2(0,) < C(0") t=::
+
8.
(15)
VG
t=1
Here L(8) EIED LTD(S,, as, Ps, $/ 8) is the TD loss, G the number of total training steps,
D ||0" -0, 0,), and Err (w, S. L. 0,) is the gradient approximation errors.
16

---

Published as a conference paper at ICLR 2025
Proof. From the definition of Gradient Descent, we have:
V (0.) (0: - 8*) = (0: 02+1) ( (0c-0% -0)
(33)
at
1
V Vgrrac (0)) ( - )²) = (10: 0c+1ll + +100; - - 02)
(34)
20t
1
(o,r (6:) (0: 0²) 20h
(1 (O))l + 110: 10 110,+1 0²113)
(35)
Then, rewrite the function VoCrae (0,) (0. 0*) as follows:
(e 0*) = (oLzzc(01)3 - 0-) VoL(0)) -0) VoL(O)) 0*)
(36)
Combining the above equations we have:
(0,) 0-) VoL(O,) 0*) VoL(O)) - -0) =
(37)
1
(Ilar VoLrd + 110, -- -10++1 1²))²
(38)
20g
VoL(O)) -0)-
(Ila, VoLrac ||0 1- ||0141 - -00
(39)
2a
(VoLrac (0x) - vol(O,)) ((0)))((000)) 0*)
(40)
Summing up the above equation for different value of t E [0, G 1] and the learning rate ag is constant a, then we have:
G-1 Σ
Σ (0 - t=
- - 8-11 - 1:0 - -0112++
+
a (aVerrac(()))
(41)
2a
2a
1=0
G-1
(V (o (0 (42)
t=
Since 100 0-1|2 2 0, we have:
G-1
G-1
VoL(O))
-0:-0))
- 110
0²1
1
||a Ve crro
(43)
2a
2a
1=0
G-1
+
((VoLrse - VoL(H))) - -0)-
(44)
From the convexity of function L(0), we have:
c(0.) - -(0') VoL(O)) 0*)
(45)
Combining the Equation 44 and Equation 45 have:
17

---

Published as a conference paper at ICLR 2025
G
G-1
((0)) ((0') 1 |10 - 0 1-
1
Σ
la @crac((1)))
(46)
2a
2a
1=0
G-1
+
Σ
(VoLrde
(o -r))
(47)
1=0
We assume that ||0 - ||0| S D. Since
VL(()]| S 0, we have:
D²
G
G-1
Σ - -(0))
D(((veCra(())) VoL(O.)))
(48)
2a
2
Then:
ΣG-1 ((0,) [(0))
D2
ao²
G-1
+
V V (0:) voL(0,)1)
(49)
G
2aG
2
G
1=
Since min(C((.) - [(0))) < Ers (((0)- have:
D
ao²
G-1 Σ
D
- -
<
+
(0)
(50)
2aG
We adopt E to denote (eerac((1)) - voL(ei)] then we have:
min(2(0.) - [()")) <
D2
+
ag
+
G-1
Σ
D
(51)
2aG
2
G
1=
Theorem A.1. The training loss original dataset always monotonically decreases with every
training epoch [(0,-1) & C(O,) if it satisfies the condition that VeL(O,)) (oCrac(O.) 2 0 for
VoL.e
Osts G and the learning rate a min; L VoL,u Velaa (P.)
Proof. Since the training loss L(() is lipschitz smooth, we have:
C(0(+1) & ((8.) + VoL(O,) A + |
(52)
where
A 0:-1 0..
(53)
Since, using SGD optimize the reduced subset training loss (oac(O)) model parameters. The update equation is:
i.t1 = - a
(54)
Combining the above two equations, we have:
L(0041) < L(0.) +
(0.) - - --VooCra((0))
(55)
18

---

Published as a conference paper at ICLR 2025
Next, we have:
(0,)) - _ ( (56)
-(0(41)-A11 - S
From the above equation, we have:
((0(+1) c(0)) if Vol(e))Vooru((O)) aL V 2
(57)
Since V (0))l2 O, we will have the necessary condition V,L(O,)" 2 0. Next, we rewrite the above condition as follows:
al (VoLrc((fe))
(58)
Therefore, the necessary condition for the learning rate is:
2 Vole
S L VoCrac (0.)² V Crac (0t)
(59)
Since the above condition needs to be true for all values for 4, we have the following conditions for the learning rate:
2 Voloo. min T. VoLrae (0.) Cx (0:)
(60)
19

---

Published as a conference paper at ICLR 2025
B ABLATION STUDY
To study the contribution of each component in our learning framework, we conduct the following ablation study. Q Target: We replace the empirical returns used to update Q functions with the standard target Q function in the TD loss function. Single Round: We set the number of data selection rounds in Figure Soo that removing any of these two modules will worsen the performance of REDOR. In case like wa ker2d-med um, ablation Single Round even decrease the performance by over 80%, and ablation Q Target results in 95% performance drop in wa 1 kerdddexpeereeeeee expert. Furthermore, we also find that the fcheetah tasks, the impact of removing the two modules relatively small. This result can be attributable to the fact that this task has limited state space, and we can directly apply OMP to the entire dataset and identify important and diverse data.
and study the function of multi-round data selection. The experimental results
Algorithm ReDOR
Egicade Returns
Single Round
Target
Fipicode Rattures
8C
Algovithm ReDOR
(3)
00
60
Targre
20
20
hooper medium
walker2d medium
holfcheetah medium
waller2d expert
halfcheetah
Figure Ablation results D4RL (Hard) tasks with the normalized score metric.
C COMPUTATIONAL COMPLEXITY
We report the computational overhead of REDOR on various datasets. All experiments are conducted on the same computational device (GeForce RTX 3090 GPU), The results the following Table indicate that
datasets containing millions of data points, the computational overhead remains low. This low computational complexity can attributed to the trajectory-basee bassdd selection technique in Sec. 4.2 (II) and the regularized constraint technique in Sec. 4.3 making our method easily scalable to large-scale datasets.
Env
Data Number
REDOR
Hopper-medium-v0
999981
8m
Walker2d-medium--w
999874
8m
Halfcheetah-mmeiimss
998999
8m
Hopper-expert-v0
999034
8m
Walker2d-expert-vv0
999304
8m
Halfcheectah-expert--00
998999
8m
Hopper-meddium-expertvvv
1199953
8m
Walkerz2--mediiu--cxprr--0
1999179
13m
Hallchhetaa-mcdiiu--eeer---
1997998
14m
Hopper-medium-replay--
200918
3m
Walkerzd--medium--eepaayy
100929
3m
Halfcheetah Halichhetth-mmediiu-rrra----
100899
3m
Table 4: The computational complexity associated with REDOR in various datasets, m represents
minutes.
20

---

Published as a conference paper at ICLR 2025
C.I VISUALIZATION RESULTS
We visualize the selected data of ReDOR on various tasks based on the same method in Section S
Figure 5: Visualization of selected data on hopper-medium-vol
Figure 6: Visualization of selected data on hopper-mmdduum--xpeettt-t.
Figure 7: Visualization of selected data on hopper-expert-vO.
21

---

Published as a conference paper at ICLR 2025
Figure 8: Visualization of selected data on faakerr2d-mcduum-epprreee
Figure 9: Visualization of selected data on walker2d-expert-vO.
3
Figure 10: Visualization of selected data on halfcheetah- sirichettah-mdiuum--ee
Figure 11: Visualization of selected data on halfcheetah-mediung medium-expert--o.....
Figure 12: Visualization of selected data on halfcheetah-exppertvv..
22

---

Published as a conference paper at ICLR 2025
D EXPERIMENTAL DETAILS
Hyper-parameters. For the Mujoco tasks, we adopt the TD3+BC as the backbone of the offline algorithms. For the Antmaze tasks, adopt the IQL as the backbone of the offline algorithms. We outline the hyper-parameters used by REDOR in Table S
Hyperparameter
Value
Optimizer
Adam
Critic learning rate
3e-4
Actor learning rate
3e-4
Mini-batch size
256
Discount factor
0.99
Target update rate
5e-3
Policy noise
0.2
Policy noise clipping
(-0.5, 0.5)
TD3+BC regularized parameter
2.5
Architecture
Value
Critic hidden dim
256
Critic hidden layers
Critic activation function
ReLU
Actor hidden dim
256
Actor hidden layers
Actor activation function
ReLU
REDOR Parameters
Value
Training rounds T
50
50
0.01
Table Hyper-parameters sheet of REDOR
23


Reviews:
[{"review_id": "gndTte10Gm", "reviewer": "Reviewer_uw5n", "summary": "Motivated by the large size of the offline dataset as well as suboptimal data quality in offline RL, this paper considers the problem of finding a coreset out of the given dataset. The authors first formulate such problem as a task to approximate the actual gradients (from the complete dataset) in the offline training process. And a line of of results are provided to support the low approximation errors. Then the method named Reduced Datasets for Offline RL (REDOR) is proposed, inspired by the orthogonal matching pursuit (OMP). Finally, the method is compared with several baseline methods on D4RL data.", "questions": "None", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "> **Originality**\n- Such new method is proposed to select a coreset from the raw offline dataset, which could contribute as an alternative approach in offline RL.\n\n> **Clarity**\n- Several informative figures are provided. Especially the one by t-SNE provides a straightforward way to understand the behaviour of such selection process.\n\n> **Significance**\n- In some of the settings concerned in the experiments, such method is quite efficient.", "weaknesses": "> **Quality**\n- Several assumptions in Theorem 4.1 are rather stronger than scenarios in actual implementations. One observation often seen in offline RL is the diverging gradients (if without proper training techniques), which, however, are assumed to be uniformly bounded in the paper, w.r.t parameters in respectively policies and Q-functions.\n- Despite the multi-round selection strategy introduced in Section 4.2, as long as the empirical returns are used, as depicted in equation (13), the targets in training steps are relatively fixed (in the sense of distributions due to behaviour policies), which then makes (13) no longer an approximation of Bellman backup errors. As a result, it is currently not clear if such approach would lead to a guaranteed good estimation of values/Q-functions.  \n- According to what the reviewer can understand about the statements and proof for results in Section 5, the theorems only consider the proposed method defined with classic TD loss, while do not consider the techniques emphasized in Section 4.2 - 4.3. As a result, such theoretical discussion is not an actual analysis of the proposed algorithm (feel free to correct me).\n- In Line 766, within the proof for Theorem 5.2, it is not justified why $S\\^k$ can always start from the cluster center ${c\\_k}$ of gradients.\n\n> **Clarity**\n- According to the way a Q-function is defined in Line 99, some index of $t$ should be included in the notation of $Q$.\n- Horizon $H$ is not explicitly defined.\n- There is not enough information for $L\\_{\\text{max}}$.\n- There lacks for an introduction to how KRLS, Log-Det and BlockGreedy are implemented in such offline RL settings.\n\n> **Significance**\n- As explained in the 'Quality' part, the theoretical results seem not to be exactly for the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2h6ON3METG", "reviewer": "Reviewer_W8YX", "summary": "This paper introduces a method for dataset selection in offline reinforcement learning (RL) using the Orthogonal Matching Pursuit (OMP) algorithm and Monte Carlo Q losses. The proposed approach selects full trajectories whose loss gradients align well with the residuals.", "questions": "1. Could you please clarify how the suboptimal datasets for MuJoCo, namely \"hard\", were generated? The paper mentioned that they were generated by adding low-quality data, but the quality or source of such data and mix ratio should also be introduced.\n\n2. Regarding $Q_\\theta$ in Algorithm 1, could you explain how $Q_{\\theta_t}$ was formulated? There is no update term in either the pseudocode or the codebase. Was $Q_\\theta$ pretrained or trained simultaneously but omitted? It would be best if the pseudocode or thorough explanations were provided.\n\n3. In Equation 14, it is stated that trajectories in the top $m%$ based on return are filtered, with $m$ set to 50, which would seem to exclude almost the entire random dataset. Could you provide the result of simply selecting trajectories with top $m (=50)\\%$ returns for comparison?\n\n4. In the codebase, it seems that in addition to the evaluation of Monte Carlo Q targets, the selection of candidate trajectories via OMP is filtered based on trajectory returns. What is the exact search space of the selected trajectories? If it is the filtered one with trajectory returns, then how can we ensure the fairness of the comparison to baselines that do not utilize such a filter?\n\n5. In the paper, the percentile $m$ is specified as 50 (Top), but in the codebase, it varies (Bottom 50, 70, and 95). Could you clarify the reason for this difference?\n\n6. In Algorithm 2, $r$ is defined as a scalar, but in Line 4, an inner product is applied. Could you kindly explain this?\n\n7. In Line 3 of Algorithm 2, the inequality appears to be reversed. Is this correct?\n\n8. Is there a reason why TD3+BC was chosen as the backbone offline RL algorithm for the MuJoCo tasks? Would using IQL, as in the Antmaze tasks, provide a more consistent comparison?\n\n9. For the MuJoCo tasks, the authors used the \"-v0\" versions, which are now outdated and differ from the more recent \"-v2\" versions. Could you explain the reasoning behind using \"-v0\"?\n\n10. For the \"Complete Dataset\" scores in the Antmaze tasks, it seems that these values are taken from the IQL paper, which does not provide standard deviations. Could you clarify how these scores were derived?\n\n11. While the baselines used in the experiments appear somewhat dated, dataset selection has recently gained increased attention in offline RL. Hence, it seems that recent algorithms should be contained as baselines. For example, \"Improving Generalization in Offline Reinforcement Learning via Adversarial Data Splitting (Wang et al., 2024)\" provides a codebase, which could allow for a straightforward comparison. Or, is there any reason why such comparisons are inappropriate?\n\n12. Could you provide more details on what is meant by the \"Complete Dataset\" baseline? Specifically, is it the original mixture of the desired dataset and the suboptimal dataset, or is it just the original dataset?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The method demonstrates improved performance compared to the baselines.\n2. The paper includes a theoretical analysis that provides a solid grounding for the approach.", "weaknesses": "1. Some key elements of the proposed algorithm are either missing or unclear, and there are some discrepancies between the paper and the accompanying codebase. For instance, the method used to generate \"hard\" datasets is not fully discussed in the paper, and the percentile $m$ mentioned in the paper differs from that in the codebase. More details are provided in the questions below.\n\n2. Certain parts of the proposed algorithm may contain logical errors or inconsistencies. For example, in Line 4 of Algorithm 2, $r$ is a scalar, yet an inner product operation is applied to it. More details are provided in the questions below.\n\n3. The baselines chosen for comparison seem somewhat outdated, which could affect the perceived significance of the performance improvements demonstrated by the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EInaQosWnX", "reviewer": "Reviewer_wKBZ", "summary": "The paper explores the interesting concept of finding a subset of the offline dataset to improve the performance of offline RL algorithms using orthogonal matching pursuit. The authors provide empirical and theoretical evidence of performance improvement on benchmark datasets.", "questions": "Q1. How is the weight $w_i$ or $\\lambda$ decided during training and the parameters $L_{max}$, $m$,$\\epsilon$ chosen in practice?\n\nQ2. Are the networks Qθ, πϕ networks first trained on the full dataset before starting with the subset selection?\n\nQ3. What is the empirical reduction percentage achieved in each dataset?\n\nQ4. In Figure 1 for the walker2d-expert-v0 environment, the reward first increases and then drops. It is also counterintuitive that the subset selected in ReDOR would perform better than a dataset containing only expert trajectories. Could the authors provide an explanation for this behavior?\n\nQ5. Q5. Could the authors elaborate more on the Prioritize baseline, what do samples with highest TD Loss mean?\n\nQ6. How does ReDOR perform on random datasets such as halfcheetah-random-v2?\n\nQ7. I could not understand Fig 3. Why are the reduced dataset points more for category 6 when it is a subset of complete dataset?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and the idea is easy to follow.\n2. The idea of subset selection is novel and interesting.\n3. The paper provides both strong theoretical study and empirical analysis of the proposed method.", "weaknesses": "1. The authors characterize the field of offline RL only in terms of OOD action penalization and constraints on the behavior policy. There should also be a short discussion on model-based methods like MOPO [1] and MoERL [2], as some of these approaches have been shown to outperform model-free methods.\n\n2. Some parts of the paper are difficult to understand without prior knowledge of orthogonal matching pursuit. Specifically, how is $F\\lambda(s) = L_{max} - min_w Err_{\\lambda} (w, S, L, \\theta)$ used in the OMP.\n\n3. If I understand correctly this method may not lead to the claimed reduction in complexity, as training $Q_{\\theta}$ and $\\pi_{\\phi}$ till requires the full dataset.\n\nMinor \n\nThe table references do not match the table numbers. On line 420, I believe the authors are referring to Table 1 instead of 6.2.\n\nSuggestion : If the authors could include a notations table in Appendix it will help in readability and understanding the proofs.\n\nReferences:\n[1] Kidambi, Rahul, et al. \"Morel: Model-based offline reinforcement learning.\" Advances in neural information processing systems 33 (2020): 21810-21823.\n[2] Yu, Tianhe, et al. \"Mopo: Model-based offline policy optimization.\" Advances in Neural Information Processing Systems 33 (2020): 14129-14142.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3Bogba8xgD", "reviewer": "Reviewer_6MgE", "summary": "The authors introduce an approach for reducing the size of a dataset for offline RL by defining this reduction as a submodular set cover problem and using orthogonal matching pursuit. The resulting algorithm is evaluated on a modified version of D4RL locomotion tasks and the original antmaze tasks.", "questions": "Additional experiments:\n- Does simply filtering the dataset by high returns recover the same performance?\n- What is the performance of ReDOR on the original version of D4RL? One might expect that reducing mixed quality datasets like medium-expert, or medium, could also result in a high performance. \n\nMissing experimental details:\n- How is the hard dataset generated? How many datapoints are added to the dataset?\n- How many datapoints are removed by ReDOR? What is the size of the reduced datasets?\n\nGeneral:\n- Is there a way to tune the resulting dataset size? \n- Is Fig 3, episode return = 99.5 for behaviors [2-7] correct or a bug?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This is an interesting and novel approach for data selection in RL. The high-level approach/formulation of the problem may be useful as a foundation for extensions. \n- Strong results on a modified version of D4RL, and the unmodified antmaze.", "weaknesses": "There is a discrepancy between the proposed objectives and the resulting objectives that makes me question where the effectiveness of the proposed approach comes from. \n\nThe problem is initially defined as finding a subset of the data which results in a higher performing policy than the policy determined by training on the original dataset (Eqn 3). However, this is immediately discarded for another optimization problem, which instead tries to limit change in the value function (Eqn 5). While discovering a smaller dataset which achieves the same performance as the original dataset is an interesting problem, the authors claim in several places (and demonstrate) that their reduced dataset actually improves the performance. So where does the performance gain come from? \n\nOne possible cause for the performance increase is how the evaluation is done (add noisy/low performing trajectories to the D4RL dataset) and the filtering of low performing trajectories (Eqn 14). I would be very curious if this filtering alone is sufficient to also recover the performance of the algorithm. This concern, along with some missing key experimental details, makes me cautious about the experimental claims made in the paper. \n\nMissing References which also filter the dataset using returns:\n- [1] Chen, Xinyue, et al. \"Bail: Best-action imitation learning for batch deep reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 18353-18363.\n- [2] Yue, Yang, et al. \"Boosting offline reinforcement learning via data rebalancing.\" arXiv preprint arXiv:2210.09241 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)