Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
Authors: Yi Ma
Jianye HAO
Xiaohan Hu
YAN ZHENG
Chenjun Xiao
Venue: NeurIPS 2024 poster (2024)
Keywords: Offline RL
Primary area: reinforcement_learning

Abstract: One of the fundamental challenges for offline reinforcement learning (RL) is ensuring robustness to data distribution. Whether the data originates from a near-optimal policy or not, we anticipate that an algorithm should demonstrate its ability to learn an effective control policy that seamlessly aligns with the inherent distribution of offline data. Unfortunately, behavior regularization, a simple yet effective offline RL algorithm, tends to struggle in this regard. In this paper, we propose a new algorithm that substantially enhances behavior-regularization based on conservative policy iteration. Our key observation is that by iteratively refining the reference policy used for behavior regularization, conservative policy update guarantees gradually improvement, while also implicitly avoiding querying out-of-sample actions to prevent catastrophic learning failures. We prove that in the tabular setting this algorithm is capable of learning the optimal policy covered by the offline dataset, commonly referred to as the in-sample optimal policy. We then explore several implementation details of the algorithm when function approximations are applied. The resulting algorithm is easy to implement, requiring only a few lines of code modification to existing methods. Experimental results on the D4RL benchmark indicate that our method outperforms previous state-of-the-art baselines in most tasks, clearly demonstrate its superiority over behavior regularization.

Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
Yi Ma Jianye Hao3 Xiaohan Hu?, Yan Zheng Chenjun Xiao
School of Computer and Information Technology, Shanxi University, mayi@sxu. eeua..ddu.com
"Key Laboratory of Computational Intelligence
and Chinese Information Processing of Ministry of Education 3 College of Intelligence and Computing. Tianjin University fjine hao, auxiachan, yanzheng)&tjju edd..euuu....uuu...ee.. edu. cn
*Noah's 's Ark Lab, Huawei
The Chinese University of Hongkong, Shenzhen, chenjunx@cuhk. edu. cn
Abstract
One of the fundamental challenges for offline reinforcement learning (RL) is ensuring robustness to data distribution. Whether the data originates from a near- optimal policy or not, we anticipate that an algorithm should demonstrate its ability to learn an effective control policy that seamlessly aligns with the inherent distribution of offline data. Unfortunately. behavior regularization, simple yet effective offline RL algorithm, tends to struggle in this regard. In this paper, we propose new algorithm that substantially enhances rehaaior-reuularizztoon based conservative policy iteration. Our key observation is that by iteratively refining the reference policy used for behavior regularization, conservative policy update guarantees gradually improvement, while also implicitly avoiding querying out-of- sample actions to prevent catastrophic learning failures. We prove that the tabular setting this algorithm is capable of learning the optimal policy covered by the offline dataset, commonly referred to as the in-sample optimal policy. We then explore several implementation details of the algorithm when function approximations are applied. The resulting algorithm is easy to implement, requiring only few lines of code modification existing methods. Experimental results the D4RL benchmark indicate that our method outperforms previous state-of-the-art baselines in most tasks, clearly demonstrate its superiority over behavior regularization.
Introduction
Reinforcement learning (RL) has achieved considerable success in various (ecision-making problems, including games M. software automation testing (2]. recommendation and advertising (3), logistics optimization [4] and robotics [5]. A critical obstacle that hinders a broader application of RL is its ffal-and-erres learning paradigm. For applications such as education, autonomous driving and healthcare, active data collection could be either impractical or dangerous [6]. Instead of learning actively, more favorable approach employ scalable data-driven learning methods that can utilize existing data and progressively improve as more training data becomes available. This motivates offline RI, of which the primary objective is to learn a control policy solely from previously collected data without online interactions with environment.
Offline datasets frequently offer limited coverage of the state-action space. Directly utilizing standard RL algorithms to such datasets can result in extrapolation errors when bootstrapping from out-of- distribution (OOD) state-actions, consequently causing significant overestimations in value functions.
"Corresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

To address this, previous works impose various types of constraints to promote pessimism towards accessing OOD state-actions. One simple yet effective approach is behavior regularization, which penalizes significant deviations from the behavior policy that collects the offline dataset 7 g 10
beggat-mnddinneesat-222
walkio 20-0::00---memmmmppppp....
Nomalledd
80
20
21
0.4
0.6
0.6
Line (abp
Time Stepst
Tune Stope (o0)
Staps
1pp Spp-
mide
bottee:
The dashed lines indicate the performance of reference policies trained on filtered sub- datasets consisting of top 5%, median 5%, and bottom 5% trajectories. The solid lines indicate the performance of TD3 with different reference policies. It can be concluded that behavior regularization has advantage over behavior cloning and the efficacy of behavior regularization is strongly contingent on the reference policy.
Figure 1:
A fundamental challenge in behavior regularization lies in its performance being closely tied to the underlying data distribution. Previous works suggest that it often yields subpar results when applied to datasets originating from suboptimal policies [12] [3]. To better understand this phenomenon, we investigate how the quality of a behavior policy influences the performance of behavior regularization through utilizing percentile behavior cloning [14. Given an offline dataset, we create three sub- datasets by filtering trajectories representing the top 5%, median 5%, and bottom 5% performance. We then leverage these sub-datasets to train three policies-referree to as top, median, and botlom-using behavior cloning. We then develop TD3 with percentile behavior cloning (TD3+%BC), which optimizes the policy by max, E.~D 00 (s) - A(=(s) T% (s)) where T% E {top, median, bottom). This variant of TD3+BC simply replaces the behavior policy with the percentile cloning policy. We refer T% the reference policy. Our main hypothesis is that better reference policy (e.g. top) can dramatically improve the efficiency of behavior regularization compared to a worse reference policy (e.g. bottom). Figure[] the results medium-replay and walker-medium--eppe First, the efficacy of behavior regularization is strongly contingent on the reference policy. For example, TD3+top%BC dominates TD3+bottomm across all benchmarks. This confirms our hypothesis. Second, behavior regularization guarantees improvement. No matter using top. median or bottom, TD3+%BC produces a better or similar policy compared to the reference policy, clearly demonstrating the advantage of behavior regularization over behavior eloning.
copper-medium--ee expert, walker- from the D4RL datasets [15]. A few key observations.
Inspired by these findings, we ask: is it possible to automatically discover good reference policies from the data increase the efficiency of behavior regularization? This paper attempts to give an affirmative answer to this question. Our key observation is that Conservative Policy Iteration, an algorithm that has been widely used for online RL [16] 17 18 191, can also be extended to the offline setting with minimal changes. The core concept behind this approach hinges on the iterative refinement of the reference policy utilized for behavior regularization. This iterative process enables the algorithm to implicitly avoid resorting to out-of-sample actions, all while ensuring continuous policy enhancement. We provide both theoretical and empirical evidence to establish that, for the tabular setting, our approach is capable of learning the optimal policy covered by the offline dataset, commonly referred to as the in-sample optimal policy [12] [3]. We then discuss several implementation details of the algorithm when function approximations are applied. A previous work STR (20) shares the similar idea with CPI. However, the theoretical foundation and the resulting algorithms are quite distinct, which is discussed in appendix. Our method can be seamlessly implemented with just few lines of code modifications built upon TD3+BC [T]. We evaluate our method on the D4RL benchmark [15]. Experiment results show that it outperform previous state-of-the-art methods on the majority of tasks, with both rapid training speed and reduced computational overhead.

---

2 Preliminaries
2.1 Markov Decision Process
We consider Markov Decision Process (MDP) determined by M = PS.A, 22 where S and A represent the state and action spaces. The discount factor is given by E 10, r: A I denotes the reward function, P S x A - ASS) defines the transition dynamics Given policy + (AA), we use E" to denote the expectation under the distribution induced by the interconnection of и and the environment. The value function specifies the future discounted total reward obtained by following policy R,
20
8 E
Σ
rr(s,, (1((s,,a))
The state-action value function is defined as
(s +BB s.0)) (.)))V'((s))
(2)
There exists an optimal policy x² that maximizes values for all states E S. The optimal value functions, V* and Q", satisfy the Bellman optimality equation,
(*(s) = maxr(s, + 7B (5'),,
(' a) = r(s,a) + 7Bs max Q' (5', (s'.
2.2 Offline Reinforcement Learning
In this work, we consider learning an optimal decision making policy from previously collected offline dataset, denoted as D = {8i, a; Ti, Ti, s,, s0)==0--- The dataset is generated following this procedure: 8; ~ p. ag & ~ P(- [2, a,), distribution over states, and FD is an unknown behuvior policy. In offline RL, the learning algorithm only take samples from D without collecting new data through interactions with the environment.
a,), where p represents an unknown probability
Behavior regularization is simple yet efficient technique for offline RL 87. It imposes a constraint on the learned policy to emulate TD according to some distance measure. A popular choice is to use the KL-divergence
max EaNT [Q(s, a]] DDL (T(s)][ (s))
where T >> a pyperr-arammter. Here, Q is some value function. Typical choices include the value of TD [10, or the value of the learned policy x 8 [1. As shown by Figure the main limitation of behavior regularization is that it relies on the datasct being generated by an expert or near-optimal TD. When used datasets derived from more suboptimal pelicies-ttyicee of those prevalent in real-world applications- these methods do not yield satisfactory results.
3 Iteratively Refined Behavior Regularization
In this paper, we introduce new offline RL algorithm by exploring idea of iteratively improving the reference policy used for behavior regularization. Our primary goal is to improve the robustness of existing behavior regularization methods while making minimal changes to existing implementation. We first introduce conservative policy optimization (CPO), commonly used technique in online RL, then describe how can also shed some light on developing offline RL algorithms.
Let 7 > 0, the CPO update the policy for any s E S with the following policy optimization problem
max
Ea [Q (s, a)] - DKL (T())
(5)
We use ((Z') to denote the set probability distributions over X for finite 'We note that although [1I] applies behavior cloning term as regularization, this is indeed KL regulariza- tion under Gaussian parameterization with standard deviation.

---

which generalizes behavior regularization using an arbitrary reference policy म. The idea is to optimize a policy without moving too far away from the reference policy increase learning stability. As shown in the following proposition, this conservative policy update rule enjoys two intriguing properties: first, it guarantees policy improvement over the reference policy тद and second, the updated policy still stays on the support of the reference policy. These properties also echo our empirical findings presented in Figure [ as a special case of CPO with मं = TD. behavior regularization is stable in offline learning and always produces better policy than TD.
Proposition 1. let Re be the optimal policy of ()) For any s E S, we have that V * (s) 2 V=(s; and R² (al () 0 given (als)) 0.
Proof. The proof of this result, as well as other results, are provided in the Appendix.
In summary, the conservative policy update (5) implicitly guarantees policy improvement constrained on the support of the reference policy æ. By extending this key observation in an iteratively manner, we obtain the following Conservative Policy Iteration (CPI) algorithm for offline RL. It starts with the behavior policy mo = TD. Then in each iteration f = 0,1,2,... the following computations are done:
* Policy evaluation: compute Q and;
Policy improvement: Vs E S, M++ = arg max, Eawr (s, DKT. (T|[m))
In this approach, the algorithm commences with the behavior policy and proceeds to iteratively refine the reference policy used for behavior regularization. Thanks to the conservative policy update, CPI ensures policy improvement while mitigating the risk of querying any OOD actions that could potentially introduce instability to the learning process.
We note that CPO is a special case of mirror descent in the online learning literature (22] Extending this technique to sequential decision making has been previously investigated in online RL [16]
18 19..
In particular, the Politex algorithm considers exactly the same update as (5) for online RL This algorithm can be viewed as a softened or averaged version of policy iteration. Such averaging reduces noise of the value estimator and increases the robustness policy update. Perhaps surprisingly, our key contribution is to show this simple yet powerful policy update rule also facilitates offline learning, as it guarantees policy improvement while implicitly avoid querying OOD actions.
17
3.1 Theoretical Analysis
We now analyze the convergence properties of CPI. In particular, we consider the tabular setting with finite state and action space. Our analysis reveals that in this setting, CPI converges to the optimal policy that are well-covered by the dataset, commonly referred to as the in-sample optimal policy. Consider the in-sample Bellman optimality equation (2
V (+)
max
{r(s, a) + +
VV% (s')
(6)
(a)s(a
This equation explicitly avoids bootstrapping from OOD actions while still guaranteeing optimality for transitions that are well-supported by the data, A fundamental challenge lies in the development
of scalable algorithms for its resolution 233 12 13
Our next result shows that CPI provides simple
yet effective solution.
Theorem We consider tabular MDPs with finite S and A. Let n. be the produced policy of CPI at iteration t. There exists a parameter T >> O such that for any E S
2 log IA
(8) m2
(*( S (1 71-
t
To ensure robustness data distribution, an offline RL algorithm must possess the stitching capability, which involves seamlessly integrating suboptimal trajectories from the dataset. In-sample optimality provides a formal definition to characterize this ability. As discussed in previous works and confirmed by our experiments, existing behavior regularization approaches, such as TD3+BC [II, fall short in this regard. In contrast, Theorem [ suggests that iteratively refining the reference policy for behavior regularization can enable the algorithm to acquire the stitching ability, marking significant improvement over existing approaches. Our empirical studies in the experiments section further support this claim.

---

Practical Implementations
In this section discuss how to implement CPI properly when function approximation is applied. Throughout this section we develop algorithms for continuous actions. Extension to discrete action setting is straightforward. We build CPI as an actor-critic algorithm. We learn an actor Tw with parameters w, and critic Qo with parameters 6. The policy is parameterized using Gaussian policy with learnable mean (24]. We also normalize features of every states in the offline dataset as discussed in [TI].
CPI consists of two steps each iteration. The first step involves policy evaluation, which is carried out using standard TD learning: we learn the critic by
min E B. f Qa(s.
where Qa is target network. We also apply the double-O - trick stabilize training 24 The policy improvement step requires more careful algorithmic design. The straightforward implementation is use gradient descent on the following
max Ba-D
- (F
Here, the reference policy # is copy of the current parameters and kept frozen during optimization. This leads to the so-called iterative actor-critic or multi-step algorithm [1O]. In their study, [[O] observe that this iterative algorithm frequently encounters practical challenges, mainly attributed to the substantial variance associated with off- oof-policy evaluation. Similar observations have also been made in experiments (See Figure 4) We conjecture that while Proposition establishes that the exact solution of (9) remains within the data support when the actor is initialized as ты Wp. practical implementations often rely limited number of gradient descent steps for optimizing (9). thus could suffer from our-of-support samples. This leads to policy optimization errors which further exacerbated iteratively. An option to approximate the in-support learning importance sampling on the dataset to mimic sampling from the current policy. However, the high variance of importance sampling could cause the training unstable 20). We instead find it useful to add the original behavior regularization,
utilizes
max EAND E
Barx 1Q0(s, (
- - (T.
(10)
which further constrains the policy on the support of data to enhance learning stability. The parameter A balances between one-step policy improvement and behavior regularization. Note that the term Dit. (Tw(s) (+) [=))))))))))))))))))))))))))([)) in CPI is the only difference compared to TD3+BC [I]]. In practice this can be done with line code modification based TD3+BC implementations.
Algorithm 1 CPI & CPI-RE
1: Initialize actors Nı, Ty and critic networks qn G2 with random parameters w w2, 01, 0²: target networks 0 < 02, -- 02, ws -- wt, 8²2 2
2: for (orl=1 2...,, T do
3: Sample a mini-batch of transitions from dataset
4: Update the parameters of critic 8 using equation
5:
t mod? 2 then
6:
// For CPI
7:
Copy the historical snapshot of w to "2 and update w using equation 10 // For CPI-RE
8:
9:
Choose the current best policy between wi and a as the reference policy according to Q-value
10:
Update wt and 22 using equation 10 respectively
11:
Update target networks
12:
end if
13:
end for
5

---

Ensembles of Reference Policy One limitation of (TO) lies in the potential for negligible difference between the learning policy and the reference policy, due to the limited gradient steps when optimizing. This minor discrepancy may restrict the policy improvement over the reference policy. To improve the efficiency of policy improvement, we explore the idea of using an ensembles of reference policies. In particular, we apply two policies with independently initialized parameters w and w² Let Qa: and Qo= be the value functions of these two policies respectively. When updating the parameters w' for E {1,2}, 2]. choose the current best policy as the reference policy, where the superiority is decided according to the current value estimate. In other words, we only enable a superior reference policy to elevate the performance of the learning policy, preventing the learning policy from being dragged down by an inferior reference policy. We call this algorithm Conservative Policy iteration with Reference Ensembles (CPI-RE). We give the pseudocode of both CPI and CPI-RE in Algorithm
1 For CPI, the training process is exactly the same with that of TD3+BC except for the policy updating (marked in pink). The difference between CPI-RE and CPI is also reflected in the policy updating (marked in teal).
Experiment
7*7-GridWerld
FourRoom- Expeet
FourRoom Random
FourRoom Missiin--sstiiii
Roum partt
0060006060000
6008809660800000
100
9800860666000seo
20
50
SO
25
15
25
50
75
100
92
100
25
15
DH0.2
Noar par tpoods
1001
100)
100
100
00000000000000000
000 po6p0060es00coo0
1000000 000.0000000000 00000000000000000000000000000000
000++--
75
100
30
100
50
75
100
25
75
Jatumn Natun-eriiion pertpinkk
00000000000000
008866898600eeecoo
0=0.7
Steps
BR 1AAC
---
CR
Figure 2: Training curves of BR, InAC and CPI on 7 and FourRoom. CPI converges to the oracle across various and environments settings, similar to the in-sample optimal policy InAC.
We subject our algorithm to series of rigorous experimental evaluations. We first present an empirical study to exemplify CPI's optimality in the tabular setting. Then, we compare the practical implementations of CPI, utilizing function approximation, against prior state-of-the-art algorithms in the D4RL benchmark tasks [15]. to highlight its superior performance. In addition, we also present the resource consumption associated with different algorithms. Finally, comprehensive analysis of various designs to scrutinize their impact the algorithm's performance are provided.
5.1 Optimality in the tabular setting
We first conduct an evaluation of CPI within two GridWorld environments. The first environment's map comprises a 7 * grid layout and the second environment's 's map is the FourRoom [13]. In both environments, the agent is tasked with navigating from the bottom-left the goal positioned in the upper-right in as few steps as possible. The agent has access to four actions: fup. down, right, left). The reward is set -1 for each movement, with substantial reward of 100 upon reaching the goal; this incentivizes the agent to minimize the number of steps taken. Each episode is terminated after 30 steps, and 7 is set to 0.9. We use an inferior behavior policy to collect 10k transitions, of which the action probability is I,,:0.1, down:0.4, right:0.1 1, left:0.4) at every state in the 7 7 grid environment. For the FourRoom environment, we use three types of behavior policy to collect data: (1) Expert dataset: collect 10k transitions with the optimal policy; (2) Random dataset: collect 10k transitions

---

with random restart and equal probability of taking each action; (3) Missing-Actionn dataset: remove all down actions transitions of the upper-left room from the Mixed dataset. Although some behavior policies are suboptimal, the optimal path is ensured to exist in the offline data, in which case clear algorithm should still be able to identify the optimal path.
We consider two bascline algorithms: InAC [13], method that guarantees to find the in-sample softmax, and method that employs policy iteration with behavior regularization (BR), which could be viewed as an extension of TD3+BC [T] for the discrete action setting, The policies derived from each method are evaluated using greedy strategy that selects actions with highest probability.
As illustrated in Figure both CPI and InAC converge to the oracle across various and environments settings. In contrast, BR underperforms when larger 7 is applied on 7*7-GridWorld and FourRoom- random, as larger r means more powerful constraint on the learning policy, BR thus become more similar the behavioral policy.
5.2 Results on Continuous Control Problems
Table 1: Average normalized scores of CPI with the mean and standard deviation and previous methods on the D4RL benchmark. D-QL is short for Diffusion-QL. CPI achieves best overall performance among all the methods and consumes quite few computing resources. The top-3 results on each dataset is marked bold.
Daraset
DT
TD348C
IQU
EDAC
S1R
D34QI.
CPI
CPI-RE
halfehretah random
2.2
11.0
31.3
13.7
29.0
28.4
19.6
20.6
22.0
29.7.1
30.7.10.4
ropper-sndoo
5.4
8.5
5.3
8.4
12.0
25.3
32.4
31.3
18.3
29.5++3.7
30.4+29
waker 2d-rnod-rnndom
6.3
5.9:1177
5.5-10.9
halfeheetul.medlum
48.3
46.9
47.4
51.8
64.4+1.3
65.9+1.6
hopprr-medium
67.6
59.3
61.9
66.3
98.2
101.6
60.3
101.3
96.6
3.0
97.9214.4
waker2d medium
74.0
83.7
79.5
78.3
81.1
92.5
82.7
85.9
87.3
85.8.0..8
86.311.0
halfcheetah medium- replay
41.3
54.611.3
55.911.
hopper medium replay
82.7
60.9
86.3
94.7
98.9
10L.
92.1
100.0
102.0
101.7:16
103.2:1.44
waker2d- medium--epiin
66.6
81.8
76.8
73.9
76.6
87. 1
69.8
85.7
98.0
91.++29
93.8+2.2
halfcheetah medium- expert
95.0
86.7
106.3
83.5
94.7+1.1
95.6+00.9
hoper-medduu-exper
107.6
98.0
96.9
91.5
90.0
110.7
93.8
111.9
112-3
106.4:43
110.1
waker2d mwdddddddoddmmeeee expert
108.1
110.1
109.1
109.6
109.1
114.7
109.0
110.2
111.2
110.910.4
111.210,5
halfcheetah- expert
87.7
97.3
94.9
93.2
106.8
93.6
95.2
96.3
96.540.2
97.4=0.44
hopper-expen expert wal 2xppert
110.4
112.2:0.5
112.3.10.5
108.3
110.2
109.3
109.7
102.9
115.1
110.6
110.1
109.5
110.6+00.1
111.2+0.2
Gym- MuJooo Total
972.6
1013.2
1052.8
1034.0
1094.7
1243,4
1019.7
1162.2
1158.6
1193.2
1207.4
antmaze- uiazze
78.6
74.0
87.5
16.7
84.8
988+111
antmase unize-diveree
$3.0
71.4
84.0
62.2
64.8
0.0
82.4
77.4
84.0
88.615.77
92.6-10.0
medium
0.0
3.0
61.2
71.2
87.2
0.0
82.6
824158
84.8.5.0
antmazz- modium-divvass
804189
80.6-11.3
antmaze- Large-plan
0.0
0.0
15.8
39.6
24.4
0.0
42.8
49.0
20.6:116.3
33.6:18.1
lamaze-large-ivvess
0.0
0.2
14.9
47.5
59.2
0.0
46.8
61.7
45.2+66
4..
Total
112.2
163.8
387.6
452.5
438.8
pen-human
73.9
-1.9
35.2
71.5
76.9
52.1
52.3
75.7
80.1+16.9
87.0-255
pen-cloned
67.3
9.6
27.2
37.3
67.6
68.2
-8.0
60.8
71.8::5522
70.7=115.8
Total
141.2
44.3
136.5
157.7
Total
1226.0
1184.7
14188
1520.8
1626.8
1380.4
1747.6
1761.2
1803.
Runtime (s'epach)
7.4
19.6
39.8
8.5
19.1
GPU Memory (GB)
1.5
1.4
In this section we provide suite of results using three continuous control tasks from D4RL 15 Mujoco, Antmaze, and Adroit. Mujoco, a benchmark often used in previous studies forms the basis of our experimental framework. Adroit is a high-dimensional robotic manipulation task with sparse rewards. Antmaze, with its sparse reward property, necessitates that the agent learns to discern segments within sub-optimal trajectories and to assemble them, thereby discovering the complete trajectory leading to rewardable position. Given that the majority of datasets for these tasks contain substantial volume of sub-optimal or low-quality data, relying solely on behavior-regularization may be detrimental to performance.
We compare CPI with several baselines, including DT [1]. TD3+BC [T]. CQL 255, IQL [12]
POR (26). EDAC 27 .Diffusion-QL. (28), InAC [13 and STR 20 The results of baseline methods are either reproduced by executing the official code or sourced directly from the original papers. Unless otherwise specified, results are depicted with 95% confidence intervals, represented by shaded areas in figures and expressed standard deviations in tables. The average normalized results of the

---

final 10 evaluations for Mujoco and Adroit, and the final 100 evaluations for Antmaze, are reported. More details of the experiments are provided in the Appendix.
Our experiment results, summarized in T clearly show CPI outperforms baselines in overall. In most Mujoco tasks, CPI surpasses the extant, widely-utilized algorithms, and it only slightly trails behind the state-of-the-an EDAC method. For Antmaze and Adroit, CPI's performance is on par with the cop-performing methods such POR and Diffusion-QL. We also evaluate the resource consumption of different algorithms from two aspects: (I) runtime per training epoch (1000 gradient steps); (2) GPU memory consumption. The results Table I show that CPI requires fewer resources which could be beneficial for practitioners.
5.3 Ablation Studies
5.3.1 Effect of Reference Ensemble
We provide learning curves of CPI and CPI-RE Antmaze in Figure to further show the efficacy of using ensemble of reference policies. CPI-RE exhibits a more stable performance compared to vanilla CPI, also outperforming IQI.. Learning curves other domains provided in the Appendix.
antimaze medintt-glas-v0
artirazz mediam divene-vO
arcmaze arcmazz-large-paae parappp------p----0 play-v0
astmuae warge-aivene----e..... divens--vl
100
1.0
0.2
Tine Stepaclo)
Tiae Stepe
Tiae Stepxllo)
Tiax Stepi(le8)
CH
CT--22
KUL
Figure 3:
The learning curves of CPI, CPI-RE, and IQL on Antmaze. With reference ensemble, CPI-RE exhibits more stable performance compared to vanilla CPI, also outperforming IQL.
5.3.2 Effect of using Behavior Regularization
direct implementation based on the theoretical results observed in the tabular setting could be derived. Specifically, we initialize Ru Tp and execute the CPI without incorporating behavior regularization. For each gradient step, we perform one-step or multi-step updates for both policy evaluation and policy improvement. The empirical outcomes presented in Fig underscore the poor performance of this straightforward approach. Similar trends are also observed in [10]. Such phenomenon arises primarily because, in the presence of function approximation, the deviation emerges when policy optimization does not necessarily remain within the defined support without the behavior regularization.
boppor-mediimmv222
begget-mediim-ccllly-22
weller24-mccium-22
100
30
20
20
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.5
0.5
1.0
0.2
0.4
0.6
0.5
1.0
0.2
0.4
0.6
0.5
1.0
Timee Suepi()
Time Stepelled)
Time Sueel (nf)
Time Stgge lefo
CP1
upuliing beherke regalanization
zaflistay spiiting
behavin regulatization
Figure 4: Effect of using Behavior Regularization. Without it, both one-step and multi-step updating methods each gradient step suffer from deviation derived from out-of-support optimization.
5.3.3 Effect of using different KL strategies
In our implementation of the CPI method, we employ the reverse KI. divergence. This is distinct from the forward KI. divergence approach adopted by (29]. comprehensive ablation of incor- porating different KL divergence strategies in CPI is presented in Fig As evidenced from the

---

results, our reverse KL-based CPI exhibits superior performance compared to the forward KL-based implementations. Implementations details of CPI with forward KL are provided in Appendix.
hopper-medium--22
Dxcper-medium-mppee
waller24-medium-22
waller2d-medium reploll-uunnn--22222
20
0.2
044
U1.
6.2
014
0.6
112
0.4
0.5
0.5
11.4
0.6
0.8
Tine SAqui()
Vieee Stepelleh)
Tinz Steper (e0)
Time Stapelect)
Cit
C31 Eorward
Figure 5: Effect of using different KL strategies. Reverse KL-based CPI exhibits superior performance compared the forward KL-based CPI. See Appendix for details of forward KL.
5.3.4 Effect of Hyper Parameters
Figure 6 illlutraate the effects of using different hyperparameters and X, offering valuable insights for algorithm tuning. The weighting coefficient À regulates the extent of behavior policy integration into the training and affects the training process. As shown in Figure Gh, when 0.1, the early- stage performance excels, as the behavior policy assists in locating appropriate actions in the dataset. However, this results in suboptimal final convergence performance, attributable to the excessive behavior policy constraint on performance improvement. For larger values, such as 0.9, the marginal weight of the behavior policy leads to performance increase during training. Unfortunately. the final performance might be poor. This is due to that the policy does not have sufficient behavior cloning guidance, leading potential distribution shift during the training process. Consequently, we predominantly select À value of 0.5 or 0.7 to strike balance between the reference policy regularization and behavior regularization. The regularization parameter т plays crucial role in determining the weightage of the joint regularization relative the Q-value component. We find that (Figure 6b) T assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).
halfcheria meSium
hopper
halfcheetuh mediun
halfihersil
50
201
20
20
0.2
0.4
9.6
D.8
0.2
0.5
1.0
0.2
1V4
01.6
3.5
3.0
0.2
0.4
246
0.8
Time Sipel
Time Skepelich)
Time Stepeliot)
Tame Skgel
2.855
A.355
T.055
0.0.1
T.1.D
(a) Effect of weighting coefficient
(b) Effect of regularization hyperparameter
Figure 6: Hyperparameters ablation studies. The tuning experience drawn from this include: (1)
could mostly be set 0.5 0.7; (2) The higher the dataset quality is, the higher should be
5.4 Online Fine-tuning
The distinctive feature of CPI lies in its policy iteration training, which makes it particularly well- suited for the online fine-tuning process. Fine adjustments to CPI enable its seamless application in online fine-tuning- In online setting, the agent interact with the environment, there is a need to progressively enhance the level of exploration during the training process. To achieve this, the À parameter in formula TO is modified. Specifically, in the online training process, the weight of (KL((T,,) (s) (s)][((s)) remains constant, while the weight of (s)] decreases exponentially, eventually reducing to 0.1. This progressive adaptive exploration mechanism allows the algorithm to adopt a conservative exploration strategy in its initial stages and then gradually expand its exploration scope, thereby enhancing the model's 's adaptability in the online learning phase.

---

Empirical evaluations were conducted on nine Mujoco datasets, including hallcheetaa--medium- haffcheetal-mmdiuu--rllaa--- halfcheetah-meddiu- expert-v2, hopper- hopper-medium--epllay
Nine tasks of Mujoco in D4RL
Normaliand Score.
v2, logper-mediimm-experryy walker2d-medium-v2,
60
walker2d--eddiu--rrpha replay... and walker2d-medium- expert-v2. We compare our algorithms with several baselines including TD3+BC, IQL, Cal-QL [30] and PEX BI. As shown in Figure 7 the CPI algorithm achieves exceptional overall performance compared to the state-of-ther the-art offline-to- to-online algorithms Cal-QU and PEX. Detailed results for each Mujoco dataset are provided in the Appendix.
200
Time Steps(1ed)
Figure 7: Overall finetuning performance on Mujoco datasets.
Conclusion
In this paper, propose innovative offline RL algorithm termed Conservative Policy iteration (CPI). By iteratively refining the policy used for regularization, CPI progressively improves itself within the behavior policy's 's support and provably converges to the in-sample optimal policy in the tabular setting. We then propose practical implementations tasks. Experimental results the D4RL benchmark show that CPI surpass previous cutting-edge methods in a majority of tasks various domains, offering both expedited training speed and diminished computational overhead.
CPI for solving continuous control
Nonetheless, our study is not devoid of limitations. For instance, our method's performance with func- tion approximation is contingent upon the selection of two hyperparameters, which may necessitate tuning for optimal results.
Acknowledgments and Disclosure of Funding
The work is supported by the National Natural Science Foundation of China (Grant Nos: 62422605, 92370132, 62406271, 62106172), the National Key R&D Program of China (Grant No. 20222D01 16402) and the Xiaomi Young Talents Program of Xiaomi Foundation.
References
[1] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-535 533, 2015.
[2] Yan Zheng, Xiaofei Xie, Ting Su, Lei Ma, Jianye Hao, Zhaopeng Meng, Yang Liu, Ruimin Shen, Yingfeng Chen, and Changjie Fan. Wuji: Automatic online combat game testing using evolutionary deep reinforcement learning. In 2019 34th IEEF/ACM International Conference on Automated Software Engineering (ASE), pages 772 784. JEEE, 2019.
[3] Xiaotian Hao, Zhaoqing Peng. Yi Ma, Guan Wang. Junqi Jin, Jianye Hao, Shan Chen, Rongquan Bai, Mingzhou Xie, Miao Xu. al. Dynamic knapsack optimization towards efficient multi-channel sequential advertising, International Conference on Muchine Learning, pages 4060-4070. PMLR, 2020.
[4] Ma, Xiaotian Hao, Jianye Hao, Jiawen Lu, Xing Liu, Tong Xialiang. Mingxuan Yuan, Zhigang Li, Jie Tang, and Zhaopeng Meng. A hierarchical reinforcement learning based optimization framework for large-scale dynamic pickup and delivery problems. Advances in Neural Information Processing Systems, 34:23609-22 236200, 2021.
Ajay Mandlekar, Fabio Ramos. Byron Boots, Silvio Savarese, Li Fei-Fei, Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale learning control from offline robot manipulation data. 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420. IEEE, 2020.
[6] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint arXiv:2005.016643 2020.
10

---

[7] Aviral Kumar, Justin Fu. Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems. 32, 2019.
[8] Yifan Wu, George Tucker, and Ofrr Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1. 1331,, 2019.
[9] Noah Y Siegel, Jost Tobias Springenberg. Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behav- ioral modelling priors for offline reinforcement learning. arXiv preprint arXiv:2002.08390 2020.
[10] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offine rl without off-policy evaluation. Advances in Neural Information Processing Systems, 34:4933 2021.
[11] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 32-20132-20145, 2021.
[12] lly Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations. 2022.
[13] Chenjun Xiao, Han Wang, Yangchen Pan, Adam White, and Martha White. The in-sample softmax for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
[14] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover. Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer; Reinforcement learning via sequence modeling, Advances in neural information processing systems, 34:15084- 15097, 2021.
[15] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
[16] Abbas Abdolmaleki, Jost Tobias Springenberg, Yuval Tassa, Remi Munos, Nicolas Heess, and Mar- tin Riedmiller. Maximum a posteriori policy optimisation. International Conference on Learning Representations, 2018.
[17] Yasin Abbasi-Yadkori, Peter Bartlett, Kush Bhatia, Nevena Lazic, Csaba Szepesvari, and Gellért Weisz. Politex: Regret bounds for policy iteration using expert prediction. In International Conference on Machine Learning, pages 6692-3702. PMLR, 2019.
[18] Jincheng Mei. Chenjun Xiao, Ruitong Huang, Dale Schuurmans, and Martin Müller. On principled entropy exploration in policy optimization. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 3130-3136, 2019.
[19] Matthieu Geist, Brumo Scherrer, and Olivier Pictquin. A theory of regularized markov decision processes. In International Conference on Machine Learning, pages 2160-2169. PMLR, 2019.
[20] Yixiu Mao, Hongchang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported trust region optimization for offline reinforcement leaming. In International Conference on Machine Learning, pages 23829-222 23851. PMLR, 2023.
[21] Martin L Puterman, Markoy decision processes: discrete stochastic dynamic programming. John Wiley Sons, 2014.
[22] Elad Hazan et al. Introduction to online convex optimization. Foundations and Trends@ in Optimization, 2(3-4):157 -25, 2016.
[23] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. International conference on machine learning, pages 2052-2062. PMLR, 2019.
[24] Scott Fujimoto, Hetke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596. PMLR, 2018.
[25] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-11999 2020.
[26] Haoran Xu, Li Jiang, Jianxiong Li, and Xiamyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. In Advances in Neural Information Processing Systems, 2022.
[27] Gaon An, Seungyong Moon, Jang- Hyyn Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified geensemble. Advances in neural information processing systems, 34:7436-744 7447, 2021.

---

[28] Zhendong Wang. Jonathan J Hunt, and Mingyuan Zhou, Diffusion policies an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208. 061933 2022.
[29] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006. 093399 2020.
[30] Mitsuhiko Nakamoto, Yuexiang Zhai, Anikait Singh, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline r1 pre-training for efficient online fine-tuning. In Workshop Reincar- nating Reinforcement Learning at ICLR 2023, 2023.
[31] Haichao Zhang, Wei Xu, and Haonan Yu. Policy expansion for bridging oflline-to-online reinforcement learning. In The Eleventh International Conference on Learning Representations, 2022.
[32] Michael Bain and Claude Sammut. A framework for behavioural cloning. In Machine Intelligence 15, pages 109, 1995.
[33] Seyed Kamyar Seyed Ghasemipour, Dale Schuurmans, and Shixiang Shane Gu. Emaq: q-lcarning operator for simple yet effective offtine and online In International Conference on Machine Learning, pages 3682-3691, PMLR, 2021.
Expected-man
[34] Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278-31291, 2022.
[35] Jianxiong Li, Xianyuan Zhan, Haoran Xu, Xiangyu Zhu, Jingjing Liu, and Ya-Qin Zhang. Distance- sensitive offiin reinforcement learning. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
[36] Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768 7778, 2020.
[37] David Brandfonbrener, William F Whitney, Rajesh Ranganath, and Joan Bruna. Quantile filtered imitation learning. arXiv preprint 112.0:2112.00950. 2021.
[38] Xue Bin Peng. Aviral Kumar, Grace Zhang, and Sergey Levine. weeintaee--eigghed regression: Simple and scalable pol-p reinforcement learning. arXiv preprint arXiv:1910.00177. 2019.
[39] Xinyue Chen, Zijian Zhou, Zheng Wang, Che Wang. Yanqiu Wu, and Keith Ross. Bail: Best-action imitation learning for batch deep reinforcement learning. Advances in Neural Information Processing Systems, 33:18353- 18363, 2020.
[40] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offline RL via supervised learning? In International Conference on Learning Representations, 2022.
[41] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning modeling problem. Advances in neural information processing systems, 34: 1273-12288,, 2021.
big sequence
[42] Rui Yang. Yiming Lu, Wenzhe Li, Hao Sun, Meng Fang, Yali Du, Xiu Li, Lei Han, and Chongjic Zhang. Rethinking goal-conditioned supervised learning and its connection to offline RL. International Conference on Learning Representations, 2022.
[43] Yixiu Mao, Hongehang Zhang, Chen Chen, Yi Xu, and Xiangyang Ji. Supported value regularization for offline reinforcement learning. Advances in Neural Information Processing Systems. 36, 2024.
[44] Jing Zhang, Chi Zhang, Wenjia Wang, and Bingyi Jing. Constrained policy optimization with explicit behavior density for offline reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[45] Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Advances in neural information processing systems, 30, 2017.
[46] Alan Chan, Hugo Silva, Sungsu Lim, Tadashi Kozuno, Rupam Mahmood, and Martha White. Greedifi- cation operators for policy optimization: Investigating forward and reverse kl divergences. The Journal of Machine Learning Research, 23(1):11474-1155 2022.
[47] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as "Lounchpad". 2022.
12

---

[48] Alex Beeson and Giovanni Montana. Improving td3-bc: Relaxed policy constraint for offline learning and stable online fine-tuning. arXiv preprint arXiv:2211.1102. 2022.
[49] Liyuan Mao, Haoran Xu, Weinan Zhang, and Xianyuan Zhan. Odice: Revealing the mystery of distribution correction estimation via orthogonal orthogonal-gradient update. arXiv preprint arXiv:2402.0060 00348, 2024.
[50] Harshit Sikchi, Qinqing Zheng, Amy Zhang, and Scott Niekum. Dual rl: Unification and new methods for reinforcement and imitation learning. arXiv preprint orXi:23302.08560, 2023.
13

---

A Related Works
In the offline RL, policy constraint is the most straightforward and intuitive method for avoiding the issue of distributional shift. The core idea is to ensure that the learning policy cannot be too far away from the behavioral policy. Based on BC [22] that actions are selected from the dataset and cloned, some attempts are made to ensure
that the distribution bounded by carcfully parameterizing the learning policy 25 33 In addition, several works explicitly apply divergence penalty terms to help adjust the learning policy. e.g., KL divergence (29 and MMD distance 77. A general framework is proposed to evaluate these divergence-dependden
methods [
More recently, other problem-specie
regularization terms have been found to be more effective. These can be accomplished by adding simple behavior cloning term that produces exceptional performance M. regularizing behavior density directly to put more precise constraints
(34 or generalizing OOD samples within a convex
hull of training data using the distance function [35]
The policy constraint of these works can generally be assumed to be implemented based on the behavioral cloned policy. When data quality is high, BC is capable of achieving competitive performance thus the ahove methods can achieve satisfactory performance. However, a naive hehavioral cloning approach is unworkable when the datasct collected by sub-optimal or multi-modal policy. To improve the quality of the behavioral policy used for constraint, several alternatives have been proposed. Weighted BC is a way to clone only the available best behavioral actions by down-weightig low-quality data or simply discarding them. Many works evaluate the relative advantages of transitions and rely on them to filter the most promising ones in the
dataset B6 37 I0 12 pick the trajectory samples based on Monte Carlo returns 38 39 9 Conditional
BC introduces an additional condition as the input of BC. 40 demonstrates the effectiveness of conditional
BC and shows that an appropriate conditional variable is crucial when offline RL takes a supervised learning approach. These approaches use return-to-go as a condition to make better sequential decisions [4] 41 or select a future goal state as condition and can extract the behavior of reaching the goal from sparse rewards [40 42 Furthermore, recent research (26) learns guide-policy and exccute-policy separately and uses the state generated by guide-poli -olicy to steer where execute- execute-policy goes. (2 proposed to employ the diffusion model as highly expressive policy class and use it to constrain the learning policy, which achieves the state-of of oate--of-hheart performance among the policy constraint offline RL methods. CPI shares the same idea of improving the policy used for constraints, but CPT does not need to employ complex policy models such as diffusion model, still achieving similar even better performance while greatly saving computational overhcad.
These methods adopt diffcrent methods to address the extrapolation error in offfine reinforcement learning. For instance, SPOT (4) directly models the support set of the behavior policy, CPED 44 utilizes a GAN model for explicit density estimation, IQL 112 avoids querying values of unseen actions by implicitly approximating the policy uses the bias of importance sampling to calculate the penalty term. While CPI's core motivation is to enhance the robustness of behavior regularization methods by iteratively refining the reference policy used for regularization. This approach aims to progressively improve the policy's performance without significantly altering existing implementations. In addition, CPI is grounded in the conservative policy iteration, which guarantees progressive policy improvement and eventual convergence to the in-sample optimal policy within the support of the reference policy. SPOT, CPED, IQI., and SVR also have their theoretical foundations. For example, SPOT R is based on the theoretical formalization of density-hasee support constraints, CPED [ provides theoretical results for the GAN estimator and performance guarantees, IQL [12] relies on the expectile regression of function approximators, and SVR (43 demonstrates the contractive property of its policy evaluation operator.
In addition, there are several previous works aiming at achicving in-sample optimality
12
improvement step, and SVR
4
A previous work named STR 20 shares the similar idea with CPL. The key difference between the two algorithms is CPI uses reverse KL to optimize policy while STR uses forward KL to optimize policy and the resulting algorithms of CPI and STR are quite distinct. Note that we discuss the implementations between the two KL approaches in Appendix C STR could be seen as adding an importance sampling weight to the Equation
40 in our paper's 's appendix.
B Proofs
We first introduce some technical lemmas that will be used in the proof.
We consider a k-armed one-step decision making problem. Let be a k-dimensional simplex and q (q(1), (k)) E R be the reward vector. Maximum entropy optimization considers
max π q + ++)(()).
(11)
The next result characterizes the solution of this problem (Lemma 4 of 45 Lemma 1. For T >0 0, let
e/T
(g)g) = T logg
adult fL(9)= Σ, eala)/7
(12)
14

---

Then there is
(q)
max π + - f. (q) q +))
(13)
EEA
The second result gives the error decomposition of applying the Politex algorithm to compute optimal policy.
This result is adopted from [17]
Lemma 2. Let no be the uniform policy and consider running the following iterative algorithm MDP for t
9 (als)
Te+1 (als) (T(aaa) exp
(14)
Then
log IA
e' (s) (()) S (1 71-
(15)
t
Prouf. We use vector and matrix operations to simply the proof. In particular, we use ot E R and 4"' RlSIXIAl, Let P [sikikaill be the transition matrix, and lst R states when applying the policy æ.
(|x|s| be the transition matrix between
We first apply the following error dccomposition
A-
R-1
Th-
(16)
1=0
1=0
For the first part,
(17)
1=0
A-1 Σ [I 7p= y-1 0: x7
(18)
7p" y- ''(MM' - 0=
(19)
log |A A:
$ (1 7)
(0
where Equation 18 follows by the value difference lemma, Equation 20 follows by applying the regret bound of mirror descent algorithm for policy optimization. For the second part.
(21)
h (1 1 6"
(22)
k (1 p 1 (M
1
- -
(23)
0
(24)
where for the last step we use that for any s E S, k-- (TT 14- S O. This follows by
k-1
2
Fk-10
Th 1/-- + 7 -H(( 7 (
(25)
2-
Th 29: + 19-1- + r TH( TH(TT-11)
(26)
1=0
(27)
न(qi + +HH(To) -H((Te-1)
(28)
k-1
Σ
(29)
k=0
15

---

where Equation 26 follows by applying Lemma and the definition of ma-1, Equation 29 definition of no- Combine the above together finishes the proof.
follows by the
Proof of Theorem 7 First recall the in-sample optimality equation
Pip (s, r(s, a) ((Bls=P()ls.a)
max 9ip 9 (s'
(30)
which could be viewed as the optimal value of MDP MD covered by the behavior policy where Mr only contains transitions starting with (s, a) E 8 A such that Ta(aas) O. Then the result be proved by two steps. First, note that the CPI algorithm will never consider actions such that (a(a))) This directly
implied by Lemma [ Second, we apply Lemma to show the error bound of using CPI on Mp. This finishes the proof.
Proposition 1. let Rt be the optimal policy of 5 For any E S. we have that v*a (s) 2 V*(s)) and #" (als) = 0 given H(als) O.
Proof of Proposition
prove the first part.
Ba [Q" a)]
> Ennx. [Q (s, a)] -Dk. (# (s)]l=(s)
(31)
2 Enca (Q"(s,)) T DDi.. (m())][*(s))
(32)
En [Q"
(33)
where the first inequality follows the non-negativity of KI. divergence, the second inequality follows is optimal policy of (34). Then the first statement can be proved by applying a standard recursive argument. The second statement is directly implied by Lemma !
max Ea-x | (s, a)] - T DKL ((s))(((s)))
(34)
C CPI with forward KL
We show how optimize
34 using the forward KL. Extension
is straightforward by picking m.
Recall
34
max En->x [Q (s, a)] - DKK. (s())](((s))
(35)
By Lemma
I
the optimal policy #* has a closed form solution.
Q"
(als) x (xals)exp exp
(36)
This implies that to optimize
can also consider the following
min DKL. ( ([T)) = # log
(37)
Easr [log x [a[s)]
(38)
x (ulss)
E
log x (als))
(39)
#(als)
exp((" (T)
E.--
log ((aa))
(40)
7(a)
The first step follows by removing terms not dependent on ((+) Σ, (als)) exp(("(xss...e is normaliza- tion term. In practice, it is often approximated by state value function [ 29
We now discuss how to connect the forward KL objective described above with the original optimization problem (34) Indeed, it can be verified that (4 corresponds to reverse KL policy
Connection to 34
16

---

optimization.
arg min DRL (*|l="" arg min log log
(41)
x
arg min nt log log exp (Q /T)
(42)
arg min log log न =Q² /T + log Z
(43)
*
mig (xL(T]]F) - =Q- /r
(44)
=agg - (
(45)
where we use vector notations for simplicity. Although forward and reverse KL has exactly the same solution in the tabular case, when function approximation is applied these two objectives can showcase different optimization properties. We refer 44 for more discussions on these two objectives.
D Detailed Experimental Settings
In this section we provide the complete details of the experiments in our paper.
D.1 The effect of different regularizations
We analyze the impact on the algorithm when different poli- cies used regularization terms. The most straightfor- ward way to obtain policies with different performances is
Table 2: 5% BC Hyperparameters
the baseline method XC mentioned in DT [4]. We set to in order to make the difference between policies more significant and let the 5%BC policy be the policy used for constraint. detail, we selectively choose different 5% data for behavioral cloning so that we can get various policies with different performances. First, we sort the trajectories in the dataset by their return (accumulated rewards) and select three different levels of data: top (highest return), middle bottom (lowest return). Each level data is sampled at total data volume. Then we can train 5%BC using the MLP network via BC and get three 5%BC BC policies with different performances. We can then use each 5%BC policy the regularization term of TD3 to implement the TD3+5%BC method. Besides, also normalize the states and set the regularization parameter a to 2.5. The only difference between the TD3+5%BC and TD3+BC the action obtained in the regularization term. In TD3+BC, the action used for constraint is directly obtained from the dataset according to the corresponding state in the dataset, while in TD3+5%BC, the action for constraint is sampled using 5%BC. We set the training steps for 5% BC to Se5 and the training steps for TD3+5%BC to IM. Table concclues the hyperparameters of 5% BC. The hyperparameters of TD3+5%BC are the same those of TD3+BC (1I].
Hyperparameter
Value
Hidden layers
Hidden dim
256
Activation function
ReLU
Mini-batch size
256
Optimizer
Adam
Dropout
0.1
Leaminng rate
3e-4
D.2 Baselines
We conduct experiments on the benchmark of D4RL and use Gym-MuloCC datasets of version v2, Antmaze datasets of version vO. and Adroit datasets of version vl. We compare CPI with BC. DT [14]. TD3+BC ([]]
CQL (25) IQL 122, POR 26). EDAC 27.. Diffusion-Q2 QL. (2 and InAC D3 In Gym-Mujoco tasks,
experimental results are preferentially selected from EDAC, Diffusion-QI. papers, or their original papers.
corresponding results are unavailable from these sources, we rerun the code provided by the authors. Specifically, we run it on the expert dataset for P For DT and IOL we follow the hyperparameters given by the authors random and expert datasets. For Diffusiion Qu set the hyperparameters for the random dataset be the same as on medium-replay dataset and the hyperparameters for the expert dataset be the same as on the medium-cxper dataset according to the similarity between these datasets. For InAC] we run on the random datasct. In Antmaze tasks, our experimental results are taken from the Diffusion -LL paper, except for EDAC and POR. The results of EDAC are obtained by running the authors' provided code and setting the
for-Offline-RR
17

---

Table 3: CPI Hyperparameters
Hyperparameter
Value
Actor hidden layers
3
Actor hidden dim
256
Actor activation function
ReLU
Architecture
Critic hidden layers
3
Critic hidden dim
256
Critic activation function
ReLU
Optimizer
Adam
Critic learning rate
3e-4 for MuJoCo and Adroit
1e-3 for Antmaze
Actor learning rate
3e-4
Mini-hatch size
256
Discount factor
0.99 for MuJoCo and Adroit
0.995 for Antmaze
Learning
Target update rate
Se-3
Policy noise
Policy noise clipping
(-0.5, 0.5)
[0.05, 0.2, 2) for MuloCo
{0,03, 0.05, 0.1, J for Antmaze
1000 200 for Adroit
A
(0.5, 0.7)
ensemble number 10 and 17 = 1.0. Even when transformed the rewards according to 25 the performance EDAC on Antmaze still not perform well, which matches report in offline reinforcement learning library CORL [47]. As the results in the POR paper are under Antmaze v2. we rerun them under Antmaze v0. In the Adroit task, we rerun the experiment for TD3+BC, DT (with return-to-go set to 3200), POR (with the same parameters as the antmaze tasks), and InAC(with tau set to 0.7).
D.3 Conservative Policy iteration
To implement our idea, we made slight modifications to TD3+BQ to obtain CPI, For CPI, we select historical snapshot policy as the reference policy. Specifically, the policy snapshot x ===, which is two gradient steps before the current step, is chosen the reference policy for the current learning policy n² CPI is trained similarly TD3. For CPI-RE, the complete network contains identical policy networks with different initial parameters, so that two policies with distinct performances can be obtained. The two policy networks CPI-RE are updated via cross-update to fully utilize the information from both value networks. During training, the value network evaluates actions induced by the two policy networks, and only the higher value action is used to pull up the performance of the learning policy. During evaluation, the two policy networks also used to select high-value actions to interact with environment. CPI only has one more actor compared TD3+BC and therefore requires less computational overhead than other otatc-of-the-art offline RL algorithms with complex algorithmic architectures, such as EDAC (an ensemble of Q functions) and Diffusion-O1 QL (Diffusion model).
According to TD3+BC, we normalize the state of the MuloCo tasks and use the original rewards in the dataset. For the Antmaze datasets, ignore the state normalization techniques and transform the rewards in the dataset according to (25). For Adroit datasets,
also do not state normalization and standardize the rewards
according to Diffusion-QU QL (28). To get the reported results, we average the returns of 10 trajectories with five random seeds evaluated every 5e3 steps for MuJoCo and Adroit, 100 trajectories with five random seeds evaluated every 5c4 steps for Antmaze. When training on Antmaze, it's observed that the randomness can heavily influence the algorithm performance reported of offline learning and selects model using finite online samples, we train several models using different random seeds randomly choose five models of which the Q-value is not divergent and report the average returns of 100 trajectories with these five models evaluated every 5e4 steps. In addition, we evaluate the runtime and the memory consumption of different algorithms train epoch (1000 gradient update steps). All experiments GeForce GTX 2080T1 GPU.
28 Compared with the
28 that violates the setting
The most critical hyperparameters in CPI are the weight coefficient À and the regularization parameter On all datasets, the choice of À or À = 0.7 the most appropriate so that the two actions (from behavioral policy 3 and the reference policy #) can be well-weighed to participate in the learning process. As mentioned in the main text, the choice of r depends heavily on the characteristics of the datasct. For high-quality dataset,
18

---

Table Regularization parameter and weighting factor À of CPI for all datasets. On most datasets, CPI and CPI-RE have the same optimal parameters. On some datasets, the values in parentheses indicate the parameters CPI, and the values outside the parentheses indicate the parameters of CPI-RE.
Dataset
regularization parameter
weighting coefficient
srrchectah-random-s
0.05
0.7
malfcheetah-mediimm-
0.05
0.7
malfcheetah-mediueee repllyy---
0.05
halfcheetah- hattteetta--mmiuu----ee
-eedium-expert-v22
0.5
halfchecttah-expert-22
0.5
hopper-randoom-22
0.05
0.5
hopper-medium-v2
0.2
0.5
popper-mdduum-cplly-yoo
0.05
0.5
hopper-meediuu--eppettt-
0.5(0.7)
hopper-expert-v2
0.5(0.7)
walker2dd-random-v22
0.5
walker22d-medium--2
0.7
malker2d-medduu--eelaay
0.2
0.7(0.5)
walker2admmcdium-erpprttz
0.7
walker2d-expert-v2
anlmazz-umaaz--vd
0.7
autmaze-umazz-diverse
0.1
0.5
aatmaaze-mmedium-plaayy
0.5
antmaze-mediiun--iveeee----
0.1(0.05)
0.5
latmaaze-arge-play-Wo
0.03
0.5(0.7)
antmaze-large-divenee-v
0.05
0.5
pen-human-vl
100(200)
0.5(0.7)
pen-cloned-vl
100
0.7
should be larger to learn in close imitation way, and for high-diversity dataset, the should be chosen be smaller to make whole learning process more similar to RL. We find that training more stable and better performance can be achieved on Antmaze when the critic learning rate is set to le-3. Also, since Antmaze is sparse reward domain, we also set the discount factor to 0.995. Table[4]giees our selections of hyperparameters and À on different datasets. Other settings given in Table 3 Codes provided in this link httttttt////gthubb htt.://gg//gggal....cao/aaannnng con///amnnnyyyy/PP
E
More experimental results
F.1
Effect of different numbers of reference policies
We also present the effect of using different numbers actors in CPI-RE. We can conclude from Figure that increasing the actor number from to 2 (i.c., introducing the reference policy) can significantly improve the performance of the learning policy. As further introducing reference policy derived from more actors does not bring significant benefits and entails significant resource consumption. Thus, set the actor number method to two.
balfchoctah-mediinee
balfchoctal-mddiu---op-----2
60
60
Nommskzzd Score
Nommalzz Scorre
40
20
0.6
Time Stepe(1o6)
Time Stepe(1o6)
Figure 8: Effect of different actor number
19

---

E.2 Results of Online Fintuning
We finetune CPI during online training after training on nine offline datasets including halfcheetah-meedium-- halficheeta--mddiu--eeeeee
malfchhetal-mediu--rlllaaa
hopper-medium--2 peopper-medium--epllaa---e
hopper medium- medium-experrt expert-v2, v2, walker2d-medium- v2, Detailed results provided in Figure 9 show that CPI with online finetuning can outperform the state-of sate-of-the-art the-art
and -eedim--eppeer---y expert-v2.
offline-to-ouie to-online algorithms Cal-QL 30 and PEX [3]
halfchectab-meclumee
haltccbetaal--eddin-----p
Normaling Scong
Normalized Sgore
Morualizgi Sonta
100
50
10
50
100
200
250
50
100
1.50
200
230
50
100
150
250
Time Stepss 1c3)
Time StepeçTe3)
Time Steps(Te3)
hupper seppper--ediiu----
hopper perpodiiu---pppaaa
hogper-mediunes medium expert-
100
Normiibed Score
Normalized Score
100
80
50
20
50
100
Time Steped 1e3)
Time Stepe(Tee)
Time Sieps(lee)
walker2d-meedim--eplla-
wullker2--mediun-xxppo
100
Jommlted Scon
Normaized Score
Normulzzd Soore
100
80
50
2220
100
200
250
50
100
1.50
200
230
50
100
150
250
Time Stepst Ic3)
Time Stepe(Te3)
Time Steps(Te3)
TD3-BC
101.
PEX
Cal-QI.
CPI
CPLRE
Figure 9: Performance of CPI offline training after further online fine-tuning
E.3 CPI with minimal hyper-parametor tuning
We provide results of CPI with little possible hyper- parameter tuning the same domain. For MuJoCo datasets, we set r 0.05 1.0, to 0.7. For Antmaze,
0.1 and to 0.5. For Antmaze, 200.0 and À to 0.7. The detailed results (average across five seeds) and setting are provided the following table The results show that compared with IQL,, CQL. and TD3BC, the overall performance of CPI with as little possible hyper-parameter tuning is still significantlly better.
to
E.4 Effects of various techniques used in CPI
Here we ablate the techniques including state normalization, reward transformation used in CPL.
In Antmaze, no state norm is used for CPI and TD3BC. shows introducing reward transformation indeed help TD3BC on certain antmaze datasets, but the overall performance is not improved and still falls behind CPL
In Mujoco, no reward transformation used for CPI and TD3BC. Table/7aablates the effectiveness of state normalization. With and without state normalization, CPI can still outperform TD3BC, demonstrating the effectiveness of the iteratively policy refinement.
E.5 In-depth comparison of TD3+BC and CPI
According to the experimental part of the ablation study and the analysis in the (34]. a is an important parameter for controlling the constraint strength. TD3+BC is set to constant value of 2.5 for each dataset, whereas
20

---

Table The performance of CPI and CPI-RE with as little as possible hypper-parameter (hp)
tuning.
Dataset
A
TD3+BC
CQI.
IQI.
CPI with minimal hp tuning
CPI
halfchcetah-ramee
0.05
0.7
11.0
31.3
13.7
29.7#:1.1
19.7:t111
halfchectah- meddim
0.05
0.7
46.9
47.4
64.4+1.3
64.4+1..3
halfcheetah- medfium-repay
0.05
0.7
45.3
44.2
54.6+1.3
54.6+
halfcheetah-
wwweheeeaaammmmu--eeeeeeee medium-exxerr
1.0
0.7
90.7
95.0
86.7
89.7.11.5
94.7.11.1
halfehectah-expere
1.0
0.7
96.7
97.3
94.9
98.1+00.6
96.5+0.2
hopper-random
0.05
0.7
8.5
5.3
8.4
29.5+3.8
29.5+3.77
hopper- medium
1.0
0.7
59.3
61.9
66.3
61411.9
98.5
hopper-mediung medium-replla mereplay
0.7
86.3
94.7
99.316.88
hopper-meding expert
0.7
96.9
106.4+4.3
106.4+-433
hupper-expert
1.0
0.7
107.8
106.5
108.8
112.2:1055
112.2:10.5
waller2ddrrnddon
1.0
0.7
1.6
5.4
5,9
3,912.2
5.911.7
1.0
0.7
83.7
79.5
78.3
85.810.88
85.810.8
billker2d-meiium--eee
0.7
81.8
76.8
73.9
89.1++3.0
91.8+2999
walker2d-
0.7
110.1
109.1
110.910.4
walker2d-ewe 2d-expert
0.7
110.2
109.3
109.7
110.6210.1
110.6110.1
mujoco-total
1013.2
1052.8
1034.0
1145.6
1193.2
antmaze-umaze
0.1
0.5
78.6
74.0
87.5
96.6+3.2
98.8+1.11
antmaze-umaze-da
0.5
84.0
62.2
88.6+5..7
8..6+5.7
0.1
0.5
3.0
61.1
71.2
82.415.8
82,415.8
antmaze- untmaze-mediume divveee diverse
0.1
0.5
10.6
53.7
70.0
78.818.9
80.418.9
antmaze- -arge-------ggg large-play
0.1
0.5
0.0
15.8
39.6
3.3+-577
20.6 16.3
antmaze- cermazz-large-dinn large-diverse
0.1
0.5
0.2
14.9
47.5
23.0.44.3
45.2.16.9
antmaaze-total
163.8
303.6
378.0
372.7
416.0
pen-human pen-cloned
200.0
0.7
-1.9
35.2
71.5
80.1=16,9
80.1=16.9
200.0
0.7
9.6
27.2
37.3
69.7+17.0
31.8++352
adroit-total
7.7
62.4
108.8
149.8
151.9
Table 6: Ablation of reward transformation.
datasets
TD3BC with reward transformation
TD3BC
CPI
antmaze-umaze
87.8+5.2
78.6+33.3
98.8+11.1
untmaze-umaze-liverr
69.8 11.1
71.41120.7
88.6115.7
untmaze-medium-
0.0=0..0
10.6110.1
82.415.8
untmaze-medium-toe
0,6=1.9
3.024.88
80.418,9
lntmaze-large-play
0.0=0.0
0.2:00.4
26.6.116.3
antmaze-large-divvrse
0.2.0.4
0.0.0.00
45.2-1..9
Total
158.4
163.8
416.0
Table 7: Ablation of state normalization.
datascts
CPI w/o
CPI
TD3BC w/o
TD3BC
halfcheetah-men medium
63.5+3.6
64.4++1.3
48.1+0.4
48.3++0.3
44.4=4.5
14.6+1..
44.020.5
44.6:10.5
halfchectah-meddee medium-cxxprrt
94.0=1.7
94.7+1.1
89,4+5.1
90.7+44.3
halfcheetah-expee expeett
96.4-0.7
96.5++0.2
96.1+1..0
96.7+111
hopper-mediug
94.2-9.77
98.5++3.00
59.5+
59.3:14.2
100.743.2
101.741.6
56.7423.6
60.9+18.8
hopper-medium-expee
104.7+6.7
106.4+-..3
98,4+4499
98.0++9.4
hopper-expert
112.1++0.3
112.2+0.5
110.7+1.7
107.8+7.0
malker2d-mediime
85.3.0,55
85.8.10.8
82,3133
83.7.12.1
89.4=1.5
91.8+229
83.0+332
81.8+5555
malker24-medium-exppe
111.9+0.3
110.9+0A
110.4+0.5
110.1+0.5
walker2dlexxerrt
110.510.1
110.6.0.11
110.2.1..3
110.2.0.3
Total
1117.1
1128.1
988.8
992.1
21

---

CPI chooses the appropriate from a set of alternatives. We note that the hyperparameter that plays role in regulating and regularization in CPI is т. which essentially be understood as the reciprocal of in TD3+BC. Therefore, for convenience comparison, we rationalize the reciprocal T as the parameter a. this section, we set the a of TD3+BC to be consistent with that of CPI in order to show that the performance improvement of CPI mainly comes from amalgamating the benefits of both Sehavior-regularizee and in in-sample algorithms. Further, swept best in the ranges 0.0001, 0.05, 0.25, 2.5, 25, 36, 50, 100, which improves TD3+BC by large margin, show the superiority of CPI. The selection parameters shown Table
also compare CPI with TD3+BC with dynamically changed [48] and TD3+BC with
results TD3+BC (vanilla), TD3+BC (same a CPI), TD3+BC (swept best o), TD3+BC with dynamically changed a and CPI are shown in Table B Comparing the variants of TD3+BC with different a choices. it can be found that changing can indeed improve the performance of TD3+BC. However, compared with TD3+BC (same and TD3+BC (swept best a). the performance of CPI significantly better, which proves the effectiveness of the mechanism for iterative refinement of policy for hehavior regularization in CPI.
Table 8: Comparison with TD3+BC and its variants.
TD3+BC (vanilla)
TD3+BC with CPI)
TD3+BC (swept best
TD3+BC
Datasct
CPI
CPI-RE
(dynamic
halfcheetah-medium
48.3
58.8
58.8
55.3
64.4
65.9
hopper-medium
59.3
69.0
69.0
100.1
98.5
97.9
maker2d-mediune
83.7
79.8
83.7
89.1
85,8
86.3
halfchcetah- medium-rrplayy
44,6
51.2
51.2
48.7
54.6
55.9
hopper-mediumen med--mm-eedllllaa
60.9
88.5
88.5
100.5
101.7
103.2
Mikkerrd-mddium medium-replay
83.0
83.0
87.9
91.8
halfchcetah- medium-expert
90.7
92.3
92.3
91.9
94.7
95.6
hopper-mediue wwape--ddddduippeeeeeee
98.0
76.1
98.0
103.9
106.4
110.1
waker2d medium-creee exppe
110.1
109.4
110.1
112.7
110.9
111.2
halfcheetah-cxpect
96.7
94.5
96.7
97.5
96.5
97.4
hopper-cxpert
107.8
110.8
110.8
112.4
112.2
112.3
walker2d-cxper Expexe
110.2
109.3
110.2
113.0
110.6
111.2
Above Total
992.1
1022.7
1052.3
1113.0
1128.1
1140.8
halfcheetah-randee
11.0
23.2
23.2
29.7
30.7
hopper-random waker2d-random
28.8
28.8
29.5
30.4
1.6
4.4
4.4
5.9
5.5
Gym Total
1013.2
1079.1
1108.7
1193.3
1207.4
millmaze-umazz
78.6
93.8
93.8
98.8
99.2
untmaze-umazz-ee
73.2
73.2
92.6
antmaze-mest
3,0
13.0
35.7
82.4
84.8
matmaaz-medliunddiver
10.6
8.0
17.5
80.4
80.6
marmaaze-largeee
0.0
0.0
0.0
20.6
33.6
Heimaze-large-diveree
0.2
0.0
0.2
45.2
48.0
Antmaze Total
163.8
188.0
220.4
416.0
438.8
E.6 Applying idea of CPI to SAC-BC
We implement SAC-BC similar to TD3-BC and integrate CPI into SAC-BC C to whether the idea of CPI could improve SAC-BC, which adding behavior cloning SAC algorithm. As show in Table [ these six datasets, CPI could bring benefits to SAC-BC.
Table 9: Results of SAC-BC and SAC-BC with Policy Iteration.
Datascts
SAC-BC
SAC-BC with Policy Iteration
setccheetth--meeiinn
42.1.10.8
$1.41008 51.01110
halfcheetah--me wew.rtata-mmmmmeeeeeee replay
38.0 38.0.22.88
hopper-medium
35.9+:1.4
40.548.1
cupper--eeduum-rella
32.2.17.9
36.4.14.1
walker2d-medium
78.8144.2
59.6113.0
wallker2d-medium--rrpp
75.8+13.2
84.343.8
Total
302.8
323.2
E.7 Comparisons with more baselines
We additionally provide comparisons of methods and recently proposed baselines here.
22

---

Table 10: Comparison with more baselines
datascts
1-DVL ODICE(49
34
CPED
CPI
CPI-RE
halfcheetth--reediuu
47.4
47.5
58.4
61.8
64.4
65.9
hopper-mest
64.1
86.0
100.1
98.5
97.9
walker2d 2d-mmmuu
84.9
81.5
86.4
90.2
85.8
86.3
hal [cheetah caterraaah--uum---- medhumm-rppay
44.0
44.7
52.2
55.8
54.6
55.9
hopper-mediume replay
99.9
98.0
100.2
98.1
101.7
103.2
walk walker2d medium-repllay
83.6
68.7
91.6
91.9
9188
93.8
sallchhetaahaee -alfcheeethh-mmediure medium-exxprr
93.2
91.2
86.9
85.4
94.7
95.6
hopper- wewterr-ddiuu-eppeee
110.8
93.3
95.3
106.4
110.1
walker2d medium--de-eeee expert
110.8
109.6
112.0
113.0
110.9
111.2
Gym-M MuloCo Total
773.0
791.7
808.8
819.9
antmaze-umaze
94.1
87.7
93.5
96.8
98.8
99.2
mntmaze-urraze-divveee
55.6
88.6
antmaze-mediuum play
86.0
71.0
74.7
85.1
82.4
84.8
meduuzeeeeeeium--eee diverse
82.7
60.2
79.1
72.1
80.4
80.6
antmaze-large-laa
55.9
41.7
35.3
34.9
20.6
33.6
antmize-large-iiver
54.0
39.3
36.3
32.3
45.2
48.0
Antmaze Total
452.2
348.3
359.6
376.8
416.0
438.8
Total
1212.9
1046.9
1132.6
1168.5
1224.8
1258.7
E.8
Learning curves of CPI
The learning of CPI all tasks are shown in Figure
pen-human-vl
umazze-v +0
medium hwwpae----gpp-p------y
Narmutaal Booes
100
50
pen-clooe pen-cloned-vl
untmaze umazz diverso-vO
antmaze medium diverse-vO
autmaze-lare large---aee-eaee diverse-vO
Nonulied Sene
100
hulfcheetah medium replay v2
halfcheetah medium v2
halfcheetah medium expert v2
hulfcheetah expert
50
hopper-msduun-ccllay-222
hopper-medium--22
hopprr-mciimm-xxpen-2
hopper-cxpert-v2
Nematied Sove Sovee
100
walker2d medium medium-replay-se replay-v2
walker2d-medium-
walker22 medium expent v2
walker2d valker2d-expert-v v2
Nomaliel Spore
100
Steps(lef)
Stepsi
Stepocled)
Steps(1e6)
Figure 10: Learning eurves of CPI and CPI-RE for Mujoco, Antmaze and Pen, evaluated every Se3 steps.
23

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We proposed a novel offline rl algorithm, the core concept behind this approach hinges
the iterative refinement of the reference policy utilized for behavior regularization. We try best accurately describe our contributions in the Abstract and Introduction (Section T We also reiterated our contribution the last paragraph Introduction section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals motivation long it is clear that these goals not attained by the paper.
2. Limitations
Question: Does paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings. model well- sell-specification asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice
what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. general, empirical results often depend on implicit assumptions, which should he articulated.
The authors should reflect on the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with datasct size.
applicable, the authors should discuss possible limitations of their approach address problems privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't it acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification:
See appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.

---

All the theorems, formulas, and proofs in the paper should be numbered and cross cross-reference..
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can cither appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide code link and describe the hyperparameters setting in appendix
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contrihution or the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., the case of a large language model), releasing of model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (c.g.. with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reprodueibility. In the case of closed source models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide code link and description of datasets the experimental section and appendix
D
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpp://nips. httppa://nips.cc//ubbbll cc/public/ for more details.
guides/Cood httss///ooouuuaasiisooooooo
While we encourage the release of code and data, we understand that this might not be possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
25

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nips.cc/puut
Buzdee//DodeSubmiissioooool
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We describe experimental details in appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: Results in our paper are depicted with 95% confidence intervals, represented by shaded areas in ligures and expressed as standard deviations in tables.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanicd by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap. etc.)
The assumptions made should be given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
See appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental nuns well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does research conducted the paper conform, every respect, with NeurIPS Code of Ethics htttp:://ner http://pccc/cbiaaaaaieeeeeeeeee.
Answer: [NA]
Justification: None
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss hoth potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: None
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
foundational research not tied to particular
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms could arise when technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: None
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions access the model implementing safety filters.
27

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Answer: [NA]
Justification: None
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (c.g., ., CC-BY 4.0) should be included for cach asset. scraped from particular source (e.g.. website), the copyright terms of service of that source should provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, paperswi thcode. www.coom//aaac /attttttstts has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-per re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: None
Guidelines:
The answer NA means that the paper does not release new assets.
Rescarchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an amonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: None
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
28

---

Answer: [NA]
Justification:
None
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, approval (or equivalent) may be required for any human subjects research. If you ohtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
29


Reviews:
[{"review_id": "1DJPy299Eh", "reviewer": "Reviewer_Y6m9", "summary": "Policy constraint is a standard approach to offline RL. Research in this area often involves using different types of divergence to regulate the distance between the current (learned) policy and the behavior (reference) policy. This paper proposes a new perspective on policy constraint offline RL: why not update the reference policy?\n\nBy doing so, policy constraint offline RL methods can handle heterogeneous datasets, where trajectories are collected by different levels of behavior policies. This is an interesting, promising, and novel idea. The example in Figure 1 clearly verifies this motivation.\n\nAs for the empirical evaluation part, many state-of-the-art baseline algorithms such as IQL, EDAC, and STR are included, covering policy constraint methods and value regularization methods. Empirical comparison on the D4RL benchmark suggests that iteratively refining the reference policy significantly improves performance, especially with the help of a non-algorithmic technique, Reference Ensembles (CPI-RE).", "questions": "I have a question regarding Proposition 1. I checked the proof in the appendix, where I find that the proof of $E_{a\\sim \\pi^*}[Q^\\pi(s,a)] \\geq E_{a\\sim \\pi}[Q^\\pi(s,a)]$ is correct. However, this is different from the conclusion that $V^{\\pi^*} \\geq V^{\\pi}$, because the notation $V^{\\pi^*}(s)$ normally refers to the expected returns by following $\\pi^*$, that is $E_{a\\sim \\pi^*}[Q^{\\pi^*}(s,a)]$.", "limitations": "please see questions.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The idea of iteratively updating the reference policy for policy constraint offline RL is well-motivated.\n2. The motivation, implementation, and convergence analysis (towards the unique fixed solution with the in-sample version of tabular MDP) are clear to me.\n3. Empirical evaluation includes comprehenseive baseline algorithms.", "weaknesses": "The writing can be improved, though it is not technical nor affects the contribution of this paper.\n    1. “behind” → “behinds”, Line 57\n    2. Proposition 1 has a restatement in the appendix, which has a different number (Proposition 2). I believe the LaTeX command`\\begin{restatable}{theorem}` would help. Besides, I think it would be better to replace \"optimal policy\" with \"solution\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## The writing can be improved\n\nWe sincerely appreciate your suggestions for improving the readability of our paper. We have modified our paper as your suggestion!\n\n## question regarding Proposition 1.\nYou’re right that $E_{a \\\\sim \\\\pi^*}\\\\left[Q^\\\\pi(s, a)\\\\right] \\\\geq E_{a \\\\sim \\\\pi}\\\\left[Q^\\\\pi(s, a)\\\\right]$ is different from the conclusion that $V^{\\\\pi^*} \\\\geq V^\\\\pi$. However, $V^{\\\\pi^*} \\\\geq V^\\\\pi$ could be proved in a quite simple way using the Policy Improvement Theorem and the corresponding proof.\n\n**Theorem:** Consider two policy $\\\\pi(a \\\\mid s), \\\\pi^{\\\\prime}(a \\\\mid s)$, and define $$ Q^\\\\pi\\\\left(s, \\\\pi^{\\\\prime}\\\\right)=\\\\mathbb{E}_{a \\\\sim \\\\pi^{\\\\prime}(a \\\\mid s)}\\\\left[Q^\\\\pi(s, a)\\\\right] . $$\n\nIf $\\\\forall s \\\\in S$, we have that $Q^\\\\pi\\\\left(s, \\\\pi^{\\\\prime}\\\\right) \\\\geq V^\\\\pi(s)$, then it holds that $V^{\\\\pi^{\\\\prime}}(s) \\\\geq V^\\\\pi(s), \\\\forall s \\\\in S$. This means that $\\\\pi^{\\\\prime}$ is atleast as good a policy as $\\\\pi$.\n\n**Proof**. Note that $Q^\\\\pi\\\\left(s, \\\\pi^{\\\\prime}\\\\right) \\\\geq V^\\\\pi(s)$. By expanding $Q^\\pi$, we can get that $\\\\forall s \\\\in S$, \n\n\\begin{aligned}\n& V^\\pi(s) \\leq Q^\\pi\\left(s, \\pi^{\\prime}\\right) \\\\\\\\\n& =\\mathbb{E}_{a \\sim \\pi^{\\prime}(a \\mid s), s^{\\prime} \\sim \\mathcal{T}^{\\prime}\\left(s^{\\prime} \\mid s, a\\right)}\\left[r(s, a)+\\gamma V^\\pi\\left(s^{\\prime}\\right)\\right] \\\\\\\\\n& \\\\leq \\\\mathbb{E}\\_{a \\\\sim \\\\pi^{\\\\prime}(a \\\\mid s), s^{\\\\prime} \\\\sim \\\\mathcal{T}^{\\\\prime}\\\\left(s^{\\\\prime} \\\\mid s, a\\\\right)}\\\\left[r(s, a)+\\\\gamma Q^\\\\pi\\\\left(s^{\\prime}, \\\\pi^{\\\\prime}\\\\right)\\\\right] \\\\\\\\\n& =\\\\mathbb{E}\\_{a, a^{\\\\prime} \\\\sim \\\\pi^{\\\\prime}}\\\\left[r(s, a)+\\\\gamma r\\\\left(s^{\\\\prime}, a^{\\\\prime}\\\\right)+\\\\gamma^2 V^\\\\pi\\\\left(s^{\\\\prime \\\\prime}\\\\right)\\\\right] \\\\\\\\\n& \\\\leq \\\\ldots \\\\\\\\\n& \\\\leq \\\\mathbb{E}\\_{a, a^{\\\\prime}, a^{\\\\prime \\\\prime} \\\\ldots \\\\sim \\\\pi^{\\\\prime}}\\\\left[r(s, a)+\\\\gamma r\\\\left(s^{\\\\prime}, a^{\\\\prime}\\\\right)+\\\\gamma^2 r\\\\left(s^{\\\\prime \\\\prime}, a^{\\\\prime \\\\prime}\\\\right)+\\\\ldots\\\\right] \\\\\\\\\n& =V^{\\\\pi^{\\\\prime}}(s) \\\\\\\\\n\\end{aligned}\n\n\nThis completes the proof that the new policy $\\\\pi'$ is at least as good as the original policy $\\\\pi$ in terms of the state-value function for all states $s$.  According to this Policy Improvement Theorem and the proof of Proposition 2 in our paper, we can directly obtain that $V^{\\pi^*}(s)=E_{a \\sim \\pi^*}[Q^{\\pi^*}(s, a)] \\geq E_{a \\sim \\pi^*}[Q^\\pi(s, a)] \\geq E_{a \\sim \\pi}[Q^\\pi(s, a)]= V^\\pi(s)$.\n\nWe sincerely hope this could address your concerns. If you feel that the manuscript meets the criteria for a higher rating, I would be immensely appreciative of any positive adjustments to the score. Your recognition of the merits of this research would encourage us a lot. I look forward to receiving your feedback and am open to any suggestions that may further enhance the quality of this manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8SkSDOnLIl", "reviewer": "Reviewer_Hcq2", "summary": "The paper introduces Conservative Policy Iteration (CPI), a new policy regularization algorithm for offline reinforcement learning. The core concept behind this approach is the iterative refinement of the reference policy for regularization. The algorithm guarantees policy improvement while avoiding out-of-sample actions and converges to the in-sample optimal policy. The paper also discusses practical implementations of CPI for continuous control tasks and evaluates its performance on the offline RL benchmarks.", "questions": "* Can you provide more detailed hyperparameter study results, including those on additional datasets?\n* Are there any specific types of tasks or environments where CPI may underperform typical behavior regularization method TD3+BC under the same regularization strength?", "limitations": "The authors have stated the limitations of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The idea of iteratively refining the reference policy provides a new approach to address the challenges of typical behavior regularization. This contribution improves the robustness and performance of behavior regularization methods.\n* The paper provides theoretical analysis to support CPI in the tabular setting and demonstrates its superior performance compared to previous methods in empirical evaluations. The experiments are well-designed and provide comprehensive results. The theoretical analysis is also presented in a clear and concise manner. \n* The presentation is good. The authors provide clear explanations of the algorithm, its implementation details, and the experimental setup.", "weaknesses": "* The authors have mentioned that one of the limitations of CPI is the need for the selection of two hyperparameters. It would be helpful to have a detailed evaluation and discussion on this. For example, ablation on additional offline datasets would offer a deeper insight.\n* The detailed experimental setup of Figure 1 is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## more detailed discussion and hyperparameter study results\n\nIn Section 5.3.4, we provide ablations of the effect of the two hyperparameters. We also summarize some empirical experiences in the section.\n\n- $\\lambda$ : When $\\lambda=0.1$, the early-stage performance excels, as the behavior policy assists in locating appropriate actions in the dataset. However, this results in suboptimal final convergence performance, attributable to the excessive behavior policy constraint on performance improvement. For larger values, such as 0.9 , the marginal weight of the behavior policy leads to performance increase during training. Unfortunately, the final performance might be poor. This is due to that the policy does not have sufficient behavior cloning guidance, leading to a potential distribution shift during the training process. Consequently, we predominantly select a $\\lambda$ value of 0.5 or 0.7 to strike a balance between the reference policy regularization and behavior regularization. In practice, for dataset of higher quality and lower diversity (e.g., expert dataset), we encourage authors to try small $\\lambda$. For datasets of lower quality and higher diversity (e.g., medium dataset), larger $\\lambda$ should be chosed.\n- $\\tau$ : The regularization parameter $\\tau$ plays a crucial role in determining the weightage of the joint regularization relative to the Q-value component. We find that (Figure 6 b) $\\tau$ assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).\n\nIn addition, we also provide results of CPI with as little as possible hyper-parameter tuning on the same domain in Table 5 in our appendix. The results show that compared with IQL, CQL and TD3BC, the overall performance of CPI with as little as possible hyper-parameter tuning is still significantly better.\n\n## ablation on additional offline datasets\n\nIn the PDF of the general response, we provide more hyperparamerts ablations. All the results are averaged across three seeds using the final obtained model after training for 1M gradient steps. The results further prove our empirical experiences of hyperparameter tuning provided above.\n\n## detailed experimental setup of Figure 1\n\nIn Figure 1 we empirically demonstrate the impact of regularization utilizing distinct policies on the 'hopper-medium-replay' and 'hopper-medium-expert' datasets in D4RL. We leverage Percentile Behavior Cloning (\\%BC) to generate policies of varied performance \\citep{chen2021decision}. Specifically, for behavioral cloning, we filter trajectories of the top 5\\%, median 5\\%, and bottom 5\\% returns. Following this, we modify TD3+BC to develop the TD3+5\\%BC algorithm by replacing the behavior regularization with the 5\\%BC policy, and subsequently train TD3+5\\%BC on the original dataset. These descriptions are also provided in the introduction section. We’ll try to make them clearer to readers. \n\n## tasks or environments where CPI may underperform typical behavior regularization method TD3+BC under the same regularization strength\n\nIn Table 8 in our appendix, we ablate different variants of TD3+BC and compare CPI with them. TD3+BC is set $\\alpha$ to a constant value of 2.5 for each dataset, whereas CPI chooses the appropriate $\\tau$ from a set of $\\tau$ alternatives. We note that the hyperparameter that plays a role in regulating Q and regularization in CPI is $\\tau$, which can essentially be understood as the reciprocal of $\\alpha$ in TD3+BC. Therefore, for the convenience of comparison, we rationalize the reciprocal of $\\tau$ as the parameter $\\alpha$. In this section, we set the $\\alpha$ of TD3+BC to be consistent with that of CPI in order to show that the performance improvement of CPI mainly comes from amalgamating the benefits of both behavior-regularized and in-sample algorithms. Further, we also compare CPI with TD3+BC with dynamically changed $\\alpha$ and TD3+BC with swept best $\\alpha$ in the ranges {0.0001, 0.05, 0.25, 2.5, 25, 36, 50, 100}, which improves TD3+BC by a large margin, to show the superiority of CPI. The selection of parameters is shown in Table 4.\n\nThe results for TD3+BC (vanilla), TD3+BC (same $\\alpha$ with CPI), TD3+BC (swept best $\\alpha$), TD3+BC with dynamically changed $\\alpha$ and CPI are shown in Table 8. Comparing the variants of TD3+BC with different $\\alpha$ choices, it can be found that changing $\\alpha$ can indeed improve the performance of TD3+BC. However, compared with TD3+BC (same $\\alpha$) and TD3+BC (swept best $\\alpha$), the performance of CPI is significantly better, which proves the effectiveness of the mechanism for iterative refinement of policy for behavior regularization in CPI.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cUD9CLZFf0", "reviewer": "Reviewer_h61i", "summary": "The authors propose a new offline RL algorithm based on the on conservative policy iteration. The main idea is that the reference policy used for behavior regularization is iteratively modified. The practical algorithm is implemented as a simple modification over TD3-BC, where an additional regularization term is added to control the distance between the current policy being trained, and a frozen version of it. The authors provide some theoretical guarantees in the tabular settings, and evaluate the algorithm on a number of standard benchmarks.", "questions": "See Weaknesses.", "limitations": "The authors discuss their limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 2, "strengths": "## Strengths\n\n1. The paper is well written, and the claims are supported with experiments.\n\n2. The proposed algorithm is simple, and can be implemented with minimal modifications to existing ones\n\n3. The experimental evaluations are extensive\n\n4. Provides theoretical results in the tabular setting", "weaknesses": "**1. The algorithm is highly dependent on the value of $\\tau$**\n\nThe algorithm seems to be dependent on $\\tau$, with some experiments requiring a value of 200, while others use values in the range [0.05,2].  Further, the authors choose the value of $\\tau$ based on the quality of the dataset, which in most real-world scenario is hard to determine.   \n\n**2. BC seems to dominate performance**\n\nHaving high values of $\\tau$ essentially reduces the offline RL problem to Behavior Cloning. I would encourage the authors to include baselines such as %BC in their results (shown in figure 1), this will help understanding the role $\\tau$ plays.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **The algorithm is highly dependent on the value of $\\tau$**\n\nIn Section 5.3.4, we provide ablations of the effect of the two hyperparameters. We also summarize some empirical experiences in the section. The regularization parameter $\\tau$ plays a crucial role in determining the weightage of the joint regularization relative to the Q-value component. We find that (Figure 6 b) $\\tau$ assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).\n\nTherefore, in real-world scenario, if you have a impression of the data quality, you could tune the parameters in a confident way. For example, in the recommendation system or the autonomous driving systems, you could directly distinguish and identify  the quality of data collected by the previous deployed algorithms via Return on Investment (ROI) or Success Rate. If the ROI of a policy is much more better than the random policy, one can try to $\\tau$ to a relative higher value. If the Success Rate of an autonomous driving car is quite low, then the  $\\tau$ should be set to a lower value.\n\nIn addition, we also provide results of CPI with as little as possible hyper-parameter tuning on the same domain in Table 5 in our appendix. The results show that compared with IQL, CQL and TD3BC, the overall performance of CPI with as little as possible hyper-parameter tuning is still significantly better.\n\n### **BC seems to dominate performance**\n\nIt’s true that high values of $\\tau$ is essential when the dataset is of higher quality (i.e., higher rewards). However, when the dataset is of lower quality (i.e., lower rewards), the Q loss plays more  important role. As you suggested, we include different \\%BC (run behavior cloning on only the top X% of timesteps in the dataset, ordered by episode returns) in our results, citing from the Decision Transformer paper:\n\n|  |  10%BC   | 25%BC  | 40%BC | 100%BC | CPI |\n| --- | --- | --- | --- | --- | --- |\n| halfcheetah-medium | 42.9  | 43.0  | 43.1 | 43.1 | **64.4** |\n| hopper-medium | 65.9  | 65.2  | 65.3  | 63.9 | **98.5** |\n| walker2d-medium |  78.8  | 80.9  | 78.8  | 77.3 | **85.8** |\n| halfcheetah-medium-replay | 40.8  | 40.9  | 41.1   | 4.3 | **54.6** |\n| hopper-medium-replay | 70.6  | 58.6 |  31.0  | 27.6 | **101.7** |\n| walker2d-medium-replay | 70.4  | 67.8 |  67.2  | 36.9 | **91.8** |\n\nIt can be observed on these datasets of lower quality, \\%BC indeed help improve over the original BC (100\\%BC). However, they still fall behind CPI.  We believe this could help you understand $\\tau$'s important role in CPI’s performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tyCJsll6er", "reviewer": "Reviewer_k25f", "summary": "The authors introduce a modified version of prior behavior-regularized offline RL methods based on conservative policy iteration, where the current policy is regularized towards an older policy. Performance benefits are demonstrated in the D4RL benchmark.", "questions": "I imagine an important hyperparameter would be the delay between the current policy and older policy, how much does this impact performance of the algorithm?", "limitations": "Satisfactory.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Easy to implement/add to TD3+BC or other closely related methods.\n- Strong performance across multiple offline datasets. \n- Significant number of ablations.", "weaknesses": "**Skepticism**\n- I’m not sure the algorithm does what the authors claim it does. If the current policy escapes the support of the behavior policy (which is possible since the KL to an older policy is a penalty and not a constraint), then the next iteration will not be a refined version of the behavior policy. Instead, like CPI, the update simply penalizes large changes to the policy. \n\n**Experiments**\n- Results are based on per-environment hyperparameter optimization. While the authors do compare against a similarly-swept version of TD3+BC, there are other possible explanations for why there are hyperparameter-specific benefits. For example, 2 hyperparameters versus 1 provides more opportunities to overfit. As mentioned below, there is some discrepancy between KL and MSE which may also explain some performance differences. \n- Anonymous code link is expired, code is unavailable. \n- Minor: The authors used KL for the behavior regularization, but talk about comparisons to TD3+BC, which uses a deterministic policy and minimizes MSE to the behavior policy instead. So I’m not sure how this is implemented and what the TD3+BC baseline uses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Whether the algorithm does what the authors claim it does.\nWhile Proposition 1 establishes that the exact solution of Eq 9 remains within the data support when the actor is initialized as $\\pi_\\omega=\\pi_D$, practical implementations often rely on a limited number of gradient descent steps for optimizing Eq 9, thus could suffer from our-of-support samples. This leads to policy optimization errors  which are further exacerbated iteratively. To approximate the in-support learning, we find it is useful to add the original behavior regularization, which further constrains the policy on the support of data to enhance learning stability. As different datasets have different support, it’s necessary to help the policy stay in the support of the dataset by tuning the hyperparameter across different environments. However, it’s not hard to tune the parameter across different environments and datasets. In Section 5.3.4, we provide ablations of the effect of the two hyperparameter and also summarize some empirical experiences in the section to help users find the suitable hyperparameters when encountering new environments.\n\n## Results are based on per-environment hyperparameter optimization.\n\nWe also provide results of CPI with as little as possible hyper-parameter tuning on the same domain in Table 5 in our appendix. The results show that compared with IQL, CQL and TD3BC, the overall performance of CPI with as little as possible hyper-parameter tuning is still significantly better.\n\nAs analyzed before, the main reason to tune the hyper-parameter is to approximate the in-support learning according to the property of different datasets. For example:\n\n- $\\lambda$ : When $\\lambda=0.1$, the early-stage performance excels, as the behavior policy assists in locating appropriate actions in the dataset. However, this results in suboptimal final convergence performance, attributable to the excessive behavior policy constraint on performance improvement. For larger values, such as 0.9 , the marginal weight of the behavior policy leads to performance increase during training. Unfortunately, the final performance might be poor. This is due to that the policy does not have sufficient behavior cloning guidance, leading to a potential distribution shift during the training process. Consequently, we predominantly select a $\\lambda$ value of 0.5 or 0.7 to strike a balance between the reference policy regularization and behavior regularization. In practice, for dataset of higher quality and lower diversity (e.g., expert dataset), we encourage authors to try small $\\lambda$. For datasets of lower quality and higher diversity (e.g., medium dataset), larger $\\lambda$ should be chosed.\n- $\\tau$ : The regularization parameter $\\tau$ plays a crucial role in determining the weightage of the joint regularization relative to the Q-value component. We find that (Figure 6 b) $\\tau$ assigned to dataset of higher quality and lower diversity (e.g., expert dataset) ought to be larger than those associated with datasets of lower quality and higher diversity (e.g., medium dataset).\n\n## Anonymous code link is expired\n\nWe have modify the link and it could be open now. Please check the anonymous  link in the paper. \n\n## discrepancy between KL and MSE\n\nActually $D_{K L}\\left(\\pi_w(s) \\| \\bar{\\pi}(s)\\right)$ was implemented in the experiment  as an MSE loss like $1 / N \\sum\\left(\\pi_w(s)-\\bar{\\pi}(s)\\right)^2$.  For two gaussians, the KL between them can be calculated as $K L(p, q)=\\log \\frac{\\sigma_2}{\\sigma_1}+\\frac{\\sigma_1^2+\\left(\\mu_1-\\mu_2\\right)^2}{2 \\sigma_2^2}-\\frac{1}{2}$. As $\\sigma_1$ equals to $\\sigma_2$ in the two gaussians, we can obtain the simplified form of this KL as an MSE loss as you described. For deterministic policies, they could be seen as the mean of a gaussian. Therefore, MSE is the specific implementation of KL in our work, so does in TD3BC. That's also the reason we claim that our  algorithm is easy to implement, requiring only a few lines of code modification to existing method, i.e., TD3+BC.  In our provided codes, you could find the corresponding implementation details. \n\n\n## the delay between the current policy and older policy, how much does this impact performance of the algorithm\n\nThe default update interval (UI) of the reference policy in CPI is set to 2 gradient steps. We ablate the update interval 4 and 8 here on six datasets across three seeds in the below table:\n\n|  | CPI  (UI=2) | CPI  (UI=4) | CPI  (UI=8) |\n| --- | --- | --- | --- |\n| halfcheetah-medium | 64.4 $\\pm$ 1.6 | 65.3 $\\pm$ 0.8 | 61.2 $\\pm$ 6.5 |\n| hopper-medium | 98.5 $\\pm$ 4.4 | 83.2 $\\pm$ 3.6 | 61.8 $\\pm$ 44.6 |\n| walker2d-medium | 85.8 $\\pm$ 1.0 | 85.6 $\\pm$ 0.9 | 81.2 $\\pm$ 5.9 |\n| halfcheetah-medium-replay | 54.6 $\\pm$ 1.5 | 52.8 $\\pm$ 0.4 | 45.4 $\\pm$ 14.0 |\n| hopper-medium-replay | 101.7 $\\pm$ 1.4 | 89.7 $\\pm$ 13.2 | 96.4 $\\pm$ 10.2 |\n| walker2d-medium-replay | 91.8 $\\pm$ 2.2 | 81.9 $\\pm$ 1.7 | 66.3 $\\pm$ 35.4 |\n\nIt could be seen that increase the update interval of reference policy in CPI could have overall negative impact on the policy’s performance. While with UI=4  on some datasets (halfcheetah-medium, walker2d-medium) it seems there isn’t significant negative influence, in other situations the performance is severely dropped. This may be attributed to the delayed update causes the performance gap between learning policy and the reference policy become larger, that is, the reference policy will remain worse performance for longer time. Thus the reference policy could be more possible to introduce more instability to the training and drag down the performance of the learning policy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
In the PDF we provide more hyperparamerts ablations.

Meta-review: The reviewers are divided into two camps: enthusiasts and skeptics. As the Area Chair, I align with the former group. I commend the clear exposition of the approach, the thoroughness and rigor of the empirical study, and the impressive performance demonstrated. Based on these strengths, I recommend accepting this submission as a poster. However, I also acknowledge the validity of Reviewer k25f's concerns regarding per-environment hyperparameter optimization. Addressing this issue more thoroughly in the final version could significantly enhance the paper's overall impact.
Decision: Accept (poster)