Title: Mildly Constrained Evaluation Policy for Offline Reinforcement Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: Offline Reinforcement Learning

Reinforcement Learning
Abstract: Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a Mildly Constrained Evaluation Policy (MCEP) for test time inference with a more constrained target policy for value estimation. Since the target policy has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3-BC [Fujimoto and Gu, 2021] and AWAC [Nair et al., 2020] algorithms. The empirical results on MuJoCo locomotion tasks show that the MCEP significantly outperforms the target policy and achieves competitive results to state-of-the-art offline RL methods. The codes are open-sourced at link.

1

Abstract

Offline reinforcement learning (RL) methodologies enforce constraints on the policy to adhere closely to the behavior policy, thereby stabilizing value learning and mitigating the selection of out-of-distribution (OOD) actions during test time. Conventional approaches apply identical constraints for both value learning and test time inference. However, our findings indicate that the constraints suitable for value estimation may in fact be excessively restrictive for action selection during test time. To address this issue, we propose a Mildly Constrained Evaluation Policy (MCEP) for test time inference with a more constrained target policy for value estimation. Since the target policy has been adopted in various prior approaches, MCEP can be seamlessly integrated with them as a plug-in. We instantiate MCEP based on TD3-BC [Fujimoto and Gu, 2021] and AWAC [Nair et al., 2020] algorithms. The empirical results on MuJoCo locomotion tasks show that the MCEP significantly outperforms the target policy and achieves competitive results to state-of-the-art offline RL methods. The codes are open-sourced at link .

1 Introduction

Offline reinforcement learning (RL) extracts a policy from data that is pre-collected by unknown policies. This setting does not require interactions with the environment thus it is well-suited for tasks where the interaction is costly or risky. Recently, it has been applied to Natural Language Processing [Snell et al., 2022], e-commerce [Degirmenci and Jones] and real-world robotics [Kalashnikov et al., 2021, Rafailov et al., 2021, Kumar et al., 2022, Shah et al., 2022] etc. Compared to the standard online setting where the policy gets improved via trial and error, learning with a static offline dataset raises novel challenges. One challenge is the distributional shift between the training data and the data encountered during deployment. To attain stable evaluation performance under the distributional shift, the policy is expected to stay close to the behavior policy. Another challenge is the "extrapolation error" [Fujimoto et al., 2019, Kumar et al., 2019] that indicates value estimate error on unseen state-action pairs or Out-Of-Distribution (OOD) actions. Worsely, this error can be amplified with bootstrapping and cause instability of the training, which is also known as deadly-triad [Van Hasselt et al., 2018]. Majorities of model-free approaches tackle these challenges by either constraining the policy to adhere closely to the behavior policy [Wu et al., 2019, Kumar et al., 2019, Fujimoto and Gu, 2021] or regularising the Q to pessimistic estimation for OOD actions [Kumar et al., 2020, Lyu et al., 2022]. In this work, we focus on policy constraints methods.

Policy constraints methods minimize the disparity between the policy distribution and the behavior

32 distribution. It is found that policy constraints introduce a tradeoff between stabilizing value estimates 33 and attaining better performance. While previous approaches focus on developing various constraints 34 for the learning policy to address this tradeoff, the tradeoff itself is not well understood. Current 35 solutions have confirmed that an excessively constrained policy enables stable values estimate 36

but degrades the evaluation performance [Kumar et al., 2019, Singh et al., 2022, Yu et al., 2023]. 37 Nevertheless, it is not clear to what extent this constraint fails to stabilize value learning and to 38 what extent this constraint leads to a performant evaluation policy. It is essential to investigate these 39 questions as their answers indicate how well a solution can be found under the tradeoff. However, 40 the investigation into the latter question is impeded by the existing tradeoff, as it requires tuning the 41 constraint without influencing the value learning. We circumvent the tradeoff and seek solutions for 42 this investigation through the critic. For actor-critic methods, [Czarnecki et al., 2019] has shed light 43 on the potential of distilling a student policy that improves over the teacher using the teacher's critic. 44 Inspired by this work, we propose to derive an extra evaluation policy from the critic to avoid solving 45 the above-mentioned tradeoff. The actor is now called target policy as it is used only to stabilize the 46 value estimation. 47

48

49

50

51

52

53

54

Based on the proposed framework, we empirically investigate the constraint strengths for 1) stabilizing value learning and 2) better evaluation performance. The results find that a milder constraint improves the evaluation performance but may fall beyond the constraint space of stable value estimation. This finding indicates that the optimal evaluation performance may not be found under the tradeoff, especially when stable value learning is the priority. Consequently, we propose a novel approach of using a Mildly Constrained Evaluation Policy (MCEP) derived from the critic to avoid solving the above-mentioned tradeoff and to achieve better evaluation performance.

55

56

57

58

59

60

61

62

63

64

As the target policy is commonly used in previous approaches, our MCEP can be integrated with them seamlessly. In this paper, we first validate the finding of [Czarnecki et al., 2019] in the offline setting by a toy maze experiment, where a constrained policy results in bad evaluation performance but its off-policy Q estimation indicates an optimal policy. After that, our experiments on D4RL [Fu et al., 2020] MoJoCo locomotion tasks showed that in most tasks milder constraint achieves better evaluation performance while more restrictive constraint stabilizes the value estimate. Finally, we instantiated MCEP on both TD3BC and AWAC algorithms. The empirical results find that the MCEP significantly outperforms the target policy and achieves competitive results to state-of-the-art offline RL methods.

2 Related Work

Policy constraints method (or behavior-regularized policy method) [Wu et al., 2019, Kumar et al., 65 2019, Siegel et al., 2020, Fujimoto and Gu, 2021] forces the policy distribution to stay close to the 66 behavior distribution. Different discrepancy measurements such as KL divergence [Jaques et al., 2019, 67 Wu et al., 2019], reverse KL divergence Cai et al. [2022] and Maximum Mean Discrepancy [Kumar 68 et al., 2019] are applied in previous approaches. [Fujimoto and Gu, 2021] simply adds a behavior69 cloning (BC) term to the online RL method Twin Delayed DDPG (TD3) [Fujimoto et al., 2018] 70 and obtains competitive performances in the offline setting. While the above-mentioned methods 71 calculate the divergence from the data, [Wu et al., 2022] estimates the density of the behavior 72 distribution using V AE, and thus the divergence can be directly calculated. Except for explicit policy 73 constraints, implicit constraints are achieved by different approaches. E.g. [Zhou et al., 2021] ensures 74 the output actions stay in support of the data distribution by using a pre-trained conditional V AE 75 (CVAE) decoder that maps latent actions to the behavior distribution. In all previous approaches, the 76 constraints are applied to the learning policy that is queried during policy evaluation and is evaluated 77 in the environment during deployment. Our approach does not count on this learning policy for the 78 deployment, instead, it is used as a target policy only for the policy evaluation. 79

While it is well-known that a policy constraint can be efficient to reduce extrapolation errors, its 80 drawback is not well-studied yet. [Kumar et al., 2019] reveals a tradeoff between reducing errors in 81 the Q estimate and reducing the suboptimality bias that degrades the evaluation policy. A constraint is 82 designed to create a policy space that ensures the resulting policy is under the support of the behavior 83 distribution for mitigating bootstrapping error. [Singh et al., 2022] discussed the inefficiency of policy 84 constraints on heteroskedastic dataset where the behavior varies across the state space in a highly 85 non-uniform manner, as the constraint is state-agnostic. A reweighting method is proposed to achieve 86 a state-aware distributional constraint to overcome this problem. Our work studies essential questions 87 about the tradeoff [Kumar et al., 2019] and overcomes this drawback [Singh et al., 2022] by using an 88 extra evaluation policy. 89

There are methods that extract an evaluation policy from a learned Q estimate. One-step RL [Brand90 fonbrener et al., 2021] first estimates the behavior policy and its Q estimate, which is later used 91 for extracting the evaluation policy. Although its simplicity, one-step RL is found to perform badly 92 in long-horizon problems due to a lack of iterative dynamic programming [Kostrikov et al., 2022]. 93 [Kostrikov et al., 2022] proposed Implicity Q learning (IQL) that avoids query of OOD actions 94 by learning an upper expectile of the state value distribution. No explicit target policy is mod95 eled during their Q learning. With the learned Q estimate, an evaluation policy is extracted using 96 advantage-weighted regression [Wang et al., 2018, Peng et al., 2019]. Our approach has a similar 97 form of extracting an evaluation from a learned Q estimate. However, one-step RL aims to avoid 98 distribution shift and iterative error exploitation during iterative dynamic programming. IQL avoids 99 error exploitation by eliminating OOD action queries and abandoning policy improvement (i.e. the 100 policy is not trained against the Q estimate). Our work instead tries to address the error exploitation 101 problem and evaluation performance by using policies of different constraint strengths. 102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

3 Background

We model the environment as a Markov Decision Process (MDP) ⟨ S, A, R, T, p 0 ( s ) , γ, ⟩ , where S is the state space, A is the action space, R is the reward function, T ( s ′ | s, a ) is the transition probability, p 0 ( s ) is initial state distribution and γ is a discount factor. In the offline setting, a static dataset D β = { ( s, a, r, s ′ ) } is pre-collected by a behavior policy π β . The goal is to learn a policy π ϕ ( s ) with the dataset D that maximizes the discounted cumulated rewards in the MDP:

Next, we introduce the general policy constraint method, where the policy π ϕ and an off-policy Q estimate Q θ are updated by iteratively taking policy improvement steps and policy evaluation steps, respectively. The policy evaluation step minimizes the Bellman error:

where the θ ′ is the parameter for a delayed-updated target Q network. The Q value for the next state is calculated with actions a t +1 from the learning policy that is updated through the policy improvement step:

where C is a constraint measuring the discrepancy between the policy distribution π ϕ and the behavior distribution π β . The w ∈ (0 , ∞ ] is a weighting factor. Different kinds of constraints were used such as Maximum Mean Discrepancy (MMD), KL divergence, and reverse KL divergence.

4 Method

In this section, we first introduce the generic algorithm that can be integrated with any policy constraints method. Next, we introduce two examples based on popular offline RL methods TD3BC and AWAC. With a mildly constrained evaluation policy, we name these two instances as TD3BC with MCEP (TD3BC-MCEP) and AWAC with MCEP (AWAC-MCEP) .

4.1 Offline RL with mildly constrained evaluation policy

The proposed method is designed for overcoming the tradeoff between a stable policy evaluation and a performant evaluation policy. In previous constrained policy methods, a restrictive policy constraint is applied to obtain stable policy evaluation. We retain this benefit but use this policy (actor) ˜ π as a target policy only to obtain stable policy evaluation. To achieve better evaluation performance, we introduce an MCEP π e that is updated by taking policy improvement steps with the critic Q ˜ π . Different from ˜ π , π e does not participate in the policy evaluation procedure. Therefore, a mild policy constraint can be applied, which helps π e go further away from the behavior distribution without influencing the stability of policy evaluation. We demonstrate the policy spaces and policy trajectories for ˜ π and π e in the l.h.s. diagram of Figure 1, where π e is updated in the wider policy space using Q ˜ π .

Qtt2

Qnt1

PE

Qπ元o

π1./Qπ1

Qit2

元

Qπ

TO!

元3

Qπto

PI

PI

Restrictive Constrained Policies

Policy Improvement

EvaluationPolicySpace

Policy Evaluation

The overall algorithm is shown as pseudo-codes 133 (Alg. 1). At each step, the Q ˜ π , ˜ π ψ and π e ϕ are 134 updated iteratively. A policy evaluation step up135 dates Q ˜ π by minimizing the TD error (line 7), 136 i.e. the deviation between the approximate Q 137 and its target value. Next, a policy improve138 ment step updates ˜ π ψ (line 6. These two steps 139 form the actor-critic algorithm. After that, π e ϕ 140 is extracted from the Q ˜ π , by taking a policy im141 provement step with a policy constraint that is 142 likely milder than the constraint for ˜ π ψ (line 7). 143 Many approaches can be taken to obtain a milder 144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

Algorithm 1 MCEP Training

1: Hyperparameters: LR α , EMA η , ˜ w and w e

2: Initialize: θ, θ ′ , ψ, and ϕ

3: for i=1, 2, ..., N do

4: θ ← θ -α L Q ( θ ) (Equation 2)

5: θ ′ ← (1 -η ) θ ′ + ηθ

6: ψ ← ψ -α L ˜ π ( ψ ; ˜ w ) (Equation 3)

7: ϕ ← ϕ -α L π e ( ϕ ; w e ) (Equation 3)

policy constraint. For example, tuning down the weight factor w e for the policy constraint term or replacing the constraint measurement with a less restrictive one. Note that the constraint for π e ϕ is necessary (the constraint term should not be dropped) as the Q ˜ π has large approximate errors for state-action pairs that are far from the data distribution.

4.2 Two Examples: TD3BC-MCEP and AWAC-MCEP

TD3BC with MCEP TD3BC takes a minimalist modification on the online RL algorithm TD3. To keep the learned policy to stay close to the behavior distribution, a behavior-cloning term is added to the policy improvement objective. TD3 learns a deterministic policy therefore the behavior cloning is achieved by directly regressing the data actions. For TD3BC-MCEP, the target policy ˜ π ψ has the same policy improvement objective as TD3BC:

where the ˜ λ = ˜ α 1 N ∑ s i ,a i | Q θ ( s i ,a i ) | is a normalizer for Q values with a hyper-parameter ˜ α : The Q θ is updated with the policy evaluation step similar to Eq. 2 using ˜ π ψ . The MCEP π e ϕ is updated by policy improvement steps with the Q ˜ π taking part in. The policy improvement objective function for π e ϕ is similar to Eq. 4 but with a higher-value α e for the Q-value normalizer λ e . The final objective for π e ϕ is

AWAC with MCEP AWAC [Nair et al., 2020] is an advantage-weighted behavior cloning method. 160 As the target policy imitates the actions from the behavior distribution, it stays close to the behavior 161 distribution during learning. In AWAC-MCEP, the policy evaluation follows the Eq. 2 with the target 162

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

(a) Toy maze MDP

(b)

V

∗

, π

∗

(c)

V

˜

π

,

˜

π

(d)

V

˜

π

,

Q

˜

π

10

8

6

4

2

policy ˜ π ψ that updates with the following objective: 163

where the advantage A ( s, a ) = Q θ ( s, a ) -Q θ ( s, ˜ π ψ ( s )) . This objective function solves an advantageweighted maximum likelihood. Note that the gradient will not be passed through the advantage term. As this objective has no policy improvement term, we use the original policy improvement with KL divergence as the policy constraint and construct the following policy improvement objective:

where the weighting factor λ e is a hyper-parameter. Although the Eq. 6 is derived by solving Eq. 8 in a parametric-policy space, the original problem (Eq. 8) is less restrictive even with ˜ λ = λ e as it includes a -Q ( s, π e ( s )) term. This difference means that even with a λ e > ˜ λ , the policy constraint for π e could still be more relaxed than the policy constraint for ˜ π .

5 Experiments

In this section, we set up 4 groups of experiments to illustrate: 1) the policy constraint might degrade the evaluation performance by forcing the policy to stay close to low-state-value transitions. 2) The suitable constraint for the final inference could be milder than the ones for safe Q estimates. 3) Our method shows significant performance improvement compared to the target policy and achieves competitive results to state-of-the-art offline RL methods on MuJoCo locomotion tasks. 4) the MCEP generally gains a higher estimate Q compared to the target policy. Additionally, we adopt 2 groups of ablation studies to verify the benefit of an MCEP and to investigate the constraint strengths of MCEP.

Environments D4RL [Fu et al., 2020] is an offline RL benchmark consisting of many task sets. Our experiments involve MuJoCo locomotion tasks ( -v2 ) and two tasks from Adroit ( -v0 ). For MuJoCo locomotion tasks, we select 4 versions of datasets: data collected by a uniformly-random agent ( random ), collected by a medium-performance policy ( medium ), a 50% -50% mixture of the medium data and the replay buffer during training a medium-performance policy ( medium-replay ), a 50% -50% mixture of the medium data and expert demonstrations ( medium-expert ). For Adroit, we select pen-human and pen-cloned , where the pen-human includes a small number of human demonstrations, and pen-cloned is a 50% -50% mixture of demonstrations and data collected by rolling out an imitation policy on the demonstrations.

5.1 Target policy that enables safe Q estimate might be overly constrained

To investigate the policy constraint under a highly suboptimal dataset, we set up a toy maze MDP that is similar to the one used in [Kostrikov et al., 2022]. The environment is depicted in Figure 2a, where

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

TD3-BChopper-medium-v2

TD3-BC walker2d-medium-replay-v2

AWAC hopper-medium-replay-v2

15

10

5

α=1

α=4

α=10

4

2

2

入=0.5

入=0.3

入=1

入=0.1

return

100

80

normalizedr

60

40

20

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

0.0

0.2

0.4

0.6

0.8

1.0

wa-me

ha-m

100

betterfinal inference

safeQestimate

08

ro-ime

60

40

29

ha-me

wa-rk

wa-mr

ha-mr

the lower left yellow grid is the starting point and the upper left green grid is the terminal state that gives a reward of 10 . Other grids give no reward. Dark blue indicates un-walkable areas. The action space is defined as 4 direction movements (arrows) and staying where the agent is (filled circles). There is a 25% probability that a random action is taken instead of the action from the agent. For the dataset, 99 trajectories are collected by a uniformly random agent and 1 trajectory is collected by an expert policy. Fig. 2b shows the optimal value function (colors) and one of the optimal policies.

We trained a constrained policy using Eq. 2 and Eq. 8 in an actor-critic manner, where the actor is constrained by a KL divergence with a weight factor of 1 . Figure 2c shows the value function and the policy. We observe that the learned value function is close to the optimal one in Figure 2b. However, the policy does not make optimal actions in the lower left areas where the state values are relatively low. As the policy improvement objective shows a trade-off between the Q and the KL divergence, when the Q value is low, the KL divergence term will obtain higher priority. i.e. in low-Q-value areas, the KL divergence takes the majority for the learning objective, which makes the policy stays closer to the transitions in low-value areas. However, we find that the corresponding value function indicates an optimal policy. In Figure 2d, we recover a greedy policy underlying the learned critic that shows an optimal policy. In conclusion, the constraint might degrade the evaluation performance although the learned critic may indicate a better policy. Although such a trade-off between the Q term and the KL divergence term can be alleviated in previous work [Fujimoto and Gu, 2021] by normalizing the Q values, in the next section, we will illustrate that the constraint required to obtain performant evaluation policy can still cause unstable value estimate.

5.2 Test-time inference requires milder constraints

The previous experiment shows that a restrictive constraint might harm the test-time inference, which motivates us to investigate what constraints make better evaluation performance. Firstly, we relax the policy constraint on TD3BC and AWAC by setting up different hyper-parameter values that control the strengths of the policy constraints. For TD3BC, we set α = { 1 , 4 , 10 } ([Fujimoto and Gu, 2021] recommends α = 2 . 5 ). For AWAC, we set λ = { 1 . 0 , 0 . 5 , 0 . 3 , 0 . 1 } ([Nair et al., 2020] recommends λ = 1 ). Finally, We visualize the evaluation performance and the learned Q estimates.

In Figure 4, the left two columns show the training of TD3BC in the hopper-medium-v2 and walker2dmedium-replay-v2 . In both domains, we found that using a milder constraint by tuning the α from 1 to 4 improves the evaluation performance, which motivates us to expect better performance with α = 10 . As shown in the lower row, we do observe higher performances in some training steps. However, unstable training is caused by the divergence in value estimate, which indicates the tradeoff between the stable Q estimate and the evaluation performance. The rightmost column shows the training of AWAC in hopper-medium-replay-v2 , we observe higher evaluation performance by relaxing the constraint ( λ > 1 ). Although the Q estimate keeps stable during the training in all λ values, higher λ results in unstable policy performance and causes the performance crash with λ = 0 . 1 .

Concluding on all these examples, a milder constraint can potentially improve the performance but may cause unstable Q estimates or unstable policy performances. As we find that relaxing the constraint on current methods triggers unstable training, which hinders the investigation of constraints

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

for better evaluation performance. We instead systematically study the constraint strengths in TD3BC and TD3BC with evaluation policy (TD3BC-EP).

We first tune the α for TD3BC to unveil the range for safe Q estimates. Then in TD3BC-EP, we tune the α e for the evaluation policy with a fixed ˜ α = 2 . 5 to approximate the constraint range of better test inference performance (i.e. where the evaluation policy outperforms the target policy). The ˜ α = 2 . 5 is selected to ensure a stable Q estimate (also the paper-recommended value). The α ( α e ) is tuned within { 2 . 5 , 5 , 10 , 20 , 30 , 40 , 50 , 60 , 70 , 80 , 90 , 100 } . For each α ( α e ), we observe the training of 5 runs with different random seeds. In Figure 5, we visualize these two ranges for each task from MuJoCo locomotion set. The blue area shows α values where the TD3BC Q estimate is stable for all seeds. The edge shows the lowest α value that causes Q value explosion. The orange area shows the range of α e where the learned evaluation policy outperforms the target policy. Its edge (the orange line) shows the lowest α e values where the evaluation policy performance is worse than the target policy. For each task, the orange area has a lower bound α e = 2 . 5 where the evaluation policy shows a similar performance to the target policy.

Note that α weighs the Q term and thus a larger α indicates a less restrictive constraint. Comparing the blue area and the orange area, we observe that in 6 out of the 9 tasks, the α for better inference performance is higher than the α that enables safe Q estimates, indicating that test-time inference requires milder constraints. In the next section, we show that with an MCEP, we can achieve much better inference performance without breaking the stable Q estimates.

5.3 Comparison on MuJoCo locomotion and Adroit

We compare the proposed method to state-of-the-art offline RL methods CQL and IQL, together with our baselines TD3BC and AWAC. Similar hyper-parameters are used for all tasks from the same domain. For our baseline methods (TD3BC and AWAC), we use the hyper-parameter recommended by their papers. TD3BC uses α = 2 . 5 for its Q value normalizer and AWAC uses 1 . 0 for the advantage value normalizer. In TD3BC-MCEP, the target policy uses ˜ α = 2 . 5 and the MCEP uses α e = 10 . In AWAC-MCEP, the target policy has ˜ λ = 1 . 0 and the MCEP has λ e = 0 . 6 . The full list of hyper-parameters can be found in the Appendix.

As is shown in Table 1, we observe that the evaluation policies with a mild constraint significantly outperform their corresponding target policy. TD3BC-MCEP gains progress on all medium and medium-replay datasets. Although the progress is superior, we observe a performance degradation on the medium-expert datasets which indicates an overly relaxed constraint for the evaluation policy. To overcome this imbalance problem, we designed a behavior-cloning normalizer. The results are shown in the Appendix. Nevertheless, the TD3BC-MCEP achieves much better general performance than the

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

target policy. In the AWAC-MCEP, we observe a consistent performance improvement over the target policy on most tasks. Additionally, evaluation policies from both TD3BC-MCEP and AWAC-MCEP outperform the CQL and IQL while the target policies have relatively low performances. On Adroit tasks, the best results are obtained by behavioral cloning agent and TD3BC with a high BC weighting factor. Other agents fail to outperform the BC agent. We observe that MCEP does not benefit these tasks where behavior cloning is essential for the evaluation performance.

5.4 Ablation Study

In this section, we design 2 groups of ablation studies to investigate the effect of the extra evaluation policy and its constraint strengths. Reported results are averaged on 5 runs of different random seeds.

Performance of the extra evaluation policy. Now, we investigate the performance of the introduced evaluation policy π e . For TD3BC, we set the parameter α = { 2 . 5 , 10 . 0 } . A large α indicates a milder constraint. After that, we train TD3BC-MCEP with ˜ α = 2 . 5 and α e = 10 . 0 . For AWAC, we trained AWAC with the λ = { 1 . 0 , 0 . 5 } and AWAC-MCEP with ˜ λ = 1 . 0 and λ e = 0 . 5 .

The results are shown in Figure 6. By comparing TD3BC of different α values, we found a milder constraint ( α = 10 . 0 ) brought performance improvement in hopper tasks but de-

TD3BCandTD3BC-MCEP

AWACandAWAC-MCEP

100

α=2.5

入=1.0

α=10

A=0.5

NormalizedReturn

08

α =2.5, α =10

62.370.2

61.9

71.9

=1,A = 0.5

61.5

71.4

69.664.870.7

60

48 47.8

50.4

52.8

32.4

43.9,

39.8

49.5

45.3

40

20

0

hopper

walker2d

hopper

walker2d

grades the performance in walker2d tasks. The degradation is potentially caused by unstable value estimates (see experiment at section 5.2). Finally, the evaluation policy trained from the critic learned with a target policy with α = 2 . 5 achieves the best performance in all three tasks. In AWAC, a lower λ value brought policy improvement in hopper tasks but degrades performances in half-cheetah and walker2d tasks. Finally, an evaluation policy obtains the best performances in all tasks.

In conclusion, we observe consistent performance improvement brought by an extra MCEP that circumvents the tradeoff brought by the constraint.

Constraint strengths of the evaluation policy. We set up two groups of ablation experiments to investigate the performance of evaluation policy under different constraint strengths. For TD3BC-MCEP, we tune the constraint strength by setting the Q normalizer hyper-parameter. The target policy hyper-parameter is fixed to α = 2 . 5 . We pick three strengths for evaluation policy α e = { 1 . 0 , 2 . 5 , 10 . 0 } to create more restrictive, similar, and milder constraints, respectively. For AWACMCEP, the target policy uses λ = 1 . 0 . However, it is not straightforward to create a similar constraint for the eval- uation policy as it has a different policy improvement objective. We set λ e = { 0 . 6 , 1 . 0 , 1 . 4 } to show how performance changes with different constraint strengths.

)

The performance improvements over the target policy are shown in Fig. 7. The left column shows a 314 significant performance drop when the evaluation policy has a more restrictive constraint ( α e = 1 . 0 315 than the target policy. A very close performance is shown when the target policy and the evaluation 316 policy have similar policy constraint strengths ( α e = 2 . 5 ). Significant policy improvements are 317

TD3BC-MCEP

AWAC-MCEP

0.4

α=1

α=2.5

0.36

0.6

A=1.0

A=1.4

0.49

0.63

Return

0.3

α=10

0.4

A=0.6

Normalized

0.1

0.2

0.05

0.08

0.06

0.16

0.2

0.08

0.170.15

0.14

0.01

0.0

0.01

0.0

-0.06

0.01

-0.1

-0.05

-0.11

0.2

0.2

0.4

0.3

-0.24

F9°0-

0.4

halfcheetah

hopper

walker2d

halfcheetah

hopper

walker2d

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

(a) medium-expert

(b) medium

(c) medium-replay

(d) random

env

˜

π

(%)

π

e

(%)

TD3BC-MCEP

wa-me

69.8

87.2

wa-m

66.2

82.7

wa-mr

71.8

88.7

wa-r

89.6

99.0

AWAC-MCEP

ha-me

63.4

70.8

ha-m

64.7

68.3

ha-mr

68.6

73.1

ha-r

75.3

95.6

target policy

evaluation policy

0

2

-1

0

2

2

0

2

1e6

0

25

50

75

100

0

Z

1.0

0.5

0.0

0.5

1.0

1

0

2

20

0

20

40

obtained with the target policy having a milder constraint ( α e = 10 ). The right column presents the results of AWAC-MCEP. Generally, the performance in hopper tasks keeps increasing with milder constraints while the half-cheetah and walker2d tasks show performances that increase from λ = 1 . 4 to λ = 1 and similar performances between λ = 1 and λ = 0 . 6 . Compared to the target policy, the evaluation policy consistently outperforms in half-cheetah and hopper tasks. On the walker2d task, a strong constraint ( λ = 1 . 4 ) causes a performance worse than the target policy but milder constraints ( λ = { 1 , 0 . 6 } ) obtain similar performance to the target policy.

In conclusion, for both algorithms, we observe that on evaluation policy, a milder constraint obtains higher performance than the target policy while a restrictive constraint may harm the performance.

5.5 Estimated Q values for the learned evaluation policies

To compare the performance of the policies learned in Section 5.3 on the learning objective (maximizing the Q values), we counted Q differences between the policy action and the data action Q ( s, π ( s )) -Q ( s, a ) in the training data (visualized in Figure 9). Proportions of data points that show positive differences are listed in Table 2, where we find that on more than half of the data, both the target policy and the MCEP have larger Q estimation than the behavior actions. Additionally, the proportions for the MCEP are higher than the proportions for the target policy in all datasets, indicating that the MCEP is able to move further toward large Q values.

6 Conclusion

This work focuses on the policy constraints methods where the constraint addresses the tradeoff between stable value estimate and evaluation performance. While to what extent the constraint achieves the best results for each end of this tradeoff remains unknown, we first investigate the constraint strength range for a stable value estimate and for evaluation performance. Our findings indicate that test time inference requires milder constraints that can go beyond the range of stable value estimates. We propose to use an auxiliary mildly constrained evaluation policy to circumvent the above-mentioned tradeoff and derive a performant evaluation policy. The empirical results show that MCEP obtains significant performance improvement compared to target policy and achieves competitive results to state-of-the-art offline RL methods. Our ablation studies show that an auxiliary evaluation policy and a milder policy constraint are essential for the proposed method. Additional empirical analysis demonstrates higher estimated Q values are obtained by the MCEP.

Limitations. Although the MCEP is able to obtain a better performance, it depends on stable value estimation. Unstable value learning may crash both the target policy and the evaluation policy. While the target policy may recover its performance by iterative policy improvement and policy evaluation, we observe that the evaluation policy may fail to do so. Therefore, a restrictive constrained target policy that stabilizes the value learning is essential for the proposed method.

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

References

David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems , 34:4933-4946, 2021.

Cai, C. Zhang, L. Zhao, W. Shen, X. Zhang, L. Song, J. Bian, T. Qin, and T. Liu. Td3 with reverse kl regularizer for offline reinforcement learning from mixed datasets. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 21-30, Los Alamitos, CA, USA, dec 2022. IEEE Computer Society. doi: 10.1109/ICDM54844.2022.00012. URL https://doi. ieeecomputersociety.org/10.1109/ICDM54844.2022.00012 .

Wojciech M Czarnecki, Razvan Pascanu, Simon Osindero, Siddhant Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. Distilling policy distillation. In The 22nd international conference on artificial intelligence and statistics , pages 1331-1340. PMLR, 2019.

Soysal Degirmenci and Chris Jones. Benchmarking offline reinforcement learning algorithms for e-commerce order fraud evaluation. In 3rd Offline RL Workshop: Offline RL as a'Launchpad' .

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actorcritic methods. In International conference on machine learning , pages 1587-1596. PMLR, 2018.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456 , 2019.

Dmitry Kalashnikov, Jacob Varley, Yevgen Chebotar, Benjamin Swanson, Rico Jonschkowski, Chelsea Finn, Sergey Levine, and Karol Hausman. Mt-opt: Continuous multi-task robotic reinforcement learning at scale. arXiv preprint arXiv:2104.08212 , 2021.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022. URL https:// openreview.net/forum?id=68n2s9ZJWF8 .

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Aviral Kumar, Anikait Singh, Stephen Tian, Chelsea Finn, and Sergey Levine. A workflow for offline model-free robotic reinforcement learning. In Conference on Robot Learning , pages 417-428. PMLR, 2022.

Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview. net/forum?id=VYYf6S67pQc .

Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359 , 2020.

Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint arXiv:1910.00177 , 2019.

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

Rafael Rafailov, Tianhe Yu, Aravind Rajeswaran, and Chelsea Finn. Offline reinforcement learning from images with latent space models. In Learning for Dynamics and Control , pages 1154-1168. PMLR, 2021.

Dhruv Shah, Arjun Bhorkar, Hrishit Leen, Ilya Kostrikov, Nicholas Rhinehart, and Sergey Levine. Offline reinforcement learning for visual navigation. In 6th Annual Conference on Robot Learning , 2022. URL https://openreview.net/forum?id=uhIfIEIiWm_ .

Noah Siegel, Jost Tobias Springenberg, Felix Berkenkamp, Abbas Abdolmaleki, Michael Neunert, Thomas Lampe, Roland Hafner, Nicolas Heess, and Martin Riedmiller. Keep doing what worked: Behavior modelling priors for offline reinforcement learning. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=rke7geHtwH .

Anikait Singh, Aviral Kumar, Quan Vuong, Yevgen Chebotar, and Sergey Levine. Offline rl with realistic datasets: Heteroskedasticity and support constraints. arXiv preprint arXiv:2211.01052 , 2022.

Charlie Snell, Ilya Kostrikov, Yi Su, Mengjiao Yang, and Sergey Levine. Offline rl for natural language generation with implicit language q learning. arXiv preprint arXiv:2206.11871 , 2022.

Hado Van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, and Joseph Modayil. Deep reinforcement learning and the deadly triad. arXiv preprint arXiv:1812.02648 , 2018.

Qing Wang, Jiechao Xiong, Lei Han, Peng Sun, Han Liu, and Tong Zhang. Exponentially weighted imitation learning for batched historical data. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , pages 6291-6300, 2018.

Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy optimization for offline reinforcement learning. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=KCXQ5HoM-fy .

Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11361 , 2019.

Lantao Yu, Tianhe Yu, Jiaming Song, Willie Neiswanger, and Stefano Ermon. Offline imitation learning with suboptimal demonstrations via relaxed distribution matching. arXiv preprint arXiv:2303.02569 , 2023.

Wenxuan Zhou, Sujay Bajracharya, and David Held. Plas: Latent action space for offline reinforcement learning. In Conference on Robot Learning , pages 1719-1735. PMLR, 2021.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the problem of policy constraints in offline reinforcement learning (RL) settings, and finds the phenomenon that milder constraints on policies during training can lead to better performances at inference tests. The proposed component MCEP can be added on existing algorithms including TD3BC and AWAC. Experiments on D4RL dataset show improved performances over vanilla TD3BC and AWAC.\n\nSTRENGTHS:\nThe strength of policy constraints in offline RL is an important problem. It is a novel perspective to separate the policies for value estimation and inference with different constraint levels. I would say the proposition that milder constraints can improve policy inference performances is an interesting problem. The experiments are thorough with necessary ablation studies, and the results indeed show some improvement by using the proposed constraining method.\n\nWEAKNESSES:\nI think one major critique of the paper is: the most essential discovery that milder constraints may be required for test-time inference is mostly from experimental evaluations. The observations are not even consistent for that only 6 out of 9 shows this pattern. This is not strong evidence showing that milder constraints are necessarily always better. Some theoretical analysis or at least insights about this observation can be provided to make it more convincing.\n\nAnother critique is that although the experiments show some improvement by using MCEP on TD3BC and AWAC and over some baselines like CQL and IQL. These are not the SOTA results on these offline datasets, there exists better algorithms proposed by the time of NeurIPS submission that should be aware of:\n\n[1] Hansen-Estruch, Philippe, et al. \"Idql: Implicit q-learning as an actor-critic method with diffusion policies.\" arXiv preprint arXiv:2304.10573 (2023).\n\n[2] Garg, Divyansh, et al. \"Extreme Q-Learning: MaxEnt RL without Entropy.\" arXiv preprint arXiv:2301.02328 (2023).\n\n[3]  Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" arXiv preprint arXiv:2208.06193 (2022).\n\nThe paper writing can be further improved.\n\nQUESTIONS:\nHow to evaluate the strength of restriction for Eq. (8) given the fact that it has an additional $Q$ term? I think it’s not fair to just say Eq. (8) is less restrictive than Eq. (6) since after taking an exponential function it is similar as Eq. (6) but with advantage $A$ replaced by $Q$.\n\nFor Fig. 5, I think it’s better to just give the $\\alpha$ values in a table.\n\nFig. 4, are the results averaged across different seeds?\n\nLIMITATIONS:\nThe limitations of the current method are discussed, that the evaluation policies in MCEP may not be consistent with the value function and can lead to unstable value estimation if used in policy evaluation.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-22"}

comments: SUMMARY:
This paper proposes Mildly Constrained Evaluation Policy (MCEP) for offline reinforcement learning to address the issue of excessively restrictive constraints for action selection during test time inference. MCEP uses a more constrained target policy for value estimation and another less restrictive policy for performance evaluation. Empirical results demonstrate the effectiveness of MCEP.

STRENGTHS:
MCEP is easy to implement and can be plugged into many policy constraint offline RL methods.

WEAKNESSES:
Since $\pi_e$ does not participate in the policy evaluation, I think line 7 of Algorithm 1 can be removed and $\pi_e$ can be extracted from Q after actor critic learning to save computational cost. The contribution of MCEP is only to extract a less restrictive policy after RL learning, which is somewhat limited.

The overall idea of the paper is quite simple. However, the notations and descriptions are a bit confusing. For example, the notations in Algorithm 1 lack a clear definition ($\psi, \phi, \tilde \pi, \pi^e, \tilde w, w^e, \mathcal L(.,.)$). And $\psi$ and $\phi$ in line 6 and 7 of Algorithm 1 are reversed, since Q evaluation in Equation 2 is associated with $\phi$. 

No theory supports MCEP in the paper.

QUESTIONS:
I do not have additional questions.

LIMITATIONS:
More hyperparameters need to be tuned for MCEP compared with the original algorithm.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-03

comments: SUMMARY:
This work addresses the issue of excessive policy constraints in stabilizing value estimation within the offline RL paradigm. A separate target policy is used solely for evaluation and stabilizing value estimation, which is more constrained than the "evaluation policy." The evaluation policy does not participate in policy evaluation and is improved by the value function estimates, with the level of constraint adjusted by the weight of the term.

STRENGTHS:
Major points:
- This procedure can be easily integrated into offline RL algorithms that utilize policy constraints, and empirical results apply it to TD3+BC and AWAC. The empirical findings demonstrate promising improvements, with baselines encompassing the standard suite of state-of-the-art offline RL algorithms.
- The paper is well-written, easy to comprehend, and thoughtfully structured. 
- The idea itself is intuitive, and the toy experiments convincingly demonstrate that over-constraint poses a significant issue. Figure 2 clearly illustrates the adverse effects of over-constraint, with the policy performing poorly in low state value regions of the maze.
- The ablation studies are extensive and demonstrate the method's effectiveness.

I believe this simple yet intuitive method is worth presenting to the broader offline RL community. I believe this work should be accepted.

WEAKNESSES:
Major points:
- While the results show promise, they do not indicate substantial improvements across many environments, and there is some inconsistency observed. The method shows a decrease in performance in the medium-expert D4RL tasks and the pen task.

QUESTIONS:
Can you explain the inconsistencies in the results, especially in the pen task? What could be a possible reason?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nOffline reinforcement learning (RL) methods frequently involve a policy constraint to mitigate error propagation when learning the Q function. Generally, a single constraint strength is used throughout training. This paper proposes instead to use different constraint strengths for learning the target policy, which is only used for learning the Q function, and the evaluation policy, which is the final policy returned by the algorithm. In particular, a stronger constraint is needed to ensure stability when training the target policy, but weakening the constraint for the evaluation policy can lead to better performance.\n\nSTRENGTHS:\n* The idea is fairly general and can be instantiated with various RL algorithms, as shown in the paper.\n* The experimental results provide insight into the role of the constraint and the tradeoff between stability and performance.\n* Conceptually, the approach allows for a continuum of algorithms between one-step RL and standard actor-critic methods.\n* The paper is clearly written and understandable.\n\nWEAKNESSES:\n* The algorithm introduces an additional hyperparameter that requires tuning, which is already a challenge in offline RL.\n* The paper found that “in 6 out of the 9 tasks, the $\\alpha$ for better inference performance is higher than the $\\alpha$ that enables safe Q estimates”. While 6/9 is a majority, this is not convincing evidence that weakening the constraints is always helpful.\n\nQUESTIONS:\nIn Section 5.2, how exactly do you determine \"the lowest $\\alpha$ value that causes Q value explosion\"? (In particular, how is \"explosion\" defined?) Could this lead to a hyperparameter tuning procedure that involves only looking at the Q values and does not require off-policy evaluation or sample collection?\n\nLIMITATIONS:\nYes, limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}