Title: ExID: Offline RL with Intuitive Expert Insights in Limited-Data Settings
Venue: NeurIPS 2024
Accepted: no
Keywords: Offline Reinforcement Learning

Domain knowledge
Abstract: With the ability to learn from static datasets, Offline Reinforcement Learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-the-art offline RL algorithms perform sub-optimally when confronted with limited data confined to specific regions within the state space. The performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This paper proposes a novel domain knowledge-based regularization technique and adaptively refines the initial domain knowledge to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase compared to ensemble of domain knowledge and existing offline RL algorithms operating on limited data.

Abstract

With the ability to learn from static datasets, Offline Reinforcement Learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-theart offline RL algorithms perform sub-optimally when confronted with limited data confined to specific regions within the state space. The performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This paper proposes a novel domain knowledgebased regularization technique and adaptively refines the initial domain knowledge to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase compared to ensemble of domain knowledge and existing offline RL algorithms operating on limited data.

1 Introduction

Offline RL [9, 1], also referred to as batch RL, is a learning approach that focuses on extracting knowledge solely from static datasets. This class of algorithms has a wider range of applications being particularly appealing to real-world data sets from business [46], healthcare [25], and robotics [35]. However, offline RL poses unique challenges, including over-fitting and the need for generalization to data not present in the dataset. To surpass the behavior policy, offline RL algorithms need to query Q values of actions not in the dataset, causing extrapolation errors [21]. Most offline RL algorithms address this problem by enforcing constraints that ensure that the learned policy does not deviate too far away from the data set's state action distribution [13, 11] or is conservative towards Out-of-Distribution (OOD) actions [21, 20]. However, such approaches are designed on coherent batches [13], which do not account for OOD states.

In many domains, such as business and healthcare, available data is scarce and often confined to expert behaviors within a limited state space.

For example, a sales recommendation system, where historic data may not contain details about many active users and operator gives coupon of higher value to

attract sales.

Learning on such limited data sets can curtail the generalization capabilities of state-of- the-art (SOTA) offline RL algorithms, resulting in sub-optimal performance [23]. We illustrate this

limitation via Fig 1. In Fig 1a) the state action space of a simple Mountain Car environment [27] is

plotted for an expert dataset [32] and a partial dataset with first 10% samples from the entire dataset. 31

Fig 1b) shows the average reward obtained over these data sets and the average difference between 32

the Q value of action taken by the under-performing Conservative Q Learning (CQL) [21] agent and 33

the action in the full expert dataset for unseen states. It can be observed that the performance of the 34

offline RL agent considerably drops. This is attributed to the critic overestimating the Q value of 35

non-optimal actions for states that do not occur in the dataset while training. 36

CQL-MCFullExpert-CQL-MCo.1Expert

-120

Reward

-140

3D Scatter Plot - Full Expert Dataset Mountain-Car Vo

3D Scatter Plot - 10% Samples Expert Dataset Mountain-Car V0

-160

Full Dataset

。10,000Samples

-180

1.75

2.00

2.00

1.75

-200

0

50

100

150

200

250

Episodes

1.00

1.50

125

1.50

125

1.00

0.6

QDifference

CQL-MCo.1Expert

0.75

0.50

0.75

0.5

0.50

0.25

0.00

0.25

0.4

0.00

0.04

0.06

0.02

0.04

0.3

0.2

1.25

0.50

Position

0.25

0.00

0.25

0.50

0.06

0.04

0.02,

0.00

0.02

0.6

Position

0.4

0.2

0.0

0.2

0.04

0.02

0.00

0.1

0

50

100

150

200

250

Episode

30

(a)

(q)

In numerous real-world applications, expert insights regarding the general behavior of a policy are 37 often accessible [33]. For example, sales operators often distribute lower discount coupons to active 38 users to maximize profit . While these insights may not be optimal, they serve as valuable guidelines 39 for understanding the overall behavior of the policy. A rich literature in knowledge distillation [18] 40 has shown that teacher networks trained on domain knowledge can transfer knowledge to another 41 network unaware of it. This work aims to leverage a teacher network mimicking simple decision 42 tree-based domain knowledge to help offline RL generalize in limited data settings. 43

45

46

47

We introduce an algorithm dubbed ExID , leveraging intuitive human obtainable expert insights. The domain expertise is incorporated into a teacher policy, which improves offline RL in limited-data settings through regularization.

48

49

50

51

52

53

54

55

The teacher based on expected performance improvement of the offline policy during training, improving the teacher network beyond initial heuristics.

We demonstrate the effectiveness of our methodology on real sales promotion dataset , several discrete OpenAI gym and Minigrid environments with standard offline RL data sets and show that ExID significantly exceeds the performance when faced with limited data.

2 Related Work

This work improves offline RL learning on batches sampled from static datasets using domain expertise. One of the major concerns in offline RL is the erroneous extrapolation of OOD actions

[13]. Two techniques have been studied in the literature to prevent such errors. 1) Constraining the 56

policy to be close to the behavior policy 2) Penalizing overly optimistic Q values [24]. We discuss a 57

58

few relevant algorithms following these principles. In Batch-Constrained deep Q-learning (BCQ)

candidate actions sampled from an adversarial generative model are considered, aiming to 59

balance proximity to the batch while enhancing action diversity. Algorithms like Random Ensemble 60

Mixture Model (REM) [2], Ensemble-Diversified Actor-Critic (EDAC) [3] and Uncertainty Weighted 61

Actor-Critic (UWAC) [42] penalize the Q value according to uncertainty by either using Q ensemble 62

networks or directly weighting the loss with uncertainty. CQL [21] enforces regularization on Q63

functions by incorporating a term that reduces Q-values for OOD actions while increasing Q-values 64

for actions within the expected distribution. However, these algorithms do not handle OOD actions 65

for states not in the static dataset and can have errors induced by changes in transition probability. 66

Integration of domain knowledge in offline RL, though an important avenue, has not yet been 67

extensively explored. Domain knowledge incorporation has improved online RL with tight regret 68 bounds [33, 4]. In offline RL, bootstrapping via blending heuristics computed using Monte-Carlo 69 returns with rewards has shown to outperform SOTA algorithms by 9% [15]. Recent works improve 70 offline RL by incorporating a safety expert [40] and preference query [44], contrary to our work 71 which improves imperfect domain knowledge. The closest to our work is Domain Knowledge guided 72 Q learning (DKQ) [46] where domain knowledge is represented in terms of action importance and 73 the Q value is weighted according to importance. However, obtaining action importance in practical 74 scenarios is nontrivial. 75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

3 Preliminaries

A DRL setting is represented by a Markov Decision Process (MDP) formalized as ( S, A, T, r, ρ 0 , γ ) . Here, S denotes the state space, A signifies the action space, T ( s ′ | s, a ) represents the transition probability distribution, r : S × A → R is the reward function, ρ 0 represents the initial state distribution, and γ ∈ (0 , 1] is the discount factor. The primary objective of any DRL algorithm is to identify an optimal policy π ( a | s ) that maximizes E s t ,a t [ ∑ ∞ t =0 γ t r ( s t , a t )] where, s 0 ∼ d 0 ( . ) , a t ∼ π ( . | s t ) , and s ′ ∼ T ( . | s t , a t ) . Deep Q networks (DQNs) [26] learn this objective by minimizing the Bellman residual ( Q θ ( s, a ) -B π θ Q θ ( s, a )) 2 where B π θ Q θ ( s, a ) = E s ′ ∼ T [ r ( s, a ) + γ E a ′ ∼ π θ ( . | s ′ ) [ Q θ ′ ( s ′ , a ′ )]] . The policy π θ chooses actions that maximize the Q value max a ′ ∈ A Q θ ( s ′ , a ′ ) . However, in offline RL where transitions are sampled from a pre-collected dataset B , the chosen action a ′ may exhibit a bias towards OOD actions with inaccurately high Q-values. To handle the erroneous propagation from OOD actions, CQL [22] learns conservative Q values by penalizing OOD actions. The CQL loss for discrete action space is given by

Eq. 1 encourages the policy to be close to the actions seen in the dataset. However, CQL works on the assumption of coherent batches, i.e., if ( s, a, s ′ ) ∈ B , then s ′ ∈ B . There is no provision for handling OOD actions for s / ∈ B , which can lead to policy failure when data is limited. In the next sections, we present ExID, a domain knowledge-based approach to improve performance in data-scarce scenarios.

4 Problem Setting and Methodology

In our problem setting, the RL agent learns the policy on a limited dataset with rare and unseen demonstrations. We define the characteristics of this dataset as follows:

Definition 4.1. A reduced buffer B r is a proper subset of the full dataset B i.e., B r ⊂ B satisfying the following conditions:

Some states in B are not present in B r , i.e., ∃ s ′ ∈ B ∧ ∀ ( s, a, s ′ ) : ( s, a, s ′ ) / ∈ B r

The number of samples N ( s, a, s ′ ) for some transitions in B are less in B r i.e, ∃ ( s, a, s ′ ) ∈ B : N ( s, a, s ′ ) B r < N ( s, a, s ′ ) B

̸

We observe, performing Q -Learning by sampling from a limited buffer B r may not converge to an optimal policy for the MDP M B representing the full buffer. This can be shown as a special case of (Theorem 1,[13]) as p B ( s ′ | s, a ) = p B r ( s ′ | s, a ) and no Q updates for ( s, a ) / ∈ B r leading to sub-optimal policy. Please refer to the App. B for analysis and example.

We also assume a set of common sense rules in the form of domain knowledge is available. Domain knowledge D is defined as hierarchical decision nodes capturing S → A as represented by Eq. 2. Each decision node T η i is represented by a constraint ϕ η i and Boolean indicator µ η i function selects the branch to be traversed based on ϕ η i .

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

{(Si,ai,S'i,ri)}

Critic Qe

2.Criticregularization

S~Br

Critic policy as =

S,r

Q

O

Tβ

argmax(Qe)

B

Reduced

Br

Buffer

a

Behaviourpolicy

Full offline data

Yes

1.Trainingteacher actorwithdomainknowledge

S~Br

Update Qe

Warm

Start?

additionalloss

(i)

Check if

↑

S|=D

(s |= D)

No

No

s~U(B(s))

S|=D

Is criticpolicy

s1>1

betterwithhigher

certainity?

s2<1

action2

(i)

O

Yes

3:UpdateTeacher

Cross

Entropyy

Commonsensedomainknowledge

a=D(s)

BehaviorclonedTeacher

Teacher action

D:S->A

3

at= Tw (s)

(a)

(q)

̸

We assume that D gives heuristically reasonable actions for s | = D and S D ∩ S B r = ∅ where S D , S B r are the state coverage of D and B r .

Training Teacher: An overview of our methodology is depicted in Fig 2. We first construct a trainable actor network π ω t parameterized by ω from D , Fig 2 step 1. For training π ω t synthetic data ˆ S is generated by sampling states from a uniform random distribution over state boundaries B ( s ) , ˆ S = U ( B ( S )) . Note that this does not represent the true state distribution and may have state combinations that will never occur. We train π ω t using behavior cloning where state ˆ s ∼ ˆ S is checked with root decision node in Eq. 2. A random action is chosen if ˆ s does not satisfy decision node T η 0 or leaf action is absent. If ˆ s satisfies a T η i , T η i is traversed and action a η i is returned from the leaf node. This is illustrated in Fig 2 (a). We term the pre-trained actor network π ω t as the teacher policy.

Regularizing Critic: We now introduce Algo 1 (App C) to train an offline RL agent on B r . Algo 1 takes B r and pretrained π ω t as input. The algorithm uses two hyper-parameters, warm start parameter k and mixing parameter λ . A critic network Q θ s with Monte-Carlo (MC) dropout and target network Q θ ′ s are initialized. ExID is divided into two phases. In the first phase, we aim to warm start the critic network Q θ s with actions from π ω t as shown in Fig 2b( i). However, this must be done selectively as the teacher's policy is random around the states that do not satisfy domain knowledge. In each iteration, we first check the states sampled from a mini-batch of B r with D . For the states which satisfy D we compute the teacher action π ω t ( s ) and critic's action argmax a ( Q θ s ( s, a )) and collect it in lists a t , a s , Algo 1 lines 4-10. Our main objective is to keep actions chosen by the critic network for s | = D close to the teacher's policy. To achieve this, we introduce a regularization term:

̸

Eq 3 incentivizes the critic to increase Q values for actions from π ω t and decreases Q values for other 129 actions when argmax a ( Q θ s ( s, a )) = π ω t ( s ) for states that satisfy domain knowledge. Note that Eq 3 130 will only be 0 when argmax a ( Q θ s ( s, a )) = π ω t ( s ) for s | = D . It is also set to 0 for s ̸| = D . However, 131 since π ω t mimicking heuristic rules is sub-optimal, it is also important to incorporate learning from 132 the data. The final loss is a combination of Eq. 1 and Eq. 3 with a mixing parameter λ ∈ [0 , 1] : 133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

The choice of λ and the warm start parameter k depends on the quality of D . In the case of perfect domain knowledge, λ would be set to 1, and setting λ to 0 would lead to the vanilla CQL loss. Mixing both the losses allows the critic to learn both from the data in B r and knowledge in D .

Updating Teacher: Given a reasonable warm start, the critic is expected to give higher Q values for optimal actions for s ∈ D ∩ B r as it learns from data. We aim to leverage this knowledge to enhance the initial teacher policy π ω t trained on heuristic domain knowledge. For s ∼ B and s | = D , we calculate the average Q values over critic actions and teacher actions and check which one is higher in Algo 1 line 11 which refers to Cond. 6. For brevity E s ∼B r ∧ s | = D is written as E . If E ( Q θ s ( s, a s )) > E ( Q θ s ( s, a t )) it denotes the critic expects a better return on an average over its own policy than the teacher's policy. Hence, we can use the critic's policy to update π ω t , making it better over D . However, only checking the critic's value can be erroneous as the critic can have high values for OOD actions. We check the average uncertainty of the predicted Q values to prevent the teacher from getting updated by OOD actions. Uncertainty has been shown to be a good metric for OOD action detection by [42, 3]. A well-established methodology to capture uncertainty is predictive variance, which takes inspiration from Bayesian formulation for the critic function and aims to maximize p ( θ | X,Y ) = p ( Y | X,θ ) p ( θ ) /p ( Y | X ) where X = ( s, a ) and Y represents the true Q value of the states. However, p ( Y | X ) is generally intractable and is approximated using Monte Carlo (MC) dropout, which involves including dropout before every layer of the critic network and using it during inference [14]. Following [42], we measure the uncertainty of prediction using Eq 5.

Eq 5 estimates the variance of Q value Q ( s, a ) for an action a using T forward passes on the Q θ s ( s, a ) with dropout where ¯ Q ( s, a ) represents the predictive mean. We check the average uncertainty of the Q value for action chosen by the critic and teacher policy over the states that match domain knowledge in a batch. The teacher network is updated using the critic's action only when the policy expects a higher average Q return on its action and the average uncertainty of taking this action is lower than the teacher action. E ( V ar T Q θ s ( s r , a s )) < E ( V ar T Q θ s ( s r , a t )) indicates the actions were learned from the expert data in the buffer and are not OOD samples. The condition is summarized in cond. 6:

We update the teacher with cross-entropy described in Eq 7:

where, π s ( s, a ) = e Q ( s,a ) ∑ a ′ Q ( s,a ′ ) . When the critic's policy is better than the teacher's policy, L r ( θ ) is set to 0 Algo 1 Lines 11 to 13. Finally, the critic network is updated using calculated loss L ( θ ) Algo 1 Lines 17-18. We theoretically analyse the implications of using ExID in propositions 4.2 and 4.3.

Proposition 4.2. Denote ˆ π as the policy learned by ExID, π u as any offline RL policy learned on B r and optimal Q function as Q ∗ and V function as V ∗ . Then it holds that

Where α = E s ∼ O [ V ∗ ( s ) -Q ∗ ( s, ˆ π ( s ))] , ¯ ρ π ( s ) = [ 1 | S ˆ π | (1 -γ ) , 1 1 -γ ] ( | S ˆ π | is the number of different states observed by ˆ π ) and O / ∈ B r . Here α denotes the quality of regularized action for s / ∈ B r . Hence, updating π ω t is important as high divergence of action from the optimal can lead to performance degradation. In offline RL, the extrapolation error for non optimal action is usually high for states not observed in dataset (as illustrated in 1b), regularization can lead performance improvement when π ω t is reasonable. Furthermore, in ExID coarse actions from π ω t are updated driving them closer to the optimal actions, improving the performance lower bound. Additionally π ω t increases | S ˆ π | making ¯ ρ π ≪ 1 in practice further improving the performance lower bound. Proof is deferred to App. A .

Proposition 4.3. ExID reduces generalization error if Q ∗ ( s, π ω t ( s )) > Q ∗ ( s, π u ( s )) for s ∈ D ∩ B r . 175

176

Proof is deferred to App. A . In the next section, we discuss our empirical evaluations.

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

5 Empirical Evaluations

We investigate the following through our empirical evaluations: 1. Does ExID perform better than combining D and offline RL algos on different environments with datasets exhibiting rare and OOD states Sec 5.2? 2. Does ExID generalize to OOD states covered by D Sec 5.4? 3. What is the effect of varying k , λ and updating π ω t Sec 5.5? 4. How does performance vary with the quality of D Sec 5.6?

5.1 Experimental Setting

We evaluate our methodology on open-AI gym [5], MiniGrid [6] and real sales promotion (SP) [30] offline data sets. All our data sets are generated using standard methodologies defined in [32, 31] except SP which is generated by human operators. All experiments have been conducted on a Ubuntu 22.04.2 LTS system with 1 NVIDIA K80 GPU, 4 CPUs, and 61GiB RAM. App. F notes the hyperparameter values and network architectures.

Dataset: We experiment on three types of data sets. Expert Data-set [10, 16, 22] generated using an optimal policy without any exploration with high trajectory quality but low state action coverage. Replay Data-set [2, 13] generated from a policy while training it online, exhibiting a mixture of multiple behavioral policies with high trajectory quality and state action coverage. Noisy Data-set [12, 13, 22, 16] generated using an optimal policy that also selects random actions with ϵ greedy strategy where ϵ = 0 . 2 having low trajectory quality and high state action coverage. Additionally we also experiment on human generated dataset for sales promotion task.

Baselines: We do comparative studies on 10 baselines for OpenAI gym datasets. The first baseline simply checks the conditions of D and applies corresponding actions in execution. The performance of this baseline shows that D is imperfect and does not achieve the optimal reward. CQL SE is from [40] where the expert is replaced by D . The other baselines are an ensemble of D and eight algorithms popular in the Offline RL literature for discrete environments. These algorithms include Behavior Cloning (BC) [29], Behaviour Value Estimation (BVE) [16], Quantile Regression DQN (QRDQN) [7], REM, MCE, BCQ, CQL and Critic Regularized Regression Q-Learning (CRR) [41]. For a fair comparison, we use actions from domain knowledge for states not in the buffer and actions from the trained policy for other states to obtain the final reward. Hence, each algorithm is renamed with the suffix D in Table 5.1.

Limiting Data: To create limited-data settings for benchmark datasets, we first extract a small percentage of samples from the full dataset and remove some of the samples based on state conditions. This is done to ensure the reduced buffer satisfies the conditions defined in Def 4.1. We describe the specific conditions of removal in the next section. Further insights and the state visualizations for selected reduced datasets are in App H. Note : no data reduction has been performed on SP dataset to demonstrate a real dataset exhibits characteristics of reduced buffer .

5.2 Performance across Different Datasets

Our results for OpenAI gym environments are summarised in Table 5.1 and Minigrid in Table 3 (App D). We observe the performance of offline RL algorithms degrades substantially when part of the data is not seen and trajectory ratios change. For these cases with only 10% partial data, ExID surpasses the performance by at least 27% in the presence of reasonable domain knowledge. The proposed method performs strongest on the replay dataset where the contribution of L r ( θ ) is significant due to state coverage, and the teacher learns from high-quality trajectories. Environment details are described in the App. D. All domain knowledge trees are shown in the App. D Fig 10. We describe limiting data conditions and domain knowledge specific to the environment as follows:

Mountain Car Environment: [27] We use simple, intuitive domain knowledge in this environment shown in the App. D Fig 10 (c), which represents taking a left action when the car is at the bottom of the valley with low velocity to gain momentum; otherwise, taking the right action to drive the car up. Fig 6 (c) shows the state action pairs this rule generates on states sampled from a uniform random distribution over the state boundaries. It can be observed that the states of D cover part of the missing

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

CQL-MCFullIExpertEXID-MC0.1Replay

/EXID-MC0.1Noisy

CQLSESPCQLDSPEXIDSP

-100

Reward

-100

Reward

800

sp_score

-120

-120

750

-140

-140

700

-160

-160

650

600

-180

-180

550

-200

0

50

100

150

200

250

Episodes-200

0

50

100

150

200

250

Episodes

500

20

40

60

80

100

120

Episodes

140

(a)

(q)

(c)

data in Fig 1 (a). For limiting datasets, we remove states with position > -0.8. The performance of CQLD and ExID are shown in Fig 3 (a),(b) where ExID surpasses CQLD for all three datasets.

Cart-pole Environment: For this environment, we use domain knowledge from [33], which aims to move in the direction opposite to the lean of the pole, keeping the cart close enough to the center. If the cart is close to an edge, the domain knowledge attempts to account for the cart's velocity and recenter the cart. The full tree is given in the App. D Fig 10 (a). We remove states with cart velocity > -1.5 to create the reduced buffer.

Lunar-Lander Environment: We borrow the decision nodes from [34] and get actions from a sub-optimal policy trained online with an average reward of 52.48. The full set of decision nodes is shown in the App. D Fig 10 (b). D focuses on keeping the lander balanced when the lander is above ground. When the lander is near the surface, D focuses on keeping the y velocity lower. To create the reduced datasets, we remove data of lander angle < -0.04.

Mini-Grid Environments: For our experiments, we choose two environments: Random Dynamic Obstacles 6X6 and LavaGapS 7X7. We use intuitive domain knowledge which avoids crashing into obstacles in front, left, or right of agent ref. App. D Fig 10 (d), (e). We remove states with obstacles on the right for creating limited data settings. Due to limitation of space we report the results of the best-performing algorithms on the replay dataset in Table 3 (App D).

5.3 Case study on real human generated Sales Promotion (SP) dataset

SP dataset and environment [30] simulates a real-world sales promotion platform. The number of coupons and the discount the user received will affect his behavior. A higher discount will promote

the sales, but the cost will also increase. The goal for the platform operator is to maximize the 245 total profit. The horizon of the dataset is 50 days for the training and 30 days for the test. Domain 246 knowledge ([30], App A] : Active users can be given more coupons with lower discount to maximize 247 profit. We model this as order number > 60 ∧ Avg fee > 0 . 8 = ⇒ [5 , 0 . 95] where action 1 is number 248 of coupons range [0,5] and action 2 is coupon value (discount value = (1-coupon value)) range 249 [0.6,0.95]. The dataset exhibits the properties in Def 4.1 as first 50 days of sales does not contain 250 many active users as reported in the coverage column of Tab 2 depicting scarcity. The domain rule is 251 imperfect as coupon value and number depend on multiple factors such as user purchase history and 252 behavior. As illustrated in the table 2 and Fig 3 (c) the intuitive domain rule enhances performance 253 by 10.49% in the real dataset. 254

5.4 Generalization to OOD states and contribution of L r ( θ ) 255

CQL-MC0.1Noisy—EXID-MC0.1Noisy

Noisy－Expert－Replay

0.6

Difference

1e+5

Difference

0.5

5

0.4

0.3

Q

1

Q

0.2

0.1

0

Episode

1e-5

Episode

0

Ste

50

100

150

200

250

300

50

100

150

200

250

300

50

100

150

200

250

300

(a)

(b)

(c)

In Fig 4 (a), (b), we plot Q θ s ( s, a expert ) -Q θ s ( s, a θ ) for CQL and EXID policies for different datasets of Mountain-Car environments. Action a expert is obtained from the full expert dataset where position > -0 . 8 . We observe that the Q value for actions of CQL policy diverges from the expert policy actions with high values for the states not in the reduced buffer, whereas ExID stays close to the expert actions for the unseen states. This empirically shows generalization to OOD states not in the dataset but covered by domain knowledge. In Fig 4 (d), we plot the contribution by L r ( θ ) during the training and observe the contribution is higher for replay data sets with more state coverage.

5.5 Performance on varying λ , k , and ablation of π ω t

256

257

258

259

260

261

262

263

We study the effect of varying λ on the algorithm for the given domain knowledge. We empirically 264 observe setting a high or a low λ can yield sub-optimal performance, and λ = 0 . 5 generally gives 265 good performance. In Fig 5 (a), we show this effect for LunarLander. Plots for other environments 266 are in the App. G Fig 11. For k we observe setting the warm start parameter to 0 yields a sub-optimal 267 policy, as the critic may update π ω t without completely learning from it. The starting performance 268 increases with an increase in k as shown in Fig 5 (b) for LunarLander. k = 30 works best according 269 to empirical evaluations. Plots for other environments are in the App. G Fig 12. We show two 270 ablations for Cart-pole in Fig 5 (c) with no teacher update after the warm start and no inclusion of 271 L r ( θ ) after the warm start. The warm start in this environment is set to 30 episodes. Fig 5 c) shows 272 without teacher updated, the sub-optimal teacher drags down the performance of the policy beyond 273 the warm start, exhibiting the necessity of π ω t update. Also, the student converges to a sub-optimal 274 policy if no L r ( θ ) is included beyond the warm start. 275

276

lambda=0一lambda=0.1-lambda=0.5

-lambda=0.8

EXID-JustWarmStart

200

Reward

200

Reward

Reward

100

100

400

0

0

300

-100

-100

200

-200

-200

100

-300

Episodes

-300

Episodes

0

Episodes

0

50

100

150

200

250

0

50

100

150

200

250

0

50

100

150

(a)

(q)

(c)

一Rule1一Rule2-Rule3-Rule4

3D Scatter Plot - State action distribution for teacher Mountain-Car V0

100000 samples

Rule 1

246.49±27.0

Rule5

一Baseline

200

Rule 2

9.65±111.80

2.00

1.75

0

125 g

1.50

Rule 3

52.48±26.51

-200

0.50

1.00

0.75

0.25

Rule 4

-110.72±48.35

-400

0.03

0.00

0.02

Rule 5

-379.25±73.05

-600

Episodes

0.90.8 0.7 0.6 0.5

0.00 g

0.01

0.01,g9

-400

-300-200-100

0

100

200

0

20

40

60

80

10

Position

-0.3 _02

0.03

0.02

AverageReward

(a)

(b)

(c)

5.6 Effect of varying D quality

We show the effect of choosing policies as D with different average rewards for Lunar-Lander expert 277 data in Fig 6 (a) and (b). Rule 1 is optimal and has almost the same effect as Rule 3, which is the D 278 used in our experiments exhibiting that updating a sub-optimal D can lead to equivalent performance 279 as optimal D . Using a rule with high uncertainty, as Rule 2, induces high uncertainty in the learned 280 policy but performs slightly better than the baseline. Rule 4, which has a lower average reward, also 281 causes gains on average performance with slower convergence. Finally, Rule 5, with very bad actions, 282 affects policy performance adversely and leads to a performance lower than baseline CQL. 283

284

6 Conclusion and Limitation

In this paper, we study the effect of limited and partial data on offline RL and observe that the 285 performance of SOTA offline RL algorithms is sub-optimal in such settings. The paper proposes a 286 methodology to handle offline RL's performance degradation using domain insights. We incorporate 287 a regularization loss in the CQL training using a teacher policy and refine the initial teacher policy 288 while training. We show that incorporating reasonable domain knowledge in offline RL enhances 289 performance, achieving a performance close to full data. However, this method is limited by the 290 quality of the domain knowledge and the overlap between domain knowledge states and reduced 291 buffer data. The study is also limited to discrete domains. In the future, the authors would like to 292 improve on capturing domain knowledge into the policy network without dependence on data and 293 extending the methodology to algorithms that handle continuous action space. 294

295

7 Broader Impact

During the trial-and-error training phase, RL agents may exhibit irrational behavior, which can be 296 risky and costly in real-world scenarios. As a more practical alternative to online RL, offline RL 297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

utilizes pre-existing collected data to eliminate the need for real-time interactions during training. However, a drawback of offline RL is its dependence on the quality and quantity of historical data, which, when sub-optimal, could adversely affect overall performance. Therefore, through this work, we use domain knowledge to suppress erroneous actions when available data is limited. However, this inclusion may facilitate harmful behavior in the presence of biased domain knowledge. Therefore, we advocate the use of well-regulated domain knowledge obtained from experts. Beyond this, we do not foresee any ethical impact on our work.

References

A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Neural Networks and Learning Systems , 2023. ISSN 21622388. doi: 10.1109/TNNLS.2023.3250269.

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , pages 104-114. PMLR, 2020.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. Advances in neural information processing systems , 34:7436-7447, 2021.

Peter L. Bartlett and Ambuj Tewari. Regal: A regularization based algorithm for reinforcement learning in weakly communicating mdps. In Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence , UAI '09, page 35-42, Arlington, Virginia, USA, 2009. AUAI Press. ISBN 9780974903958.

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv:1606.01540 , 2016.

Maxime Chevalier-Boisvert, Bolun Dai, Mark Towers, Rodrigo de Lazcano, Lucas Willems, Salem Lahlou, Suman Pal, Pablo Samuel Castro, and Jordan Terry. Minigrid & miniworld: Modular & customizable reinforcement learning environments for goal-oriented tasks. CoRR , abs/2306.13831, 2023.

Will Dabney, Mark Rowland, Marc Bellemare, and Rémi Munos. Distributional reinforcement learning with quantile regression. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 32, 2018.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Damien Ernst, Pierre Geurts, and Louis Wehenkel. Tree-based batch mode reinforcement learning. Journal of Machine Learning Research , 6, 2005.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems , 34:20132-20145, 2021.

Scott Fujimoto, Edoardo Conti, Mohammad Ghavamzadeh, and Joelle Pineau. Benchmarking batch deep reinforcement learning algorithms. arXiv preprint arXiv:1910.01708 , 2019.

Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning , pages 2052-2062. PMLR, 2019.

Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout in recurrent neural networks. Advances in neural information processing systems , 29, 2016.

Sinong Geng, Aldo Pacchiano, Andrey Kolobov, and Ching-An Cheng. Improving offline rl by blending heuristics. arXiv preprint arXiv:2306.00321 , 2023.

Caglar Gulcehre, Sergio Gómez Colmenarejo, Ziyu Wang, Jakub Sygnowski, Thomas Paine, Konrad Zolna, Yutian Chen, Matthew Hoffman, Razvan Pascanu, and Nando de Freitas. Regularized behavior value estimation. arXiv preprint arXiv:2103.09575 , 2021.

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

Zhiting Hu, Xuezhe Ma, Zhengzhong Liu, Eduard Hovy, and Eric Xing. Harnessing deep neural networks with logic rules. arXiv preprint arXiv:1603.06318 , 2016.

Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In 347 Proceedings of the Nineteenth International Conference on Machine Learning , pages 267-274, 2002. 348

349

350

351

Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning , pages 5774-5783. PMLR, 2021.

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems , 32, 2019.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems , 33:1179-1191, 2020.

Sergey Levine, Aviral Kumar, G. Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. ArXiv , abs/2005.01643, 2020. URL https://api. semanticscholar.org/CorpusID:218486979 .

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Siqi Liu, Kay Choong See, Kee Yuan Ngiam, Leo Anthony Celi, Xingzhi Sun, and Mengling Feng. Reinforcement learning for clinical decision support in critical care: comprehensive review. Journal of medical Internet research , 22(7):e18477, 2020.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature , 518(7540):529-533, 2015.

Andrew William Moore. Efficient memory-based learning for robot control. Technical report, University of Cambridge, Computer Laboratory, 1990.

Susan A Murphy. A generalization error for q-learning. 2005.

Dean A Pomerleau. Efficient training of artificial neural networks for autonomous navigation. Neural computation , 3(1):88-97, 1991.

Rong-Jun Qin, Xingyuan Zhang, Songyi Gao, Xiong-Hui Chen, Zewen Li, Weinan Zhang, and Yang Yu. Neorl: A near real-world benchmark for offline reinforcement learning. Advances in Neural Information Processing Systems , 35:24753-24765, 2022.

Kajetan Schweighofer, Markus Hofmarcher, Marius-Constantin Dinu, Philipp Renz, Angela Bitto-Nemling, Vihang Prakash Patil, and Sepp Hochreiter. Understanding the effects of dataset characteristics on offline reinforcement learning. In Deep RL Workshop NeurIPS 2021 , 2021. URL https://openreview.net/ forum?id=A4EWtf-TO3Y .

Kajetan Schweighofer, Marius-constantin Dinu, Andreas Radler, Markus Hofmarcher, Vihang Prakash Patil, Angela Bitto-Nemling, Hamid Eghbal-zadeh, and Sepp Hochreiter. A dataset perspective on offline reinforcement learning. In Conference on Lifelong Learning Agents , pages 470-517. PMLR, 2022.

Andrew Silva and Matthew Gombolay. Encoding human domain knowledge to warm start reinforcement learning. In Proceedings of the AAAI conference on artificial intelligence , volume 35, pages 5042-5050, 2021.

Andrew Silva, Matthew Gombolay, Taylor Killian, Ivan Jimenez, and Sung-Hyun Son. Optimization methods for interpretable differentiable decision trees applied to reinforcement learning. In International conference on artificial intelligence and statistics , pages 1855-1865. PMLR, 2020.

Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4rl: Surprisingly simple self-supervision for offline reinforcement learning in robotics. In Aleksandra Faust, David Hsu, and Gerhard Neumann, editors, Proceedings of the 5th Conference on Robot Learning , volume 164 of Proceedings of Machine Learning Research , pages 907-917. PMLR, 08-11 Nov 2022.

Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A simple semi-supervised learning framework for object detection. arXiv preprint arXiv:2005.04757 , 2020.

Jiaxi Tang and Ke Wang. Ranking distillation: Learning compact ranking models with high performance for recommender system. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2289-2298, 2018.

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

Raphael Tang, Yao Lu, Linqing Liu, Lili Mou, Olga Vechtomova, and Jimmy Lin. Distilling task-specific knowledge from bert into simple neural networks. arXiv preprint arXiv:1903.12136 , 2019.

Wei-Cheng Tseng, Tsun-Hsuan Johnson Wang, Yen-Chen Lin, and Phillip Isola. Offline multi-agent reinforcement learning with knowledge distillation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 226-237. Curran Associates, Inc., 2022.

Richa Verma, Durgesh Kalwar, Harshad Khadilkar, and Balaraman Ravindran. Guiding offline reinforcement learning using a safety expert. In Proceedings of the 7th Joint International Conference on Data Science & Management of Data (11th ACM IKDD CODS and 29th COMAD) , pages 82-90, 2024.

Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems , 33:7768-7778, 2020.

Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua M. Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. In International Conference on Machine Learning , 2021. URL https://api.semanticscholar.org/CorpusID:234763307 .

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10687-10698, 2020.

Qisen Yang, Shenzhi Wang, Matthieu Gaetan Lin, Shiji Song, and Gao Huang. Boosting offline reinforcement learning with action preference query. In International Conference on Machine Learning , pages 39509-39523. PMLR, 2023.

Li Yuan, Francis EH Tay, Guilin Li, Tao Wang, and Jiashi Feng. Revisiting knowledge distillation via label smoothing regularization. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3902-3910, 2020. doi: 10.1109/CVPR42600.2020.00396.

Xiaoxuan Zhang and S Zhang Y Yu. Domain knowledge guided offline q learning. In Second Offline Reinforcement Learning Workshop at Neurips , volume 2021, 2021.

Ying Zheng, Haoyu Chen, Qingyang Duan, Lixiang Lin, Yiyang Shao, Wei Wang, Xin Wang, and Yuedong Xu. Leveraging domain knowledge for robust deep reinforcement learning in networking. In IEEE INFOCOM 2021 - IEEE Conference on Computer Communications , pages 1-10, 2021. doi: 10.1109/INFOCOM42981.2021.9488863.

A Theoretical Analysis 427

428

429

430

431

432

433

434

435

Notations

For any deterministic policy π the performance return is formulated as η ( π ) = E τ ∼ π [ ∑ ∞ t =0 γ t r ( s t , a t )]

For any policy π , ρ π is the (unormalized) discounted visitation frequency given by ρ π ( s ) = ∑ ∞ t =0 γ t P ( s t = s ) where s 0 ∼ ρ 0 ( s 0 ) and the trajectory ( s 0 , s 1 , . . . ) is sampled from the policy π and ρ π ( s ) ∈ [0 , 1 1 -γ ] . ¯ ρ π ( s ) = sup { ρ π ( s ) , s ∈ S } ∈ [ 1 | S π | (1 -γ ) , 1 (1 -γ ) ]

We denote the regularized policy learned by ExID on B r as ˆ π and the unregularized policy as π u .

Lemmas

We introduce the following Lemma required for our theoretical analysis.

Lemma A.1. ([44]) Given two policies π 1 and π 2 436

Proof. Please refer to Lemma A.1 Eq 17 in [44] 437

Proposition A.2. (4.2) Denote ˆ π as the policy learned by ExID, π u as any offline RL policy learned on B r and 438 optimal Q function as Q ∗ and V function as V ∗ . Then it holds that 439

Proof. According to [19] performance improvement between two policies if given by 440

Replacing π 1 by ˆ π and π 2 by π u and by following Lemma A.1 441

Dividing the state space into in dataset domain states (I) and OOD states (O). The 442

Since the regularization loss facilitates visitation to OOD states via knowledge distillation we assume ρ ˆ π = ρ π u -∆ i for s ∈ i and ρ ˆ π = ρ π u +∆ o for s ∈ o where ∆ i ∈ [0 , ρ π u ( s ) ] and ∆ o ∈ [0 , 1 1 -γ -ρ π u ( s ) ] 443

Under assumption in distribution action can be learned from the dataset due to conservatism of offline RL ( Q ∗ ( s, ˆ π ( s )) -Q ∗ ( s, π u ( s ))) ≈ 0 , a ≥ 0 444

Further loosening the lower bound 445

Combining Eq 14, 17 and 19, and denoting α = E s ∼ O [ V ∗ ( s ) -Q ∗ ( s, ˆ π ( s ))]

Hence, Proposition 4.2 follows Q.E.D

446

447

448

Proposition A.3. (4.3) Algo 1 reduces generalization error if Q ∗ ( s, π ω t ( s )) > Q ∗ ( s, π ( s )) for s ∈ D ∩ B r , 449 where π is vanilla offline RL policy learnt on B r . 450

Proof. Generalization error for any policy π as defined by [28] can be written as: 451

Here, E τ ∼ π represents sampling trajectories with policy π . Since the state space is continuous, we can represent the expectation as an integral over the state space 452

Analysing with respect to s ∈ D ∩ B r we can break the integration into two parts 453

For a policy ˆ π learnt in Algo 1 the action for s t = s ∈ D is regularized to be close to π ω t which either follows domain knowledge or expert demonstrations. Hence, it is reasonable to assume Q ∗ ( s t , π ω t ( s t )) > Q ∗ ( s t , π ( s t )) . It follows 454

Note for s / ∈ D , f ( s | ˆ π ) ≈ f ( s | π ) . This is because the regularization term assigns max Q value to a different action for s ∈ D but max a ( Q ( s, a )) remains same 455

Hence, G ˆ π < G π Proposition 2 follows Q.E.D 456

457

458

459

460

461

462

463

464

465

466

481

B Missing Examples

Performing Q -Learning by sampling from a reduced batch B r may not converge to an optimal policy for the MDP M B representing the full buffer.

Example (Theorem 1,[13]) defines MDP M B of B from same state action space of the original MDP M with transition probabilities p B ( s ′ | s, a ) = N ( s,a,s ′ ) ∑ ˜ s N ( s,a, ˜ s ) where N ( s, a, s ′ ) is the number of times ( s, a, s ′ ) occurs in B and an terminal state s init . It states p B ( s init | s, a ) = 1 when ∑ ˜ s N ( s, a, ˜ s ) = 0 . This happens when transitions of some s ′ of ( s, a, s ′ ) are missing from the buffer, which may occur in B r when B r ⊂ B . r ( s init , s, a ) is initialized to Q ( s, a ) . We assume that a policy learned on reduced dataset B r converges to optimal value function and disprove it using the following counterexample:

Original MDP M

Buffer MDP M

Reduced Buffer MDP M,Br

0,2

0,0

0,2,0.2

0,0,0.3

0,2,0.6

0,10,1

1

2

3

1

2

3

1

2

Sinit

1,5

1,2

1,3,0.3

1,2,0.2

1,3,0.4

1,3,1

0

1

0

1

0

1

1

7

2

1

4

6

1

8

7

2

10

3

2

0

2

2

10

3

3

0

0

3

0

0

sinit

0

0

Qtableafterone

Qtableafterone

Initialized q table

step backup for M

step backup for MBr

467

468

469

470

471

472

473

474

475

476

477

478

479

480

DomainKnowledge

ReducedBuffer

statecoverage

StateCoverage

statecoverage

Unseen states during

training

Actions corrected by

Actions corrected by

regularization

knowledge distillation

482

483

484

485

486

487

488

489

490

A visualization is shown in Fig 8.

C Algorithm

The pseudo code of the algorithm is described in Algo 1. 491

̸

We take a simple MDP illustrated in Fig 7 with 3 states and 2 actions (0,1). The reward of each action is marked along the transition. The sampled MDP is constructed the following samples (1,0,2)2,(1,1,2)-3, (2,0,3)-3, and (2,1,3)-2 and the reduced buffer MDP with samples (1,0,2)-2 and (1,1,2)-1. The probabilities are marked along the transition. It is easy to see that the policy learned under the reduced MDP converges to a nonoptimal policy after one step of the Q table update with Q ( s, a ) = r ( s, a ) + p ( s ′ | s, a ) ∗ max a ′ ( Q ( s ′ , a ′ )) . This happens because of transition probability shift on reducing samples p B ( s ′ | s, a ) = p B r ( s ′ | s, a ) and no Q updates for ( s, a ) / ∈ B r .

Our methodology addresses these issues as follows:

For s ∈ D ∩B r better actions are enforced through regularization using π ω t even when the transition probabilities are low for optimal transitions.

Incorporating regularization distills the teacher's knowledge in the critic-enhancing generalization.

Algorithm 1 Pseudo code for EXID

Input: Reduced buffer B r , Initial teacher network π ω t , Training steps N , Warm-up steps k , Soft update τ , hyperparameters: λ, α

̸

D Environments and Domain Knowledge Trees 492

(e

b)

c)

0

25

50

75

100

125

150

175

0

50

100

150

d)

(a

The graphical visualization of each environment is depicted in Fig 9. The choice of environment in this paper 493 depended on two factors: a) Pre-existing standard methods of generating offline RL datasets. b) Possibility of 494 creating intuitive decision tree-based domain knowledge. All datasets have been created via [31]. We explain the 495 environments in detail as follows: 496

Mountain-car Environment: This environment Fig 9 a) has two state variables, position and velocity, and three 497 discrete actions: left push, right push, and no action [27]. The goal is to drive a car up a valley to reach the flag. 498 This environment is challenging for offline RL because of sparse rewards, which are only obtained on reaching 499 the flag. 500

501

502

Cart-pole Environment The environment Fig 9 b) has 4 states and 2 actions representing left force and right force. The objective is to balance a pole on a moving cart.

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

Lunar-Lander Environment: The task is to land a lunar rover between two flags Fig 9 c) by observing 8 states and applying one of 4 actions.

Minigrid Environments: Mini-grid [6] is an environment suite containing 2D grid-worlds with goal oriented tasks. As explained in the main text, we experiment using MiniGrid-LavaGapS7-v0 and MiniGrid-DynamicObstacles-Random-6x6-v0 from this environment suite is shown in Fig 9 d) and e). In MiniGrid-LavaGapS7-v0, the agent has to avoid Lava and pass through the gap to reach the goal. Dynamic obstacles are similar; however, the agent can start at a random position and has to avoid dynamically moving balls to reach the goal. The environment has image observation with 3 channels (OBJECT_ID, COLOR_ID, STATE). Following [31] experiments, we flatten the image to an array of 98 observations and restrict action space to three actions: Turn left, Turn Right, and Move forward. The results of minigrid environment are reported in Table 3. Since this environment uses a semantic map from image observation, we collect states from a fixed policy with random actions to generate the teacher's state distribution. CQL on the full dataset achieves the average reward of 0 . 92 ± 0 . 1 for DynamicObstacles and 0 . 53 ± 0 . 01 for LavaGapS.

The domain knowledge trees for all the environments are shown in Fig 10. The cart pole domain knowledge tree Fig 10 a) is taken from [33] (Fig 7). The Lunar Lander decision nodes Fig 10 b) have been taken from [34] (Fig4). For the mini-grid environments, we construct intuitive decision trees shown in Fig 10 d) and Fig 10 e). Positions 52, 40, and 68 represent positions front, right, and left of the agent. Value 0.2 represents a wall, 0.9 represents Lava, and 0.6 represents a ball. We check positions 52, 40, and 68 for these obstacles and choose the recommended actions as domain knowledge.

Angular velocity >0.04

Position>-1

Position<1

Pole Angle<0

a = Tsub(s)

Y velocity >-0.35

Pole Angle<0

PoleAngle<0

Velocity<0

GoLeft

a = TIsub(s)

Angular velocity > 0.22

GoLeft

GoRight

GoLeft

Angular Vel.>0

Go Right

Velocity>0

GoRight

GoLeft

a = Tsub(s)

Angle> -0.04

Angular Vel.<0

Pole Angle <0

a = TTsub(s)

random

GoLeft

GoRight

GoLeft

GoRight

(a)

(b)

MountainCar

Minigrid lavagaps

Minigrid dynamic obstacles

Ball or wallin front

Lava or wall in front

Pos<-0.5

not Ball or wall in left

not Lava or wallin left

forward

forward

Vel < -0.01

left

not Lava or wall in

right

left

not Ball or wall in right

Left

Right

Right

Random

random

right

random

right

(c)

(d)

(e)

E Related Work: Knowledge Distillation

Knowledge distillation is a well-embraced technique of incorporating additional information in neural networks 523 and has been applied to various fields like computer vision [43, 36], natural language processing [8, 38], and 524 recommendation systems [37]. [17] introduced the concept of distilling knowledge from a complex, pre-trained 525 model (teacher) into a smaller model (student). In recent years, researchers have explored the integration 526 of rule-based regularization techniques within the context of knowledge distillation. Rule regularization 527 introduces additional constraints based on predefined rules, guiding the learning process of the student model 528

[18, 45]. These techniques have shown to reduce overfitting and enhance generalization [38]. Knowledge 529 distillation is also prevalent in the field of RL [47] and offline RL [39]. Contrary to prevalent teacher-student 530 knowledge distillation techniques, our work does not enforce parameter sharing among the networks. Through 531 experiments, we demonstrate that a simple regularization loss and expected performance-based updates can 532 improve generalization to unobserved states covered by domain knowledge. There are also no constraints on 533 keeping the same network structure for the teacher, paving ways for capturing the domain knowledge into more 534 structured networks such as Differentiable Decision Trees (DDTs). 535

536

537

538

539

540

541

542

543

544

545

546

F Network Architecture and Hyper-parameters

We follow the network architecture and hyper-parameters proposed by [31] for all our networks, including the baseline networks. The teacher BC network π t ω and Critic network Q θ s ( s, a ) consists of 3 linear layers, each having a hidden size of 256 neurons. The number of input and output neurons depends on the environment's state and action size. All layers except the last are SELU activation functions; the final layer uses linear activation. π t ω uses a softmax activation function in the last layer for producing action probabilities. A learning rate of 0.0001 with batch size 32 and α = 0 . 1 is used for all environments. MC dropout probability of 0.5 and number of stochastic passes T=10 have been used for the critic network. The uncertainty check is performed every 15 episodes after the warm start to avoid computational overhead. The hyper-parameters specific to our algorithm for OpenAI gym are reported in Table F. The hyper-parameters specific to our algorithm for Minigrid environments are reported in Table 5.

lambda0.1-lambda0-lamda0.8-lambda0.5

-120

Reward

200

Reward

Reward

400

100

-140

300

0

-160

200

-100

-180

100

-200

Episodes

-200

Episodes

0

Episodes

-300

0

50

100

150

200

250

0

50

100

150

200

250

0

50

100

150

MountainCarenvironment

Cartpoleenvironment

LunarLanderenvironment

0.6

Reward

0.5

0.5

0.4

U

0.3

0.2

0.5

0.1

Episodes

0

Episodes

50

100

150

0

50

100

150

MiniGrid-Dynamic-Obstacles-Random-6x6-v0

MiniGrid-LavaGapS7-v0

G Effect of k and λ and Evaluation Plots 547

We empirically evaluate the effect of λ In Fig 11 and k in Fig 12. We believe these parameters depend on the 548 quality of D . For the given D in the environments we empirically observe, λ = 0 . 5 generally performs well, 549 except for Minigrid environments where λ = 0 . 1 works better. Increasing the warm start parameter k generally 550 increases the initial performance of the policy, allowing it to learn from the teacher. Meanwhile, no warm start 551 adversely affects policy performance as the critic may erroneously update the teacher. From empirical evaluation, 552 we observe that k = 30 gives a reasonable start to the policy. All the evaluation plots are shown in Fig 13, where 553 it can be observed that ExID performs better than baseline CQL. 554

-100

Reward

Reward

200

-120

400

100

-140

300

0

-160

200

-100

-180

-200

-200

100

Episodes

Episodes

-300

Episodes

0

50

100

150

200

250

0

0

50

100

150

0

50

100

150

200

250

Mountaincarenvironment

Cartpoleenvironment

LunarLanderenvironment

Reward

0.6

Reward

0.5

0.5

0.4

0.3

-0.5

0.2

-1

0.1

Episodes

Episodes

0

50

100

150

0

0

50

100

150

MiniGrid-Dynamic-Obstacles-Random-6x6-v0

MiniGrid-LavaGapS7-v0

H Data reduction design and data distribution visualization of reduced 555 dataset 556

In this section, we discuss the intuition behind our data-limiting choices. We also visually represent selected 557 reduced datasets for the OpenAI gym environments. 558

-EXID=CQL

Cartpole Expert

100

150

0.6

0.5

0.4

0.3

0.2

0.5

0.1

Episodes

Episodes

50

100

150

0

50

100

150

MiniGrid-LavaGapS7-v0

MiniGrid-Dynamic-Obstacles-Random-6x6-v0

CQL-MC-0.1EX>-0.8

CQL-MC-EXFull>-0.8

velocity>0.01

vlocity<0.01

position<-0.5

CQL-MC-0.1EX

CQL-MC-FulIEX

一

position>-0.5

CQLMC-EX0.05

-100

-100

Reward

Reward

-120

-120

-140

-140

-160

-160

-180

-180

-200

Episodes

Episodes

-200

0

50

100

150

200

250

0

50

100

150

200

250

(a)

(b)

Reducing transitions from the dataset: For all datasets, 10% of the data samples were extracted from the full 559 dataset. This experimental design choice is based on the observation shown in Fig 14 (a). Performance degrades 560 on reducing samples to 0.1% of the dataset and reduces further on reducing samples to 0.05% of the dataset. 561 However, this drop is not substantial. The performance also reduces on removing part of the dataset from the 562 full dataset with states > -0 . 8 . However, the worst performance is observed when both samples are reduced 563 and data is omitted, attributing to accumulated errors from probability ratio shift contributing to an increase in 564 generalization error. Our methodology aims to address this gap in performance. 565

Removing part of the state space: Due to the simplicity of the Mountain-Car environment, we analyze the 566 Mountain-Car expert dataset to show the effect of removing data matching state conditions of the different nodes 567 in the decision tree in Fig 10 (c). The performance for each condition is summarised in Table 6. The most 568 informative node in the tree is position > -0 . 5 ; removing states matching this condition causes a performance 569 drop in the algorithm as the domain knowledge regularization does not contribute significant information to the 570 policy. Similarly, removing data with velocity < 0 . 01 causes a performance drop. However, both performances 571 are higher than the baseline CQL trained on reduced data. Based on this observation, we choose state removal 572 conditions that preserve states matching part of the information in the tree such that the regularization term 573 contributes substantially to the policy. Fig 15 shows the data distribution plot of 10% samples extracted from 574 mountain car replay and noisy data with states > -0 . 8 removed. Fig 16 shows visualizations for 10% samples 575

extracted from expert data with velocity > -1 . 5 removed. Fig 17 shows visualizations for 10% samples 576 extracted from expert data with lander angle < -0 . 04 removed. 577

MountainCarReplaydataset

MountainCarNoisyDataset

3D Scatter Plot - Full Dataset

3D Scatter Plot - 10% Samples Replay Dataset Mountain-Car V0

3D Scatter Plot - Full Dataset

3D Scatter Plot - 10% Samples Noisy Dataset Mountain-Car V0

Full Dataset

10,000 Samples

Full Dataset

。10,000 Samples

2.00

2.00

2.00

2.00

0.75

125

1.00

0.50

0.25

0.00

1.75

1.50

0.75

1.00

0.50

0.25

1.50

1.25

1.75

0.00

0.50

1.00

0.75

1.50

1.25

000

0.25

1.75

0.25

0.50

1.00

0.75

1.50

1.25

0.00

1.75

1.25

0.50

Position

0.25

0.00

0.25

0.50

0.06

0.04

0.02

0.00

0.02

0.04

0.06

-0.8

0.6

0.4

Pesition

0.2

0.0

0.2

0.4

90°0

0.04

0.02,

0.02

0.04

0.06

1.25

0.50

Position

0.25

0.00

0.25

0.50

0.06

0.04

-0.02

0.00

0.02

0.04

0.06

8°0-

0.6

0.4

Position

-0.2

0.0

0.2

0.4

0.06

0.04

0.02

0.00

0.02

0.04

0.06

Full Dataset - Distribution - Dimension 1

Full Dataset - Distribution - Dimension 2

Full Dataset - Distribution - Dimension 3

Full Dataset - Distribution - Dimension 4

7000

3500

12000

6000

8000

3000

10000

5000

2500

6000

8000

Frequency

Kouanbay

4000

2000

Frequency

nbay

6000

3000

4000

1500

4000

2000

1000

2000

2000

1000

500

-2

-1

Dimension 1

Dimension 2

0.2

0.1

uojsuaia

0.0

0.1

0.2

2

Dimension 4

10%Reduced CartpoleExpert dataset

2500

600

008

500

800

2000

009

400

600

1500

Frequency

fouanbay

300

1000

400

400

200

500

0

200

100

-1

Dimension 1

0

Dimension 2

0.2

0.1

Dimension 3

0.0

0.1

0.2

-2

-1

0

Dimension 4

Full LunarLander Expert Data (100K)

12000

40000

60000

20000

60000

30000

10000

35000

17500

5000

50000

25000

30000

15000

8000

40000

12500

20000

25000

40000

Kou

Frequency

Frequency

Kouan

Frequency

6000

15000

20000

30000

30000

4000

7500

10000

15000

20000

20000

5000

10000

2000

5000

10000

10000

2500

5000

1.0

0.5

Dimension 1

0.0

0.5

0

0.0

Dimension 2

0.5

1.0

1.5

2

Dimension 3

-1

0

~

Dimension 4

-1

0

Dimension 5

0

2

2

Dimension 6

0

2

10%Sampled LunarLanderExpertData

1600

3000

3000

3500

5000

6000

1400

2500

2500

3000

1200

4000

5000

2500

2000

2000

1000

4000

uency

2000

Frequency

ba

800

a

000E

1500

600

2000

1000

1000

2000

1000

400

1000

200

500

500

500

1000

1.0

0.5

0.0

0.5

0.0

0.5

10

1.5

2

1

0

2

-1

0

0

2

0

2

0

Dimension 1

Dimension 2

Dimension 3

Dimension 4

Dimension 5

Dimension 6

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the paper have been experimented on different settings for validity and generalization. Please refer to sec 5.2.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper acknowledges the dependency on reasonable domain knowledge and coverage please refer to sec 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Please refer to App A in the supplement material for theoretical analysis and proofs. Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Yes all hyper-parameters and experimental setting have been clearly listed in the paper. Please refer to App F and sec 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code is provided with the submission in a zip file with Readme for instructions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

692

693

694

695

696

697

698

699

700

701

702

703

704

705

706

707

708

709

710

711

712

713

714

715

716

717

718

719

720

721

722

723

724

725

726

727

728

729

730

731

732

733

734

735

736

737

738

739

740

741

742

743

744

745

746

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper uses open source code to create the dataset and lists the modifications in details in the main paper and supplement material. Please refer to App F and sec 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: All experiments have been run on 3 random seeds and the error bounds have been reported in Table 5.1, Table 2 and Table 3.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Please refer to the Experimental setup section in the main paper sec 5.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the code of ethics and the paper adheres to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: Please refer to the section broader impacts 7.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The algorithm proposed in this paper does not not pose any such risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All codes and datasets used in this paper are under MIT licence and the original owners have been cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: No new assets have been introduced in this paper.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper did not require any crowdsourcing or human subject for experimentation.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper did not require any crowdsourcing or human subject for experimentation.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ExID, an offline reinforcement learning algorithm that enhances learning performance in limited data scenarios by combining domain knowledge in the form of simple decision trees with agent experience replay data.\n\nSTRENGTHS:\n* Domain Knowledge Utilization: ExID incorporates domain knowledge to guide decision-making in data-limited scenarios\n* Teacher-Student Architecture: A teacher network, informed by domain knowledge, regularizes a student critic network to improve generalization.\n* Regularization with Domain Knowledge: The algorithm uses a regularization term to align the critic's decisions with the teacher's advice for states covered by domain knowledge.\n\nWEAKNESSES:\n* Discrete Action Space Limitation: The algorithm is currently limited to discrete action spaces, necessitating future extensions for continuous action domains.\n* Hyperparameter Tuning Challenge: The need for precise hyperparameter tuning complicates the deployment of ExID in scenarios where extensive optimization is impractical.\n* The paper does not have enough strong experiment comparisions. The methods of the paper is related with offline RL methods, such as SCQ[1], ReDS[2], A2PR[3], CPED[4]. But it lacks the experiments comparisions with offlien RL methods. I think adding some SOTA baseline methods will improve your paper. It is not required that experimental comparisons must be given, but at least add some discussion with these methods to the paper.  \n\nReferences：\n\n[1] Shimizu, Yutaka, et al. \"Strategically Conservative Q-Learning.\" arXiv preprint arXiv:2406.04534 (2024).\n\n[2] Singh, Anikait, et al. \"ReDS: offline reinforcement learning with heteroskedastic datasets via support constraints.\" Proceedings of the 37th International Conference on Neural Information Processing Systems. 2023.\n\n[3] Liu, Tenglong, et al. \"Adaptive Advantage-Guided Policy Regularization for Offline Reinforcement Learning.\" In International Conference on Machine Learning (ICML). PMLR, 2024.\n\n[4] Zhang, Jing, et al. \"Constrained policy optimization with explicit behavior density for offline reinforcement learning.\" Advances in Neural Information Processing Systems. 2023\n\nQUESTIONS:\n* How does ExID perform when the true optimal policy deviates significantly from the provided domain knowledge? Can the algorithm recover from incorrect domain knowledge without additional corrective mechanisms?\n* What measures are in place to ensure robustness when the domain knowledge contains errors or biases? How sensitive is the algorithm to such inaccuracies?\n* The paper mentions a real sales promotion dataset, but how does ExID perform in other real-world scenarios? Are there any plans for more extensive real-world testing to validate the algorithm's practical utility?\n* How should practitioners select the hyperparameters $\\lambda$ and $k$ in the absence of extensive computational resources?\n\nLIMITATIONS:\n* The paper only conducts experiments in several simulated environments and a real-world sales promotion dataset, which may not fully verify the effectiveness and applicability of the algorithm in more diverse and complex real-world scenarios.\n* The performance of the ExID algorithm heavily relies on the quality of the domain knowledge. If the domain knowledge is incomplete, inaccurate, or biased, it may mislead the learning process and result in suboptimal policies. Moreover, obtaining high-quality domain knowledge can be challenging and time-consuming in practice.\n* The proposed method mainly concentrates on discrete action spaces, and its performance and applicability in continuous action spaces are not clear. This limits the algorithm's utility in many real-world control tasks that involve continuous action spaces.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies offline RL when data is limited. The authors propose a domain knowledge-based regularization technique to learn from an initial tracker network and limited data buffer. The experiments verified the effectiveness of the proposal, which outperforms the classic RL baseline methods.\n\nSTRENGTHS:\n1. The proposed method is simple and technically reasonable.\n2. The experimental results on the real sales promotion dataset show the proposal is a promising solution in real-world applications.\n\nWEAKNESSES:\n1. The technical novelty is limited. Despite the claimed use of expert knowledge, the method adopted by the paper is to directly train a policy from the knowledge, which assumes that the information provided by the domain knowledge is at the state-action level (a decision tree in this paper), which limits the feasibility of this method. Compared to the use of knowledge between latent concepts discussed in neuro-symbolic learning, I think it's more like traditional model distillation. \n2. In practice, limited offline data may come from domain knowledge-based strategies, such as human-designed rules, thus I have great concerns about whether these two can promote each other. Empirical studies on more real-world datasets or rigorous theoretical analysis will provide support to this issue and further improve this work.  \n3. The introduction uses the sales task as an example, but the visualization is based on the Mountain Car dataset.\n4. Definition 4.1 seems strange, why not directly define the offline dataset as a subset of the complete state spaces?\n5. The $\\eta$ in Proposition 4.2 is not well defined.\n\nQUESTIONS:\n1. Could the “Intuitive Expert Insights” be formally defined?\n2. The difference between the proposal and the traditional model distillation.\n\n\n---\n\nAfter rebuttal, my concerns have been addressed. I decide to raise the score to 6.\n\nLIMITATIONS:\nThe authors have provided a discussion about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel technique ExID, a domain knowledge-based regularization method, that adaptively refines initial domain knowledge to boost performance of offline reinforcement learning (RL) in limited-data scenarios. The key insight is leveraging a teacher policy, trained with domain knowledge, to guide the learning process of the offline-optimized RL agent (student policy). This mitigates the issue of erroneous actions in sparse samples and unobserved states by having the domain knowledge-induced teacher network to cover them. And the initial domain knowledge would be improved when the student policy reaches a better perform than the teacher policy.  Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase compared to traditional offline RL algorithms operating on limited data\n\nSTRENGTHS:\n1. Originality: The paper's originality lies in its integration of domain knowledge into offline RL through a teacher policy network. This approach addresses performance degradation in limited-data settings, which is a novel and underexplored area. The introduction of the domain knowledge-based regularization technique and adaptive refinement of initial domain knowledge are particularly innovative.\n\n2. Quality: The quality of the work is evidenced by the solid theoretical analysis and the thorough empirical evaluations conducted on multiple standard datasets, including OpenAI Gym environments (Mountain Car, Cart-Pole, Lunar Lander) and MiniGrid environments, as well as a real-world sales promotion dataset. The results consistently show that ExID outperforms existing offline RL algorithms in these settings.\n\n3. Clarity: The paper is well-structured, with clear explanations of the problem, methodology, and results. The use of diagrams and tables helps understand the motivation of the problem (figure 1), the proposed method (figure 2), illustrate the effectiveness of ExID (Table 1-2). Each section logically follows from the previous one, making the overall argument easy to follow.\n\n4. Significance: By tackling the challenge of limited data in offline RL, the paper makes a significant contribution to the field. The proposed approach has practical implications for various real-world applications where data is scarce and expert knowledge is available, such as in business, healthcare, and robotics.\n\nWEAKNESSES:\n1. Generalization to Continuous Domains: The paper is limited to discrete action spaces, which restricts its applicability to a broader range of RL problems involving continuous action spaces. This limitation is acknowledged by the authors.\n\n\n2. Scalability: The scalability of ExID to more complex environments that requires a complex representation (e.g., a significant large tree) of domain knowledge is not thoroughly explored.  It would be beneficial to understand how the method performs in such settings and what challenges might arise because the challenging of updating the domain knowledge represented in a complex representation could hinder the learning process of the student policy in the proposed method ExID.\n\nQUESTIONS:\n1. Comparative Analysis: Can the authors provide a more detailed comparison with other domain knowledge-based methods? Specifically, how does ExID differ in its approach to leveraging domain knowledge compared to methods like DKQ and CQL SE that \n are mentioned in Related Work?\n\n2. Examples of Domain Knowledge Improvement: having some examples of the improved domain knowledge could make the paper stronger and more convincing.\n\nLIMITATIONS:\nThe authors acknowledge several limitations of their work, including the reliance on the quality of domain knowledge and the focus on discrete action spaces. While these limitations are well-addressed in the paper, it may be worth to consider a broad evaluation:\n   * Conducting experiments on a wider variety of environments that have larger state and action spaces, would provide a more comprehensive evaluation of the method's applicability.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}