Title: How to Leverage Imperfect Demonstrations in Offline Imitation Learning
Venue: NeurIPS 2023
Accepted: no
Keywords: offline imitation learning

imperfect demonstration

imitation learning
Abstract: Offline imitation learning (IL) with imperfect data has garnered increasing attention due to the scarcity of expert data in many real-world domains. A fundamental problem in this scenario is how to extract good behaviors from noisy demonstrations. In general, current approaches to the problem build upon state-action similarity to the expert, neglecting the valuable information in (potentially abundant) diverse behaviors that deviate from given expert demonstrations. In this paper, we introduce a simple yet effective data selection method that identifies the positive behavior based on its "resultant state", which is a more informative criterion that enables explicit utilization of dynamics information and the extraction of both expert-like and beneficial diverse behaviors. Further, we devise a lightweight constrained behavior cloning algorithm capable of leveraging the expert and selected data correctly. We term our proposed method iLID and evaluate it on a suite of complex and high-dimensional offline IL benchmarks, including MuJoCo and Adroit tasks. The results demonstrate that iLID achieves state-of-the-art performance, significantly outperforming existing methods often by 2-5x while maintaining a comparable runtime to behavior cloning (BC).

Abstract

Offline imitation learning (IL) with imperfect data has garnered increasing attention due to the scarcity of expert data in many real-world domains. A fundamental problem in this scenario is how to extract good behaviors from noisy demonstrations . In general, current approaches to the problem build upon state-action similarity to the expert, neglecting the valuable information in (potentially abundant) diverse behaviors that deviate from given expert demonstrations. In this paper, we introduce a simple yet effective data selection method that identifies the positive behavior based on its resultant state , which is a more informative criterion that enables explicit utilization of dynamics information and the extraction of both expert-like and beneficial diverse behaviors. Further, we devise a lightweight constrained behavior cloning algorithm capable of leveraging the expert and selected data correctly. We term our proposed method iLID and evaluate it on a suite of complex and high-dimensional offline IL benchmarks, including MuJoCo and Adroit tasks. The results demonstrate that iLID achieves state-of-the-art performance, significantly outperforming existing methods often by 2-5x while maintaining a comparable runtime to behavior cloning (BC).

1 Introduction

Offline imitation learning (IL) is the study of learning from demonstrated behaviors without reinforcement signals or further interaction with the environment. It has been deemed as a promising solution for safety-sensitive applications, such as autonomous driving and healthcare, where manually identifying a reward function is difficult but historical human demonstrations are readily available. Traditionally, offline IL methods such as behavior cloning (BC) (Pomerleau, 1988) often require an expert dataset with sufficient coverage over state-action spaces to combat error compounding (Ross and Bagnell, 2010; Jarrett et al., 2020; Chan and van der Schaar, 2021), which can be prohibitively expensive for many real-world domains. Instead, a more realistic scenario might allow for a small expert dataset, combined with a large amount of imperfect data sampled from unknown policies (Wu et al., 2019; Xu et al., 2022a; Yu et al., 2022). For example, autonomous vehicle companies may have limited high-quality data from experienced drivers but can obtain a wealth of mixed-quality data from ordinary drivers. Clearly, effective utilization of these imperfect demonstrations would significantly enhance the robustness and generalization of offline IL.

A fundamental question raised in this scenario is: how can we extract good behaviors from noisy 31 data? To answer this question, several prior works have attempted to explore and imitate the imperfect 32 behaviors that resemble expert ones (as in Xu et al. (2022a); Sasaki and Yamashina (2020)). However, 33 due to the scarcity of expert data, such methods are ill-equipped to leverage valuable knowledge 34 in (potentially abundant) diverse behaviors that deviate from limited expert demonstrations. Of 35 course, a natural solution to incorporate these behaviors is inferring a reward function and labeling 36

Expert behaviors

Imperfect behaviors

Beneficial behaviors

A state unobserved

from expert data

Leading to an

expert state

all imperfect data, followed by an offline reinforcement learning (RL) progress (as in Zolna et al. 37 (2020); Chang et al. (2022); Yue et al. (2023)). Unfortunately, it is highly challenging to define and 38 learn meaningful reward functions without environmental interaction. As a result, current offline 39 reward learning methods typically rely on complex adversarial optimization using a learned dynamics 40 model. They easily suffer from hyperparameter sensitivity, learning instability, and limited scalability 41 in high-dimensional environments (Yu et al., 2022; Arjovsky et al., 2017; Garg et al., 2021). 42

In this paper, we introduce a simpler data selection method along with a lightweight policy learning 43 algorithm to fully exploit both expert-like and positive diverse behaviors in imperfect demonstrations 44 without indirect reward learning procedures. Specifically, instead of examining a behavior's similarity 45 to expert demonstrations in and of itself, we assess its value based on whether its resultant state , to 46 which environment transitions after performing that behavior, falls within the expert data manifold. 47 In other words, we (properly) select the state-actions that can lead to expert states, even if they bear 48 no similarity to expert demonstrations. As illustrated in Fig. 1 and supported by the theoretical results 49 in Section 3.1, the underlying rationale is: when the agent encounters a state unobserved in expert 50 demonstrations, compared to taking a random action, a more reasonable way is to return to the states 51 where it knows expert behaviors; otherwise, it may keep making mistakes and remain out-of-expert52 distribution for the remainder of time steps. Notably, the resultant state is more informative than 53 the state-action similarity, as it can explicitly utilize the dynamics information and identify both 54 expert-like and beneficial diverse state-actions. 55

56

57

58

59

60

61

62

63

64

65

Drawing on this insight, we first train a state-only discriminator to distinguish expert and non-expert states in imperfect demonstrations. Based on the identified expert-like states, we appropriately select their causal state-actions and build a complementary training dataset. In light of the suboptimality of the complementary data, we further devise a lightweight constrained BC algorithm to mitigate the potential interference among behaviors. We term our proposed method Offline Imitation Learning with Imperfect Demonstrations (iLID) and evaluate it on a suite of offline IL benchmarks, including widely-used MuJoCo tasks as well as more complex and high-dimensional Adroit tasks. iLID achieves state-of-the-art performance, significantly outperforming existing baseline methods often by 2-5x while maintaining a comparable runtime to BC. In a nutshell, the main contributions of this paper are as follows:

66

67

68

69

70

71

72

We introduce a simple yet effective method to select potentially useful behaviors in noisy data. It can explicitly exploit the dynamics information and extract both expert-like and positive diverse behaviors, achieving a significant improvement in the utilization of imperfect demonstrations.

To avoid behavior interference induced by the suboptimality of complementary behaviors, we propose a constrained BC algorithm that can correctly leverage the expert and extracted behaviors.

Extensive experiments on complex and high-dimensional domains corroborate that iLID can surpass the existing baseline methods in terms of performance and computational cost.

2 Preliminaries 73

Episodic Markov decision process (MDP) can be specified by M . = 〈 S , A , T, H, r, µ 〉 , consisting of 74 state space S , action space A , transition dynamics T : S × A → P ( S ) , episode horizon H , reward 75

function r : S × A → [0 , 1] , and initial state distribution µ : S → [0 , 1] . A stationary stochastic 76 policy maps states to distributions over actions, denoted as π : S → P ( A ) . The policy value of π 77 is defined as the expected cumulative reward, V π . = E [ ∑ H h =1 r ( s h , a h )] , where the expectation is 78 computed w.r.t. the distribution over trajectories induced by rolling out π in the environment. The 79 objective of reinforcement learning (RL) can be expressed as max π ∈ Π V π , where Π is the set of 80 all stationary stochastic policies taking actions in A given states in S . We denote the average state 81 distribution of policy π as ρ π ( s ) . = 1 H ∑ H h =1 Pr( s h = s | π , T, µ ) , where Pr( s h = s | π , T, µ ) denotes 82 the probability of visiting s at time step h by rolling out π with M . When clear from context, we 83 overload notation and denote the average state-action distribution as ρ π ( s, a ) . = ρ π ( s ) π ( a | s ) . 84

85

86

87

88

89

90

91

Offline IL with imperfect demonstrations is the setting where the algorithm is neither allowed to interact with the environment nor provided ground-truth reward signals. Rather, it has access to an expert dataset and a mix-quality imperfect/noisy dataset, collected from unknown expert policy π e and (perhaps highly suboptimal) behavior policy π s , respectively. To be specific, the expert and imperfect datasets are denoted by D e . = { τ j } n e j =1 and D s . = { τ i } n s i =1 , where τ i . = ( s i, 1 , a i, 1 , . . . , s i,H , a i,H ) represents a trajectory. Our goal is to learn the best policy with regard to optimizing V π from static offline data D o . = D e ∪ D s without querying the expert or interacting with the environment.

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

Behavior cloning (BC) is a classical offline IL approach, which seeks to learn a policy via supervised learning. The standard objective of BC is to maximize the negative log-likelihood over D e :

However, standard BC does not utilize the information in D s . Due to the limited state coverage of D e , the learned policy may suffer from severe compounding errors, i.e., the inability for the policy to get back on track if it encounters a state not seen in the expert demonstrations.

3 Offline imitation learning with imperfect demonstrations

In this section, we provide a detailed description of our approach. We begin by presenting the theoretical findings on the benefits of utilizing diverse transitions. Building on the theoretical insights, we then design our data selection and policy learning methods.

3.1 How to extract good behaviors from noisy data

To discard low-quality demonstrations from D s , existing approaches often rely on the state-action dissimilarity between D s and D e . For example, Xu et al. (2022a); Zolna et al. (2020); Kim et al. (2022) propose to learn a weighting function f ( s, a ) by pushing up its value on ( s, a ) ∈ D e while pushing down that on ( s, a ) ∈ D s . Based on f ( s, a ) , they perform weighed BC to implicitly select expert-like state-actions, i.e., max π ∈ Π E ( s,a ) ∼ D o [ f ( s, a ) log( π ( a | s ))] . However, due to the limitation of expert demonstrations, the learned f ( s, a ) can be overly conservative and neglect the useful information in diverse state-actions. Therefore, it calls for a more informative criterion to assess the value of imperfect behaviors.

Before preceding, we first provide the following theoretical results under deterministic transition dynamics to gain insights into this problem. Denote S h ( D ) as the set of h -step visited states of D and S ( D ) . = ⋃ H h =1 S h ( D ) all the states thereof. Assume that π e is optimal and deterministic, and there exists a supplementary dataset consisting of transition tuples from initial states to given expert states, i.e., ˜ D . = { ( s i , a i , s ′ i ) | s i ∼ µ, s ′ i ∼ S ( D e ) , T ( s i , a i ) = s ′ i , i = 1 , 2 , . . . , ˜ n } . Consider a policy ˜ π such that in expert states S ( D e ) , it takes the corresponding expert actions, and in states S 1 ( ˜ D ) \S 1 ( D e ) , it takes the actions in ˜ D , that is,

We bound the suboptimality gap and sample complexity of ˜ π in the next theorem and corollary. 117

Theorem 3.1. For any finite and episodic MDP with deterministic transition dynamics and µ = U ( S ) , 118 the following fact holds: 119

where /epsilon1 . = E [ E s 1 ∼ µ [ ( s 1 / ∈ S 1 ( D e ))]] and δ . = E [ E s 1 ∼ µ [ ( s 1 / ∈ S 1 ( ˜ D ))]] represent the missing mass over the initial distribution w.r.t. S 1 ( D e ) and S 1 ( ˜ D )) . U ( S ) is the uniform distribution over S .

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

Proof Sketch. Note that the error stems from the initial states that are not covered by S 1 ( D e ) . We bound the errors generated from the states not in S ( D e ) ∪ S ( ˜ D ) and from the states in S ( ˜ D ) \S ( D e ) by H δ/epsilon1 and ( H/ 2 + 1 /H )(1 -δ ) /epsilon1 , respectively. Combining these two errors yields the result. For a detailed proof, please refer to Appendix B.

Building on Theorem 3.1, we can obtain the following sample complexity result (where we retain the constant 1 2 in the asymptotic result to highlight the improvement over BC).

Corollary 3.2. Suppose ˜ D is sufficiently large. For any finite and episodic MDP with deterministic transition dynamics and µ = U ( S ) , to obtain an ε -optimal policy, V π e -E [ V ˜ π ] ≤ ε , ˜ π requires at most O ( |S| H/ (2 · ε )) expert trajectories.

Proof. Invoking Xu et al. (2021, Theorem 2) yields the bounds for the missing mass:

where e is the Euler's number. If ˜ D is sufficiently large, then δ → 0 . Using Theorem 3.1, the result can be easily derived.

Remarks. It is worth noting that the minimax suboptimality of BC is limited to H /epsilon1 in this setting (Rajaraman et al., 2020), and beating the O ( H ) barrier is unattainable. The reason is that when the agent encounters a state beyond given demonstrations during the interaction with the environment, it has no prior knowledge about the expert. As a result, the agent is essentially forced to take an arbitrary action in these states, potentially leading to mistakes for H time steps. Whereas, as revealed by Theorem 3.1 and Corollary 3.2, ˜ π provably alleviates the error compounding and reduces the sample complexity bound of BC (which is O ( | S | H/ ε ) ) by approximately half. The reason behind is that ˜ D can empower ˜ π to recover from mistakes. Combined with Eq. (2), this provides an important insight for us: in the states uncovered by D e , if an action can lead to known expert states, mimicking it can benefit the performance of imitation policy.

Thus motivated, we propose to assess the imperfect behavior based on its resultant states rather than the state-action in and of itself. For example, if there exists ( s 1 , a 1 , s 2 , a 2 , s 3 ) ∈ D s such that s ′ ∈ D e , one can select ( s 1 , a 1 ) and ( s 2 , a 2 ) (or only ( s 2 , a 2 ) ), even if these behaviors do not bear similarity to any ( s, a ) ∈ D e . To this end, we consider learning a state-only discriminator to contrast expert and non-expert states in D s , e.g.,

However, optimizing Problem (4) can lead to the problem of false negative , where the learned discriminator assigns 1 to all transitions from D e and 0 to all transitions from D s . This problem is analogous to the positive-unlabeled (PU) classification problem (Elkan and Noto, 2008), where both positive (expert) and negative (imperfect) samples exist in the unlabeled data (imperfect demonstrations). Akin to Xu et al. (2022a); Zolna et al. (2020), we adopt the reweighting method from PU learning to address this issue:

where η > 0 is a reweighting parameter, corresponding to the proportion of expert states to imperfect 155 states. Intuitively, the third term in Eq. (5) could avoid d ∗ ( s ) of the states from S ( D s ) but similar to 156 S ( D e ) becoming 0. 157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

An imperfect traj.

· · ·

· · ·

Selected causal state-actions

Rollback steps: 2

s

h

-

3

a

h

-

3

s

h

s

h

-

2

a

h

-

2

s

h

-

1

a

h

-

1

a

h

An imperfect traj.

Data selection. The learned discriminator d ∗ is able to identify the expert states in D s . Based on these states, we in turn select their causal states and actions to construct a complementary dataset ˜ D . Specifically, given threshold σ ∈ [0 , 1] and rollback steps K ≥ 1 , if there exist h > 1 and i ∈ { 1 , . . . , n s } (where n s represents the number of trajectories in D s ) such that d ( s i,h ) ≥ σ , we include K causal state-action pairs from s i,h into ˜ D .:

We iterate the above process for all identified expert-like states. To clarify, we illustrate the process in Fig. 2. It is evident that ˜ D comprises of both the positive diverse state-actions in D s and those similar to D e therein. This highlights that using resultant states is a more informative way to extract useful behaviors.

/negationslash

Behavior interference. After obtaining ˜ D , a natural solution to learn an imitation policy is carrying out BC from the union of D e and ˜ D . However, due to the suboptimality of ˜ D , this naïve solution will suffer from potential interference among behaviors. That is, for a selected ( s, a, s ′ ) , if s, s ′ ∈ D e but a = π e ( s ) , action a will affect mimicking the expert behavior in expert state s when learning via the naïve solution. Furthermore, this interference issue also exists in the states of complementary dataset ˜ D , but in a more subtle manner. Consider a state s ∈ ˜ D where two actions a 1 , a 2 are selected, i.e., ( k 1 , s, a 1 ) , ( k 2 , s, a 2 ) ∈ ˜ D . Owing to the stochasticity of MDPs, if k 1 < k 2 , one may prefer a 1 over a 2 , whereas the naïve solution will imitate both actions equally. In Section 3.2, we address this problem and propose a lightweight algorithm to correctly learn from D e and ˜ D .

3.2 How to learn an imitation policy from expert and extracted data

Due to the suboptimality of D s and the stochasticity of MDPs, direct cloning the behaviors in ˜ D ∪ D e can lead to the interference issue. In fact, the solution has been implied in Eq. (2), which suggests that the policy should be constrained to D e ( ·| s ) in the known expert states. Accordingly, we cast the problem of learning policy from D e and ˜ D as follows:

where ˜ π e = arg max π ∈ Π E ( s,a ) ∼ D e [log( π ( a | s ))] is the BC policy learned on D e , and /epsilon1 ≥ 0 is the threshold. In Eq. (7), we use discount factor γ ∈ (0 , 1] to mitigate the impact of stochasticity of MDPs. It is easy to see that with a sufficiently small /epsilon1 , the optimal solution of Problem (7) enjoys at least the same theoretical guarantee of BC in general stochastic MDPs, i.e., suboptimality upper-bound O ( |S| H 2 log n e /n e ) compared to V π e (Rajaraman et al., 2020).

Problem (7) is a convex optimization problem. From Slater's condition, the strong duality holds, and thus the optimization is equal to

where α is the dual variable, and ˜ H is the expected entropy of the empirical expert policy, which is 188 derived from: 189

191

Algorithm 1: Offline Imitation Learning with Imperfect Demonstrations (iLID)

Require: expert data D e , imperfect data D s , learning rate λ , parameter K , /epsilon1

1 Train discriminator d using D e and D s based on Eq. (5);

2 Select data from D s and build complementary dataset ˜ D based on Eq. (6);

3 Train BC policy ˜ π e only using D e and compute expected entropy ˜ H . = -E s,a ∼ D e [log ˜ π e ( a | s )] ;

4 Initialize policy π θ and dual variable α ;

5 while not done do

6 Sample a training batch from D e and ˜ D ;

7 Update policy parameter θ ← θ -λ ˜ ∇ L ( θ ) based on Eq. (10);

8 Update dual variable α ← α -λ ˜ ∇ L ( α ) based on Eq. (11);

9 end while

Parameterize the learned policy by θ and denote the loss functions for θ and α as follows: 190

Problem (8) can be optimized by approximating dual gradient descent that alternates between the gradient steps w.r.t. L ( θ ) and L ( α ) , which has been shown to converge under convexity assumptions (Boyd and Vandenberghe, 2004) and work very well in the case of nonlinear function approximators such as neural networks (Haarnoja et al., 2018).

Our algorithm, named Offline Imitation Learning with Imperfect Demonstrations (iLID) , is outlined in Algorithm 1. Notably, while iLID pretrains a discriminator and a BC policy (using D e ), the progress can converge within a small number of gradient steps, especially when D e is limited. In light of the negligible cost in updating α , iLID is indeed computationally cheap.

4 Experiments

In this section, we use experimental studies to test the proposed method and answer the following questions: 1) Can iLID effectively utilize imperfect demonstrations? 2) What is the convergence properity of iLID? 3) How does iLID perform given different numbers of expert demonstrations or different qualities of imperfect demonstrations? 4) What is the impact of the rollback steps? 5) What is the runtime of iLID? 6) Is the constrained BC an overkill?

Baselines. We evaluate our method against five strong baseline methods in the offline IL setting: 1) Behavior Cloning with Expert Data (BCE) , the standard BC trained only on the expert dataset (Pomerleau, 1988); 2) Behavior Cloning with Union Data (BCU) , BC on both the expert and diverse datasets; 3) Discriminator-Weighted Behavioral Cloning (DWBC) (Chang et al., 2022), a recent offline IL algorithm capable of leveraging suboptimal demonstrations; 4) Using Imperfect Demonstration via Stationary Distribution Correction Estimation (DemoDICE) (Kim et al., 2022), another recent offline IL algorithm that can leverage suboptimal demonstrations; 5) Conservative offLine model-bAsed Reward lEarning (CLARE) (Yue et al., 2023), a recent model-based offline inverse RL algorithm trained from both expert and imperfect datasets.

Datasets. We conduct experiments on both widely-used MuJoCo tasks (including HalfCheetah, Walker2d, Hopper, and Ant) and more complex and highdimensional Adroit tasks (including Pen, Hammer, Relocate, and Door, shown on the right). We use the D4RL datasets (Fu et al., 2020) and utilize the random and expert data for each MuJoCo task, and cloned and expert data for Adroit tasks. 1 Similar to Xu et al. (2022a); Kim et al. (2022), we generate D e and D s as follows:

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

Expert datasets: For MuJoCo, we sample 1 trajectory (including less than 1000 state-action pairs) from the expert D4RL dataset to constitute expert datasets. For Adroit, we sample 10 expert trajectories (each includes less than 100 state-action pairs) to form the datasets.

Imperfect datasets: For MuJoCo tasks, we sample 1000 random trajectories mixed with 10 and 20 expert trajectories to constitute the low-quality and high-quality imperfect datasets. Regarding Adroit tasks, we sample 1000 cloned trajectories mixed with 100 and 200 expert trajectories to constitute the low-quality and high-quality datasets.

Comparative results. To answer the first and second questions, we show the comparative results under both low-quality and high-quality imperfect data in Section 4 and their corresponding learning curves in Fig. 4. iLID outperforms baseline algorithms on most of the tasks (13 out of 16) often by a wide margin and reaches near-expert scores on many tasks. It indicates that iLID can effectively extract and leverage positive behaviors from imperfect demonstrations over the approaches based on state-action similarity such as DWBC and DemoDICE. Unsurprisingly, BCE fails to fulfill most of the tasks, while BCU learns a mediocre policy. CLARE also performs poorly because the learned reward function could become too pessimistic due to the scarcity of expert demonstrations. Clearly, the model-based approach struggle in high-dimensional environments.

Expert demonstrations. To answer the third question, we vary the numbers of expert trajectories from 1 to 50 and present the results on Fig. 3(a). iLID reaches the expert with sufficient expert data.

(a) Expert demonstrations.

(b) Rollback steps.

(c) Ablation Study.

Albeit with very limited expert trajectories, iLID also achieves strong performance, revealing its 239 advantages in extracting good behaviors. DemoDICE performs relatively poorly with larger n e . The 240 reason is that it learns on both expert and random data, whereas the random data of HalfCheetah is 241 highly suboptimal. 242

243

244

245

246

247

248

249

250

Rollback steps. To answer the fourth question, we vary the rollback steps from 1 to 20 and show the corresponding results in Fig. 3(b). With larger K , the performance increases at the beginning. This is due to more positive diverse data included. An excessively large K may have a negative impact due to the dynamics stochasticity and behavior interference. However, it is worth noting that, as K increases further, the performance does not significantly deteriorate. This is because we apply a discount factor to penalize the potential uncertainty in the resulting states, capable of mitigating the issue. In practice, K can be treated as a hyper-parameter to tune. Intuitively, it can be set relatively smaller in a more stochastic environment.

251

252

253

254

Ablation study. We compare iLID to the naïve solution mentioned in Section 3.1, i.e., directly imitating the union of expert and select data. Fig. 3(c) validates the necessity of the constrained BC procedure. The result of direct imitation is passable as we select a number of positive data. However, it fails to deal with the behavior interference issue caused by the suboptimality of imperfect data.

Runtime. Weevaluate the runtime of iLID compared with baseline 255 algorithms for 250,000 training steps, utilizing the same network 256 size and batch size. We reproduce the reported results in Xu et al. 257 (2022a) on an NVIDIA V100 GPU. As illustrated by the figure 258 on the right, the runtime of iLID is nearly the same as BC. It 259 substantiates that the iLID is indeed a lightweight method. Due 260 to the cooperation training between the discriminator and policy, 261 DWBC requires additional computation than iLID. CLARE is 262 costly due to the effort to solve an intermediate offline RL problem. 263

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

5 Related work

Offline IL deals with training an agent to mimic the actions of a demonstrator in an entirely offline fashion. BC (Ross and Bagnell, 2010) is an intrinsically offline solution, but it is prone to covariate shift and inevitably suffers from error compounding, i.e., there is no way for the policy to learn how to recover if it deviates from the expert behavior to a state not seen in the expert demonstrations (Levine et al., 2020). Considerable research has been devoted to developing new offline IL methods to remedy this problem, e.g., Jarrett et al. (2020); Chan and van der Schaar (2021); Garg et al. (2021); Klein et al. (2011, 2012); Piot et al. (2014); Herman et al. (2016); Kostrikov et al. (2019); Swamy et al. (2021); Florence et al. (2022). However, since these methods imitate all given demonstrations, they often require a large amount of clean expert data, which can be expensive for real-world tasks.

Recently, there has been growing interest in exploring how to effectively leverage imperfect data in offline IL (Xu et al., 2022a; Yu et al., 2022; Sasaki and Yamashina, 2020; Kim et al., 2022). Sasaki and Yamashina (2020) analyze why the imitation policy trained by BC deteriorates its performance when using noisy demonstrations. They reuse an ensemble of policies learned from the previous iteration as the weight of the original BC objective to extract the expert behaviors. However, this requires that expert data occupies the majority proportion of the offline dataset, otherwise the policy will be misguided to imitate the suboptimal data. Kim et al. (2022) retrofit the BC objective with an additional KL-divergence term to regularize the learned policy to stay close to the behavior policy. Although it can implicitly extract the behaviors that bear similarity to the expert demonstrations, it easily fails to achieve satisfactory performance when the diverse data is highly suboptimal. Xu et al. (2022a) cope with this issue by introducing an additional discriminator, the outputs of which serve as the weights of the original BC loss, so as to imitate demonstrations selectively. Unfortunately, it selects behaviors building on state-action similarity, which does not suffice to leverage the dynamics information and diverse behaviors. In offline RL, Yu et al. (2022) propose to utilize unlabeled data by applying zero rewards, but this method necessitates a large amount of labeled offline data. In contrast, this paper focuses on the setting with no access to any reward signals.

Offline inverse reinforcement learning (IRL) explicitly learns a reward function from offline datasets, aiming to comprehend and generalize the underlying intentions behind expert actions (Lee et al., 2019). Zolna et al. (2020) propose ORIL that constructs a reward function that discriminates expert and exploratory trajectories, followed by an offline RL progress. Chan and van der Schaar (2021) use a variational method to jointly learn an approximate posterior distribution over the reward and policy. Garg et al. (2021) propose to learn a soft Q-function that implicitly represents both reward and policy, which can stabilize the training. To cope with the reward extrapolation error, Chang et al. (2022) introduce a model-based offline IRL algorithms that uses a model inaccuracy estimate to penalize the learned reward function on out-of-distribution state-actions. Recently, Yue et al. (2023) also propose a model-based offline IRL approach, named CLARE. In contrast to Chang et al. (2022), they compute a conservative element-wise weight to implicitly penalize out-of-distribution behaviors. However, it is highly challenging to define and learn meaningful reward functions without environmental interaction (Xu et al., 2022b). The model-based approaches often struggle to scale in high-dimensional environments, and their min-max progress usually causes training to be unstable and inefficient.

6 Conclusion

In this paper, we introduce a simple yet effective data selection method along with a lightweight 306 behavior cloning algorithm to fully leverage the imperfect demonstrations in offline IL. In contrast 307 to the prior methods, we exploit the resultant states to access the value of behaviors, which is an 308 informative criterion that enables explicit utilization of dynamics information and the extraction of 309 both expert-like and beneficial diverse behaviors. We provide necessary theoretical guarantees for the 310 proposed method, and extensive experiments corroborate that iLID outperforms existing methods 311 in continuous, high-dimensional environments by a significant margin. In future work, we plan to 312 establish theoretical guarantees for iLID in the general stochastic MDPs and explore whether the 313 proposed methods can benefit offline RL in terms of data selection and policy optimization. 314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

References

Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Proc. of NeurIPS , 1988.

Stéphane Ross and Drew Bagnell. Efficient reductions for imitation learning. In Proc. of AISTATS , pages 661-668, 2010.

Daniel Jarrett, Ioana Bica, and Mihaela van der Schaar. Strictly batch imitation learning by energybased distribution matching. Proc. of NeurIPS , pages 7354-7365, 2020.

Alex J Chan and M van der Schaar. Scalable bayesian inverse reinforcement learning. In Proc. of ICLR , 2021.

Yueh-Hua Wu, Nontawat Charoenphakdee, Han Bao, Voot Tangkaratt, and Masashi Sugiyama. Imitation learning from imperfect demonstration. In Proc. of ICML , pages 6818-6827, 2019.

Haoran Xu, Xianyuan Zhan, Honglei Yin, and Huiling Qin. Discriminator-weighted offline imitation learning from suboptimal demonstrations. In Proc. of ICML , pages 24725-24742, 2022a.

Tianhe Yu, Aviral Kumar, Yevgen Chebotar, Karol Hausman, Chelsea Finn, and Sergey Levine. How to leverage unlabeled data in offline reinforcement learning. In Proc. of ICML , pages 25611-25635, 2022.

Fumihiro Sasaki and Ryota Yamashina. Behavioral cloning from noisy demonstrations. In Proc. of ICLR , 2020.

Konrad Zolna, Alexander Novikov, Ksenia Konyushkova, Caglar Gulcehre, Ziyu Wang, Yusuf Aytar, Misha Denil, Nando de Freitas, and Scott Reed. Offline learning from demonstrations and unlabeled experience. In Proc. of NeurIPS Workshop , 2020.

Jonathan Chang, Masatoshi Uehara, Dhruv Sreenivas, Rahul Kidambi, and Wen Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Proc. of NeurIPS , pages 965-979, 2022.

Sheng Yue, Guanbo Wang, Wei Shao, Zhaofeng Zhang, Sen Lin, Ju Ren, and Junshan Zhang. Clare: Conservative model-based reward learning for offline inverse reinforcement learning. In Proc. of ICLR , 2023.

Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In Proc. of ICML , pages 214-223, 2017.

Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse soft-q learning for imitation. Proc. of NeurIPS , pages 4028-4039, 2021.

Geon-Hyeong Kim, Seokin Seo, Jongmin Lee, Wonseok Jeon, HyeongJoo Hwang, Hongseok Yang, and Kee-Eung Kim. Demodice: Offline imitation learning with supplementary imperfect demonstrations. In Proc. of ICLR , 2022.

Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. On generalization of adversarial imitation learning and beyond. arXiv preprint arXiv:2106.10424 , 2021.

Nived Rajaraman, Lin Yang, Jiantao Jiao, and Kannan Ramchandran. Toward the fundamental limits of imitation learning. Proc. of NeurIPS , pages 2914-2924, 2020.

Charles Elkan and Keith Noto. Learning classifiers from only positive and unlabeled data. In Proc. of KDD , pages 213-220, 2008.

Stephen P Boyd and Lieven Vandenberghe. Convex optimization . Cambridge university press, 2004.

Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905 , 2018.

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 , 2020.

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 , 2020.

Edouard Klein, Matthieu Geist, and Olivier Pietquin. Batch, off-policy and model-free apprenticeship learning. In Proc. of EWRL , pages 285-296, 2011.

Edouard Klein, Matthieu Geist, Bilal Piot, and Olivier Pietquin. Inverse reinforcement learning through structured classification. Proc. of NeurIPS , 2012.

Bilal Piot, Matthieu Geist, and Olivier Pietquin. Boosted and reward-regularized classification for apprenticeship learning. In Proc. of AAMAS , pages 1249-1256, 2014.

Michael Herman, Tobias Gindele, Jörg Wagner, Felix Schmitt, and Wolfram Burgard. Inverse reinforcement learning with simultaneous estimation of rewards and dynamics. In Proc. of AISTATS , pages 102-110, 2016.

Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In Proc. of ICLR , 2019.

Gokul Swamy, Sanjiban Choudhury, J Andrew Bagnell, and Steven Wu. Of moments and matching: A game-theoretic framework for closing the imitation gap. In Proc. of ICML , pages 10022-10032, 2021.

Pete Florence, Corey Lynch, Andy Zeng, Oscar A Ramirez, Ayzaan Wahid, Laura Downs, Adrian Wong, Johnny Lee, Igor Mordatch, and Jonathan Tompson. Implicit behavioral cloning. In Proc. of CoRL , pages 158-168, 2022.

Donghun Lee, Srivatsan Srinivasan, and Finale Doshi-Velez. Truly batch apprenticeship learning with deep successor features. In Proc. of IJCAI , 2019.

Tian Xu, Ziniu Li, Yang Yu, and Zhi-Quan Luo. Understanding adversarial imitation learning in small sample regime: A stage-coupled analysis. arXiv preprint arXiv:2208.01899 , 2022b.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the challenge of learning good imitative policies from offline data, in which abundant imperfect demonstrations are mixed with few expert ones. Unlike previous work that measures the state-action similarity between imperfect and expert data, the present work proposes iLID, which leverages trajectories in imperfect data that lead to expert states in several steps. The sample complexity analysis indicates that this approach benefits the performance of imitation policy, and the empirical results suggest that iLID outperforms baselines including state-of-the-art offline imitation learning methods.\n\nSTRENGTHS:\n* As illustrated in Figure 1, the original idea of selecting imperfect demonstrations leading to expert states is novel and makes intuitive sense.\n* The policy optimization problem is well-posed, and is straightforward to implement with alternating dual ascent.\n* Empirical results suggest a large performance gain for iLID compared to state-of-the-art baselines, especially when the dataset contains very few expert demonstrations. In particular, the ablation study in Figure 3 does a good job of explaining why the constrained optimization problem leads to a better policy than the naive direct imitation approach.\n\nWEAKNESSES:\nThe quality of presentation can be improved. \n1) $\\tilde{\\mathcal{D}}$ in equation (6) overloads the notation that was originally presented in Section 3.1 without time indices. \n2) Remarks in Section 3.1 state that the sample complexity of the proposed approach is better than the vanilla BC, but there’s no citation for the BC sample complexity. \n3) The explanation on the behavior interference for the complementary dataset $\\tilde{\\mathcal{D}}$ did not make full sense and requires further clarification. In particular, it is unclear why more recent actions are preferred when the same state appears multiple times in the trajectory, even though the underlying MDP does not have any discount factor in the definition of the value function. (What would happen if the discount factor $\\gamma$ in equation (7) is set to 1 for all the experiments?)\n\nQUESTIONS:\n* In Figure 4 and Table 1, it appears that all the methods resulted in poor performance for the Door task. Is it a failure case, and if so what was so difficult about the task?\n* It was not easy for me to follow the proof of Theorem 3.1. There are quite a few questions about the proof steps:\n1) What is the outer expectation over in the definition of $\\epsilon$ and $\\delta$? Is it over the randomness in the choice of the datasets? If so, do you require any assumptions on the distributions of $\\mathcal{D}_e$ and $\\tilde{\\mathcal{D}}$, such as independence?\n2) How do you get from the 1st inequality to the 2nd in equation (13)? Specifically, how do you show $\\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e)) \\mathbb{1}(s \\notin \\mathcal{S}_1(\\tilde{\\mathcal{D}}))\\right] \\leq \\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e))\\right] \\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\tilde{\\mathcal{D}}))\\right]$?\n3) Whether $\\epsilon = \\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e))\\right]$ and $\\delta = \\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\tilde{\\mathcal{D}}))\\right]$ hold or not seem to depend on the distribution of $ \\mathcal{D}_e $ and $ \\mathcal{\\tilde{D}} $. For example, $\\epsilon = \\mathbb{E}\\left[ \\mathbb{E}_s [ \\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e)) ]\\right] = \\mathbb{E}_s \\left[ \\mathbb{E} [ \\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e)) ]\\right] = \\sum_s \\frac{1}{|S|} \\mathbb{E} [\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e)) ] = \\sum_s \\frac{1}{|S|} \\mathbb{P}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e))$ , and the quantity $\\mathbb{P}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e))$ may differ for different $s \\in \\mathcal{S}$ depending on the distribution of $\\mathcal{D}_e$.\n4) The 1st equality of equation (15) is not obvious. How do you show that $\\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e)) \\mathbb{1}(s \\in \\mathcal{S}_1(\\tilde{\\mathcal{D}}))\\right] = \\mathbb{E}\\left[\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e))\\right] \\mathbb{E}\\left[\\mathbb{1}(s \\in \\mathcal{S}_1(\\tilde{\\mathcal{D}}))\\right]$? Do you assume that the events $\\mathbb{1}(s \\notin \\mathcal{S}_1(\\mathcal{D}_e))$ and $\\mathbb{1}(s \\in \\mathcal{S}_1(\\tilde{\\mathcal{D}}))$ are independent for all $s \\in \\mathcal{S}$?\n4) The 1st inequality of equation (15) does not make sense. Did you mean $\\epsilon(1 - \\delta) V^{\\pi_e}$ rather than $\\epsilon(1 - \\delta) V^{\\pi_e}(s)$? Where does the inequality come from?\n5) Can you elaborate on how to derive $\\mathbb{E}_{s' \\sim \\tilde{D}(\\cdot | s)} V'(s')$ in equation (16)? Does it hold because we only have to consider the case $s \\in \\mathcal{S}_1(\\mathcal{\\tilde{D}}) \\backslash \\mathcal{S}_1(\\mathcal{D}_e)$ there, which would correspond to the 2nd case in equation (2)?\n\nPlease elaborate on the points above and clarify any underlying assumptions that are used implicitly.\n\nLIMITATIONS:\nThe authors have not explicitly provided limitations in the paper. One conceivable limitation is that one requires the datasets of expert and imperfect demonstrations to be labeled as such, although it seems unavoidable for any methods of this kind. Another limitation could be that the resulting policy may still fail to discover diverse modes to accomplish the task, if the expert demonstration only has a single mode. For instance, in a goal-reaching navigation task similar to the one depicted in Figure 1, the expert policy still needs to exhibit the two different paths so that iLID learns to discover both modes to approach the goal.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe submission presents a novel method called Offline Imitation Learning with Imperfect Demonstrations (iLID) for Offline Imitation Learning, which aims to improve policy learning from both expert and imperfect demonstrations. Compared with previous IL methods, which only consider the state-action pairs during learning, this paper also considers the dynamics of the non-expert data. To this end, the submission proposed employs a data selection technique using a discriminator on the resultant state of behavior, meanwhile integrating lightweight-constrained behavior cloning. Some empirical studies showed the outperformance of the proposed method compared with other baselines.\n\nSTRENGTHS:\n1. Overall the paper is well-written.\n2. The motivation for including dynamics in behavior cloning makes sense and easy-to-follow.\n3. The experimental tasks and baselines are sufficient.\n\nWEAKNESSES:\n1. Some notations are so confusing that the descriptions of these notations can not lead to corresponding theoretical results.\n2. The assumption of Theorem 3.1 is too strong but without any discussion.\n3. The proposed algorithm and the motivation in the introduction (Figure 1) are isolated.\n\nQUESTIONS:\n1. In line 76 the initial state distribution $\\mu$ is defined as the mapping function $\\mathcal{S} \\rightarrow [0, 1]$. Then how is it possible to sample $s_i \\sim \\mu$ but $s'_i \\sim \\mathcal{S}(\\mathcal{D}_e)$ in the definition of $\\widetilde{\\mathcal{D}}$? After that, in Theorem 3.1, $\\mu$ is again defined by $U(\\mathcal{S})$, but its meaning is completely different from the meaning of the initially defined mapping function.\n2. $\\mu$ is usually defined as a subset of the state space $\\mathcal{S}$. The assumption in Theorem 3.1 that $\\mu=U(\\mathcal{S})$ is too strong and almost impossible for decision-making environments.\n3. There exists no empirical study or statement to illustrate how the proposed algorithm reflects the motivation in the introduction.\n\nLIMITATIONS:\nThere is no discussion about the limitation or societal impact in the submission.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of offline imitation learning (IL) from demonstrations that are noisy/suboptimal. To this end, the authors propose iLID which is a two-step process—a data selection step which only retains those $(s,a)$ transitions from suboptimal demonstrations which lead to states in the expert demonstrations, thereby maintaining a supplementary data buffer; then policy learning is performed by behavior cloning on samples from the supplementary buffer while also regularizing the policy to not stray too far from a BC expert policy (on samples from the expert data buffer.)  \n\nThe authors establish competitive upper bounds on the suboptimality and sample complexity of iLID. Extensive experimental on 8 complex robotic tasks, and accompanying sensitivity studies show that iLID outperforms 5 competing baselines, and limited sensitivity analysis is performed.\n\nSTRENGTHS:\n-\tThe paper tackles the important but challenging issue of offline imitation learning from suboptimal demonstrations. In this regard, the paper addresses a pertinent problem in the research area.\n-\tThe rationale behind the formulation is simple yet powerful. Specifically, the data selection step trains a discriminator to select only those transitions in the suboptimal data $(s,a) \\in \\mathcal{D}_\\mathcal{s}$ which lead to a state in the expert data $(s,a) \\in \\mathcal{D}_\\mathcal{e}$ within some specified $K$ timesteps. This is a simple way of leveraging offline data to distil only useful knowledge from suboptimal data for policy learning and may aid the agent in correcting towards expert behavior from non-expert states. The formulation of the policy learning step as a regularized version of BC yields demonstrable improvements in training time.\n-\tEmpirical results on the D4RL robotics benchmark dataset are impressive (Table 1) and hold across all but one environment. \n-\tOverall, the paper is very well-written and provides helpful illustrations and examples to present ideas.\n\nWEAKNESSES:\nExperimentally, seeding imperfect dataset with expert data (~1-20%) seems like a strong assumption. Given that the data selection method explicitly selects $(s,a)$ pairs based on whether they lead to expert states $s \\in \\mathcal{D}_\\mathcal{e}$. If the expert and suboptimal trajectories only share the seeded expert transitions i.e., $\\mathcal{D}_\\mathcal{e} \\cap\\mathcal{D}_\\mathcal{s} = \\mathcal{D}_\\mathcal{\\text{seeded}}$ (a realistic assumption in real-world cases), then the proposed selection criterion will select only the (seeded) “expert” transitions from $\\mathcal{D}_\\mathcal{s}$ to add to supplementary data $\\tilde{\\mathcal{D}}$ (since only states from the seeded expert trajectories in $\\mathcal{D}_\\mathcal{s}$ would lead to successor states in $\\mathcal{D}_\\mathcal{e}$). In this case does iLID reduce to just pure BC (Pomerlau, 1998) on just expert data with additional policy regularization? Further, Figure 3a also shows that iLID does rely heavily on expert demonstrations for it to perform well. Some ways to address this – \n- Experiment showing results when the imperfect data is not seeded with expert demonstrations would best clarify this issue.\n- Experiment showing results for varying # of expert trajectories (as in Fig 3a) for different environments (including the unseeded case).\n- As an alternative methodology, to bypass seeding, the imperfect demonstration set could be generated rolling out trajectories from the noise-injected expert BC policy $\\tilde{\\pi}_{\\mathcal{e}}$.\n\nQUESTIONS:\n-\tThe only major concern is seeding suboptimal data with expert data. How does iLID perform across different environments when this is not done at all?\n-\t(Table 1) Why does BC with expert data (BCE) perform worse than BC with union data (BCU)? Do they use different number of demonstrations/samples? Compounding errors should have affected both?\n-\tCan the runtime for DemoDICE be included in Figure 5 (missing figure caption?)\n\nLIMITATIONS:\nWhile the iLID formulation is interesting, some weaknesses associated with data seeding could be discussed in more detail as per suggestions above.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an algorithm for offline imitation learning on a mixture of “perfect” expert demonstrations and “imperfect” sub-optimal demonstrations. The authors provide a theoretical motivation for their approach and exhibit results on various Mujoco and Adroit tasks. The authors also conduct ablation studies to justify their design choices.\n\nSTRENGTHS:\n- The paper proposes a novel and interesting approach for learning from suboptimal data. It also provides some theoretical motivation for why filtering data based on resultant states might be better than simply using the state-action distribution. \n- The paper is written clearly and concisely.\n- In addition to the novelty in filtering the data, the paper proposes a novel constrained BC algorithm where a discount factor is used to reduce the impact of stochasticity in the MDP during the optimization process.\n- The author’s compare their approach with a bunch of recently published offline imitation learning algorithms capable of learning of suboptimal data. The algorithms are evaluated across 4 MoJoCo tasks and 4 Adroit tasks with two variations (low and high expert data) for each task.\n- The authors justify their design choices using ablation studies showing the impact of number of expert demonstrations, the number of rollback steps considered, the use of the discount factor and a comparison between runtimes of iLID and other algorithms.\n\nWEAKNESSES:\n- Though the paper has some good ablation studies, it would be interesting to have a study of the effect of quality of the suboptimal data on the performance of the method. The paper currently only considers random trajectories as suboptimal data which might not be very useful for harder tasks. Instead, imagine demonstrations that can complete a part of the task but not the entire task (for instance, can pick up the hammer but not hit the nail). These can probably be collected by rolling out an expert agent and taking random actions in between (varying the percentage of random actions can give different levels of suboptimality). Such a study can help (1) highlight the relevance of the work in the real world where collecting perfect demos might be hard but it is often possible to do parts of the task, and (2) highlight the importance of expert demonstrations in this problem setting (for instance, can we reduce the amount of expert data if the amount of suboptimality in the remaining data reduces).\n- Fig. 4 plots the average return against the number of training steps. However, since the models are completely trained on offline data, a better metric might be a comparison of maximum performance attained by different algorithms. Also, training it for too long might result in the model overfitting on the data, thus, reducing the average return over time (as can be seen in quite a few tasks in Fig. 4).\n- It would be great if the ablations in Fig. 3 could be shown on a few more tasks in the appendix.\n- The paper is missing a limitations section.\n\nQUESTIONS:\nIt would be great if the authors could address the points mentioned in the “Weaknesses” section.\n\nLIMITATIONS:\nThe paper is missing a limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}