Discriminator-Guided Model-Based Offline Imitation Learning
Authors: Wenjia Zhang
Haoran Xu
Haoyi Niu
Peng Cheng
Ming Li
Heming Zhang
Guyue Zhou
Xianyuan Zhan
Venue: CoRL 2022 Poster (2022)
Keywords: Offline Imitation Learning
Model-based Learning

Abstract: Offline imitation learning (IL) is a powerful method to solve decision-making problems from expert demonstrations without reward labels. Existing offline IL methods suffer from severe performance degeneration under limited expert data. Including a learned dynamics model can potentially improve the state-action space coverage of expert data, however, it also faces challenging issues like model approximation/generalization errors and suboptimality of rollout data. In this paper, we propose the Discriminator-guided Model-based offline Imitation Learning (DMIL) framework, which introduces a discriminator to simultaneously distinguish the dynamics correctness and sub-optimality of model rollout data against real expert demonstrations. DMIL adopts a novel cooperative-yet-adversarial learning strategy, which uses the discriminator to guide and couple the learning process of the policy and dynamics model, resulting in improved model performance and robustness. Our framework can also be extended to the case when demonstrations contain a large proportion of suboptimal data. Experimental results show that DMIL and its extension achieve superior performance and robustness compared to state-of-the-art offline IL methods under small datasets.

Discrininator-Guidee Model-Based Offline Imitation Learning
Wenjia Zhang', Haoran Xu², Haoyi Niu', Peng Cheng
Ming Li", Heming Zhang', Guyue Zhou', Xianyuan Zhan 1.4x
Tsinghua University, Beijing. China
JD Technology. Beijing, China
3 Beijing Jiaotong University, Beijing, China
4 Shanghai AT Laboratory, Shanghas, China
zhanguj 10@mails......mmmm wwingnui..n.. edu. cn
(ryanxhr, t6. da thu, , 1 @@gmall com
201 201120410bjtu.edu. hmz@mail teazmmmailttsiinnuua.eeeeee
t taingaannnaaaaaiiiiattiiiaaaaaa edu. cn
Offline imitation learning (IL) is powerful method to solve decision- making problems from expert demonstrations without reward labels. Existing offline IL methods suffer from severe performance degeneration under limited expert data. Including a learned dynamics model can potentially improve the state-action space coverage of expert data, however, it also faces challenging is- sues like model
Abstract:
errors and suboptimality of rollout data. In this paper, we propose the Discriminator-guicl Imitation Learning (DMIL) framework, which introduces a discriminator to simul- taneously distinguish the dynamics correctness and suboptimality of model rollout data against real expert demonstrations. DMIL adopts a novel cooperative-yet adversarial learning strategy, which uses the discriminator to guide and couple the learning process of the policy and dynamiic model, resulting in improved model performance and robustness. Our framework can also be extended the case when demonstrations contain a large proportion of suboptimal data. Experimental results show that DMIL and its extension achieve superior performance and robustness compared to state of the art offline IL. methods under small datasets.
Model-based offline
Keywords: Offline Imitation Learning, Model-based Learning, Sample Elficiency
1
Introduction
Offline imitation learning (IL) that trains a policy from expert demonstrations without additional online environment interactions has become attractive solution for many real-world dccision- making applications, such as robotic manipulation [1] and autonomous driving [2, 3], etc. bypasses several major obstacles practice, such as the difficult reward function design [4] in reinforcement learning (RL) approaches, and the requirement of simulation or real-world system interactions during model training in online IL methods [5, 6, 7, 8, 9], which be costly dangerous.
Despite these desirable features, the performance of offline IL methods heavily depends on the size and quality of demonstration data. Due to its supervised learning nature, learning IL policy in parts of the state space not covered by expert data could make arbitrary mistakes, which leads to severe compounding errors. This phenomenon, called covariate shift [10, 11, 12], is core issuc in IL and greatly hurts the policy gencralization capability. In practice, collecting a large number of expert demonstrations expert data distribution can lead to limited state space coverage, causing poor policy performance. On the other hand, involving non-expert suboptimal offline demonstration data although can potentially
be costly infeasible. The reduction in data size coupled with the narrow
Corresponding author.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

68-
Expeet date
Expert data
Model rolieuts
66
Model rollouus
20
-75
-40
Normall
40
Good policy and model
(b) Good policy and bad model
20-
ralliuus
Different settings
-20
palicy model
Good policy and bad model
Bad policy and good
-40
Bad policy and model
-40
-40
-30
-40
20
Original
(c) Bad policy and good model
(d) Bad policy and model
(e) Evaluation scores
Figure 1: Empirical observations on the impacts of involving dynamics model rollouts in BC. (a)-(d) TSNE visualizations of expert data and dynamics model rollouts under different BC policies on MuJoCO Hopper task with only 20,000 expert data transitions (2% of the D4RL [29] Hopper-expert dataset). The good policy and dynamics model are two-layer MLPs with 256 hidden umits, and are trained until convergence, The bad policy and model are trained with fewer steps, and the hidden layers of the latter are reduced to 128 umits. It can be observed that model rollouts under well-learned policy and dynamics model align well with the expert data, while noticeable discrepancies are observed when the policy or the model is problematic. (e) shows the final performance BC policy trained with 1:1 expert and model rollout data under the four cases (a)-(d). found that under small expert datasets, including a dynamics model in many cases is beneficial, but the quality of rollout policy and dynamics model could have great impact on the final policy performance.
improve state-action space coverage, is shown in previous studies [13, 14] to result in reduced performance in traditional offline IL. methods like behavior cloning (BC) [2]. Many of these problems can be alleviated the online II. setting, either by interactively querying an expert to collect more data [5, 15, 16), or by resorting to inverse reinforcement learning (IRL) to learn rewards function or match the state-action distribution induced by the expert policy [6, 8, 9, 17]. However, such treatments do not apply to the offline setting, since additional environment interaction is not possible. Moreover, utilizing additional suboptimal offline data through offline IRL approaches [13, 18] also shows inferior performance compared with online IRL counterpart methods, due to the involvement of offline RL sub-problems that is prone to training instability and bootstrapping error accumulation [19, 20J. Hence, the ability leverage limited expert data for robust policy learning remains to be a key challenge for the successful real-world deployment offline IL methods.
The sample efficiency requirement for offline IL methods reminds of the success of model-based approaches the online and offline RL domains [21, 22, 23, 24, 25J. Dynamics models learned from the data can greatly supplement the limited expert data to improve state-action space coverage, leading to potentially improved policy performance and generalizability [22, 25, 26, 27]. However, adopting model-based approach in offine IL is still an underexplored area [26, 27, 28]. Many existing methods bear some limitations, such as requiring an additional suboptimal dataset [26] or a low-fidelity simulator [28] for training. fully trusting the learned dynamics model [27]. The key challenges of introducing a learned dynamics model in IL policy learning is twofold (see Figure I for an empirical illustration): 1) the learned dynamics model has poprroximation/generaiiazz directly using model rollouts for imitation learning can be problematic; 2) using the learned policy as the rollout policy may generate suboptimal data, causing performance degeneration that similar to the case of learning with suboptimal data in IL [13, 20). In model-based RL, the second problem is less severe, as the reward function can be used to distinguish the optimality of data. However, this is typically not possible in IL settings.
errors,
In this work, we develop novel model-based offline IL framework to tackle the above challenges. We introduce a discriminator to simultancously distinguish the dynamics discrepancy and suboptimality of the model rollout data against the real expert demonstrations. This gives rise to special cooperative-
2

---

Expert data 1>,
Model rollouts
Expert data D,
Suboptimal date D>
Model rollouts D>
Roll out
Roll out
Traiming
Imitation policy 3 9)
Dynamics model f( ()
Training
Imitation policy X(a (5)
Dynamics model
f (2,44)
Adversarial
Cooperative information sharing
Adversarial learning
Adversarial learning
Cooperative information sharing
learning
d,(s, logo
[s; log Jog Rollout discriminetor
Discriminator d(s, log.loooooooooooo log
Optimality discriminetor
DMIL: learning from expert demonstrations only (b) D2MIL: learning from both expert and suboptimal data
Figure 2: Illustration of the proposed model-based offinn IL framework DMIL and its extension D2MIL
yet-adversarial Shrree-party game". Both the dynamics model and the policy provide information as inputs to the discriminator, while also challenging to establish worst-case error minimization. Under this design, the discriminator can use more information to make better judgment on the dynamics correctness and optimality of the rollout data, and the worst-case optimization scheme also substantially improves the robustness of all three models (policy, dynamics model and discriminator). Interestingly, we can show that this design leads to new policy and dynamics model learning objectives, where the outputs of the discriminator sever weights in their original loss functions. Moreover, the resulting algorithm can be efficiently solved a simple supervised learning manner, which avoids explicitly solving the complex min-max optimization problems in adversarial learning [30, 31]. We thus term our algorithm Discriminator-guided Model-based Imitation Learning (DMIL). Our proposed framework expert and larger proportion of unknown quality, potentially suboptimal data [13, 20). This can be achieved by simply introducing the second discriminator contrast the expert and suboptimal data, which we refer this variant as Soaul-Dicrimmnatoo guided Model-based Imitation Learning (D2MIL). Through extensive experiments D4RL benchmarks [29] and real-world robotic tasks, we show that both DMIL and D2MIL achieve superior performance and robustness against state-of-the-an methods under small datasets. These promising results demonstrate the potential of adopting model-based learning in world offline IL applications under limited expert demonstrations.
also be extended the offline IL setting that involves limited
2 Method
2.1 Problem Setting
We consider the fully observed Markov Decision Process (MDP) setting, which can be described as
M = MS, A, P. do, 7),, where S and A are the state and action space, respectively, P(s'|s, a) is the transition probability, ds(s) is the initial state distribution, r(s, a) the reward function, and y E [0, is the discount factor. Under offline IL setting, we have an expert dataset D. = {(s, a, } collected from some expert policy ne. Our goal is to learn a policy (a]s) to minimize its gap with the expert policy In the simplest case, behavior cloning (BC) trains the policy by minimizing the negative log- likelihood the observed expert actions:
min C :=
(a|s))
2.2 Aiseriminator-Guide Model-Based Imitation Learning (DMIL)
Traditional offline IL methods like BC suffer greatly from covariate shift under small expert datasets due to extremely sparse state space coverage of data. Our idea is to mitigate this issue by involving dynamics model rollouts while also carefully handling these potentially problematic data through the guidance of an additional discriminator in a coupled and cooperative-yet-adveeaarr learning process. Figure provides illustration of the proposed DMIL framework well its extension D2MIL.
Incorporating the Dynamics Model. Model-based approaches have been widely adopted in RL to improve sample efficiency and shows good performance and generalization ability in recent offline

---

RL studies [22, 24, 25]. In our work, we introduce probabilistic dynamics model implemented using a neural network that outputs a Gaussian distribution over the difference between the current and next state, i.e., [[s'|s, a) (s + μo, (s,a), Eo, (s, a)), where 14o (s, a) and Eg, (s, a) are the parameterized mean and diagonal covariance matrix. We predict the difference of states rather than
the next states as it has been shown in past studies [21, 22] to yield better dynamics predictions. The dynamics model can be learned using the following maximum log-likeling log-likelihood objective:
min J == (s.,, |- log f a)]
(2)
cooperative-ye--AAveraaa Learning Scheme. Directly using the rollout data Dr generated by the learned BC policy и and dynamics model f in subsequent imitation learning can be problematic. Under small datasets, it is usually difficult to obtain an accurate dynamics model, and the rollouts from less-well learned policy can be suboptimal compare with the true expert data. To solve this issue, we use a discriminator d to measure dynamics discrepancy and suboptimality in rollout data D, Moreover, we introduce a special discriminator
learning scheme, and use the bridge to couple the learning process of æ, and d. The key idea is to first include the element-wise loss information from both policy x and dynamics model f (i.c., log π and log f) into the input of the discriminator (i.c., d(s,, log ((a s), log f(s'|s, a))) to establish cooperative information sharing, And then make π and f challenge d to establish adversarial learning. This Ieads to special learning objective for the discriminator d, which can be expressed as:
coperraive-yet--dveere
min max La == v.
(",,,))~ - llg d(ss, a, log (a) s), log f( a]]]+
(3)
(s,a,5')~,, |- log(1 - d(s, a log अ (a s), log f( a)))
This design has number of attractive properties. First, element-wise loss information from f and ı reflects the confidence of these models on the rollout data, Suppose f and π are well-learned, then they will assign high probabilities (large log π and log f) on good rollouts with reasonable dynamics and expert-like samples. This can provide valuable information to facilitate the judgment of the discriminator. Second, the adversarial component forms GAN-like problem [30], where and f jointly serve as a generator to challenge the discriminator. This will force the discriminator to minimize the worst-case error [31, 32], which makes its robustness significantly improved. In return, a stronger d can better guide the learning of π and f to further improve their performance and make better use of the generalization power of the dynamics model. Consequently, this cooperative- yet-adversarial learning scheme enables coupling among policy, dynamics model and discriminator, which can potentially lead to boosted performance for all three models.
Loss Correction for Policy and Dynamies Model. Jointly solving (3) together with minimiza- tion problems in (1) and (2) can be rather complex. As both and f appear in the input of the discriminator, d becomes a functional of π and f (i.c.. function a function). (3. is a functional min-max optimization problem, which is itself quite challenging solve. Fortunately, based on calcu- lus of variation [33] and the analysis method introduced in DWBC [20], we can avoid directly solving this complex functional min-max optimization problem by introducing discriminator-dependee correction terms Ceor and Lgor on the losses of policy C and dynamics model Ly. In this way, т, f and d can be efficiently learned by solving three simple minimization problems: min, On Ln +L min, a L + Lior and mina La where an, a 2 1 are weight factors for the original losses of and f. In the follows, we briefly describe the essential steps of deriving Caer and Lear, , and provide detailed derivations in Appendix A. The outline of DMIL is presented in Appendix B.1.
loss
Denote I (s, s') and Peas' its domain. Note that the functional La(d, log n, log f) can be written as the integral of a new functional FIr, log म, log f) with the following form:
Lu = / (PD. log d) PD, (x) (- (-11 log logd)) d))
[ JA.. F(x, d, log log f)dx (4) Jea.
where we slightly abuse the notations and write the output of d(s, a. log ((a)s) s), log f f(' a)) as and F(x, d, log म, log f) as F hereafter, PD. and PD, are distributions of x in D. and D,-. To simplify the analysis, we focus the inner maximization problem in (3) According to calculus of variation, maximizing La with respect to function π and f requires to find the extrema of La. which can be achieved by solving the following associate Faler-Lagrangian equations:
F-
à F
Fr 0
(5)
Fj
For = =F = 0 On
dx

---

where Fy stands for ar Let 0₂ and fy denote the network parameters of policy x and dynamics yy
model f. Using the analysis policy
example. Assuming F and d are continuously differentiable with respect to and log respectively, from the first equation in Eq.(5), we have Ōx
80 alog ad Vo, log = 0. As d is determined by ad the outer minimization problem of (3)(3), thus Flog not obtainable by solely inspecting the
Fx An af ad Blog
09, ad
alog=
inner maximization problem. To ensure the previous equation hold, can instead consider relaxed condition by letting äF Vo, log я O. The integration of this condition is still Vo, log udæ = 0), which leads to the following tractable condition:
Vornn af 3d
1 d
E (s,0,5")~,
Vo, log
+ (s,,)")),
Ve, log
=
(6)
Above can be equivalently perceived as the first-order optimality condition of minimizing the following corrective loss term C for policy a:
Ccorr
log s(ls))
E
1 - log T(als)
(7)
(5,a,8))-2
(s,,,))~,,
Similarly, we can obtain the corrective loss term Egor for dynamics model f fas:
2/2 log ((''ss,
E (s,0.5")~D,
E d log ((s"ss,))
(8)
(.,,5')-,,
2.3 Extensions to Scenarios with Additional Suboptimal Dataset
The DMIL framework can be easily extended to IL scenarios with small expert dataset D. and larger dataset D, sampled from one or multiple potentially suboptimal policies [13, 20, 34). Under this setting, we can add second optimality discriminator d, additional to the original rollout discriminator in DMIL (referred as d, in this setting), dedicated to differentiate between expert and suboptimal samples in both DD and D,-- follow learning [35] objective for d,, and also introduce second pair of adversarial relationship between and do. PU-learning enables learning from positive (expert data D.) and unlabeled data (D, DD in case) with hyperparameter 7) capture the proportion of positive samples to unlabeled samples.
al. [20] to adopt positive-unlabelee (PU)
min max Cd, = (s,a)>D - log do a, (als)) d,
(s.D)-D.UD, [ log(1 - dols, a, [ag=(s)))) (s.a)~D.
E |- log(1 do a, (ogg((()))))
Similar to the derivation in previous section, when jointly solving above functional min-max opti- mization problem together with (q)(1),)3), we can obtain the following updated corrective loss term for policy x, which now depends on outputs of both discriminators d, and d,, with Bo and 3, being the weight parameters for the two discriminators. We term this extension as Dual-Diseriminttr guided Model-based Imitation Learning (D2MIL). Complete derivation can be found in Appendix A.
I do (1 do)
BaT
3r
[1-
Bo do
Br 3
LEOrT
E (s.,,5"
(1-))
log ((lss)
E
log (als)
d,
(s,0,5"~>D,
( 1- do
Bo
Br
E (e,,
log Taga(as)
(10)
1
dr
3 Experiments
We evaluate methods against offline IL baseline methods on both D4RL benchmark datasets [29] and a real-world wheel-legged robot. Our methods achieve superior performance and robustness compared with baselines, especially under small datasets. Experiment setups and results described below. Ablation study on the impact different design elements of DMIL can be found in Appendix C.2. Implementation details and extra comparative results reported in Appendix B and C.

---

Table I: Normalized scores for models trained on different proportion of DARL MuloCo-expert datasets and Adroit-human lasks. Results are averaged over 3 random sceds.
Ratio
BC
BC+d
2-phase BC+d
DWBC+d
ValueDICE
IQ-Leamn
DMIL
100%
95.06:1:2.38
106.78::4.4
110.59:0.63
60.34= 10.12
25,49:1:.34
110.22:11.22
Hopper
10%
33.52:1:0.58
10.
104.3519.44 44
91.32124.81
58.77=10.45
25.16116.69
111.56::.51
5%
73.35:137.04
19.72
99. 19.96
88.35:228.1
44.94=13.71
0.58:10.1
111.14:1.83
2%
53.54:1:36.9
61,57:1:0.0 18
88 23..24335.663
81.70:132.27
31.38 12.84
3.72:00.6
108.51
100%
91,95 .24
8..23
91.48.10.33
83.75=6.57
5..07
38.12:19.96
93,34:11.29
Halfcheetah
10%
90.64412221
89.71
11.27+19.33
17.49
48.77
18.36=11..09
92.69:1182
5%
82. 11.77
16.40=16.94
70.89+23.06
65.76+200.5
30.61
7.12+6.77
90,18+4.43
2%
23.58+ 16.36
21.48+16.86
27.48=25.63
30.10=22.27
17.47=7.63
1.63+1.37
76.87=15.31
100%
107.35+:229
10.3
008.15
103.92-1..53
86.42.=11.20
100.96 1.23
107.65
Walker2d
10%
105.36:t4.38
107.61=1.14
106,40
91.17=25.05
86.76=13.04
73.65+12.64
107.62:1:.83
5%
103.21
105.42.3.93
104.51+4.54
89.78=24.81
83.91=12.96
59.47=23.17
107.89::..1
2%
58.34:135.88
60.64:135.00 10
86.71#21.20
65.19136.27
78 28.84123.16
34.19420.11
105.55:1:.42
pen human
57.911555.05
15.87
68.57:153.57
18.611226.66
52.51 19.58
4.94=
67.56:157.87
hammer-human
1.05
1.18
1.64:11.30
0.67-10.64
1.12=0.64
0.37-+0.13
2.06:1:1.91
door-human
0.47:10.65
0.16::0.29
0.94+1.22
0.01+0021
0.22==0001
-0.28
6.06:17.56
3.1 Experiment Setup
Baselines. We compare DMIL with 5 baselines: 1) BC: vanilla BC [2]: 2) BC+d: learns dynamics model alongside BC to generate rollouts, and the policy trained on both expert and rollout data; 3) 2-phase BC+d: first pretrains the dynamics model and BC policy expert data, then uses BC+d to fine-tune the policy; 4) DWBC+d: we use pretrained dynamics model and a BC policy to generate the suboptimal dataset required in DWBC, and then run DWBC learn the policy; 5) ValueDICE: we implement an offline version of the original ValueDICE [9], which uses a learned dynamics model to serve as the online sampling environment; 6) IQ-Learn [36]: a recent IL method that Iearns Q function to implicitly represent the policy, and can work offline. For D2MIL, we compare it with BC trained on expert data only (BC-exp) and on all data (BC-all), as well as two recent methods ORIL [13], DemoDICE [34] and DWBC [20] which are designed for the same problem setting.
Simulation Tasks. We conduct the experiments on the widely-used D4RL [29] MuJoCo ex- pert/medium datasets and the more complex Adroit human datasets (Pen, Hammer, Door). To investigate the impact of sample size on model performance, of transitions from MuJoCo expert datasets to construct set of much smaller datasets for evaluation.
randomly sample certain proportions
real-world robot which stands on pair of wheels to get balanced, as shown in Figure 3. The states of robot are composed of its forward tilt angle 8, displacement I, angular velocity 0 and linear velocity ă The robot controlled by the torque 7 of motors at two wheels. We evaluate method two tasks: (1) Standing still: keep the robot balanced and not fall down; (2) Moving straight: keep the robot balanced and move forward with a target velocity The dataset for these tasks are collected from very few human demonstrations (10,000 transitions from about 50s human control a sampling frequency of 200Hz).
Real-world Robotic Tasks. We also experiment
Figure 3: Wheel-legged robot
3.2 Results
Comparative Evaluation on D4RL Benchmarks. The comparative results are presented in Table We can see that many tasks, naively incorporating dynamics model with BC only leads to marginal improvement. This is due to the lack of discrimination on the quality of rollout data. 2-phase BC+d that use pretrained, high quality dynamics model and rollout policy in some cases can result in improved performance under small datasei. Besides, offline ValueDICE performs poorly owing to its reliance on accurate online interaction. IQ-Learn performs badly on the continuous control tasks with high digensiimnnional sate-action space. For DWBC+d, we can see that simply incorporating

---

Disgbacment
Angle
10000
(redt
-0.2
tie (9)
Biee (6)
time
time
BC+d
ValueDice
IQ-1eam
BC-d
ValueCice
10-Leam
(a) Standing still
(b) Moving straight
Figure 4: Evaluation results real-world wheel-legged robot
Normalized scares
100
300
150
DMIL-no-de rov
p-paase
DMIL
10%
5%
2%
10%
5%
2%
10%
5%
2%
Hopper
Haltcheetah
Walker2d
Figure 5: Evaluation results on policy robustness. For differrnnt sizes of expert datasets, we randomly pick 20% samples and add a Gaussian noise on the states to make policy learning more challenging.
Nermalized scgres
100
DemoDICE
LWWCC
exp-med-0.3 exp-med-0.6 Hopper
exp-med- 0.3 exp-med-0.6 Halfcheetah
exp-med-0.3 Exp-med-0.6 Walker2d
D2MLL
Figure 6: Evaluation results D2MIL under small datasets. We first sample 1% trajectories from the D4RL MuloCo expert datasets. We then sample X proportion of these trajectories and combine them with the 2% medium dataset to constitute the suboptimal dataset D.. The remaining 1-X trajectories constitute the expert dataset De. The larger X, D, contains more high quality data, but corresponds smaller expert dataset D.. We label each lask exp-med-X in the figure.
rollouts from dynamics model suboptimal dataset in DWBC brings no benefit dynamics model learning, and insufficient leverage of information in the limited data, which leads to substaintial performance drop with smaller dataset. By contrast, our method achieves the best performance in almost all tasks with small variance. Most importantly, we find DMIL performs surprisingly well under small datasets while other baselines suffer from comparable performance even if the training data is reduced to 5% or 2% of its original size.
performance degeneration. It achieves
Comparative Evaluation on Real-World Tasks. The imitation performance of two tasks wheel-legged robot are shown in figure 4. these two tasks, we only use 50s human demonstrations to learn the policy. For the Standing still task, despite some small drifts, the robot using DMIL policy can maintain in a balanced state for over 30s, which achieves the most stable performance both displacement measure and tilt angle. The robot with other control policies either quickly bump to the ground CC, methods make the robot move forward within certain speed range, but DMIL policy maintains a closest speed to the target speed v=0.2m/s and also keeps a relatively more balanced state.
or dashes forward (IQ-learn). For the Moving straight task, most
Evaluation on Policy Robustness. We further evaluate the policy robustness of DMIL under small and noisy training data on MuJoCo tasks in Figure 5. We compare with three stronger baselines in Table 1: BC, BC+d and 2-phase BC+d. To further examine the effectiveness of the cooperative- yet-adversarial learning scheme on the learned dynamics model, we add an additional baseline DMIL-no-d-ady, which removes Cfor as well as log f in the input of discriminator d from DMIL. We observe that the performances of BC and BC+d drop with the introduction of noise, mainly due to the lack of discrimination on data quality. 2-phase BC+d is slightly better, but still perform worse than DMIL and DMIL-no-d-adv. Due to the absence of adversarial learning in dynamics model, OMMI-no-d-acd is generally less performant compared with DMIL due the noisy training data. In all tasks, DMIL shows great robustness to training noise and achieves almost the same performance

---

as the case without noise (Table 1). This is because that the discriminator of DMIL in this setting not only distinguishes dynamics correctness and optimality rollouts, but can also serve as a denoiser to identify and alleviate the negative impact of noisy inputs for policy and dynamics model.
Evaluation of D2MIL. We also evaluate the performance of D2MIL when learning with small expert dataset and a larger suboptimal dataset in Figure 6. The results show that D2MIL outperforms Mlac-offthheart method DWBC [20] and other basclines all tasks. The introduction of the dynamies model f and the two discriminators (d, and d,) indeed help with improving the generalization performance of imitating policy under small datasets, which demonstrates the effectiveness of D2MIL in scenarios with suboptimal data.
4 Related Work
Model-based Imitation Learning. To combat the covariate shift and improve sample efficiency, many online IL studies have incorporated dynamics models during policy learning [27, 37, 38, 39]. These methods typically require online system interactions or additional expert guidance to correct model errors. Under offline settings without environment interaction, incorporating the model- approach is much more challenging and less explored. A few existing works all bear some limitations, such requiring an extra suboptimal dataset [26] or a misspecified simulator [28], only applicable to imagery input [40], or simply fully trust the learned model [27]. Many of these methods assume sufficient coverage of demonstration data, which can be fragile in scenarios with small datasets.
Offline Imitation Learning. Offline IL methods that imitate expert demonstrations can be calego- rized into two paradigms, behavior cloning (BC) and offline inverse reinforcement learning (offline IRL). BC [2] is the simplest IL method, it trains a policy by maximizing the log- logliielihood of ob- served actions. Some recent works enhance BC by using energy- baaeyy model [41, 42] or introducing curriculum training strategy [43]. Offline IRL methods [9, 44, 45, 42, 36] on the other hand, consider matching the reward or state-action distribution of the expert policy. This can be done explicitly by learning a reward function [44] or implicitly by learning a Q-function that represents both reward and policy [9, 36). Although these recent methods can mitigate covariate shift to some extent, they still struggle to work under limited expert data and suffer from the involvement of suboptimal data.
Another stream of studies focus on the problem when demonstrations contain suboptimal data. Some studics [46, 47] leverage previously Iearned policics [47] or entropy of the model [46] as weights to penalize noisy demonstrations. However, they require the clean expert data occupy the majority of the offline dataset. When both the expert demonstrations and additional suboptimal data are given, some IRL-based methods [13, 26, 34] first construct reward function to distinguish expert and suboptimal data, and then use it to solve an offline RL. problems. The drawbacks of these methods that the reward learning through offline IRL is costly, and the inner-loop offline RL problem also suffers from training instability [19]. The recently proposed DWBC [20] trains a discriminator to distinguish expert and non-expert data and uses its outputs to re-weight the IL objective, so to imitate demonstrations selectively, Our method shares some similarity with DWBC, however, we use the discriminator to distinguish both the dynamics discrepancy and suboptimality of model rollout data, and re- re-weight the objectives of both the IL policy and the dynamics model.
5 Conclusion and Limitations
We propose a model-based offline IL framework DMIL for scenarios with limited expert data, which is composed of an imitation policy, dynamics model and discriminator. We use the discriminator as a bridge to couple the learning process of all three models through a cooperative-yel--dvrrsss learning scheme. This design allows fully leverage the generalizability of dynamics model to improve state action space coverage, while also alleviating the negative impacts from potentially problematic rollouts. Our framework can also be extended to scenarios with suboptimal data (D2MIL). Through comprehensive experiments, we show that our method achieves strong performance and robustness under small datasets, which can be a nice tool for many real-world IL tasks
Our method also has some limitations. When the state-action space large or the MDP is partially observed, the dynamics model might need to be specially designed. For future directions, adopting temporal models, or learning the dynamics in latent state space might be a solution to achieve improved model performance.
8

---

Acknowledgments
We thank the anonymous reviewers for their thoughtful feedback. This work is supported by funding from Ha
References
[1] B. Fang, S. Jia, D. Guo, M. Xu, S. Wen, and Sun. Survey of imitation learning for robotic manipulation. International Journal of Intelligent Robotics and Applications. 3, 12 2019.
[2] D. A. Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in neural information processing systems, 1988.
[3] L. Le Mero. D. Yi, M. Dianati, and A. Mouzakitis. survey on imitation learning techniques for end-to-end autonomous vehicles. IEEE Transactions on Intelligent Transportation Systems, pages 1-20, 2022.
[4] A. Y. Ng. D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. In Proceedings of the Sixteenth International Conference on Machine Learning (ICML 1999), 1999, pages 278-287, 1999.
[5] Ross, G. Gordon, and J. A. Bagnell. reduction of imitation learning and structured prediction no-regret online learning. In AISTATS, 2011.
[6] J. Ho and S. Ermon. Generative adversarial imitation learning. In Proceedings of the 30th International Conference on Neural Information Processing Systems, NIPS' 16, page 4572-4580, 2016.
[7] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-actor-coo Addressing sample inefficiency and reward bias in adversarial imitation learning. In ICLR, 2019.
[8] B. D. Ziebart, A. L. Maas, J. A. Bagnell, A. K. Dey, et al. Maximum entropy inverse reinforce- ment learning. In AAAL, volume 8, pages 1433-1438. Chicago, IL. USA, 2008
[9] I. Kostrikov, O. Nachum, and Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020.
[10] N. Rajaraman, L. F. Yang, J. Jiao, and K. Ramchandran. Toward the fundamental limits of imitation learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, 20, 2020.
[11] C. Spencer, S. Choudhury, A. Venkatraman, B. D. Ziebart, and A. Bagnell. Feedback in imitation learning: The three regimes of covariate shift. CoRR, 0bs/2102.0228 2021.
[12] G. Tennenholtz, A. Hallak, G. Dalal, S. Mannor, G. Chechik, and U. Shalit. On covariate shift of latent confounders in imitation and reinforcement learning. International Conference on Learning Representationss 2022.
[13] K. Zolna, A. Novikov, K. Konyushkova, Ç. Gülçehre, Z. Wang. Y. Aytar, M. Denil, N. de Freitas, and S. Reed. Offline learning from demonstrations and unlabeled experience. CoRR. abs/201 1.2200111188 13885, 2020.
[14] A. Mandlekar, D. Xu, J. Wong, S. Nasiriany, C. Wang. R. Kulkarni, L. Fei-Fei, S. Savarese, Zhu, and R. Martfn-Marttn. What matters in learning from offline human demonstrations for robot manipulation. In Conference on Robot Learning, pages 1678-1690. PMLR, 2022.
[15] J. Zhang and K. Cho. Query-elficient imitation learning for end-to-end simulated driving. In Proceedings of the Thirty-Firs AAAI Conference on Artificial Intelligence, AAAI'17, page 2891-2897, AAAI Press, 2017.
[16] M. Kelly, C. Sidrane, K. CragbelCa and M. Kochenderfer. Hg-dagger: Interactive imitation learning with human experts. In 2019 International Conference on Robotics and Automation, ICRA 2019, pages 8077-8083, May 2019.

---

[17) X. B. Peng, A. Kanazawa, S. Toyer, P. Abbeel, and S. Levine. Variational discriminator bottleneck: Improving imitation learning, inverse rl, and gans by constraining information flow. In 7th International Conference on Learning Representations, ICLR, 2019.
[18] K. Konyushkova, K. Zolna, Y. Aytar, A. Novikov, S. Reed, S. Cabi, and N. de Freitas. Semi- supervised reward learning for offline reinforcement learning. arXiv preprint arXiv:2012.06899, 2020.
[19] A. Kumar, J. Fu, M. Soh, G. Tucker, and S. Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. In Advances in Neural Information Processing Systems, pages 11761-11771, 2019.
[20] H. Xu, Zhan, H. Yin, and H. Qin. Discriminnatr--eiighed' offline imitation learning from suboptimal demonstrations. In International Conference on Machine Learning, 2022.
[21] M. Janner, J. Fu, M. Zhang, and S. Levine When to trust your model: Model-based policy optimization. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 12498 12509, 2019.
[22] T. Yu, G. Thomas, L.. Yu, Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. Mopo: Model- based offline policy optimization. In Proceedings of the 34th International Conference on Neural Information Processing Systems, pages 14129-14142, 2020.
[23] K. Chua, R. Calandra, R. McAllister and Levine. Deep reinforcement learning in a handful of trials using prohabilistic dynamics models. In Advances in Neural Information Processing Systems, pages 4759 4770, 2018.
[24] X. Zhan, H. Xu, Y. Zhang. X. Zhu, H. Yin, and Y. Zheng. Deepthermal: Combustion optimiza- tion for thermal power generating units using offline reinforcement Icarning, In Proceedings of the AAAI Conference on Artificial Intellience, 2022.
[25] X. Zhan, X. Zhu, and H. Xu. Model-based offline planning with trajectory pruning. In Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI 2022, pages 3716-3722, 2022.
[26] J. Chang, M. Uehara, D. Sreenivas, R. Kidambi, and W. Sun. Mitigating covariate shift in imitation learning via offline data with partial coverage. Advances in Neural Information Processing Systems, 34:965-979, 2021.
[27] H. Lin, B. Li, X. Zhou, J. Wang, and M. Q.H. Meng. No need for interactions: Robust model-based imitation Icarning using neural odc. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 11088 11094, 2021.
[28] S. Jiang, J. Pang, and Y. Yu. Offline imitation learning with misspecified simulator. In Advances in Neural Information Processing Systems, 2020.
[29] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072 2020.
[30] Goodfellow, Pouget-ouaaeeeee Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
[31] N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, A. Madry, and A. Kurakin. On evaluating adversarial robustness. arXiv preprint arXiv: 10Xvx:100,.06755, 2019.
[32] I. J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, ICLR 2015, 2015.
[33] I. M. Gelfand, R. A. Silverman, et al. Calculus of variations. Courier Corporation, 2000.
[34] G.-H. Kim, S. Seo, J. Lee, W. Jeon, H. Hwang, H. Yang, and K.E Kim. DemoDICE: Offline imitation learning with supplementary imperfect demonstrations. International Conference on Learning Representations, 2022.
10

---

[35] K. Zoina, A. Novikov, K. Konyushkova, Ç. Gülgehre, Z. Wang, Y. Aytar, M. Denil, N. de Freitas, and Reed. Offine learning from demonstrations and unlabeled experience. CoRR, abs/2011 133885, 2020.
[36] D. Garg, S. Chakraborty, C. Cundy, J. Song, and S. Ermon. Iq-leam: Inverse soft-q learning for imitation. In Thirty-Fijh Fifth Conference on Neural Information Processing Systems, 2021.
[37] N. D. Palo and B. Johns. SAFARI: safe and active robot imitation learning with imagination. CoRR, abs/201 1. 1.09586, 2020.
[38] R. Rafailov, Yu, Rajeswaran, and Finn. Visual adversarial imitation learning using variational models. In Advances in Neural Information Processing Systems, NeurIPS 2021, pages 3016 3028, 2021.
[39] Englert, A. Paraschos, M. P. Deisenroth, and J. Peters. Probabilistic model-based imitation learning, Adaptive Behavior, 21(5):388-005 2013.
[40] A. Wu, A. J. Piergiovanni, and M. S. Ryoo. llddel-based robot imitation with future image similarity. Int. J. Comput. Vision 12865):130--3374, may 2020. ISSN 0920-5691.
[41] P. Florence, C. Lynch, A. Zeng, O. A. Ramirez, A. Wahid, L. Downs, A. Wong, J. Lee, I. Mordatch, and Tompson. Implicit behavioral cloning. In 5th Annual Conference Robot Learning, 2021.
[42] D. Jarrett, I. Bica, and M. van der Schaar. Strictly batch imitation learning by energy-based distribution matching. Advances in Neural Information Processing Systems, 33:7354-7365, 2020.
[43] M. Liu, H. Zhao, Z. Yang, J. Shen, W. Zhang, L. Zhao, and T.-Y. Liu. Curriculum offline imitating learning. In Advances in Neural Information Processing Systems, 2021.
[44] G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: game- theoretic framework for closing the imitation gap. In International Conference Machine Learning, pages 10022-10032. PMLR, 2021.
[45] O. Nachum, Y. Chow, B. Dai, and L. Li. Dualdice: Behavior-agnostic estimation of discounted stationary distribution corrections. Advances in Neural Information Processing Systems, 32, 2019.
[46] M. Hussein, B. Crowe, M. Petrik, and M. Begum. Robust maximum entropy behavior cloning. CoRR, abs/2101 01251, 2021.
[47] Sasaki and R. Yamashina. Behavioral cloning from noisy demonstrations. In International Conference Learning Representations, 2021.
[48] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, pages 214-223. PMLR, 2017.
[49] E. Hewitt. Rings of veal-v continuous functions. Transactions of the American Mathemati- cal Society, 64(1):45-99 1948.
11


Reviews:
[{"review_id": "j75C22aWs1p", "reviewer": "Reviewer_wiFv", "summary": "The paper presents a new approach for offline imitation learning. The key idea is to learn a dynamics model along with the policy. To do so they use a discriminator that takes into account both dynamics and policy simultaneously. They show that this approach improves performance of agents even when the number of expert demonstrations available is low. They also show on a real robot their methods works better than other baselines.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) The proposed approach has solid performance over different expert dataset sizes. \n2) The proposed method also outperforms many different offline imitation learning algorithms. \n3) One of the baselines proposed by the authors \"2 phase BC+d\" is a very reasonable baseline that outperforms more complicated algorithms. More details on this baseline would be useful.\n4) Real robot experiments showing the effectiveness of their approach.\n\nWeakness\n1) Ablation experiments missing the show the importance of each term in the loss function. This will give more insight into why their method is performing so well. It seems only in Fig. 5 when the authors test the robustness of policy with noisy datasets, they have an ablation experiment. \n\nMinor Weakness\n1) \"performance degeneration under limited expert data due to covariate shift\" - this statement in the abstract is not backed up by any experiment. Performance degeneration is usually due to drift from expert demonstrations' data distribution rather than covariate shift.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The proposed approach performs well when the number of demonstrations available are low and there is noise present in the data. The authors have made extensive comparison with existing methods. They have also shown their method works on a real robot. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "1. Adding ablation experiments with different loss terms and sensitivity of their method to different hyper-parameter settings.\n", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "hSdJg9OuEf0", "reviewer": "Reviewer_ihkf", "summary": "The paper presents a model-based offline imitation leaning algorithm – DMIL. The method jointly trains a policy, a dynamics model, and a discriminator in an iterative algorithm. The policy and the dynamics model are trained with maximum likelihood on the expert data, with further regularization (corrective loss terms) using the discriminator. The discriminator is trained with a GAN-style objective on the expert and rolled-out data. Crucially, the discriminator includes the log-likelihood of the policy and the dynamics model as inputs to facilitate information sharing. An extension (D2MIL) is also presented for when sub-optimal data is available. Experiments on D4RL MuJoCo and Adroit tasks, and a real-world wheeled robotic task show that DMIL compare favorably to the baseline offline imitation methods especially when expert data is limited. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Improving the sample-efficiency (in terms of expert data size) of offline imitation learning algorithms is important for their practical application. The paper provides an interesting approach to achieve this and backs it up with nice experiments -- for instance, Table 1 shows that DMIL can achieve competitive performance with only 2% of the original data in Hopper and Walker. Another strong point is the inclusion of results on real robot tasks. I feel that the paper has some shortcomings in the current form – presentation of material and clear differentiation from prior work. These are expanded further below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "My pre-rebuttal rating is a weak-reject since I have some reservations about the presentation, comparison to prior work, and experiments (see issues below). The paper has several nice elements to it and I would be happy to reconsider after discussion with the authors.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Differentiation from prior work: The “cooperative information sharing” for the discriminator was introduced in DWBC (Xu et al.), where they used $\\log\\pi$ in the discriminator input. DMIL builds upon this idea by introducing a learned dynamics model and augmenting the discriminator input with the model likelihood. The corrective loss terms follow the same derivation as DWBC. Although I believe that the extensions are important, the current presentation in Section 2.2 could be improved to cover DWBC in a bit more detail and clearly outline the new contributions of this paper.\n\n\n2. Other suggestions regarding presentation:\n\n* &nbsp; It would really help to move the DMIL algorithm box (Appendix B.1) to the main paper. The algorithm box provides a full picture of the approach that is currently difficult to glean from Section 2. \n\n* &nbsp; I would also recommend that you move the no-$\\log f$-no-$\\log\\pi$ ablation results from the Appendix (Table 5) to the main paper results, and comment on the performance comparison to DMIL. This seems to be an important ablation to justify the “cooperative information sharing” argument.\n\n* &nbsp; Section 2.3: some details could be included here to make the paper self-contained. For instance, the acronym “PU” is not expanded, $\\eta$ is not defined, etc.\n\n3. MuJoCo environments – it’s nice to see that for Hopper/Walker/Cheetah, using only about 5% of the expert data recovers the performance of using the full data. However, I wonder if this is largely due to the nature of the environments – they are deterministic and involve repetitive (cyclic) motions, so only a few expert transitions are usually sufficient. On the more complicated tasks, it seems there’s still big gap to the expert’s performance (especially hammer/door). Could the authors give some intuition/ideas for this?\n\n4. Clarifying question: there’s a relaxation that is made in the derivation of equation 6, where $\\frac{\\partial d}{\\partial \\log\\pi}$ is ignored. In an iterative algorithm, can this term be computed for the current discriminator using the auto-differentiation tools and used for the policy/dynamics gradient? "}}, {"review_id": "DKdoYC3W4Sb", "reviewer": "Reviewer_F1vv", "summary": "The paper introduces an approach to offline imitation learning that extends a dynamic forward model with an adversarial discriminator. The idea is to use the discriminator to estimate the forward model correctness and optimality of the model rollouts at the same time. Additionally, the authors extend the method to incorporate sub-optimal data as well.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n* The method is novel. I like the idea of re-evaluating the correctness of the predictions of the forward model using a discriminator. The existing techniques use online interactions (see [1, 2]) instead of a model that is accurate and does not require re-evaluation. Or they use ensembles of forward models, which are computationally expensive (PETS, [3]). Using a discriminator mitigates the issues associated with computational costs of PETS.\n* The method is theoretically sound and outperforms on the baselines. In particularly, the method outperforms the baselines in the low-data regime which is considered to be the most challenging for offline learning.\n\n# Weaknesses\n* The clarity of writing can be improved. It is not clear how the method is implemented. The paper can be improved by adding a pseudo-code for the algorithm.\n\n[1] Generative adversarial imitation learning, J. Ho, S. Ermon\n\n[2] Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning, I. Kostrikov, K. Agrawal, D. Dwibedi, S. Levine, J. Tompson\n\n[3] Deep reinforcement learning in a handful of trials using probabilistic dynamics models, K. Chua, R. Calandra, R. McAllister, S. Levine", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "In general, I recommend this paper for acceptance. However, the clarity of presentation can be improved. See my comments above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "* Clarity of presentations. See the comments above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "C8fctnyREt0", "reviewer": "Reviewer_8kmH", "summary": "This paper presents a model-based offline imitation learning method that trains a model, a policy and a discriminator in both a cooperative and adversarial way to improve the robustness of the dynamics model and the the performance of the policy. The method is evaluated in simulated domains with both expert demonstrations and sub-optimal offline data settings and is shown to be superior compared to prior offline imitation learning methods. The authors also include the evaluation on the real-world wheel-legged robot where the method appears to be effective.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The paper presents a new model-based offline method that trains a discriminator along with the learned policy and dynamics model, unlike prior model-based IL methods. The discriminator takes the dynamics model as an additional input, which is different from the prior discriminator-based IL works [19]. Such a scheme seems to effectively achieve a cooperative yet adversarial game among the discriminator, dynamics model and the policy. Moreover, the method achieves strong results in the simulated D4RL domains with fully expert or human data.\n\n2. The authors include ablations that show the importance of $L_corr$ and the log-likelihood of the dynamics model as inputs to the discriminator, showing the effectiveness of the algorithmic contributions in practice.\n\n**Weakness**: \n\n1. Despite that the method gives the cooperative yet adversarial game, the technical part of the method appears to be a model-based version of the prior work [19], which is rather straightforward and makes the novelty a bit limited. The derivation of the algorithm is also heavily based on [19].\n\n2. I'm not sure why comparisons to [19] are not included in Table 1 and Figure 19 as the difference between the proposed method and [19] is rather small in Figure 5 and 6. It is important to compare to [19] everywhere as it appears to be the strongest baseline. Moreover, I think it is important to compare to some model-based IL methods such as [35, 36]. While these methods are proposed to be online, they should be easy to be applied in the fully offline setting. I also think DemoDICE [45] should be a good baseline to include as it handles offline IL with suboptimal data.\n\n3. Finally, I wonder how the hyperparameters are tuned. The method introduces lots of hyperparameters and it's known that tuning hyperparameters in offline IL/RL usually requires online rollouts, which kind of defeats the purpose of fully offline training particularly in the case where there are so many hyperparameters to tune.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Based on the Strengths And Weaknesses section, I'm leaning towards a weak reject of the paper but I can change my rating if all the concerns are properly addressed.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "1. Compare to model-based baselines such as [35, 36] and also DemoDICE [45].\n\n2. Include comparisons to [19] in all domains.\n\n3. Clarify hyperparameter selection schemes.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)