Plan Your Target and Learn Your Skills: State-Only Imitation Learning via Decoupled Policy Optimization
Authors: Minghuan Liu
Zhengbang Zhu
Yuzheng Zhuang
Weinan Zhang
Jian Shen
Jianye HAO
Yong Yu
Jun Wang
Venue: NeurIPS 2021 Submitted (2021)
Keywords: reinforcement learning
imitation learning

Abstract: State-only imitation learning (SOIL) enables agents to learn from massive demonstrations without explicit action or reward information.
However, due to the agnostic executed skills caused by the incomplete guidance, matching the state sequences of the expert by a state-to-action mapping is ambiguous.
In this paper, we overcome this issue by introducing hyper-policy as sets of policies that share the same state transition to characterize the optimality in SOIL. Accordingly, we propose decoupled policy optimization (DPO) via explicitly decoupling the state-to-action mapping policy as a state transition predictor and an inverse dynamics model. Intuitively, we teach the agent to plan the target to go and then learn its own skills to reach. Beyond simple supervised learning objectives, we also analyze the compounding error caused by both parts and employ effective solutions to reduce it. 
Experiments on challenging benchmarks and a real-world driving dataset demonstrate the effectiveness of DPO and its potential of bridging the gap between reality and reinforcement learning simulations.

Plan Your Target and Learn Your Skills: State-Only Imitation Learning via Decoupled Policy Optimization
Anonymous Author(s)
Affiliation
Address
email
Abstract
State-only imitation learning (SOIL) enables agents learn from massive demon- strations without explicit action or reward information. However, previous methods attempt learn the implicit state-to-action mapping policy directly from state-only data, which results in ambiguity and inefficiency. In this paper, we overcome this issue by introducing hyper-polle policy as sets of policies that share the same state tran- sition to characterize the optimality in SOIL. Accordingly, we propose Decoupled Policy Optimization (DPO) via explicitly decoupling the state-to-action mapping policy state transition predictor and an inverse dynamics model. Intuitively, we teach the agent to plan the target to go and then learn its own skills to reach. Experiments standard benchmarks and real-world driving dataset demonstrate the effectiveness DPO and its potential bridging the gap between reality and simulations of reinforcement learning.
11
Introduction
13
14
Imitation learning offers a way to train an intelligent agent from demonstrations by mimicking the expert's behaviors without constructing hand-crafted reward functions [7] The corresponding methods normally require the expert demonstrations include information of both states and actions. Unfortunately, the action information is not always accessible from many real-world demonstration resources, e.g., online video recordings of car driving or sports. Thus a natural desire to take advantage of these massive and valuable resources motivates the study of state-only imitation learning (SOIL),
15
16
17
18
20 also known as learning from observations (LfO) 24 Analogy to human beings, SOIL a more
21 intuitive way to approach imitation by only matching the expert's 's state sequences without having
22 explicit knowledge of the exact actions.
23 A wide range of algorithms have been proposed to solve SOIL by matching the state sequence of the
24 expert 22 25 However, the action agnostic setting in SOIL makes challenging determine
25 the optimal action because of the partial observability of the expert demonstrations that multiple
26 policies could be chosen to match the same expert state sequence. Thus learning a state-to-action
27 policy is implicit, leading to less efficient modeling of the explicit information from demonstrations,
28 and in result could cause suboptimality.
29 To this end, in this paper, we introduce the concept of hyper-policy denoting a family of policies that
30 share the same state transition. Based on that, instead of recovering the expert policy, characterize
31 the optimality in SOIL by finding the expert hyper-policy. The proposed method is called decoupled
32 policy optimization (DPO), which separates the policy into two modules: an expert state transition
33 predictor that finds the optimal hyper-policy, followed by an inverse dynamics model that builds the
34 executable policy deliver actions. Intuitively, the expert state transition predictor predicts the target,
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

35 while the inverse dynamics model enables the agent to learn its own skills to reach the target. DPO
36 takes the advantage of such a decoupled structure by separately learning two kinds of data: (1) the expert state transition that is directly accessible in the demonstration; (2) the action to be performed which should be obtained by interacting with the environment.
37
38
39 To ensure the benefit of DPO, these two modules should work coherently to provide accurate foresight for targets and corresponding skills. To achieve this, we regularize the state transition predictor to prevent the model from predicting non-neighboring states via multi-step and cycle training style. Further, to improve the learning efficiency by encouraging the agent to reach the expert states, we augment reward and apply policy gradient to DPO with additional generative adversarial objective.
40
41
42
43
Experiments standard benchmarking tasks show the advantage of the decoupled structure and the higher efficiency of DPO. We also evaluate DPO on a real-world driving dataset with state-only demonstrations, and the result shows that DPO can learn driving behaviors closer to human drivers when compared with baseline methods.
45
47
Preliminaries
48
49
Markoy Decision Process. We consider y-discounted infinite horizon Markov decision process
50
(MDP) as a tuple M (S, A, Po: r, 7). where S is the of states, AA represents the action space. Pu. T S x A x S -> [0, 1] is environment dynamics distribution, Po S - [0, 1] is the distribution of
51
52 the initial state 8o. and y E [0, 1] the discount factor. The agent makes decisions through a policy
53 x (als): S x A + [0, and receives rewards r S A R.
Occupancy Measure. The concept of occupancy measure (OM) 10 is proposed to characterize the statistical properties of a certain policy interacting with an MDP. Specifically, the state OM is 56 defined as the time-discounted eumulative stationary density the states under given policy m.
55
57 Pa (8) = Ei P(( = a/m). Following such a definition we can define different OM:
58 a) State-action OM: (r(s, a) = ((a)s)px(s)
59 b) State transition OM: Ps (s, - Pa aT
60
c) Joint OM: Pm a, 8') (TT'
61
Imitation Learning from State-Only Demonstrations. Imitation learning (IL) [3] studies the
62 task of learning from demonstrations (LfD), which aims to learn policy from expert demonstrations without getting access to the reward signals. The expert demonstrations typically consist of expert state-action pairs. General IL. objective minimizes the state-action OM discrepancy:
63
64
arg min Ea-ep [ (-)), (-[s))] + arg min & (pr (s, a), Pa (s)
(1)
65
where denotes some distance metric. For example, GAIL [1O] chooses to minimize the JS divergence (Ps(Prs (s, a) a)) a)), and AIRL (5] utilizes the KL divergence DxL (Prg (3. a)ll (s, a)) instead, which corresponds to a maximum entropy solution with the recovered reward [17] However, for the scenario studied in this paper, the action information is absent in state-only demonstrations, known as state-only imitation learning (SOIL) or learning from observations (LfO) problems, where the action
67
69
70 spaces between the expert and the agent can even be different. Such challenges prevent applying typical IL solutions. An existing method for this problem is to instead optimize the discrepancy of the state transition OM with the state-to-action policy Talss) 23
71
72
x
= arg min[E (pai (s, s Pm ss))s'))))
(2)
73 However, the solution to this problem is ambiguous since there is no one-to-one correspondence between p(s, s') and и as we will show in the following section. As such, the optimality of SOIL should be reconsidered.
74
75
3
Methodology
78
3.1
Characterizing the Optimality in SOIL
78 In standard II. tasks, when the expert actions are accessible demonstrations, perfectly imitating the expert policy corresponds to matching the state-action OM due to the one-to-one correspondence

---

80
between and Pn (s. a) 10 21 However, such correspondence is not applicable for the state transition OM matching SOIL.
81
82
Proposition 1. Suppose 1I is the policy space and P = {p: 0} is feasiible set of OM, then policy π E II corresponds one state transition OM px P. However, a state transition OM p E P can correspond to more than one policy in II.
83
84
85
The proof can be found in Appendix B As a result, if we choose to optimize state-to-ace -t-tttto- mapping policy, then the optimal solution to Eq. (2) is ambiguous. The ambiguity also comes from the fact that Eq. (2) does not correspond to maximum policy entropy solution as in normal IL tasks (see Appendix C for details). Therefore, a state-to-action mapping function may be too implicit for matching the state sequence, which could cause training instability and lead to sub-optimal policies. 90 In that case, we must find a one-to-one corresponding solution to solve SOIL explicitly and efficiently. Before continuing, we introduce the definition of hyper-policy.
86
87
88
92
Definition 1. hyper-policy Q is set of policies such that for any TT, 42 E S2 we have Px, (s. s') =
93
Pr (s, s.).
Then by definition, there is one-to-one correspondence between the hyper-policy Ω and the state transition OM po((s,'), Similar to the normal state-to-action mapping policy, a hyper-policy St can be regarded as state-to-state mapping function (s'ss( which predicts the state transition such that for any π E Ω::
95
96
97
( ha(s'(s) [g Pa(s, ds
- (6(
98
Proposition 2. Suppose the state transition predictor is defined as in Eq. 3 and I its space, 99
P {p: p 0}. then hyper-policy state transition predictor ha E I corresponds one state transition OM pQ E P, and a state transition OM p E P only corresponds to one hyper-policy state transition predictor such that hp p(s,s')) J3 p(s, ds.
101
103
Therefore, we find a one-to-one correspondence between the optimization term p(s, s') and a practical target (s'ss,,, which indicates that we do not have to infer the expert actions under state-only demonstrations but only need to recover the state transition predictor of the hyper-policy Ωg:
105
arg min[{ (pag (s, pa(s,,
arg min Ea-n! [ (hog (s'(s), (s(s'(s))]
(4)
ha
However, SOIL still requires to learn policy interact with the MDP environment to match the state transition OM of the expert. This is achievable since we do not have to recover the expert policy Tg exactly but can learn any policy ı fgg according to Eq. (4
107
109
3.2 Policy Decoupling
To construct unambiguous objective for SOIL,, we define hyper-policy and solve the problem by finding the state transition predictor of the expert hyper-policy. Intuitively, this tells the agent the
111
112 targer that the expert will reach without informing any feasible skill that require the agent to learn
113 itself. Therefore, to recover π E Ωgg we can construct an inverse dynamics such that
π
T
T(TE)
(5)
Inverse dynamics Expert state transition prodictor 114 Formally, the expert policy can be decoupled as
TE(e|s)= = /. T(s'ls, a)Te(als) ds' :o'==fo =f. PTo PTo (x, do' = f Pr (rg(s,)')Tx((all ds' (rg(s)
(s)
(6)
= / Aag (s)(ss) (als,, d's
115
Notice that both the state transition predictor h and the inverse dynamics model / is policy dependent. 117 Nevertheless, recall that the optimality in SOIL only requires us to recover π E Qgg we do not have
to learn about Lag but just one feasible skill [aa)s, Then policy can be recovered by
((als,s))
hig (s'
(7)
skill
3

---

119
Here the inverse dynamics model offers an arbitrary skill to reach the expected targer state provided by the state transition predictor h. In fact, it does not depend on the hyper-policy 2g but sampling policy æø to construct 7 ag We only need a mild requirement for Tв that it covers the support of P0g (8,s') 5') so that the learned I can provide a possible action to achieve the target state. In both experiments and theoretical analysis we show that this require- the inverse
Deccupted Poliy
Insicenant
120
Expert State Transiion Predictor
Iwerie Dynamic Model
122
123
124
126
vrlvence Faww
Expart Devronstioiin
Training Row
128
ment alleviates the dependence dynamics. Furthermore, if the environment and the expert policy are both deterministic (which usually the case in real-world scenarios such robotics), the state transition is single-point distribution (or known the Dirac delta func- tion), and we can simply model / as a deterministic function. By decoupling the policy, which is ttatttto-action mapping function, as state-to-state mapping function (the transition predictor) and slate-pair-t--acte
The architecture of Decoupled Policy Opti- mization (DPO), which consists of an expert state tran- sition predictor (to plan where go) followed by an inverse dynamics model (to decide how to reach).
Figure 1:
130
132
134
135
136
mapping function (the inverse dynamics model), we can mimic the expert policy from state-only demonstrations by optimizing these two modules. The whole architecture is illustrated in Fig.
137
138
139
State Transition Predictor. In practice, we construct a parameterized expert state transition predic- tor be which predicts the subsequent state of the expert taking the input as a current state á = hop (s).
140
141
The state transition predictor models the explicit information of the expert, and it can be learned from KI. divergence minimization:
142 the demonstration data only. Thence, we implement Eq.
min E (,,""~~ DKL (ha, (s' (s)ss)))
which can be optimized in supervised manner. Specifically, sample state transitions (s, s') from the expert demonstrations D and optimize the L2 loss:
144
c E (A, []]s' - he (s))
145
Inverse Dynamics Model. Knowing where to go is not enough since the agent has interact with the environment to reach the target. This can be achieved via an inverse dynamics model, which predicts the action given two consecutive states. Formally, let the d-parameterized inverse dynamics model To take input the state pair and predict the feasible action to achieve the state transition: = (s, s'). Intuitively, we want the inverse dynamics to learn from possible transitions sampled by the agent. Recall that we only need the support of learned / [a)s. s, s') of the sampling policy covers the support of the expert state transition OM, from which we can infer at least one possible action. Hence, we can optimize the KL divergence between the inverse dynamics of sampling policy тв
147
149
151
159 and Iz:
min (s,,"")~ [Drr
(xxi(Iep (Tr (al 8, s'
( (alss,
(10)
and we can choose to optimize L2 loss in supervised manner by sampling from the replay buffer B:
C/ (',,,"''> [Ila (s s
(11)
In implementation, both the state predictor and the inverse dynamics can be constructed as Gaussian distributions similar to a normal stochastic policy, thus encouraging exploration.
156
157
3.3
Tackling Compounding Error Challenges
In our formulation, have decoupled the cate-to-action mapping policy as state-to-state mapping function and a state-pair-te perce-ppii---teee sair-to--ction mapping function. Unfortunately, the compounding error problem exists such that the agent cannot reach where it plans due to the fitting errors of these two parts.
159
Theorem 1 (Error Bound of DPO). Consider deterministic environment whose dynamics transition function T(x, a) deterministic and T.-Iipschit Assume the ground-truth state transition bag (s) is deterministic, and for each policy E 11, its inverse dynamics Lx is also deterministic and
162
4

---

184 C-Lipschitz. Then for any state the distance between the desired state s'g and reaching state s' sampled by the decoupled policy is bounded by
185
I| -''' S LCIlbas (s) (5)()) + (s. - [s(ss'')l
(12)
186
where m is a sampling policy that covers the state transition support of the expert hyper-policy and 8' h (8) is the predicted next state.
167
168
The proof can be found in Appendix where we also induce a similar error bound for rollout with state- to-action policy as BCO [22] to show the advantage
SOW Losk
189
170
of the decoupled structure. From Theorem [ we know that the compounding error can be enlarged
171
Figure 2: Multi-step optimization. Given an ex- pert state sg. he predicts the next possible state 8' which is further fed to target network her to predict the following sequence. The total loss computes the MSE loss along the state sequence. where it plans to. To alleviate the error, we further propose regularization on these two modules.
172
173
due to each part's ss fitting error, where the first term corresponds the error of predicted states and the second term indicates whether the agent can reach
174
175
176
177
3.3.1
Regularization on Target Planning
178
One major problem is that the state transition predictor may suggest non-neighboring states instead of predicting one-step reachable states. To overcome this, we draw inspiration from Asadi et al. [2] and Edwards et al. [31, and regularize state transition predictor to prevent the model from predicting noinneigg states via multi-step and cycle training style.
179
180
181
182
Multi-Step Optimization. We first explain the details of the multi-step optimization objective. This idea is motivated by Asadi et al. 12]. which optimizes a multi-step outcome by executing a sequence of actions the dynamics model. Here we optimize the state sequence instead. As shown in Fig. 2 given an expert state sg, he predicts the next possible state s' that the expert will reach; the predicted state is then fed into the predictor to output the predicted two-step state s" As such, the multi-step training loss is the L2 loss computed along the k-step outcome sequence:
183
184
185
186
187
hs.ms 4
E DDD
11/ - (x)) +
1108 (s
(13)
(s. C
188
Intuitively, such a regularization makes the state prediction 3' close to the expert state distribution in order to make accurate long step predictions. It is worth noting that the gradient of the cascading state transition predictors should be dropped since we already have an accurate input at each time step, and for each training step, we only update the first one. We use a target network har in practice.
Mulf'le o
190
SE
Nateis.
192
194
Figure 3: Cycle training style. Given an ex- pert state sg, [s 8') takes input the pre- dicted state 3' and sg to get the exccution action then additional forward dynam- ies model M,, is used simulated one step rollout using (#5, a) and get a forward next state &'. The total loss computes the MSE loss between the two predicted states.
Cycle Training Style. Another way to regularize the transition predictor's output to a neighboring state is to keep additional function to ensure the eycle consistency, which is also an important technique in (3. In particular, illustrated in Fig. 3 given expert state sg, take the predicted state s' and &E into the inverse dynamics and get the action a, then train additional forward dynamics model M., to simulate one step rollout that takes the input (Sg, a) and gets a forward next state 8/:
196
198
200
202
CM = ((,, [|]s" as
(14)
cl hyyycee E, E [[]s' (s)) + [l((s) (s.ss
203 In other words, the cycle training scheme provides a regularization on hss to make predictions consistent with the forward dynamics model.
5

---

205
3.3.2
Efficient Skills Learning via Decoupled Policy Gradient
206 In previous sections, we have mentioned that learning to reach specific place requires the data- collecting policy to cover the support of the expert hyper-policy. This is easy to achieve on simple low -wwddiennionnl tasks, but may not be satisfied in high-dimensional continuous environments. To this end, we encourage the agent to approach those state transitions from the expert's 'ss hyper-policy Ag by minimizing the JS divergence of the state transition occupancy using a to-e-to- mapping s, s') (s, s')). This can be done by producing informative rewards via GAN-like methods [10 [3) and updating the decoupled policy with policy gradients (PG).
207
209
211
policy DJs ( (8,
213 In detail, we construct a parameterized discriminator Dw(s, s') to compute the reward r(s, a) A r(s, s') as log D (s, s') and the decoupled policy served as the generator. In addition, since we decouple the policy as two parameterized modules. i.e., a state transition predictor and an inverse dynamics model, then by chain rule, the PG for the decoupled policy can be accomplished by
214
215
216
V g.v Err [Q(s, a) Vo.
d.v log (a)s(
Ex [((s.a) /. (v log (s'ls) + V log (s(as, ds'
(15)
217
where Q is the state-action value function; the first term is the gradient for updating the state transition predictor; and the second term is for the inverse dynamics model. Thus, the optimization for both the state transition predictor and the inverse dynamics model can augment the supervised learning objectives with any (G-based learning algorithms (e.g., TRPO, PPO, SAC). As the training proceeds,
218
219
220
221
the agent will sample more transition data around Qgg and thus the support of the sampling policy will progressively cover the support of pag (s,
223
3.4
Overall Algorithm
224
By combining the idea of generative adversarial training, we obtain our final algorithm, composed with three essential parts: the state transition predictor h used for predicting the possible future states sampled by the expert; the inverse dynamics model used for inferring the possible actions conditioned on two adjacent states; and the discriminator D used for offering intermediate reward signals for training the decoupled policy π. = [hh). The overall objective of DPO is
225
226
228
6.v >c + An A1C
(16)
229
where ÀG, Xd and A are hyperparameters for trading off the training among each loss. In practice, we try less than ten combinations for these parameters as shown in Appendix D.3] and we directly optimize f7.y instead of iterative training. The detailed algorithm summarized in Appendix A
230
231
232
Besides, it is worth noting that both the inverse dynamics model and the state transition predictor
can be pre-trained, where we optimize c using the state-only demonstration and optimize Co using
233
234
samples collected by a randomized agent.
Table 1: Comparison between different methods.
4
Related Work
Method
Inverse Dynumies
State Predictor
Decoupted Policy
Task
235
BCO [2]
X
x
SOIL.
GAIO 23
SOI.
236
State-only imitation learning (SOIL) endows the agent with the ability to learn from expert states. Al- though lacking the expert decision information, most
SOIL
OPOLO
27
SOIL
PID-GAIL
11
X
II.
238
QSS
of the previous works still optimize state-to-action mapping policy to match the expert state transition distribution. For example, Torabi et al. [22) used model-based approach to apply behavioral cloning to state-only demonstrations, while Torabi et al. [23] employed a similar structure to GAIL
SAIL
16
IL
DPO (Ours)
SOIL
240
242
to match the state transition distribution. Yang et al. [25] analyzed the optimization gap between SOIL and naive II. and introduced a mutual information term to narrow it. Huang et al. [11] applied
244
SOIL autonomous driving tasks by decoupling the policy into a neural decision module and non dinfdifferentiable execution module in a hierarchical way.
246
Our work decouples the state-to-actio -ctiin policy into two modules. However, both the inverse dynamics model and the state transition predictor have been widely used by many previous works on RL. and II. tasks, For instance, Torabi al. [22] and Guo et al. [6] trained an inverse dynamics model to label the
248

---

Table 2: Eventual performance against different methods on 6 casy-to-hard continuous control benchmarks. The means and the standard deviations evaluated over more than 5 random seeds.
InvertedPeadulum
InverteDouliceeauullmm
Hopper
Walker2d
HalfCheetah
Random
25.28 1. 5.53
78.28 1: 10.73
13.09 0.10
7.07 a 0.13
74,08 1: 12.39
713.59 1. 203 92
BCO
1000.00 1: 0.00
413,01 1: 118.46
1130.6 1 198.81
261 36 25.17
-13.06 1: 179.91
397.19 1: 279.16
GALIO
1000.00
0.00
781K077 :: 1778.67
5068.10 26.32
3865 20 : 341.90
8953,35 4: 1079.1
5122.29
807. 19
GABO-DD
1000,00
7305.01
3031.84 152.13
4003.06 241.34
8675.42
5535.9
62.74
DPO (wlo PC)
1000.00
0.00
3545.70
738.
629.84 34
334.23 85.42
-172.00
132.81
-196.96
124.26
DPO (w PC)
1000.00
0.00
7846.40
+ 1541.20
3165.72 + 68.44
4407.53
+ 266.72
10801.96
438.01
5538,AN
+ 107.2
(SAC)
1000.00 0100
9.5387 + 0.10
5402.94 + 446.45
5639
+ 29.97
1371164
111,47
5204.5 + 15:0149
250
state-only demonstrations with inferred actions. Nair et al. [19] proposed a method for manipulating
251
ropes to match single
image sequence, in which an inverse dynamics model is
252
trained in a self-supervised manner and used to generate control signals. Kimura et al. [4] utilized a
253
state transition predictor to fit the state transition probability in the expert data, which is further used
254
to compute predefined reward function. Liu et al. [66] constructed policy prior using the inverse dynamics and the state transition predictor, but the policy prior was only used for regularizing the
255
256
policy network. However, as shown in this paper, the policy can be exactly decoupled as these two parts, which can be uniformly optimized through policy gradient without keeping
extra policy. Edwards et al. 3 estimated Q(s. s') for RL tasks which employs similar policy form as Eq. (6) and updates the state transition predictor through a deterministic policy gradient similar to DDPG [5]
258
280
To sort out the difference between these methods and ours, we summarize the key factors in Tab.
Experiments
261
We conduct four sets of experiments to investigate the following research questions:
263
RQ1
Is decoupled learning structure superior than state-to-action structure on SOIL tasks?
264
RQ2
Does DPO achieve higher efficiency or better performance than basclines on SOIL tasks?
285
RQ3
Can agent reach where it plans and does the proposed regularization help mitigate the compounding error?
266
Expert
DPo
267
RQ4 How can DPO be applied on real-world data?
268
To answer RQ1, we conduct toy experiments with a simple 2D grid world environment and compare both qualitative and quantitative results of DPO
289
270
against BCO and GAIfO, Regarding RQ2. we empirically evaluate DPO on easy-to-hard continuous control benchmarking tasks. And for RQ3, we evaluate the difference between the predicted states that the agent plans to reach and the consecutive state that the agent actually reaches in the environment for the proposed regularization. Finally, we try to imitate real-world traffic surveillance recordings in simulated environment to investigate RQ4, which shows the potential of using real-world data for
272
273
274
(a) Rollout density.
276
human behavior simulation. Due the space limit, we leave experiment details, additional results and ablation study in Appendix.
278
279
5.1
Understanding the Decoupled Structure
280
In this paper we design decoupled policy optimization (DPO) to perform SOIL tasks, and in previous sections we propose that the key technical contribution of DPO is the decoupled structure of policy that models the explicit state transition information and the latent action information from demonstrations, which solves the ambiguity and enhances the learning efficiency. Therefore, in this set of experiments, we aim to demonstrate how DPO is superior than Mate-to-action policy methods (RQI). We first generate expert demonstrations in a 2D 6x6 grid world environment, in which the agent starts at the upper left corner and aims to reach the upper right corner. In each grid the agent has k x 4 actions, which means that the agent has l possible actions to reach the neighboring block and in our experiment choose k- = 5 to enlarge the action space.
281
283
coon
(b) Loss curves.
285
Figure 4: Toy example.
287
289
290
291
The density of the expert trajectories and the trajectories sampled by different methods are shown in Fig. 4 a)). We show that both BCO and GAIfO have troubles in directly learning the implicit action the major trajectory and omit the other choice and BCO also stucks in the middle right. By contrast, DPO recovers the expert demonstrations
293
from state-only behaviors. Notably, GAIFO only imitates
7

---

InveriedPendullumx22
InverfaeDDebieeenndduua..
Hopper-v2
1000
3000
Aranaged retum
800
Averaged return
2000
Averaged retur
12000
1000
1000
200
2000
1.6c6
1.82
182
Walker-v2
HallChhetah-v2
Amb-v2
10000
5000
Avergged relurn
3000
Averoged return
Averoged return
4000
6500
5000
2000
4000
2000
2000
1000
0
0.6
1.2 stops
2.0
0.5 0.6
0.9
1.5 211 244 steps
0x7
0.2 0.4
0.6 1.0
1.2 1.4 1.07
steps
GAID Figure 5: Learning curves casy-to-hard continuous control benchmarks, where the solid line and the shade represent the mean and the standard deviation of the averaged return over more than 5 random seeds. We pre-train BCO and DPO for 50k steps and show it in figures.
DPO (w PGI
DPO (wls PG)
GAMD De
BCO
295
much better, benefiting from the decoupled structure that first determining the target and then taking the action achieve it. To further illustrate the learning efficiency advantage of DPO, we illustrate the IS divergence curves of DPO and BCO during training in Fig. bb. Besides, we show the policy loss
286
287
298
for BCO, the state predictor (SP) loss for DPO, and the inverse dynamics (ID) loss for both methods.
299
Except that the JS divergence of DPO decreases more quickly than BCO, it is also observable that
300
DPO relies less on the inverse dynamics than BCO, since the inverse dynamics loss of DPO converges
301
to higher level. We further provide a theoretic analysis of the dependence on inverse dynamics with
302
BCO and DPO in Appendix
5.2
Comparative Evaluations
We compare the qualitative results of DPO against other baseline methods on easy-to-hard continuous control benchmarking environments (RQ2), including InvertedPendulum, InveriedDoubbeeeedduum Hopper, Walker2d, HalfCheetah and Ant. In each environment, besides GAIfO and BCO, we also evaluate GAILO with decoupled policy (denoted as GAIFO-DP). For DPO we compare the reward augmented version of DPO (denoted as DPO w PG with the supervised learning version of DPO, ie., Ag = 0 (denoted as DPO w/o PG). For fairness, we re-implement all the algorithms based on a Pytorch code framework"] and adopt Soft Actor-Crite (SAC) (7] as the RL learning algorithm for
305
306
307
309
311
GAIFO and DPO. For all environments, we first train an SAC agent to collect 4 state-only expert trajectories and then train agents with such data. All algorithms
evaluated by deterministic policy. The eventual results are summarized in Tab. 2 and the learning curves are shown in Fig. [ It worth noting that for DPO, we choose the best performance among the experiments that use multi-step or cycle regularization, and we put the full experiment results in Appendix [
313
314
315
One can casily observe that on simple environments, BCO is able to achieve good performance, and GAIFO also does well on harder tasks. Even so, DPO can still gain the best or comparable performance against its counterparts. Particularly, without augmented reward, DPO is able to reach the optimality with the highest sample efficiency on simple tasks like InvertedPendulum. By contrast, on dimens-aimei tasks such as Hopper, Walker2d, HalfCheetah and Ant, it is difficult to construct accurate inverse dynamics that covers the support of the expert hyper-polic, from scratch. However, by combining generative adversarial policy gradients, the agent finally recovers good policy from the expert hyper-policy. This is particularly evident on HalfCheetah where DPO behaves poorly at the beginning but improves fast the training proceeds. Besides, as illustrated in Fig. 5
317
319
321
323
324
325
DPO benefits from better sample efficiency in most of the environments, but the improvements are
328
limited on the hardest tasks. We think that this may be due to larger state spaces (111 dimensions
327
for Ant) that makes it difficult to recover a good state predictor or an inverse dynamics model. In all
Without ambiguity we simply denote DPO for this version algorithm the following sections. Bttts///ithub.com/Kaaaaaaa com/
swiss
8

---

Hosparvi2
Adk-22
1200
Bape
1800 Reg
380 Rog
IPO Sy Reg
Figure 6: Compounding error of the predicted consecutive states and the real states the agent reaches when rollout in the cnvironments.
328
experiments, GAIfO-DP achieves similar results as GAIfO, indicating that the network structure does not count much for the performance.
330
5.3
Compounding Error Reduction
In this section, we aim to study whether the agent can effectively reach the target it plans (RQ3). Therefore, we analyze the distance of the reaching states and the predicted consecutive states, and draw the mean square error (MSE) along the training procedure in Fig. [ We also compare our regularization including multi-step optimization (denoted as M.S.-K. where k is the number of rollout
332
333
334
335
steps) and cycle training style (denoted as Cycle). Note that DPO needs at least 1-step rollout for training the state transition predictor. As shown in Fig- 6 the agent still has gaps to get to where it plans to, and the mismatch always deteriorates on harder tasks. Combining regularization can always achieve lower compounding error, and the cycle training is effective in most of the environments. In Appendix D.6, we further illustrate the correlation between the final performance and the distance.
336
337
338
339
340
5.4
Learn to Drive from Real-Worid Traffic Data
341
The rapid development of autonomous driving has brought a lot of demand for simulating and training RL agent in the simulator, which requires realistic interactions with various social vehicles 26). However, driver's detailed actions are not easily to obtain yet we adopt SOIL from a traffic surveillance recording dataset (NGSIM I-80 (8) that contains kinds of recorded human driving
342
343
344
345
trajectories. We wish to further examine the potential of DPO for decreasing the gap between the real world and simulation (RQ4). We utilize the simulator provided by Henaff et al. 9 our simulation platform and learn to imitate real-world driving behaviors. We compare DPO against
348
347
348
GAIfO and BCO. and choose Success Rate, Mean Distance and KL Divergence as evaluation metrics, Specifically, Success Rate is the percentage of driving across the entire area without crashing into other vehicles or driving off the road, Mean Distance the distance traveled before the episode ends, and KL. Divergence measures the position distribution distance between the expert and the agent.
350
351
352 As shown in Tab. 3 DPO outperforms baseline methods in all three metrics while possessing higher stability. The de- coupled policy allows the state predictor focus match-
Table 3: Performance on NGSIM 1-80 driv- ing task over 5 random seeds.
353
354
Method
Success Rate (%)
Mean Distance (m)
KL Divergence
355
ing the distribution of expert trajectories, thus achieving smaller deviations from the expert position distribution.
BCO GAItO
27.4 = 1.1
1298 = 2.0
24.4 : 22 11.5
356
77.5 0.8
188.3
357
Furthermore, since the policy gradient can be computed with non-differentiable inverse dynamics, we can generate stable action sequences [1 by replacing the inverse
DPO
80.3 + 0.5
192.7 = 0.6
9,5 1.8
358
Expert
100
210.0
359
360
dy tynami model with classical controllers, which can be generalized to realistic applications.
Conclusion
361
382
In this paper, we characterize the optimality and investigate the ambiguity problem in state-only imitation learning, and accordingly propose Decoupled Policy Optimization (DPO), which splits the state-to-action mapping policy into a state-to-state mapping state transition predictor and a state- state-pair-to- pair-to-action mapping inverse dynamics model. Furthermore, we employ regularization and generative adversarial methods to mitigate the compounding error caused by the decoupled modules. The flexibility of the decoupled architecture allows a wide range of interesting future works, such as replacing the inverse dynamics with a classic control module to produce stable control signals, learning specific skills with shared state transition and multi-task target learning with shared pre-
383
364
365
366
367
369
trained skills.

---

References
371
372
[1]
Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning. In Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004), 2004.
374
375
[2] Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L Littman. Combating the compounding-crrod problem with a multi-step model. arXiv preprint arXiv:1905.0 1905.133200 2019.
378
377
[3] Ashley D. Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q(s.s') with deep deterministic dynamics gradients. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 2020.
378
379
[4] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International Conference on Machine Learning, pages 49-58, 2016.
382
383
384
[5] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In 6th International Conference on Learning Representations, ICLR 2018, 2018.
385
386
387
[6] Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforce- ment learning with expert state sequences. In The Thirty-Third AAAI Conference Artificial Intelligence, AAAI 2019, pages 3739 0046,776,2222 2019.
389
380
[7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the 35th International Conference on Machine Learning, ICML 2018, pages 1856-1865, 2018.
381
392
383
[8] John Halkias and James Colyar. Next generation simulation fact sheet. US Department of Transportation: Federal Highway Administration, 2006.
394
395
99] Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model Model-predictive policy learning with uncertainty regularization for driving in dense traffic. In 7th International Conference Learning Representations, ICLR 2019, 2019.
397
398
[10] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29, pages 4565-4573, 2016
399
400
[1]] Junning Huang, Sirui Xie, Jiankai Sun, Qiurui Ma, Chunxiao Liu, Dahua Lin, and Bolei Zhou. Learning a decision module by imitating driver's 's control behaviors. In Proceedings of the Conference on Robot Learning (CoRI.) 2020, 2020.
402
403
[12] Sandy H. Huang, Nicolas Paperot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial attacks neural network policies. In 5th International Conference Learning Representations, ICLR 2017, 2017.
405
406
[13] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning: survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1-35, 2017.
407
408
[14] Daiki Kimura, Subhajit Chaudhury, Ryuki Tachibana, and Sakyasingha Dasgupta. Internal model from observations for reward shaping. arXiv preprint arXiv:1806.01266 2018
[15] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In 4th International Conference on Learning Representations, ICLR 2016., 2016.
411
412
413
[16] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment- imitation learning. In 8th International Conference on Learning Representations, ICLR 2020, 2020.
414
[17] Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. In 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021,
416
2021.
10

---

[18] Leland McInnes and John Healy. UMAP: uniform manifold approximation and projection for dimension reduction. CoRR, abs/1802.03326, 2018.
419
420
[19]
Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Combining self-supervised learning and imitation for vision-based rope manipulation.
422
In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, pages 2146- 2153, 2017.
[20]
nonparametric estimates of density function. The Annals of Mathematical Statistics, pages 832-837, 1956.
Murray Rosenblatt. Remarks
425
428
[21] Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear programming. In Machine Learning. Proceedings of the Twenty-Fifth International Conference (ICML 2008), pages 1032-1039. 2008
427
[22] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In Proceedings of the Twenty-Seventh International Joint Conference Artificial Intelligence, IJCAI 2018, pages 4950-4957, 2018.
430
[23] Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from state-only demonstrations. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS 19, pages 2229-222222222222222222222222222 2231,,,, 2019.
433
[24] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from observation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, pages 6325-6331, 2019.
436
437
438
[25] Chao Yang, Xiaojian Ma, Wenbing Huang. Fuchun Sun, Huaping Liu, Junzhou Huang, and Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagree- ment. In Advances Neural Information Processing Systems 32, pages 239-249, 2019.
440
441
[26] Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang, al. Smarts: Scalable multi-agent reinforcement learning training school for autonomous driving. In Conference on Robot Learning, 2020.
443
444
[27] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from observations. In Advances Neural Information Processing Systems 33, 2020.

---

Checklist
448
447
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
448
contributions and scope? [Yes]
450
(b) Did you descrihe the limitations of your work? [Yes] See Section
451
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See
Section
452
453
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results... 455
456
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theorem
457
(b) Did you include complete proofs of all theoretical results? [Yes] See Section B
458
3. If you ran experiments..
459
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix [D.3]
462
463
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We ran our results with more than 5 random seeds as said in Section 5.2
464
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [No]
467
468
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
469
(a) If your work uses existing assets, did you eite the creators? [Yes] We re-implement all
470
algorithms based on an existed code base, as said in Section 5.2
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or URL? [No] We will public our codes after publication.
473
474
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
475
476
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
478
5. If you used crowdsourcing conducted research with human subjects...
479
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
481
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
482
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
484
12


Reviews:
[{"review_id": "yoHM0rv5FsZ", "reviewer": "Reviewer_wdvB", "summary": "The paper presents a method for state-only imitation learning (SOIL) that decouples learning the expert state-transition distribution from the actual policy needed to reach desired states. In particular, given a dataset of expert state sequences, the method trains two models: one that predicts next state given the current state, and another one that learns inverse dynamics, i.e. which action is needed to reach the next state from the current state. Where the state prediction can be directly learned from the state-only expert data, environment interactions are collected using a sampling policy to learn the inverse dynamics model. Additional techniques, such as multi-step optimization and cycle training are used to improve prediction accuracy of the state-transition model. In addition, training is improved by introducing a discriminator with a GAN-like reward that can provide intermediate reward signals to the policy. The method is evaluated on several MuJoCo continuous control environments, where it is shown to outperform other SOIL baselines, and on a real-world human driving dataset where it can learn effective driving policies just from human driving trajectories. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- State-only imitation is important for initializing robotic policies without access to ground truth actions, especially when imitating non-robotic agents like humans.\n- The paper is well-written, and straightforward to understand and follow. Mathematical derivations are coherent.\n- The method is shown to outperform multiple SOIL baselines and work with a real-world dataset.\n\nCriticism\n- One of the motivations for the presented work, which is discussed in Section 2, is that there is often not a one-to-one correspondence between expert state transitions and the underlying policy. However, as described later in the paper, both state-transition function and the inverse dynamics model are deterministic, which prevents them from representing a multi-modal policy. It would be interesting to see a discussion how the ambiguity of the underlying expert policy can be represented within this framework.\n- To better understand scalability of the presented approach, it would be interesting to see evaluations on higher-dimensional problems, e.g. using visual inputs.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed limitations of their work. The proposed approach does not lead to any obvious negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "dkX3HSVu8jw", "reviewer": "Reviewer_GQAi", "summary": "This paper proposes a decoupled policy approach to state-only imitation learning, wherein the policy is decomposed into a next-state prediction network, which aims to predict the desired next state given the current state under the expert policy, and an inverse dynamics module, which given the current state and the desired next state, predicts the action. The utility of this split is that it allows for additional supervision during policy optimization, as the state transition network can be trained to match the observed expert transitions, and the inverse dynamics module can be trained on the transitions in the trajectories obtained during policy optimization. These supervised losses are placed on top of a standard policy gradient method, as gradients to both components of the decoupled policy can also be obtained through a standard likelihood-ratio policy gradient applied to a reward function corresponding to likelihood of the executed state-transitions being drawn from the expert policy. This reward function is obtained by jointly training a discriminator to classify sample transitions from the learned policy from those from the expert data.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, this presents a novel approach to state-only imitation learning, which, to the best of my knowledge seems to be different from prior work primarily due to the fact that it combines state-only adversarial imitation via policy gradient (as in GAIfO) with a decoupled policy structure which allows additional supervision to each component separately. Intuitively this makes sense, and in the experiments the authors show that the approach can achieve better performance than baselines. I also appreciated the comparisons with GAIfO applied to the same decoupled network policy to show that the improved performance comes from the additional supervision, and not just the network architecture. I also appreciated the system diagrams showing both the overall framework and the regularization schemes.\n\nThe paper however also has some weaknesses in its clarity and experimental evaluation.\n\nFirst, I found the organization of the paper to be confusing. The authors spend a large portion of the paper discussing the fact that the solution to the SOIL optimization problem may not be unique. While I agree with this analysis, I do not agree that this lack of uniqueness should pose problems for optimization -- gradient based optimization works well even if there might be a family of equally performing policies. The notion of \"optimality\" stays the same, as the authors aren't arguing that the objective of SOIL should be reconsidered. Instead, the purpose of the paper is to argue for an parameterization of the policy which takes into account the non-uniqueness of the solution. \nFor this reason, section 3.1 felt superfluous: The end result (equation 4) is a fairly direct conclusion from the problem statement of SOIL.\n\nIn contrast, sections 3.3 and 3.4 felt lacking in detail. In the discussion of the multi-step optimization, it was unclear why certain gradients were dropped -- the reasoning \"we already have an accurate input at each time step and for each training step, we only update the first one\" was not clear. Furthermore, in equation 15, it is unclear how the Q function is estimated, and also how the integral over $s'$ is computed in practice. The details in the appendix also do not provide this information. In 3.4, it is unclear what it means to \"directly optimize $\\mathcal{L}^\\pi_{\\phi,\\psi}$ instead of iterative training.\" Also the authors state in the appendix that pre-training does not help much, and is can sometimes hurt performance, and indeed, only a few environments use a pretraining step. This should be made clear in the body of the paper, as it seems to be an important factor in performance.\n\nThe algorithm seems to be designed for deterministic environments: this is an assumption in the theorems, and the MSE-based regularization schemes do not take into account stochasticity in the dynamics or policy. While the approach seems to be effective on NGSIM environment, which is stochastic, it is evident from the appendix that neither form of regularization was used. The authors should be more transparent about this limitation of their work in the body of the paper.\n\nFinally, it is unclear what procedure was used to tune hyperparameters for the experiments. The hyperparameters vary quite a bit from task to task, with different forms of regularization used, different forms of the reward function, different amounts of pre-training, and different amounts of gradient penalty. This suggests that the performance of the proposed method is quite sensitive to hyperparameter selection. Indeed, this may also be a flaw of prior work in this domain (as GANs are notoriously difficult to tune), but such limitations should also be more transparently discussed in the body of the paper.\n\nMinor comments:\n\t- Proof of proposition 1 is not rigorous -- it should either be made rigorous, or the body of the paper be revised to say \"proof sketch is in the appendix\"\n\t- Tables should bold all entries that could statistically be the best -- for example, in Table 2, InvertedDoublePendulum, GAIfO and DPO (w PG) have overlapping confidence intervals, so bolding only DPO (w PG) is misleading.\n\t- In the toy experiment, it is unclear what the effect of the added action ambiguity achieves. It would be good to also include experiments comparing to k=1 to show if empirically the baseline algorithms suffer from environments with action ambiguity.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The discussion of limitations and negative societal impacts are minimal and inadequate. The limitations of an approach go beyond error bounds, but also into more practical considerations. I would encourage the authors to include a broader discussion, as I state earlier. Furthermore, while the checklist says that section 6 includes a discussion of potential negative societal impact, I do not see any discussion to this effect there.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "bRKRiM4CO76", "reviewer": "Reviewer_M4Kk", "summary": "The paper proposes a new method for state-only imitation learning, based around explicitly representing the transition from one state to the next (based on state-only expert demonstrations) and an inverse dynamics model providing feasible actions to produce that transition (based on rollouts from a sampling policy). Theoretical analysis attempts to show that this reduces policy ambiguity. Empirical results show improved performance relative to baselines, with the most striking results in a gridworld and on real-world driving data: on standard MuJoCo continuous control tasks, the mean of the method tends to be higher, but the confidence intervals overlap with baselines.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "There are many data sources that only provide observations. To exploit such data sources, we need to be able to learn from data without actions (the focus of this paper) and from partially observable data (not the focus). The basic idea of exploiting the structure of the problem by separately learning to predict the expert state transitions and to predict actions that lead to them is plausible. Unfortunately, the paper is confusingly written, key theoretical details are either incorrect or stated too informally to verify, and the empirical results while promising are not fully convincing. Given these deficiencies, I do not think this manuscript is suitable for presentation at NeurIPS.\n\n# Issues with derivation\n\nThe proof for Proposition 1 (appendix B) is very informal. While the overall claim seems approximately correct, it is missing some key assumptions (which you would likely have noticed had the proof been written rigorously). In general one may not be able to recover the policy from the state-action OM, since the OM $\\rho(s,a)$ may be zero for all actions $a$ transitioning out of a particular state $s$. For example, this can occur if $s$ is not reachable from the starting state of the MDP. Granted you may not care about this unindentifiability (although it does matter in the transfer case), but it does mean there is policy ambiguity even in the state-action case, and there is no one-to-one correspondence. Additionally, the “proof” that there is no one-to-one correspondence between $\\rho(s,s')$ and $\\pi(a\\mid s)$ seems to be missing a step – simply writing down the marginal formula does not establish this. Indeed, there exist MDPs where the transition dynamics are invertible, so that we can establish $(s,a)$ from $(s,s')$.\n\nDefinition 1 is also rather imprecise. A literal reading implies that a hyper-policy is any set of policies where all policies have the same state transition OM. This would include, of course, the empty set and all singleton sets of policies. In this case the following claim that “there is a one-to-one correspondence between the hyper-policy $\\Omega$ and the state transition OM $\\rho_{\\Omega}(s,s')$ is clearly false. For any $\\Omega$ with size $|\\Omega| > 2$, any subset of it is still a hyper-policy and has the same OM. I think you need the hyper-policy to be the set of *all* policies with this OM for this claim to go through.\n\nProposition 2 is stated without proof, and I do not believe it to be true due to the issue discussed with the Definition, although the intuition behind it seems plausible so I imagine this can be fixed.\n\n# Experimental results\n\nI appreciated the gridworld experiment -- although simple, this does provide a striking qualitative illustration of the difference between the methods. However, it would help to have some discussion as to why your method performs better in this case. In particular, I suspect the GAIfO failure mode was due to mode collapse, which often happens with GAIL (and GANs more generally). Should we expect your algorithm to systematically avoid these problems, or will similar problems occur given enough task complexity?\n\nTable 2: It appears the confidence interval for DPO overlaps with the confidence interval for at least one other method in every column. Although the means being >= other methods in 5/6 cases is certainly promising, it would be more convincing if the difference was statistically significant. Can you run more seeds to reduce variance?\n\n# Minor points\n\nCorrectness:\n\n- “Furthermore, if the environment and the expert policy are both deterministic (which is usually the case in real-world scenarios such as robotics)” – this seems to be claiming that human demonstrations are deterministic. While humans may be deterministic in a philosophical sense, unless you are observing all relevant state (how much sleep the human had, how caffeinated they are, etc) I very much doubt this will be true in practice.\n\n- “we also induce a similar error bound for rollout with a state-to-action policy as BCO [22] to show the advantage of the decoupled structure”. How is the decoupled structure an advantage when the error bound is similar (not stricly less)?\n\nClarity:\n\n- Line 26: “state-to-action policy” – don't all policies mapping states to (distributions) over actions?\n\n- Line 30: “state transition” is unclear. Do you mean visitation distribution? State sequence?\n\n- Line 89-90: I did not understand any of this: “Therefore, a state-to-action mapping function may be too implicit for matching the state sequence, which could cause training instability and lead to sub-optimal policies. In that case, we must find a one-to-one corresponding solution to solve SOIL explicitly and efficiently.”\n\n- Line 99: “and $\\Gamma$ is its space” – what does this mean? Clarity is always important, but is absolutely essential when stating definitions and theorems.\n\n- Would be helpful to introduce BCO somewhere as it's referred to repeatedly in the experiments. The only place it is defined seems to be in a table in related work.\n\n# Update after rebuttal\n\nI would like to thank the authors for a detailed response to my and others reviews. The propositions are now much more clearly stated and although I did not have time to check every detail of the derivation for Proposition 1 it appears to now be sound. In light of this, I am increasing my score.\n\nThank for you running the additional experiments. In particular it is helpful to see from Figure 3 that BCO learns the correct behavior (albeit slowly) when there is an invertible transition dynamics. I may be willing to increase my score further depending on the confidence intervals (see question below).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "There is no discussion of social impact, and little discussion of limitations -- there is a brief discussion of future work in the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "Ydo2lRBgRPl", "reviewer": "Reviewer_DMhm", "summary": "This paper tackles the state-only imitation learning problem by decomposing it into two components: a state-transition predictor and an inverse dynamics model. Specifically, it argues that the problem is difficult because there are potentially multiple policies that could match the same state trajectory of the expert. \n\nInstead, it proposes to learn what they call a hyper-policy, which is a family of policies that lead to the same distribution over state trajectories. Then, after recovering the optimal hyper-policy, it learns an inverse dynamics model to recover the actions, and this model can be learned by interacting with the environment.\n\nHowever, to actually achieve good performance in practice, they propose two regularizations, i.e., a multi-step prediction loss and a cycle training loss. The parameters of the hyper-policy and the inverse dynamics model are also trained with a policy gradient objective where the reward is based on a discriminator (which is trained to predict whether the transition (s, s’) is from the expert). ", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: \n- This paper decomposes the state-only imitation learning problem into learning a state predictor and an inverse dynamics model. As noted by the related work, this decomposition has previously been studied in the context of imitation learning [16] and reinforcement learning [3]. \n\n- Notably, the QSS work also shows that their method can be adapted to the state-only imitation learning setting. It would be good to clarify the differences and to include a comparison in the experiments if possible.\n\nQuality:\n- The method is compared to a couple of prior methods. Comparatively, their full method demonstrates better performance. However, it seems like the reward from the discriminator is what enables good performance, more so than the decoupled structure.\n\n- The evaluation is on Mujoco continuous control tasks and a simulated driving task. These have relatively low-dimensional state-action spaces and relatively deterministic dynamics, which makes learning a forward dynamics model, required by the cycle training, easy. I’m curious how well this would scale to higher-dimensional observations however.\n\nClarity:\n- The paper is overall well-written and organized, and provides a good overview of the existing works in the literature in the related work section and through Table 1.\n\nSignificance:\n- Imitation learning from observations is an important problem setting with real-world applications (as shown through the driving experiment).\n\n- Since the results are quite close and the differences less discernible in many of the Mujoco tasks, evaluation on a set of harder tasks would be more convincing.\n\n\n==========================\n\nThanks for the detailed response, and for the additional comparison to QSS. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes. See the main review for other limitations.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a method for state-only-imitation learning. The key ideas are: 
(a) Predicting future expert states from the current state. 
(b) Using the inverse model to transition from current to the predicted state. 
(c) Use of cycle consistency loss to improve the training of expert state prediction. 
(d) Error bound on tackling the challenge of compounding error. 
(e) Use of discriminator training to further improve performance. 

Two reviewers score the paper as 6, one as 7, and the other with a score of 5 to reject the paper. The considerations are: 

- Reviewers asked for a comparison to QSS and other baselines that the authors provided. 
- Authors clarified the proof of their propositions. 
- Authors clarified concerns about modeling stochastic distributions. 
- Reviewers were split on the clarity of the writing, that authors promised to improve. 

At the end of the rebuttal, M4Kk believes that updates both in terms of the writing and the experiments are substantial and therefore require another round of review. Considering the fact that the authors had addressed a majority of the concerns and the rebuttal incorporated all that would go in the revised version, I was initially inclined to recommend the paper for acceptance. 

However, studying the paper in more detail revealed an issue with the lack of comparison with prior work that I mentioned in my comment to the authors. After considering the author's response my opinion is: (details in the discussion and the thread below): 

The main premise of the paper is that state-only IL is hard because there are multiple policies consistent with a sequence of expert states. The paper proposes decoupling the learning into predicting expert states (i.e., what to do) and using the inverse model to follow the predicted expert states. However, as reviewer DMhm remarks, the most gain can be attributed to the use of discriminator rewards. The authors acknowledge this in the rebuttal, but also mention that additional supervision signals can be used due to the de-coupled policy structure which can improve performance/sample efficiency. The paper can thus be seen as proposing a set of ideas that encourage the imitator to follow the expert demonstrations better than a naive inverse model. In this view, 

- Comparison with an improved version of inverse models that uses cycle consistency [1] is critical. It is because both cycle consistency in training of inverse model and the proposed method of using discriminator rewards (possible due to de-coupling) are both encouraging state-matching, albeit in different ways. 

Further, if the proposed method outperforms [1], then the paper should be re-written to highlight that PG fine-tuning is critical (which experimentally leads to maximum gain, Figure 5) and decoupling allows this. The way the paper is written, it's hard to understand why de-coupling. 

In light of missing comparison and that paper would have gone a substantial re-write to incorporate suggestions from all the reviewers, I am siding by reviewer M4Kk, "revisions are substantial enough that I think the paper would benefit from another review cycle, especially given the general lack of polish in the initial submission."

[1] Zero-shot visual imitation, ICLR 2018.

Decision: Reject