Plan Your Target and Learn Your Skills: State-Only Imitation Learning via Decoupled Policy Optimization

Abstract:
State-only imitation learning (SOIL) enables agents to learn from massive demon1 strations without explicit action or reward information. However, previous methods 2 attempt to learn the implicit state-to-action mapping policy directly from state-only 3 data, which results in ambiguity and inefficiency. In this paper, we overcome this 4 issue by introducing hyper-policy as sets of policies that share the same state tran5 sition to characterize the optimality in SOIL. Accordingly, we propose Decoupled 6 Policy Optimization (DPO) via explicitly decoupling the state-to-action mapping 7 policy as a state transition predictor and an inverse dynamics model. Intuitively, 8 we teach the agent to plan the target to go and then learn its own skills to reach. 9 Experiments on standard benchmarks and a real-world driving dataset demonstrate 10 the effectiveness of DPO and its potential of bridging the gap between reality and 11 simulations of reinforcement learning. 12

State-only imitation learning (SOIL) enables agents to learn from massive demon-1 strations without explicit action or reward information. However, previous methods2 attempt to learn the implicit state-to-action mapping policy directly from state-only3 data, which results in ambiguity and inefficiency. In this paper, we overcome this4 issue by introducing hyper-policy as sets of policies that share the same state tran-5 sition to characterize the optimality in SOIL. Accordingly, we propose Decoupled6 Policy Optimization (DPO) via explicitly decoupling the state-to-action mapping7 policy as a state transition predictor and an inverse dynamics model. Intuitively,8 we teach the agent to plan the target to go and then learn its own skills to reach.9 Experiments on standard benchmarks and a real-world driving dataset demonstrate10 the effectiveness of DPO and its potential of bridging the gap between reality and11 simulations of reinforcement learning.12
1 Introduction13
Imitation learning offers a way to train an intelligent agent from demonstrations by mimicking the14 expert’s behaviors without constructing hand-crafted reward functions [13, 17]. The corresponding15 methods normally require the expert demonstrations include information of both states and actions.16 Unfortunately, the action information is not always accessible from many real-world demonstration17 resources, e.g., online video recordings of car driving or sports. Thus a natural desire to take advantage18 of these massive and valuable resources motivates the study of state-only imitation learning (SOIL),19 also known as learning from observations (LfO) [24]. Analogy to human beings, SOIL is a more20 intuitive way to approach imitation by only matching the expert’s state sequences without having21 explicit knowledge of the exact actions.22
A wide range of algorithms have been proposed to solve SOIL by matching the state sequence of the23 expert [22, 23, 25]. However, the action agnostic setting in SOIL makes it challenging to determine24 the optimal action because of the partial observability of the expert demonstrations that multiple25 policies could be chosen to match the same expert state sequence. Thus learning a state-to-action26 policy is implicit, leading to a less efficient modeling of the explicit information from demonstrations,27 and in result could cause suboptimality.28
To this end, in this paper, we introduce the concept of hyper-policy denoting a family of policies that29 share the same state transition. Based on that, instead of recovering the expert policy, we characterize30 the optimality in SOIL by finding the expert hyper-policy. The proposed method is called decoupled31 policy optimization (DPO), which separates the policy into two modules: an expert state transition32 predictor that finds the optimal hyper-policy, followed by an inverse dynamics model that builds the33 executable policy to deliver actions. Intuitively, the expert state transition predictor predicts the target,34
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.
while the inverse dynamics model enables the agent to learn its own skills to reach the target. DPO35 takes the advantage of such a decoupled structure by separately learning two kinds of data: (1) the36 expert state transition that is directly accessible in the demonstration; (2) the action to be performed37 which should be obtained by interacting with the environment.38
To ensure the benefit of DPO, these two modules should work coherently to provide accurate foresight39 for targets and corresponding skills. To achieve this, we regularize the state transition predictor to40 prevent the model from predicting non-neighboring states via multi-step and cycle training style.41 Further, to improve the learning efficiency by encouraging the agent to reach the expert states, we42 augment reward and apply policy gradient to DPO with additional generative adversarial objective.43
Experiments on standard benchmarking tasks show the advantage of the decoupled structure and44 the higher efficiency of DPO. We also evaluate DPO on a real-world driving dataset with state-only45 demonstrations, and the result shows that DPO can learn driving behaviors closer to human drivers46 when compared with baseline methods.47
2 Preliminaries48
Markov Decision Process. We consider a γ-discounted infinite horizon Markov decision process49 (MDP) as a tupleM = 〈S,A, T , ρ0, r, γ〉, where S is the set of states, A represents the action space,50 T : S ×A× S → [0, 1] is environment dynamics distribution, ρ0 : S → [0, 1] is the distribution of51 the initial state s0, and γ ∈ [0, 1] is the discount factor. The agent makes decisions through a policy52 π(a|s) : S ×A → [0, 1] and receives rewards r : S ×A → R.53
Occupancy Measure. The concept of occupancy measure (OM) [10] is proposed to characterize54 the statistical properties of a certain policy interacting with an MDP. Specifically, the state OM is55 defined as the time-discounted cumulative stationary density over the states under a given policy π:56 ρπ(s) = ∑∞ t=0 γ tP (st = s|π). Following such a definition we can define different OM:57
a) State-action OM: ρπ(s, a) = π(a|s)ρπ(s)58 b) State transition OM: ρπ(s, s′) = ∫ A ρπ(s, a)T (s
′|s, a) da59 c) Joint OM: ρπ(s, a, s′) = ρπ(s, a)T (s′|s, a)60
Imitation Learning from State-Only Demonstrations. Imitation learning (IL) [13] studies the61 task of learning from demonstrations (LfD), which aims to learn a policy from expert demonstrations62 without getting access to the reward signals. The expert demonstrations typically consist of expert63 state-action pairs. General IL objective minimizes the state-action OM discrepancy:64
π∗ = arg min π Es∼ρsπ [` (πE(·|s), π(·|s))]⇒ arg min π ` (ρπE (s, a), ρπ(s, a)) , (1)
where ` denotes some distance metric. For example, GAIL [10] chooses to minimize the JS divergence65 DJS(ρπE (s, a)‖ρπ(s, a)), and AIRL [5] utilizes the KL divergence DKL(ρπE (s, a)‖ρπ(s, a)) instead,66 which corresponds to a maximum entropy solution with the recovered reward [17]. However, for the67 scenario studied in this paper, the action information is absent in state-only demonstrations, known as68 state-only imitation learning (SOIL) or learning from observations (LfO) problems, where the action69 spaces between the expert and the agent can even be different. Such challenges prevent applying70 typical IL solutions. An existing method for this problem is to instead optimize the discrepancy of71 the state transition OM with the state-to-action policy π(a|s) [23]:72
π∗ = arg min π [` (ρπE (s, s ′), ρπ(s, s ′))] . (2)
However, the solution to this problem is ambiguous since there is no one-to-one correspondence73 between ρ(s, s′) and π as we will show in the following section. As such, the optimality of SOIL74 should be reconsidered.75
3 Methodology76
3.1 Characterizing the Optimality in SOIL77
In standard IL tasks, when the expert actions are accessible in demonstrations, perfectly imitating the78 expert policy corresponds to matching the state-action OM due to the one-to-one correspondence79
between π and ρπ(s, a) [10, 21]. However, such correspondence is not applicable for the state80 transition OM matching in SOIL.81 Proposition 1. Suppose Π is the policy space and P = {ρ : ρ ≥ 0} is a feasible set of OM, then a82 policy π ∈ Π corresponds to one state transition OM ρπ ∈ P . However, a state transition OM ρ ∈ P83 can correspond to more than one policy in Π.84
The proof can be found in Appendix B. As a result, if we choose to optimize a state-to-action mapping85 policy, then the optimal solution to Eq. (2) is ambiguous. The ambiguity also comes from the fact86 that Eq. (2) does not correspond to a maximum policy entropy solution as in normal IL tasks (see87 Appendix C for details). Therefore, a state-to-action mapping function may be too implicit for88 matching the state sequence, which could cause training instability and lead to sub-optimal policies.89 In that case, we must find a one-to-one corresponding solution to solve SOIL explicitly and efficiently.90 Before continuing, we introduce the definition of hyper-policy.91 Definition 1. A hyper-policy Ω is a set of policies such that for any π1, π2 ∈ Ω, we have ρπ1(s, s′) =92 ρπ2(s, s ′).93
Then by definition, there is a one-to-one correspondence between the hyper-policy Ω and the state94 transition OM ρΩ(s, s′). Similar to the normal state-to-action mapping policy, a hyper-policy Ω can95 be regarded as a state-to-state mapping function hΩ(s′|s) which predicts the state transition such that96 for any π ∈ Ω:97
hΩ(s ′|s) = ρΩ(s, s ′)∫ s̃ ρΩ(s, s̃) ds̃ = ∫ a π(a|s)T (s′|s, a) da . (3) 98
Proposition 2. Suppose the state transition predictor is defined as in Eq. (3) and Γ is its space,99 P = {ρ : ρ ≥ 0}, then a hyper-policy state transition predictor hΩ ∈ Γ corresponds to one state100 transition OM ρΩ ∈ P; and a state transition OM ρ ∈ P only corresponds to one hyper-policy state101 transition predictor such that hρ = ρ(s, s′)/ ∫ s̃ ρ(s, s̃) ds̃.102
Therefore, we find a one-to-one correspondence between the optimization term ρ(s, s′) and a practical103 target hΩ(s′|s), which indicates that we do not have to infer the expert actions under state-only104 demonstrations but only need to recover the state transition predictor of the hyper-policy ΩE :105
arg min Ω
[` (ρΩE (s, s ′), ρΩ(s, s ′))]⇒ arg min hΩ Es∼Ω[` (hΩE (s′|s), hΩ(s′|s))] . (4)
However, SOIL still requires to learn a policy to interact with the MDP environment to match the106 state transition OM of the expert. This is achievable since we do not have to recover the expert policy107 πE exactly but can learn any policy π ∈ ΩE according to Eq. (4).108
3.2 Policy Decoupling109
To construct an unambiguous objective for SOIL, we define hyper-policy and solve the problem by110 finding the state transition predictor of the expert hyper-policy. Intuitively, this tells the agent the111 target that the expert will reach without informing any feasible skill that require the agent to learn112 itself. Therefore, to recover a π ∈ ΩE , we can construct an inverse dynamics such that113
π = T −1π︸︷︷︸ Inverse dynamics ( T (πE)︸ ︷︷ ︸ Expert state transition predictor ) . (5)
Formally, the expert policy can be decoupled as114 πE(a|s) = ∫ s′ T (s′|s, a)πE(a|s) ds′ = ∫ s′ ρπE (s, a, s ′) ρπE (s) ds′ = ∫ s′ ρπE (s, s ′)IπE (a|s, s ′) ρπE (s) ds′
= ∫ s′ hπE (s ′|s)IπE (a|s, s ′) ds′ .
(6)
115
Notice that both the state transition predictor h and the inverse dynamics model I is policy dependent.116 Nevertheless, recall that the optimality in SOIL only requires us to recover π ∈ ΩE , we do not have117 to learn about IπE but just one feasible skill I(a|s, s′). Then a policy can be recovered by118
π = Es′ ∼ hΩE (s′|s)︸ ︷︷ ︸ target
[ I(a|s, s′)︸ ︷︷ ︸
skill
] .
(7)
Here the inverse dynamics model I offers an119 arbitrary skill to reach the expected target state120 provided by the state transition predictor h. In121 fact, it does not depend on the hyper-policy122 ΩE but a sampling policy πB to construct I =123 IπB . We only need a mild requirement for πB124 that it covers the support of ρΩE (s, s
′) so that125 the learned I can provide a possible action to126 achieve the target state. In both experiments and127 theoretical analysis we show that this require-128 ment alleviates the dependence on the inverse129 dynamics. Furthermore, if the environment and130 the expert policy are both deterministic (which131 is usually the case in real-world scenarios such132
as robotics), the state transition is a single-point distribution (or known as the Dirac delta func-133 tion), and we can simply model h as a deterministic function. By decoupling the policy, which is134 a state-to-action mapping function, as a state-to-state mapping function (the transition predictor)135 and a state-pair-to-action mapping function (the inverse dynamics model), we can mimic the expert136 policy from state-only demonstrations by optimizing these two modules. The whole architecture is137 illustrated in Fig. 1.138
State Transition Predictor. In practice, we construct a parameterized expert state transition predic-139 tor hψ which predicts the subsequent state of the expert taking the input as a current state ŝ′ = hψ(s).140
The state transition predictor models the explicit information of the expert, and it can be learned from141 the demonstration data only. Thence, we implement Eq. (4) as a KL divergence minimization:142
min ψ
E(s,s′)∼ΩE [DKL(hΩE (s ′|s)‖hψ(s′|s))] , (8)
which can be optimized in a supervised manner. Specifically, we sample state transitions (s, s′) from143 the expert demonstrations D and optimize the L2 loss:144
Lhψ = E(s,s′)∼D [ ‖s′ − hψ(s)‖2 ] . (9)
Inverse Dynamics Model. Knowing where to go is not enough since the agent has to interact with145 the environment to reach the target. This can be achieved via an inverse dynamics model, which146 predicts the action given two consecutive states. Formally, let the φ-parameterized inverse dynamics147 model Iφ take input the state pair and predict the feasible action to achieve the state transition:148 â = Iφ(s, s
′). Intuitively, we want the inverse dynamics to learn from possible transitions sampled149 by the agent. Recall that we only need the support of learned I(a|s, s′) of the sampling policy covers150 the support of the expert state transition OM, from which we can infer at least one possible action.151 Hence, we can optimize the KL divergence between the inverse dynamics of a sampling policy πB152 and Iφ:153
min φ
E(s,s′)∼πB [DKL(IπB(a|s, s ′)‖Iφ(a|s, s′))] , (10)
and we can choose to optimize L2 loss in a supervised manner by sampling from the replay buffer B:154 LIφ = E(s,a,s′)∼B [ ‖a− Iφ(s, s′)‖2 ] . (11)
In our implementation, both the state predictor and the inverse dynamics can be constructed as155 Gaussian distributions similar to a normal stochastic policy, thus encouraging exploration.156
3.3 Tackling Compounding Error Challenges157
In our formulation, we have decoupled the state-to-action mapping policy as a state-to-state mapping158 function and a state-pair-to-action mapping function. Unfortunately, the compounding error problem159 exists such that the agent cannot reach where it plans due to the fitting errors of these two parts.160 Theorem 1 (Error Bound of DPO). Consider a deterministic environment whose dynamics transition161 function T (s, a) is deterministic and L-Lipschitz. Assume the ground-truth state transition hΩE (s)162 is deterministic, and for each policy π ∈ Π, its inverse dynamics Iπ is also deterministic and163
C-Lipschitz. Then for any state s, the distance between the desired state s′E and reaching state s ′164 sampled by the decoupled policy is bounded by165
‖s′ − s′E‖ ≤ LC‖hΩE (s)− hψ(s)‖+ L‖IπB(s, ŝ′)− Iφ(s, ŝ′)‖ , (12)
where πB is a sampling policy that covers the state transition support of the expert hyper-policy and166 ŝ′ = hψ(s) is the predicted next state.167
The proof can be found in Appendix B, where we also168 induce a similar error bound for rollout with a state-169 to-action policy as BCO [22] to show the advantage170 of the decoupled structure. From Theorem 1 we171 know that the compounding error can be enlarged172 due to each part’s fitting error, where the first term173 corresponds to the error of predicted states and the174 second term indicates whether the agent can reach175
where it plans to. To alleviate the error, we further propose regularization on these two modules.176
3.3.1 Regularization on Target Planning177
One major problem is that the state transition predictor may suggest non-neighboring states instead178 of predicting one-step reachable states. To overcome this, we draw inspiration from Asadi et al. [2]179 and Edwards et al. [3], and regularize state transition predictor to prevent the model from predicting180 non-neighboring states via multi-step and cycle training style.181
Multi-Step Optimization. We first explain the details of the multi-step optimization objective.182 This idea is motivated by Asadi et al. [2], which optimizes a multi-step outcome by executing a183 sequence of actions in the dynamics model. Here we optimize the state sequence instead. As shown184 in Fig. 2, given an expert state sE , hψ predicts the next possible state ŝ′ that the expert will reach; the185 predicted state is then fed into the predictor to output the predicted two-step state ŝ′′. As such, the186 multi-step training loss is the L2 loss computed along the k-step outcome sequence:187
Lh,msψ = E (s,{s′iE}ki=1)∼D
[ ‖s′1E − hψ(s)‖2 + k∑ i=2 ‖s′iE − hψ′(s′i−1)‖2 ] . (13)
Intuitively, such a regularization makes the state prediction188 ŝ′ close to the expert state distribution in order to make189 accurate long step predictions. It is worth noting that the190 gradient of the cascading state transition predictors should191 be dropped since we already have an accurate input at each192 time step, and for each training step, we only update the193 first one. We use a target network hψ′ in practice.194
Cycle Training Style. Another way to regularize the195 transition predictor’s output to a neighboring state is to196 keep an additional function to ensure the cycle consistency,197 which is also an important technique in [3]. In particular,198 as illustrated in Fig. 3, given an expert state sE , we take199 the predicted state ŝ′ and sE into the inverse dynamics and200
get the action a, then we train an additional forward dynamics model Mω to simulate one step rollout201 that takes the input (sE , a) and gets a forward next state s̃′:202
LMω = E(s,a,s′)∼B [ ‖s′ −Mω(s, a)‖2 ] Lh,cycleψ = E(s,s′)∼D [ ‖s′ − hψ(s)‖2 + ‖h(s)−Mω(s, Iφ(s, s′))‖2 ] .
(14)
In other words, the cycle training scheme provides a regularization on hψ to make predictions203 consistent with the forward dynamics model.204
3.3.2 Efficient Skills Learning via Decoupled Policy Gradient205
In previous sections, we have mentioned that learning to reach a specific place requires the data-206 collecting policy to cover the support of the expert hyper-policy. This is easy to achieve on simple207 low-dimensional tasks, but may not be satisfied in high-dimensional continuous environments. To208 this end, we encourage the agent to approach those state transitions from the expert’s hyper-policy209 ΩE by minimizing the JS divergence of the state transition occupancy using a state-to-action mapping210 policy DJS (ρπE (s, s ′), ρπ(s, s ′)). This can be done by producing informative rewards via GAN-like211 methods [10, 23], and updating the decoupled policy with policy gradients (PG).212
In detail, we construct a parameterized discriminator Dω(s, s′) to compute the reward r(s, a) ,213 r(s, s′) as logDω(s, s′) and the decoupled policy served as the generator. In addition, since we214 decouple the policy as two parameterized modules, i.e., a state transition predictor and an inverse215 dynamics model, then by chain rule, the PG for the decoupled policy can be accomplished by216
∇Lπφ,ψ = Eπ [Q(s, a)∇φ,ψ log πφ,ψ(a|s)] = Eπ [ Q(s, a) ∫ s′ ( ∇ψ log hψ(s′|s) +∇φ log Iφ(a|s, s′) ) ds′ ] ,
(15)
whereQ is the state-action value function; the first term is the gradient for updating the state transition217 predictor; and the second term is for the inverse dynamics model. Thus, the optimization for both218 the state transition predictor and the inverse dynamics model can augment the supervised learning219 objectives with any PG-based learning algorithms (e.g., TRPO, PPO, SAC). As the training proceeds,220 the agent will sample more transition data around ΩE , and thus the support of the sampling policy221 will progressively cover the support of ρΩE (s, s ′).222
3.4 Overall Algorithm223
By combining the idea of generative adversarial training, we obtain our final algorithm, composed224 with three essential parts: the state transition predictor h used for predicting the possible future225 states sampled by the expert; the inverse dynamics model I used for inferring the possible actions226 conditioned on two adjacent states; and the discriminator D used for offering intermediate reward227 signals for training the decoupled policy π = I(h). The overall objective of DPO is228
Lπ,h,Iφ,ψ = λGL π φ,ψ + λhLhψ + λILIφ , (16)
where λG, λh and λI are hyperparameters for trading off the training among each loss. In practice,229 we try less than ten combinations for these parameters as shown in Appendix D.3, and we directly230 optimize Lπφ,ψ instead of iterative training. The detailed algorithm is summarized in Appendix A.231 Besides, it is worth noting that both the inverse dynamics model and the state transition predictor232 can be pre-trained, where we optimize Lhψ using the state-only demonstration and optimize LIφ using233 samples collected by a randomized agent.234
4 Related Work235
Table 1: Comparison between different methods.
Method Inverse State Decoupled TaskDynamics Predictor Policy
BCO [22] 3 7 7 SOIL GAIfO [23] 7 7 7 SOIL IDDM [25] 7 7 7 SOIL
OPOLO [27] 3 7 7 SOIL PID-GAIL [11] 7 7 3 IL
QSS [3] 3 3 3 RL SAIL [16] 3 3 7 IL
DPO (Ours) 3 3 3 SOIL State-only imitation learning (SOIL) endows the236 agent with the ability to learn from expert states. Al-237 though lacking the expert decision information, most238 of the previous works still optimize a state-to-action239 mapping policy to match the expert state transition240 distribution. For example, Torabi et al. [22] used a model-based approach to apply behavioral241 cloning to state-only demonstrations, while Torabi et al. [23] employed a similar structure to GAIL242 to match the state transition distribution. Yang et al. [25] analyzed the optimization gap between243 SOIL and naive IL and introduced a mutual information term to narrow it. Huang et al. [11] applied244 SOIL on autonomous driving tasks by decoupling the policy into a neural decision module and a245 non-differentiable execution module in a hierarchical way.246
Our work decouples the state-to-action policy into two modules. However, both the inverse dynamics247 model and the state transition predictor have been widely used by many previous works on RL and IL248 tasks. For instance, Torabi et al. [22] and Guo et al. [6] trained an inverse dynamics model to label the249
state-only demonstrations with inferred actions. Nair et al. [19] proposed a method for manipulating250 ropes to match a single human-specified image sequence, in which an inverse dynamics model is251 trained in a self-supervised manner and used to generate control signals. Kimura et al. [14] utilized a252 state transition predictor to fit the state transition probability in the expert data, which is further used253 to compute a predefined reward function. Liu et al. [16] constructed a policy prior using the inverse254 dynamics and the state transition predictor, but the policy prior was only used for regularizing the255 policy network. However, as shown in this paper, the policy can be exactly decoupled as these two256 parts, which can be uniformly optimized through policy gradient without keeping an extra policy.257 Edwards et al. [3] estimated Q(s, s′) for RL tasks which employs a similar policy form as Eq. (6) and258 updates the state transition predictor through a deterministic policy gradient similar to DDPG [15].259 To sort out the difference between these methods and ours, we summarize the key factors in Tab. 1.260
5 Experiments261
We conduct four sets of experiments to investigate the following research questions:262
RQ1 Is decoupled learning structure superior than state-to-action structure on SOIL tasks?263 RQ2 Does DPO achieve higher efficiency or better performance than baselines on SOIL tasks?264 RQ3 Can agent reach where it plans and does the proposed regularization help mitigate the265 compounding error?266 RQ4 How can DPO be applied on real-world data?267 Expert DPO
To answer RQ1, we conduct toy experiments with a simple 2D grid world268 environment and compare both qualitative and quantitative results of DPO269 against BCO and GAIfO. Regarding RQ2, we empirically evaluate DPO270 on easy-to-hard continuous control benchmarking tasks. And for RQ3,271 we evaluate the difference between the predicted states that the agent272 plans to reach and the consecutive state that the agent actually reaches in273 the environment for the proposed regularization. Finally, we try to imitate274 real-world traffic surveillance recordings in a simulated environment to275 investigate RQ4, which shows the potential of using real-world data for276 human behavior simulation. Due to the space limit, we leave experiment277 details, additional results and ablation study in Appendix.278
5.1 Understanding the Decoupled Structure279
In this paper we design decoupled policy optimization (DPO) to perform280 SOIL tasks, and in previous sections we propose that the key technical281 contribution of DPO is the decoupled structure of policy that models the282 explicit state transition information and the latent action information from283 demonstrations, which solves the ambiguity and enhances the learning284 efficiency. Therefore, in this set of experiments, we aim to demonstrate285 how DPO is superior than state-to-action policy methods (RQ1). We first286
generate expert demonstrations in a 2D 6×6 grid world environment, in which the agent starts at287 the upper left corner and aims to reach the upper right corner. In each grid the agent has k × 4288 actions, which means that the agent has k possible actions to reach the neighboring block and in our289 experiment we choose k = 5 to enlarge the action space.290
The density of the expert trajectories and the trajectories sampled by different methods are shown in291 Fig. 4(a). We show that both BCO and GAIfO have troubles in directly learning the implicit action292 from state-only behaviors. Notably, GAIfO only imitates the major trajectory and omit the other293 choice and BCO also stucks in the middle right. By contrast, DPO recovers the expert demonstrations294
much better, benefiting from the decoupled structure that first determining the target and then taking295 the action to achieve it. To further illustrate the learning efficiency advantage of DPO, we illustrate the296 JS divergence curves of DPO and BCO during training in Fig. 4(b). Besides, we show the policy loss297 for BCO, the state predictor (SP) loss for DPO, and the inverse dynamics (ID) loss for both methods.298 Except that the JS divergence of DPO decreases more quickly than BCO, it is also observable that299 DPO relies less on the inverse dynamics than BCO, since the inverse dynamics loss of DPO converges300 to a higher level. We further provide a theoretic analysis of the dependence on inverse dynamics with301 BCO and DPO in Appendix B.302
5.2 Comparative Evaluations303
We compare the qualitative results of DPO against other baseline methods on easy-to-hard continuous304 control benchmarking environments (RQ2), including InvertedPendulum, InvertedDoublePendulum,305 Hopper, Walker2d, HalfCheetah and Ant. In each environment, besides GAIfO and BCO, we also306 evaluate GAIfO with decoupled policy (denoted as GAIfO-DP). For DPO we compare the reward307 augmented version of DPO (denoted as DPO w PG)1 with the supervised learning version of DPO,308 i.e., λG = 0 (denoted as DPO w/o PG). For fairness, we re-implement all the algorithms based on a309 Pytorch code framework2 and adopt Soft Actor-Critic (SAC) [7] as the RL learning algorithm for310 GAIfO and DPO. For all environments, we first train an SAC agent to collect 4 state-only expert311 trajectories and then train agents with such data. All algorithms are evaluated by a deterministic312 policy. The eventual results are summarized in Tab. 2, and the learning curves are shown in Fig. 5.313 It is worth noting that for DPO, we choose the best performance among the experiments that use314 multi-step or cycle regularization, and we put the full experiment results in Appendix D.315
One can easily observe that on simple environments, BCO is able to achieve a good performance,316 and GAIfO also does well on harder tasks. Even so, DPO can still gain the best or comparable317 performance against its counterparts. Particularly, without augmented reward, DPO is able to reach318 the optimality with the highest sample efficiency on simple tasks like InvertedPendulum. By contrast,319 on higher-dimensional tasks such as Hopper, Walker2d, HalfCheetah and Ant, it is difficult to320 construct accurate inverse dynamics that covers the support of the expert hyper-policy from scratch.321 However, by combining generative adversarial policy gradients, the agent finally recovers a good322 policy from the expert hyper-policy. This is particularly evident on HalfCheetah where DPO behaves323 poorly at the beginning but improves fast as the training proceeds. Besides, as illustrated in Fig. 5,324 DPO benefits from better sample efficiency in most of the environments, but the improvements are325 limited on the hardest tasks. We think that this may be due to larger state spaces (111 dimensions326 for Ant) that makes it difficult to recover a good state predictor or an inverse dynamics model. In all327
1Without ambiguity we simply denote DPO for this version of algorithm in the following sections. 2https://github.com/KamyarGh/rl_swiss
experiments, GAIfO-DP achieves similar results as GAIfO, indicating that the network structure does328 not count much for the performance.329
5.3 Compounding Error Reduction330
In this section, we aim to study whether the agent can effectively reach the target as it plans (RQ3).331 Therefore, we analyze the distance of the reaching states and the predicted consecutive states, and332 draw the mean square error (MSE) along the training procedure in Fig. 6. We also compare our333 regularization including multi-step optimization (denoted as M.S.-k, where k is the number of rollout334 steps) and cycle training style (denoted as Cycle). Note that DPO needs at least 1-step rollout for335 training the state transition predictor. As shown in Fig. 6, the agent still has gaps to get to where it336 plans to, and the mismatch always deteriorates on harder tasks. Combining regularization can always337 achieve lower compounding error, and the cycle training is effective in most of the environments. In338 Appendix D.6, we further illustrate the correlation between the final performance and the distance.339
5.4 Learn to Drive from Real-World Traffic Data340
The rapid development of autonomous driving has brought a lot of demand for simulating and341 training an RL agent in the simulator, which requires realistic interactions with various social vehicles342 [26]. However, driver’s detailed actions are not easily to obtain yet we adopt SOIL from a traffic343 surveillance recording dataset (NGSIM I-80 [8]) that contains kinds of recorded human driving344 trajectories. We wish to further examine the potential of DPO for decreasing the gap between the345 real world and simulation (RQ4). We utilize the simulator provided by Henaff et al. [9] as our346 simulation platform and learn to imitate real-world driving behaviors. We compare DPO against347 GAIfO and BCO, and choose Success Rate, Mean Distance and KL Divergence as evaluation metrics.348 Specifically, Success Rate is the percentage of driving across the entire area without crashing into349 other vehicles or driving off the road, Mean Distance is the distance traveled before the episode ends,350 and KL Divergence measures the position distribution distance between the expert and the agent.351
As shown in Tab. 3, DPO outperforms baseline methods in352 all three metrics while possessing higher stability. The de-353 coupled policy allows the state predictor to focus on match-354 ing the distribution of expert trajectories, thus achieving355 smaller deviations from the expert position distribution.356 Furthermore, since the policy gradient can be computed357 with non-differentiable inverse dynamics, we can generate358 stable action sequences [12, 11] by replacing the inverse359
dynamics model with classical controllers, which can be generalized to realistic applications.360
6 Conclusion361
In this paper, we characterize the optimality and investigate the ambiguity problem in state-only362 imitation learning, and accordingly propose Decoupled Policy Optimization (DPO), which splits363 the state-to-action mapping policy into a state-to-state mapping state transition predictor and a364 state-pair-to-action mapping inverse dynamics model. Furthermore, we employ regularization and365 generative adversarial methods to mitigate the compounding error caused by the decoupled modules.366 The flexibility of the decoupled architecture allows a wide range of interesting future works, such367 as replacing the inverse dynamics with a classic control module to produce stable control signals,368 learning specific skills with shared state transition and multi-task target learning with shared pre-369 trained skills.370
References371 [1] Pieter Abbeel and Andrew Y. Ng. Apprenticeship learning via inverse reinforcement learning.372 In Machine Learning, Proceedings of the Twenty-first International Conference (ICML 2004),373 2004.374
[2] Kavosh Asadi, Dipendra Misra, Seungchan Kim, and Michel L Littman. Combating the375 compounding-error problem with a multi-step model. arXiv preprint arXiv:1905.13320, 2019.376
[3] Ashley D. Edwards, Himanshu Sahni, Rosanne Liu, Jane Hung, Ankit Jain, Rui Wang, Adrien377 Ecoffet, Thomas Miconi, Charles Isbell, and Jason Yosinski. Estimating q(s,s’) with deep378 deterministic dynamics gradients. In Proceedings of the 37th International Conference on379 Machine Learning, ICML 2020, 2020.380
[4] Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal381 control via policy optimization. In International Conference on Machine Learning, pages 49–58,382 2016.383
[5] Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse384 reinforcement learning. In 6th International Conference on Learning Representations, ICLR385 2018, 2018.386
[6] Xiaoxiao Guo, Shiyu Chang, Mo Yu, Gerald Tesauro, and Murray Campbell. Hybrid reinforce-387 ment learning with expert state sequences. In The Thirty-Third AAAI Conference on Artificial388 Intelligence, AAAI 2019, pages 3739–3746, 2019.389
[7] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy390 maximum entropy deep reinforcement learning with a stochastic actor. In Proceedings of the391 35th International Conference on Machine Learning, ICML 2018, pages 1856–1865, 2018.392
[8] John Halkias and James Colyar. Next generation simulation fact sheet. US Department of393 Transportation: Federal Highway Administration, 2006.394
[9] Mikael Henaff, Alfredo Canziani, and Yann LeCun. Model-predictive policy learning with395 uncertainty regularization for driving in dense traffic. In 7th International Conference on396 Learning Representations, ICLR 2019, 2019.397
[10] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in398 Neural Information Processing Systems 29, pages 4565–4573, 2016.399
[11] Junning Huang, Sirui Xie, Jiankai Sun, Qiurui Ma, Chunxiao Liu, Dahua Lin, and Bolei Zhou.400 Learning a decision module by imitating driver’s control behaviors. In Proceedings of the401 Conference on Robot Learning (CoRL) 2020, 2020.402
[12] Sandy H. Huang, Nicolas Papernot, Ian J. Goodfellow, Yan Duan, and Pieter Abbeel. Adversarial403 attacks on neural network policies. In 5th International Conference on Learning Representations,404 ICLR 2017, 2017.405
[13] Ahmed Hussein, Mohamed Medhat Gaber, Eyad Elyan, and Chrisina Jayne. Imitation learning:406 A survey of learning methods. ACM Computing Surveys (CSUR), 50(2):1–35, 2017.407
[14] Daiki Kimura, Subhajit Chaudhury, Ryuki Tachibana, and Sakyasingha Dasgupta. Internal408 model from observations for reward shaping. arXiv preprint arXiv:1806.01267, 2018.409
[15] Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval410 Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning.411 In 4th International Conference on Learning Representations, ICLR 2016,, 2016.412
[16] Fangchen Liu, Zhan Ling, Tongzhou Mu, and Hao Su. State alignment-based imitation learning.413 In 8th International Conference on Learning Representations, ICLR 2020, 2020.414
[17] Minghuan Liu, Tairan He, Minkai Xu, and Weinan Zhang. Energy-based imitation learning. In415 20th International Conference on Autonomous Agents and Multiagent Systems, AAMAS 2021,416 2021.417
[18] Leland McInnes and John Healy. UMAP: uniform manifold approximation and projection for418 dimension reduction. CoRR, abs/1802.03426, 2018.419
[19] Ashvin Nair, Dian Chen, Pulkit Agrawal, Phillip Isola, Pieter Abbeel, Jitendra Malik, and Sergey420 Levine. Combining self-supervised learning and imitation for vision-based rope manipulation.421 In 2017 IEEE International Conference on Robotics and Automation, ICRA 2017, pages 2146–422 2153, 2017.423
[20] Murray Rosenblatt. Remarks on some nonparametric estimates of a density function. The424 Annals of Mathematical Statistics, pages 832–837, 1956.425
[21] Umar Syed, Michael H. Bowling, and Robert E. Schapire. Apprenticeship learning using linear426 programming. In Machine Learning, Proceedings of the Twenty-Fifth International Conference427 (ICML 2008), pages 1032–1039, 2008.428
[22] Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In429 Proceedings of the Twenty-Seventh International Joint Conference on Artificial Intelligence,430 IJCAI 2018, pages 4950–4957, 2018.431
[23] Faraz Torabi, Garrett Warnell, and Peter Stone. Adversarial imitation learning from state-only432 demonstrations. In Proceedings of the 18th International Conference on Autonomous Agents433 and MultiAgent Systems, AAMAS ’19, pages 2229–2231, 2019.434
[24] Faraz Torabi, Garrett Warnell, and Peter Stone. Recent advances in imitation learning from435 observation. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial436 Intelligence, IJCAI 2019, pages 6325–6331, 2019.437
[25] Chao Yang, Xiaojian Ma, Wenbing Huang, Fuchun Sun, Huaping Liu, Junzhou Huang, and438 Chuang Gan. Imitation learning from observations by minimizing inverse dynamics disagree-439 ment. In Advances in Neural Information Processing Systems 32, pages 239–249, 2019.440
[26] Ming Zhou, Jun Luo, Julian Villela, Yaodong Yang, David Rusu, Jiayu Miao, Weinan Zhang,441 et al. Smarts: Scalable multi-agent reinforcement learning training school for autonomous442 driving. In Conference on Robot Learning, 2020.443
[27] Zhuangdi Zhu, Kaixiang Lin, Bo Dai, and Jiayu Zhou. Off-policy imitation learning from444 observations. In Advances in Neural Information Processing Systems 33, 2020.445
Checklist446
1. For all authors...447 (a) Do the main claims made in the abstract and introduction accurately reflect the paper’s448 contributions and scope? [Yes]449 (b) Did you describe the limitations of your work? [Yes] See Section 3.3.450 (c) Did you discuss any potential negative societal impacts of your work? [Yes] See451 Section 6.452 (d) Have you read the ethics review guidelines and ensured that your paper conforms to453 them? [Yes]454 2. If you are including theoretical results...455
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Theorem 1.456 (b) Did you include complete proofs of all theoretical results? [Yes] See Section B.457
3. If you ran experiments...458 (a) Did you include the code, data, and instructions needed to reproduce the main experi-459 mental results (either in the supplemental material or as a URL)? [Yes]460 (b) Did you specify all the training details (e.g., data splits, hyperparameters, how they461 were chosen)? [Yes] See Appendix D.3.462 (c) Did you report error bars (e.g., with respect to the random seed after running experi-463 ments multiple times)? [Yes] We ran our results with more than 5 random seeds as said464 in Section 5.2.465
(d) Did you include the total amount of compute and the type of resources used (e.g., type466 of GPUs, internal cluster, or cloud provider)? [No]467
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...468 (a) If your work uses existing assets, did you cite the creators? [Yes] We re-implement all469 algorithms based on an existed code base, as said in Section 5.2.470 (b) Did you mention the license of the assets? [No]471 (c) Did you include any new assets either in the supplemental material or as a URL? [No]472 We will public our codes after publication.473 (d) Did you discuss whether and how consent was obtained from people whose data you’re474 using/curating? [N/A]475 (e) Did you discuss whether the data you are using/curating contains personally identifiable476 information or offensive content? [N/A]477 5. If you used crowdsourcing or conducted research with human subjects...478
(a) Did you include the full text of instructions given to participants and screenshots, if479 applicable? [N/A]480
(b) Did you describe any potential participant risks, with links to Institutional Review481 Board (IRB) approvals, if applicable? [N/A]482
(c) Did you include the estimated hourly wage paid to participants and the total amount483 spent on participant compensation? [N/A]484

Review summaries:
This paper tackles the state-only imitation learning problem by decomposing it into two components: a state-transition predictor and an inverse dynamics model. Specifically, it argues that the problem is difficult because there are potentially multiple policies that could match the same state trajectory of the expert.   Instead, it proposes to learn what they call a hyper-policy, which is a family of policies that lead to the same distribution over state trajectories. Then, after recovering the optimal hyper-policy, it learns an inverse dynamics model to recover the actions, and this model can be learned by interacting with the environment.  However, to actually achieve good performance in practice, they propose two regularizations, i.e., a multi-step prediction loss and a cycle training loss. The parameters of the hyper-policy and the inverse dynamics model are also trained with a policy gradient objective where the reward is based on a discriminator (which is trained to predict whether the transition (s, s’) is from the expert). 

The paper proposes a new method for state-only imitation learning, based around explicitly representing the transition from one state to the next (based on state-only expert demonstrations) and an inverse dynamics model providing feasible actions to produce that transition (based on rollouts from a sampling policy). Theoretical analysis attempts to show that this reduces policy ambiguity. Empirical results show improved performance relative to baselines, with the most striking results in a gridworld and on real-world driving data: on standard MuJoCo continuous control tasks, the mean of the method tends to be higher, but the confidence intervals overlap with baselines.

This paper proposes a decoupled policy approach to state-only imitation learning, wherein the policy is decomposed into a next-state prediction network, which aims to predict the desired next state given the current state under the expert policy, and an inverse dynamics module, which given the current state and the desired next state, predicts the action. The utility of this split is that it allows for additional supervision during policy optimization, as the state transition network can be trained to match the observed expert transitions, and the inverse dynamics module can be trained on the transitions in the trajectories obtained during policy optimization. These supervised losses are placed on top of a standard policy gradient method, as gradients to both components of the decoupled policy can also be obtained through a standard likelihood-ratio policy gradient applied to a reward function corresponding to likelihood of the executed state-transitions being drawn from the expert policy. This reward function is obtained by jointly training a discriminator to classify sample transitions from the learned policy from those from the expert data. 

The paper presents a method for state-only imitation learning (SOIL) that decouples learning the expert state-transition distribution from the actual policy needed to reach desired states. In particular, given a dataset of expert state sequences, the method trains two models: one that predicts next state given the current state, and another one that learns inverse dynamics, i.e. which action is needed to reach the next state from the current state. Where the state prediction can be directly learned from the state-only expert data, environment interactions are collected using a sampling policy to learn the inverse dynamics model. Additional techniques, such as multi-step optimization and cycle training are used to improve prediction accuracy of the state-transition model. In addition, training is improved by introducing a discriminator with a GAN-like reward that can provide intermediate reward signals to the policy. The method is evaluated on several MuJoCo continuous control environments, where it is shown to outperform other SOIL baselines, and on a real-world human driving dataset where it can learn effective driving policies just from human driving trajectories.