SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards

1 INTRODUCTION . Many sequential decision-making problems can be tackled by imitation learning : an expert demonstrates near-optimal behavior to an agent , and the agent attempts to replicate that behavior in novel situations ( Argall et al. , 2009 ) . This paper considers the problem of training an agent to imitate an expert , given expert action demonstrations and the ability to interact with the environment . The agent does not observe a reward signal or query the expert , and does not know the state transition dynamics . Standard approaches based on behavioral cloning ( BC ) use supervised learning to greedily imitate demonstrated actions , without reasoning about the consequences of actions ( Pomerleau , 1991 ) . As a result , compounding errors cause the agent to drift away from the demonstrated states ( Ross et al. , 2011 ) . The problem with BC is that , when the agent drifts and encounters out-of-distribution states , the agent does not know how to return to the demonstrated states . Recent methods based on inverse reinforcement learning ( IRL ) overcome this issue by training an RL agent not only to imitate demonstrated actions , but also to visit demonstrated states ( Ng et al. , 2000 ; Wulfmeier et al. , 2015 ; Finn et al. , 2016b ; Fu et al. , 2017 ) . This is also the core idea behind generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) , which implements IRL using generative adversarial networks ( Goodfellow et al. , 2014 ; Finn et al. , 2016a ) . Since the true reward function for the task is unknown , these methods construct a reward signal from the demonstrations through adversarial training , making them difficult to implement and use in practice ( Kurach et al. , 2018 ) . The main idea in this paper is that the effectiveness of adversarial imitation methods can be achieved by a much simpler approach that does not require adversarial training , or indeed learning a reward function at all . Intuitively , adversarial methods encourage long-horizon imitation by providing the agent with ( 1 ) an incentive to imitate the demonstrated actions in demonstrated states , and ( 2 ) an incentive to take actions that lead it back to demonstrated states when it encounters new , out-ofdistribution states . One of the reasons why adversarial methods outperform greedy methods , such as BC , is that greedy methods only do ( 1 ) , while adversarial methods do both ( 1 ) and ( 2 ) . Our approach is intended to do both ( 1 ) and ( 2 ) without adversarial training , by using constant rewards instead of learned rewards . The key idea is that , instead of using a learned reward function to provide a reward signal to the agent , we can simply give the agent a constant reward of r = +1 for matching the demonstrated action in a demonstrated state , and a constant reward of r = 0 for all other behavior . We motivate this approach theoretically , by showing that it implements a regularized variant of BC that learns long-horizon imitation by ( a ) imposing a sparsity prior on the reward function implied by the imitation policy , and ( b ) incorporating information about the state transition dynamics into the imitation policy . Intuitively , our method accomplishes ( a ) by training the agent using an extremely sparse reward function – +1 for demonstrations , 0 everywhere else – and accomplishes ( b ) by training the agent with RL instead of supervised learning . We instantiate our approach with soft Q-learning ( Haarnoja et al. , 2017 ) by initializing the agent ’ s experience replay buffer with expert demonstrations , setting the rewards to a constant r = +1 in the demonstration experiences , and setting rewards to a constant r = 0 in all of the new experiences the agent collects while interacting with the environment . Since soft Q-learning is an off-policy algorithm , the agent does not necessarily have to visit the demonstrated states in order to experience positive rewards . Instead , the agent replays the demonstrations that were initially added to its buffer . Thus , our method can be applied in environments with stochastic dynamics and continuous states , where the demonstrated states are not necessarily reachable by the agent . We call this method soft Q imitation learning ( SQIL ) . The main contribution of this paper is SQIL : a simple and general imitation learning algorithm that is effective in MDPs with high-dimensional , continuous observations and unknown dynamics . We run experiments in four image-based environments – Car Racing , Pong , Breakout , and Space Invaders – and three low-dimensional environments – Humanoid , HalfCheetah , and Lunar Lander – to compare SQIL to two prior methods : BC and GAIL . We find that SQIL outperforms BC and achieves competitive results compared to GAIL . Our experiments illustrate two key benefits of SQIL : ( 1 ) that it can overcome the state distribution shift problem of BC without adversarial training or learning a reward function , which makes it easier to use , e.g. , with images , and ( 2 ) that it is simple to implement using existing Q-learning or off-policy actor-critic algorithms . 2 SOFT Q IMITATION LEARNING . SQIL performs soft Q-learning ( Haarnoja et al. , 2017 ) with three small , but important , modifications : ( 1 ) it initially fills the agent ’ s experience replay buffer with demonstrations , where the rewards are set to a constant r = +1 ; ( 2 ) as the agent interacts with the world and accumulates new experiences , it adds them to the replay buffer , and sets the rewards for these new experiences to a constant r = 0 ; and ( 3 ) it balances the number of demonstration experiences and new experiences ( 50 % each ) in each sample from the replay buffer.1 These three modifications are motivated theoretically in Section 3 , via an equivalence to a regularized variant of BC . Intuitively , these modifications create a simple reward structure that gives the agent an incentive to imitate the expert in demonstrated states , and to take actions that lead it back to demonstrated states when it strays from the demonstrations . 1 SQIL resembles the Deep Q-learning from Demonstrations ( DQfD ) ( Hester et al. , 2017 ) and Normalized Actor-Critic ( NAC ) algorithms ( Gao et al. , 2018 ) , in that it initially fills the agent ’ s experience replay buffer with demonstrations . The key difference is that DQfD and NAC are RL algorithms that assume access to a reward signal , while SQIL is an imitation learning algorithm that does not require an extrinsic reward signal from the environment . Instead , SQIL automatically constructs a reward signal from the demonstrations . Algorithm 1 Soft Q Imitation Learning ( SQIL ) 1 : Require λsamp ∈ R≥0 , Ddemo 2 : Initialize Dsamp ← ∅ 3 : while Qθ not converged do 4 : θ ← θ − η∇θ ( δ2 ( Ddemo , 1 ) + λsampδ2 ( Dsamp , 0 ) ) { See Equation 1 } 5 : Sample transition ( s , a , s′ ) with imitation policy π ( a|s ) ∝ exp ( Qθ ( s , a ) ) 6 : Dsamp ← Dsamp ∪ { ( s , a , s′ ) } 7 : end while Crucially , since soft Q-learning is an off-policy algorithm , the agent does not necessarily have to visit the demonstrated states in order to experience positive rewards . Instead , the agent replays the demonstrations that were initially added to its buffer . Thus , SQIL can be used in stochastic environments with high-dimensional , continuous states , where the demonstrated states may never actually be encountered by the agent . SQIL is summarized in Algorithm 1 , where Qθ is the soft Q function , Ddemo are demonstrations , δ2 is the squared soft Bellman error , δ2 ( D , r ) , 1|D| ∑ ( s , a , s′ ) ∈D ( Qθ ( s , a ) − ( r + γ log ( ∑ a′∈A exp ( Qθ ( s ′ , a′ ) ) ) ) ) 2 , ( 1 ) and r ∈ { 0 , 1 } is a constant reward.2 The experiments in Section 4 use a convolutional neural network or multi-layer perceptron to model Qθ , where θ are the weights of the neural network . Section A.3 in the appendix contains additional implementation details , including values for the hyperparameter λsamp ; note that the simple default value of λsamp = 1 works well across a variety of environments . As the imitation policy in line 5 of Algorithm 1 learns to behave more like the expert , a growing number of expert-like transitions get added to the bufferDsamp with an assigned reward of zero . This causes the effective reward for mimicking the expert to decay over time . Balancing the number of demonstration experiences and new experiences ( 50 % each ) sampled for the gradient step in line 4 ensures that this effective reward remains at least 1/ ( 1 + λsamp ) , instead of decaying to zero . In practice , we find that this reward decay does not degrade performance if SQIL is halted once the squared soft Bellman error loss converges to a minimum ( e.g. , see Figure 8 in the appendix ) . Note that prior methods also require similar techniques : both GAIL and adversarial IRL ( AIRL ) ( Fu et al. , 2017 ) balance the number of positive and negative examples in the training set of the discriminator , and AIRL tends to require early stopping to avoid overfitting . 3 INTERPRETING SQIL AS REGULARIZED BEHAVIORAL CLONING . To understand why SQIL works , we sketch a surprising theoretical result : SQIL is equivalent to a variant of behavioral cloning ( BC ) that uses regularization to overcome state distribution shift . BC is a simple approach that seeks to imitate the expert ’ s actions using supervised learning – in particular , greedily maximizing the conditional likelihood of the demonstrated actions given the demonstrated states , without reasoning about the consequences of actions . Thus , when the agent makes small mistakes and enters states that are slightly different from those in the demonstrations , the distribution mismatch between the states in the demonstrations and those actually encountered by the agent leads to compounding errors ( Ross et al. , 2011 ) . We show that SQIL is equivalent to augmenting BC with a regularization term that incorporates information about the state transition dynamics into the imitation policy , and thus enables long-horizon imitation . 3.1 PRELIMINARIES . Maximum entropy model of expert behavior . SQIL is built on soft Q-learning , which assumes that expert behavior follows the maximum entropy model ( Ziebart et al. , 2010 ; Levine , 2018 ) . In 2Equation 1 assumes discrete actions , but SQIL can also be used with continuous actions , as shown in Section 4.3. an infinite-horizon Markov Decision Process ( MDP ) with a continuous state space S and discrete action space A,3 the expert is assumed to follow a policy π that maximizes reward R ( s , a ) . The policy π forms a Boltzmann distribution over actions , π ( a|s ) , exp ( Q ( s , a ) ) ∑ a′∈A exp ( Q ( s , a ′ ) ) , ( 2 ) where Q is the soft Q function . The soft Q values are a function of the rewards and dynamics , given by the soft Bellman equation , Q ( s , a ) , R ( s , a ) + γEs′ [ log ( ∑ a′∈A exp ( Q ( s′ , a′ ) ) ) ] . ( 3 ) In our imitation setting , the rewards and dynamics are unknown . The expert generates a fixed set of demonstrations Ddemo , by rolling out their policy π in the environment and generating state transitions ( s , a , s′ ) ∈ Ddemo . Behavioral cloning ( BC ) . Training an imitation policy with standard BC corresponds to fitting a parametric model πθ that minimizes the negative log-likelihood loss , ` BC ( θ ) , ∑ ( s , a ) ∈Ddemo − log πθ ( a|s ) . ( 4 ) In our setting , instead of explicitly modeling the policy πθ , we can represent the policy π in terms of a soft Q function Qθ via Equation 2 : π ( a|s ) , exp ( Qθ ( s , a ) ) ∑ a′∈A exp ( Qθ ( s , a ′ ) ) . ( 5 ) Using this representation of the policy , we can train Qθ via the maximum-likelihood objective in Equation 4 : ` BC ( θ ) , ∑ ( s , a ) ∈Ddemo − ( Qθ ( s , a ) − log ( ∑ a′∈A exp ( Qθ ( s , a ′ ) ) ) ) . ( 6 ) However , optimizing the BC loss in Equation 6 does not in general yield a valid soft Q function Qθ – i.e. , a soft Q function that satisfies the soft Bellman equation ( Equation 3 ) with respect to the dynamics and some reward function . The problem is that the BC loss does not incorporate any information about the dynamics into the learning objective , so Qθ learns to greedily assign high values to demonstrated actions , without considering the state transitions that occur as a consequence of actions . As a result , Qθ may output arbitrary values in states that are off-distribution from the demonstrations Ddemo . In Section 3.2 , we describe a regularized BC algorithm that adds constraints to ensure that Qθ is a valid soft Q function with respect to some implicitly-represented reward function , and further regularizes the implicit rewards with a sparsity prior . In Section 3.3 , we show that this approach recovers an algorithm similar to SQIL .

Summary: The authors propose SQUIL, an off-policy imitation learning (IL) algorithm which attempts to overcome the classic drift problems of behavioral cloning (BC). The idea is to reduce IL to a standard RL problem with a reward that incentivizes the agent to take expert actions in states observed in the demonstrations. The algorithm is tested on both image-based tasks (Atari) and continuous control tasks (Mujoco) and shown effective against GAIL and a simple supervised BC approach.