Visual Imitation Learning with Recurrent Siamese Networks

1 INTRODUCTION . Often in RL the designer formulates a reward function to elicit some desired behaviour in the policy . However , people often modify or refine their objectives as they learn . For example , a gymnast that is learning how to perform a flip can understand the overall motion from a few demonstrations . However , over time the gymnast , along with their previous experience , will learn to understand the less obvious but significant state features that determine a good flipping motion . In this same vein we want to gradually learn a distance function where , as the agent explores and gets more skilled , the agent refines its state space understanding and therefore the distance metric can further refine its accuracy . Robots and people may plan using an internal pose space understanding ; however , typically when people observe others performing tasks only visual information is available . Often , using distances in pose-space is ill-suited for imitation as changing some features will have result in drastically different visual appearance . In order to understand how to perform tasks from visual observation some mapping/transformation is used pose = φ ( image ) , which allows for the minimization of φ ( image ) −agentpose . Even with a method to transform observations to a similar pose every person has different capabilities . Because of this , people must learn how to transform demonstrations into a representation where they can reproduce the behaviour to the best of their ability . In our work here we construct a distance metric derived from the agent ’ s visual perceptions without the need for an intermediate pose representation by allowing the agent to observe itself externally and compare that perception with a demonstration . Searching for a distance function has been an active topic of research ( Abbeel & Ng , 2004 ; Argall et al. , 2009 ) . Given some vector of features the goal is to find an optimal transformation of these features such that when differences are computed in this transformed space there exists a strong contextual meaning . For example , if we wanted a transformation that computed the distance between an agent ’ s standing pose and its current pose , a good schema may prioritize the joint angles of the legs and torso while ignoring momentum . With a meaningful transformation function φ ( · ) a distance can be computed between an agent and a demonstration . Previous work has explored the area of state-based distance functions , but many rely pose based metrics ( Ho & Ermon , 2016 ; Merel et al. , 2017a ) that come from an expert . Few use image based inputs and none consider the importance of learning a distance function in time as well as space ( Sermanet et al. , 2017 ; Finn et al. , 2017 ; Liu et al. , 2017 ; Dwibedi et al. , 2018 ) . In this work we use a recurrent siamese network to learn the distance metric ( Chopra et al. , 2005 ) . An important detail of imitating demonstrations is their sequential and causal nature . There is both an ordering and speed in which the demonstration is performed . It is important to match the demonstrations state distribution . However , similarity between states may force the agent to imitate the same timing as the demonstration . This can be highly effective and lead to learning smooth motions but it also constrains the result to have similar timing as the demonstration . However , when the agent ’ s motion becomes desynchronized with the demonstration the agent will receive low reward . Consider the case when a robot has learned to stand before it can walk . This pose exists inside the demonstration and should be encouraged . Therefore we learned an RNN-based distance function that can give reward for out of sync but similar behaviour . The work in Liu et al . ( 2017 ) ; Dwibedi et al . ( 2018 ) ; Sermanet et al . ( 2017 ) also performs imitation from video observation but each assumes some sort of time alignment between the agent and demonstration . Considering the data sparsity of the problem we include data from other tasks in order to learn a more robust distance function in visual sequence space . Our method has similarities to both Inverse Reinforcement Learning ( IRL ) ( Abbeel & Ng , 2004 ) and Generative Advisarial Imitation Learning ( GAIL ) ( Ho & Ermon , 2016 ) . The process of learning a cost function that will understand the space of policies in order to find an optimal policy given a demonstration is fundamentally IRL . While using positive examples from the expert and negative examples from the policy is similar to the method GAIL used to train a discriminator to understand the desired distribution . In this work we build upon these techniques by constructing a method that can learn polices using only noisy partially observable visual data . We also construct a cost function that takes into account the demonstration timing as well as pose by using a recurrent Siamese network . Our contribution rests on proposing and exploring this form of recurrent Siamese network as a way to address the key problem of defining the reward structure for imitation learning for deep RL agents . 2 PRELIMINARIES . In this section we outline some key details of the general RL framework and specific specialized formulations for RL that we rely upon when developing our method in Section : 3 . 2.1 REINFORCEMENT LEARNING . Using the RL framework formulated with a Markov Dynamic Process ( MDP ) : at every time step t , the world ( including the agent ) exists in a state st ∈ S , wherein the agent is able to perform actions at ∈ A , sampled from a policy π ( st , at ) which results in a new state st+1 ∈ S according to the transition probability function T ( st , at , st+1 ) . Performing action at from state st produces a reward rt from the environment ; the expected future discounted reward from executing a policy π is : J ( π ) = Er0 , ... , rT [ T∑ t=0 γtrt ] ( 1 ) where T is the max time horizon , and γ is the discount factor , indicating the planning horizon length . The agent ’ s goal is to optimize its policy , π , by maximizing J ( π ) . Given policy parameters θπ , the goal is reformulated to identify the optimal parameters θ∗π : θ∗π = arg max θπ J ( π ( ·|θπ ) ) ( 2 ) Using a Gaussian distribution to model the stochastic policy µθt ( st ) . Our stochastic policy is formulated as follows : at ∼ π ( at | st , θπ ) = N ( µ ( st | θπ ) , Σ ) Σ = diag { σ2i } ( 3 ) where Σ is a diagonal covariance matrix with entries σ2i on the diagonal , similar to ( Peng et al. , 2017 ) . For policy optimization we employ stochastic policy gradient methods ( Sutton et al. , 2000 ) . The gradient of the expected future discounted reward with respect to the policy parameters , ∇θπJ ( π ( ·|θπ ) ) , is given by : ∇θπJ ( π ( ·|θπ ) ) = ∫ S dθ ( s ) ∫ A ∇θπ log ( π ( a , s|θπ ) ) Aπ ( s , a ) da ds ( 4 ) where dθ = ∫ S ∑T t=0 γ tp0 ( s0 ) ( s0 → s | t , π0 ) ds0 is the discounted state distribution , p0 ( s ) represents the initial state distribution , and p0 ( s0 ) ( s0 → s | t , π0 ) models the likelihood of reaching state s by starting at state s0 and following the policy π ( a , s|θπ ) for T steps ( Silver et al. , 2014 ) . Aπ ( s , a ) represents an advantage function ( Schulman et al. , 2016 ) . 2.2 IMITATION LEARNING . Imitation learning is the process of training a new policy to reproduce the behaviour of some expert policy . Behavioural Cloning ( BC ) is a fundamental method for imitation learning . Given an expert policy πE possibly represented as a collection of trajectories τ < ( s0 , a0 ) , . . . , ( sT , aT ) > a new policy π can be learned to match this trajectory using supervised learning . max θ EπE [ T∑ t=0 logπ ( at|st , θπ ) ] ( 5 ) While this simple method can work well , it often suffers from distribution mismatch issues leading to compounding errors as the learned policy deviates from the expert ’ s behaviour . Similar to BC , IRL also learns to replicate some desired behaviour . However , IRL makes use of the environment , using the RL environment without a defined reward function . Here we describe maximal entropy IRL ( Ziebart et al. , 2008 ) . Given an expert trajectory τ < ( s0 , a0 ) , . . . , ( sT , aT ) > a policy π can be trained to produce similar trajectories by discovering a distance metric between the expert trajectory and trajectories produced by the policy π. max c∈C min π ( Eπ [ c ( s , a ) ] −H ( π ) ) − EπE [ c ( s , a ) ] ( 6 ) where c is some learned cost function and H ( π ) is a causal entropy term . πE is the expert policy that is represented by a collection of trajectories . IRL is searching for a cost function c that is low for the expert πE and high for other policies . Then , a policy can be optimized by maximizing the reward function rt = −c ( st , at ) . GAIL ( Ho & Ermon , 2016 ) uses a Generative Advasarial Network ( GAN ) -based ( Goodfellow et al. , 2014 ) framework where the discriminator is trained with positive examples from the expert trajectories and negative examples from the policy . The generator is a combination of the environment and the current state visitation probability induced by the policy pπ ( s ) . min θπ max θφ EπE [ log ( D ( s , a|θφ ) ) ] + Eπθπ [ log ( 1−D ( s , a|θφ ) ) ] ( 7 ) In this framework the discriminator provides rewards for the RL policy to optimize , as the probability of a state generated by the policy being in the distribution rt = D ( st , at|θφ ) . 3 OUR APPROACH . In this section we describe our method to perform recurrent vision based imitation learning . 3.1 PARTIAL OBSERVABLE IMITATION LEARNING WITHOUT ACTIONS . For many problems we want to learn how to replicate the behaviour of some expert πE without access to the experts actions . Instead , we may only have access to an actionless noisy observation of the expert that we call a demonstration . Recent work uses BC to learn a new model to estimate the actions used via maximum-likelihood estimation ( Torabi et al. , 2018 ) . Still , BC often needs many expert examples and tends to suffer from state distribution mismatch issues between the expert policy and the student ( Ross et al. , 2011 ) . Work in ( Merel et al. , 2017b ) proposes a system based on GAIL that can learn a policy from a partial observation of the demonstration . In this work the state input to the discriminator is a customized version of the expert ’ s pose and does not take into account the demonstration ’ s sequential nature . The work in ( Wang et al. , 2017 ) provides a more robust GAIL framework along with a new model to encode motions for few-shot imitation . This model uses an RNN to encode a demonstration but uses expert state and action observations . In our work we limit the agent to only a partial visual observation as a demonstration . Additional works learn implicit models of distance ( Yu et al. , 2018 ; Pathak et al. , 2018 ; Finn et al. , 2017 ; Sermanet et al. , 2017 ) , none of these explicitly learn a sequential model to use the demonstration timing . Another version of GAIL , infoGAIL ( Li et al. , 2017 ) , was used on some pixel based inputs . In contrast , here we train a recurrent siamese model that can be used to enable curriculum learning and allow for reasonable distances to be computed even when the agent and demonstration are out of sync or have different capabilities .

Summary: This paper aims to imitate, via Imitation Learning, the actions of a humanoid agent given only video demonstrations of the desired task, including walking, running, back-flipping, and front-flipping. Since the algorithm does not have direct access to the underlying actions or rewards, the agent aims to learn an embedding space over instances with the hope that distance in this embedding space corresponds to reward. Deep RL is used to optimize a policy to maximize cumulative reward in an effort to reproduce the behavior of the expert.