Goal-Driven Imitation Learning from Observation by Inferring Goal Proximity

1 INTRODUCTION . Humans are capable of effectively leveraging demonstrations from experts to solve a variety of tasks . Specifically , by watching others performing a task , we can learn to infer how close we are to completing the task , and then take actions towards states closer to the goal of the task . For example , after watching a few tutorial videos for chair assembly , we learn to infer how close an intermediate configuration of a chair is to completion . With the guidance of such a task progress estimate , we can efficiently learn to assemble the chair to progressively get closer to and eventually reach , the fully assembled chair . Can machines likewise first learn an estimate of progress towards a goal from demonstrations and then use this estimate as guidance to move closer to and eventually reach the goal ? Typical learning from demonstration ( LfD ) approaches ( Pomerleau , 1989 ; Pathak et al. , 2018 ; Finn et al. , 2016 ) greedily imitate the expert policy and therefore suffer from accumulated errors causing a drift away from states seen in the demonstrations . On the other hand , adversarial imitation learning approaches ( Ho & Ermon , 2016 ; Fu et al. , 2018 ) encourage the agent to imitate expert trajectories with a learned reward that distinguishes agent and expert behaviors . However , such adversarially learned reward functions often overfit to the expert demonstrations and do not generalize to states not covered in the demonstrations ( Zolna et al. , 2019 ) , leading to unsuccessful policy learning . Inspired by how humans leverage demonstrations to measure progress and complete tasks , we devise an imitation learning from observation ( LfO ) method which learns a task progress estimator and uses the task progress estimate as a dense reward signal for training a policy as illustrated in Figure 1 . To measure the progress of a goal-driven task , we define goal proximity as an estimate of temporal distance to the goal ( i.e. , the number of actions required to reach the goal ) . In contrast to prior adversarial imitation learning algorithms , by having additional supervision of task progress and learning to predict it , the goal proximity function can acquire more structured task-relevant information , and hence generalize better to unseen states and provide better reward signals . However , the goal proximity function can still output inaccurate predictions on states not in demonstrations , which results in unstable policy training . To improve the accuracy of the goal proximity function , we continually update the proximity function with trajectories both from expert and agent . In addition , we penalize trajectories with the uncertainty of the goal proximity prediction , which prevents the policy from exploiting high proximity estimates with high uncertainty . As a result , by leveraging the agent experience and predicting the proximity function uncertainty , our method can achieve more efficient and stable policy learning . The main contributions of this paper include ( 1 ) an algorithm for imitation from observation that uses estimated goal proximity to inform an agent of the task progress ; ( 2 ) modeling uncertainty of goal proximity estimation to prevent exploiting uncertain predictions ; and ( 3 ) a joint training algorithm of the goal proximity function and policy . We show that the policy learned with our proposed goal proximity function is more effective and generalizes better than the state-of-the-art LfO algorithms on various domains , such as navigation , robot manipulation , and locomotion . Moreover , our method demonstrates comparable results with GAIL ( Ho & Ermon , 2016 ) , which learns from expert actions . 2 RELATED WORK . Imitation learning ( Schaal , 1997 ) aims to leverage expert demonstrations to acquire skills . While behavioral cloning ( Pomerleau , 1989 ) is simple but effective with a large number of demonstrations , it suffers from compounding errors caused by the distributional drift ( Ross et al. , 2011 ) . On the other hand , inverse reinforcement learning ( Ng & Russell , 2000 ; Abbeel & Ng , 2004 ; Ziebart et al. , 2008 ) estimates the underlying reward from demonstrations and learns a policy through reinforcement learning with this reward , which can better handle the compounding errors . Specifically , generative adversarial imitation learning ( GAIL ) ( Ho & Ermon , 2016 ) and its variants ( Fu et al. , 2018 ; Kostrikov et al. , 2020 ) shows improved demonstration efficiency by training a discriminator to distinguish expert and agent transitions and using the discriminator output as a reward for policy training . While most imitation learning algorithms require expert actions , imitation learning from observation ( LfO ) approaches learn from state-only demonstrations . This enables the LfO methods to learn from diverse sources of demonstrations , such as human videos , demonstrations with different controllers , and other robots . To imitate demonstrations without expert actions , inverse dynamics models ( Niekum et al. , 2015 ; Torabi et al. , 2018a ; Pathak et al. , 2018 ) or learned reward functions ( Edwards et al. , 2016 ; Sermanet et al. , 2017 ; 2018 ; Liu et al. , 2018 ; Lee et al. , 2019a ) can be used to train the policy . However , these methods require large amounts of data to train inverse dynamics models or representations . On the other hand , state-only adversarial imitation learning ( Torabi et al. , 2018b ; Yang et al. , 2019 ) can imitate an expert with few demonstrations , similar to GAIL . In addition to discriminating expert and agent trajectories , our method proposes to also estimate the proximity to the goal , which can provide more informed reward signals and generalize better . Closely related works to our approach are reinforcement learning algorithms that learn a value function or proximity estimator from successful trajectories and use them as an auxiliary reward ( Mataric , 1994 ; Edwards & Isbell , 2019 ; Lee et al. , 2019b ) . While these value function and proximity estimator are similar to our proposed goal proximity function , these works require environment reward signals , and do not incorporate adversarial online training and uncertainty estimates . Moreover , demonstrating the value of learning a proximity estimate for imitation learning , Angelov et al . ( 2020 ) utilizes the learned proximity to choose a proper sub-policy but does not train a policy from the learned proximity . Similar to our method , Burke et al . ( 2020 ) proposes to learn a reward function using a ranking model and use it for policy optimization , demonstrating the advantage of using goal proximity as a reward for training a policy . However , they learn the proximity function from demonstrations alone and solely provide proximity as a reward . This hinders agent learning when the proximity function fails to generalize to agent experience , allowing the agent to exploit inaccurate proximity predictions for reward . By incorporating the online update , uncertainty estimates , and difference-based proximity reward , our method can robustly imitate state-only demonstrations to solve goal-driven tasks without access to the true environment reward . 3 METHOD . In this paper , we address the problem of learning from observations for goal-driven tasks . Adversarial imitation learning methods ( Torabi et al. , 2018b ; Yang et al. , 2019 ) suggest learning a reward function that penalizes an agent state transition off the expert trajectories . However , these learned reward functions often overfit to expert demonstrations and do not generalize to states which are not covered in the demonstrations , leading to unsuccessful policy learning . To acquire a more structured and generalizable reward function from demonstrations , we propose to learn a goal proximity function that estimates proximity to the goal distribution in terms of temporal distance ( i.e. , number of actions required to reach the goal ) . Then , a policy learns to reach states with higher proximity ( i.e. , that are closer to the goal ) predicted by the goal proximity function . Moreover , during policy training , we propose to measure the uncertainty of the goal proximity function which prevents the policy from exploiting over-optimistic proximity predictions and yielding undesired behaviors . In Section 3.2 , we describe the goal proximity function in detail . Then , in Section 3.3 , we elaborate how the policy is jointly trained with the goal proximity function . 3.1 PRELIMINARIES . We formulate our learning problem as a Markov decision process ( Sutton , 1984 ) defined through a tuple ( S , A , R , P , ρ0 , γ ) for the state space S , action space A , reward function R ( s , a ) , transition distribution P ( s′|s , a ) , initial state distribution ρ0 , and discounting factor γ . We define a policy π ( a|s ) that maps from states to actions and correspondingly moves an agent to a new state according to the transition probabilities . The policy is trained to maximize the expected sum of discounted rewards , E ( s , a ) ∼π [ ∑Ti t=0 γ tR ( st , at ) ] , where Ti represents the variable length of episode i . In imitation learning , the learner receives a fixed set of expert demonstrations , De = { τe1 , . . . , τeN } . In this paper , we specifically consider the learning from observation ( LfO ) setup where each demonstration τei is a sequence of states . Moreover , we assume that all expert demonstrations are successful ; therefore , the final state of an expert trajectory reaches the task goal . 3.2 LEARNING GOAL PROXIMITY FUNCTION . In goal-driven tasks , an estimate of how close an agent is to the goal can be utilized as a direct learning signal . Therefore , instead of learning to discriminate agent and expert trajectories ( Ho & Ermon , 2016 ; Torabi et al. , 2018b ) , we propose a goal proximity function , f : S → R , that learns how close states are to the goal distribution . Specifically , we define goal proximity as a proximity that is discounted based on its temporal distance to the goal ( i.e. , inversely proportional to the number of actions required to reach the goal ) . Note that the goal proximity function measures the temporal distance , not the spatial distance , between the current and goal states . Therefore , a single proximity value can entail all information about the task , goal , and any roadblocks . In this paper , we define goal proximity of a state st as the linearly discounted proximity f ( st ) = 1− δ · ( Ti − t ) , where δ ∈ ( 0 , 1 ) is a discounting factor and Ti is the episode horizon . In this paper , we set δ to 1/H to evenly distribute the proximity between 0 and 1 , where H is the maximum task horizon . Note that we use the maximum episode length H , instead of the variable episode length Ti , to define a fixed δ for the temporal discounting to be consistent between episodes . We use mean squared error as the objective for training the goal proximity function fφ parameterized by φ : Lφ = Eτei ∼De , st∼τei [ fφ ( st ) − ( 1− δ · ( Ti − t ) ) ] 2 . ( 1 ) Algorithm 1 Imitation learning with goal proximity function Require : Expert demonstrations De = { τe1 , . . . , τeN } 1 : Initialize weights of goal proximity function fφ and policy πθ 2 : for i = 0 , 1 , ... , M do 3 : Sample expert demonstration τe ∼ De . Offline proximity function training 4 : Update goal proximity function fφ with τe to minimize Equation 1 5 : end for 6 : for i = 0 , 1 , ... , L do 7 : Rollout trajectories τi = ( s0 , . . . , sTi ) with πθ . Policy training 8 : Compute proximity reward Rφ ( st , st+1 ) for ( st , st+1 ) ∼ τi using Equation 5 9 : Update πθ using any RL algorithm 10 : Update fφ with τi and τe ∼ De to minimize Equation 4 11 : end for There are alternative ways to represent and learn goal proximity , such as exponentially discounted proximity and ranking-based proximity ( Brown et al. , 2019 ) . But , in our experiments , linearly discounted proximity consistently performed better than alternatives ; therefore , the linearly discounted proximity is used throughout this paper ( see Figure 5b and Figure 11 ) . By learning to predict the goal proximity , the goal proximity function not only learns to discriminate agent and expert trajectories ( i.e. , predict 0 proximity for an agent trajectory and positive proximity for an expert trajectory with Equation 4 ) , but also acquires the task information about temporal progress entailed in the trajectories . From this additional supervision , the proximity function provides more informative learning signals to the policy and generalizes better to unseen states as empirically shown in Section 4 .

Summary: The authors propose a new method for imitation learning from observation that attempts to estimate and leverage a notion of goal proximity in order to help the learning process. The authors provide a framework for computing this estimate, and a technique for using that estimate -- along with a measure of uncertainty -- to perform imitation learning from observation. Experimental results for several domains are presented in which the proposed technique achieves better performance than the comparison methods.