Year: 2020

Abstract:
Humans can effectively learn to estimate how close they are to completing a desired task simply by watching others fulfill the task. To solve the task, they can then take actions towards states with higher estimated proximity to the goal. From this intuition, we propose a simple yet effective method for imitation learning that learns a goal proximity function from expert demonstrations and online agent experience, and then uses the learned proximity to provide a dense reward signal for training a policy to solve the task. By predicting task progress as the temporal distance to the goal, the goal proximity function improves generalization to unseen states over methods that aim to directly imitate expert behaviors. We demonstrate that our proposed method efficiently learns a set of goal-driven tasks from state-only demonstrations in navigation, robotic arm manipulation, and locomotion tasks.

heading: 1 INTRODUCTION
text: Humans are capable of effectively leveraging demonstrations from experts to solve a variety of tasks. Specifically, by watching others performing a task, we can learn to infer how close we are to completing the task, and then take actions towards states closer to the goal of the task. For example, after watching a few tutorial videos for chair assembly, we learn to infer how close an intermediate configuration of a chair is to completion. With the guidance of such a task progress estimate, we can efficiently learn to assemble the chair to progressively get closer to and eventually reach, the fully assembled chair.
Can machines likewise first learn an estimate of progress towards a goal from demonstrations and then use this estimate as guidance to move closer to and eventually reach the goal? Typical learning from demonstration (LfD) approaches (Pomerleau, 1989; Pathak et al., 2018; Finn et al., 2016) greedily imitate the expert policy and therefore suffer from accumulated errors causing a drift away from states seen in the demonstrations. On the other hand, adversarial imitation learning approaches (Ho & Ermon, 2016; Fu et al., 2018) encourage the agent to imitate expert trajectories with a learned reward that distinguishes agent and expert behaviors. However, such adversarially learned reward functions often overfit to the expert demonstrations and do not generalize to states not covered in the demonstrations (Zolna et al., 2019), leading to unsuccessful policy learning.
Inspired by how humans leverage demonstrations to measure progress and complete tasks, we devise an imitation learning from observation (LfO) method which learns a task progress estimator and uses the task progress estimate as a dense reward signal for training a policy as illustrated in Figure 1. To measure the progress of a goal-driven task, we define goal proximity as an estimate of temporal distance to the goal (i.e., the number of actions required to reach the goal). In contrast to prior adversarial imitation learning algorithms, by having additional supervision of task progress and learning to predict it, the goal proximity function can acquire more structured task-relevant information, and hence generalize better to unseen states and provide better reward signals.
However, the goal proximity function can still output inaccurate predictions on states not in demonstrations, which results in unstable policy training. To improve the accuracy of the goal proximity function, we continually update the proximity function with trajectories both from expert and agent. In addition, we penalize trajectories with the uncertainty of the goal proximity prediction, which prevents the policy from exploiting high proximity estimates with high uncertainty. As a result, by leveraging the agent experience and predicting the proximity function uncertainty, our method can achieve more efficient and stable policy learning.
The main contributions of this paper include (1) an algorithm for imitation from observation that uses estimated goal proximity to inform an agent of the task progress; (2) modeling uncertainty of goal proximity estimation to prevent exploiting uncertain predictions; and (3) a joint training algorithm of the goal proximity function and policy. We show that the policy learned with our proposed goal proximity function is more effective and generalizes better than the state-of-the-art LfO algorithms on various domains, such as navigation, robot manipulation, and locomotion. Moreover, our method demonstrates comparable results with GAIL (Ho & Ermon, 2016), which learns from expert actions.

heading: 2 RELATED WORK
text: Imitation learning (Schaal, 1997) aims to leverage expert demonstrations to acquire skills. While behavioral cloning (Pomerleau, 1989) is simple but effective with a large number of demonstrations, it suffers from compounding errors caused by the distributional drift (Ross et al., 2011). On the other hand, inverse reinforcement learning (Ng & Russell, 2000; Abbeel & Ng, 2004; Ziebart et al., 2008) estimates the underlying reward from demonstrations and learns a policy through reinforcement learning with this reward, which can better handle the compounding errors. Specifically, generative adversarial imitation learning (GAIL) (Ho & Ermon, 2016) and its variants (Fu et al., 2018; Kostrikov et al., 2020) shows improved demonstration efficiency by training a discriminator to distinguish expert and agent transitions and using the discriminator output as a reward for policy training.
While most imitation learning algorithms require expert actions, imitation learning from observation (LfO) approaches learn from state-only demonstrations. This enables the LfO methods to learn from diverse sources of demonstrations, such as human videos, demonstrations with different controllers, and other robots. To imitate demonstrations without expert actions, inverse dynamics models (Niekum et al., 2015; Torabi et al., 2018a; Pathak et al., 2018) or learned reward functions (Edwards et al., 2016; Sermanet et al., 2017; 2018; Liu et al., 2018; Lee et al., 2019a) can be used to train the policy. However, these methods require large amounts of data to train inverse dynamics models or representations. On the other hand, state-only adversarial imitation learning (Torabi et al., 2018b; Yang et al., 2019) can imitate an expert with few demonstrations, similar to GAIL. In addition to discriminating expert and agent trajectories, our method proposes to also estimate the proximity to the goal, which can provide more informed reward signals and generalize better.
Closely related works to our approach are reinforcement learning algorithms that learn a value function or proximity estimator from successful trajectories and use them as an auxiliary reward (Mataric, 1994; Edwards & Isbell, 2019; Lee et al., 2019b). While these value function and proximity estimator are similar to our proposed goal proximity function, these works require environment reward signals, and do not incorporate adversarial online training and uncertainty estimates.
Moreover, demonstrating the value of learning a proximity estimate for imitation learning, Angelov et al. (2020) utilizes the learned proximity to choose a proper sub-policy but does not train a policy
from the learned proximity. Similar to our method, Burke et al. (2020) proposes to learn a reward function using a ranking model and use it for policy optimization, demonstrating the advantage of using goal proximity as a reward for training a policy. However, they learn the proximity function from demonstrations alone and solely provide proximity as a reward. This hinders agent learning when the proximity function fails to generalize to agent experience, allowing the agent to exploit inaccurate proximity predictions for reward. By incorporating the online update, uncertainty estimates, and difference-based proximity reward, our method can robustly imitate state-only demonstrations to solve goal-driven tasks without access to the true environment reward.

heading: 3 METHOD
text: In this paper, we address the problem of learning from observations for goal-driven tasks. Adversarial imitation learning methods (Torabi et al., 2018b; Yang et al., 2019) suggest learning a reward function that penalizes an agent state transition off the expert trajectories. However, these learned reward functions often overfit to expert demonstrations and do not generalize to states which are not covered in the demonstrations, leading to unsuccessful policy learning.
To acquire a more structured and generalizable reward function from demonstrations, we propose to learn a goal proximity function that estimates proximity to the goal distribution in terms of temporal distance (i.e., number of actions required to reach the goal). Then, a policy learns to reach states with higher proximity (i.e., that are closer to the goal) predicted by the goal proximity function. Moreover, during policy training, we propose to measure the uncertainty of the goal proximity function which prevents the policy from exploiting over-optimistic proximity predictions and yielding undesired behaviors. In Section 3.2, we describe the goal proximity function in detail. Then, in Section 3.3, we elaborate how the policy is jointly trained with the goal proximity function.

heading: 3.1 PRELIMINARIES
text: We formulate our learning problem as a Markov decision process (Sutton, 1984) defined through a tuple (S,A, R, P, ρ0, γ) for the state space S, action space A, reward function R(s, a), transition distribution P (s′|s, a), initial state distribution ρ0, and discounting factor γ. We define a policy π(a|s) that maps from states to actions and correspondingly moves an agent to a new state according to the transition probabilities. The policy is trained to maximize the expected sum of discounted rewards, E(s,a)∼π [∑Ti t=0 γ tR(st, at) ] , where Ti represents the variable length of episode i.
In imitation learning, the learner receives a fixed set of expert demonstrations, De = {τe1 , . . . , τeN}. In this paper, we specifically consider the learning from observation (LfO) setup where each demonstration τei is a sequence of states. Moreover, we assume that all expert demonstrations are successful; therefore, the final state of an expert trajectory reaches the task goal.

heading: 3.2 LEARNING GOAL PROXIMITY FUNCTION
text: In goal-driven tasks, an estimate of how close an agent is to the goal can be utilized as a direct learning signal. Therefore, instead of learning to discriminate agent and expert trajectories (Ho & Ermon, 2016; Torabi et al., 2018b), we propose a goal proximity function, f : S → R, that learns how close states are to the goal distribution. Specifically, we define goal proximity as a proximity that is discounted based on its temporal distance to the goal (i.e., inversely proportional to the number of actions required to reach the goal). Note that the goal proximity function measures the temporal distance, not the spatial distance, between the current and goal states. Therefore, a single proximity value can entail all information about the task, goal, and any roadblocks.
In this paper, we define goal proximity of a state st as the linearly discounted proximity f(st) = 1− δ · (Ti − t), where δ ∈ (0, 1) is a discounting factor and Ti is the episode horizon. In this paper, we set δ to 1/H to evenly distribute the proximity between 0 and 1, where H is the maximum task horizon. Note that we use the maximum episode length H , instead of the variable episode length Ti, to define a fixed δ for the temporal discounting to be consistent between episodes. We use mean squared error as the objective for training the goal proximity function fφ parameterized by φ:
Lφ = Eτei ∼De,st∼τei [fφ(st)− (1− δ · (Ti − t))] 2 . (1)
Algorithm 1 Imitation learning with goal proximity function Require: Expert demonstrations De = {τe1 , . . . , τeN}
1: Initialize weights of goal proximity function fφ and policy πθ 2: for i = 0, 1, ...,M do 3: Sample expert demonstration τe ∼ De . Offline proximity function training 4: Update goal proximity function fφ with τe to minimize Equation 1 5: end for 6: for i = 0, 1, ..., L do 7: Rollout trajectories τi = (s0, . . . , sTi) with πθ . Policy training 8: Compute proximity reward Rφ(st, st+1) for (st, st+1) ∼ τi using Equation 5 9: Update πθ using any RL algorithm
10: Update fφ with τi and τe ∼ De to minimize Equation 4 11: end for
There are alternative ways to represent and learn goal proximity, such as exponentially discounted proximity and ranking-based proximity (Brown et al., 2019). But, in our experiments, linearly discounted proximity consistently performed better than alternatives; therefore, the linearly discounted proximity is used throughout this paper (see Figure 5b and Figure 11).
By learning to predict the goal proximity, the goal proximity function not only learns to discriminate agent and expert trajectories (i.e., predict 0 proximity for an agent trajectory and positive proximity for an expert trajectory with Equation 4), but also acquires the task information about temporal progress entailed in the trajectories. From this additional supervision, the proximity function provides more informative learning signals to the policy and generalizes better to unseen states as empirically shown in Section 4.

heading: 3.3 TRAINING POLICY WITH PROXIMITY REWARD
text: In a goal-driven task, a policy πθ aims to get close to and eventually reach the goal. We can formalize this objective as maximizing the goal proximity at the final state fφ(sTi), which can be used as a sparse proximity reward. In addition, to encourage the agent to make consistent progress towards the goal, we devise a dense proximity reward based on the increase in proximity, fφ(st+1)− fφ(st), at every timestep. By combining the sparse and dense proximity rewards, our total proximity reward can be defined as
Rφ(st, st+1) = { fφ(st+1)− fφ(st) t 6= Ti − 1 2 · fφ(sTi)− fφ(st) t = Ti − 1 . (2)
Given the proximity reward, the policy is trained to maximize the expected discounted return:
E(s0,...,sTi )∼πθ [ γT fφ(sTi) + Ti−1∑ t=0 γt(fφ(st+1)− fφ(st)) ] . (3)
However, a policy trained with the proximity reward can sometimes perform undesired behaviors by exploiting over-optimistic proximity predictions on states not seen in the expert demonstrations. This becomes critical when the expert demonstrations are limited and cannot cover the state space sufficiently. To avoid inaccurate predictions leading an agent to undesired states, we propose to (1) fine-tune the goal proximity function with online agent experience to reduce optimistic proximity evaluations; and (2) penalize agent trajectories with high uncertainty in goal proximity predictions.
First, we set the target proximity of states in agent trajectories to 0, similar to adversarial imitation learning methods (Ho & Ermon, 2016), and train the proximity function with both expert demonstrations and agent experience by minimizing the following loss:
Lφ = Eτei ∼De,st∼τei [fφ(st)− (1− δ · (Ti − t))] 2 + Eτ∼πθ,st∼τ [fφ(st)] 2 . (4)
Although successful agent experience is also used as negative examples for training the proximity function, in practice, this is not problematic since the proximity function ideally converges to the average of expert and agent labels (e.g., 1/2 − δ · (Ti − t)/2 for ours and 0.5 for GAIL). Early stopping and learning rate decay can be used to further ease this problem (Zolna et al., 2019). Also,
the online training of the goal proximity function can lower the goal proximity estimate in a local optimum, which helps the policy escape from such local optima.
To alleviate the effect of inaccurate proximity estimation in policy training, we discourage the policy from visiting states with uncertain proximity estimates. Specifically, we model the uncertainty U(st) as the disagreement in terms of standard deviation of an ensemble of proximity functions (Osband et al., 2016; Lakshminarayanan et al., 2017). Then we use the estimated uncertainty to penalize exploration of states with uncertain proximity estimates. The proximity estimate f(st) is the average prediction of the ensemble. With the uncertainty penalty, the modified reward can be written as:
Rφ(st, st+1) = { fφ(st+1)− fφ(st)− λ · U(st+1) t 6= Ti − 1 2 · fφ(sTi)− fφ(st)− λ · U(sTi) t = Ti − 1 , (5)
where λ is a tunable hyperparameter to balance the proximity reward and uncertainty penalty. A larger λ results in more conservative exploration outside the states covered by the expert demonstrations
In summary, we propose to learn a goal proximity function that estimates how close the current state is to the goal distribution, and train a policy to maximize the goal proximity while avoiding states with inaccurate proximity estimates using the uncertainty measure. We jointly train the proximity function and policy as described in Figure 1 and Algorithm 1.

heading: 4 EXPERIMENTS
text: In our experiments, we aim to answer the following questions: (1) How does our method’s efficiency and final performance compare against prior work in imitation from observation and imitation learning with expert actions? (2) Does our method lead to policies that generalize better to states unseen in the demonstrations? (3) What factors contribute to the performance of our method? To answer these questions we consider diverse goal-driven tasks: navigation, robot manipulation, and locomotion.
To demonstrate the improved generalization capabilities of policies trained with the proximity reward, we benchmark our method under two different setups: expert demonstrations are collected from (1) only a fraction of the possible initial states (e.g., 25%, 50%, 75% coverage) and (2) initial states with smaller amounts of noise. (1) measures the ability of an agent to interpolate between states covered by the demonstrations while (2) evaluates extrapolating beyond the demonstrations to added noise during agent learning. In both setups, our method shows superior generalization capability and thus, achieves higher final rewards than LfO baselines. Moreover, our method achieves comparable results with LfD methods that use expert actions.
These generalization experiments serve to mimic the reality that expert demonstrations may be collected in a different setting from agent learning. For instance, due to the cost of demonstration collection, the demonstrations may poorly cover the state space. An agent would then have to learn in an area of the state space not covered by the demonstrations. We measure this in the experimental setup (1), where the demonstrations cover a fraction of the possible learner starting and goal states. Likewise, demonstrations may be collected in controlled circumstances with little environment noise. Then, an agent learning in an actual environment would encounter more noise than presented in
the demonstrations. We quantify this in the experimental setup (2), where less noise is applied to demonstration starting states.

heading: 4.1 BASELINES
text: We compare our method to the state-of-the-art prior works in both imitation learning from observations and standard imitation learning with actions, which are listed below:
• BCO (Torabi et al., 2018a) learns an inverse model from environment interaction to provide action labels in demonstrations for behavioral cloning. • GAIfO (Torabi et al., 2018b) is a variant of GAIL (Ho & Ermon, 2016) which trains a discrimi-
nator to discriminate state transitions (s, s′) instead of state-action pairs (s, a). • GAIfO-s, as compared to in Yang et al. (2019), learns a discriminator based off only a single
state, not a state transition as with GAIfO. • BC (Pomerleau, 1989) fits a policy to the demonstration actions with supervised learning. This
method requires expert action labels while our method does not. • GAIL (Ho & Ermon, 2016) uses adversarial imitation learning with a discriminator trained on
state-action pairs (s, a). This method also uses actions whereas ours does not.
Also, we study several variations of our method to evaluate the importance of different design choices:
• Ours (No Uncert): Removes the uncertainty penalty from the reward function. • Ours (No Online): Learns the proximity function offline from the demonstrations and does not
refine it using agent experience during policy learning. This approach may fail as the proximity function will not learn outside of the demonstrations and thus provide a poor reward signal. • Ours (No Offline): Does not pre-train the proximity function. This should be less efficient than
our method, which pre-trains the proximity function using the demonstrations. • Ours (Exp): Uses the exponentially discounted goal proximity f(st) = δ(T−t).

heading: 4.2 EXPERIMENTAL SETUP
text: By default, our primary method uses the linearly discounted version of the proximity function as this empirically lead to the best results (see details in Figure 11) and set the discounting factor as δ = 1/H , whereH is the task horizon length. For modeling uncertainty, we use an ensemble of size 5 across all tasks. For all tasks, we pre-train the proximity function for 5 epochs on the demonstrations. During online training (i.e., policy learning), we sample a batch of 128 elements from the expert and agent experience buffers. The mean and standard deviation of outputs from the ensemble networks are used as the proximity prediction and uncertainty, respectively.
The same network architecture is used for proximity function, discriminator (for the baselines), and policy. Details of the network architecture can be found in Section G.2. Any reinforcement learning algorithm can be used for policy optimization, but we choose to use PPO (Schulman et al., 2017) and the hyperparameters of PPO are tuned appropriately for each method and task (see Table 2). Each baseline implementation is verified against the results reported in its original paper. We train each task with 5 different random seeds and report mean and standard deviation divided by 2.

heading: 4.3 NAVIGATION
text: In the first set of experiments, we examine the NAVIGATION task between four rooms shown in Figure 2a. The purpose of this environment is to show the benefits of our method in a simple setting where we can easily visualize and verify the learned goal proximity function. The agent start and goal positions are randomly sampled and the agent has 100 steps to navigate to the goal. We provide 250 expert demonstrations obtained using a shortest path algorithm. During demonstration collection, we hold out 50% of the possible agent start and goal positions determined by uniform random sampling. In contrast, during agent learning and evaluation, start and goal positions are sampled from all possible positions.
As can be seen in Figure 3a, our method achieves near 100% success rate in 3M environment steps, while all GAIL variants fail to learn the task. Although BC and BCO could achieve the goal for about 60% and 30% cases respectively, they show limited generalization to unseen configurations. This
result proves that learning with the goal proximity reward is effective and the learned goal proximity function generalizes well to unseen configurations.
To verify whether the proximity function learns meaningful information of proximity to the goal, we visualize the proximity from all agent positions in Figure 4d. Our proximity function predicts a qualitatively meaningful goal proximity: high estimated proximity near the goal and lower estimated proximity when the agent is farther away from the goal. The corners of the rooms show low goal proximity since less expert trajectories pass over those regions compared to the center of each room.
Finally, we investigate our hypothesis that the goal proximity function allows for greater generalization, which results in better task performance with smaller demonstration coverage. To test this hypothesis, we compare the cases where extreme (25% coverage), moderate, and no generalization (100% coverage) are required. Figure 4 demonstrates that our method consistently achieves 100% success rates in 3M steps with 50%-100% demonstration coverages and is not as affected by increasingly difficult generalization as baselines. In contrast, GAIL and all LfO baselines fail to learn the NAVIGATION task when expert demonstrations do not cover all configurations. This supports our hypothesis that the goal proximity function is able to capture the task structure and therefore, generalize better to unseen configurations.

heading: 4.4 ROBOT MANIPULATION
text: We further evaluate our method in two continuous control tasks: FETCH PICK and FETCH PUSH from Plappert et al. (2018). In the FETCH PICK task shown in Figure 2b, a Fetch robotic arm must
grasp and move a block to a target position. In FETCH PUSH, the Fetch arm pushes a block to a target position, as shown in Figure 2c. Both the initial position of the block and target are randomly initialized. For each, we provide 1k demonstrations, consisting of 33k and 28k transitions for FETCH PICK and FETCH PUSH respectively, generated using a hand-engineered policy . We create a 50% holdout set of starting states for agent learning by splitting the continuous state space into a 4 by 4 grid and holding out two cells per row to sample the block and target starting positions from.
In the FETCH PICK task, our method achieves more than 80% success rate while the success rates of GAIfO and GAIfO-s are upper-bounded by 50% due to the limited coverage of expert demonstrations (see Figure 3b). Our method learns slower than GAIL but achieves comparable final performance even though GAIL learns from expert actions. The FETCH PUSH task is more challenging than FETCH PICK due to the more complicated contact dynamics for pushing interactions. In Figure 3c, the demonstrations are collected with full coverage but the policy is trained in a version of the environment with 2 times larger noise applied to the starting state. All methods fail to learn diagonal pushing movements but our method still learns horizontal pushing faster and achieves higher performance than all other baselines. We evaluate both FETCH tasks under two different generalization setups, different demonstration coverages (Figure 8) and different amounts of noise (Figure 9), and the results consistently show that our proximity function is able to accelerate policy learning in continuous control environments with superior generalization capability.

heading: 4.5 ANT LOCOMOTION
text: We used the ANT REACH environment proposed in Ghosh et al. (2018), simulated in the MuJoCo physics engine (Todorov et al., 2012). In this task, the quadruped ant is tasked to reach a randomly generated goal, which is along the perimeter of a half circle of radius 5m centered around the ant (see Figure 2d). We provide 1k demonstrations, which contain 25k transitions in total. When demonstrations are collected, no noise is added to the initial pose of the ant whereas random noise is added during policy learning, which requires the reward functions to generalize to unseen states.
In Figure 3d, with 0.03 added noise, our method achieves 35% success rate while BCO, GAIfO, and GAIfO-s achieve 1%, 2%, and 7%, respectively. This result illustrates the importance of learning proximity over learning to discriminate expert and agent states for generalization to unseen states. The performance of GAIfO and GAIfO-s drops drastically with larger joint angle randomness as shown in Figure 9. As the ANT REACH task is not as sensitive to noise in actions compared to other tasks, BC and GAIL show superior results but our method still achieves comparable performance.
We also ablate the various aspects of our method in Figure 5. First, we verify the effect of the uncertainty penalty used in the proximity reward. The learning curves with different λ are plotted in Figure 5a and demonstrate that our method works the best with λ = 0.1. Both too low and too high uncertainty penalties degrade the performance. Figure 5b shows the linearly discounted proximity function learns marginally faster than the exponentially discounted proximity function. In Figure 5c, we test the importance of online and offline training of the proximity function. The result shows that the agent fails to learn the task without online updates using agent trajectories. Meanwhile, no proximity function pre-training lowers performance.

heading: 4.6 ABLATION STUDY
text: Finally, we analyze the contribution of the proximity function, reward formulation, and uncertainty to our method’s performance in Figure 6. Adding uncertainty to GAIfO-s (GAIfO-s+Uncert) produced a 5.8% boost in average success rate compared to regular GAIfO-s, which is not a significant improvement. Proximity supervision, without the uncertainty penalty, resulted in a 28.1% increase in average performance over GAIfO-s with the difference-based reward R(st, st+1) = f(st+1)− f(st) (Prox+Diff) and 15.9% with the absolute rewardR(st) = f(st) (Prox+Abs). This higher performance means modeling proximity is more important than using the uncertainty penalty for our method.
We also found that the uncertainty penalty and proximity function have a synergistic interaction. Combining both the proximity and uncertainty gives a 43.3% increase with the difference-based reward (Prox+Diff+Uncert) and 33.0% increase with the absolute reward (Prox+Abs+Uncert). We can observe that the difference-based reward consistently outperforms the absolute reward except on ANT REACH, where the bias of the absolute reward Kostrikov et al. (2019) helps the agent survive
longer and reach the goal. Firstly, this shows the uncertainty penalty is more important for the proximity function as it models fine-grain temporal information where inaccuracies can be exploited, as opposed to the binary classification given by other adversarial imitation learning discriminators. Secondly, both with and without the uncertainty penalty, the difference-based proximity reward performs better than the absolute proximity reward. In conclusion, all three components of proximity, uncertainty, and difference-based reward are crucial for our method.
In Figure 6, we also evaluate the advantage of the additional sparse proximity reward given at the final time step. Compared to our method without this final reward, it results in a minor 0.9% average performance improvement, meaning this component is not critical to our method.

heading: 5 CONCLUSION
text: In this work, we propose a learning from observation (LfO) method inspired by how humans acquire skills by watching others performing tasks. Specifically, we propose to learn a goal proximity function from demonstrations which provides an estimate of temporal distance to the goal. Then, we utilize this learned proximity to encourage the policy to progressively move to states with higher proximity and eventually reach the goal. The experimental results on navigation, robotic manipulation, and locomotion tasks demonstrate that our goal proximity function improves the generalization capability to unseen states, which results in better learning efficiency and superior performance of our method compared to the state-of-the-art LfO approaches. Moreover, our method achieves comparable performance with LfD approaches.

heading: A OVERVIEW
text: As supplementary material for enhancing the main paper, we provide the following information:
• Further analysis: We first provide analyses of how our model and baselines generalize to unseen states. We also include qualitative results demonstrating what the proximity function learns. Finally, we show ablation experiments on additional environments.
• Implementation details: We describe the environments and provide training hyperparameters and architectures for our models.
• Code: Complete code of our model, environments, and experiments can be found in the code directory. The code/README.md file documents the installation process, how to download the expert datasets, and commands for running experiments.
• Video: We provide the results.mp4 file to present qualitative results of our method and baseline methods.

heading: B COMPARISON WITH GAIL
text: Our method shares a similar adversarial training process with GAIL. The following steps describe how to achieve our method starting from GAIL. Firstly, like the discriminator in GAIfO-s, our proximity function takes only states as input. Next, rather than training the discriminator to classify expert from agent, we train the proximity function to regress to the proximity labels which are 0 for agent and the time discounted value between 0 and 1 for expert. Our reward formulation also differs from GAIL approaches which give a log probability reward based on the discriminator output. We instead incorporate a proximity estimation uncertainty penalty, a difference-based reward, and a sparse reward given for the proximity of the final state, as shown in Equation 2.

heading: C ANALYSIS ON GENERALIZATION OF OUR METHOD AND BASELINES
text: By learning to predict the goal proximity, the proximity function not only learns to discriminate expert and agent states but also models task progress, which provides more information about the task. With additional supervision on learning goal proximity, we expect the proximity function to provide a more informative learning signal to the policy and generalize better to unseen states than baselines which overfit the reward function to expert demonstrations. To analyze how well our method and the baselines can generalize to unseen states, we vary the difference between the states encountered in expert demonstrations and agent learning.
One way, we vary the difference between expert demonstrations and agent learning is by restricting the expert demonstrations to only cover parts of the state space. For NAVIGATION, FETCH PICK and FETCH PUSH we show results for demonstrations that cover 100% 75%, 50% and 25% of the state
space. For the discrete state space in NAVIGATION we restricted expert demonstrations to the fraction of possible agent start and goal configurations. For the two continuous state FETCH tasks, we break the 2D continuous sampling region a 4 by 4 grid and hold out one cell per row for 75% coverage and three cells per row for 25% coverage to sample the block and target starting positions from.
Likewise, we also measured generalization by varying the difference between expert demonstrations and agent learning by increasing the initial state noise during agent learning. On FETCH PICK, FETCH PUSH and ANT REACH we show results for four different noise settings for the two FETCH tasks, the 2D sampling region is scaled by the noise factor. For ANT REACH, uniform noise, scaled by the noise factor, is added to the initial joint angles, whereas the demonstrations have no noise. If our method allows for greater generalization from the expert demonstrations, our method should perform well even under states different than those in the expert demonstrations.
The results of our method and baselines across varying degrees of generalization are shown in Figure 8 and Figure 9. Note that the results in the main paper are for 50% coverage in FETCH PICK, 2x noise in FETCH PUSH, and 0.05 noise in ANT REACH. Across both harder and easier generalization, our method demonstrates more consistent performance. While GAIfO-s performs well on high coverage, which requires little generalization in agent learning, its performance deteriorates as the expert demonstration coverage decreases.

heading: D QUALITATIVE RESULTS
text: In this section, we aim to qualitatively verify if the learned goal proximity function gives a meaningful measure progress towards the goal for the agent. It is important for agent learning, that this proximity function gives higher values for states which are temporally closer to the goal. To verify these intuitions, we visualize the proximity values predicted by the proximity function in a successful episode from agent learning in Figure 10.
From the qualitative results of FETCH PICK and FETCH PUSH in Figure 10, we can observe that as the agent moves the block closer to the goal, the predicted proximity increases. This provides an example of the proximity function generalizing to agent experience and providing a meaningful reward signal for agent learning. We notice that while the predictions increase as the agent nears the goal, the proximity prediction values are often low (<0.1) as in Figure 10a. We hypothesize this is due to the adversarial training which labels agent experience with 0 proximity and lowers the average proximity predicted across states. For videos of more qualitative evaluations for our method and baselines, refer to results.mp4, also included in the submission.

heading: E FURTHER ABLATIONS
text: We include additional ablations to further highlight the advantages of our main proposed method over its variants. We evaluate against the same ablations proposed in the main paper, but across all environments. We present all these experiments in Figure 11.
We also attempted to compare to an ablation which learns the proximity function through a ranking based loss from Brown et al. (2019). However, we empirically found it to be ineffective and therefore did not include it. This ranking based loss uses the criterion that for two states from an expert trajectory st1 , st2 , the proximities should obey f(st1) < f(st2) if t1 < t2. We therefore
train the proximity function with the cross entropy loss −∑ti<tj log exp fφ(stj )exp fφ(sti )+exp fφ(stj ) . We incorporate agent experience by adding an additional loss which ranks expert states above agent states for randomly sampled pairs of expert and agent states (se, sa), through the cross entropy loss −∑sa∼De,se∼πθ log exp fφ(se)exp fφ(sa)+exp fφ(se) . Unlike the discounting factor in the discounting based proximity function, the ranking based training requires no hyperparameters. However, the lack of supervision on ground truth proximity scores could result in less meaningful predicted proximities and a worse learning signal for the agent, which could explain the observed poor performance.
As seen from the additional ablation analysis in Figure 11, our method has the best performance in the majority of environments. In each task, incorporating uncertainty and online updates is crucial. Our main method using the linear proximity function always performs slightly better than the exponential proximity function. Using offline updates is helpful in all environments except NAVIGATION.

heading: F FURTHER BASELINES
text: In Figure 12, we compare to SQIL (Reddy et al., 2020), an imitation learning approach which demonstrates higher sample efficiency with off-policy RL. SQIL modifies the replay buffer with the expert demonstrations which are assigned reward +1 while all agent experience is assigned reward 0. We use Soft Actor-Critic (Haarnoja et al., 2018) using the same hyperparameters as in Reddy et al. (2020).
Our method consistently outperforms SQIL in the Fetch tasks, despite SQIL using actions whereas our method does not. This can be because the Fetch tasks require precise actions (i.e., it is difficult to reverse moving the block in the wrong direction) which results in covariate shift that negatively impacts BC and SQIL (a form of regularized BC). On the other hand, SQIL learns the Ant Reach task well and performs similar to the behavioral cloning (BC) baseline since Ant Reach is not sensitive to small errors (i.e., the agent can recover from bad actions and change heading). We observed unstable SQIL results in Figure 12c, which can also be observed in Figure 2,3 of (Reddy et al., 2020), further training or hyperparameter tuning could stabilize training.
G IMPLEMENTATION DETAILS
G.1 ENVIRONMENT DETAILS
The implementations of NAVIGATION, FETCH (which includes FETCH PICK and FETCH PUSH), and ANT REACH environments are based on Chevalier-Boisvert et al. (2018), Plappert et al. (2018), and Ghosh et al. (2018), respectively. The actions in the FETCH experiments use end effector position control and continuous control for the gripper. A 15-dimensional state in FETCH tasks consists of
the relative position of the goal from the object, relative position of the end effector to the object, and robot state. We found that not including the velocity information was beneficial for all learning from observation approaches. Meanwhile, in NAVIGATION the state consists of a one-hot vector for each grid cell encoding wall, empty space, agent or goal. Finally, in ANT REACH the state consists of velocity, force and the relative goal position with the action space consisting of joint control. Each environment also randomly initializes the starting state and goal. The details of observation spaces, action spaces, and episode lengths are described in Table 1. All units in this section are in meters unless otherwise specified.
For NAVIGATION we collect 250 expert demonstrations and for all other environments we collect 1000 expert demonstrations. In NAVIGATION we use BFS search to collect expert demonstrations. In FETCH PICK we generate demonstrations by hard coding the Sawyer Robot to first reach above the object, then reach down and grasp, and finally move to the target position. Similarly, in FETCH PUSH, we reach behind the object and then execute a planar push forward towards to the goal. For ANT REACH, we collect demonstrations using an expert policy trained using PPO (Schulman et al., 2017) based on the reward function R(s, a) = 1− 0.2 · ||pant − pgoal||2 − 0.005 · ||a||22, where pant and pgoal are (x, y)-positions of the ant and goal, respectively, and a is an action. Please refer to the code for more details.
To evaluate the generalization capability of our method and baselines, we constrain the coverage of expert demonstrations or add additional starting state noise during agent learning as discussed in Section C.
G.2 NETWORK ARCHITECTURES
Actor and critic networks: We use the same architecture for actor and critic networks except for the output layer where the actor network outputs an action distribution while the critic network outputs a critic value. For NAVIGATION, the actor and critic network consists of CONV (3, 2, 16)−ReLU − MaxPool(2, 2)−CONV (3, 2, 32)−ReLU −CONV (3, 2, 64) followed by two fully-connected layers with hidden layer size 64, where CONV (k, s, c) represents a c-channel convolutional layer with kernel size k and stride s. For other tasks, we model the actor and critic networks as a two separate 3-layer MLP with hidden layer size 256. For the continuous control tasks, the layer of the actor MLP is two-headed to output the mean and standard deviation of the action distribution.
Goal proximity function and discriminator: The goal proximity function and discriminator use a CNN encoder followed by a hidden layer of size 64 for NAVIGATION and a 3-layer MLP with a hidden layer size of size 64 for other tasks.
G.3 TRAINING DETAILS
For our method and all baselines except BC (Pomerleau, 1989) and BCO (Torabi et al., 2018a), we train policies using PPO. The hyperparameters for policy training are shown in Table 2, while the hyperparameters for the proximity and discriminator function are shown in Table 3. For our method, in some tasks, we also found it slightly helpful to scale the reward by a constant factor and these values are also included in Table 2.
In BC, the demonstrations were split into 80% training data and 20% validation data. The policy was trained on the training data until the validation loss stopped decreasing. The policy is then evaluated for 1,000 episodes to get an average success rate. In GAIfO-s and GAIL the AIRL reward from Fu et al. (2018) is used.

Review summaries:
The authors propose a new method for imitation learning from observation that attempts to estimate and leverage a notion of goal proximity in order to help the learning process. The authors provide a framework for computing this estimate, and a technique for using that estimate -- along with a measure of uncertainty -- to perform imitation learning from observation. Experimental results for several domains are presented in which the proposed technique achieves better performance than the comparison methods.