Goal-Driven Imitation Learning from Observation by Inferring Goal Proximity
Authors: Andrew Szot
Youngwoon Lee
Shao-Hua Sun
Joseph J Lim
Venue:  (2021)
Keywords: Imitation Learning
Learning from Observation

Abstract: Humans can effectively learn to estimate how close they are to completing a desired task simply by watching others fulfill the task. To solve the task, they can then take actions towards states with higher estimated proximity to the goal. From this intuition, we propose a simple yet effective method for imitation learning that learns a goal proximity function from expert demonstrations and online agent experience, and then uses the learned proximity to provide a dense reward signal for training a policy to solve the task. By predicting task progress as the temporal distance to the goal, the goal proximity function improves generalization to unseen states over methods that aim to directly imitate expert behaviors. We demonstrate that our proposed method efficiently learns a set of goal-driven tasks from state-only demonstrations in navigation, robotic arm manipulation, and locomotion tasks.

Under review as a conference paper at ICLR 2021
GOAL-DRIVEN IMITATION LEARNING FROM OBSERVA- TION BY INFERRING GOAL PROXIMITY
Anonymous authors Paper under double-blind review
ABSTRACT
Humans effectively learn estimate how close they are completing desired task simply by watching others fulfill the task. To solve the task, they can then take actions towards states with higher estimated proximity to the goal. From this intuition, we propose a simple yet effective method for imitation learning that learns goal proximity function from expert demonstrations and online agent experience, and then uses the learned proximity to provide a dense reward signal for training a policy to solve the task. By predicting task progress as the temporal distance to the goal, the goal proximity function improves generalization to unseen states over methods that aim to directly imitate expert behaviors. We demonstrate that our proposed method efficiently learns a set of goal-driven tasks from state-only demonstrations in navigation, robotic arm manipulation, and locomotion tasks.
INTRODUCTION
Humans capable of effectively leveraging demonstrations from experts solve variety of tasks. Specifically, by watching others performing a task, we can learn to infer how close we are to completing the task, and then take actions towards states closer the goal of the task. For example, after watching few tutorial videos for chair assembly, we learn to infer how close an intermediate configuration of chair is to completion. With the guidance of such a task progress estimate, we can efficiently learn to assemble the chair to progressively get closer to and eventually reach, the fully assembled chair.
Can machines likewise first learn an estimate of progress towards goal from demonstrations and then use this estimate as guidance to move closer to and eventually reach the goal? Typical learning from demonstration (Lff) approaches (Pomerleau 1989: Pathak et al. 2018 Finn et al. (016) greedily imitate the expert policy and therefore suffer from accumulated errors causing a drift away from states seen in the demonstrations, On the other hand, adversarial imitation learning approaches (Ho & Ermon 2016 Fu et al. 2018) encourage the agent to imitate expert trajectories with a learned reward that distinguishes agent and expert behaviors. However, such adversarially learned reward functions often overfit to the expert demonstrations and do not generalize to states not covered in the demonstrations (Zolna al. 2019). leading to unsuccessful policy learning.
Inspired by how humans leverage demonstrations to measure progress and complete tasks, we devise an imitation learning from observation (LfO) method which learns a task progress estimator and uses the task progress estimate as a dense reward signal for training policy as illustrated in Figure I To measure the progress of goal-driven task, we define goal proximity as an estimate of temporal distance to the goal (i.e., the number of actions required to reach the goal). In contrast to prior adversarial imitation learning algorithms, by having additional supervision of task progress and learning to predict it, the goal proximity function can acquire more structured task-relevant information, and hence generalize better to unseen states and provide better reward signals.
However, the goal proximity function can still output inaccurate predictions states not in demon- strations, which results in unstable policy training. To improve the accuracy of the goal proximity function, we continually update the proximity function with trajectories both from expert and agent. In addition, we penalize trajectories with the uncertainty of the goal proximity prediction, which prevents the policy from exploiting high proximity estimates with high uncertainty. As result, by leveraging the agent experience and predicting the proximity function uncertainty, our method can achieve more efficient and stable policy learning.

---

Under review as a conference paper at ICLR 2021
Expert Demonstrations
Learning Proximity Function
Agent Experience under Policy the
Demo:
fd
Praximity
Rollout
1.0 (Cloml)
0.5
08
1.0 (Casn)
Alternte Training
Rollout 2
:
Learing Pollcy
0.8
0.2
0.6
1.0 (Gow)
Demoo N
Obsevvations
R
0.8
0.8
(Doa)
Prosimity to
Proxdmity Reward = fo(Ces)) fa(e))
04
02(F3))
Figure 1: In goal-driven tasks, states toward the goal as the expert proceeds and fulfills a task. Inspired by this intuition, we propose to learn proximity function fo from expert demonstrations and agent experience, which provides an estimate of temporal distance to the goal of a task. Then, using this learned proximity function, we train policy ve progressively move states with higher proximity and eventually reach the goal to solve the task. We alternate these two learning phases to improve both the proximity function and the policy, leading to not only better learning efficiency but also superior performance.
expert trajectory have gradually increasing proximity
The main contributions of this paper include (1) an algorithm for imitation from observation that uses estimated goal proximity to inform an agent of the task progress; (2) modeling uncertainty of goal proximity estimation to prevent exploiting uncertain predictions; and (3) joint training algorithm of the goal proximity function and policy. We show that the policy learned with our proposed goal proximity function is more effective and generalizes better than the state-of-the-ard LfO algorithms on various domains, such as navigation, robot manipulation, and locomotion. Moreover, our method demonstrates comparable results with GAIL (Ho & Ermon 2016), which learns from expert actions.
2 RELATED WORK
Imitation learning Schaal 1997) aims leverage expert demonstrations to acquire skills. While behavioral cloning (Pomerleau 1989) is simple but effective with a large number of demonstrations,
it suffers from compounding errors caused by the distributional drift (Ross et al. 2011). On the other hand, inverse reinforcement learning (Ng & Russell 20000 Abbeel & Ng. 2004 Ziebart al.) 2008) estimates the underlying reward from demonstrations and learns a policy through reinforcement learning with this reward, which can better handle the compounding errors. Specifically, generative adversarial imitation learning (GAIL) (Ho Ermon 2016) and its variants (Fu et al.] 2018 Kostrikov et al., 2020 shows improved demonstration efficiency by training discriminator distinguish expert and agent transitions and using the discriminator output as a reward for policy training.
While most imitation learning algorithms require expert actions, imitation learning from observation (LfO) approaches learn from state-only demonstrations. This enables the LfO methods to learn from diverse sources of demonstrations, such human videos, demonstrations with different controllers, and other robots. To imitate demonstrations without expert actions. inverse dynamics models (Nickum al. 2015 Torabi al. 2018a; Pathak al., 2018 or learned reward functions (Edwards al. 2016 Sermanet et al.) 2017 2018 Liu et al. 2018 Lee et al.) 2019a) can be used to train the policy. However, these methods require large amounts of data to train inverse dynamics models or representations. On the other hand, state-only adversarial imitation learning (Torabi et al. [018b] Yang et al.) 2019) can imitate an expert with few demonstrations, similar to GAIL. In addition to discriminating expert and agent trajectories, our method proposes to also estimate the proximity to the goal, which can provide more informed reward signals and generalize better.
Closely related related or proximity estimator from successful trajectories and use them as an auxiliary reward (Mataric 1994 Edwards & Isbell 2019 Lee al. 2019b). While these value function and proximity estimator are simlar to our proposed goal proximity function, these works require environment reward signals, and do not incorporate adversarial online training and uncertainty estimates.
works
approach are reinforcement learning algorithms that learn value function
Moreover, demonstrating the value of learning a proximity estimate for imitation learning, Angeloy all 2020) utilizes the learned proximity to choose proper sub-policy but does not train a policy
2

---

Under review as a conference paper at ICLR 2021
from the learned proximity. Similar to our method, Burke et al. (2020) proposes to learn reward function using a ranking model and use it for policy optimization, demonstrating the advantage of using goal proximity as reward for training policy. However, they learn the proximity function from demonstrations alone and solely provide proximity as a reward. This hinders agent learning when the proximity function fails to generalize agent experience, allowing the agent to exploit inaccurate proximity predictions for reward. By incorporating the online update, uncertainty estimates, and difference-based proximity reward, our method can robustly imitate state-only demonstrations to solve goal goal-driven tasks without access to the true environment reward.
3 METHOD
In this paper, we address the problem of learning from observations for goal-driven tasks. Adversarial imitation learning methods (Torabi et al. 2018b Yang al. 2019) suggest learning a reward function that penalizes an agent state transition off the expert trajectories. However, these learned reward functions often overfit expert demonstrations and do not generalize states which are not covered in the demonstrations, leading to unsuccessful policy learning.
To acquire more structured and generalizable reward function from demonstrations, we propose to learn a goal proximity function that estimates proximity to the goal distribution terms of temporal distance (i.e., number of actions required to reach the goal). Then, policy learns to reach states with higher proximity (i.e., that are closer to the goal) predicted by the goal proximity function. Moreover, during policy training, we propose to measure the uncertainty of the goal proximity function which prevents the policy from exploiting over vver-optmmitti proximity predictions and yielding undesired behaviors, In Section 33.2] we describe the goal proximity function in detail. Then, in Section 3.3 we elaborate how the policy is jointly trained with the goal proximity function.
3.1 PRELIMINARIES
We formulate our learning problem as a Markov decision process (Sutton 1984) defined through a tuple (S, A. R, P. Po: 7) for the state space S, action space A, reward function R(s, a), transition distribution P(s'|s s, a), initial state distribution po, and discounting factor 7. We define a policy (als) that maps from states to actions and correspondingly moves an agent to new state according to the transition probabilities. The policy is trained to maximize the expected sum of discounted rewards, B)s,a)= E T3 > (R(ss,, a.). where T; represents the variable length of episode i.
In imitation learning, the learner receives fixed set of expert demonstrations, D {-f,-- - N In this paper, we specifically consider the learning from observation (LfO) setup where each demon- stration 7 is sequence of states. Moreover, assume that all expert demonstrations are successful; therefore, the final state of an expert trajectory reaches the task goal.
3.2 LEARNING GOAL PROXIMITY FUNCTION
In goal-driven tasks, an estimate of how close an agent to the goal can be utilized as a direct learning signal. Therefore, instead of learning to discriminate agent and expert trajectories (Ho & Ermon 2016 Torabi et al. 2018b), we propose goal proximity function, f + IR that learns how close states are to the goal distribution. Specifically, we define goal proximity as a proximity that discounted based on its temporal distance to the goal (i.e.. inversely proportional to the number of actions required to reach the goal). Note that the goal proximity function measures the temporal distance, not the spatial distance, between the current and goal states. Therefore, a single proximity value can entail all information about the task, goal, and any roadblocks.
In this paper, we define goal proximity of a state s, as the linearly discounted proximity f(s.) = 1 - 5- (T; -t)) t), where ö E (0. 1) is discounting factor and T, is the episode horizon. In this paper, we set 8 to 1/H to evenly distribute the proximity between and 1, where H is the maximum task horizon. Note that use the maximum episode length H. instead of the variable episode length T, to define fixed 8 for the temporal discounting to be consistent between episodes. We use mean squared error the objective for training the goal proximity function Sa parameterized by de
L. BrgnDD aidt (f((s)) (1 - 5 (T.-
- =)))
(1)
3

---

Under review as a conference paper at ICLR 2021
Algorithm 1 Imitation learning with goal proximity function
Require: Expert demonstrations D
{ T
1: Initialize weights of goal proximity function So and policy To
2: for i 0, 1,,,,. ,.... M do
3: Sample expert demonstration re ~ De
Offline proximity function training
4: Update goal proximity function fa with r" to minimize Equation
5: end for
6: for i i= 0, 1,..... I.
7: Rollout trajectories T; (80. ................................ 8T. with no
D Policy training
8:
Compute proximity reward Rø(st, 8141) for (st, 8641) ~ 75 using Equation
9: Update me using any RL algorithm
10: Update fa with T; and 7 ~ D to minimize Equation
11: end for
There are alternative ways to represent and learn goal proximity, such exponentially discounted proximity and ranking- proximity (Brown et al., 2019). But, our experiments, linearly discounted proximity consistently performed better than alternatives; therefore, the linearly discounted proximity is used throughout this paper (see Figure 5b and Figure [I].
By learning to predict the goal proximity, the goal proximity function not only learns to discriminate agent and expert trajectories (i.e., predict 0 proximity for an agent trajectory and positive proximity for an expert trajectory with Equation 4). but also acquires the task information about temporal progress entailed in the trajectories. From this additional supervision, the proximity function provides more informative learning signals the policy and generalizes better to unseen states as empirically shown in Section 4
3.3 TRAINING POLICY WITH PROXIMITY REWARD
In goal-driven task, policy Ti aims to get close to and eventually reach the goal. We can formalize this objective as maximizing the goal proximity at the final state fo(S),), which can be used as sparse proximity reward. In addition, to encourage the agent to make consistent progress towards the goal, devise dense proximity reward based the increase in proximity, fo($(+1) (s()). at every timestep. By combining the sparse and dense proximity rewards, our total proximity reward can be defined
Ro St+1) = [o((1+1) - fo((t) t t 7 1
(2)
(2. 2- - (a())
l 1= - 1
Given the proximity reward, the policy is trained to maximize the expected discounted return:
T--1
Eiso.....r, JosTa x fT((T))
Σ ((ddle))) fo(s)))
(3)
However, a policy trained with the proximity reward can sometimes perform undesired behaviors by exploiting ovtr-optiiistic proximity predictions on states not seen the expert demonstrations. This becomes critical when the expert demonstrations are limited and cannot cover the state space sufficiently. To avoid inaccurate predictions leading an agent to undesired states, we propose to (1) fine-tune the goal proximity function with online agent experience to reduce optimistic proximity evaluations: and (2) penalize agent trajectories with high uncertainty in goal proximity predictions.
First, we set the target proximity of states in agent trajectories to O, similar to adversarial imi- tation learning methods (Ho & Ermon 2016), and train the proximity function with both expert demonstrations and agent experience by minimizing the following loss:
C. EE [f,))) (1 (1-6 8- (T: t)>} E (s)))
(4)
Although successful agent experience is also used as negative examples for training the proximity function, in practice, this is not problematic since the proximity function ideally converges to the average of expert and agent labels (e.g.. 1/2 - (T; t)/2 for ours and 0.5 for GAIL). Early stopping and learning rate decay be used to further ease this problem (Zolna et al. 2019). Also,
4

---

Under review as a conference paper at ICLR 2021
(a) NAVIGATION
(b) FETCH PICK
(c) FRTCH PUSH
(d) ANT REACH
Figure 2: Four goal-driven tasks are used to evaluate our proposed method and the baselines. (a) The agent (yellow) must navigate across rooms to reach the goal (green). (b, c) The robotic arm is required to pick up or push the yellow block towards the goal (red). (d) A quadruped ant agent must walk towards the green flag.
the online training of the goal proximity function can lower the goal proximity estimate a local optimum, which helps the policy escape from such local optima.
To alleviate the effect of inaccurate proximity estimation in policy training, we discourage the policy from visiting states with uncertain proximity estimates. Specifically, we model the uncertainty (s+)) as the disagreement in terms of standard deviation of an ensemble of proximity functions (Osband al. 2016 Lakshminarayanan al., 2017). Then we use the estimated uncertainty to penalize exploration of states with uncertain proximity estimates. The proximity estimate f(s)) is the average prediction of the ensemble. With the uncertainty penalty, the modified reward can be written as:
[o((E-1) fo(*)) U+++1) 2 fo(ST.) fs:):) X UST))
T
Ro 8141)
T
where À is a tunable hyperparameter to balance the proximity reward and uncertainty penalty, A larger results in more conservative exploration outside the states covered by the expert demonstrations
In summary, we propose to learn goal proximity function that estimates how close the current state is to the goal distribution, and train a policy maximize the goal proximity while avoiding states with inaccurate proximity estimates using the uncertainty measure. We jointly train the proximity function and policy described in Figure and Algorithm
EXPERIMENTS
In our experiments, we aim to answer the following questions: (1) How does our method's efficiency and final performance compare against prior work in imitation from observation and imitation learning with expert actions? (2) Does
method lead to policies that generalize better to states unseen in the demonstrations? (3) What factors contribute to the performance of our method? To answer these questions consider diverse goal-driven tasks: navigation, robot manipulation, and locomotion.
To demonstrate the improved generalization capabilities of policies trained with the proximity reward, we benchmark our method under two different setups: expert demonstrations are collected from (1) only a fraction of the possible initial states (e.g., 25%, 50%, 75% coverage) and (2) initial states with smaller amounts of noise. (1) measures the ability of an agent to interpolate between states covered by the demonstrations while (2) evaluates extrapolating beyond the demonstrations to added noise during agent learning. In both setups, our method shows superior generalization capability and thus, achieves higher final rewards than LfO baselines. Moreover, our method achieves comparable results with LfD methods that use expert actions.
These generalization experiments serve to mimic the reality that expert demonstrations may be collected a different setting from agent learning. For instance, due to the cost demonstration collection, the demonstrations may poorly cover the state space. An agent would then have to learn in an area of the state space not covered by the demonstrations. We measure this in the experimental setup (1), where the demonstrations cover fraction of the possible learner starting and goal states. Likewise, demonstrations may collected in controlled circumstances with little environment noise. Then, agent learning in an actual environment would encounter noise than presented in
5

---

Under review as a conference paper at ICLR 2021
the demonstrations. We quantify this in the experimental setup (2), where less noise is applied to demonstration starting states.
4.1 BASELINES
We compare our method to the late-of-the-arr prior works in both imitation learning from observations and standard imitation learning with actions, which are listed below:
BCO (Torabi et al. 201 learns an inverse model from environment interaction to provide
action labels in demonstrations for behavioral cloning.
GAIfO (Torabi et al. 2018b) a variant of GAIL (Ho Ermon 2016) which trains a discrimi-
nator to discriminate state transitions (s, s') instead of state-action pairs (s, a).
GAIfO-s, as compared to in Yang et al. (2019), learns a discriminator based off only a single
state, not state transition with GAIFO.
1989) fits a policy to the demonstration actions with supervised learning. This method requires expert action labels while method does not.
BC (Pomerleau)
GAIL (Ho & Ermon 2016) uses adversarial imitation learning with discriminator trained on state-action pairs (5,a). This method also uses actions whereas ours does not.
Also, we study several variations of our method to evaluate the importance of different design choices:
Ours (No Uncert): Removes the uncertainty penalty from the reward function.
Ours (No Online): Learns the proximity function offline from the demonstrations and does not refine it using agent experience during policy learning. This approach may fail as the proximity function will not learn outside of the demonstrations and thus provide poor reward signal.
Ours (No Offiine): Does not pre-train the proximity function. This should less efficient than our method, which pre-trains the proximity function using the demonstrations.
Ours (Exp): Uses the exponentially discounted goal proximity f(tt) = 5(
4.2 EXPERIMENTAL SETUP
By default, our primary method uses the linearly discounted version of the proximity function as this empirically lead the best results (see details in Figure [1] and the discounting factor as 5 1/H, where H is the task horizon length. For modeling uncertainty, we use an ensemble of size across all tasks. For all tasks, we pre-train the proximity function for 5 epochs on the demonstrations, During online training (i.e., policy learning), we sample a batch of 128 elements from the expert and agent experience buffers. The mean and standard deviation of outputs from the ensemble networks are used as the proximity prediction and uncertainty, respectively.
The same network architecture is used for proximity function, discriminator (for the baselines), and policy. Details of the network architecture can be found in Section G.2 Any reinforcement learning algorithm can be used for policy optimization, but we choose to use PPO (Schulman et al. 2017) and the hyperparameters of PPO are tuned appropriately for each method and task (see Table 2 Each bascline implementation verified against the results reported its original paper. We train each task with 5 different random seeds and report mean and standard deviation divided by 2.
4.3 NAVIGATION
In the first set of experiments, we examine the NAVIGATION task between four rooms shown in Figure [a] The purpose of this environment is to show the benefits our method in a simple setting where we can easily visualize and verify the learned goal proximity function. The agent start and goal positions are randomly sampled and the agent has 100 steps to navigate to the goal. We provide 250 expert demonstrations obtained using shortest path algorithm. During demonstration collection, hold out 50% of the possible agent start and goal positions determined by uniform random sampling. In contrast, during agent learning and evaluation, start and goal positions sampled from all possible positions.
As can be seen in Figure 3a while all GAIL variants fail to learn the task. Although BC and BCO could achieve the goal for about 60% and 30% cases respectively, they show limited generalization unseen configurations. This
method achieves near 100% success rate in 3M environment steps,

---

Under review as a conference paper at ICLR 2021
GAIOO-S
Or Deepkree (h)
Gral Conpente
Gram
20
2002
St02
(a) NAVIGATION 50%
(b) FETCH PICK 50%
(c) FETCH PUSH w/ noise (d) ANT REACH w/ noise
Figure 3: Goal completion rates of our method and baselines. The agent must generalize to wider state distribution than seen in the expert demonstrations. Demonstrations in (a.b) cover only 50% of states and in (c.d) are generated with less noise. Note that GAIL and BC (dashed lines) expert actions whereas all other methods, including ours, learn from observations only. Our method learns more stably, faster and achieves higher goal completion rates than baseline L.O algorithms. Moreover, our method outperforms BC and GAIL in NAVIGATION and FETCH PUSH, and achieves comparable results in all other tasks.
GANO-S
GAITO
BCO
GAIL
CON CURDARSO
(a) 100% Coverage
(b) 75% Coverage
(c) 25% Coverage
(d) Proximity Heatmap
Figure 4: Analyzing the effect of improved generalization the cause for performance increase in our method. (a) performance with no generalization required (i.c., same start and goal distribution for demonstrations and online learning). (b, c) performance with increasing difference between start and goal distributions of demonstrations and online learning. (d) visualization of the learned proximity function for fixed goal (green). The proximity function was evaluated for every state the grid; lighter cells correspond to states which higher estimated proximity to the goal.
result proves that learning with the goal proximity reward is effective and the learned goal proximity function generalizes well to unseen configurations.
To verify whether the proximity function learns meaningful information of proximity to the goal, we visualize the proximity from all agent positions in Figure Our proximity function predicts a qualitatively meaningful goal proximity: high estimated proximity near the goal and lower estimated proximity when the agent is farther away from the goal. The corners of the rooms show low goal proximity since less expert trajectories pass over those regions compared to the center of each room.
Finally, we investigate our hypothesis that the goal proximity function allows for greater general- ization, which results in better task performance with smaller demonstration coverage. To test this hypothesis, we compare the cases where extreme (25% coverage), moderate, and no generalization (100% coverage)
required. Figure 4 demonstrates that our method consistently achieves 100% success rates in 3M steps with 50%-1009 demonstration coverages and is not as affected by in- creasingly difficult generalization as baselines. In contrast, GAIL and all LfO baselines fail to learn the NAVIGATION task when expert demonstrations do not cover all configurations. This supports our hypothesis that the goal proximity function able capture the task structure and therefore, generalize better to unseen configurations.
4.4 ROBOT MANIPULATION
We further evaluate our method in two continuous control tasks: FETCH PICK and FETCH PUSH from Plappert al, (2018). In the FETCH PICK task shown in Figure 2b Fetch robotic arm must
7

---

Under review as a conference paper at ICLR 2021
grasp and move block to target position. In FETCH PUSH, the Fetch arm pushes block to target position, as shown in Figure 2c Both the initial position of the block and target are randomly initialized. For each, we provide 1k demonstrations, consisting of 33k and 28k transitions for FETCH PICK and FETCH PUSH respectively, generated using a hand-engineered policy. We create a 50% holdout set of starting states for agent learning by splitting the continuous state space into 4 by 4 grid and holding out two cells per row to sample the block and target starting positions from.
In the FETCH PICK task, our method achieves more than 80% success rate while the success rates of GAIFO and GAIfO-s are upper-bounded by 50% due to the limited coverage of expert demonstrations (see Figure 3b). Our method learns slower than GAIL but achieves comparable final performance even though GAIL learns from expert actions. The FETCH PUSH task is more challenging than FETCH PICK due to the more complicated contact dynamics for pushing interactions. In Figure 3cl the demonstrations are collected with full coverage but the policy is trained in version of the environment with 2 times larger noise applied to the starting state. All methods fail to learn diagonal pushing movements but method still learns horizontal pushing faster and achieves higher performance than all other baselines. We evaluate both FETCH tasks under two different generalization setups, different demonstration coverages (Figure 8) and different amounts of noise (Figure 1 and the results consistently show that our proximity function is able to accelerate policy learning in continuous control environments with superior generalization capability.
4.5 ANT LOCOMOTION
We used the ANT REACH environment proposed in Ghosh et al. (2018), simulated in the MuJoCo physics engine Todorov et al.) 2012). In this task, the quadruped ant is tasked to reach randomly generated goal, which is along the perimeter of half circle of radius 5m centered around the ant (see Figure[d) We provide 1k demonstrations, which contain 25k transitions in total. When demonstrations are collected, no noise is added the initial pose of the ant whereas random noise is added during policy learning, which requires the reward functions to generalize to unseen states.
In Figure 3d with 0.03 added noise, our method achieves 35% success rate while BCO, GAIfO, and GAIfO-s achieve 1%, 2%, and 7%, respectively. This result illustrates the importance of learning proximity over learning to discriminate expert and agent states for gemeralization to unseen states. The performance of GAIfO and GAIfO-s drops drastically with larger joint angle randomness as shown in Figure 9 As the ANT REACH task is not as sensitive to noise in actions compared to other tasks, BC and GAIL show superior results but method still achieves comparable performance.
We also ablate the various aspects of our method in Figure 5 First, we verify the effect of the uncertainty penalty used in the proximity reward. The learning curves with different À are plotted in Figure and demonstrate that our method works the best with X = 0.1. Both too low and too high uncertainty penalties degrade the performance. Figure shows the linearly discounted proximity function learns marginally faster than the exponentially discounted proximity function. In Figure [5c we test the importance of online and offline training of the proximity function. The result shows that the agent fails to learn the task without online updates using agent trajectories. Meanwhile, no proximity function pre-training lowers performance.
4.6 ABLATION STUDY
Finally, we analyze the contribution of the proximity function, reward formulation, and uncertainty to our method's performance in Figure Adding uncertainty to GAIfO-s (GAIfO-s+ Uncert)) produced a 5.8% boost in average success rate compared to regular GAIfO-s, which is not a significant improvement. Proximity supervision, without the uncertainty penalty, resulted in a 28.1 1% increase in average performance over GAIfO-s with the difference-based reward R(st, 8t-1) = [*++11 (*x) (Prox+Diff) and 15.9% with the absolute reward R(s;) = f(x) (Prox+Abs). This higher performance means modeling proximity is more important than using the uncertainty penalty for our method.
We also found that the uncertainty penalty and proximity function have synergistic interaction. Combining both the proximity and uncertainty gives a 43.3% increase with the difference- based reward (Prox+Diff-Uncett and 33.0% increase with the absolute reward (Prox+AbsUUncen)) can observe that the difference-basee reward consistently outperforms the absolute reward except on ANT REACH, where the bias of the absolute reward Kostrikov et al. (2019) helps the agent survive
We
8

---

Under review as a conference paper at ICLR 2021
(Exp)
(Ne
130
Goal Completion (N)
0.01
Goal Completionn (N)
Completion (N)
0.1
0.2
0.3
40
40
40
Step
Ablation on À
(b) Ablation on proximity functions (c) Ablation on proximity training
Figure 5: Ablation analysis of our method on ANT REACH. (a) Comparing different À values to show the effect of the uncertainty penalty. = corresponds to no uncertainty penalty. (b) Contrasting two alternate formulations of the proximity function. (c) Analyzing the effect of online and offline proximity function training.
Prox Diff Uncert (Ours)
Prox Abs Uncert
Prox+ Diff
Prox+Abs
Ours Final)
GAifO-s+Uncer
GAIfO-s
0314 ComprNttn
Chep
(a) Navigation 50%
(b) Pick w/ noise
(c) Push w/ noise
(d) Ant w/ noise
Figure 6: Ablation analysis of the contribution of proximity, uncertainty penalty, and reward formulation to our method's performance. "Prox" uses the goal proximity function while "GAlfO-s" does not. "+Diff" uses R(st, per-time step reward. "+Uncert" adds the uncertainty penalty to the reward. Finally, "No Final" removes the sparse proximity reward at the final time step.
f((++1) f(:)) and "+Abs" R(s;) f(s) as the
longer and reach the goal. Firstly, this shows the uncertainty penalty is more important for the proximity function as models ine-grain temporal information where inaccuracies can be exploited, opposed to the binary classification given by other adversarial imitation learning discriminators. Secondly, both with and without the uncertainty penalty, the difference-based proximity reward performs better than the absolute proximity reward. In conclusion, all three components of proximity, uncertainty, and difference-basee reward are crucial for our method.
In Figure 6 also evaluate the advantage of the additional sparse proximity reward given the final time step. Compared to our method without this final reward, results a minor 0.9% average performance improvement, meaning this component not critical our method.
CONCLUSION
In this work, we propose learning from observation (LIO) method inspired by how humans acquire skills by watching others performing tasks. Specifically, we propose to learn a goal proximity function from demonstrations which provides an estimate of temporal distance to the goal. Then, we utilize this learned proximity to encourage the policy to progressively move to states with higher proximity and eventually reach the goal. The experimental results on navigation, robotic manipulation, and locomotion tasks demonstrate that our goal proximity function improves the generalization capability to unseen states, which results better learning efficiency and superior performance our method compared to the ofte--of-the-at LfO approaches. Moreover, our method achieves comparable performance with LfD approaches.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning, 2004.
Daniel Angelov, Yordan Hristov, Michael Burke, and Subramanian Ramamoorthy. Composing diverse policies for temporally extended tasks. IEEE Robotics and Automation Letters, 5(2): 2658-2665, 2020.
Daniel S. Brown, Wonjoon Goo, Prabhat Nagarajan, and Scott Niekum. Extrapolating beyond suboptimal demonstrations via inverse reinforcement learning from observations. International Conference on Machine Learning, 2019.
Michael Burke, Katie Lu, Daniel Angelov, Artüras Straižys, Craig Innes, Kartic Subr, and Subrama- nian Ramamoorthy. Learning robotic ultrasound scanning using probabilistic temporal ranking. arXiv preprint ar arXiv:2002.012400 2020.
Maxime Chevalier- Lucas Willems, and Suman Pal. Minimalistic gridworld environment for openai gym. https::/gi com /woo/maxiiie max ietaximeee mwcb/
wwtpm/.wwmcb////cccccco.ctttccct cateeecbbyymmeeeeeeee gym mi n grid 2018
Ashley D. Edwards and Charles L. Isbell. Perceptual values from observation. arXiv preprint arXiv: 07666,, 2019.
Ashley D. Edwards, Charles L. Isbell, and Atsuo Takanishi. Perceptual reward functions. arXiv 2016.
preprint /608.
Chelsea Finn, Xin Yu Tan, Yan Duan, Trevor Darrell, Sergey Levine, and Picter Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference on Robotics and Automation, pp. 512-519, 2016.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforce- ment learning. In International Conference on Learning Representations, 2018.
Dibya Ghosh, Avi Singh, Aravind Rajeswaran, Vikash Kumar, and Sergey Levine. Divide and conquer reinforcement learning. In International Conference on Learning Representations, 2018.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et al. Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905. 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems, pp. 4565-4573, 2016.
Ilya Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tompson. Diseriminator-actorrrittt:: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference Learning Representations, 2019.
Ilya Kostrikov, Ofir Nachum, and Jonathan Tompson. Imitation learning via off-policy distribution matching. In International Conference on Learning Representations, 2020.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in Neural Information Processing Systems, pp. 6402-6413, 2017.
Youngwoon Lee, Edward S. Hu, Zhengyu Yang, and Joseph J. Lim. To follow not to follow: Selective imitation learning from observations. Conference on Robot Learning, 2019a.
Youngwoon Lee, Shao-Hur Hua Sun, Sriram Somasundaram, Edward S Hu, and Joseph J Lim. Com- posing complex skills by learning transition policies. In International Conference on Learning Representations, 2019b.
YuXuan Liu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In IEEE International Conference Robotics and Automation, pp. 11255 2018.
10

---

Under review as a conference paper at ICLR 2021
Maja Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994, Pp. 181-189. Elsevier, 1994.
Andrew Y. Ng and Stuart Russell. Algorithms for inverse reinforcement learning. International Conference on Machine Learning, volume 1, pp. 2, 2000.
Scott Niekum, Sarah Osentoski, George Konidaris, Sachin Chitta, Bhaskara Marthi, and Andrew G Barto. Learning grounded finite-state representations from unstructured demonstrations, The International Journal of Robotics Research, 34(2)131-157, 2015.
lan Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, pp. 4026-4034, 2016.
Deepak Pathak, Parsa Mahmoudich, Michael Luo, Pulkit Agrawal, Dian Chen, Fred Shentu, Evan Shelhamer, Jitendra Malik, Alexci A. Efros, and Trevor Darrell. Zero-shot visual imitation. In International Conference Learning Representations, 2018.
Matthias Plappert, Marcin Andrychowicz, Alex Ray, Bob McGrew, Bowen Baker, Glenn Powell, Jonas Schneider, Josh Tobin, Maciek Chociej, Peter Welinder, et al. Multi-goal reinforcement learn- ing: Challenging robotics environments and request for research. arXiv preprint arXiv:1802.00 044400944, 2018.
Dean A Pomerleau. Alvinn: An autonomous land vehicle in a neural network. Advances in Neural Information Processing Systems, pp. 305-313, 1989.
Siddharth Reddy, Anca D. Dragan, and Sergey Levine. SQIL: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020.
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In International Conference on Artificial Intelligence and Statistics, pp. 627-635. 2011.
Stefan Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems, pp. 1040-1046, 1997.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 177.001777.06347 2017.
Pierre Sermanet, Kelvin Xu, and Sergey Levine. Unsupervised perceptual rewards for imitation learning. Robotics: Science and Systems, 2017.
Pierre Sermanet, Corey Lynch, Yevgen Chebotar, Jasmine Hsu, Eric Jang, Stefan Schaal, Sergey Levine, and Google Brain. Time-contrastive networks: Self-supervised learning from video. In IEEE International Conference Robotics and Automation, pp. 1134 1141. IEEE, 2018.
Richard Stuart Sutton. Temporal credit assignment in reinforcement learning. PhD) thesis, University of Massachusetts Amherst, 1984.
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- based control. In JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033, 2012.
Faraz Torabi, Garrett Warnell, and Peter Stone. Behavioral cloning from observation. In International Joint Conference on Artificial Intelligence, pp. 4950-4957, 20.8a.
Faraz Torabi, Garrett Warnell, and Peter Stone. Generative adversarial imitation from observation. 2018b
arXiv preprint
Chao Yang et al. Imitation learning from observations by minimizing inverse dynamics disagreement. In Neural Information Processing Systems, 2019.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence, volume pp. 1433-1438. Chicago, IL, USA, 2008.
Konrad ZoIna et al. Task-relevant adversarial imitation learning. arXiv preprint arXiv: 2019.
11

---

Under review as a conference paper at ICLR 2021
A OVERVIEW
As supplementary material for enhancing the main paper, we provide the following information:
Further analysis: We first provide analyses of how our model and baselines generalize unseen states. We also include qualitative results demonstrating what the proximity function learns. Finally, we show ablation experiments on additional environments.
Implementation details: We describe the environments and provide training hyper- parameters and architectures for our models.
Code: Complete code our model, environments, and experiments can be found in the code directory. The code / README. md file documents the installation process, how to download the expert datasets, and commands for running experiments.
We provide the resulta ts. mp4 file to present qualitative results our method and baseline methods.
Video:
B COMPARISON WITH GAIL
Our method shares similar adversarial training process with GAIL. The following steps describe how to achieve our method starting from GAIL. Firstly, like the discriminator in GAIfO-s, our proximity function takes only states input. Next, rather than training the discriminator classify expert from agent, we train the proximity function to regress to the proximity labels which are for agent and the time discounted value between 0 and for expert. Our reward formulation also differs from GAIL approaches which give a log probability reward based on the discriminator output. We instead incorporate proximity estimation uncertainty penalty, difference-batte reward, and sparse reward given for the proximity of the final state, as shown in Equation
(With Action)
Ours (Rank)
5,033
Gasl Camreitton 280
Gun Canreleenn 2812
2.2
(a) Pick w/ noise
(b) Push w/ noise
(c) Ant w/ noise
(d) Ant
Figure 7: Examining different ways of training the proximity function. (a-c) Comparing training the
proximity function with actions input with a ranking-based objective. (d) Analyzing different choices for training the proximity function. The model learns similarly well over a range of values around 1/ H, which in ANT REACH is 0.02, but struggles for large as many proximity values will be clipped O.
C ANALYSIS ON GENERALIZATION OF OUR METHOD AND BASELINES
By learning to predict the goal proximity, the proximity function not only learns to discriminate expert and agent states but also models task progress, which provides more information about the task. With additional supervision on learning goal proximity, we expect the proximity function to provide more informative learning signal the policy and generalize better to unseen states than baselines which overfit the reward function to expert demonstrations. To analyze how well our method and the baselines generalize to unseen states, vary the difference between the states encountered in expert demonstrations and agent learning.
One way, vary the difference between expert demonstrations and agent learning is by restricting the expert demonstrations to only cover parts of the state space. For NAVIGATION, FETCH PICK and FETCH PUSH show results for demonstrations that cover 100% 75%, 50% and 25% of the state
12

---

Under review as a conference paper at ICLR 2021
space. For the discrete state space in NAVIGATION we restricted expert demonstrations to the fraction of possible agent start and goal configurations. For the two continuous state FETCH tasks, we break the 2D continuous sampling region by 4 grid and hold out one cell per for 75% coverage and three cells per row for 25 coverage to sample the block and target starting positions from.
Ours
GAIFD
GAIL
Goaa
(a) FETCH PICK 100%
(b) FETCH PICK 75%
(c) FETCH PICK 50%
(d) FETCH PICK 25%
Godl P4)
(c) FETCH PUSH 100%
(f) FETCH PUSH 75%
(g) FETCH PUSH 50%
(h) FETCH PUSH 25%
Figure 8: Analyzing generalization under different demonstration coverages. The percentage refers to the percentage of regions the expert demonstrations cover. A higher coverage percentage indicates less generalization, with 100% requiring no generalization.
Likewise, also measured generalization by varying the difference between expert demonstrations and agent learning by increasing the initial state noise during agent learning. On FETCH PICK, FETCH PUSH and ANT REACH we show results for four different noise settings for the two FETCH tasks, the 2D sampling region is scaled by the noise factor. For ANT REACH, uniform noise, scaled by the noise factor, is added to the initial joint angles, whereas the demonstrations have noise. If our method allows for greater generalization from the expert demonstrations, our method should perform well even under states different than those in the expert demonstrations.
The results of our method and baselines across varying degrees of generalization are shown in Figure 8 and Figure g Note that the results in the main paper are 50% coverage FETCH PICK, 2x noise in FETCH PUSH, and 0.05 noise in ANT REACH. Across both harder and easier generalization, our method demonstrates more consistent performance. While GAIfO-s performs well on high coverage, which requires little generalization in agent learning, its performance deteriorates as the expert demonstration coverage decreases.
D QUALITATIVE RESULTS
In this section, we aim to qualitatively verify if the learned goal proximity function gives meaningful measure progress towards the goal for the agent. is important for agent learning, that this proximity function gives higher values for states which are temporally closer to the goal. To verify these intuitions, we visualize the proximity values predicted by the proximity function in a successful episode from agent learning in Figure TO
From the qualitative results of FETCH PICK and FETCH PUSH Figure TO we can observe that as the agent moves the block closer to the goal, the predicted proximity increases. This provides an example the proximity function generalizing to agent experience and providing a meaningful reward signal for agent learning. We notice that while the predictions increase the agent nears the goal, the proximity prediction values are often low (<0..) as in Figure 10a We hypothesize this is due to the adversarial training which labels agent experience with 0 proximity and lowers the average proximity predicted across states. For videos more qualitative evaluations for our method and baselines, refer to resule al mp4, also included in the submission.
13

---

Under review as a conference paper at ICLR 2021
Or Oeepkcee (N)
Goa CompAitenn
20
200
(a) FETCH PICK noise
(b) FETCH PICK 1.25x
(c) FETCH PICK 1.75x
(d) FETCH
Cuatoretonn 260
CastCardeeen 280
380
(c) FETCH PUSH 1x noise
(f) FETCH PUsh 1.25x
(g) FETCH PUSH 1.75x
(h) FETCH PUSH 2x
(i) ANT REACH 0.00 noise
(i) ANT REACH 0.01
(k) ANT REACH 0.03
(1) ANT REACH 0.05
Figure 9: Analyzing generalization to more noisy environments. The number indicates the amount of additional noise in agent learning compared to that in the expert demonstrations, with more noise requiring harder generalization.
Proximity
Proximity
Proximity
Proximity
FETCH PICK
Proximity 0.004
Proximity 0.011
Proximity 0.021
Proximity 0.988
(b) FETCH PUSH
Figure 10: Visualizing the proximity predictions for a successful trajectory from agent learning in (a) FETCH PICK and (b) FETCH PUSH. Four informative frames are selected from the overall trajectory and the predicted proximity value displayed below.
E FURTHER ABLATIONS
We include additional ablations to further highlight the advantages of main proposed method over its variants. We evaluate against the same ablations proposed in the main paper, but across all environments. We present all these experiments in Figure
14

---

Under review as a conference paper at ICLR 2021
We also attempted to compare an ablation which learns the proximity function through a ranking based loss from Brown et al. (2019). However, we empirically found to be ineffective and therefore did not include it. This ranking based loss uses the criterion that for two states from an expert trajectory Stz the proximities should ohey [((t,) < [(8tt) if l < 12- We therefore train the proximity function with the cross entropy loss - Etset, log exp
exp56((9))
(+,,, We
incorporate agent experience by adding an additional loss which ranks expert states above agent states for randomly sampled pairs of expert and agent states (Se; 8a), through the cross entropy loss
-E (e
log
exp fo(*.)
exp (e. (e()))
Unlike the discounting factor the discounting based proximity function, the ranking based training requires hyperparameters. However, the lack of supervision ground truth proximity scores could result in less meaningful predicted proximities and a worse learning signal for the agent, which could explain the observed poor performance.
As seen from the additional ablation analysis in Figure 11 our method has the best performance the majority of environments. In each task, incorporating uncertainty and online updates is crucial. Our main method using the linear proximity function always performs slightly better than the exponential proximity function, Using offline updates is helpful in all environments except NAVIGATION.
Ours
Ours (Exp)
Ours (No Offline)
Ours (No Online)
Gaal Completion (6)
Stop
2002
2655
S252
(a) NAVIGATION 50%
(b) FETCH PICK 50%
(c) FETCH PUSH 50%
(d) ANT REACH
Figure 11: Ablation experiments across additional environments. Our method shows consistently superior performance over all ablations.
F FURTHER BASELINES
In Figure 12
we compare to SQIL Reddy et al. 2020). an imitation learning approach which demonstrates higher sample efficiency with off-policy RL. SQIL modifies the replay buffer with the expert demonstrations which are assigned reward +1 while all agent experience is assigned reward O. We use Soft Actor-Critic (Haarnoja al. 2018) using the same hyperparameters as in Reddy et al. (2020).
Our method consistently outperforms SQIL in the Fetch tasks, despite SQIL using actions whereas our method does not. This can be because the Fetch tasks require precise actions (i.c.. it is difficult to reverse moving the block in the wrong direction) which results in covariate shift that negatively impacts BC and SQIL. (a form of regularized BC). On the other hand, SQIL learns the Ant Reach task well and performs similar to the behavioral eloning (BC) baseline since Ant Reach not sensitive to small errors (i.e., the agent can recover from bad actions and change heading). We observed unstable SQIL results in Figure 12c which can also observed in Figure 2,3 of (Reddy et al. (020). further training or hyperparameter tuning could stabilize training.
G IMPLEMENTATION DETAILS
G.1 ENVIRONMENT DETAILS
The implementations of NAVIGATION, FETCH (which includes FETCH PICK and FETCH PUSH), and ANT REACH environments are based on Chevalier-BBe Schevalie-Boiiserrtt et al. 22018) Plappert et al. (2018), and Ghosh et al. (2018), respectively. The actions in the FETCH experiments use end effector position control and continuous control for the gripper. A 15-dimensional state in FETCH tasks consists of
15

---

Under review as a conference paper at ICLR 2021
Ours
BC
SQIL
Goal Completion (16)
Completion
Completion
Fetch Pick 1.75x
(h) Fetch Push 1.75x
(c) Ant Reach 0.03
1.00
1:00
Goal Completion (56)
Goal Completion
40
Coal
Step
Stop
(d) Fetch Pick 2x
(c) Fetch Push 2x
(f) Ant Reach 0.05
Figure 12: Comparing method to SQIL across three tasks in the most challenging generalization settings. SQIL results are for seeds while our method and BC are for seeds. Note that SQIL and BC use actions whereas method does not.
the relative position of the goal from the object, relative position of the end effector to the object, and robot state. We found that not including the velocity information was beneficial for all learning from observation approaches. Meanwhile, in NAVIGATION the state consists of one-hot vector for each grid cell encoding wall, empty space, agent or goal. Finally, in ANT REACH the state consists of velocity, force and the relative goal position with the action space consisting of joint control. Each environment also randomly initializes the starting state and goal. The details of observation spaces, action spaces, and episode lengths described in Table I All units this section are in meters unless otherwise specified.
For NAVIGATION collect 250 expert demonstrations and for all other environments we collect 1000 expert demonstrations. In NAVIGATION we use BFS search to collect expert demonstrations. In FETCH PICK we generate demonstrations by hard coding the Sawyer Robot to first reach above the object, then reach down and grasp, and finally move to the target position. Similarly, in FETCH PUSH, we reach behind the object and then execute planar push forward towards to the goal. For ANT REACH, collect demonstrations using expert policy trained using PPO (Schulman et al., 2017) based on the reward function R(s, a) = 0.2 Pent - Pooot ||2 0.005 ||a 12 where Pant and Pgoul (Ir posiiiion of the ant and goal, respectively, and is action. Please refer the code for more details.
To evaluate the generalization capability of our method and baselines, constrain the coverage of expert demonstrations or add additional starting state noise during agent learning as discussed in Section C
16

---

Under review as a conference paper at ICLR 2021
Table Environment details. In NAVIGATION the goal and agent are randomly initialized anywhere on the grid. The goal and object noise in FETCH describes the amount of uniform noise applied to the (ar, y) coordinates. In ANT REACH, the angle of the goal and the velocity of the agent are randomly initialized.
NAVIGATION
FETCH PICK
FETCH PUSH
ANT REACH
Observation Space
(19,19.4)
16
16
132 E [0, x]
Goal Noise Object/ / Agent Noise
(a.y) E [..02, +0.05]
(r.
0..02, 10.05
(2, E +0.02 +0.02
(x.y)
+0.02, +0.05
V E +..005
Action Space
4
3
8
Episode length
50
50
60
50
G.2 NETWORK ARCHITECTURES
Actor and critic networks: We use the same architecture for actor and critic networks except for the output layer where the actor network outputs an action distribution while the critic network outputs critic value. For NAVIGATION, the actor and critic network consists of CONV 2, 16) RRLL
MaxPool (2, 2) CONV(3. (3. 2, 32) ReLU CONV 2, 64) followed by two fully-connected layers with hidden layer size 64, where CONV 8, e) represents e-channel convolutional layer with kernel size l and stride For other tasks, model the actor and critic networks two separate 3-layer MLP with hidden layer size 256, For the continuous control tasks, the layer of the actor MLP is two-headed to output the mean and standard deviation of the action distribution.
Goal proximity function and discriminator: The goal proximity function and discriminator use CNN encoder followed by a hidden layer of size 64 for NAVIGATION and a 3-layer MLP with hidden layer size of size 64 for other tasks.
G.3 TRAINING DETAILS
For our method and all baselines except BC (Pomerleau 1989) and BCO (Torabi et al., 2018a). we train policies using PPO. The hyperparameters for policy training are shown in Table 2 while the hyperparameters for the proximity and discriminator function are shown in Table 3 For our method, in some tasks, we also found slightly helpful to scale the reward by a constant factor and these values are also included in Table 2
In BC, the demonstrations were split into 80% training data and 20% validation data. The policy was trained on the training data until the validation loss stopped decreasing. The policy is then evaluated for 1,000 episodes to get an average success rate. In GAIfO-s and GAIL the AIRL reward from Fu al. (2018) is used.
Table 2: PPO hyperparameters.
Hyperparameter
Value
Learning Rate
0.001 (NAvIGATION, FETCH PUSH), 0.0003 (ANT, FETCH PICK)
Learning Rate Decay
linear decay 32 (FETCH, ANT), (NAVIgATION)
Mini-batches
# Epochs per Update
10 (FETCH, ANT), (NAVIGATION)
Discount Factor
0,99
Rollout Size
4,096 (NAVIGATION, FETCH). 16,000 (ANT) 0.001 (ANT), 0.01 (NAVIGATION, FETCH) (NAVIGATION), (FETCH), 50 (ANT REACH) True (FETCH, ANT), False (NAVIGATION)
Entropy Coefficient
Reward Scale
State Normalization
17

---

Under review as a conference paper at ICLR 2021
Table 3: Hyperparameters for goal proximity functions and discriminators.
Hyperparameter
Value
# Networks for Ensemble
5
Uncertainty Coefficient À Discounting Factor Learning Rate (Ours)
0.01 (ANT, FETCH PUSH), 0.1 (FETCH PICK, NAVIGATION) 0.01 (NAVIGATION, ANT Reach), 0.02 (FETCH PICK, FETCH PUSH)
0.001 (NAVIGATION, FETCH), 0.0001 (ANT)
Learning Rate (GAIIO-s)
0.0001
Learning Rate (GAIfO)
0.001
Batch Size
128 (FETCH, ANT), 32 (NAVIGATION)
# Updates per Agent Update
I
Agent Experience Buffer Size
16,000 (ANT), 4096 (NAVIGATION, FETCH)
18


Reviews:
[{"review_id": "r-RN0RFl5_9", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "To accelerate and improve imitation learning for goal-driven tasks, the authors introduce a goal proximity function that is learned from the observation of expert demonstrations and online agent experience.\nThe inferred goal proximity is used as an additional reward signal. The authors showed that heuristics efficiently improve the performance of imitation learning.\n\nThe method is simple and looks effective, as shown in the experiment. \nHowever, from the theoretical viewpoint, this proposal looks a heuristic method.\nIt is better to clarify the theoretical foundation.\n\nThe relationship with GAIL is mentioned several times. However, the explicit comparison between the proposed method and GAIL is not given.  \n(For example, \"First, we set the target proximity of states in agent trajectories to 0, similar to adversarial imitation learning methods (Ho & Ermon, 2016), and train the proximity function with both expert demonstrations and agent experience by minimizing the following loss.\" )\nDescribing the comparison (as an appendix) may help readers to understand the key idea.\n\nTo my understanding, the paper focus on LfO. However, the relationship between LfO and \"goal proximity\" is not clear. The \"goal proximity\" can be used for LfD as well?\n\nIf we consider \"goal proximity function\" as a goal-related reward function, the method is regarded as the integration of an imitation learning, e.g., GAIL, and a goal-driven reinforcement learning.\nFrom this view, this work looks related to the following paper. \n\n-Kinose, Akira, and Tadahiro Taniguchi. \"Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model.\" Advanced Robotics 34.16 (2020): 1055-1067.", "extra_scores": {}, "extra_text": {}}, {"review_id": "nK7--QR2Maj", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "SUMMARY:\nThe authors propose a new method for imitation learning from observation that attempts to estimate and leverage a notion of goal proximity in order to help the learning process. The authors provide a framework for computing this estimate, and a technique for using that estimate -- along with a measure of uncertainty -- to perform imitation learning from observation. Experimental results for several domains are presented in which the proposed technique achieves better performance than the comparison methods. \n\n\nSTRENGTHS:\n\t(S1) The paper seeks to solve an interesting and relevant problem in IfO.\n\t(S2) The proposed technique -- estimating and using a notion of task completion proximity -- is (as far as I'm aware) a novel take on the IfO that seems to have the potential to advance the state of the art.\n\n\nWEAKNESSES:\n\t(W1) Fundamental experimental results are missing. The paper proposes a technique with two major components: (a) an estimated proximity function, and (b) a method to exploit uncertainty information in that estimate. However, its not clear from the results which if it is the unique combination of these components that leads to the good results, or just one of them -- especially given the results in Figure 5 which shows how critical (b) is. One way to get at that would be to apply (b) to some adversarial imitation learning techniques (eg, GAIfO) and see how they perform. Without something like this, one cannot tell if it is the proximity function, the use of uncertainty information, or the combination that truly leads to improvement. This is a fundamental question that must be addressed.\n\t(W2) As written, some of the details of the proposed method are not clear to where it would likely be difficult to reproduce. For example:\n\t\t(a) There appears to be an unstated assumption that all demonstration trajectories terminate at a known time $T$, which is not typically true. Meanwhile, it seems like such trajectories would necessitate different choices of $\\delta$, but the authors only discuss how to set $\\delta$ in general according to some parameter $H$ which \n\t\t(b) It seems as though the training objective for $f_\\phi$ articulated in Equation (1) is highly dependent on $\\delta$, but it doesn't seem as though the choice of delta is discussed.\n\n\nRECOMMENDATION STATEMENT:\nThe proposed method describes a novel approach to a good problem and seems to hold promise. However, I feel that important experimental results need to be added before the paper should be published.\n\n\nQUESTIONS FOR AUTHORS:\n\t(Q1) How would a standard imitation from observation algorithm perform if endowed with the same uncertainty information as the proposed method?\n\t(Q2) In Figure 4d, why are cells in the NW quadrant seemingly just as proximal as cells in the SW/NE? It seems as though they should be less proximal.\n\n\nMINOR COMMENTS:\n\t(MC1) The two legends present on Figure 5 are confusing while looking at (a), and its unclear where some of the curves in the legend at the top are in (b) and (c). The figure should be revised to be more clear.", "extra_scores": {}, "extra_text": {}}, {"review_id": "LmJVjPG4AVj", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Paper and Review Summary \n\nThis paper introduces a goal proximity approach to learning rewards from observations captured during demonstrations in goal-oriented tasks. This metric allocates rewards based on the temporal distance to a goal, relying on a model trained to predict this distance. The paper also proposes an adversarial learning approach to policy optimisation with this reward, which helps to improve policies in regions where demonstrations were not captured.\n\nI like the idea, but, unfortunately, this paper has missed key related work [Angelov et al.], [Burke et al.] on reward inference and policy scheduling using temporal progress metrics, which have been proposed previously. Although I do see potential novelty and additions going beyond the work in [Angelov et al.], [Burke et al.] (adversarial training, ablations the incorporation of uncertainty in the metric) - the primary contribution is severely reduced. As a result, I am inclined to recommend *rejecting* this work, unless the paper revisions can convince me that there are sufficient differences and novelty. However, this may require a substantial rewrite of the introduction and conclusions and major revisions. \n\n## Pros \n\n- The paper is well written, with detailed experiments and nice ablations\n- I like the idea of combining a goal progress metric with adversarial learning\n- The inclusion of the uncertainty metric is interesting, potentially allowing for risk-based policy optimisation\n\n### Cons\nLimited novelty: \n- A linear goal progress metric has already proposed by Angelov et al. in the paper [Composing Diverse Policies for Temporally Extended Tasks](https://arxiv.org/pdf/1907.08199.pdf). This paper trains a model to predict the normalised time to goal using observed demonstrations, and uses this to select sub-policies for long-horizon tasks, in a model-based setting. Angelov et al. do not optimise policies directly using the goal metric.\n- In follow on work by Burke et al. in the paper [Learning rewards for robotic ultrasound scanning\nusing probabilistic temporal ranking](https://arxiv.org/abs/2002.01240) this goal progress metric of Angelov et al. is extended to a probabilistic temporal ranking metric that allows for non-monotonically increasing goal progress in demonstrations to be captured. Here, the goal progress reward is used for policy optimisation (value iteration for grid world experiments) and online learning.\n\n### Questions\n\n- On page 3, the paper states \"*There are alternative ways to represent and learn goal proximity, such as exponentially discounted proximity and ranking-based proximity (Brown et al., 2019). But, in our experiments, linearly discounted proximity consistently performed better than alternatives*\" - I am curious that linear proximity models performed better as experiments in Brown et al. and Burke et al. show that non-linear temporal progress metrics are more effective. In particular, ranking-based methods allow for non-monotonically increasing progress metrics. Is this comment based only on the comparison between exponentially and linearly increasing progress, or was a temporal ranking approach considered? If not, I would recommend exploring a temporal ranking approach going forward.\n- In Equation (2), I see the progress metric is not used directly, instead a derived reward was used. Did you experiment with using the progress metric directly instead?\n- Page 7 FETCH-PUSH - Why can't the methods learn diagonal pushing policies?\n- Figure 5 - Why does the policy using offline only training fail completely?  Is this a function of the way/ amount the training data was collected or something else?", "extra_scores": {}, "extra_text": {}}, {"review_id": "IsW3Rh015v9", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------\nThe paper proposes a method for imitation learning for goal-directed tasks that uses a learned proximity function for computing rewards.\nAn ensemble of proximity functions is trained in supervised way to predict the time step (rescaled to the range $[0,1]$) for the expert's states. The ensemble is improved online based on the additional objective of assigning zero proximity to the states encountered by the agent. The agent is improved using PPO where the gain in proximity serves as reward function with an additional cost based on the uncertainty about the proximity (estimated by the standard deviation of the ensemble).\n\nStrong points\n-------------\n- Predicting time from states is an interesting idea and assigning higher reward to later states is sensible for goal directed tasks.\n- The paper is well-written and sufficiently clear  \n\nConcerns\n---------\n- Soundness\nThere is barely any theoretical justification for the approach. Furthermore, several hacks / hyperparameters have been added to achieve the reported results (uncertainty penalty, bonus reward, proximity \"discounting\", reward scaling).\n\n- Relevance\nThe approach is limited to goal-directed tasks and thus much less applicable than comparable methods.\n\n- Evaluation\nI think that the evaluations are not fair for several reasons.\n1. The competitors are not intended for learning under different initial state distributions. It is well known that behavioral cloning often performs very bad for out-of-distribution data. Adversarial methods like GAIL and GAIL-s are based on distribution matching which is in general not possible if the initial state is significantly different from the demonstrations. Especially GAIL-s can suffer here, since it would move to the initial state when starting at the goal position. Matching state transitions as GAILfO would be a bit more reasonable here. Methods that can guide the agent towards the demonstrations, such as SQIL (Reddy et al. 2019) might be even more suitable.\n\n2. The paper mentions that GAIL makes use of additional information by taking into account the actions, but completely ignores the fact that the proposed method makes use of time-labels which are much more relevant for tasks that are essentially defined by the state at the last time step. It would be easy to provide this information also to the adversarial methods by weighting the discriminator samples during training dependent on the time step (in the extrem case by only using the samples form the final time steps). Another naive baseline would be to use a non-parameteric reward function by placing radial basis functions at every expert sample (again weighted dependent on the time step). Instead, the evaluation does not seem to make any use of the strong assumption of goal-directed tasks for any of the competing methods.\n\n3. It seems that much more hyper-parameter tuning was involved for obtaining the results for the proposed method than for competing methods. Even for hyperparameters that are applicable to other methods, such as learning rates and reward scaling the competing methods share the same parameters during all experiments, whereas the proposed method has different hyperparameters depending on the environment. \n\n\nAdditional Feedback\n-------------------\nClarity could be improved at some points. Section 3.2. introduces $\\delta$ as a discounting but doesn't mention that it is typically set to $1/T$ which essentially scales the predicted time step to the range $[0,1]$. Without this information, it is hard to make sense of Eq. 1 because for a \"discount factor\" close to 1 (which is typical for the discount factor of the MDP) the proximity function would actually be negative for most time steps and by setting the proximity of agent states to 0 one would actually assign them very high goal proximity.  \n\n\nQuestions\n---------\n1. Why provide an additional reward for last time step? Is this really necessary to obtain good results?\n\n2. I don't see how the optimal proximity function for Eq.4 converges to $\\frac{1}{2} - \\delta \\frac{(T-t)}{2}$. This is not really relevant to the algorithm but still I'm wondering whether this claim is correct. Can you sketch a proof?\n\n3. Can you specify the initial distributions for the test and train experiments more precisely. For example, for the navigation task you mention that 50% of the possible initial states  and goals where used for collecting demonstrations. But, I can neither find the set of possible initial states, nor how the 50% where chosen (uniformly, or selected?).  \n\n\nAssessment\n----------\nThe proposed (heuristic) approach seems reasonable for the considered setting, however, overall the contribution is too small. Strong theoretical results or a thorough empirical evaluation (with suitable baselines) would both be fine for me. However, the current submission lacks quite severly on both sides and is therefore in my opinion not suitable for publication.\n\n\nReferences\n----------\nReddy, S., Dragan, A. D., Levine, S. SQIL: Imitation Learning via Reinforcement Learning with Sparse Rewards. ICLR. 2019", "extra_scores": {}, "extra_text": {}}, {"review_id": "AQ86I-AzmFO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Paper summary**\n\nThis paper proposes a method for imitation learning from observations based on learning a goal proximity function from expert demonstrations and using it as a dense reward for training an imitator. The authors show that this method improves generalization to unseen states in comparison to several baselines. \n\n**Pros:**\n\n- The idea is simple, well-motivated and the paper is easy to read, clear and well-written.\n\n- The experiments are well designed and the results are explained with sufficient detail. \n\n**Cons:**\n\nWhile I find the idea interesting and the experiments well-designed, I have a few concerns about the method and would need some clarifications to evaluate it further:\n\n1. Figure 5 and 9 demonstrate that a key component of the method is the adversarial training of the proximity function. Without it, the method completely fails. In comparison, not using the uncertainty part, or not training offline, makes less of a difference. This seems to indicate that the adversarial part is more important than the temporal aspect of it. If we ignore the temporal aspect and replace the target for f, $1-\\delta (T-t)$, by its upper value of 1, we obtain a loss: $E_{expert} [f -1]^2 + E_{imitator} f^2$ which is very similar to the one used in GAIL: $E_{expert} \\log(\\frac{1}{1-D}) + E_{imitator} \\log \\frac{1}{D})$ for values of $D$ and $f$ between 0 and 1 (plotting the 4 functions recommended). Given this similarity, I think the paper would benefit from one more ablation of this form: removing the temporal part and keeping the adversarial component. \n\n2. Following from point 1, I wonder why are the GAIL results so different (e.g GAIfO). One concern I have is the final reward. The method proposed in this paper makes the assumption that the last state of the demonstration is the goal. Therefore the authors use an extra reward at this time step (Eq. 2). If I understand correctly, GAIL in general doesn’t make this assumption, so I wonder if this is at least in part responsible for the differences between these methods. I think the paper would benefit from clarifying this by performing an ablation that removes this final reward.\n\n3. One maybe more superficial concern is whether the name ‘goal proximity’ is appropriate. Given that the function $f$ is not only trained to follow time, but also adversarially to be zero on imitator data, this name is confusing to me. In fact, as shown in Fig 4d, the value of $f$ doesn’t end up corresponding to the time (or number of actions) that would be required to reach the goal in the general sense. \n\n4. Regarding Fig 4d. I’m not sure why the centers of all quadrants are light. Shouldn’t only the path to the goal used by the expert be light?\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method for learning intristic reward from demonstrations. The inartistic reward is computed as time-to-reach and generalizes to unseen states.  The reviewers agree that the method is novel useful, and of interest to ICLR community. 

Although the authors' significantly improved the manuscript during the rebuttal phase with new results, and addressed many of the reviewers' comments, the overall novelty of the paper is still somewhat limited, making it unsuitable for ICLR in its current form.   

The future  version of the paper should address the comments below and go through a detailed pass for clarity.

Additional comments that did not influence the final decision:

The idea of learning temporal distance to the goal is not novel [1], although the application as an intristic reward is. The authors should connect the temporal difference to the reachability theory and solving two-point boundary problem for systems with non-linear dynamics, as a theoretical foundation of the method [1].

I am curious about the decision to use the time to reach as a reward directly, instead of delta between the states. Some empirical work provides evidence [2,3] that delta yield less side effects in behaviors. 

[1] https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8772207
[2]https://arxiv.org/abs/1803.10227
[3] https://arxiv.org/abs/2003.06906
Decision: Reject