Generalizable Imitation Learning from Observation via Inferring Goal Proximity
Authors: Youngwoon Lee
Andrew Szot
Shao-Hua Sun
Joseph J Lim
Venue: NeurIPS 2021 Poster (2021)
Keywords: Imitation Learning
Learning from Observation

Abstract: Task progress is intuitive and readily available task information that can guide an agent closer to the desired goal. Furthermore, a task progress estimator can generalize to new situations. From this intuition, we propose a simple yet effective imitation learning from observation method for a goal-directed task using a learned goal proximity function as a task progress estimator for better generalization to unseen states and goals. We obtain this goal proximity function from expert demonstrations and online agent experience, and then use the learned goal proximity as a dense reward for policy training. We demonstrate that our proposed method can robustly generalize compared to prior imitation learning methods on a set of goal-directed tasks in navigation, locomotion, and robotic manipulation, even with demonstrations that cover only a part of the states.

Generalizable Imitation Learning from Observation via Inferring Goal Proximity
Youngwoon Lee
Andrew Szot
Shao-Hua Sun
Joseph J. Lim 11
University of Southern California https://clvrai.
Georgia Institute of Technology
Abstract
Task progress is intuitive and readily available task information that can guide an agent closer to the desired goal. Furthermore, a task progress estimator can generalize to new situations. From this intuition, we propose simple yet effective imitation learning from observation method for a goal-directed task using a learned goal proximity function as task progress estimator for better generalization unseen states and goals. We obtain this goal proximity function from expert demon- strations and online agent experience, and then use the learned goal proximity as dense reward for policy training. We demonstrate that our proposed method can robustly generalize compared to prior imitation learning methods on a set goal-directed tasks in navigation, locomotion, and robotic manipulation, even with demonstrations that cover only a part of the states.
Introduction
Humans are effective at learning a task from demonstrations and applying the learned behaviors to other situations. We achieve this by extracting the underlying structure of the task when observing others fulfilling the task, instead of simply memorizing the demonstrator's 's low-level actions [4] [8]. This high-level task structure generalizes to new situations and thus helps to quickly learn the task in new situations. One intuitive and readily available instance such high-level task structure task progress, measuring how much of the task the agent completed. Inspired by this insight, propose a novel imitation learning method that utilizes task progress for better generalization to unseen states and goals.
Typical learning from demonstration (LfD) approaches [3 35 greedily imitate the expert policy and thus suffer from accumulated errors causing a drift away from states seen in the demonstrations [38 To make the imitation policy more robust to states not demonstrations, adversarial imitation learning methods [14] [1]] encourage the agent to stay near the expert trajectories using learned reward that distinguishes expert and agent behaviors. However, such learned reward functions often overfit to the expert demonstrations by learning spurious correlations between task iask-irreeevant features and expert/agent labels (52), and thus suffer from generalization to slightly different initial and goal configurations from the ones seen in the demonstrations (e.g. holdout goal regions larger perturbation in goal sampling).
To learn more generalizable and informative reward from demonstrations, propose imitation learning from observation (LfO) method, which learns a task progress estimator and uses the task progress estimate dense reward for training policy illustrated in Figure I Unlike discriminating expert and agent behaviors by predicting binary labels prior adversarial imitation
"Equal contribution. Correspondence to: lee5048usc. edu and aszot30gatech. This work was partially carried out during an internship at NAVER Al Lab. I AI Advisor NAVER Al Lab.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Experi Demonstrations
Agent Experience under Policy Ta
Learning Proximity Function
fe
Goel Proximity
Rollout
0.2
Joint Training
2
0.6
C.7
0.8
0.8
Learning Policy
0.6
1.0 (Cuud)
Otremsalions
Pullout
Picaictoo Goal
Phoomnly Iglne)
0.8
0.8
1.0 15ca))
** Goal Phoamik
Proximity Reward: fo(S1e1) - feles)
0.5
0.8
1.0./Goa)
Figure I: In goal-directed tasks, states on an expert trajectory have increasing proximity toward the goal as the expert makes progress towards fulfilling a task. Inspired by this intuition, we propose to learn a proximity function f& from expert demonstrations and agent experience, which predicts goal proximity (i.e. estimate of temporal distance to the goal). Then, using this proximity function, we train a policy Ty to progressively move to states with higher predicted goal proximity (italicized numbers) and eventually reach the goal. We alternate these two learning phases improve both the proximity function and policy, leading to not only better generalization but also superior performance.
learning methods, which is prone overfitting task-irrelevan features, the task progress estimator is required to learn more task-relevant information to precisely predict the task progress on a continuous scale. Hence, can generalize better to unseen states and provide more informative rewards.
As a measure of progress in goal-directed tasks, we define goal proximity, which is an estimate of temporal distance to the goal (i.e. the number of actions required to reach the goal) and entails all semantic information about how to reach the goal. We then train a proximity function to predict the goal proximity from expert demonstrations and agent experience. This proximity function acts dense reward to guide a reinforcement learning agent to reach states with high proximity, leading to the goal. In this paper, we focus on learning the proximity function and policy in state space shared by the expert and learner, and leave generalizing to different embodiments as future work.
However, the predicted goal proximity resulting in unstable policy learning. To improve the accuracy of the proximity function, we contin- ually update it with trajectories from both the expert and learning agent. In addition, trajectories with the uncertainty of the proximity prediction to prevent the policy from exploiting inaccurate high proximity predictions. By leveraging the agent experience and predicting proximity function uncertainty, the proposed method achieves more efficient and stable policy learning.
still be inaccurate states not in the demonstrations,
penalize
The main contribution of this paper is an LIO algorithm for goal -dirediee tasks with better generaliza- tion to new goals or states not demonstrations using goal proximity that informs an agent of the task progress. Together with difference- estimation, our method provides more informative and robust rewards. Our extensive experiments show that the policy learned with the goal proximity function generalizes better than the state-of- the-art LfO algorithms on various goal-directed tasks, including navigation, locomotion, and robotic manipulation. Moreover, our method shows comparable results with LfD methods which learn from expert actions and a goal-conditioned imitation learning method which uses sparse task reward.
reward and uncertainty penalty of goal proximity
2 Related Work
Imitation learning [39] aims to leverage expert demonstrations to acquire skills. While behavioral cloning 335 is simple but effective with a large number of demonstrations, it suffers from com- pounding errors caused by covariate shift [38). On the other hand, inverse reinforcement learning (IRL) 29 5l) estimates the underlying reward from demonstrations and trains a policy through reinforcement learning (RL) with this reward, which can better handle the compounding errors. Specifically, generative adversarial imitation learning (GAIL) 17 shows improved demonstration efficiency by training a discriminator to distinguish expert and agent transitions and using the dis- criminator output as a reward for policy training. GoalGAII. [9] further improves sample efficiency for goal-din direeeiee tasks by relabeling transitions 2 and using true environment rewards.

---

While these imitation learning algorithms require expert actions, imitation learning from observation (LfO) approaches learn from state-only demonstrations, such as videos and kinesthetic demonstrations. To imitate demonstrations without expert actions, inverse dynamics models 30 33 46
reachability
functions [33, or learned reward functions [1] 26 42 can be learned and used for policy
training, but training such models requires large amount of quality data or additional test-time
demonstrations. On the other hand, state-only adversarial imitation learning 44 can imitate from a few demonstrations,
However, in such adversarial imitation learning approaches, the discriminator tends to find spurious associations between task- irreeeaetvnn features and expert/agent labels [52]. This becomes problematic when the agent encounters unseen states and the discriminator erroneously assigns agent behaviors low scores based on these -reeeerteleantt features, providing poor reward for the agent. To overcome finding spurious associations, in addition to discriminating expert and agent trajectories, we propose to also estimate the proximity to the goal, which requires more task-relevant information and thus generalizes better to new states.
Temporal progress estimation has shown its effectiveness as an auxiliary reward for RL 10 24 27
and decision making criteria [ []. However, these methods learn the progress estimator only from the given demonstrations. This hinders policy learning when the progress estimator fails to generalize to agent experience, allowing the agent to exploit inaccurate progress predictions for higher reward. Moreover, greedily choosing an action with the highest predicted temporal progress [3 6 could lead low long-term returns. By incorporating online updates, uncertainty estimates, and a difference-basee proximity reward, our method robustly learns from demonstrations to solve gial-directed tasks without access to expert actions or the true environment reward.
3 Method
In this paper, we address the problem of LfO for goal-directed tasks with a focus on generalization to states or goals not covered in the demonstrations. Adversarial LiO methods 47 40 suggest learning a reward function that penalizes agent state transitions deviating from the expert trajectories. However, these learned reward functions often focus on task-irrelevant features states not in the demonstrations, leading to unsuccessful policy training.
[52] and do not generalize to
To learn generalizable reward, we propose leverage task progress information freely available in demonstrations, in terms of goal proximity, which estimates temporal distance to the goal (i.e. number of actions required to reach the goal). Predicting precise goal proximity rather than simply distinguishing expert and agent states, requires the model to capture task-relevant information, allowing the proximity prediction to generalize
a continuous scale,
states not in the demonstrations (Section 3.2). Then, a policy learns to reach states with higher proximity prediction, leading to the goal (Section 3.3). Moreover, we propose to use the uncertainty of the proximity prediction to prevent the policy from exploiting over-optimistic proximity predictions and yielding undesired behaviors.
3.1 Preliminaries
We formulate our problem AS,A, R, P, ()) of the state space S, action space A, reward function R(s, a, 8'), transition distri- bution P(s"|s, [s, a), initial state distribution Po. and discounting factor 7. We define policy (aass) (a]s) that maps from a state 8 to an action a and correspondingly moves an agent to a new state according to the transition probability P(s'|s,,) a). The policy trained to maximize the sum of discounted rewards, (o..a..... Jom E 7¹ 7' au, sc+1] where T; is the variable episode length.
Markov decision process
47
defined through a tuple
In imitation learning, the learner receives a set of N expert demonstrations, D .. J In this paper, we specifically consider the Lf0 setup where each demonstration rf is a sequence of states. Moreover, we assume that goal information is explicitly or implicitly included in the state s, and all demonstrations are successful; therefore, the final state of each trajectory achieves the task goal.
3.2 Learning Goal Proximity Function
To effectively leverage expert demonstrations and generalize to new states or new goals, learning generalizable reward function is essential. In goal-directed tasks, estimate of how close an
3

---

agent is to the goal can be utilized as dense and direct learning signal. Moreover, predicting the continuous goal proximity requires understanding the task structure and thus encourages finding more task-relevant features, resulting in better generalization.
Therefore, instead of learning to simply discriminate agent and expert trajectories, we propose to learn a goal proximity function, f: : + |o, 1J, which predicts goal proximity of a state which is a discounted value based on the temporal distance the goal (i.c. inversely proportional to the number of actions required to reach the goal). In this paper, we define goal proximity as the exponentially discounted proximity [(x) = -02 where ā E (0, 1) is discounting factor and T; is the episode length. Note that the goal proximity function measures the temporal distance, not the spatial distance, between the current and goal states. Therefore, a single proximity value can entail all information about the task, goal, and any roadblocks. There are alternative ways define goal proximity, such as linearly discounted proximity [24] and ranking-based proximity [5] 6]. But, in this paper, we use the exponentially discounted proximity as performs better across most tasks (see appendix, Figure g
We train a goal proximity function f parameterized by ø to minimize the following objective:
Ls = BrgwDD,eerr [fa()) - 8(T:-1)
Since the goal proximity function trained only on expert demonstrations can overfit to the data, we further train the goal proximity function with online agent experience by setting the target proximity of states in agent trajectories to 0, similar to adversarial imitation learning methods [17
L Brewwrnsaant (fo(()) 5(T:-0) E. [fo)s) (s:))
-
(2)
By learning to predict the goal proximity, Jo not only learns to discriminate agent and expert trajectories (i.e. predict proximity for an agent trajectory and positive proximity for an expert trajectory) but also acquires the task information about temporal progress entailed in the trajectories. From this freely available additional supervision, the proximity function is required to learn task- relevant features. Hence, the resulting proximity function generalizes better to unseen states and provides more informative learning signals to the policy as empirically shown in Section 4
Due to the lack of environment reward, successful agent experience is also used as negative examples for proximity function training, and thus the proximity function learns to predict low goal proximity even for successful trajectories. However, early stopping and learning rate decay can ease this problem [52] and the optimal proximity function still outputs the average of expert and agent labels, which is 6'"1-t)/2 /2 for ours and 0.5 for GAIL [4]
3.3 Training Policy with Proximity Reward
In a goal-directed task, a policy ma aims to get close to and eventually reach the goal. We can formalize this objective as maximizing the difference-basee proximity reward R2, the increase in goal proximity, at every timestep, which corresponds to making consistent progress towards the goal:
(s $141) = fe($1+1) - (st)..
Given the proximity reward Rø, the policy is trained to maximize the expected discounted return:
I,-1
E Es )oSTT
Σ
(s)
(4)
However, a policy trained with the proximity reward can sometimes acquire undesired behaviors by exploiting over-optimistic proximity predictions states not seen in the expert demonstrations. This becomes critical when the expert demonstrations are limited and cannot sufficiently cover the state space. To avoid inaccurate predictions leading agent to undesired states, propose to (1) fine-tune the proximity function with online agent experience to reduce optimistic proximity predictions; and (2) penalize agent trajectories with high uncertainty in goal proximity prediction.
To alleviate the effect of inaccurate proximity estimation policy training, we discourage the policy from visiting states with uncertain proximity estimates. Specifically, we model the uncertainty (*()) as the disagreement of an ensemble of proximity functions by computing the standard deviation of their outputs 222 31. Then, use this estimated uncertainty to penalize exploration of states with

---

(a) NAVIGATION
(b) MAZE2D
(c) ANT REACH (d) FRTCH PICK (e) FETCH PUSH (f) HAND ROTATE
Figure 2: Six goal-directed tasks used for experiments. (a) The agent must navigate across rooms to reach the goal. (b) The agent needs to navigate the maze to reach the goal. (c) The ant agent must walk towards the flag. (d, e) The robotic arm required pick up push the block towards the goal (red). (f) The dexterous robot hand needs to rotate the block in-hand to the desired rotation.
high uncertainty. The proximity estimate (a() is the average prediction the ensemble. With the uncertainty penalty, the modified proximity reward can be written
(5) where À is a tunable hyperparameter to balance the proximity reward and uncertainty penalty, A larger results in more conservative exploration outside the states covered by the expert demonstrations.
(s, $141) = fa($++1) - fs()s À (
In summary, propose to learn a goal proximity function to robustly provide reward signal on states or goals not covered by demonstrations. We train the goal proximity function to estimate how close the current state is to the goal, and train a policy maximize the goal proximity while avoiding states with uncertain proximity predictions. We jointly train the proximity function and policy as described in appendix, Algorithm
4 Experiments
In this paper, we propose generalizable LfO algorithm that leverages task progress information (i.e. goal proximity) freely acquired from demonstrations. Hence, in our experiments, we aim to answer the following questions: (1) Does our method lead to policies that generalize better to states and goals not in the demonstrations? (2) How does our method's efficiency and performance compare against prior work in LfO and LID? (3) What factors contribute to the performance of our method? To answer these questions consider diverse goal-directed tasks; navigation, locomotion, and robotic manipulation.
4.1 Experimental Setup
To demonstrate the improved generalization capabilities of policies trained with the goal proximity, benchmark our method under two different setups: expert demonstrations are collected from (1) only a fraction of the possible initial and goal states (e.g. 25%, 50% coverage) and (2) initial states with smaller amounts noise. These generalization experimental setups serve mimic the reality that expert demonstrations may be collected in a different setting from agent learning. For instance, due the cost of demonstration collection, the demonstrations may poorly cover the state space, which corresponds to the setup (1). Likewise, in the setup (2), demonstrations may be collected in controlled circumstances with little noise. Then, an agent in an actual environment would encounter more noise than presented in the demonstrations, leading to wider initial state distribution.
In experiments, use the discounting factor 5 0.95 for the goal proximity. We use ensemble of proximity functions to model uncertainty across all tasks. For policy optimization, we tuned for
PPO 440). which is widely used in LfO and L.fD methods, and its hyperparameters each method and task (see appendix, Table 2). Each baseline impiementation is verified against the results reported in its original paper. We train each task with 5 random seeds and report mean and standard deviation. See Section[[ [ for further implementation details.
4.2 Baselines
We compare our method to the state-of-the-art methods in LfO (BCO, GAIFO, GAIfO-s) well as L.F0 with reward (GoaIGAll.) and LfD (BC, GAII,, SQIL) approaches, which require additional supervisions, such as task reward and expert actions:
5

---

GAND
GABO
800
SQIL
Goa
Cook Compttloonn 190)
Goo Compledinn 090
Comstettin
30
Step
(a) NAVIGATION 25%
(b) MAZE2D 50%
(c) ANT REACH 0.05 noisc
130
Our (GAIL)
000 Comolation (S1
Grew Completton 044
60
20
16
(d) FETCH PICK 1.75x noise
(c) FETCH PUSH 1.75x noise
HAND ROTATE 0.35 noise
Figure 3: Goal completion rates of our method and baselines. The agent must generalize to wider state and goal distributions than seen the demonstrations. Demonstrations cover only a part of states (a, b) or are generated with less noise (c, d, e, f). Our method learns more stably, faster, and achieves higher goal completion rates than prior LfO methods. Moreover, our method outperforms the LfD baselines in NAVIGATION FETCH tasks, and MAZE2D, and achieves comparable results in ANT REACH. GoalGAIL performs well MAZE2D since it can easily acquire environment rewards.
learns inverse dynamics model from environment interaction provide action labels in demonstrations for behavioral cloning.
BCO [46]
GAIFO
trains a discriminator with state transitions (s, s'), instead of (s, a) as in GAIL.
GAIFO
40
learns discriminator based off single state, not a state transition as with GAIfO.
GoalGAIL uses goal reaching reward and relabeling to improve sample efficiency GAIL.
BC 33 fits policy the demonstration state-action pairs (s, a) with supervised learning.
GAIL 17 is adversarial imitation learning with a discriminator trained state-action pairs (8,a) from both expert and agent.
SQIL [37 a sample-effictn imitation learning method which adds expert transitions (s, a) with reward to the replay buffer of off-policy RL and assigns O reward to all agent experience.
4.3 Navigation
We first examine the NAVIGATION task between four rooms shown in Figure 2a to demonstrate generalization capability of our method, and visualize the learned goal proximity function. The agent observes the 19 19 x 2D map of the maze and moves in one of four directions. In this task, the agent starting and goal positions provide 250 expert demonstrations obtained using a shortest path algorithm. During demonstration collection, we hold out 0%, 25%, 50%, and 75% of the possible agent starting and goal positions uniformly at random. contrast, during agent learning and evaluation, start and goal positions are sampled from all possible positions.
randomly sampled (see example in appendix, Figure [2]. We
Figure 3al shows that our method achieves near 100% success rate in 2M environment steps even with demonstrations only covering 25% of starting and goal states, while other LfO methods fail to learn the task. Although BC, GAIL, and BCO achieve success rates of about 60%. 30%, and 35%, respectively, they show limited generalization unseen configurations. This result shows that the learned goal proximity function generalizes well to unseen configurations.
Figure 4d visualizes the proximity function trained with 50% coverage demonstrations and 250k steps of agent training. Our proximity function predicts high proximity near the goal and lower proximity when the agent is farther away from the goal. This demonstrates that proximity function can learn

---

LiO
Ours
6010
LTD
X
Compleston
Consetttnn
S002
(a) 100% coverage
(b) 75% coverage
(c) 50% coverage
(d) Proximity heatmap
Figure 4: Analyzing the effect of improved generalization the cause for performance increase in our method. (a) Performance with no generalization required. (b, c) Performance with increasing difference between start and goal distributions demonstrations and agent learning. (d) Visualization of the learned proximity function for fixed goal (green) in the 50% coverage case. The proximity function was evaluated for every state on the grid; lighter cells correspond states with higher estimated proximity to the goal.
the semantic, non-euclidean relationship between high- -ighesimensional observations and goals. Since the proximity function is conditioned on the state, similar states likely to have similar predicted proximity, and thus the proximity function learns a spatially consistent measure of proximity from temporal supervision. Moreover, the task progress is relative position within trajectory, both slow and fast demonstrations result in the same task progress. More visualizations can be found in appendix, Section E
Finally, we investigate our hypothesis that the goal proximity function allows for greater generaliza- tion, which results in better performance with smaller demonstration coverages. We compare the cases where extreme (25% coverage), moderate (50% and 75% coverage), and no generalization (100% coverage) required. Figure 3a and Figure show that method consistently achieves almost 100% success rates in 2M steps across all coverages and is not as affected by the increasingly difficult generalization settings as baselines. In contrast, all LfO baselines struggle to learn the task when the demonstrations do not cover all configurations. LiD methods also shows limited generalization in 25% coverage since the discriminator the actions and labels, which hurts generalization to new actions. This supports our hypothesis that the goal proximity function is able to capture the task structure and therefore, generalize better to unseen configurations.
easily learn spurious associations between
4.4 Maze2D
We further evaluate our method in MAZE2D [5] with the medium maze, a continuous version of NAVIGATION. The agent observes its position, velocity, and goal position, and then outputs and y-velocity to navigate the maze. The agent starting and goal positions are randomly sampled. We collect 100 demonstrations (18k transitions) using planner from Fu et al. [15).
Our method outperforms LfO baselines over all demonstration coverages (see appendix, Figure (7). More importantly, in the low coverage case, our method outperforms BC, which has access to expert actions, as shown Figure 3b This could be because our proximity function generalizes well whereas BC is not robust to unseen states under small demonstration coverages. On the other hand, GoalGAIL shows the best performance regardless coverages as task can be easily solved with the sparse reward and goal relabeling, which is not available for method and other baselines.
4.5 Ant Locomotion
In ANT REACH [6], the quadruped ant is tasked to reach randomly generated goal, which is along the perimeter of half circle of radius centered around the ant (see Figure 2cp. The 132D state consists of joint angle, velocity, contact force, and the goal position relative to the agent. We collect demonstrations (25k transitions) using pre-trained policy (trained 40M steps). When demonstrations are collected, no noise is added to the initial pose of the ant whereas random noise is added to the initial pose during policy learning, which requires the reward functions to generalize to unseen states.

---

Prix Unfff-nneett (Ound
Proc+Drf
Prac-dbx--treert
Proz-Abs
GAIO
* GARC-s-rreeblle
Coa CAPPE CompLeti PM
Ooal Cherr Mor.auunnn00
0
2x4p
Staz
Stop
Stap
(a) NAVIGATION 50%
(b) PETCH PICK 1.75x
(c) FETCH PUSH 1.75x
(d) ANT REACH 0.03
Figure 5: Analysis of the contribution of goal proximity function, uncertainty penalty, and reward formulation to the performance. "Prox" uses the goal proximity function while "GAIfO-s" does not. "+Diff" uses R(st, S(+1) 1) = f(8++1) f(*)) and "+Abs" uses R(st) = ((t) as reward. "+Uncert" adds the uncertainty penalty to the reward. "+Ensemble" uses an ensemble for the discriminator.
In Figure 3c with 0.05 added noise, our method achieves 35% success rate while BCO, GAIfO, and GAIfO-s achieve 1%, 2%, and 7%, respectively. This result illustrates the importance of learning proximity as opposed to discriminating expert and agent states for generalization to unseen states. The performance of GAIfO and GAIfO-s drops drastically with larger joint angle randomness as shown appendix, Figure As ANT REACH not as sensitive to noise in actions compared to other tasks, BC and GAIL show superior results but our method still achieves comparable performance.
4.6 Robotic Manipulation
We evaluate our method in two robotic manipulation tasks with the 7-DoF Fetch robotics arm: FETCH PICK and FETCH PUSH 34 The robot must grasp and move block target position for FETCH PICK, and push a block to a target position for FETCH PUSH. The 16D state consists of the gripper pose, object pose, the gripper pose relative the object, and goal position. Both the initial and target positions of the block are randomly initialized. We generate 1k demonstrations using a hard-coded policy, consisting of 33k and 28k transitions for FETCH PICK and FETCH PUSII, respectively. The policy is trained in an environment with larger noise applied to the starting and target block positions.
In FETCH PICK, our method achieves about 80% success rate outperforming all baselines. despite LíD methods learning with expert actions (see Figure[3d] The best performing baseline BC only obtains around 40% success rate. The high variance performance between seeds comes from the difficulty of learning the grasping behavior with large noise. In FETCH PUSH, our method outperforms baselines in generalization to unseen states by achieving more than 90% success rate (see Figure Be). This shows that our proximity function is able to accelerate policy learning in continuous control environments with superior generalization capability.
4.7 Dexterous Hand Manipulation
We evaluate our method in a challenging in-hand object manipulation task [34], HAND ROTATE as shown in Figure 21 In HAND ROTATE, a 24-DoF Shadow Dexterous Hand must in-hand rotate a block to target z-axis rotation. The state consists of the agent's joint angles and velocities, object pose, and the target rotation. Due to the high dimension of the state (68D) and action space (20D), HAND ROTATE is extremely challenging for both RL and IL without dense reward. We therefore ease the task by constraining the possible initial and target = rotations collect 10k demonstrations (98k transitions) using pre-trained policy (trained for 8M steps).
32 32 and [ g). We
In Figure 31 GAIfO-s performs well because its reward function biased to provide large negative rewards encouraging the agent to end the episode early which is only possible by succeeding. In contrast, our difference-basse reward designed to provide positive rewards, which does not exploit this task property, and performs poorly even with an additional constant penalty -0.005 every step. To test the generalization capability of our proximity function, we additionally examine variant of our method GAIL), which uses the same reward formulation as GAIfO-s, log fa(st) log(1 fa(s:)) With this biased reward function, our method outperforms both GAIFO-s and GAIL, which verifies the benefit of our proximity function in generalization to noisy environments. While BC achieves the high success rate with 10x more demonstrations compared to other tasks, SQIL shows poor performance due to the lack of the negative reward bias.
8

---

130
100
Ours
120
Ours
0.0001
Ours (No Offinn)
Coal Completion (N)
GAIrO.4
Ours (Lincar)
s Norm
Completiion
A= 0.1
Ours tw/ Action)
INo Uncert)
Ours (Rank]
Ours INo Online)
40
Goal
20
274
2.4
Step
Step
Step
(a) Uncertainty penalty
(b) Proximity function design
(c) Baseline regularization
Figure 6: Ablation of method and comparison to regularized baseline on FETCH PICK 1.75x investigate (a) effects of uncertainty penalty coefficient X; (b) effects of proximity function design and online/offline training; and (c) generalization capability of regularization technique and our
proximity function.
4.8 Ablation Study
Dissecting proximity reward We analyze the contribution of the proximity function, reward formulation, and uncertainty penalty method's performance across four tasks in Figure Adding uncertainty to GAIfÖ-s (GAlfO-s+Uncerr) produced 18.4% boost in average success rate compared regular GAIfO-s. Proximity supervision, without the uncertainty penalty, resulted in 66.7% increase in average performance over GAIfO-s with the difference- f(;) (Prox+Diff) and 25.8% with the absolute reward f(r:) (Prox+Abs). This higher performance means modeling proximity is more important than the uncertainty penalty for our method.
reward [*
Although we choose difference-batte reward with exponentially discounted goal proximity, the goal proximity can be either linear or exponential discounting, and both can be used for either a difference- based or absolute reward, which perform differently across tasks. For example, the difference- based proximity reward is better for policy learning than the absolute proximity reward except on ANT REACH and HAND ROTATE, where the bias of the absolute reward (2) helps the agent survive longer and reach the goal. This a fundamental problem in IRI., where inductive bias in reward functions is show that our goal proximity reward provides more stable and generalizable learning signal than baselines under the same reward bias.
crucial and varies by tasks (20). Nonetheless, our extensive experiments (Figure 5 6b
Moreover, we found that the uncertainty penalty and proximity function have a synergistic in- teraction. Combining both the proximity and uncertainty gives 68.7% increase with the difference- based reward (Prox+Diff+nneet and 26.4% increase with the absolute reward Abrox+Abs+Unen Uncert). The uncertainty penalty is especially important for the proximity function models fine-grain tem- poral information where inaccuracies can be easily exploited, as opposed to the binary classification of other adversarial imitation learning methods.
Ensemble networks Next, study if the robustness of method comes from the use of ensemble networks or task progress. We verify this by applying ensemble of discriminators to the best performing baseline, GAIfO-s. 5 that GAIfO-s with ensemble networks (GAIfO- s+Ensemble) only achieves 19.6% higher success rates, but this is still 99.7% lower than our method on average. Therefore, the use of task progress key learn a generalizable reward, the use of ensemble networks.
Regularization of discriminators In our experiments above, we show that our goal proximity function generalizable to unseen states and goals, which leads to successful imitation learning.
We verify whether standard regularization techniques, such as spectral normalization 28 can also
provide the same generalization benefit. In the FETCH PICK 1.75x noise setting (Figure (c). GAIfO-s without regularization struggles to learn, achieving only a 1.43% success rate. Not surprisingly, applying spectral normalization (28 the discriminator of GAIfO-s improves the success rate to 14.56%, which suggests that generalization of the reward function is key to imitation learning with insufficient demonstration coverage. Despite this improvement, our method performs much better 75.45% success. In summary, predicting goal proximity enables significantly better

---

generalization than regularization on the baselines. Figure 10 in appendix show similar results across most other tasks.
Uncertainty penalty coefficient À In Figure bal investigate how the uncertainty penalty coeffi- cient À affects the performance, showing that our method performs the best with À = 0.001. higher or lower À yields worse performance since higher À prevents exploring unseen states while lower encourages exploiting uncertain predictions.
Proximity function training In Figure 6b we test the importance of online and offline training of the proximity function. Note that we update the policy with online interactions in both scenarios. The result shows that online proximity function update is crucial for our method as the agent fails without online update. Meanwhile, no pre-training, Ours (No Offline), slows down training. Similar results can be observed across all tasks (see appendix, Figure 3
Our ablation experiments show that (1) goal proximity generalizes better and is more informative for policy learning: (2) the difference-based proximity reward generally performs better than the absolute one; and (3) the uncertainty penalty boosts the performance of our method. In conclusion, all three components of proximity, difference- reward, and uncertainty are crucial for our method.
Conclusion
We propose a generalizable learning from observation (LfO) method inspired by how humans acquire generalizable task information and learn skills in situations by watching others performing goal-directed tasks. We specifically propose to use task progress, which is intuitive and readily available task information that can guide an agent closer to the goal. Inspired by this insight, we learn a goal proximity function and utilize it as a dense reward for policy learning. We hypothesize that predicting the task progress requires more task-relevant information than estimating occupancy measure 17). and thus generalizes to states not seen the demonstrations. Our extensive experiments on navigation, locomotion, and robotic manipulation show that our goal proximity function improves generalization in imitation learning, which results in better performance compared to LfO methods and comparable performance with LfD methods which learn from expert actions.
In imitation learning. the generalization ability can include generalization to (1) unseen states and goals, (2) new visual enviromments (c.g. background), (3) unseen objects, and (4) different embodiments (e.g. humans to robots or different dynamics). this paper, we focus on generalization to (1) unseen states and goals. This is especially important in imitation learning when the number of demonstrations is not sufficient to cover all possible states and goals. This is very common in imitation learning due costly demonstration collection. Our approach suggests effective way of using the demonstrations with limited coverage by learning the generalizable goal proximity reward.
Generalization to different environment and embodiment is another important research direction and this indeed our immediate future work. Recent advances in generalizable representation learning [42 43 48), robust policy learning [19 [21), and cross-domain correspondence [50) enable us to train a policy that generalizes to new environments and embodiments. Yet, these approaches are orthogonal and complementary to our method our goal proximity function can be trained top of
the learned representations [9 21 42 43 48
55). We believe that our method can be combined with these approaches and improve their performance with better demonstration efficiency and additional supervision about task progress.
Societal Impact Our method aims to increase the ability of autonomous agents, such as robots and self-driving cars, to imitate experts (e.g. humans) from observation alone. This enables autonomous agents to utilize data even without expert actions, such as kinesthetic demonstrations and video demonstrations. Ultimately, it could allow autonomous agents to acquire skills even from watching Youtube videos. Since our method learns from experts, it inherits any biases of the demonstrator, such as sub-optimal or unsafe behaviors. Additionally, demonstrations are easy and intuitive way to specify behaviors, its potential for automation is threat to job security. However, we overall see enormous benefit with this technology increasing human quality of life and automating difficult jobs.
10

---

Acknowledgments and Disclosure of Funding
This research is partially supported by the Annenberg Fellowship from USC and the funding from National Science Foundation (NSF NRI-2024768) and NAVER AI Lab. We thank members of the USC CLVR lab for constructive feedback.
References
[1] Abbeel and A. Ng. Apprenticeship learning via inverse reinforcement learning. In International Conference on Machine Learning. 2004.
[2] M. Andrychowicz. F. Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, B. McGrew, J. Tobin, Abbeel, and W. Zaremba. Hindsight experience replay. In Advances in Neural Information Processing Systems, 2017.
[3] D. Angelov, Y. Hristov, M. Burke, and Ramamoorthy. Composing diverse policies for temporally extended tasks. IEEE Robotics and Automation Letters, 5(2):2658-2663 2020.
[4] H. Bekkering, A. Wohlschlager, and M. Gattis. Imitation of gestures in children is goal-directed. The Quarterly Journal of Experimental Psychology: Section A, 53(1):153-164, 2000.
[5] D. S. Brown, W. Goo, P. Nagarajan, and S. Niekum. Extrapolating beyond suboptimal demon- strations via inverse reinforcement learning from observations. International Conference on Machine Learning, 2019.
[6] M. Burke, K. Lu, D. Angelov, A. StraiZys, C. Innes, K. Subr, and S. Ramamoorthy, Learning robotic ultrasound scanning using probabilistic temporal ranking. arXiv preprint arXiv:2002.01240. 2020.
[7] M. Chevalier-Boisvert, L. Willems, and S. Pal. Minimalistic gridworld environment for openai gym. https:// cww http///xxxmc/////////mmm////ccct gym comm-inngrrdd 2018.
[8] H.-T. L. Chiang. J. Hsu, M. Fiser, L. Tapia, and A. Faust. Rl-rrt: Kinodynamic motion planning via learning reachability estimators from policies. IEEE Robotics and Automation Letters, 2019.
[9] Y. Ding, C. Florensa, M. Phielipp, and P. Abbeel. Goal-conditioned imitation learning. In Neural Information Processing Systems, 2019.
[10] A. D. Edwards and C. L. Isbell. Perceptual values from observation. arXiv preprint /905. 2019.
[11] A. D. Edwards, C. L. Isbell, and A. Takanishi. Perceptual reward functions. arXiv preprint arXiv: 1600: 03824, 2016.
[12] C. Finn, P. Christiano, P. Abbeel, and Levine. connection between generative adversarial networks, inverse reinforeement learning, and energy-based models. NeurIPS Workshop on Adversarial Training, 2016.
[13] C. Finn, X. Y. Tan, Y. Duan, T. Darrell, S. Levine, and P. Abbeel. Deep spatial autoencoders for visuomotor learning. In IEEE International Conference Robotics and Automation, 2016.
[14] Fu, K. Luo, and S. Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
[15] J. Fu, A. Kumar, O. Nachum, G. Tucker, and S. Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint Www.:i002400021999 2020.
[16] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumar, and S. Levine. Divide and conquer reinforcement learning. In International Conference on Learning Representations, 2018.
[17] Ho and Ermon. Generative adversarial imitation learning. Advances in Neural Informa-
tion Processing Systems, 2016.

---

[18] N. J. Hodges, A. M. Williams, J. Hayes, and G. Breslin. What is modelled during observa- tional learning? Journal of sports sciences, 25(5):531-545, 2007.
[19] S. James, A. J. Davison, and E. Johns. Transferring end-to-end visuomotor control from simulation to real world for multi-stage task. In Conference on Robot Learning, pages 334-343 2017.
[20] I. Kostrikov, K. K. Agrawal, D. Dwibedi, S. Levine, and J. Tompson. Discriminator-ccor:: critic: Addressing sample inefficiency and reward bias in adversarial imitation learning. In International Conference on Learning Representations, 2019.
[21] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots, In Robotics: Science and Systems, 2021.
[22] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, 2017.
[23] Y. Lee, S. Hu, Z. Yang. and J. Lim, To follow or not follow: Selective imitation learning from observations. In Conference on Robot Learning. 2019.
[24] Y. Lee, S.-H. Sun, S. Somasundaram, E. S. Hu, and J. J. Lim. Composing complex skills by learning transition policies. In International Conference on Learning Representations, 2019.
[25] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. Continuous control with deep reinforcement learning. In International Conference on Learning Representations, 2016.
[26] Y. Liu, A. Gupta, P. Abbeel, and S. Levine. Imitation from observation: Learning to imitate behaviors from raw video via context translation. In IEEE International Conference on Robotics and Automation, 2018.
[27] M. J. Mataric. Reward functions for accelerated learning. In Machine learning proceedings 1994, pages 181-189. Elsevier, 1994.
[28] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations, 2018
[29] A. Y. Ng and S. J. Russell. Algorithms for inverse reinforcement learning. In International Conference Machine Learning. 2000.
[30] S. Niekum, S. Osentoski, G. Konidaris, S. Chitta, B. Marthi, and A. G. Barto. Learning grounded finite-state representations from unstructured demonstrations. The International Journal of Robotics Research, 34(2): 131-173 2015.
[31] I. Osband, C. Blundell, A. Pritzel, and B. Van Roy. Deep exploration via bootstrapped dqn. In Advances in Neural Information Processing Systems, 2016.
[32] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. Vito,, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. Automatic differentiation PyTorch. NIPS Autodiff Workshop. 2017.
[33] D. Pathak, P. Mahmoudieh, M. Luo, P. Agrawal, D. Chen, F. Shentu, Shelhamer, J. Malik, A. A. Efros, and T. Darrell, Zero-shot visual imitation. In International Conference on Learning Representations, 2018.
[34] M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, al. Multi-goal reinforcement learning: Challenging robotics environments and request for research. arXiv preprint 18X2v::880.09944 2018.
[35] D. A. Pomerleau. Alvinn: An autonomous land vehicle in neural network. In Advances in Neural Information Processing Systems, 1989.
[36] R. Rafailov, Yu, A. Rajeswaran, and Finn. Visual adversarial imitation learning using variational models. arXiv preprint 2rXiv:2107.08829, 2021.
12

---

[37] S. Reddy, A. D. Dragan, and S. Levine SQIL: Imitation learning via reinforcement learning with sparse rewards. In International Conference on Learning Representations, 2020.
[38] Ross, G. Gordon, and D. Bagnell. reduction imitation learning and structured prediction to no-regret online learning. In International Conference on Arlificial Intelligence and Stafistics, 2011.
[39] S. Schaal. Learning from demonstration. In Advances in Neural Information Processing Systems, 1997.
[40] Schulman, Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms. arXiv preprint T.XXvv1777.0647.. 2017.
[41] P. Sermanet, K. Xu, and S. Levine. Unsupervised perceptual rewards for imitation learning. Robotics: Science and Systems, 2017.
[42] Sermanet, C. Lynch, Y. Chebotar, Hsu, Jang, S. Schaal, S. Levine, and Brain. Time- contrastive networks: Self-supervised learning from video. In IEEE International Conference on Robotics and Automation, 2018.
[43] A. Srinivas, A. Jabri, P. Abbeel, S. Levine, and C. Finn. Universal planning networks: Learning generalizable representations for visuomotor control. In International Conference on Machine Learning, pages 4732-4741, 2018.
[44] R. S. Sutton. Temporal credit assignment in reinforcement learning. PhD thesis, University of Massachusetts Amherst, 1984.
[45] G. Swamy, S. Choudhury, J. A. Bagnell, and S. Wu. Of moments and matching: game- theoretic framework for closing the imitation gap. In International Conference on Machine Learning, pages 10022-10032, 2021.
[46] Torabi, G. Warnell, and P. Stone. Behavioral cloning from observation. In International Joint Conference Artificial Intelligence, 2018.
[47] F. Torabi, G. Warnell, and P. Stone. Generative adversarial imitation from observation. arXiv preprint arXiv: 1807,06/7 58, 2018.
[48] T. Wang, R. Liao, J. Ba, and S. Fidler. Nervenet: Learning structured policy with graph neural networks. International Conference on Learning Representations, 2018.
[49] C. Yang, Ma, W. Huang, Sun, H. Liu, J. Huang, and C. Gan. Imitation learning from observations by minimizing inverse dynamics disagreement. In Neural Information Processing Systems, 2019.
[50] G. Zhang. L. Zhong, Y. Lee, and J. J. Lim. Policy transfer across visual and dynamics domain gaps via iterative grounding. In Robotics: Science and Systems, 2021.
[51] B. D. Ziebart, A. Maas, A. Bagnell, and A. K. Dey. Maximum entropy inverse reinforcement learning. In Association for the Advancement of Artificial Intelligence, 2008
[52] K. Zolna, S. Reed, A. Novikov, S. G. Colmenarej. D. Budden, S. Cabi, M. Denil, N. de Freitas, and Z. Wang. Task-relevant adversarial imitation learning. In Conference on Robot Learning. 2020.
13


Reviews:
[{"review_id": "lBSaLWzKBs", "reviewer": "Reviewer_YZMg", "summary": "Disclaimer: \nI was the reviewer of this paper's older version submitted to ICML 2021. Along the reviewing process, the authors' information was kept anonymous. \n\nOverall I'm glad to see this improved version. As the authors stated, 1) comparison to Goal-GAIl added; 2) clearly specify the generalization is limited, and not referring to out-of-demo goals/states which including a different embodiment (for example unseen objects) of environments. Thanks for the authors' improved clarity.\n\nThis paper is well written with sound evaluation results, falling in approaches learning reward function from human demonstrations, though I prefer viewing the generalization problem of adapting human demonstrations to unseen states or actions as more of a representation learning or meta-learning problem since such problem settings will flexibly apply to the manipulation of unseen objects, which is important to learn generalizable manipulation skills.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Learning a robust reward function from human demonstrations is an old topic studied in literature like IRL, GAIL, etc. To overcome overfitting to the human demonstrations, typically, online agent experience is used to continuously improve the reward function estimation. This paper utilizes the same approach by introducing a goal proximity function in a common adversarial training scenario.\n\nThe goal proximity function estimates how many discounted time steps are still needed to reach the target. After that, the reward function is simply the proximity difference between successive states. As a result, two inductive biases are assumed here:\n- (1). the goal proximity, i.e., how much time steps are left to reach the target.\n- (2). The task reward is simply a function of S_{t} and S_{t+1} that is independent of robot actions.\n\nMajor comments:\n\n- (1). While I have the impression that the above two assumptions are too strong in many RL tasks in general, the evaluation results presented in the six tasks convince me that the proposed method will still apply well in a variety of tasks. \n- (2). The claimed generalization from human demonstration seems only mean to generalize to different initial and target configurations in all the tasks. And in the Conclusion section, the authors discussed the possibility of generalizing to tasks with different task settings, for example, objects or backgrounds, but leaving such generalizations as future works. This actually limit the proposed method's applicability since such generalization is now widely studied in approaches leveraging both expert demonstration data and agent online training samples.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, it was discussed in the last paragraph of Page 9.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}, {"review_id": "Rkm_pPKpDa", "reviewer": "Reviewer_AFg8", "summary": "This paper presents a learning from observation technique for solving goal-conditioned tasks from demonstrations and environment interaction, without access to environment provided rewards. The method takes an adversarial learning approach like prior works, where the agent tries to learn behavior that is indistinguishable from that of the demonstrator. However the key difference in this work is that instead of using a binary discriminative objective, the paper uses a temporal progress prediction objective, where the learning agent’s experience is labeled as having 0 temporal progress and the demonstrators experience is densely labeled based on progress in the episode. The learning RL agent then aims to maximize progress (while also avoiding high uncertainty regions) \n\nThe intuition is such an objective will provide (1) a denser reward signal and (2) generalize more effectively to unseen states/goals. The paper demonstrates over a range of domains including navigation, locomotion, and manipulation this seems to hold true, with improved stability and performance over most baselines when generalizing to unseen states.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "*Strengths*\n\nOverall the paper presents a simple and effective idea. \n- Since the data comes from a demonstrator, it is reasonable to assume that the distances are near optimal, and thus that learning a dense measure of temporal task progress from the demos would be effective. \n- It also is reasonable that such an objective will force the discriminator to pay attention to features which measure progress throughout the task, and thus may be able to generalize to new states and goals more effectively than the binary 0/1 adversarial objectives used in prior works. \n- The experiments are thorough, comparing against the relevant baselines in GAIfO and SQIL. They show that across several task domains the when generalizing to unseen states and goals performance using this method is much stronger than the baselines. Figure 4 also nicely shows how GAILs performance degrades as the more and more states are held out while the proposed method does not.\n- The ablations also suggest that the different components of the method, including the uncertainty penalty and using the difference in score between states is important. \n\n*Weaknesses*\n\nThere are a few results in the experiments which could use more explanation:\n- First, I found it surprising how poorly SQIL and GaifO performed across all environments, failing out outperform even vanilla BC.  Is the main failure mode simply that these methods cannot effectively handle unseen states? I think that could make sense, but it would be valuable to confirm then that if the evaluation states/goals are the same as what exist in the demos then these methods should perform well. Figure 4a seems to be testing this exactly, yet GAIfO and SQIL still fail completely. Why do they fail in this setting?\n- Second, I was confused by the difference in performance between using the absolute proximity vs the difference in proximity between time-steps. On 3/4 tasks it seems using the absolute fails completely, and on the 4th using the absolute significantly outperforms using the difference. Could the authors explain further why this difference exists, and which types tasks benefit which approach?\n- Figure 6a seems to indicate that the methods performance is actually quite sensitive to the weighting of the uncertainty parameter, so not sure the paper can claim “our method is robust to different choices of the uncertainty penalty coefficient λ“.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5"}, "extra_text": {}}, {"review_id": "1kl3bBVa99J", "reviewer": "Reviewer_RPMC", "summary": "The paper presents a learning from observation approach that uses a temporal goal proximity estimate as reward function for policy learning. The temporal goal proximity function is learned from observations and measures how many steps are left to reach the target goal state. Additionally, an ensemble of proximity functions is used to empirically estimate the uncertainty function that is added to the reward function to discourage exploration of uncertain proximity states. Results on a number of simulated tasks show performance improvement over GAIL, goal-GAIL, GAIfO, BCO and other competing methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written, experiments are clean and well-presented, the ablation study helps understand the role of how different components of reward function are interacting. Results are also convincing on the presented tasks.\n\nMy main concern is that the task progress indicator seems to be a limiting signal to learn from expert observations. Although the heuristic is simple and works well on the example tasks, its usability may well be limited to low-dimensional and simple short-horizon tasks only. There needs to be some semantic mapping of the goal state and the current state (possibly in a latent space) besides temporal steps to reach the goal state. This would likely be more evident in long-horizon tasks and/or high-dimensional tasks.\n\n- Limitation of temporal goal proximity function would also be evident if the expert demonstrations are sampled from optimal policy roll outs of different reward functions. The uncertainty estimate would likely be higher in such situations, even if the observations lie within the feasible set.\n\n- It would be useful to describe what domain randomization (objects, backgrounds) are performed in the experiments, as the mapping of state to temporal progress should remain invariant.\n\n- Results in Fig. 6(b) indicate that the offline imitation scenario is sub-optimal and online interactions are required for task completion. It would be useful to comment on the sample and time complexity in offline and online scenarios.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above. no negative societal impact.", "review_text": "", "extra_scores": {"ethical_concerns": "no", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "0lQQD2OGxG", "reviewer": "Reviewer_W3nz", "summary": "The paper addresses the problem of learning from observation for directed tasks with a focus on generalization beyond the demonstrations. The main contribution of the paper is a goal proximity function (also referred as goal progress in the paper) which predicts the agent progress towards the goal. The paper has extensive experiments on navigation, locomotion and robotic manipulation which proves that the proposed goal proximity function helps to generalize in imitation learning. The results show that proposed method achieves better performance in case of learning from observation tasks and comparable performance with learning from demonstrations method.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "- The paper is well-written and easy to follow.\n- The proposed method is evaluated on good range of experiments\n- The framework of learning the proximity function from expert demos and agent experience are nice contributions.\n- good comparison with other related methods (GAIL, Goal GAIL etc.,)", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "- for societal impact, the authors mention about the method increases \"the ability of the autonomous agents\", Can you specify what types of ability will be improved or mention specific ability for autonomous agents that will be improved with the proposed method?\n-  Given that goal proximity is a simple measure of steps to the goal from current state, I am having difficulty in understanding that how the proposed method with help in generalization of states and task? The author talks about generalization in the future work, could the authors elaborate it? If space is limited then authors are encouraged talk/discuss about generalization in a separate section in the Appendix.  \n-What kind of new goals the method can generalize to?", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The author response convinced all reviewers, they now unanimously vote for an accept.
Decision: Accept (Poster)