Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: The paper addresses the problem of learning from observation for directed tasks with a focus on generalization beyond the demonstrations. The main contribution of the paper is a goal proximity function (also referred as goal progress in the paper) which predicts the agent progress towards the goal. The paper has extensive experiments on navigation, locomotion and robotic manipulation which proves that the proposed goal proximity function helps to generalize in imitation learning. The results show that proposed method achieves better performance in case of learning from observation tasks and comparable performance with learning from demonstrations method.
main_review: - The paper is well-written and easy to follow.
- The proposed method is evaluated on good range of experiments
- The framework of learning the proximity function from expert demos and agent experience are nice contributions.
- good comparison with other related methods (GAIL, Goal GAIL etc.,)
limitations_and_societal_impact: - for societal impact, the authors mention about the method increases "the ability of the autonomous agents", Can you specify what types of ability will be improved or mention specific ability for autonomous agents that will be improved with the proposed method?
-  Given that goal proximity is a simple measure of steps to the goal from current state, I am having difficulty in understanding that how the proposed method with help in generalization of states and task? The author talks about generalization in the future work, could the authors elaborate it? If space is limited then authors are encouraged talk/discuss about generalization in a separate section in the Appendix.  
-What kind of new goals the method can generalize to?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a learning from observation approach that uses a temporal goal proximity estimate as reward function for policy learning. The temporal goal proximity function is learned from observations and measures how many steps are left to reach the target goal state. Additionally, an ensemble of proximity functions is used to empirically estimate the uncertainty function that is added to the reward function to discourage exploration of uncertain proximity states. Results on a number of simulated tasks show performance improvement over GAIL, goal-GAIL, GAIfO, BCO and other competing methods.
main_review: The paper is well-written, experiments are clean and well-presented, the ablation study helps understand the role of how different components of reward function are interacting. Results are also convincing on the presented tasks.

My main concern is that the task progress indicator seems to be a limiting signal to learn from expert observations. Although the heuristic is simple and works well on the example tasks, its usability may well be limited to low-dimensional and simple short-horizon tasks only. There needs to be some semantic mapping of the goal state and the current state (possibly in a latent space) besides temporal steps to reach the goal state. This would likely be more evident in long-horizon tasks and/or high-dimensional tasks.

- Limitation of temporal goal proximity function would also be evident if the expert demonstrations are sampled from optimal policy roll outs of different reward functions. The uncertainty estimate would likely be higher in such situations, even if the observations lie within the feasible set.

- It would be useful to describe what domain randomization (objects, backgrounds) are performed in the experiments, as the mapping of state to temporal progress should remain invariant.

- Results in Fig. 6(b) indicate that the offline imitation scenario is sub-optimal and online interactions are required for task completion. It would be useful to comment on the sample and time complexity in offline and online scenarios.
limitations_and_societal_impact: see above. no negative societal impact.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a learning from observation technique for solving goal-conditioned tasks from demonstrations and environment interaction, without access to environment provided rewards. The method takes an adversarial learning approach like prior works, where the agent tries to learn behavior that is indistinguishable from that of the demonstrator. However the key difference in this work is that instead of using a binary discriminative objective, the paper uses a temporal progress prediction objective, where the learning agent’s experience is labeled as having 0 temporal progress and the demonstrators experience is densely labeled based on progress in the episode. The learning RL agent then aims to maximize progress (while also avoiding high uncertainty regions) 

The intuition is such an objective will provide (1) a denser reward signal and (2) generalize more effectively to unseen states/goals. The paper demonstrates over a range of domains including navigation, locomotion, and manipulation this seems to hold true, with improved stability and performance over most baselines when generalizing to unseen states.
main_review: *Strengths*

Overall the paper presents a simple and effective idea. 
- Since the data comes from a demonstrator, it is reasonable to assume that the distances are near optimal, and thus that learning a dense measure of temporal task progress from the demos would be effective. 
- It also is reasonable that such an objective will force the discriminator to pay attention to features which measure progress throughout the task, and thus may be able to generalize to new states and goals more effectively than the binary 0/1 adversarial objectives used in prior works. 
- The experiments are thorough, comparing against the relevant baselines in GAIfO and SQIL. They show that across several task domains the when generalizing to unseen states and goals performance using this method is much stronger than the baselines. Figure 4 also nicely shows how GAILs performance degrades as the more and more states are held out while the proposed method does not.
- The ablations also suggest that the different components of the method, including the uncertainty penalty and using the difference in score between states is important. 

*Weaknesses*

There are a few results in the experiments which could use more explanation:
- First, I found it surprising how poorly SQIL and GaifO performed across all environments, failing out outperform even vanilla BC.  Is the main failure mode simply that these methods cannot effectively handle unseen states? I think that could make sense, but it would be valuable to confirm then that if the evaluation states/goals are the same as what exist in the demos then these methods should perform well. Figure 4a seems to be testing this exactly, yet GAIfO and SQIL still fail completely. Why do they fail in this setting?
- Second, I was confused by the difference in performance between using the absolute proximity vs the difference in proximity between time-steps. On 3/4 tasks it seems using the absolute fails completely, and on the 4th using the absolute significantly outperforms using the difference. Could the authors explain further why this difference exists, and which types tasks benefit which approach?
- Figure 6a seems to indicate that the methods performance is actually quite sensitive to the weighting of the uncertainty parameter, so not sure the paper can claim “our method is robust to different choices of the uncertainty penalty coefficient λ“.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: Disclaimer: 
I was the reviewer of this paper's older version submitted to ICML 2021. Along the reviewing process, the authors' information was kept anonymous. 

Overall I'm glad to see this improved version. As the authors stated, 1) comparison to Goal-GAIl added; 2) clearly specify the generalization is limited, and not referring to out-of-demo goals/states which including a different embodiment (for example unseen objects) of environments. Thanks for the authors' improved clarity.

This paper is well written with sound evaluation results, falling in approaches learning reward function from human demonstrations, though I prefer viewing the generalization problem of adapting human demonstrations to unseen states or actions as more of a representation learning or meta-learning problem since such problem settings will flexibly apply to the manipulation of unseen objects, which is important to learn generalizable manipulation skills.
main_review: Learning a robust reward function from human demonstrations is an old topic studied in literature like IRL, GAIL, etc. To overcome overfitting to the human demonstrations, typically, online agent experience is used to continuously improve the reward function estimation. This paper utilizes the same approach by introducing a goal proximity function in a common adversarial training scenario.

The goal proximity function estimates how many discounted time steps are still needed to reach the target. After that, the reward function is simply the proximity difference between successive states. As a result, two inductive biases are assumed here:
- (1). the goal proximity, i.e., how much time steps are left to reach the target.
- (2). The task reward is simply a function of S_{t} and S_{t+1} that is independent of robot actions.

Major comments:

- (1). While I have the impression that the above two assumptions are too strong in many RL tasks in general, the evaluation results presented in the six tasks convince me that the proposed method will still apply well in a variety of tasks. 
- (2). The claimed generalization from human demonstration seems only mean to generalize to different initial and target configurations in all the tasks. And in the Conclusion section, the authors discussed the possibility of generalizing to tasks with different task settings, for example, objects or backgrounds, but leaving such generalizations as future works. This actually limit the proposed method's applicability since such generalization is now widely studied in approaches leveraging both expert demonstration data and agent online training samples.

limitations_and_societal_impact: Yes, it was discussed in the last paragraph of Page 9.
review_title: null

Meta-review: The author response convinced all reviewers, they now unanimously vote for an accept.
Decision: Accept (Poster)