Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper presented a novel method that extracts object priors from egocentric videos to guide the learning of a robotic agent to interact with objects. The key idea is to model the presence of objects and their co-occurrence from naturalistic activities in first person videos (so called activity context), which can be further used as an auxiliary reward in RL for learning to perform complex object manipulation tasks. The proposed method was demonstrated in a simulated robotic learning environment, with promising results across several tasks. 
main_review: **Originality**

Teaching a robotic agent by learning from first person visual experience is a quite exciting idea. It is great to see the first successful demonstration of such an idea. 

The design of activity context using statistics of object presence and co-occurrence is smart, avoiding the potentially complicated modeling of human body motion. The idea of using object prior as auxiliary reward in RL is interesting, though the RL part is quite standard. 

Overall, this is a well executed paper with a great idea. 

**Quality**

The proposed method is technically sound. The experiments are solid and the results are promising. I appreciate the video demo included in the supplement.

My only concern is that all results are reported under simulation. It will be quite interesting to see some demonstrations in a real world setting. This is less of a concern but more of a hope. I do believe that the ability to demonstrate this idea in the real world could be quite impactful. 

**Clarity**

The paper is well written and easy to follow. The key idea is clearly illustrated with nice visuals across the paper. 

The authors claimed that “the model did not use the interaction / action label” in L148. Yet L242 suggests that the experiment considered 40K clips cropped from the video, where the starting and ending time of these clips are given by the action annotations. I think this part should be clarified.

**Significance**

Teaching robotic agents using real world videos is a critical problem in computer vision and robotics. The success of this paradigm can potentially have a major impact in robotic learning. 

**Post-rebuttal Update**

Thanks for the authors' response. After reading other reviews and responses, I remain quite positive about this paper. Thus, I stand by my previous rating and recommend to accept this paper. 

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this paper, the authors train a vision model to learn an activity context from unlabeled video, use this activity context to derive a reward augmentation strategy, and train a model-free algorithm (PPO) to bring compatible objects (matching the distribution of objects scene during training) together.
main_review: * Originality
The paper's novel contribution is threefold. The authors propose a novel method for learning an embedding of activity priors for video, the authors use these priors to create a reward augmentation strategy, and the authors show that when trained with this strategy, pre-existing model-free algorithms are able to outperform SOTA on interaction heavy tasks.  Learning an embedding of activity priors is not, in itself new [1]. As noted by the author's memory-based methods for reinforcement learning, are also, not in themselves, novel [2]. The authors' primary contribution is reformulating the representation of the memory in terms of activity context, and deriving the reward augmentation from this representation.

* Quality
The paper shows strong results against four competing algorithms (NavExp, IntExp, ScenePriors). The results use both real-world image data and a realistic simulation environment. The figures are clear, and support the author’s central claim about the utility of activity context priors in driving the object aggregation behavior.

* Clarity
The paper is clear, the authors describe the formulation of the activity context and the adaption of the activity context to the reward. The description of how to generate and store the activity context, and how this is then used by the agent policy is also clear.

Figure 4. was a bit hard to parse, the contrast of the auxiliary reward against the background made it difficult to interpret, and the blue lines (which may represent the link of the supporting activity context to a given auxiliary reward) were not explained in the caption.

* Significance
Video datasets are abundant in comparison to supervised datasets of agents performing home activities. Developing methods that can learn useful priors from unlabeled videos would be an extreme boon to the embodied learning community. This work presents an early step towards that goal, as it does a dataset with a comparatively narrow distribution of objects and features (kitchen) Furthermore, this work presents a novel reward augmentation strategy derived from the learned object-centric priors.

limitations_and_societal_impact: There are a few areas that if addressed may increase the score of my review.

The paper could benefit from internal ablations, as it is unclear the contributions of each component of the model to overall performance. I would be interested to see how replacing individual components of the model (ResNet18 backbone, LSTM aggregator) contribute to the overall model performance.

One limitation is that although the activity-context model derives from unlabeled video data, the model itself relies heavily on priors learned from a dataset that includes a large (100K) subset of labeled data [2]. It would be beneficial to see if an action recognition model trained explicitly on the Epic Kitchens dataset only showed strong generalization.

On the topic of generalization, the authors make the claim that the method generalizes to unseen environments. I think it’s important to note the distinction that these activity context priors all share a common underlying distribution (drawn from common household kitchen objects). One relevant ablation would be to alter the distribution of objects to preserve object categories, but such that the distribution no longer matched the distribution of a kitchen.

The authors do not directly address societal impact in the paper. As noted, the Epic Kitchens Dataset is anonymous and publicly available. It may be the case that as data-driven approaches to learning contexts on unsupervised data evolve, ethical concerns will arise about the nature of entrained biases.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces an approach to discover activity-context priors, that is, for a given object, the environment states that are preconditions for attempting given activities with objects. Such priors are acquired from in-the-wild egocentric videos that are captured from a camera that is worn by people performing daily activities. A visual model is trained to detect how humans prepare their environment for their activities from egocentric videos. Video-based priors are then used as auxiliary rewards to encourage agents to seek out similar activity-context states. The proposed approach accelerates learning and generalises to unseen environments.

main_review: Originality: The paper presents a method that is based on the insight that while humans and embodied agents have very different action spaces and bodies, they operate in similar environments. Hence, instead of tackling the gap and transfer between embodiments, the goal of the authors here is to guide agents' interactions towards states that are "demonstrated" by humans and that enables meaningful object interactions. The tasks and methods introduced have limited novelty, and the work is a novel and interesting combination of well-known techniques and ideas. It is clear how this work differs from previous contributions, and related work is adequately cited.

Quality: The submission is technically sound and claims are supported by theoretical and experimental analysis. The proposed method and the alternative methods used as comparison are used in an appropriate way. Comparisons with other existing methods are analysed across different experiments and tests, which completes the evaluation of the proposed method. Limitations of the proposed method are also discussed, and failure cases are presented.

Clarity: The paper is clear and well organised. Details about the implementation and algorithms used are provided, so that enough information is available to reproduce the results.

Significance: The results presented are significant and interesting, the ideas at the basis of this work are valuable and can be used by other researchers in the field to improve performance on complex tasks that entail sequences of object interactions. The idea of guiding agents' interactions towards states that enable meaningful object interactions from human demonstration is not completely novel, but its realisation in this work is neat. 

Other comments:
Would the learned visual model work on real robot/real environments as well? Can you elaborate on the simulation vs. reality gap?
Videos are a great addition to the submission as they make it easier to appreciate the behavior of the proposed method w.r.t. others.

limitations_and_societal_impact: The authors adequately addressed limitations and edge cases. The work does not have societal impact is such.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces activity-context priors learned from real-world egocentric video frames, which are used as an auxiliary reward to encourage RL agents to solve longer-term object interaction tasks. Instead of directly imitating human videos, the prior is computed automatically using statistics from an off-the-shelf object recognition model run on the EPIC-Kitchens dataset. 
main_review: Originality: Although there are other methods which try to compute exploration rewards based on object interaction and novelty, the authors clearly explain how their method is novel. Compared to prior methods, this method specifically tries to encode not just which objects occur near each other, but which objects are actually used by the agent to interact together, which can reduce the impact of spurious correspondences. The experimental comparison is very thorough and the methods which the authors choose to compare to are very relevant. 

Quality: The evaluation tasks are very challenging for RL and the method makes significant improvements on these difficult multi-step tasks.

The compatibility score ablations are thorough and insightful. The method is able to accelerate early training performance, when the algorithm still needs to rely on the dense provided rewards for exploration.

Clarity: The paper is very well written. Code will be released at a later date, but the appendix provides sufficient details for reproducibility at the moment. Addressing the following points may make things even better:

It took me a bit to figure out whether $\phi(o_i, o_j) == \phi(o_j, o_i)$, but I see that it is not. It could be helpful to mention this explicitly.

How sensitive is the method to the auxiliary reward scaling hyperparameter $\lambda_\phi$? 

The intro states “Specifically, we 1) train visual models to detect how humans prepare their environment for activities from egocentric video”, however, maybe this is not quite precise -- the scores seem to model which objects occur in activities together, which is not solely preparation? 

The discussion of how the memory $\mathcal{M}$ is maintained is rather complex and difficult to understand as written. It could be helpful to give a more general description of what the memory is keeping track of and when rewards are given before introducing details.

Significance: The proposed method can use existing egocentric real-world video to learn the priors, and does not need any additional demonstration data. This is impactful in that it may allow methods to better use existing large-scale data to improve the learning of RL algorithms.

However, the performance on tasks with minimal object interactions, while competitive, is not quite as strong as prior work. This is rather surprising as it seems like the proposed method should prioritize object interactions which solve the task and still perform better than navigation-based methods which weigh all object interactions equally. Is there an explanation for why performance is not as strong on those tasks?

Other comments:

Figure 4 is challenging to parse -- it is difficult to see what the correct sequence of object movements to solve the task should be (and what we would like the corresponding red dots to look like).

Figure 2 is great!

Recommendation:

The idea of learning priors specifically to encode activities rather than encouraging any object interactions seems like a promising direction. Comparisons are made to state of the art methods and the method performs well on the most difficult tasks. The paper is well written, the work is positioned and explained well compared to relevant prior work, and the idea is presented clearly. The main negative I see is the performance on the simpler (less object interaction) tasks. So, I think this paper meets the threshold and recommend acceptance.
limitations_and_societal_impact: The authors do well in discussing limitations of their method, including the ACO detection errors and cases where simpler methods seem to perform better. One significant limitation is that the method relies on a hand-object interaction detection model, which restricts the possible sources of data which it can use for learning (other datasets than epic kitchens).
review_title: null

Meta-review: This paper considers an interesting problem of aiming to use egocentric human videos as a prior for guiding the behavior of an embodied agent completing tasks in AI2-iTHOR virtual environment. All of the reviews are positive about the paper, and the author response also helped address several points of feedback from the reviewers. The authors are strongly encouraged to incorporate the feedback and the changes mentioned in the author response into the revised paper.
Decision: Accept (Spotlight)