Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper introduces an inverse reinforcement learning approach for learning from diverse third-person videos. The key component of the method is representing the frames of videos with graphs, which capture scene semantics and object interactions while leaving out visual details such as textures. They generate the graphs by running a pre-trained object detector then computing geometric relationships between each pair of detected objects; then, they apply temporal cycle consistency-based self-supervised learning on the graph representations, which yields an embedding space from which they can generate a dense reward signal given an observation and a goal. They show much improved performance with their method on a simulated sweeping task and three robotic manipulation tasks on a real robot.
strengths_and_weaknesses: Strengths:
1. Graph-based method is intuitive, well-explained, and shown to be very effective.
2. Experiments are thorough.
3. Writing is very clear.

Weaknesses:
1. The selection of architecture for processing the graph and building the graph could be better motivated. For instance, were other graph convolutional architectures considered? Why use a fully-connected graph? etc.
2. Regarding the graph representation, it is unclear exactly how exactly it differs from prior work in object-centric representations.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper shows that graph representations can be used to greatly improve the performance of inverse RL when learning from diverse data. The method is elegant, clearly explained, and highly effective, and the experiments are thorough, with results shown on a real robot. 
issues: no specific issues, addressed in weaknesses
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes a method for learning a reward function which is both cross-embodiment and cross-domain. This allows for the reward function to be learned from human videos, for the learned reward to be used for learning in simulation, and for the learned policy to be deployed in the real world again on a physical robot. A graph representation is used to cross the domain gap, and temporal cycle consistency loss is used to cross the embodiment gap. Experiments show improved performance on a cross-embodiment benchmark, and successful deployment on a physical robot.
strengths_and_weaknesses: Strengths:
1. The combination of graph representations and IRL appears to be novel and is likely to have significant impact, as this general approach can be used for many tasks.
1. The experimental finding that learned reward functions can be better than hand-designed rewards for these tasks is intriguing, and may be useful for the community.
1. The paper does well at proposing a vision for this learning pipeline: “learning rewards from real-world human demonstrations, learning a policy in simulation using learned rewards, and finally deployment of the learned policy on physical hardware”. This seems like a promising way forward for robot learning, and therefore raises the impact potential of this work.
1. Evaluation on a real robot shows that this method can be readily applied in the real world, supporting the vision for the full pipeline.
1. Data efficiency. As suggested by an ablation, the graph abstraction is much more data-efficient. A comparison using this metric to other methods may be interesting.

Limitations:
1. The node features only use the bounding box and centroids of each object, in the current draft of the paper. Other visual information is discarded. This is what helps the method cross the domain gap, but also drastically cuts the number of tasks that the method can be applied for, if it does not use visual features beyond the sizes of objects. For example, if the object to be pushed and the goal are the same size, then it is unclear how the network would be able to tell them apart.
1. X-MAGICAL is a useful benchmark, as it is designed specifically for evaluating cross-embodiment methods, but it is not very clear why strong performance in this benchmark would necessarily translate well to strong performance in real robot experiments. The policy is state-based rather than image-based. It seems like objects do not collide or interact much, so this might not show the full power of the Interaction Networks used in this method. The results on this benchmark suggest that when trained on not diverse videos but one visual style of videos, the proposed method generalises better to videos with different visual features. This makes sense because these visual features are all discarded by the representation, keeping only bounding boxes. However, in the second row of Figure 4, it seems like the baseline method XIRL performs just as well as the proposed method when the training dataset contains visually diverse videos, presumably because the visual encoder learns which features to focus on. This might lessen the impact of the current work, because it is also motivated by the context of learning from diverse videos.
1. The real-world experiments address a scenario which also might not show the full potential of Interaction Networks, as there are only 2-3 objects involved. The results are not very convincing for the proposed method unfortunately, in the current draft. The number of demonstrations is in the range of 162-256, which is high considering the simplicity of the task. For the peg-in-hole experiment, it does not seem as though 3D geometric understanding is needed to complete the task. In the video on the website, simply moving in a 2D line without raising peg allows peg to be dragged over wall and into box, due to compliance of string. In the demo videos, the peg is lifted over the wall -- it seems like the robot does not succeed in learning this behaviour. Additionally, it is not clear how the limited visual features in the graph representation would allow this 3D information to be captured. On Line 250, it is claimed that the proposed method outperforms the oracle method. This conclusion is not fully proven by the data as it stands. This is because (a) of the variance of the reward curves and (b) because this conclusion seems highly dependent on this specific termination point for the reward curve, when before that SAC was significantly out-performing GraphIRL, and after this point it is unclear which method will perform better, since neither curve looks like it has reached convergence.
1. The limitations section does not explore the limitations of the method in detail, such as those mentioned in the review.

Questions:
In Eq (4), why is g not just the final frame, but an aggregation across frames?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: The paper proposes a novel idea and with high impact potential. However, there are some limitations introduced as a result of design decisions, and experimental evidence does not yet conclusively show the potential of this work. I will happily reconsider my weak reject recommendation if the visual features limitation is addressed and additional experiments demonstrate cleanly the advantages of this method.
- - -
EDIT: thank you to the authors for their hard work in completing the rebuttal. Taking the paper and discussion into consideration, I have now updated my recommendation to **Weak Accept**. The main reasons for this are as follows:
* The authors have clarified how 3D reasoning can still be performed by their method. Thank you for this: I now understand and agree.
* The authors have clarified that the object identity is included in the representation, albeit without visual features. With object identity and size, this is sufficient to solve some tasks already which are useful in the real world, and so it is sufficiently impactful as a method to merit acceptance. However, the fact remains that removing visual features is what allows this method to cross the sim2real gap, so extending this method to tasks which require the robot to use visual features will require significant changes, therefore this remains a limitation.
* The new experiment on OOD performance shows that the prior XIRL work does struggle with overcoming these visual distractor features, so the value of the proposed method has been made clearer.

Additional comments on the rebuttal:
* I appreciate the data efficiency experiment. This is a promising hypothesis to demonstrate an advantage of the graph representation. However, a comparison to other methods is required to demonstrate this advantage, since it is possible that the performance of other methods changes similarly when the training dataset size is changed. I understand that the authors have promised to run this later and include it in the final version if the paper is accepted, presumably due to time constraints.

Overall, graph representations seem like a useful and novel direction for IRL, and so this paper will be a valuable addition to CoRL 2022.

issues: 1. Line 32: another important advantage of third-person videos, such as those used in this approach, compared to demonstrations involving a robot is the availability of in-the-wild data.
2. The description of TCC was quite concise, but it may be worth spending more space on this to explain it in more detail as it is quite key to the paper. The diagram from the original TCC paper conveys the intuition well.
3. In Eq (3), lowercase s is not defined, it seems. Probably it should be the same as uppercase S.
4. It may be helpful to emphasise that SAC is an oracle method in the real world experiments, in that it has access to reward function information
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work proposes a framework for robot learning from third-person human demonstration videos. To learn from a diverse set of videos that have different viewpoints, motions, and embodiments, the IRL is performed at an abstract, graph level. They show that the reward function, indicating task progression, that is learned with this abstraction can be used to train an RL agent and the learned policy can be successfully transferred to a real robot. 
strengths_and_weaknesses: Plus
- The research problem is well motivated. The components in the proposed pipeline are designed well. There are no major flaws in experiments and evaluations.
- Choosing the right representation for an IRL problem is a fundamental research question. Scene graphs have a lot of potential in this direction and this work has demonstrated its effectiveness.
- No environment nor additional hand-crafted reward is required, which sets this paper apart from some previous works.

Minus
- This work assumes an object detector is given, which is a reasonable assumption. But there are several unanswered questions. What off-the-shelf detector is being used here? What’s the detection accuracy in the tasks presented? More critically, how accurate the perception module needs to be for the entire pipeline to be effective?
- Graph abstraction has all the advantages mentioned in the paper, but like any abstraction, some information is lost. What is some critical information that is lost in the abstraction? Does this prevent the proposed approach to generalize to tasks that heavily rely on, for example, object pose information, 3D geometric information, etc. This should be at least discussed in the limitations section.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: Overall this is solid work. See strengths for the reasons for acceptance. Although the concerns raised should be addressed and discussed in detail. 
issues: - How is occlusion handled? Do we always assume that all objects are visible?
- What is the action space for robotic tasks?
- Sim2real results: we need more information on what causes the performance drop here. What are some typical failure cases? Is that the policy? Perception? Action execution?
- Sim2real results: for Peg-in-Box, the performance in the real world is better than the one in simulation? 
- Ln73-75: can you explain why approaches like BCO [14] assume shared observation / action space? At least not according to this review paper: Recent Advances in Imitation Learning from Observation (Torabi et al., 2019) see section 3.1.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a technique they call GraphIRL, for learning reward abstractions across domains with diverse embodiment gaps. The method decomposes the underlying scene structure and agent-object interaction into a graph, which the authors claim is invariant to distractor features (like texture, agent embodiment, etc). The learned reward is then used in an IRL setting to learn policies from third-person demonstrations and they show successful cross-embodiment transfer on both simulated and real-world setups.
strengths_and_weaknesses: Strengths

- The paper is highly polished and easy to read. The videos are high quality and the experimental results are quite thorough (with some details missing… see below).

- Real robot experiments are appreciated.

- This is an incredibly challenging problem domain so any improvement in performance could be impactful for the community.

Weaknesses

- Some references are missing. There have been quite a few papers that use object-centric representations to define (by definition) invariant state representations. One example would be Manuelli., “Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learning.” I suggest the authors perform an additional literature search.

- The reward is not conditioned on policy actions and so implies some limitations in the form of reward that can be learned. e.g. learned rewards cannot be directly action dependent - or assumes the learned reward can infer the policy action if this is what is needed (concrete example: rewards that introduce penalties or costs for certain actions). Please add details discussing the fundamental limitations of the assertions made, especially since you have a “Limitations” section.

- Similarly, the reward is defined on a “representative goal frame” which also limits the rewards that can be learned (effectively, you can only represent rewards where some individual goal state is sufficient to solve the task). This should be discussed to avoid overselling the method.

- I’ve read the “Our Approach” section a few times and while I think I understand how the graph is constructed, a visual aid would be immensely helpful. I understand the component parts (psi, f_o, f_s, f_in), and I can follow these into S_i and S_j, but it would be nice to see the flow from image to bounding box, to embeddings to construction of the graph. Figure 2 is a good start, but doesn’t actually contain any of the terms from the “Our Approach” section so is somewhat hard to follow.

- Table 1: please explain why the environment reward underperforms your method? Is this a sparse reward (and if so why did you compare a dense reward with a sparse one)? Otherwise, I’d expect a well crafted true reward to be an upper bound so this requires some explanation.

- For the real robot experiments, did you train on any real data? Or did you train the reward only in sim and then deploy in real? Likewise, did you train the policy in real (or reward + policy trained in sim and deployed in real)? Sorry if this is spelled out, but I wasn’t able to confirm. If it’s the case that the reward was trained in sim only, it would make sense to me that a method that first crops objects learns a reward function that can better transfer sim to real, while a method like XIRL, that uses full image embeddings, won't perform as well (since background distractor features are going to result in generalization issues). This isn’t necessarily a ding against the method at all (and in fact using bounding boxes is a reasonable contribution), but an experiment that also trains the full stack in real would be necessary to make sure the graph representation accounts for the claimed performance gains (and it’s not just because of cropping out of the background).

- Nit: It would be helpful if the number of videos was in the Table 3 caption.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I’m on the fence between weak reject and weak accept. The proposed method is sound, however I have some questions pertaining to the real-robot experiments so that readers can better understand the performance implications of the designed reward. Likewise, an honest and comprehensive review of the method's limitations is needed (the discussion around requiring human demonstrations seems like a throwaway statement), however this seems easily addressable and go a long way to help frame the paper within the context of solutions to the problem.

Update: Bumping up to weak accept after rebuttal. The authors did a great job addressing some of my concerns in the updated version.
issues: Issues listed in the summary above.
review_title: null

Meta-review: metareview: This paper proposes an interesting approach to learning rewards from a diverse set of videos and has received some strong reviews. The approach is original, and the paper well-motivated and well-written. The experimental section is thorough with good results, and includes experiments on a real robot. Using a graph-based method for IRL is novel and could have significant impact. However, there were a few weaknesses found by the reviewers. In particular, more justification for the choice of architecture and visual features could improve the paper. How would different choices impact performance? Along with this, a more in-depth discussion of the limitations of the approach would greatly improve the paper. The authors are encouraged to address these concerns along with other concerns raised by the reviewers.

=========

Thank you to the authors for the detailed response and updated experiments. 
confidence: 5: The area chair is absolutely certain
Decision: Accept (Oral)