Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a graph prediction method designed to predict how object move in a household as a result of a person's daily activities. The paper also presents a novel simulated dataset for this problem. The method outperforms naive baselines. A demonstration with a robot shows that a robot could in principle use this method to anticipate where objects may be needed in the future. 
strengths_and_weaknesses: 
Strengths:

The paper is written well and the description of the problem and method are clear. The simulator used to generate the data is novel and potentially could be expanded into a paper on its own. A technical demonstration shows that in principle, a robot could use this method. 

Weaknesses:

Results are only reported on simulated data. The baselines are rather naive while the proposed graph prediction method is fairly straightforward and very similar to other methods applied on different domains. A major limitation is that it is unclear how a learned model would transfer to a new household. 

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: 
The recommendation is based on the fact that the graph prediction method in itself is routine while at the same time, the results are based on just simulated data. 
issues: 
The main issue that I see is with the problem formulation itself. Why not learn from multiple households and people at the same time? Or at least formulate the problem in a way that a learned model from one household/person can be adapted to a new household/person through, let's say, domain adaptation? 

Can you also elaborate on what you mean by "open-world"? That phrase appears a few times and it has a very precise meaning in the AI literature, e.g., worlds in which things change (e.g., all of a sudden the human's behavior pattern changes, or new objects appear, or a new set of location is added to the household, etc.). To me, it doesn't appear that the simulated environment is actually an open world as defined in the AI literature. 

The baseline methods do not seem to improve as more training data is available. Any explanation why?

Finally, the  motivation needs to address the issue is what happens when the user loses some locus of control -- how would people actually react if objects in their house start moving without their knowledge?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work focuses on proactive robot assistance for daily/routine tasks in a home environment, e.g., bringing the breakfast materials on to the table. This assistance relies on an object-centric prediction pipeline, based on a graph neural network model. With such an object-centric focus, the dataset provided also differs from the existing ones that are mainly based on activity recognition aspect. 
The data covers a diverse set of activities while spanning a long-horizon (several weeks) timeframe. Human behavioral patterns are segregated into five personas to capture commonalities in the activity patterns. The evaluations consider comparisons to two baselines. The proposed approach achieves a better performance on several metrics. 
strengths_and_weaknesses: strengths:
- a different take on proactive assistance by focusing on object-centric modeling showcasing the pros and cons of such an approach
- better performance on a few metrics compared to a couple of other approaches
- a new dataset containing information on spatio-temporal dynamics of household objects

weaknesses:
- no methodological novelty
- without explicit activity recognition, the predictive model is ignorant of the activity and object states, possibly resulting in inaccurate/unhelpful/redundant assistance
- human behavioral patterns/personas are explicitly (pre-)identified, i.e., independent of the predictive model architecture -> possibly limits the application of the proposed GNN architecture out-of-the-box
- assistance time-window seems to be too coarse (10 minutes), i.e., for tasks that (might) require quicker assistance cannot be tackled out-of-the-box -> diminishing the value proposition of the approach
- the data comes from a simulator, and it's unclear how well it captures daily routines of people, especially in terms of finer time scales and variability
- not much applicability in a real setting, because it still acts as an activity/time-series recognizer/predictor without much focus on robot awareness / autonomy / assistance

robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper offers a different perspective on proactive robot assistance by focusing on how and when objects are interacted with during routine tasks. Considering its weaknesses (some already acknowledged by the authors), it might turn out to be a good stepping stone for future work, especially if/when the model can also capture the 'true state' of the environment/activity. However, at its current state, I do not think that it provides a significant contribution in robot assistance tasks. Even though a superior performance is highlighted compared to two baselines, these baselines also consider object-tracking based predictions, and it's unclear how this method compares against a more traditional activity recognition based approach.
issues: - a better explanation on how/why different personas are selected/clustered. 
- some evaluations on assistance with a finer time window
- any possibility to compare against activity-recognition based approaches?
- time assignments for action sequences: how reliable/accurate they are?
- what do you mean by 'combined the activity schedules and action sequences' when they come from two different set of people/environment/setup?
- clarification on the 'type' of assistance that the proposed method aims for... be more precise in the title and in general within the main text.
- why do you train your model per persona? how do you decide which model to use during testing (e.g., if the persona is unclear)?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a model of object spatio-temporal dynamics and show how it can be used in the context of proactive robotic object relocation. They model the state of objects in an environment with a graph and learn a model that predicts how the graph will evolve over time based on a novel simulated dataset of longitudinal object relocation activities in homes. A robot can then leverage this model to predict that some objects would be needed (e.g. a bowl in the morning around breakfast time) and proactively relocate them. 
strengths_and_weaknesses: Strengths

* A pillar of the author’s approach is that an object-centric model (which does not explicitly represent agents in the environment at all) can be useful for predicting when an object should be relocated proactively. This is an interesting contrast with most work that tries to make robots proactive, which generally have some model of what a human is doing and generate interventions that would assist with the speculated action. Models of people are hard, so I think exploring other routes is interesting.

* The dataset and the method the authors used to collect it are interesting. It’s hard to collect longitudinal data on human activities, especially if you care about timescales like weeks or months. The way in which the authors interleaved the collection of coarse distributional information about habits, then used this to make personas which guide fine grained activity data collection in simulation is a good way to circumvent this problem.

* The quantitative evaluation is good. Comparison to baselines is appreciated. The performance of the baselines is also well connected to the limitations of their respective models.

Weaknesses

* Limited qualitative evaluation of the proposed model. What kinds of mistakes does the model make? When the wrong relocation is predicted, is it more often the object, the destination, or the time that is wrong? How severe are mispredictions? Do objects end up put in weird places? Do they usually get returned later? There’s a good discussion of how the baselines fail on lines 228-237, but no similar discussion for the proposed approach.

* The title of the paper, “Proactive Robot Assistance” may prime users to expect some of the more common forms of assistance from the industrial human-robot collaboration literature like pre-kitting or changing the state of objects with tools (like tightening bolts). It’s not obvious how the approach could provide “assistance” in a general sense like this. The authors study object relocation, which is one (valuable!) type of assistance.

* The absolute level of performance is low. While the same can be said for many robot learning endeavors, here it works against the core argument that you can provide useful proactive assistance with a spatio-temporal object model. 70% of the times the robot chooses to move something, the robot relocates something that didn’t need to be relocated, relocates something to the wrong place, or both. Regardless of whether the approach beats other object dynamics models, this would seem to indicate that the difficulty of predicting even just the consequences of human actions (relocation) is high, and unlike approaches that predict interventions based on human activities, this method doesn’t afford natural interactions where the robot can ask if it should do something in cases of uncertainty. Something like “It looks like you might be getting ready for bed. Should I go ahead and lay out your night clothes?” is easy if you are detecting activities as the basis for predicting interventions. But here, you might just ask, “should I move a glass onto the counter?” around 9PM. Depending on the true context that the user is in, this could seem like a nonsensical question.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The authors show an interesting take on how to provide proactive assistance, but given the limitations of the framing, and given the robotics audience for a paper like this, a deeper discussion of the performance is warranted. A strength of the work is in highlighting that, fundamentally, there's no reason why a robot couldn't realize some basic, useful assistance with a relatively constrained understanding of the world. But the paper seems to gloss over some of the practicalities that come with the approach in its current state.

**Update after rebuttal period:**

Your response has alleviated the bulk of my concerns. I remain concerned about whether this formulation of assistance is ultimately desirable or workable. I agree that making the system more conservative is a reasonable path forward, but the path towards natural interaction to clarify or gather corrections is inherently challenging in this formulation. I agree that this isn't within the reasonable scope of this work. It is also, as another reviewer pointed, an open question whether a system that completely satisfied the objective would actually be liked by users. There is some risk that future work may focus on improving performance while leaving these substantially more important questions unresolved

I think there will be interested readers for this work, so I am updating my rating accordingly.
issues: * Provide discussion of the qualitative performance of the approach. What kinds of errors does the model make? What are plausible paths to addressing these deficiencies?

* Is a limitation that the robot needs to have a full grounding of the graph at any instead before predicting forward? How feasible would it be for future work to relax this need?

* Minor: The small number of users sampled for their habit information struck me as odd. Some mention of why the size of the sample was selected would be helpful. Were there existing, alternative sources for this kind of information? It seems like the kind of thing that would be surveyed or collected in diary studies for any number of purposes, and likely would’ve come from more representative sampling methods.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This work provides a solution to the problem of robotic proactive assistance in everyday human routines. The paper introduces a novel method for successfully solving the tasks at hand. The proposed method is a generative graph neural network that utilises attention and time indexing and provides a useful and fairly large-scale behavioural simulation dataset. This paper was pleasant to read and my opinion is overall positive. 
strengths_and_weaknesses: ### Strengths
- novel solution to an important problem
- clearly written and well evaluated work
- a large scale dataset

### Weaknesses
- Depends on hand-crafted features
- The method does not explicitly take into account the user
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although I am not an expert on proactive robotic assistance I think this paper should be accepted to the conference. It provides a **clearly written** and very **well evaluated novel solution** to a valuable set of problems. In addition, it provides a seemingly **challenging relatively large-scale dataset** that can be instrumental for future research on the problem.
issues: ### Presentation
There are not many issues with this work in my opinion. The paper may become easier to parse if some of the figures were made a bit less complex. For example, Figure 1 can be simplified to just illustrate the idea. Right now it seems overly cluttered which makes it a challenge (at least for me) to grasp the idea from looking at it. Figure 2 could also be made clearer. Right now it is hard to tell what are the individual components (e.g. which part is the attention, where is the time indexing) what are the different network sizes. The figure is also a bit too small.

### Literature
[i] this work seems relevant in the context of collaborative assistance so it may be worth citing. 

[i] Penkov, S., Bordallo, A. and Ramamoorthy, S., 2017, May. Physical symbol grounding and instance learning through demonstration and eye tracking. In 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp. 5921-5928). IEEE.
review_title: null

Meta-review: metareview: The reviewers commended the paper on being well-written and proposing a new dataset on spatio-temporal dynamics of objects in households which could prove beneficial to the research community. The perspective on proactive robot assistance and object-centric modeling is interesting and relevant to the field. I encourage the authors to further address the remaining points raised in the reviews for the final version of the paper.
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)