Predicting Routine Object Usage for Proactive Robot Assistance
Authors: Maithili Patel
Aswin Gururaj Prakash
Sonia Chernova
Venue: CoRL 2023 Poster (2023)
Keywords: Proactive Robot Assistance
User Routine Understanding
Interactive Clarification
Robot Learning

Abstract: Proactivity in robot assistance refers to the robot's ability to anticipate user needs and perform assistive actions without explicit requests. This requires understanding user routines, predicting consistent activities, and actively seeking information to predict inconsistent behaviors. We propose SLaTe-PRO (Sequential Latent Temporal model for Predicting Routine Object usage), which improves upon prior state-of-the-art by combining object and user action information, and conditioning object usage predictions on past history. Additionally, we find some human behavior to be inherently stochastic and lacking in contextual cues that the robot can use for proactive assistance.  To address such cases, we introduce an interactive query mechanism that can be used to ask queries about the user's intended activities and object use to improve prediction.  We evaluate our approach on longitudinal data from three households, spanning 24 activity classes.  SLaTe-PRO performance raises the F1 score metric to 0.57 without queries, and 0.60 with user queries, over a score of 0.43 from prior work.  We additionally present a case study with a fully autonomous household robot.


Reviews:
[{"review_id": "wYqrcv8FqZ", "reviewer": "Reviewer_4vMT", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a new model for the task of predicting routine object usage, which has a direct application in enabling a robot to rearrange objects proactively to assist users. The model extends prior approaches (which predicted solely on the basis of previous object locations) by including historical object locations and activity recognition information. Intuitively, variations in a user’s activities are likely to lead to variations in the objects they interact with. Further, the authors incorporate a query mechanism which enables the robot to ask for a next activity or next object when the model has no confident predictions. The new method is compared against previous work on an enhanced dataset, and results show that the enhancements lead to better prediction accuracy.", "summary_of_recommendation": "This work addresses important shortcomings of existing approaches, and does so with good clarity and reasonable experiments. This paper will inform and motivate future work in designing assistance systems.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Figure 2. What quantity do the error bars represent? \n\nFigure 2b. Why does SLaTE-PRO show lower variance and improved accuracy as the horizon extends?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I am enthusiastic about the approach the authors present as it is close to a maximal use of the information that is readily available to an observant robot (where objects are and what people are doing, over time). At a high level, I believe this paper gives us an indication of how well this kind of proactive assistance can work, at least quantitatively; we should expect it to act appropriately only about half the time. The exact picture of what we should expect is obscured by using F1, because we can imagine situations where either precision or recall are more important. In an assistive use case, we may prioritize recall to reduce the burden on the user. In other settings, users may be perturbed by the robot moving an incorrect object (if for instance, they notice that something has moved without apparent cause, and are worried). While breaking out precision and recall would help, what this really indicates is that the questions readers are likely to have are now largely about the qualitative picture for this kind of assistance, something which is squarely future work.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "jsVC8DonHf", "reviewer": "Reviewer_DVTZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to learn the assistive actions of robots that can proactively engage in tasks based on previous observations of human activities. The paper seems to be highly influenced by the prior work Spatio-Temporal Object Tracking (STOT). However, the paper reveals some of the limitations of STOT and proposes improvements. While STOT was based solely on object information, this manuscript extends that idea with models of user activities and a component of interactive queries. Real-world experiments show that the proposed method outperforms STOT.\n", "summary_of_recommendation": "Despite being incremental w.r.t. STOT, and being dependent on the performance of other methods (Kimera, SparseInst), the motivation and story of the paper is quite clear and convincing. The paper is also well written and evaluations were done to an appropriate level that supports the claims of why the original STOT was enough without human activities and active queries.\n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "About STOT you said \"The resulting model performed well on more consistent user routines, such as using  a plate for dinner, but was unable to predict less consistent activities, such as socializing.\"\nHowever, I failed to see in this submission if your method predicts social activities. Was there support for that sentence that I simply missed?\n\nIs the location of objects learned in absolute coordinates of the observed scenario? Can the method learn the relative position of objects such that if the subject moves to a new house, the previous relational plan is directly applicable without additional (or perhaps minimal) re-training?\n\nHow was the threshold of 0.5 for the information gain tuned? Could you evaluate the user's reaction when the robot decides to make too many questions? \n\nCould you evaluate the user's reaction when the robot ends up doing the wrong task? Is it more frustrating to have a robot that shows too much uncertainty, and makes queries in obvious situations; or a robot that is very assertive but decides to do the wrong actions?\n\n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Although the work seems highly inspired by STOT, it does provide a clear motivation, formulates and presents solutions, and finalizes with experimental evidence on convincing evaluations. The pipeline appears to be highly dependent on other existing methods (Kimera and SparseInst), which makes it a bit questionable how much of the final performance is actually dependent or affected by the performance of those elementary components. The final results are nevertheless, quite impressive. The paper is well-written and easy to follow.\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "gY5HrFL67pU", "reviewer": "Reviewer_ymJh", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper addresses the problem of predicting how users move the objects around them over the course of daily activities and presents an architecture that can also seek user feedback when making such predictions. The method is based on graph-based encoder-decoder network which is trained on each specific user. The method is evaluated on a dataset and also a small demonstration is conducted on a physical robot. \n\n", "summary_of_recommendation": "My recommendation is due to the following:\n\n1. There are major limitations to the representations and assumption that would make this difficult to deploy in practice. \n2. The paper is mostly an applied machine learning paper rather than a robotics paper. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "See questions in review. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nOn the positive side, the paper is very well written and mostly clear. The dataset can be a good contribution for people developing assistive technologies. \n\nThe method itself is a fairly straightforward combination of existing machine learning techniques, it seems the primary novelty is the ability to seek feedback, though the method for selecting what questions to ask is also based on existing information-theoretic techniques for minimizing uncertainty. \n\nWhile the work is definitely related to assistive tech and assistive robotics, it is still a bit difficult for me to say that this is a robotics paper rather than an applied machine learning paper. The robot demonstration shows mainly that you can do pick and place which isn't that novel in itself. \n\nThe framework does not handle objects that may be unobservable at certain times, e.g., objects in containers, drawers, etc., i.e., the framework assumes that all objects' locations are always known. This is a major limitation to applying this in practice as most objects in a household are covered typically. \n\nAt a high level, there needs to be some discussion and ideally a way to address, out of distribution settings at test time, e.g., when something novel is happening (e.g., user is having a party) that was not captured in training data; encoders-decoders generally do pretty bad in novel situations.\n\n\nSome questions and minor comments:\n\nIn paragraph 2 of the introduction, does this mean that you assume the user is still able to perform all activities as for the robot to get observations? what happens when the user is actually disabled and such data is not available or limited to the actions the user can actually perform?\n\nAt the end of the introduction when reporting F-measures, at this point, it not entirely clear what the classification task actually is and hence difficult to tell how good or bad these scores are. \n\nFirst paragraph of section 3, wouldn't the set of locations be continuous? how is an individual location represented?\n\nSecond paragraph of section 3, do you assume time is discrete or continuous? If the former, how do you discretize it? Also, is u_t represented as a string or something else?\n\nWith regards to encoding time of day, is that absolute or relative to let's say, when the user wakes up, has a dinner, etc? Wouldn't this require the user to do their routine at the same time each day? It is possible a user wakes up earlier or later than usual, or goes home at a different time than usual\n\nWhen describing the dataset and stating that there are 93 entities+locations, this seems like a very low number of objects and locations for several-weeks worth of data -- were any objects filtered out? There are typically hundreds of objects in a household. \n\n\n\n\n\n\n", "extra_scores": {"impact": "2: Unsure, difficult to assess."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposed a system that  learn to anticipate user's needs from past observations and use the prediction to provide proactive assistance.
While the work is highly inspired by STOT, it applies the method in a new and unique setting. The paper provides a clear formulation and method. The experimental evidence is convincing and informative. Overall, it is a valuable contribution to the field.  The reviewer points out a few limitations of the proposed approach (unobservable objects, out-of-distribution settings). AC encourages the author to include these discussions in the paper and clearly describe what are the remaining issues that could be addressed by future work.
Decision: Accept (Poster)