Proactive Robot Assistance via Spatio-Temporal Object Modeling
Authors: Maithili Patel
Sonia Chernova
Venue: CoRL 2022 Poster (2022)
Keywords: Proactive Robot Assistance
Spatio-Temporal Object Tracking

Abstract: Proactive robot assistance enables a robot to anticipate and provide for a user's needs without being explicitly asked. We formulate proactive assistance as the problem of the robot anticipating temporal patterns of object movements associated with everyday user routines, and proactively assisting the user by placing objects to adapt the environment to their needs. We introduce a generative graph neural network to learn a unified spatio-temporal predictive model of object dynamics from temporal sequences of object arrangements. We additionally contribute the Household Object Movements from Everyday Routines (HOMER) dataset, which tracks household objects associated with human activities of daily living across 50+ days for five simulated households. Our model outperforms the leading baseline in predicting object movement, correctly predicting locations for 11.1% more objects and wrongly predicting locations for 11.5% fewer objects used by the human user.

Proactive Robot Assistance via
Spatio-Tempora Object Modeling
Maithili Patel
Sonia Chernova Georgia Institute of Technology United States chernova@gatech. edu
Georgia Institute of Technology United States
maithili@gatech.ern
Abstract: Proactive robot assistance enables robot to anticipate and provide for a user's needs without being explicitly asked. We formulate proactive assistance the problem of the robot anticipating temporal patterns of object movements associated with everyday user routines, and proactively assisting the user by plac- ing objects to adapt the environment to their needs. We introduce generative graph neural network to learn a unified spatio-temporal predictive model of ob- ject dynamics from temporal sequences of object arrangements. We additionally contribute the Household Object Movements from Everyday Routines (HOMER) dataset, which tracks household objects associated with human activities of daily living across 50+ days for five simulated households. Our model outperforms the leading baseline in predicting object movement,correst predicting locations for 11.1% more objects and wrongly predicting locations for 11.5% fewer objects used by the human user.
Keywords:
Proactive Assistance, Robot Learning. Graph Neural Network, Object Tracking
robot accordingts, bowl to the table
ceraa
The robot learta anticipate user mag. reguinementt bissed sequential observations user's enviromment
Bow
The robot observes and leamss patterns of object movements resulting from everyday activities, which then uses to predict execute proactive object relocations, such that the objects are locations where the user will need them in the near future.
Figure 1:
1 Introduction
Service robots that operate in human environments and assist users with everyday tasks promise to be valuable domestic [1] as well as workplace (2) settings, and can cater to the needs a wide range of users, including children [3], older adults [4], and people with disabilities [5]. Critically, prior studies have shown that in longitudinal assistive scenarios, users show a strong preference for proactive robot assistance, such that the robot able anticipate and provide for the user's needs without being explicitly asked [6. 7, 8]. Prior work has focused on two proactive assistance scenarios: assistance with fixed pre-determined tasks (e.g., manufacturing assembly [9, 10], object arrangement [11]) and assistance with daily human activities in home or office setting [7, 12, 13]
In this work, we focus the problem of proactive robot assistance towards daily activities in the form of object placement. Examples include the robot fetching objects for the user in anticipation well restoring
of their need (e.g., taking out the breakfast cereal and bowl in the morning), objects after use (e.g., cleaning up after a meal), all without being asked. Prior work has addressed proactive assistance by relying on detailed models of human activity requiring the robot to perform
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

human activity recognition or human action prediction in order to determine which action to perform next [12, 14, 15]. However, it may be impractical for the robot to keep the person in view contin- nously throughout the day, and building activity recognition models that comprehensively cover all human activities remains a challenging problem due to the diversity of human behavior. Moreover, since prior models are limited to assisting only with the user's current activity and predicting only a few minutes into the future, they may not provide sufficient time for a mobile manipulator to locate and fetch objects across house.
Our work contributes novel approach for longitudinal proactive robot assistance over large spaces (e.g. a whole house), and long time horizons (minutes to hours) without requiring recognition of the underlying user activity. Our key insight that objects provide perspective into the user's activities and needs, without explicit activity recognition. Since proactive assistance primarily takes the form of object relocation, tracking the movement of objects in the environment has the benefit of providing actionable object-level information that can be utilized to provide anticipatory assistance. Therefore, as illustrated in Figure 1, our work aims to understand patterns in temporal daily routines of users, anticipate user needs, and plan assistive actions.
Specifically, the contributions of our work are as follows. First, we contribute the Household Object Movements from Everyday Routines (HOMER) dataset, which tracks household objects associated with human activities across 50+ days for five simulated households. Second, we provide a formal definition for longitudinal proactive assistance as an object dynamics modeling problem. Third, we contribute a novel generative graph neural network model that performs spatio-temporal object tracking and facilitates proactive robot assistance by modeling future movement of objects in an environment. We validate our approach against prior works that perform object tracking based on object-object relation frequencies [16] and previously observed periodic routines [17]. Our approach outperforms both baselines in predicting object movement even with as little as 5 days of training data, correctly predicting locations for 11.1% more objects and wrongly predicting locations for 11.5% fewer objects used by the human user. Additionally, we demonstrate the use of our system on a physical robot proactively assisting user with their morning routine.
2 Prior Work
In this section, we discuss prior work relating to the problem of object tracking, the representation of object arrangements, and the computational models relating to our tracking formulation.
Object tracking methods have been developed towards manipulation in home robots [18], collision avoidance [19], augmented reality [20], robot localization [21], etc. Most closely related to our work methods aimed at aiding object search. Probabilistic models have been used to represent beliefs over object locations by combining prior knowledge, observations, and known constraints [22], and additionally leveraging correlational information from datasets to generate semantic priors on the likelihood of inter-object relations [16]. FreMEn [17] and temporal persistence modeling [23] leverage past experience in the environment to model a temporal function of existence of objects of interest in a location. We seek to combine spatial inter-object relations and observed object location histories into a unified spatio-temporal model of object dynamics.
Object tracking for proactive assistance requires us to construct object models that span both space and time. To model space, we utilize scene graphs, which are represented by set of < object, relationship. subject > triplets and model their progression in time through a sequence of scene graphs at discrete time intervals. Probabilistic models [24] or graphical LSTMs [25] are typically used for inference over spatio-temporal graphs, but learning a predictive model of graphi- cal object arrangements requires a generative model that can represent the future scene graph given the current scene graph. Generative Graph Neural Network models predict graphs using encoder- decoder frameworks [26, 27, 28], or step-wise modification [29. 30, 31]. We find the latter more suitable for our application. because of sparse changes in scenes over consecutive time steps. Due to lack of domain specific heuristics that some modification-based networks [29, 3 require, we derive inspiration from a general graph translation method, Node-Edge Co-evolving Deep Graph Transla- tor (NEC-DGT) [30] to learn our predictive model only based on data. Graph Neural Networks have been adapted to model temporal dynamics [32], by exploiting relationships between graphs proximal in time [33], or related by known temporal periodicities in data [34]. However, our model also needs to derive information from the absolute value of time (e.g. the user usually has breakfast around 8am, in addition to after brushing their teeth). Hence, we use a time representation inspired from prior work [35] as global context for our network.

---

Time
Gogglt
Generative graph neural network to predict a future seene, given the current seene and time. The input graph edges are embedded into latent space and then passed through attention- aggregation to obtain information from neighboring edges and temporal context predict the outpul edge existence.
Figure 2:
3
Problem Formulation
We model the state of the environment, Xx, at time t, set of object-location pairs (o,,4) repre- senting the placement of object E O at location 4 E C. An embodied agent (human or robot) can modify the environment by performing a relocation action r(o, (,, (2) move object location 4 to location lg. We allow objects to serve as locations for placement of other objects (e.g., food items placed on plate), such that O n C + 0. We assume all objects in O be relocatable, and instances of the same class (e.g., one of multiple cereal instances) to be uniquely identifiable.
O from
Based on the above formulation, we model the object relocation problem consisting of two parts. First, given a set of previous observations of the environment Xo:M over some time span M, learn the model & (x, (X,,,, () + Xe+5 that takes the time / and the state of the environment X, and predicts the future state Xe+s some fixed 5 timesteps in the future. Second, define the function Vi (X,, Xx+8) > R which returns the set of relocations R required to transition the environment from X, to Xe+5-
Opatio-Temporal Object Dynamics Model
We aim to learn the dynamics of object locations over graphical representations to conserve spa- tial relationship information. We represent the environment state, X, as a directed graph, G, (V,, E. and learn a model ((G,, t) - p(G++6) to predict a fully-connected probabilistic graph from data. G2 is an in-tree with nodes V = {us} representing each object/location instances, and one edge {er,} } E E originating from every node, except the root node, leading to its parent location node. The output probabilistic graph p(G(+5) is distribution over in-trees, and we can infer the posterior, G:45 arg maXG.44 p(G2+5), by picking the most likely out-edge for every node.
To learn the dynamics 4 from graph sequences, create graph translation model'. The inputs to our model are one-hot encodings identifying each object instance as node embeddings, the adjacency matrix of the input graph, and encoding of the output graph timestamp. Our time encoding is sim- ilar to Time2 Vec [35], which uses sinusoidal functions parametrized by time periods T to generate representation of time that can model periodicity. Instead of learning the time periods end-to-end with the prediction task, as done in Time2 Vec, we use ni pre-specified periods to induce meaningful priors on temporal periodicities, and use both sines and cosines for our periodic functions.
Our model generates latent edge features from the input edge existence and the associated node embeddings using Multi Layer Perceptron (MLP), and passes them through aggregation layers to predict output edge existence probabilities as outlined in Figure 2. The first aggregation step oper- ates on the input graph topology. For every edge eijj aggregate the neighboring edge features belonging to the four categories of sharing the origin Ci,A, sharing the destination ét.j. originating from destination cj.m: and ending origin Cn,á by summation and concatenate the four resulting vectors along with the feature of edge Gj and time context to produce the output feature for edge eij. In this manner, the first aggregation layer generates edge features containing information about itself, its neighbors, and time, which are passed through an MLP to generate attention weights for each edge. In the second round, aggregation done in a similar fashion, but on fully connected graph topology. with edge features being weighed by the attention weights before being summed, and the resulting features are passed through an MLP to predict output edge existence likelihoods.
Our model is inspired by the the edge evolution framework in NEC-DGT [30], which operates directly on a fully connected graph topology. We found such unweighted message passing over all edges to be sub-optimal for scaling to the large number of objects in a house, potentially due to
'Implementation shared at https: : http://.ww.....aaa.t.ootooootaaa
3

---

(a) Source samples of what time each activity is done
(b) Extract habits for every activity by clus- tering the samples
(c) Compose complete distri- butions using habit for every activity
(d) Source action
scripts for each activity
BREPWASST
PAMICERATT UNIVERSITY
BRUSHING TEETH
BRUGHIN TRETH
2015
1800
24:55
las tret
1908 thg
(c) Comparison of activity distributions and schedule samples of two personas shows common activities such as showering and breakfast, as well as activities unique to each household, for instance Household the user does cleaning and watches TV in the evening, and in Household B the user leaves the house after lunch and does laundry in the morning-
Figure 3: Process for generating the HOMER behavioral dataset.
over-squashing, so we employed the above attention mechanism. By predicting attention weights based on the sparse input graph topology, we allow the signal from those edges to be stronger. To avoid limiting the model's receptive field to existing neighbors, we allow all edges to contribute in the second step of aggregation, expecting the learned weights to emphasize important neighbors. Empirical results shown in S.c.7.3 confirm that such a mechanism is indeed beneficial.
5 Behavioral Simulation Dataset
To train and validate spatio-temporal object tracking for proactive open world robot assistance, we require a dataset capturing the locations of household objects as they move over the course of routine day-to-day human activities. Specifically, we require data that is longitudinal (spanning weeks or months) and captures naturally occurring variations in human behavior. Prior work has extensively documented routine activities in the home, particularly through datasets for activity recognition [36, 37, 38, 39]. However, existing datasets that contain object location data span only a few hours of data [40, 41, 42], and longitudinal datasets lack object location data [37, 43, 44).
We introduce the Houschold Object Movements from Everyday Routines (HOMER) dataset com- posed of routine behaviors spanning several weeks generated for five houscholds representing habits of different individuals. The dataset is based on an apartment setting, where each apartment consists of four rooms and contains 108 objects. We used the VirtualHome simulator [45] to model human behavior, as it supports human agents, object interaction, and 33 high-level semantic command, such as find, walk, grab, without requiring low-level motion control. Figure 3 presents an overview of the dataset generation process. First, we obtained list of activities of daily living relevant in-home routines from the activity recognition literature [46]. Figure 3e shows the complete set of activities. We then separately crowd-sourced high level activity schedules and the low level action sequences to perform each activity, and used the resulting data to sample the daily routines.
Activity Schedules: To obtain realistic activity schedules, collected data from 21 workers² on Amazon Mechanical Turk about which hours on a typical day from 6am to 12am they are likely to
4F/17M, nearly equal participation from age groups of 25-35, 35-45, and over 45, Four of original 25 workers were omitted due nonsensical answers, leaving 21 workers.

---

be doing each activity, Figure 3a. To build a probabilistic model that realistically captures variations in human behavior, we use clustering to extract diverse underlying habits of how each activity is performed, Figure 3b, and define household using combination of such habits for every activity, Figure 3c (see appendix for more details).
In parallel to the above, we used the VirtualHome simulator to collect execution scripts emulating each activity of daily living, detailing the step-by-step action sequences, well as the minimum and maximum time duration needed to do each action as shown in Figure 3d. We sourced this data from 23 participants, resulting set of action sequences, along with time duration ranges needed to perform each action, for all activities.
Action Sequences:
Schedule Sampling: Finally, using both the activity schedules and action sequences together, generated the HOMER dataset capturing how objects move throughout the environment over time. We characterized each of our five households by a temporal activity distribution and action script per activity, and used Monte Carlo sampling to generate samples of their daily routines', which are visualized in Figure 3e. When executed on the simulator, these routines provide sequence of object arrangements, from which we derive states X₁ to train and evaluate our model.
The sample routines generated by our dataset emulate natural stochasticity in human routines, thus providing a realistic and challenging benchmark for a predictive system such as ours. Figure 3e depicts two example schedules from our dataset, which represent the distribution of activities in two different households. While day-to-day activity patterns in each household differ in timing, duration, order, and frequency, strong commonalities in behavior occur within each household. The user in household A tends to have breakfast only about half of the days and on some days they do so later, after using their computer for a little while. They also take breaks from computer work through the day to read, watch TV and socialize. They tend to work on the computer in the morning followed by taking a break around noon, similar Household B, but then they skip lunch and continue working, whereas the user in Household B most often leaves the house after lunch. Other variations include allocating different times of the day for longer chores, such as Household A takes time to do cleaning towards the evening, whereas Household B does laundry earlier in the day. Thus, we create a challenging test bed for our models, as they need to make good predictions to capitalize on the opportunity to assist the user, but also avoid making incorrect predictions, particularly for toredict activities, which might cause disruptions to the user's natural routine.
6 Evaluation
We evaluate our predictive model sequential scenes from each of our five household datasets and report results averaged across all five. We use TÚ of 1 day, 12 hours, 6 hours, hours, 1 hour, 30 mins, and 10 mins, construct each of the our MLPs with a single hidden layer consisting of 20 neurons and ReLU activation, and use Adam optimizer with a learning rate of 10-3, Starting with a known graph G, at time f, we use our network predict the probabilistic graph one step (10 mins) into the future p(G++1), and feed it back into the model iteratively to predict further into the future. Finally we derive the desired posterior estimate G.+5.
6.1 Baselines
We compare our predictive model against two prior works on open world object tracking:
Static Semantic, adapted from [16]. calculates static priors over object locations using observed object-location relation frequencies, and uses these priors update the estimate on every step. To adapt the noise model to our topological formulation, we use tunable probability of change and spread belief uniformly over all potential locations, as opposed nearby areas in metric space.
FreMEn, adapted from [17], uses past experience to model the prior probability of existence of poject-location relationships as periodic functions in time. We maintain beliefs over topological relations instead of the metric occupancy grid used in [17]. The final belief is a combination of the given state and periodic temporal priors, with tunable time-decaying weight suggested in [17].
6.2 Metrics
Given the current state X,, represented by the graph G,, our objective is to predict the state graph 5 timesteps in the future, G(+5. and use it to infer object relocations, r(o,, 41, 1e) caused by human
'The complete dataset can be found at https: /tttps://ttnn //github. Bat.aaooo/////A/rra...... htter//AAA/raaaaaasssssss/////// httpt//eeeeee/eeeeeeeee/eeeeeeet routines, and further implementation details available in the appendix.
5

---

actions, signifying the movement of object o; from its original location 1 to destination 1z. The set of such relocations, Ru++1, predicted to happen one step into the future t + 1 can be written as Ru.+11 = {r(o,, 1,, (e)le,, GG e, E G L # 11) To assist d-steps into the future, the robot needs to predict relocations R1:+6 obtained from sequen- tial predictions from time t 5 and reconciling multiple relocations:
Re:+5 = Rea+8-1 U {r(o;, 4, 12) (r 4, l2) E Ri+8 1:466, r(o;, (,) & R In addition, we can extract the set of objects that relocated a part of relocations Ras ((R) (o:|r(o,, 1,, (2) RR
We evaluate model selected relocations against the ground truth relocations that would have been made by the human user in the absence of any assistance (Ru++5). The exact order and precise tim- ing of the robot's ss relocations is not critical as long as they occur before the numan-generated event (e.g, the bowl cereal could be taken out first). Hence, measure the predictive performance of relocations over the entire proactivity
For our evaluations, separately consider objects that were used by the human user between time t and 1+6, 6, ((Re+4)), and the remaining unused objects. While moving the unused objects might not cause a direct disruption in the user's routine, it would be disconcerting if objects randomly moved in the house. Hence, we want to maximize the fraction of unused objects that are left in their places (fo,lo; & ((RE++)), O, & ((Rea-)))). The more interesting cases are the objects that were used by the human user. Robot's predictions for these objects fall into one of three categories. First, objects whose predicted movements are correct (O(Rex+s n Rex+6)) Second, objects that predicted move to the wrong locations ((O(Rea+s) ((RE++)) \ (RE:++8 n Rt:++))); these are especially undesirable as the robot would move objects to random place, making it harder for the user to find them. Third, objects that were missed (O(Re:+))) \ O(Rea+s)) and hence were left untouched, which is still undesirable but the user would presumably easily find them in their original locations. Of these three categories, want to maximize the correct relocations, while minimizing the wrong relocations on the objects that the user uses through their routine.
Results
Below, we compare the performance of our model against baselines, investigate the effect of fewer observation days for training, show the importance of our attention mechanism and time encoding, and finally validate our model on a real robot. Our quantitative comparisons are based on 50 days of training data and 30 minute proactivity window, however show results varying proactivity windows in Section 7.1, and discuss the effect of fewer training observations in Section 7.2.
7.1 Performance comparison against Baselines
Our model moves 11.1% more objects correct locations and 1 11.5% fewer objects to wrong loca- tions out of those used by the user, compared to FreMEn, our leading baseline. Of the objects not used by the user, model mistakenly moves only 0.8% compared to over 4% objects moved by our baselines. These results are summarized in Table 1. For these results, we train each model separately on each household, and test withheld set of 10 days from the same household distribution.
Method
& Used Objects
% Unused Objects
Correct
Wrong
Missed
Correct
Wrong
Static Semantic
23.04
16.16
60.80
91.06
8.94
FreMEn
29.42
15.28
55.29
95.30
4.70
Ours
40.51
3.79
55.70
99.19
0.81
Table Comparison predictive performance on used and unused objects for a minute proactivity window our method against baselines
Performance over used and unused objects are shown in Figure for a predictive window of 10 minutes (leftmost bar) to 120 minutes (rightmost bar) with intervals of 10. More objects are used for longer proactivity windows, as can be seen by the total height of each bar representing the total number of used or unused objects in that window. On increasing the proactivity window upto 40 minutes (the first four bars), our method as well as FreMEn predict a larger fraction of moved objects correctly, because larger proactivity window allows for some error in predicting the time of a future event. However beyond the 40 minutes, the performance starts dropping owing to the natural uncertainty in predicting farther out in the future.

---

30
Missed
Whong
1.8
20
Corract
Whorg
Stat
FraMEn
Stat Sem
FraMEn
(a) Breakdown of Used Objects
(b) Breakdown of Unused Objects
Figure 4: Our methods moves the used objects correct places and fewest used objects the wrong places for proactivity 5 10 minutes (leftmost bar) to 120 minutes(rightmost bar), at intervals of 10. Morcover, our method touches fewer unused objects than our baselines.
Across different proactivity windows we observe that FreMEn makes lot of wrong relocation pre- dictions on both used and unused objects. This is because FreMEn rearrangements are based on a prior that is conditioned entirely time, without considering current environment state X,, As result, natural deviations in user behavior (e.g., early breakfast) will cause the model to place ob- jects in their typical locations (e.g., bowl back in the cabinet) even when it should remain unmoved while still in use, or when they should be moved to the next location (e.g.. bowl moved to sink). By contrast, model's ability model typical state progressions, allows it to handle such situ- ations (e.g. bowl moves to the sink after coming out). The Statie Semantic baseline demonstrates weaknesses similar to FreMEn because its prior is agnostic to both X, and time. In real world scenario, both baseline methods would cause the robot to constantly rearrange objects in the house, often misplacing objects that would be needed and moving unrelated items.
Our model robustly predicts routine object movements, while maintaining a conservative behavior over non-routine movements. In Household B (Figure 3e) the takes shower routinely and consistently, so our model is able to reliably predict associated movement of the towel. In situations that especially challenging to predict (e.g.. highly inconsistent breakfast routine in household A), our model acts conservatively and avoids moving objects, resulting in fewer wrong movements (3.79%). A consequence of the model's conservative nature is that when the user does have break- fast, no proactive assistance is provided. This results in missed assistive opportunities (55.7%), but believe real world users would prefer avoid having objects moved unnecessarily. Sometimes our model fails to predict assistive actions for activities with a large time variation, such as relocating the TV remote for household B (Figure 3c), despite them watching TV almost everyday.
7.2 Training Data Efficiency
We evaluate data efficiency by training each model using varying amounts of training data, from 5 to 50 days of observations. As shown in Figure 5, our method improves performance as more data becomes available, and outperforms both baselines with as little as 5 days of observations. The relatively simpler representation of the baseline methods enables those models to achieve high performance sooner, but hurt their improvement with more data as they cannot utilize it to learn more complex patterns. For instance, FreMEn independently models each object being in each location, and cannot learn correlations between different objects. Our model leverage such correlations (e.g., cereal and milk movements being correlated).
Higher
Lower
1 anjects)
45
14
40
20
WWrongly moved 1% unused
12
Wrongly moved 1% used
35
15
10
30
Preliained
25
10
6
TreMiin
Cerecty
20
Stat Som
15
10
20
30
20
30
40
50
20
40
50
Num. Days Observations
Days Observations
Num. Days Observations
(a) Used Objects
(b) Unused Objects
Figure 5: Our method, when trained from scratch (red triangles), outperforms baselines for as little as 5 days of observations, and shows improved performance when pre-trained (orange dots) on other houscholds.
Additionally, we evaluate the benefits of cross-training across households to take advantage of com- monalities in human behavior. We use data from four households to pre-train model, and fine-tune and test on the train and test splits, respectively, fifth houschold. Compared to a model trained
7

---

from scratch, the pre-trained model starts off with higher performance over the used objects (Fig- ure Sa), but slightly lower performance over unused objects (Figure 5b) as it imposes other house- holds' patterns on the new household. As more data is made available, the performance gap on % correct predictions closes, but the pre-trained model makes slightly fewer wrong predictions on both used and unused objects, implying that training on more varied data provides good initializa- tion. By comparison, for both baselines cross-training across households reduces performance.
7.3 Model Analysis
We perform an analysis to measure the impact of our attention mechanism and time representation. The purpose of our attention mechanism to avoid over- squashing caused by message passing over fully connected graph topology with large branching fac-
Method
% Correct
% Wrong
Ours
40,51
3.79
Without Attention
25. 25.85
8.20
Without Time Enc.
24.84
8.29
Table 2: Model Analysis
tor. Empirical results over used objects shown in Table 2 prove its benefit as our model outperforms a version without such a mechanism. Our model also exhibits superior performance over using a linear representation of time as a single number in minutes.
7.4 Robot Validation
Finally, we present proof-of-concept demo of our system on a Fetch robot assisting with a break- fast routine, shown in Figure and detailed in our accompanying video. The shown scenario demonstrates anticipatory assistance through taking out of cereal, milk and bowl prior to the user's s breakfast. Additionally, the robot anticipates the need to clean up. resulting the restorative actions of placing the cereal box and milk back, and leaving the bowl in the sink. One benefit of our object- based proactive assistance model is that is robust to certain variations in the user's routine. When the user skips breakfast, even though the robot prepared the breakfast items, it will also replace them once breakfast time has elapsed.
Tesse brtbegg Estra Fardda
End 2o Peart
(a)
(b)
(c)
(d)
Figure 6: The robot observes user behavior and learns patterns therein (a). The robot uses the learned model for proactive assistance (b), reducing effort to the user (c) both before the breakfast activity, and after (d).
8 Discussion, Limitations and Future Work
In this paper, we introduce an object-centric perspective to providing proactive robot assistance, present a novel dataset for longitudinal object tracking, and contribute novel generative graph neural network formulation for predicting future environmental scenes, Our results show that our model significantly outperforms two prior techniques for modeling environmental dynamics, even if given as little as 5 days of training data. We additionally demonstrate that the model can be used on a real robot to accurately predict future objects movements and plan anticipatory and restorative actions that benefit the user, without the need for activity recognition and subsequent analysis of used objects in each activity. Such assistance does not require explicit user commands, reducing the burden on the user.
As discussed earlier, leveraging only object data removes reliance on potentially incomplete or inaccurate activity recognition models, which can be a strength in complex real-world deployment scenarios. However, the lack of activity modeling is also limitation because the robot is unaware of the user's current actions. For example, if the user is taking an unusually long time to eat breakfast, the robot may clean up too early. Future work should combine the capabilities of both object- based and activity- based systems. An additional limitation is that our method does not currently account for object state. Future work should expand the representation to condition object placement on state information (e.g., place clean bowls in cabinet and dirty bowls the sink). Finally, human behavior is rarely deterministic and user preferences may hard to predict (e.g., cereal or yogurt for breakfast). Future work should integrate user interaction to enable the robot to confirm with the user before performing actions with high uncertainty, and include a user study to understand how users react to such a system.
L Limitations and Future Work:
8

---

Acknowledgments
This work is sponsored in part by NSF IIS 21 2112633.
References
[1] G. Wilson, C. Pereyda, N. Raghunath, G. de la Cruz, S. Goel, S. Nesaci, B. Minor, M. Schmitter-Edgecombb. M. Taylor, and D. J. Cook. Robot-enabled support of daily activities in smart home environments. Cognitive Systems Research, 54:258-272, 2019, ISSN 1389-0417.
10.032.
[2] R. Stock, M. Merkle, D. Eidens, M. Hannig, P. Heineck, M. A. Nguyen, and J. Völker. When robots enter our workplace: understanding employee trust assistive robots. ICIS 2019 Pro- ceedings, 2019.
[3] R. J. López-Sastre, M. Baptista-Ríos, F. Acevedo-Rodriguez, S. Pacheco-da Costa, S. Maldonado- Bascón, and S. Lafuente-Arroyy A low-cost assistive robot for children with neurodevelopmennal disorders to aid in daily living activities. International Jour- nal of Environmental Research and Public Health, 18(8), 2021. ISSN 1660-4601. doi: 10.3390/ijerph18888. 80039744
[4] G. Mois and J. M. Beer. The role of healthcare robotics in providing support to older adults: a socio-ecological perspective. Current Geriatrics Reports, 922)822-9, 2020.
[5] W. Brose, D. Weber, B. A. Salatin, G. G. Grindle, H. Wang. J. J. Vazquez, and R. A. Cooper. The role of assistive robotics in the lives of persons with disability. American Journal of Physical Medicine & Rehabilitation, 89(6):509 521, 2010.
[6] H.-M. Gross, S. Mueller, C. Schroeter, M. Volkhardt, A. Scheidig, K. Debes, K. Richter, and N. Doering. Robot companion for domestic health assistance: Implementation, test and case study under everyday conditions in private apartments. In 2015 JEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS), pages 5992-5999, 2015. doi: R0.109//RO..001577555
[7] G. Peleka, A. Kargakos, E. Skartados, I. Kostavelis, D. Giakoumis, Sarantopoulos, Z. Doul- geri, M. Foukarakis. M. Antona, S. Hirche, E. Ruffaldi, B. Stanczyk, A. Zompas. J. Hernandez- Farigola, N. Roberto, K. Rejdak, and D. Tzovaras. Ramcip- service robot for mci patients at home. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 1-9, 2018 10. 109/TROS 201 201.8594444... 8594214.
[8] J. Baraglia, M. Cakmak, Y. Nagai, R. P. Rao, and M. Asada. Efficient human-robot collabora- tion: when should robot take initiative? The International Journal of Robotics Research, 36 (5-7):563-579, 2017.
[9] B. Hayes and B. Scassellati. Effective robot teammate behaviors for supporting sequential manipulation tasks. 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 6374-6380. IEEE, 2015.
[10] V. Unhelkar, A. Lasota, Q. Tyroller, R.-D. Buhai, Marceau, B. Deml, and J. Shah. Human-aware robotic assistant for collaborative assembly: Integrating human motion predic- tion with planning in time. IEEE Robotics and Automation Letters, 3(3):2394-2400 2018. 10.: 109/LRA.200 2.028899.966..
[11] Baraglia, M. Cakmak, Y. Nagai, R. Rao, and M. Asada. Initiative in robot assistance during collaborative task execution. In 2016 11th ACM/IEEE international conference on human- robot interaction (HRI), pages 67-74, IEEE, 2016.
[12] H. Harman and P. Simoens. Action graphs for proactive robot assistance smart environments. Journal Ambient Intelligence and Smart Environments, 12(2):79-99, 2020.
[13] M. Ali, S. Alili, M. Warnier, and R. Alami. architecture supporting proactive robot com- panion behavior. In AISB 2009 Convention, 2009.

---

[14] X. Puig, T. Shu, S. Li, Z. Wang. Y.-H. Liao, J. B. Tenenbaum, S. Fidler, and A. Torralba. Watch- and-help: challenge for social perception and human-AI collaboration. In International Conference Learning Representations, 2021.
[15] N. Oh, J. Park, J. Ho Kwak, and S. Jo. A robot capable of proactive assistance through han- dovers for sequential tasks. In 2021 18th International Conference on Ubiquitous Robots (UR), pages 296-301, 2021. doi: 10.1 1099/R/2253.200 1994448888
[16] Z. Zeng, A. Röfer, and O. Jenkins. Semantic linking maps for active visual object search. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1984-1990. IEEE, 2020.
[17] T. Krajník, M. Kulich, L. Mudrová, R. Ambrus, and T. Duckett. Where's waldo time t? using spatio-temporal models for mobile robot search. In 2015 IEEE International Conference on Robotics and Automation (ICRA), pages 2140-2146, 2015, 1001000109/1CRR22115..77
[18] B. A. Erol, A. Majumdar, J. Lwowski, Benavidez, P. Rad. and M. Jamshidi. Improved deep neural network object tracking system for applications in home robotics. In Computational Intelligence for Pattern Recognition, pages 369-395, Springer, 2018.
[19] A. Mukhtar, L. Xia, and T. B. Tang. Vehicle detection techniques for collision avoidance systems: A review. IEEE transactions intelligent transportation systems, (15):2318-23392 2015.
[20] H. Uchiyama and E. Marchand. Object detection and pose tracking for augmented reality: Recent approaches. In 18th Korea-Japan Joint Workshop on Frontiers of Computer Vision (FCV), 2012.
[21] C.-C. Wang, C. Thorpe, S. Thrun, M. Hebert, and H. Carrant-Whyte Simultaneous localiza- tion, mapping and moving object tracking. The International Journal of Robotics Research, 26 (9):889 916, 2007.
[22] M. Lorbach, S. Höfer, and O. Brock. Prior-assisted propagation of spatial information for object search. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 2904-2909. IEEE, 2014.
[23] R. Toris and S. Chernova. Temporal persistence modeling for object search. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3215-3222. IEEE, 2017.
[24] B. Douillard, D. Fox, and F. Ramos. A spatio-temporal probabilistic model for multi-sensor object recognition. In 2007 IEEF/RSJ International Conference on Intelligent Robots and Systems, pages 2402-24002 2408 IEEE, 2007.
[25] A. Jain, A. Zamir, S. Savarese, and A. Saxena. Structural-rnn: Deep learning on spatio- temporal graphs. In Proceedings of the ieee conference on computer vision and patlern recog- nition, pages 5308-5317, 2016.
[26] W. Jin, R. Barzilay, and T. Jaakkola. Junction tree variational autoencoder for molecular graph generation. In International conference machine learning, pages 2323-2333 PMLR, 2018.
[27] W. Jin, K. Yang, R. Barzilay, and T. Jaakkol.. Learning multimodal graph-to- traphh translation for molecule optimization. In International Conference on Learning Representations, 2019.
[28] D. Zhou, L. Zheng, J. Xu, and J. He. Misc-gan: A multi-scale generative model for graphs. Frontiers in big Data, 2:3, 2019.
[29] M. Sacha, M. Blaz, P. Byrski, P. Sabrowski-Tumanske M. Chrominski, R. Loska, P. Whodarzzyk-Pruszynkki and S. Jastrzebski. Molecule edit graph attention network: mod- eling chemical reactions as sequences of graph edits. Journal of Chemical Information and Modeling, (1()):3273333 32844 2021.
[30] X. Guo, L. Zhao, C. Nowzari, S. Rafatirad, H. Homayoun, and S. M. P. Dinakarrao. Deep aulti-at graph translation with node-edge co-evolution. In 2019 IEEE International Conference on Data Mining (ICDM), pages 250-259. IEFF, 2019.
10

---

[31] Z. Zhou, S. Kearnes, L. Li, R. N. Zare, and P. Riley. Optimization of molecules via deep reinforcement learning. Scientific reports, 9(1):1-10, 2019.
[32] M. Kazemi, R. Goel, K. Jain, I. Kobyzev, A. Sethi, P. Forsyth, and P. Poupart. Representation learning for dynamic graphs: A survey. J. Mach. Learn. Res., 21(7))11-73, 2020.
[33] S. Yan, Y. Xiong, and D. Lin. Spatial temporal graph convolutional networks for skeleton- based action recognition. In Thirty-second AAAI conference on artificial intelligence, 2018.
[34] H. Peng, H. Wang, B. Du, M. Z. A. Bhuiyan, H. Ma, J. Liu, L. Wang, Z. Yang, I. Du, S. Wang, al. Spatial temporal incidence dynamic graph neural networks for traffic flow forecasting. Information Sciences, 521:277-290, 2020.
[35] V. Peñaloza. Time2vec embedding on seq2seq bi-directional lstm network for pedestrian trajectory prediction. Res. Comput. Sci., 14:(11)) 2020.
[36] R. Singh, A. Sonawane, and R. Srivastava. Recent evolution of modern datasets for human activity recognition: deep survey. Multimedia Systems, 26(2):83-106 2020.
[37] D. J. Cook, A. S. Crandall, B. L.. Thomas, and N. C. Krishnan. Casas: A smart home in box. Computer, 46(7):62-65 69, 2012.
[38] BaD, R. Dai, M. Koperski, L. Minciullo, L. Garattoni, F. Bremond, and G. Francesca. Toyota smarthome: Real-world activities of daily living. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pages 833-842, 2019.
[39] D. F. Fouhey, W. Kuo, A. A. Efros, and J. Malik, From lifestyle vlogs to everyday interactions. In CVPR, 2018.
[40] Stein and McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, pages 729-738, 2013.
[41] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel, al. The" something something" video database
I. Fruend, P. Yianilos, M. Mueller-Freitage for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
[42] G. A. Sigurdsson, A. Gupta, C. Schmid, A. Farhadi, and Alahari. Charades-ego: A large- scale dataset of paired third and first person videos. arXiv preprint arXiv: 1rXiv:1804.09626, 2018.
[43] J. Cumin, G. Lefebvre, F. Ramparany, and J. L. Crowley. A dataset of routine daily activities an instrumented home. International Conference on Ubiquitous Computing and Ambient Intelligence, pages 413-425. Springer, 2017.
[44] U. Köckemann, M. Alirezaie, J. Renoux, N. Tsiftes, M. U. Ahmed, D. Morberg, M. Lindén, and A. Loutfi. Open-source data collection and data sets for activity recognition in smart homes. Sensors, 20(3):879, 2020.
[45] X. Puig, K. Ra, M. Boben, J. Li, T. Wang. S. Fidler, and A. Torralba. Virtualhome: Simulating household activities via programs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8494-8502, 2018.
[46] Ramasamy Ramamurthy and N. Roy. Recent trends in machine learning for human ac- tivity recognition-4 survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4)ee254, 2018
11


Reviews:
[{"review_id": "pvydAdHjjGE", "reviewer": "Reviewer_EX28", "summary": "This work focuses on proactive robot assistance for daily/routine tasks in a home environment, e.g., bringing the breakfast materials on to the table. This assistance relies on an object-centric prediction pipeline, based on a graph neural network model. With such an object-centric focus, the dataset provided also differs from the existing ones that are mainly based on activity recognition aspect. \nThe data covers a diverse set of activities while spanning a long-horizon (several weeks) timeframe. Human behavioral patterns are segregated into five personas to capture commonalities in the activity patterns. The evaluations consider comparisons to two baselines. The proposed approach achieves a better performance on several metrics. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strengths:\n- a different take on proactive assistance by focusing on object-centric modeling showcasing the pros and cons of such an approach\n- better performance on a few metrics compared to a couple of other approaches\n- a new dataset containing information on spatio-temporal dynamics of household objects\n\nweaknesses:\n- no methodological novelty\n- without explicit activity recognition, the predictive model is ignorant of the activity and object states, possibly resulting in inaccurate/unhelpful/redundant assistance\n- human behavioral patterns/personas are explicitly (pre-)identified, i.e., independent of the predictive model architecture -> possibly limits the application of the proposed GNN architecture out-of-the-box\n- assistance time-window seems to be too coarse (10 minutes), i.e., for tasks that (might) require quicker assistance cannot be tackled out-of-the-box -> diminishing the value proposition of the approach\n- the data comes from a simulator, and it's unclear how well it captures daily routines of people, especially in terms of finer time scales and variability\n- not much applicability in a real setting, because it still acts as an activity/time-series recognizer/predictor without much focus on robot awareness / autonomy / assistance\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper offers a different perspective on proactive robot assistance by focusing on how and when objects are interacted with during routine tasks. Considering its weaknesses (some already acknowledged by the authors), it might turn out to be a good stepping stone for future work, especially if/when the model can also capture the 'true state' of the environment/activity. However, at its current state, I do not think that it provides a significant contribution in robot assistance tasks. Even though a superior performance is highlighted compared to two baselines, these baselines also consider object-tracking based predictions, and it's unclear how this method compares against a more traditional activity recognition based approach.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- a better explanation on how/why different personas are selected/clustered. \n- some evaluations on assistance with a finer time window\n- any possibility to compare against activity-recognition based approaches?\n- time assignments for action sequences: how reliable/accurate they are?\n- what do you mean by 'combined the activity schedules and action sequences' when they come from two different set of people/environment/setup?\n- clarification on the 'type' of assistance that the proposed method aims for... be more precise in the title and in general within the main text.\n- why do you train your model per persona? how do you decide which model to use during testing (e.g., if the persona is unclear)? \n"}}, {"review_id": "oyugUWjtfI-", "reviewer": "Reviewer_fzEn", "summary": "The authors present a model of object spatio-temporal dynamics and show how it can be used in the context of proactive robotic object relocation. They model the state of objects in an environment with a graph and learn a model that predicts how the graph will evolve over time based on a novel simulated dataset of longitudinal object relocation activities in homes. A robot can then leverage this model to predict that some objects would be needed (e.g. a bowl in the morning around breakfast time) and proactively relocate them. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n* A pillar of the author’s approach is that an object-centric model (which does not explicitly represent agents in the environment at all) can be useful for predicting when an object should be relocated proactively. This is an interesting contrast with most work that tries to make robots proactive, which generally have some model of what a human is doing and generate interventions that would assist with the speculated action. Models of people are hard, so I think exploring other routes is interesting.\n\n* The dataset and the method the authors used to collect it are interesting. It’s hard to collect longitudinal data on human activities, especially if you care about timescales like weeks or months. The way in which the authors interleaved the collection of coarse distributional information about habits, then used this to make personas which guide fine grained activity data collection in simulation is a good way to circumvent this problem.\n\n* The quantitative evaluation is good. Comparison to baselines is appreciated. The performance of the baselines is also well connected to the limitations of their respective models.\n\nWeaknesses\n\n* Limited qualitative evaluation of the proposed model. What kinds of mistakes does the model make? When the wrong relocation is predicted, is it more often the object, the destination, or the time that is wrong? How severe are mispredictions? Do objects end up put in weird places? Do they usually get returned later? There’s a good discussion of how the baselines fail on lines 228-237, but no similar discussion for the proposed approach.\n\n* The title of the paper, “Proactive Robot Assistance” may prime users to expect some of the more common forms of assistance from the industrial human-robot collaboration literature like pre-kitting or changing the state of objects with tools (like tightening bolts). It’s not obvious how the approach could provide “assistance” in a general sense like this. The authors study object relocation, which is one (valuable!) type of assistance.\n\n* The absolute level of performance is low. While the same can be said for many robot learning endeavors, here it works against the core argument that you can provide useful proactive assistance with a spatio-temporal object model. 70% of the times the robot chooses to move something, the robot relocates something that didn’t need to be relocated, relocates something to the wrong place, or both. Regardless of whether the approach beats other object dynamics models, this would seem to indicate that the difficulty of predicting even just the consequences of human actions (relocation) is high, and unlike approaches that predict interventions based on human activities, this method doesn’t afford natural interactions where the robot can ask if it should do something in cases of uncertainty. Something like “It looks like you might be getting ready for bed. Should I go ahead and lay out your night clothes?” is easy if you are detecting activities as the basis for predicting interventions. But here, you might just ask, “should I move a glass onto the counter?” around 9PM. Depending on the true context that the user is in, this could seem like a nonsensical question.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors show an interesting take on how to provide proactive assistance, but given the limitations of the framing, and given the robotics audience for a paper like this, a deeper discussion of the performance is warranted. A strength of the work is in highlighting that, fundamentally, there's no reason why a robot couldn't realize some basic, useful assistance with a relatively constrained understanding of the world. But the paper seems to gloss over some of the practicalities that come with the approach in its current state.\n\n**Update after rebuttal period:**\n\nYour response has alleviated the bulk of my concerns. I remain concerned about whether this formulation of assistance is ultimately desirable or workable. I agree that making the system more conservative is a reasonable path forward, but the path towards natural interaction to clarify or gather corrections is inherently challenging in this formulation. I agree that this isn't within the reasonable scope of this work. It is also, as another reviewer pointed, an open question whether a system that completely satisfied the objective would actually be liked by users. There is some risk that future work may focus on improving performance while leaving these substantially more important questions unresolved\n\nI think there will be interested readers for this work, so I am updating my rating accordingly.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "* Provide discussion of the qualitative performance of the approach. What kinds of errors does the model make? What are plausible paths to addressing these deficiencies?\n\n* Is a limitation that the robot needs to have a full grounding of the graph at any instead before predicting forward? How feasible would it be for future work to relax this need?\n\n* Minor: The small number of users sampled for their habit information struck me as odd. Some mention of why the size of the sample was selected would be helpful. Were there existing, alternative sources for this kind of information? It seems like the kind of thing that would be surveyed or collected in diary studies for any number of purposes, and likely would’ve come from more representative sampling methods.\n"}}, {"review_id": "ir8klm-hgUS", "reviewer": "Reviewer_KdUK", "summary": "This work provides a solution to the problem of robotic proactive assistance in everyday human routines. The paper introduces a novel method for successfully solving the tasks at hand. The proposed method is a generative graph neural network that utilises attention and time indexing and provides a useful and fairly large-scale behavioural simulation dataset. This paper was pleasant to read and my opinion is overall positive. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- novel solution to an important problem\n- clearly written and well evaluated work\n- a large scale dataset\n\n### Weaknesses\n- Depends on hand-crafted features\n- The method does not explicitly take into account the user", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although I am not an expert on proactive robotic assistance I think this paper should be accepted to the conference. It provides a **clearly written** and very **well evaluated novel solution** to a valuable set of problems. In addition, it provides a seemingly **challenging relatively large-scale dataset** that can be instrumental for future research on the problem.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "### Presentation\nThere are not many issues with this work in my opinion. The paper may become easier to parse if some of the figures were made a bit less complex. For example, Figure 1 can be simplified to just illustrate the idea. Right now it seems overly cluttered which makes it a challenge (at least for me) to grasp the idea from looking at it. Figure 2 could also be made clearer. Right now it is hard to tell what are the individual components (e.g. which part is the attention, where is the time indexing) what are the different network sizes. The figure is also a bit too small.\n\n### Literature\n[i] this work seems relevant in the context of collaborative assistance so it may be worth citing. \n\n[i] Penkov, S., Bordallo, A. and Ramamoorthy, S., 2017, May. Physical symbol grounding and instance learning through demonstration and eye tracking. In 2017 IEEE International Conference on Robotics and Automation (ICRA) (pp. 5921-5928). IEEE."}}, {"review_id": "8WepicU-b_8x", "reviewer": "Reviewer_QKuy", "summary": "The paper presents a graph prediction method designed to predict how object move in a household as a result of a person's daily activities. The paper also presents a novel simulated dataset for this problem. The method outperforms naive baselines. A demonstration with a robot shows that a robot could in principle use this method to anticipate where objects may be needed in the future. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nStrengths:\n\nThe paper is written well and the description of the problem and method are clear. The simulator used to generate the data is novel and potentially could be expanded into a paper on its own. A technical demonstration shows that in principle, a robot could use this method. \n\nWeaknesses:\n\nResults are only reported on simulated data. The baselines are rather naive while the proposed graph prediction method is fairly straightforward and very similar to other methods applied on different domains. A major limitation is that it is unclear how a learned model would transfer to a new household. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "\nThe recommendation is based on the fact that the graph prediction method in itself is routine while at the same time, the results are based on just simulated data. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "\nThe main issue that I see is with the problem formulation itself. Why not learn from multiple households and people at the same time? Or at least formulate the problem in a way that a learned model from one household/person can be adapted to a new household/person through, let's say, domain adaptation? \n\nCan you also elaborate on what you mean by \"open-world\"? That phrase appears a few times and it has a very precise meaning in the AI literature, e.g., worlds in which things change (e.g., all of a sudden the human's behavior pattern changes, or new objects appear, or a new set of location is added to the household, etc.). To me, it doesn't appear that the simulated environment is actually an open world as defined in the AI literature. \n\nThe baseline methods do not seem to improve as more training data is available. Any explanation why?\n\nFinally, the  motivation needs to address the issue is what happens when the user loses some locus of control -- how would people actually react if objects in their house start moving without their knowledge?\n\n\n"}}]

Author rebuttal:

Decision: Accept (poster)