Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups
Authors: Jan Ondras
Abrar Anwar
Tong Wu
Fanjun Bu
Malte Jung
Jorge Jose Ortiz
Tapomayukh Bhattacharjee
Venue: CoRL 2022 Poster (2022)
Keywords: Multimodal Learning
HRI
Assistive Robotics
Group Dynamics

Abstract: We develop data-driven models to predict when a robot should feed during social dining scenarios. Being able to eat independently with friends and family is considered one of the most memorable and important activities for people with mobility limitations. While existing robotic systems for feeding people with mobility limitations focus on solitary dining, commensality, the act of eating together, is often the practice of choice. Sharing meals with others introduces the problem of socially appropriate bite timing for a robot, i.e. the appropriate timing for the robot to feed without disrupting the social dynamics of a shared meal. Our key insight is that bite timing strategies that take into account the delicate balance of social cues can lead to seamless interactions during robot-assisted feeding in a social dining scenario. We approach this problem by collecting a Human-Human Commensality Dataset (HHCD) containing 30 groups of three people eating together. We use this dataset to analyze human-human commensality behaviors and develop bite timing prediction models in social dining scenarios. We also transfer these models to human-robot commensality scenarios. Our user studies show that prediction improves when our algorithm uses multimodal social signaling cues between diners to model bite timing. The HHCD dataset, videos of user studies, and code are available at https://emprise.cs.cornell.edu/hrcom/

Human-Robot Commensality: Bite Timing Prediction for Robot-Assisted Feeding in Groups
Jan Ondras" Cornell University j ankoθcs. coriess.ooeeeeeeee
Abrar Anwar*
Tong Wu' Rutgers University
University of Southern California abrar anwargusc. edu
tong wu9eeutgere edu
Fanjun Bu
Malte Jung
Jorge Jose Ortiz Rutgers University jorge. w wrw.iouttttttttteeeeeeeee
Cornell Tech
Cornell University mf mfjj2@ccrnell.ed
fb266@cornell.ed
Tapomayukh Bhattacharjee
Coreell University tapomayukhh@crrell.cco
We develop data-driven models to predict when a robot should feed during social dining scenarios. Being able eat independently with friends and family is considered one of the most memorable and important activities for peo- ple with mobility limitations. While existing robotic systems for feeding people with mobility limitations focus on solitary dining, commensality, the act of cating together, is often the practice of choice. Sharing meals with others introduces the problem of socially appropriate bite timing for robot, i.e. the appropriate timing for the robot to feed without disrupting the social dynamics of shared meal. Our key insight is that bite timing strategies that take into account the delicate balance social cues can lead to seamless interactions during robot-assisted feeding in a Humann-Human
Abstract:
social dining scenario. We approach this problem by collecting Commensality Dataset (HHCD) containing 30 groups of three people eating to- gether. We use this dataset to analyze human-human commensality behaviors and develop bite timing prediction models in social dining scenarios. We also transfer these models to human-robot commensality scenarios. Our user studies show that prediction improves when our algorithm uses multimodal social signaling cues between diners to model bite timing. The HHCD dataset, videos of user studies, and code are available at https://mpprise.sscomm wwtppriiieess..oollleeee. comnel..ed//hrccm
Keywords: Multimodal Learning, HRI, Assistive Robotics, Group Dynamics
Introduction
Nearly 27% of people living in the United States have disability, and close to 24 million people aged 18 years or older need assistance with activities of daily living (ADL) [1]. Key among these activities is feeding, which is both tine-consuming for the caregiver, and challenging for the care recipient (patient) to accept socially [2]. Indeed, needing help with one or more ADLs the most cited reason for moving assisted or institutionalized living [3, 4]. Although there several au- tomated feeding systems on the market [5-13], they have lacked widespread acceptance. One the key reasons is that all of them require manual triggering of bite timing by the user, which is chal- lenging for users with cognitive disabilities and inconvenient in social settings. A key challenge for the realization of autonomous robotic feeding systems is therefore infer proper bite timing [14].
While existing systems focus on solitary dining (e-g. [15-32]), commensality, the act of eating together, is often the practice of choice. People like to share meals with others. The social experience of a shared meal is an important part of the overall cating experience and current robot feeding systems are not designed with that experience in mind. Transferring the challenge of inferring appropriate bite timing to a social dining setting requires not only attuning to the user's cating
'These authors contributed equally to this work.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Human-Human Commensality Dataset
Social Bite Timing Model
Transfer to Numan-RRbot Commensality
For Right, Left, and User
User
3x RGBD cam
seconda
Faoc Aepsiree
Gasr Speaking Statu
4 mic
Deploy
Trained Model
Feed User with robot?
Does User intend take a bite?
Right
Left
Does not take bite
a bite
sessions
> 18 hours
Figure 1: Our bite timing prediction workflow: (Left) Human- Human Commensality Dataset collec- tion: We record audio and video of participants eating food in triads. (Middle) Our Social Nibbling NETwork (SoNNET) learns to predict whether a user intends to take a bite based on a 6-second win- dow of social signals. (Right) We conduct social robot-assisted feeding user study by deploying variation of SoNNET on a robot. We refer to the User also as a Target user.
behavior but also to the complex social dynamics of the group. For example, a robot should not attempt feed user who is actively engaged in conversation. Motivated by growing body of research that seeks to develop models for robots to function in group settings [33, 34] we ask the seemingly simple question: How should an assistive feeding robot decide the right timing for feeding a user in ever-changing and dynamic social dining scenarios?
We developed intelligent autonomous robot-assisted feeding system that uses multimodal sensing to feed people in dynamic social dining scenarios. We collected a novel audio-visual Human-Humme Commensality Dataset (HHCD) capturing human social eating behaviors. Using this data, we then trained multimodal machine learning models to predict bite timing in numan-human commensality. We explored how models trained on human -umman commensality scenarios performed in human-robot commensality setting and evaluated them in a user study. The overall workflow is shown in Fig. 1. We made algorithmic and experimental design decisions by consulting with care recipients, caregivers, and occupational therapists. We find that bite timing prediction improves when model accounts for social signaling among diners, and such model is preferred over manual trigger and a fixed-interval trigger. Our main contributions include:
A SOcial Nibbling NETwork (SoNNET) which captures the subtle inter-persone social dynamics in numan-humaa and human-robot groups for predicting bite timing in social doiiilg scenarios. Methods that can successfully transfer bite timing strategies learned from human- -umann com- mensality cues to human-robot commensality situations, which we evaluate in a user study with a robot in 10 triadic human groups
socially-awar robot-assisted feeding system that extends our capacity to feed people in solitary settings to groups of people sharing meal.
An analysis of various social and functional factors that affect human feeding behaviors during human-human commensality.
A novel Human-fluman Commensality Dataset (HHCD) containing multi-view RGBD video and directional audio recordings capturing 30 groups of three people sharing a meal.
Human-Robot Commensality
Eating is a complex activity that requires the sensitive coordination of several motor and sensory functions. Anyone who has fed another knows that feeding, particularly social feeding where person is being fed in social setting, is delicate dance of multimodal signaling (via gaze, facial expressions, gestures, and speech, to name few). Research on commensality, the practice of eating together, has highlighted the importance of the social nature of eating for social communion, order,

---

health, and well-being [35]. As consequence. digital commensality has focused on understanding the role of technology in facilitating or inhibiting the more pleasurable social aspects of dining [36].
When a person relies on assisted feeding, meals require that patient and caregiver coordinate their behavior [37]. To achieve this subtle cooperation, the people involved must be able to initiate, perceive, and interpret each other's verbal and non-verbal behavior. The main responsibility for this cooperation lies with caregivers, whose experiences, educational background, and personal beliefs may influence the course of the mealtime [38]. Our goal in this work is to understand the rhythm and timing of this dance to enable an automated feeding assistant to be thoughtful of when it should feed the user in social dining settings. We introduce the concept of Human-Robot Commensality at the intersection of commensality and robot-assisted feeding in social group settings.
Our research is motivated by the key insight that bite timing strategies that take into account ever- changing social signals and group dynamics can lead to a seamless human-robot collaboration in social dining scenarios. Fueled by this insight, we believe feeding device that takes the initiative and offers bites proactively during the meal at times when a bite is likely be desired will create more seamless dining experience than a device that requires the user to initiate bites. Herlant [39] designed an HMM to predict bite timing in dyadic robot- robot-assisted feeding. However, her model only considered the social cues of the user. Bhattacharjee et al. [40] found users preferred less intrusive interfaces in a social dining scenario, specifically a web interface over a voice interface. Our work aims to build non-intrune intrusive bite timing strategies by focusing on learning when to feed user in triadic scenarios while using implicit social features from all diners.
Particularly, bite timing is important because the consequences of presenting a bite to the diner earlier than expected is poorly tolerated. This can include an interruption to conversation or to finishing chewing the prior bite. The consequences of presenting bite later than desired can include frustration towards the robot and disruption of the natural flow of conversation during the meal. Parallels can be drawn to interruptibility research on finding the most appropriate timing to probe a user. Researchers have found that people performed best on a task if interruptions were mediated rather than timed immediately or on scheduled intervals [41, 42], often mediated based on modeling contextual and social factors [43-46].
A socially-aware robot-assisted feeding system should be designed such that if needed, the user should be able to communicate these intentions via multiple different modalities such as body lan- guage, gaze, or speech. These various modalities have been found to be effective in modeling social interactions [47-50]. Capturing these natural social interactions in computational models are likely crucial to provide accurate bite timing without distracting users from the social ambiance.
3 Problem Formulation
The objective of the bite timing prediction problem in robot-assiste feeding with single diner is to predict the timing of when this user will take a bite of food by capturing their signals U such as voice, body gestures, head movements or speaking status. We define the proper timing for when a robot should feed as when the user intends to take a bite of food. It takes input signals U(t : f) from time to to time t and learns a function F(U) to predict Boolean y(t + h) = F (U (to : t)). which indicates whether the user intends to take a bite in the time horizon h and trigger a bite transfer at time & 1. When a person lifts their fork off the plate to eat, they intend to take bite of food, where this time horizon i is the time it takes to transfer the food to their mouth from their plate.
In this paper, we consider a social variant of the bite timing prediction problem where a user is interacting with two co-diners. Our goal is to predict the timing of user to take a bite of food based on the social cues within the interaction. From an initial time ty to time f, the user receives social signals L(to t) and R(to= t) from their left and right conversational co-diners, respectively. Given these external social signals and the target user's 's own history of signals U(to t). we aim to predict y. We note that may not always be possible to track the same set of features for a user and their co-diners. Therefore, for some time range k- l-- - to and feature dimensions n, m for the user and co-diners respectively, U E T kxn while L, R E Rxmm where n does not necessarily equal m. The function to learn is:
1)++1==3 = F (U (to : f). L (to : f), (to t))
4 Model: SOcial Nibbling NETwork (SoNNET)
We present the SOcial Nibbling NETwork (SONNET) that predicts when a user has the intention to eat based on various social signals. We selected features to represent both human eating and social
3

---

Cony
Left
Dense Layers
Does User intend
Right
Yes
No
Demav toar Coupue SOWVET
User
User's Features
Figure 2: Triplet-SoNNETT contains three interacting channels for features of a target user and two co-diners. Each channel concatenates the input of time, gaze, speaking and skeleton features from each single diner. Couplet-SoNNETT eliminates all features from the target user by dropping the last channel; however, continues to use the user's bite features. Batch normalization layers are not shown in the figure.
bite features, which include the number of bites taken so far and the time since the last bite of food RR,, a diner's gaze and head pose direction R binary speaking status 8 E {0, 1), and face and body keypoints features b are computed only for the user and not the co-diners, since we do not estimate in real-time whether co-diner is taking bite of food. Thus, for a time interval k = I lo. these features are temporally stacked to construct the input signals U E kR7x175 E RRx117 R E R*x173 for the user, left co-diner, and right co-diner, respectively.
behavior:
I 168 from OpenPose [51]. We note that, in our case, the bite
Recently, convolutional neural networks (CNNs) have demonstrated significant success for multi- channel time series classification from various kinds of signals [52-54] Wu et al. [46] proposed PazNet: a multi-channel deep convolutional neural network which able to handle inputs of dif- ferent dimensions. PazNet is designed to predict the interruptibility of individual drivers. However, the information of different channels is not shared, and it lacks ability to capture social interactions among multiple people.
We design the Social Nibbling NETwork (SoNNET), a new model architecture which follows a multi-channel pattern allowing multiple interconnected branches to interleave and fuse at different stages. We create input processing channels for each diner, then add interleaving tunnels between each convolutional module and adjacent branches. The information capturing visually-observable behaviors between the diners allowed to flow between the frames and channels. We conjecture that our model will learn socially-coherent structure, allowing the model to implicitly represent the diners in an embedding space. Therefore, each channel has the same structure but does not share the same weight parameters. To help capture informative features, we performed dimension--edduction after the interleaving components using max pooling layers and 1 x convolutional layers. These per-diner channels are concatenated and then followed by two dense layers for classification, which decides whether the user intends to feed or not. For SoNNET, the range between t and to is six seconds. The social signals in this range are used to predict user's bite intentions.
Triplet-SoNNET. For modeling the bite-timing prediction of three users with no mobility limita- tions, we propose Triplet-SoNNETT which uses social signals from the left and right co-diners L, R and signals from the user U. Depicted Fig. 2, TriNEet-SoNNET ensures that the features from other co-diners L, R interleave into the target user's features U.
Couplet-SoNNET.. To run TriNEE in robot-assisted feeding setting, there would be distribution shift in the kinds of signals target user outputs. Our goal is to feed people with mobility limitations while they are engaged in social conversations. The features from someone self-feeding are inherently different from someone using a aosit-assisted feeding system. In the case of body pose, a target user with C3-C5 SCI would be largely still, which different from the training data. Our Human-Humai Commensality Dataset consists of adult diners with no mobility limitations, thus applying a trained Triplet-SoNNETT model to robot-assisted feeding of a user with mobility limitations would be out-of-distributie Although our target users with C3-C5 SCI cannot

---

move their arms to feed themselves, there is still a large spectrum of severity in mobility limitations depending on the users' conditions. From our discussions with key stakcholders, caregivers, and occupational therapists we design, Couplet-SoNNETT, where we ignore most social signals from the target user by removing the last channel in Triple--oNNNE. Therefore, the intention to feed +(++1) F (U, (fo t) L (to t), R (fo : f)), where U, E R are the user's bite features for = t to. The user's bite features, such as the time since the last bite and the number of bites since the onset of the feeding activity, are the only social signals from the target user. Additional discussion on this design choice can be found in App. 8.2.2.
5 Huuman-Human Commensality Dataset (HHCD)
We introduce a novel fluman-Humal Commensality Dataset (HHCD) of three participants with no mobility limitations eating in a social scenario. We used this dataset to develop models that predict a diner's intention take a bite of food while taking into account subtle social cues. We deployed the trained models in social robot-assisted feeding setting where one diner is fed by a robot. Beyond predicting bite timing, we are excited for the robot learning community to find other interesting challenges within our dataset that leverage understanding social dynamics.
Data Collection Setup. We recruited 90 people among our ceriitution-fffliatte fully-vaccinated students, faculty, and staff to eat meal in a triadic dining scenario. Each participant was 18+ years old and took part in the study only once. The study setup is illustrated in Fig. I (left). There are three cameras (mutually at 120") in the middle of the table, each capturing one participant, and a fourth camera capturing the whole scene. All four cameras are Intel RealSense Depth Cameras D455 [55]. The scene audio was captured by ReSpeaker Mic Array v2.0 [56] placed in the middle of the table. The ReSpeaker microphone array has four microphones arranged at the corners of a square and estimates the direction of sound arrival. For the study setup measures, see App. 8.1.2.
Participants were free to bring any kind of food and any utensil with them. They could also bring drink (some drank from a cup, others from a bottle or both, with or without a straw) and were pro- vided with napkins. Before the study, each participant was asked to fill in pre-study questionnaire about their demographic background, relationship to other participants, and social dining habits. The experimenter then asked them eat their meals and have natural conversations. At this point, the experimenter started the recording and left the room. When all three participants finished cating or after 60 minutes have passed, whichever was earlier, the experimenter stopped the recording and asked participants to fill in a post-study questionnaire about their dining experience. The specific questions asked in both pre/post-study questionnaires can be found in App. 1.3. The study was approved by Cornell's IRB.
Data Annotation. We annotated each participant's video based on their interactions with mouth, food, drink, and napkins. In particular, we annotated foodeentered, foodJifted, mouth drink entered, drink drink_io_mouth napkin_entered mouth oopen events. We chose these events as they are key transition points during feeding. We spent 151 hours annotating and used the ELAN annotation tool [57]. We assigned the annotation value E {fork, knife, spoon, chopsticks, hand) based on the utensil performing the food-to-mout touth handover. While annotating, we also noted down per-participant food types and observations of interesting behaviors. All annotation types with detailed rules are provided in App. 8.1.4.
supkin_lifted. napkin mouth, and
Data Statistics. There were 56 female and 34 male participants, and their ages ranged 18-38 (μ
22, a = 3) years. Session durations ranged 21-55 (μ1 37, a = 9) minutes and 1 session was at breakfast, 10 at lunch, and 19 dinner time. For additional dataset statistics, see App. 8.1.5.
For a summary of all available data in the dataset and its detailed analysis, see App. 8.1. For the purposes of this work, only consider bite features, speaking status, gaze and head pose, and body and face keypoints.
6 Model Evaluation on Human-Human Commensality Dataset
In this section, we evaluate Triplet- and Couplet-SoNNET against other models on the HHCD. In particular, we compare against regularized linear SVM trained with SGD to evaluate performance of linear classifiers. We also consider a Temporal Convolution Network (TCN) [58, 59], which uses causal convolutions and dilations to represent temporal data. TCNs have been found to perform better than LSTMs and GRUs on temporal anomaly detection [60] and robot food manipulation tasks [20], therefore they would provide strong bascline to compare models to. We also perform an
5

---

Table 1: Ablation study on different modali- ties from various data sources. We use aver- age over LOSO cross-validation.
Table 2: Model comparison on LOSO cross- validation over 29 sessions.
Method
Acc. Prec. Rec.
Fl nMCC
Method
Acc. Prec.
Rec.
F1
nMCC
Always Feed
0.72 0.72 1
0.83
0.5
Triple--oNNNET
0.820 0,861 0.871 0,862 0.772
Linear SVM (SGD)
0.68 0.82 0.77 0.74
0.64
Speaking Status
0,816 0,864 0.863 0,856
0.771
Triplet-TCN
0,82 0.82 0.96 0,88
0.72
Gaze & Head Pose
0.815 0.863 0.863 0,856
0.769
Trplet--SNNET
0.82 0.86 0.87 0.86
0.77
Bite Features
0.781 0.832 0.855 0.834
0.727
Body & Face
0,820 0.854 0.886 0.865 0.771
Couplet-
0.73 0.73 0.98 0.83
0.55
Couplet-SoNNET 0.76 0.78 0.96 0.85 0.66
ablation study to investigate the importance of various modalities. Implementation details about baseline models, SoNNET, and training procedure can be found in App. 8.2.
For training. we use 6811 food lifted annotations as positive training labels since they precede an actual bite of food and indicate an intention to eat. We use time interval of k = t - to = 6 seconds because takes roughly seconds for the robot to move from its wait position to feeding the user. Since bite actions sparsely distributed over time, we select 2486 6-second clips as negative samples that are in the middle of two food food_lifted annotations. All reported models are trained with leave-one-session-out (LOSO) cross-validation to evaluate generalizability to new groups of people. Due to an issue with recording, we train over 29 sessions if speaking status features are used.
The user's bite features b E R²2 (time since last bite and the number of bites eaten since the start) are indicators of eating rate. To ensure this feature is not dominated by higher dimensional features, we scale the size of the input by 7. This hyperparameter scales b E R + b E ? We selected 7 = 100 100 after a grid search over the training set on the TCN and SONNET models.
Evaluation Metries. A high recall indicates that our model can reliably feed when it should. In contrast, a high precision indicates that a model tends be stricter in deciding when to feed. Due to our dataset imbalance, the average accuracy across 29 sessions for model that predicts it should always feed is 71.56%. This classifier achieves perfect recall, and relatively high precision, causing the model to have high F1 score. It is clear that given this class imbalance, a high F1 score poorly represents the capabilities of this model. To evaluate our model effectively, we consider the normalized Matthews Correlation Coefficient (nMCC) in addition to F1 score, precision, recall, and accuracy. Unlike F1 score, nMCC considers the size of the majority and minority classes, and can only produce high scores if a classifier is able to make correct predictions for majority of both the negative and positive classes [61]. A value of 0.5 indicates random prediction, while 0 is inverse prediction and is perfect prediction.
Effects of Modality. We are interested in investigating features that are the most informative for designing good bite timing predictor in social dining. We perform a feature ablation study on the Triplet-SoNNE" model, as shown in Table 1. We selectively remove feature streams, such as body and face data from OpenPose, gaze and head features from RT-GENE, speaking status signals, and the user's bite features. We find that users' bite features such as the time since last bite and the number of bites are important, as accuracy drops drastically without them. Intuitively, we believe this feature is important because a user's bite features are proxy for their level of eating rate. We also see that without body and face features, Fl and recall slightly increase. This could be due to the fact these data streams are noisy: however, as indicated by the lower accuracy and MMCC when removing OpenPose features, these features are useful.
Effects of Model Type. Table 2 shows the outcomes of various model comparisons when trained using LOSO. We compare performance of Triplet-SoNNET against a linear SVM and TCN trained on all three diners. We call this TCN a Triplet-TCN. Triplet-TCN has all the diners' features con- catenated per-timestep, and we compare this result to TriNlet-SNNNET. We find that Triplet-SoNNet achieves higher accuracy and nMCC compared to all other models; however, it performs worse on recall and Fl score compared to Triplet-TCN. In our scenario, we want to ensure that the robot feeds when it should and does not feed when it should not. A bite prediction model that overfeeds or un- derfeeds is not ideal. A high nMCC balances the roles of recall and precision and better represents whether a classifier should or should not feed. Therefore, for our scenarios, Triplet-SoNNET is a more effective predictor of bite timing than other models trained on all three diners.
Effeets of Social Scenario. We are interested in comparing the ability of models to learn social behaviors using only two co-diners' features rather than having full observability. We compare

---

Couplet-SoNNETT to camilarly-namees Couplet-TCN trained on two co-diners' features and user's s bite features. As expected. Couplet-TCN and Couplet-SoNNET perform worse than their Triplet- counterparts, with Couplet-TCN being close to random prediction with an nMCC of 0.5539 while Couplet-SoNNET has an nMCC of 0.6648. We find that Couplet-SoNNET performs better than Couplet-TCN, This result reveals Couplet-SoNNET is able to understand social signals better than a predictor that always feeds. This implies that it is possible to predict the behavior of a user using only their co-diner information, which indicates that there is social coordination in human-human commensality. These findings also suggest that social signals were captured by the interleaving structure of the SoNNET models.
7 Transferring from Human-Human to Human-Robot Commensality
Our objective to develop a bite timing strategy for robot that feeds a user in a social dining setting. We design a study where users evaluate the effect of different bite timing strategies on their overall social dining experience. To simulate robotic caregiving scenarios for people with upper- extremity mobility limitations, we instructed users to not move their upper body. This study was approved by Cornell's IRB.
We evaluate a learned bite timing strategy against two baseline bite timing strategies inspired by our conversations with care recipients, occupational therapists, and caregivers who told us how they know when to feed. The strategies are further depicted in Fig. 3:
Experimental Setup.
1. Learned Timing. This social, fully autonomous bite timing strategy feeds based on our Couplet- SoNNET model's output. We sample this model every three seconds with the last six seconds of preprocessed features at rate of 15 frames per second. This approach takes into account the social context. Since we want to evaluate the generalization performance, we train Couplet- SoNNET on 80% of the HHCD data and use the remaining 20% of HHCD data select early- stopping criteria. 2. Fixed-Interval Timing, This fully autonomous bite timing strategy feeds every 44.5 seconds, which is scaled average time robot should take to feed user after it has picked up a food item. To derive this value, we first find the appropriate scaling factor between human motion from the HHCD and robot motion. We note the average time for a human from the food entered transition to food lifted transition 9.9 seconds. The robot end-effector motion is not designed to match the human speed but rather to be perceived as safe and comfortable to user being fed. We find the equivalent key transitions for the robot to be 5cr slower than a human. Since we define the intention to take bite as when the food is lifted, the robot should take 49.5 seconds to feed user after picking up a food item. Given the robot takes roughly 5 seconds to move to its wait position after picking the food, the robot waits 44.5 seconds. Further details about this wait-time can be found in App. 8.3.2.
3. Mouth-Open Timing. This partially autonomous bite timing strategy feeds only when the user prompts the robot by opening their mouth. The target user is prompted each time by the robot saying "When ready, look at me and open your mouth". This approach gives the user explicit control of when the robot should feed [40].
The robot user is seated a wheelchair mounted with Kinova Gen3 6-DoF arm [62], which is used to feed the participant (Fig. 3, left). For discussion of our bite timing strategies, the use of voice prompting, and implementation details of the robot study, see App. 8.3.2-8.3.4.
Experimental Procedure. In this study, participants are seated in a similar setup as that used for HHCD data collection in Sec. 5. All participants were asked to bring their own food, and each group chose who would be fed by the robot. We recruited 30 participants over 10 sessions. There were 16 female and 14 male participants, and their ages ranged from 19-70 (u- 27, a = 9) years.
A single trial consists of bite acquisition, followed by one of the three bite timing strategies, then bite transfer. For bite acquisition, the robot alternates feeding the user cantaloupes and strawberries. We chose these fruits due to their high acquisition success rates [24]. We used the bite acquisition strategies and bite transfer strategies from [25, 26]. All participants take survey after each trial, which administers a forced-choice question on the participants' preferences between the previous and current conditions. Each pair of comparisons between any two conditions occurs three times, leading to ten trials. The condition orderings are counterbalanced over ten trials. Additionally, we ask participants whether they felt the robot fed them too early, on-time, or too late. The experiment questionnaire after each trial further includes questions about bite timing appropriateness, distrac- tions due to the robot, ability to have natural conversations, ability to feel comfortable around the
7

---

proforccc votes
100+
Learned
Learned Timing
Mouthl-ppu
Kinova Arm
Model
Fbond-Intervll
RGBD Camera
Walt 3 seconds food
G0H
Forco-Torque Sansor ('Forque')
Fixed- Intervall Timing
wait 44.5 seconds
Rovi Wheelchair
Open Timing
when ready
User
cordiners All thre
Lmaal distrunsion
User
Two co diaces
All three
Figure 3: Left: We use a 6-DoF Kinova Gen3 robotic arm [62] on Rovi wheelchair [63]. Middle: User study conditions/bite timing strategies: Learned, Fixed-Interval, and Mouth-Open Timings. Top right: Preferences for bite timing strategies rated by users, two co-diners, and all three diners. Bottom right: Level of distraction by the robot perceived by users, two co-diners, and all three diners on a Likert scale 1-5 (agreement with "I felt distracted by the robot"), for each bite timing strategy. *, **, *** denote statistically significant differences with po.0s- Po.005, Po.00os respectively.
robot, as well as system reliability and trust in the robot [64]. For details on user study question- naires see App. 8.3.6. To avoid interruptions in social conversations due to the presence of a robot in human groups, we provide the participants with a list of questions (see App. 8.3.5), which they could optionally use to help get the conversation started at each trial, similarly to previous work [39]. Results and Discussion. As shown in Fig. 3 (top right), users and co-diners preferred the Learned strategy for bite timing as compared to Fixed-Interval or Mouth-Open Timing. This confirms that our insight incorporate social signals in model structure (SoNNET) improves bite timing predic- tion. These results using Couplet-SoNNET also imply that it is possible to predict the behavior of a user using only their co-diner information, which indicates that there social coordination in human groups even in the presence of robot. In Fig. 3 (bottom right) we further compared the level of dis- traction by the robot as perceived by participants. We performed Kruskal- H-tests and Tukey HSD post-hoc tests and found that Mouth-Open Timing distracts dining participants significantly more than Learned or Fixed-Interval Timing. We believe this because the Mouth-Open strategy disrupt the rhythm of conversation. Even though
prompts the user using voice interface, which the participants had a clear preference for the Learned strategy when given a forced-choice, when asked to individually rate the conditions using a 5-point Likert scale, interestingly we could not find any statistically significant differences between Mouth-Open Timing and Learned Timing. This is probably because the Mouth- Open Timing strategy provides full control of bite timing to the users themselves. Note, regardless of the conditions, the users found the system confortable, reliable, and trustworthy. Detailed analysis is given in App. 8.3.7.
Limitations. There is a risk that our results from human-robol user studies on adults with no mobil- ity limitations may not generalize to those with people with mobility limitations. People with mo- bility limitations may have different preferences and cognitive workload associated with a robotic intervention. Although our target diner is not person with such C3-C5 SCI, our model does not use their movements to infer when to feed. As transferability is a function of their behavior, our experiments demonstrate good transferability across scenarios. We expect it to similarly transfer to users with C3-C5 SCI, though it remains to be investigated in future work. We also made mul- human-robot commensality
tiple assumptions when transferring our results from human-human scenarios. During buman-human commensality. the user was self-feeding whereas in human-robot commensality the user being fed. We also assumed that the addition of robot into human- human commensality scenario does not change the social dynamics of the diners significantly. Given these assumptions, it would be interesting to see how our models perform when trained on similar human-robot commensality scenarios. Finally, it is an open question as to how these models would perform with groups of different cultures. Social science literature on commensality studied the interplay between factors such as culture [65, 66], age [67], and social context [68] on how long cating takes and what people are eating. We are excited about the potential to study how the pres- ence of robot can alter the communal act of cating together across cultures. This motivates further investigation into numan-robor commensality, both from technical and societal perspectives.
8

---

Acknowledgments
The authors would like to thank Rajat Jenamani, Rishabh Madan, and Sidharth Vasudev for their help with setting up and running the robotics user study. This work was funded in part by the National Science Foundation IIS (#2132846). This work also in part sponsored by the Office of Naval Research (N00014-19-1-2299), Any opinions, findings, and conclusions or recommendations expressed in this material those of the author(s) and do not necessarily reflect the views of the Office of Naval Research.
References
[1] U. S. C. Bureau. Americans with disabilities: 2014. 2014.
[2] L.. Perry. Assisted feeding. Journal of advanced nursing, 62(5):511-5111 2008
[3] M. E. Mlinac and M. C. Feng. Assessment of activities of daily living, self-care, and independence. Archives of Clinical Neuropsychology. (36))556-116 2016.
[4] Agis living. http::/www. agis wawpa/s...///Dc/meee/////////
[5] Obi, 2018. https://mettobi.com/..noom Retriveee
Retrieved on 25th January, 2018].
[6] My spoon, 2018. http/////ooo/o/.oooooooooooooaaa Retrieved on 25th January, 2018].
2018. http://////////////////////////a Id=8& Ott.gyyy.BBBooiiiaaaaaaaaaaaaaea Retrieved on 25th January, 2018].
[7]
[8] Meal buddy, 2018. bttps://wwrr htrrr/////w/errrrrrrrrrrrrcaaaa
http://pwwppppaaaaaaaaaccaaaaatp Retrieved on 25th January, 2018J.
[9] Winsford feeder, 2018.
Retrieved 15th February, 2018].
[10] Bestic, 2019. https://ww..ceeeec.cooc///ssstss Retrieved 18th April, 2019].
Retrivee on
[11] The mealtime partner dining system, 2018. http://baalliippprttrrss.coooood Retrieved on 15th February, 2018].
[12] Neater eater robot, 2019, http:///wwr..aaoorrroooo//tttttr Retrieved on 18th April, 2019].
[13] Beeson automaddak feeder, 2019. https:
//abledata. acl. http:///we....eeetttaaaaaaaddaad Retrieved on 18th April, 2019].
[14] Bhattacharjee, Lee, H. Song, and S. Srinivasa. Towards robotic feeding: Role of haptics in fork-based food manipulation. IEEE Robotics and Automation Letters, 2019.
[15] A. Candeias, T. Rhodes, M. Marques, M. Veloso, al. Vision augmented robot feeding. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018.
[16] E. K. Gordon, S. Roychowdhury, T. Bhattacharjee, K. Jamieson, and S. S. Srinivasa. Leveraging post hoc context for faster learning in bandit settings with applications in robot-assisted feeding. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 10528-10534 IEEE, 2021.
[17] H. Higa, K. Kurisu, and H. Uehara. A vision-based assistive robotic arm for people with severe disabilities. Transactions on Machine Learning and Artificial Intelligence, 2(4): 12-23,, 2014.

---

[18] T. Bhattacharjee, M. E. Cabrera, A. Caspi, M. Cakmak, and S. S. Srinivasa. A community-centers design framework for robot-assisted feeding systems. In The 21st international ACM SIGACCESS conference on computers and accessibility, pages 482-494, 2019.
[19] A. Jardón, C. A. Monje, and C. Balaguer. Functional evaluation of asibot: A new approach on portable robotic system for disabled people. Applied Bionics and Biomechanics, 9(1):85 2012.
[20] Bhattacharjee, Lee, H. Song, and Srinivasa. Towards robotic feeding: Role of haptics in -oake food manipulation. JEEE Robotics and Automation Letters, 4(2): 1485-1492, 2019.
[21] R. M. Alqasemi, H. J. McCaffrey, K. D. Edwards, and R. V. Dubey. Wheelchair-mounted robotic arms: Analysis, evaluation and development. Proceedings, 2005 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pages 1164-1169. IEEE, 2005.
[22] Z. Bien, M.-J. Chung, P.-H. Chang, D.-S. Kwon, D.-J. Kim, J.-S. Han, J.-H. Kim, D.-H. Kim, H.-S. Park, S.-H. Kang, et al. Integration a rehabilitation robotic system (kares ii) with human-friendly, man-machine interaction units. Autonomous robots, 16(2):165-191, 2004.
[23] D. Park, Y. K. Kim, Z. M. Erickson, and C. Kemp. Towards assistive feeding with general-purpose mobile manipulator. arXiv preprint 1.Xiv:6605.079900 2016.
[24] R. Feng, Kim, Lee, E. Gordon, M. Schmittle, S. Kumar, T. Bhattacharjee, and S. S. Srinivasa. Robot-assisted feeding: Generalizing skewering strategies across food items plate. In The International Symposium of Robotics Research, pages 427-442. Springer, 2019.
[25] D. Gallenberger, T. Bhattacharjee, Y. Kim, and S. S. Srinivasa. Transfer depends on acquisition: Analyzing manipulation strategies for robotic feeding. 2019 14th ACM/IEEE International Conference on Hluman-Robul Interaction (HRI), pages 267-276. IEEE, 2019.
[26] Belkhale, E. K. Gordon, Y. Chen, S. Srinivasa, T. Bhattacharjee, and D. Sadigh. Balancing efficiency and comfort robot-assisted bite transfer. arXiv preprint vrXiv:2111.11400) 2021.
[27] G. Canal, G. Alenyå, and C. Torras. Personalization framework for adaptive robotic feeding assistance. In International conference on social robotics, pages 22-31. Springer, 2016.
[28] Rhodes and Veloso. Robot-driven trajectory improvement for feeding tasks. 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2991-2996. IEEE, 2018
[29] I. Naotunna, C. J. Perera, C. Sandaruwan, R. Gopura, and T. D. Lalitharatne. Meal assistance robots: review on current status, challenges and future directions. In 2015 IEEE/SICE International Symposium on System Inlegration (SII), pages 21 1-216. IEEE, 2015.
[30] E. Gordon, X. Meng, M. Barnes, Bhattacharjee, and S. S. Srinivasa. Learning from failures in robot-assisted feeding: Using online learning to develop manipulation strategies for bite acquisition. 2019.
[31] D. Park, Y. Hoshi, and C. C. Kemp. A multimodal anomaly detector for robot-assisted feeding using an lstm- variational autoencoder. IEEE Robotics and Automation Letters, 1(3):1544-1551, 2018.
[32] D. Park, H. Kim, Hoshi, Z. Erickson, A. Kapusta, and C. Kemp. multimodal execution monitor with anomaly classification for robot-assisted feeding. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5406-5413. IEEE, 2017.
[33] Sebo, Stoll, B. Scassellati, and Jung. Robots groups and teams: a literature review. Proceedings of the ACM on Human- Human-Compute Computer Interaction, 4(CSCW2): 2020.
10

---

[34] M. F. Jung, D. DiFranzo, S. Shen, B. Stoll, H. Claure, and A. Lawrence. Robot-assisted tower construction method study the impact of robot's allocation behavior on interpersonal dynamics and collaboration in groups. ACM Transactions on Human-Roud Roboot Interaction (THRI), 10(1):1-23, 2020.
[35] H. Jönsson, M. Michaud, and N. Neuman. What is commensality? a critical discussion of an expanding research field. International Journal of Environmental Research and Public Health, 18(12):6235, 2021.
[36] C. Spence. M. Mancini, and G. Huisman. Digital commensality: Eating and drinking in the company of technology. Frontiers in psychology, 10:2252, 2019.
[37] E. Athlin, A. Norberg, and K. Asplund. Caregivers' perceptions and interpretations of severely demented patients during feeding in task assignment system. Scandinavian Journal of Caring Sciences, 4(4); 147-164,, 1990.
[38] E. Athlin and A. Norberg. Interaction between the severely demented patient and his caregiver during feeding. Scandinavian Journal of Caring Sciences, 1(3-4): 17-123, 1987.
[39] L. V. Herlant. Algorithms, implementation, and studies on eating with shared control robot arm. PhD Dissertation, 2018. URL
http::/wwet cs. cmu wom.ueuu/aee. afau/ff//aa wttp:d//ff..a////as/////////////
pdf..
[40] T. Bhattacharjee, E. K. Gordon, Scalise, M. E. Cabrera, A. Caspi, M. Cakmak, and S. Srinivasa. Is more autonomy always better? exploring preferences of users with mobility impairments robot-assisted feeding. In 2020 15th ACM/IEEE International Conference on Human-Roke Interaction (HRI), pages 181- 190- IEEE, 2020.
[41] D. C. McFarlane. Comparison of four primary methods for coordinating the interruption of people in human-computer interaction. Human- Caaman-Computer Interaction, 17(1):63-11) 113, 2002.
[42] M. Czerwinski, E. Cutrell, and E. Horvitz. Instant messaging: Effects of relevance and timing. In People and computers XIV: Proceedings of HCI, volume 2, pages 71-76, 2000.
[43] M. Pielot, B. Cardoso, K. Katevas, J. Serrà, A. Matic, and N. Oliver. Beyond interruptibility: Predicting opportune moments to engage mobile phone users. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 1(3), Sept. 2017. doi: 10. 145/311333 URL
https://doi. http:///ooo.rog//////// 1145/3130955
[44] Y. Takemae, T. Ohno, I. Yoda, and S. Ozawa. Estimating interruptibility in the home for remote communication based audio-visual tracking. IPSJ Digital Courier, 3:125 133, 2007.
[45] S. Banerjee, A. Silva, K. Feigh, and S. Chernova. Effects of interruptiiliiy-wwarr robot behavior. arXiv preprint arXiv: 1804:0644430 2018
[46] T. Wu, N. Martelaro, S. Stent, J. Ortiz, and W. Ju. Learning when agents can talk to drivers using the inagt dataset and multisensor fusion. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 5(3):1-28, 2021.
[47] J. Wang, H. Xu, M. Narasimhan, and X. Wang. Multi-person 3d motion prediction with multi-range transformers. Advances in Neural Information Processing Systems, 34, 2021.
[48] P. Müller, M. X. Huang, X. Zhang, and A. Bulling. Robust eye contact detection in natural multi-person interactions using gaze and speaking behaviour. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications, pages 1-10, 2018
[49] H. Park, F. Jain, and Y. Sheikh. 3d social saliency from head-mounted cameras. Advances in Neural Information Processing Systems, 25, 2012.
[50] H. Joo, T. Simon, M. Cikara, and Y. Sheikh. Towards social artificial intelligence: Nonverbal social signal prediction a triadic interaction. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10873-10883, 2019.
11

---

[51] Z. Cao, G. Hidalgo, T. Simon, S.-E. Wei, and Y. Sheikh. Openpose: realtime multi-person 2d pose estimation using part affinity fields. arXiv preprint arXi::/812.08006 2018.
[52] B. Zhao, H. Lu, S. Chen, J. Liu, and D. Wu. Convolutional neural networks for time series classification. Journal of Systems Engineering and Electronics, 28(1):162-169 2017.
[53] J. Yang, M. N. Nguyen, P. P. San, X. Li, and S. Krishnaswamy. Deep convolutional neural networks on multichannel time series for human activity recognition. In Ijcai, volume 15, pages 3995-4001. Buenos Aires, Argentina, 2015.
[54] C.-I. Liu, W.-H. Hsaio, and Y.-C. Tu. Time series classification with multivariate convolutional neural network. IEEE Transactions Industrial Electronics, 66(6): 4788-4792 2018.
[55] RealSense. Introducing the Intel@ realsense depth camera D455, 2020. URL com/ capph--camma---455:
[56] B. Zuo. ReSpeaker Mic Array v2.0, 2018. URL attps://wiki: seeedstudio com com//Rsjeeetee......rrraarr/
Mic
[57] Wittenburg, H. Brugman, Russel, Klassmann, and H. Sloetjes. Elan: professional framework for multimodality research. In 5th International Conference on Language Resources and Evaluation (LREC 2006), pages 1556-1559, 2006.
[58] C. Lea, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks: A unified approach to action segmentation. European conference on computer vision, pages 47-54. Springer, 2016.
[59] C. Lea, M. D. Flynn, R. Vidal, A. Reiter, and G. D. Hager. Temporal convolutional networks for action segmentation and detection. In proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 156-165, 2017.
[60] Y. He and J. Zhao. Temporal convolutional networks for anomaly detection in time series. In Journal of Physics: Conference Series, volume 1213, page 042050. IOP Publishing, 2019.
[61] D. Chicco and G. Jurman. The advantages of the matthews correlation coefficient (mee) over fl score and accuracy in binary classification evaluation. BMC genomics, 21(1):1-13, 2020.
[62] Kinova gen3. https: Accessed: 2022-06-14
2022.
[63] Rovi wheelchair. httpa://aaw.rovimbblittt com/, 2022.
2022. Accessed: 2022-06-14. 14.
[64] J.-Y. Jian, A. M. Bisantz, and C. G. Drury. Foundations for empirically determined scale of trust in automated systems. International journal of cognitive ergonomics, 4(1):53-711 -3, 2000.
[65] G. Danesi. Commensality in french and german young adults: An ethnographic study. Hospitality & Sociery, 1(2):153-11) 172, 2012.
[66] C. Fischler. Commensality, society and culture. Social science information, 50(3-4):528-548, 2011.
[67] S. Biggs and Haapala. Intergenerational commensality: A critical discussion on non non-familial age groups cating together. International Journal of Environmental Research and Public Health, 18(15):7905, 2021.
[68] M. Morrison. Sharing food at home and school: perspectives on commensality. The Sociological Review, 44(4):648 -674, 1996.
[69] T. Fischer, H. J. Chang, and Y. Demiris. Rt-gene: Real-time eye gaze estimation in natural environments. In Proceedings of the European Conference on Computer Vision (ECCV), pages 334-352, 2018
12

---

[70] T. Fischer and Y. Demiris. Markerless perspective taking for humanoid robots in unconstrained environments. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 3309-3316. IEEE, 2016.
[71] ython interface to the WebRTC Voice Activity Detector. https://gii http://ggttt..../// wwt.ccoo//ii..eeeeco wttp///eeaeaaaawweteeoooeeeeeeee 2022. Accessed: 2022-06-21.
[72] Remy. Temporal convolutional networks for keras.
https://gi http//ttthhhu...o htt.s/hh...ccooooolppppeeeeaarar 2020.
[73] S. Sankaran, Derechin, and Christakis. Curmelo: The theory and practice of forced-choice approach to producing preference rankings. PloS one, 16(5):e025115, 2021.
[74] M. B. Mehrani and C. Peterson. Recency tendency: Responses to forced-choice questions. Applied Cognitive Psychology, 29(3):418 424,, 2015.
13


Reviews:
[{"review_id": "g_r4G_aglgr", "reviewer": "Reviewer_iJwW", "summary": "The paper provides a multi-modal human-human commensality Dataset (HHCD) containing 30 groups of triads eating together, with rich annotation of keyframes of activities (e.g. picking food, taking bites, etc). Based on this, the authors present a data-driven model to predict when a robot should feed a user during social dining scenarios based on bite history and features of only co-diners (e.g. facial features, gaze, etc). Their user studies show that their model is more natural and less interruptive than other baselines. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper is very well-motivated and investigates a very underexplored research area of robot assistive feeding in social scenarios. The problem domain is focused and well-defined: predict the right timing to feed the human. The collected dataset HHCD is relatively large-scale and contains diverse demographics and social cues, so it seems to be generally useful for people to further investigate the topic of commensality. The details of the data collection and annotation are well-documented and explained in the paper and in the appendix. On the model side, the authors proposed an interesting interleaving architecture to fuse features from co-diners; their method is also able to consistently beat a strong baseline (TCN) in different scenarios. The most interesting finding is that their model shows evidence that it’s possible to predict the behavior of a user using only their co-diner information, suggesting some level of social coordination. The paper overall is clearly written and easy to follow. \n\nHowever, the paper is not without shortcomings. In section 7, which is the ultimate test of deploying the model on a human-robot commensality scenario, the two alternative bite timing strategies raise some questions. For the fixed-interval timing, it’s unclear what the authors mean by “the equivalent key transitions for the robot to be 5x slower than a human” and why scaling the time by 5x is ideal. If the average time between food_lifted and food_entered is around 10 seconds, presumably the target user wants the robot arm to do the same, rather than waiting for 40+ seconds? More importantly, for mouth-open timing, it seems unnecessary for the target user to be prompted each time by the robot via the voice interface. It seems more natural for the user to be prompted only once at the beginning of the trial so that the user understands how things work. Going forward for every bite, the user should only need to look at the robot and open their mouth. As a result, it’s not fully convincing that the learned method is necessarily preferred over the other two baselines. Another interesting result that raises questions is the ablation study in Table 1. It seems that the model is mostly relying on the bite features from the target user alone to make predictions. It’s a bit unclear whether the gaze, head pose, body, and face features of the diners actually help the model.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper tackles a very interesting, underexplored domain and did a thorough job of collecting a large-scale dataset and conducting human subject experiments, the main result in Section 7 isn’t fully convincing. Studying and understanding robot-assisted commensality is an extremely important topic, but it’s still unclear whether predicting bite timing is the key and whether a learned method necessarily is preferred over a much simpler heuristics-based method or a user-triggered mechanism. However, I am happy to hear the authors’ responses and revise my rating/recommendation.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Here are a couple of concrete suggestions that might help further improve the paper.\n- If time allows, re-evaluate the Mouth-Open Timing method with only one-time prompting at the beginning.\n- Clarify why Fixed-Interval Timing needs a 5x scaling factor\n- Clarify what modalities the model relies on when making predictions. The model is clearly learning something meaningful (otherwise nMCC would be 0.5), so it’s a bit confusing why the nMCC only drops by a little in most ablations in Table 1. In the paper, it’s mentioned that Body & Face feature from OpenPose is important, but it’s unclear that is the case from Table 1. \n"}}, {"review_id": "ROUBZfAD38o", "reviewer": "Reviewer_gH2e", "summary": "In this paper, the authors build a multimodal Human-Human Commensality Dataset and use it to train a SOcial Nibbling NETwork (SoNNET) and use it on a robot-feeding platform for people with mobility limitations. The approach is evaluated both on the dataset and on a real robot with people taking part to the experiment.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and well structured\n* The dataset is potentially useful for several applications and is going to be released after acceptance\n* The problem is relevant\n* The approach is evaluated on the real robot\n\nWeaknesses:\n* It would be nice to have a clearer view of what existing approaches there are out there\n* While the approach and the problem are relevant, as you state in the limitations, this might not generalize to real people with mobility limitations (and maybe it also depends on the degree of such limitation)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is well written and well structured. It deals with a relevant problem that is not considered\nsufficiently in robotics, and it does deliver good results, including a publicly released dataset and\nseveral experiments that involve both real people and a real robotic platform. The paper has \nobviously some limitations, but I believe it has good quality as well.\n\nIt would be great if the authors could contextualize their work a little better within what's the\ncurrent status of the research in this field.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "* Related work", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "IPY5-HVEFie", "reviewer": "Reviewer_tBiN", "summary": "The paper proposes a method for autonomous assistive feeding, where a robotic arm is able to feed its patient.\nIn particular, they propose SOcial Nibbling NETwork (SoNNET), a model that predicts when a user has the intention to eat based on various social signals. Experiments include evaluation on their novel dataset with human participants participating in a social meal with 2 other people, in addition to a simplified robot evaluation with the robot feeding fruit to the human participant.\nOverall, the main contributions of the paper are the model which captures the subtle interpersonal social dynamics, in human-human and human-robot groups for predicting bite timing, a socially-aware robot-assisted feeding system, and a novel Human-Human Commensality Dataset (HHCD) containing multi-view RGBD video and directional audio recordings capturing 30 groups of three people sharing a meal.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The authors aim to address an important task, that of assistive feeding, that should it be successful has tremendous societal impact to help people with mobility challenges. \n- The ablation study is very interesting. Particularly, they note that \"users’ bite features such as the time since last bite and the number of bites are important, as accuracy drops drastically without them. Intuitively, we believe this feature is important because a user’s bite features are a proxy for their level of hunger.\" \n- The authors propose a very interesting solution, where they not only propose a new model but also collect a novel dataset and deploy their proposed methodology on a real robotic platform.\n- The paper is well-written and easy to understand and follow.\n- Overall, the paper contains a range of practical insights that will be useful for robot learning practitioners.\n\nWeaknesses\n- The authors seek to answer the following question: \"How should an assistive feeding robot decide the right timing for feeding a user in everchanging and dynamic social dining scenarios?\" In my opinion, from their proposed methodology, this hypothesis is not fully met. In this question, they key words are \"everchanging and dynamic social dining scenarios\". In their model training/evaluation, they carry out their experiments in an isolated room, without environment noise, without moving people in the background as you would expect in social dining scenarios. Hence, their environment is not really representative on what they would like to achieve. I understand that this could be due to facilitation of data collection, and it might also be representative of for example a feeding scenario in a nursing home, but I hope the authors would agree with me that in social scenarios there would be more external disturbances. \n- The authors base their model on predicting the timing of a user to take a bite of food based\non the social cues within the interaction. I am wondering if they have any thoughts on whether this would change in different cultures? For e.g. different cultures would have different societal norms (think Mediterranean vs Asian population).\n- The authors show that their approach achieves >70% accuracy w.r.t. the recorded dataset. However, they did not present any quantitative results in terms of accuracy w.r.t. the real robot experiment. Would you be able to quantify this for e.g. failure cases = the robot dropped the food, or the robot fed the person but the food fell out. Also, regarding failure cases, could you highlight some examples when it failed and why? \n- In the assessment, the authors point out that they understand the limitation of their work is the fact that the used healthy participants. I was wondering how confident they are that their model would work to people with mobility issues? In the video, it is evident that the healthy subject moves closer to the robotic arm for feeding and I am sure there might be cases where this is not possible for humans with severe mobility issues. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper provides an interesting pipeline for an application with high societal impact, but lacks generalisation across different foods and does not seem to take into account safety, for e.g. if the person moves forward and the robot hits the person. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- The abstract is slightly confusing. I would encourage the authors to start from motivating why their problem is interesting to the robotics community, what they did and finally how, i.e. the opening sentence of the abstract is too abrupt.\n- I am not sure it was necessary to say that the people in the collected dataset were fully-vaccinated.\n- I understand that for simplicity fruit were chosen, but what was the motivation for this? Also, did you explore other food and in that case what happened? I am also assuming that for the recorded dataset a vast array of food could be used and different cutlery as well so was the robot experiment simplified as a matter of safety? It would be helpful to the research community if you shed some light on your experience with different food/cutlery for this task.\n- How did you ensure overall safety with the robot? Could you elaborate further, maybe in the appendix, the control and planning method used on the kinova platform?\n- The authors mention the challenge with the dataset in terms of class imbalance. Did you explore strategies to mitigate this for e.g. a weighted loss to take this into account?\n\nI would also encourage the authors to address the weaknesses of the paper to improve it."}}]

Author rebuttal:

Decision: Accept (poster)