Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: In this paper, the authors build a multimodal Human-Human Commensality Dataset and use it to train a SOcial Nibbling NETwork (SoNNET) and use it on a robot-feeding platform for people with mobility limitations. The approach is evaluated both on the dataset and on a real robot with people taking part to the experiment.
strengths_and_weaknesses: Strengths:
* The paper is well written and well structured
* The dataset is potentially useful for several applications and is going to be released after acceptance
* The problem is relevant
* The approach is evaluated on the real robot

Weaknesses:
* It would be nice to have a clearer view of what existing approaches there are out there
* While the approach and the problem are relevant, as you state in the limitations, this might not generalize to real people with mobility limitations (and maybe it also depends on the degree of such limitation)
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is well written and well structured. It deals with a relevant problem that is not considered
sufficiently in robotics, and it does deliver good results, including a publicly released dataset and
several experiments that involve both real people and a real robotic platform. The paper has 
obviously some limitations, but I believe it has good quality as well.

It would be great if the authors could contextualize their work a little better within what's the
current status of the research in this field.
issues: * Related work
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a method for autonomous assistive feeding, where a robotic arm is able to feed its patient.
In particular, they propose SOcial Nibbling NETwork (SoNNET), a model that predicts when a user has the intention to eat based on various social signals. Experiments include evaluation on their novel dataset with human participants participating in a social meal with 2 other people, in addition to a simplified robot evaluation with the robot feeding fruit to the human participant.
Overall, the main contributions of the paper are the model which captures the subtle interpersonal social dynamics, in human-human and human-robot groups for predicting bite timing, a socially-aware robot-assisted feeding system, and a novel Human-Human Commensality Dataset (HHCD) containing multi-view RGBD video and directional audio recordings capturing 30 groups of three people sharing a meal.

strengths_and_weaknesses: Strengths
- The authors aim to address an important task, that of assistive feeding, that should it be successful has tremendous societal impact to help people with mobility challenges. 
- The ablation study is very interesting. Particularly, they note that "users’ bite features such as the time since last bite and the number of bites are important, as accuracy drops drastically without them. Intuitively, we believe this feature is important because a user’s bite features are a proxy for their level of hunger." 
- The authors propose a very interesting solution, where they not only propose a new model but also collect a novel dataset and deploy their proposed methodology on a real robotic platform.
- The paper is well-written and easy to understand and follow.
- Overall, the paper contains a range of practical insights that will be useful for robot learning practitioners.

Weaknesses
- The authors seek to answer the following question: "How should an assistive feeding robot decide the right timing for feeding a user in everchanging and dynamic social dining scenarios?" In my opinion, from their proposed methodology, this hypothesis is not fully met. In this question, they key words are "everchanging and dynamic social dining scenarios". In their model training/evaluation, they carry out their experiments in an isolated room, without environment noise, without moving people in the background as you would expect in social dining scenarios. Hence, their environment is not really representative on what they would like to achieve. I understand that this could be due to facilitation of data collection, and it might also be representative of for example a feeding scenario in a nursing home, but I hope the authors would agree with me that in social scenarios there would be more external disturbances. 
- The authors base their model on predicting the timing of a user to take a bite of food based
on the social cues within the interaction. I am wondering if they have any thoughts on whether this would change in different cultures? For e.g. different cultures would have different societal norms (think Mediterranean vs Asian population).
- The authors show that their approach achieves >70% accuracy w.r.t. the recorded dataset. However, they did not present any quantitative results in terms of accuracy w.r.t. the real robot experiment. Would you be able to quantify this for e.g. failure cases = the robot dropped the food, or the robot fed the person but the food fell out. Also, regarding failure cases, could you highlight some examples when it failed and why? 
- In the assessment, the authors point out that they understand the limitation of their work is the fact that the used healthy participants. I was wondering how confident they are that their model would work to people with mobility issues? In the video, it is evident that the healthy subject moves closer to the robotic arm for feeding and I am sure there might be cases where this is not possible for humans with severe mobility issues. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper provides an interesting pipeline for an application with high societal impact, but lacks generalisation across different foods and does not seem to take into account safety, for e.g. if the person moves forward and the robot hits the person. 
issues: - The abstract is slightly confusing. I would encourage the authors to start from motivating why their problem is interesting to the robotics community, what they did and finally how, i.e. the opening sentence of the abstract is too abrupt.
- I am not sure it was necessary to say that the people in the collected dataset were fully-vaccinated.
- I understand that for simplicity fruit were chosen, but what was the motivation for this? Also, did you explore other food and in that case what happened? I am also assuming that for the recorded dataset a vast array of food could be used and different cutlery as well so was the robot experiment simplified as a matter of safety? It would be helpful to the research community if you shed some light on your experience with different food/cutlery for this task.
- How did you ensure overall safety with the robot? Could you elaborate further, maybe in the appendix, the control and planning method used on the kinova platform?
- The authors mention the challenge with the dataset in terms of class imbalance. Did you explore strategies to mitigate this for e.g. a weighted loss to take this into account?

I would also encourage the authors to address the weaknesses of the paper to improve it.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper provides a multi-modal human-human commensality Dataset (HHCD) containing 30 groups of triads eating together, with rich annotation of keyframes of activities (e.g. picking food, taking bites, etc). Based on this, the authors present a data-driven model to predict when a robot should feed a user during social dining scenarios based on bite history and features of only co-diners (e.g. facial features, gaze, etc). Their user studies show that their model is more natural and less interruptive than other baselines. 
strengths_and_weaknesses: This paper is very well-motivated and investigates a very underexplored research area of robot assistive feeding in social scenarios. The problem domain is focused and well-defined: predict the right timing to feed the human. The collected dataset HHCD is relatively large-scale and contains diverse demographics and social cues, so it seems to be generally useful for people to further investigate the topic of commensality. The details of the data collection and annotation are well-documented and explained in the paper and in the appendix. On the model side, the authors proposed an interesting interleaving architecture to fuse features from co-diners; their method is also able to consistently beat a strong baseline (TCN) in different scenarios. The most interesting finding is that their model shows evidence that it’s possible to predict the behavior of a user using only their co-diner information, suggesting some level of social coordination. The paper overall is clearly written and easy to follow. 

However, the paper is not without shortcomings. In section 7, which is the ultimate test of deploying the model on a human-robot commensality scenario, the two alternative bite timing strategies raise some questions. For the fixed-interval timing, it’s unclear what the authors mean by “the equivalent key transitions for the robot to be 5x slower than a human” and why scaling the time by 5x is ideal. If the average time between food_lifted and food_entered is around 10 seconds, presumably the target user wants the robot arm to do the same, rather than waiting for 40+ seconds? More importantly, for mouth-open timing, it seems unnecessary for the target user to be prompted each time by the robot via the voice interface. It seems more natural for the user to be prompted only once at the beginning of the trial so that the user understands how things work. Going forward for every bite, the user should only need to look at the robot and open their mouth. As a result, it’s not fully convincing that the learned method is necessarily preferred over the other two baselines. Another interesting result that raises questions is the ablation study in Table 1. It seems that the model is mostly relying on the bite features from the target user alone to make predictions. It’s a bit unclear whether the gaze, head pose, body, and face features of the diners actually help the model.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although the paper tackles a very interesting, underexplored domain and did a thorough job of collecting a large-scale dataset and conducting human subject experiments, the main result in Section 7 isn’t fully convincing. Studying and understanding robot-assisted commensality is an extremely important topic, but it’s still unclear whether predicting bite timing is the key and whether a learned method necessarily is preferred over a much simpler heuristics-based method or a user-triggered mechanism. However, I am happy to hear the authors’ responses and revise my rating/recommendation.
issues: Here are a couple of concrete suggestions that might help further improve the paper.
- If time allows, re-evaluate the Mouth-Open Timing method with only one-time prompting at the beginning.
- Clarify why Fixed-Interval Timing needs a 5x scaling factor
- Clarify what modalities the model relies on when making predictions. The model is clearly learning something meaningful (otherwise nMCC would be 0.5), so it’s a bit confusing why the nMCC only drops by a little in most ablations in Table 1. In the paper, it’s mentioned that Body & Face feature from OpenPose is important, but it’s unclear that is the case from Table 1.
review_title: null

Meta-review: metareview: Post discussion update
---------------------------

Thank you to the authors for their thorough answers to my and the reviewers' questions. 

The rebuttal raises interesting points about the value of using co-diners social signals only, such as transferability and robustness. I still think a study with the target population is needed to properly gauge satisfaction (the only metric in the real world test, since there is no notion of accuracy, as the rebuttal points out). However, the rebuttal acknowledges this as well, and no paper can be fully complete, so I think in the end it's fine.

Like reviewer iJwW, I'm also not totally satisfied by the explanation for voice prompting in the open-mouth baseline. However, again, I don't think this issue sinks the paper.

Throughout the rebuttal, there is mention of working closely with key stakeholders like people with SCI and clinicians. I would love to see more information about that in the paper, including when those conversations led to certain decisions about the model's implementation or evaluation.

Overall, I believe the main points have been addressed. I'll note that the authors promised to update the following aspects of their paper:
- add more details about the value of using only co-diners' social signals in making the timing decision
- add more details about study design such as the scaling factor in the fixed-time heuristic
- clarify study design decisions for voice prompting in the open-mouth condition
- revise minor language like "hunger"-->"eating rate"


Pre-rebuttal review
----------------

This paper presents an algorithm for robotic eating assistance in social settings. Data about the bite timing from groups of humans was used to train a model that is then applied on a robot. The paper's contributions include a CNN-based architecture, a large hand-annotated dataset, and a user study with able-bodied adults demonstrating the bite timing algorithm. 

Strengths
* Reviewers all noted the value of the data set contribution. This work includes an impressively sized data set, with 151 hours annotations from 90 users in groups of 3. Clearly lots of effort was put into this data collection, and it's likely the data will be useable in the future.

* Reviewers agreed that the paper is well written.

* Reviewers agreed that the paper addresses an important research area. 

* Reviewers noted that the model worked well compared to strong baselines. The ablation studies revealed interesting information about human-human eating such as the importance of user bite features and the potential social coordination among eaters.


Weaknesses
* Reviewers point out that this approach has not been tested on users with motor impairments, and might not transfer well. Explaining why the results are likely to transfer would be very helpful for justifying the work.

* Reviewers had some critiques about the real-world study. For example, they wanted to know more about results about model accuracy in the real world. They were interested to understand the failure cases of the real world system. They also felt the baseline conditions did not present a fair comparison. Additional information here would strengthen the paper.

* In couplet-SoNNET, social signals from the user are almost entirely ignored. Why? Doesn't it matter, for example, that the user is talking, or looking away, or providing some other social signal that they're not ready for a bite?

* Also, please don't use "healthy" to refer to unimpaired people, as it sets up the notion that people with disabilities are "unhealthy."

* The paper says that bite features are a proxy for hunger. Is this really the case, or are they just a proxy for the rate at which people eat? Is there any evidence that hunger is what drives the bite features?
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)