Towards General Single-Utensil Food Acquisition with Human-Informed Actions
Authors: Ethan Kroll Gordon
Amal Nanavati
Ramya Challa
Bernie Hao Zhu
Taylor Annette Kessler Faulkner
Siddhartha Srinivasa
Venue: CoRL 2023 Poster (2023)
Keywords: Manipulation
Learning from Demonstration
Assistive Robotics

Abstract: Food acquisition with common general-purpose utensils is a necessary component of robot applications like in-home assistive feeding. Learning acquisition policies in this space is difficult in part because any model will need to contend with extensive state and actions spaces. Food is extremely diverse and generally difficult to simulate, and acquisition actions like skewers, scoops, wiggles, and twirls can be parameterized in myriad ways. However, food's visual diversity can belie a degree of physical homogeneity, and many foods allow flexibility in how they are acquired. Due to these facts, our key insight is that a small subset of actions is sufficient to acquire a wide variety of food items. In this work, we present a methodology for identifying such a subset from limited human trajectory data. We first develop an over-parameterized action space of robot acquisition trajectories that capture the variety of human food acquisition technique. By mapping human trajectories into this space and clustering, we construct a discrete set of 11 actions. We demonstrate that this set is capable of acquiring a variety of food items with $\geq80\%$ success rate, a rate that users have said is sufficient for in-home robot-assisted feeding. Furthermore, since this set is so small, we also show that we can use online learning to determine a sufficiently optimal action for a previously-unseen food item over the course of a single meal.

Towards General Single-Utensil Food Acquisition with Human-Informed Actions
Ethan K. Gordon" University of Washington ekgordon@cs uw. edu
Amal Nanavati" University of Washington amaln0ca uw edu
Ramya Challa Oregon State University
Bernie Hao Zhu University of Washington
Taylor A. Kessler Faulkner University Washington
Siddhartha S. Srinivasa University of Washington
Abstract: Food acquisition with common general-purposs. purpose utensils is necessary component of robot applications like in-home assistive feeding. Learning acquisi- tion policies in this space is difficult in part because any model will need to contend with extensive state and actions spaces. Food is extremely diverse and generally difficult to simulate, and acquisition actions like skewers, scoops, wiggles, and twirls can be parameterized in myriad ways. However, food's visual diversity can belie degree of physical homogeneity, and many foods allow flexibility in how they are acquired. Due to these facts, our key insight that small subset of actions is sufficient to acquire wide variety of food items. In this work, we present methodology for identifying such subset from limited human trajectory data. We first develop an wvrrpparameterized action space of robot acquisition trajectories that capture the variety of human food acquisition technique. By map- ping human trajectories into this space and clustering, we construct a discrete set of 11 actions. We demonstrate that this food items with 2 80% success rate, rate that users have said is sufficient for in-home robot-assisted feeding. Furthermore, since this set is small, we also show that we can use online learning to determine sufficiently optimal action for a previously-unseen food item over the course of single meal.
capable of acquiring variety of
Introduction
Eating is a fundamental part of the human experience, and robots can play an important role in facilitating the transfer of food from farm to kitchen to plate to mouth. But the prerequisite process of actually picking food up can be tricky. Food is fragile and can fall apart with excessive forces stick to both the robot and the environment. Food is visually diverse and hard simulate. In industrial utilize suction [1] soft enveloping
transport and packaging settings, specialized end-effectors links [2] to safely and reliably grasp variety of foods. However, for in-home food manipulation, such hardware can be unavailable. impractical due to limited space, or uncomfortable for humans to interact with. In particular, this work is motivated by the application of aobot-assisted feeding for those who cannot eat on their own (1.8 million people the United States alone [3]). In this setting, there strong coupling between the problems food acquisition and mouth transfer [4]. Therefore, we focus on the problem of acquiring food with common, single-piece utensils (i.e. fork or spoon) with which users are already familiar.
Formally, our goal is to learn map from food context (e.g., RGBD images of the food item) to sufficiently good utensil trajectory and control strategy to acquire the food. In the assisted feeding context, previous work [5] suggests that that "sufficiently good" is on the order of an 80% success rate, depending on the level of mobility impairment.
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Pork Orastation
Appraak
Liiie
Boeen Dutn Calletionn
Dalise
Ohbor Deetet
Apprssua
(A in
Brgen
Dirette Action Space
Doruin
Agynac
Salrsi
Unadd
Scham Magores
Leaktr Paity
Prase
Olorre Caue 18,
Arris Sviction
Claserin
Figure 1: (Left): Visual description of the action schema. Robot motions are in orange. Reference frames are represented as three-color axes with in red, Y in green, and Z in blue. (Right): General food acquisition pipeline. Human data is collected, mapped into the schema, and clustered into discrete action space. This space is small enough to treat food acquisition as a contextual bandit to learn online the optimal action for new food items.
Humans are generally experts food acquisition, so a natural approach to this problem is to use human dataset collected on variety of food items learn this map directly. This approach runs into sample complexity issues, food is hard to simulate, and it is difficult to capture the quantity of real world data necessary to cover the diversity of food and the diversity of possible ways to acquire it. Previous work [6, 7] got around this issue by using human data qualitative starting point for the manual design of a small set of actions with some success on limited set of food items.
Supported by this work and other work exploiting the haptic similarities between food items [8], key insight is that very small subset of the space of possible acquisition actions is sufficient to acquire almost all food items that human is capable of picking up with fork. We capture this subset in principled, data-driven way by observing a relatively small number of human acquisition trajectories on arbitrary food. By parameterizing these trajectories within an interpretable metric space, we can use off-the-shelf clustering to create an even smaller representative discrete space. We empirically show that this discrete space is simultaneously large enough to effectively cover the space of food contexts and small enough that online learning can identify the best action for multiple new food contexts the course of 30 minute meal.
In summary, this work presents three contributions in the field of robotic food acquisition. (1) In Section 3 we present the schema that defines an over-parameterized action space to capture human food acquisition techniques with a common single-piece utensil (i.e. fork or spoon). (2) In Section 4, we present dataset of human food acquisition trajectories on user-motivated variety of food items. (3) In Section 4.2, we describe both method for distilling discrete actions from human data and set of 11 actions constructed from our dataset. This method is tailored to in-home robot-assisted feeding, but we believe similar structure could be used for other applications where the key insight holds.
In Section we demonstrate that the discrete action set is sufficient to acquire a variety of food items that are visually dissimilar from those used in the human study. We demonstrate in Section that we can quickly use off-the-shelf online learning techniques to determine sufficiently optimal action for previously-unseen food items within 13 trials per item. Finally, Section discusses avenues for future work in both general food manipulation research and the application of rsbst-assisted feeding.
Related Work
2.1 Robot-Assisted Feeding: Food Manipulation
Robot-assisted feeding has been explored in industry and academia, yet still contains many unsolved problems. Commercial table-mounted systems [9, 10, 11, 12, 13, 14, 15, 16] available. but work with fixed trajectories for food acquisition. This can result in issues such as food being pushed off of

---

the plate, as found in study of the Bestic system [17]. There are also teleoperated options available, such as [18], but users can have difficulties using a fully teleoperated system [19].
Food acquisition a fundamental part of robotic feeding devices [20]. Some prior work creates specialized tools for food manipulation [1, 21, 22, 2]. We focus instead on food acquisition with fork, it is a common houschold utensil with which users are familiar. Other research that has focused on picking up food with forks uses limited set of skewering actions, which works well for small set of food items but has difficulties on more varied items [23, 24, 7]. Other work added variety with utensil swapping [25], which adds hardware complexity, but kept the set of food items and action trajectories small.
Further work in food acquisition uses vision or haptic data improve the choice of acquisition action. Vision can be used to classify visually different food items, and haptic data assist in identifying foods that look different but require similar actions (e.g. grapes and cherry tomatoes). Some prior works require additional "probing" actions for every food item [26, 27, 6] or specialized sensors beyond force torque sensors [28]. Other work uses haptic data to improve food acquisition during the feeding process, but the necessary for in-h applications [8, 29]. This paper leverages the learning approaches of past work with human-informed action space that is likely cover a wider variety of food items that users might want to cat.
action space does not cover the variety of food
2.2 Learning Grasps from Human Demonstrations
As humans are generally expert tool users, plethora of work has gone into transferring those skills to robots [30, 31, 32]. Some work focuses on higher-level task planning [33, 34]. Others learn more granular motions by generating dynamic motion primitives that a model learns to stitch together [35, 36). Still others investigate a hand-design restricted action space for use with end-to-end models [37]. In contrast, this work looks leveraging ioplication-specifi structure and human data to systematically restrict the action space prior to learning model.
Other works [38, 39, 40, 41, 42] utilize simulations or extensive human and environment data to learn offline RL. policies that yield good generalization performance at test time. In contrast, this work exploits peplication-speciffe structure and the simpler contextual-bandee setting to learn a simpler model that can be refined online without the need for simulation or large datasets.
Acquisition Action Schema
Previous work [6] qualitatively captured a taxonomy of human food acquisition techniques with fork. This taxonomy included skewering and scooping and highlighted the importance of the approach angle and in-food manipulation strategies (e.g.. wiggling for greater pressure or rotation for greater contact area). Yet, that work does not provide quantitative way to represent those motions so a robot can execute them.
Filling this gap, the following schema describes an acquisition action space that is narrow enough to distill these taxonomic elements but flexible enough to capture variants of those elements (e.g.. additional wiggling, or different approach angle). The action defined by 26 continuous parameters divided into three phases: approach, grasp, and extraction (Fig. 1, left).
3.1 Approach (Pre-Grasp)
This phase captures the fork tilt and and approach angle elements of the qualitative grasp taxonomy.
Define the world frame with an arbitrary origin and orientation such that -Z the direction of gravity. We assume the existence of a food manipulation target bounded by an ellipsoid which may possibly intersect a flat plane defined by table/plate/other surface parallel the X-Y plane (from here just referred to as the plate). The projection of this ellipsoid onto the plate is the bounding ellipse of the food. The food frame is the default reference frame in which all parameters
Frame Definitions:
3

---

are defined unless otherwise specified. The origin is defined as the center of the bounding ellipse. +Z aligned with that of the world frame and the X-axis is aligned with the major axis of the bounding ellipse.
The approach consists of the following 9 parameters: Fork orientation (3)(3), Approach polar and azimuthal angle ((0, 3 (T)), Target approach point within the food (R²) Force threshold (+R).
Parameters:
During implementation, the utensil begins an arbitrary distance from the food and moves in a straight line towards the target approach point until either that point reached or the force on the utensil exceeds the threshold. Note that the definition of the food frame introduces æ-rotation symmetry depending on which direction along the X-axis is +X. In this work, this symmetry is broken during the on-robot experiments based which approach direction is within the robot's workspace and easiest for the on-board planning algorithm.
Implementation:
3.2 Grasp
This phase captures the wiggling, twirling, and in-food scooping motions of the qualitative taxonomy.
Frame Definitions: Define the approach frame as the food frame rotated by the azimuthal angle of the approach direction. Define the utensil frame with an origin at the very tip of of the utensil (e.g.. between the middle two tines on a fork) such that +Z points along the handle of the utensil and the X-axis goes across the face of the utensil. In other words, the Euler angles in this frame correspond with roll (Z), pitch (X), and yaw (Y). Using this frame instead the food frame allows the approach and grasp to be parameterized independently. For example, approaching from the side instead of the front should not result in a grasp rotation yawing instead pitching the fork.
The grasp consists of the following 9 parameters: Angular velocity in utensil frame (RR'), Linear velocity in approach frame (R*), Duration (+R), Force and torque thresholds (+R x +R).
Parameters:
During implementation, the utensil will execute the provided velocities for the provided duration, cut short if the force or torque thresholds are reached.
Implementation:
3.3 Extraction
This phase captures any stabilizing rotations that take place after the food is on the fork.
Parameters: Similarly to grasp, the extraction consists of the following 7 parameters: Angular velocity in utensil frame (R2 x +R), Linear velocity in approach frame (R*), Duration (+R). While force and torque threshold can be introduced, it is rendered unnecessary in this work by requiring the extraction motion to move against gravity away from the plate.
Implementation: Extraction is implemented the same way "grasp" is.
4 Human Bite Acquisition Strategies
Although the 26 dimensional action schema is large, hypothesized that only specific points (acquisition actions) within this schema will actually be commonly used to acquire food items. To identify those points, we had able-bodied participants acquire a variety of food items and feed them to an actuated mouth. The food items were chosen based on the meals an end-user with C1 quadripledia' eats a week, and included bready items like bagels and pizza, heterogenous items like sandwiches rice and beans, gelatenous items like jello, and stringy items like noodles. As the users acquired the food, motion capture system captured fork motion, a force-torque sensor on the fork captured haptic factors, and RGB-D camera above the plate captured visual aspects of the food manipulation (Fig. 2a). More details about the data collection can be found in Appendix A.
'CI quadriplegia refers to paralysis of all four limbs as a result of an injury to the first, or top-most, cervical vertebrate.

---

Intell bense
Parteipant
RABD Camena
Ogtitaak Maries
Camera
Astuated Moath
Capture
Erpertmenter
Tock +
Puren/Tburuur Sensor
Shalla Adtoo Star
Food Plute
Orarra
ATI c000
Mona 2D
or terter
(a) Human food acquisition data collection setup.
(b) On-robot experiment setup.
Figure 2: Food acquisition trials. For each trial, single food item was acquired. Food perception (both center-of-mass and orientation) was performed with classical computer vision through fiducial and color-based background rejection.
4.1 Dataset
We published the data gathered here [43] to facilitate future research in food acquisition. This dataset consists 496 bite acqusion trials, totaling over 1.25 hours food acquisition data across participants and 13 unique food items.
4.2 Human Data Analysis
4.2.1 Extracting an Action Schema Point
For each bite's acquisition data, we extracted point within the acquisition action schema that was close that motion. We developed the procedure by iteratively extracting action schema points from bites, and then visualizing the actual participant's motion superimposed with the extracted action on a random subset of bites to determine how to improve the extraction procedure. Complete implementation details are covered Appendix B and our code repository?.
4.2.2 Clustering Actions
We ran k-medoids on the entire dataset of 407 extracted actions, where each action has 26 dimensions. Notably, the clustering did not consider aspects the action that would be outputted by the perception system, such as food reference frame, and only included the aspects of the acquísition motion that might generalize across food items. This resulted in k 11 actions (corresponding the within- cluster-sume -funooo--uuur----ee -iittreedsstanees elbow point). A full quantitative parameterization and video of each action are provided in the supplementary materials. Qualitatively, we observed emergent behavior consistent with findings in previous work [6], such as in-food wiggling, tilted extraction, and the use of vertical tines for high force.
4.2.3 From Human to Robot Actions
Although this procedure outputs representative actions for the motions that participants took when acquiring food items, it also learns some aspects of motion that are particular to the morphology of human arm. For example, participants' motions tended to approach food from the right, since they right-handed and feeding a person to their left. However, since the robot approaches food from above and feeds someone behind it (sitting in the wheelchair), it no longer needs to be constrained to right-to-left motion. As described in Section 3, the definition of the food frame is ambiguous with respect to a rotation about the axis of gravity. Since the food location was fixed, we manually broke this symmetry by choosing the orientation that was easiest for the on-board planning algorithms.
actpp: Ptttpp:///ithie //tithab. pars Loo / co http://rco...dddaaaaeeao

---

Further, some in-food grasp motions that humans executed, specifically tiny rotations of the fork (3 or less), produced negligible motion of the robot with significant planning and collision checking time; so we truncated those rotations to 0%.
5 Experiment 1: Action Evaluation
This experiment was designed to test the utility of the discrete 1 11-action space variety of food items. Previous work in assisted feeding suggests that, depending on level of mobility impairment, users can generally tolerate up to 20% failure rate in food acquisition [5]. Our hypothesis was two-fold: (1) Coveruge: for each food item, least one action would meet or outperform baseline performance and meet the 80% user threshold. (2) Minimal Bad Actions: each action would have acceptable performance
least type of food. If the action set lacks coverage, it is likely too small to adequately acquire new food items in the home, while if there are many bad actions, it is likely too big and will be difficult to use for online learning.
Our hardware setup is summarized in Fig. 2b, and details, including the hardware description, trial description, and success metric are fleshed out in Appendix C.
5.1 Experiment Design
We evaluated our action space 14 diverse food items. Some food items were identical to those used during the human acquisition data collection (Section 4): fries, broccoli, mashed potatoes, spinach mix, and jello (cut into ~ 1.5cm slices to obviate the need for cutting). Some food items had similar properties to those in the human data collection with different visual characteristics: powdered doughnut holes, white rice, white bread sandwich, and flat noodles. Finally, some food items were new: baby carrots, grapes, half-strawberries, banana slices, and kiwi slices.
The baseline action set consisted of 3 skewering techniques pulled from past fork-acquisition work [7]. Vertical skewer (VS) orients the handle of the fork to be orthogonal to the table and moves straight down applying up to 15N of force before moving straight back up. Tines vertical (TV) orients the tines of the fork to be orthogonal to the table and again applies 15N of force straight down before moving back up. Finally, tilted angle (TA) orients the handle of the fork 45 degrees off the table normal with the fork flat facing up and approaches the food that same angle, moving straight upwards after skewering.
For each food item, we perform 10 trials with each of the baseline and 11 human-informed actions for total of 14 actions x 14 food types x 10 = 1960 trials. At about minute per trial, data collection took about 33 hours.
5.2 Results
These results summarized in Fig. 3(Left). All error bars represent Wilson Binomial Proportion 95% Confidence Intervals (n 140 in aggregate, n 10 per food item). Overall, the best action for each food item from the human-informed set significantly outperforms both the best action from the baseline set and the the user-defined benchmark with a success rate of 94.6% (p < 0.05 necessarily by non-overlapping confidence intervals).
Coverage All food items except for spinach exhibited success rate of 90% or higher within the new action space, exceeding the 80% user benchmark. Single-leaf spinach, difficult to acquire due to its thinness, came close with a 70% success rate with the best -nforam-dde action. The nearly complete coverage suggests that this action space large enough to handle the variety of food items necessary for in-home deployment. Additionally, reducing k maintained a subset of the 11 medioids down to at least & = and coverage is achieved for k 2 8, as below that, the action space does not include the only action that covers jello.

---

actions exhibited good performance on at least one food item. Actions 0, 1, 2, 3, 6, 8, and 10 were the optimal action for spinach, carrot, banana, strawherry, potato, jello, and sandwich respectively. While not an optimal action for any food item, actions 4, 5, and 7 exhibited 2 70% success on at least one food item. The only exception was action 9. This action captured the "cutting" motion that humans used on the full, undivided jello cups. Therefore, only the side of the fork comes into contact with the food, making success less likely. That 10/11 actions exhibited good performance suggests that this action space is not excessively large.
Bad Actions Almost all
Baseline Comparison Carrots, grapes, strawberries, bananas, and broccoli exhibited good (2 80%) performance with the optimal baseline actions that were designed for them in previous work [4], with the human-informed actions performing as well or slightly better. Sandwich, fries, noodles, and rice exhibited 60 - 70% baseline performance, with insufficient in-food contact (e.g. not enough of the fork present inside of every sandwich layer) as the primary failure mode. These failures were remedied by the increased in-food grasp motion the human-informed actions. Finally, jello and spinach were completely impossible for the baseline actions acquire. Jello needed a significant rotation during extraction to prevent the heavy chunk from slipping off the fork. Spinach needed significant lateral force during the grasp phase to wedge the fork between the flat leaf and the plate. Finally, the human-informe actions were generally able to acquire a greater mass of rice (258mg vs. 212mg) and potato (890mg vs 5930mg) than TA, the only baseline action with any form of scooping-like like motion.
Experiment 2: Online Action Selection
As described in Section 1, we assert that online learning is a necessary component of any in-home robot food acquisition system to handle previously-unseen food items. This is supported by the extensive in-lab evaluation time required for the previous experiment. In this experiment, we evaluate the ability of an off ffftthe-sheff online learning procedure to identify sufficiently good human-informed actions for acquiring previously unseen food items. Our hypothesis is that, given that there are only 11 actions (and as few 4 can cover the space), such system should be able reach the user benchmark the order of 11 trials for each type of food. At about 1 minute per trial, in an assistive feeding context, this would happen well within the bounds of 20-30min meal.
6.1 Learning System
As in Experiment 1, this experiment constituted a series of trials. We cycled through all 14 food items, with 1 trial per item. A group of 14 trials, one per food item, constitutes a round. Data collection ceased once the success rate over the course of a round exceeded 90% (approaching the 94% optimal). For each trial, food orientation, manual perception, action execution, and success definition, are identical what is described in Section 5.1.
Our online learning procedure models food acquisition as contextual bandit [44, 23] with visual context and augmented with haptic post hoc context based on related work [8]. Specifically, each trial prior to action selection, we manually annotate a bounding box around the food item in the RGBD image. The cropped image is run through the SPANet [7] model to create a feature vector that constitutes the visual context. During the approach phase of action execution, we collect raw 6D force-torque data, isolate the period immediately following food contact using a z-foree threshold, and run the result through a HapticNet multi-layer perceptron model [6] to generate feature vector that constitutes the haptic context.
We assume linear relationship holds between both forms of context and the expected reward (i.e. the success rate) of each of the 11 actions. Therefore, we utilize LinUCB [45], which uses the data collected so far to assign an upper confidence bound to the reward of each action and optimistically selects the action that has the highest upper confidence bound. Previous work [8] suggests that, despite being collected after action selection, optimizing the linear model with the haptic context can decrease the time needed to converge to the optimal action.
7

---

Online Action Selection
Chrm Bosline
Doe Bonchmack
Succnse Iuse
0.5
0.25
quibed
Adll
Diee
brcenul
etrasberr
crape
pestato
danglaot
6
10 11
Round
Figure 3: (Left) Best action for each food item from both the baseline and our action spaces. The specific action is labeled above each bar. (Right) Acquisition success rate for each round of 14 trials (I per food item) using LinUCB. Error bars represent the 95% confidence interval.
6.2 Results
These results are summarized in Fig. 3(Right), where we plot the success rate across all 14 food items each round (with Wilson 95% Confidence Interval, n = 14). Since most human-informer actions perform well on most food items, we see that we get 50% success rate even in the first round. By round 8, the action performance is par with the user benchmark. And by round 13, have approached the expected optimal performance with this action space (i.e. the Average shown in Fig. 3). At one minute per trial, this suggests that this system can successfully learn an acceptable acquisition action for 2-4 new types food within the span of at 30min meal, assuming that all foods of a given type have similar success rate for each action. And that number is likely higher for foods with similar haptic properties. Most food items converged to a sufficiently good action (e.g. Action 1) with 5 rounds. The food items that took the longest to learn were the haptically "unusual" food items like jello, rice, noodles, and mashed potato, which exhibited partieularly poor performance on the skewering actions that worked well on the firmer food items.
7 Limitations and Discussion
In this work, we present methodology use human trajectory data to identify a subset of food acquisition actions that can acquire wide variety of food items for robot-assisted feeding applications. The 11 actions we distill from our publicly available dataset [43] are sufficient to pick up 14 food items including hard carrots, soft bananas, slippery jello, compound sandwiches, and continuous mashed potatoes. And the set so small that we can reasonably expect to determine the optimal action for 2-4 food items over the course of a 30-minute meal.
A major avenue for future work is evaluation in a real in-home context. The foods selected for both the human data collection and on-robot experiments were motivated by surveying a participant and co-designer with mobility impairments about their eating habits. We believe they cover a wide variety of theological contexts, but there may still exist food types that are sufficiently distinct as to be covered by the actions presented here. Future work in the home can help identify such foods. These can possibly be addressed with online action space expansion. For example, the caregiver can provide some kinesthetic demonstrations that be mapped into the action schema and averaged. Additional work can investigate the coupling between these actions and the bite transfer process [4].
One significant hurdle to any future in-home experiments is food perception, a prerequisite for any long-term in-home deployment. this work, food localization was done manually by clicking on the food center. Autonomous food perception is currently being investigated utilizing general segmentation models [46] and could introduce errors that lower the overall success rate.
In general, the interaction between sources of error (e.g. perception, calibration, handling arbitrary food orientation, environmental hazards like obstacles or a non-level table) and optimal action selection is a key realm for future investigation. Since many foods were well-covered by multiple actions, is possible that the current action space will robust to such disturbances, but further study is needed. Overall, this work represents a step towards in-home general food manipulation in both feeding and food preparation contexts, and we hope that the provided method, data, and actions can help enable in-home experimentation in the near future.
8

---

Acknowledgments
This work was (partially) funded by the NSF GRFP (DGE-1762114)) National Science Foundation NRI (#2132848) and CHS (#2007011), the Office of Naval Research (#N0001441T711-211--P0000 and #2022-016-01 UW), and Amazon.
References
[1] A. Owen-Hill. Food www.-rrbotswwth-eeee
How
Robots
With
Vacuum
Grippers
Improve
Safety, 2019.
baaloogroboti..com wwwp-avaanuuuueeeeeeeeeeeee grippers improve food-
URL bttps://blog/
[2] Z. Wang, Y. Torigoe, and S. Hirai. A Prestressed Soft Gripper: Design, Modeling, Fabrication, and Tests for Food Handling. IEEE Robolics and Automation Letters, 1999:190,,, Oct. 2017. ISSN 2377-3766. 1o1110.11001RRA227::1114444 Conference Name: IEEE Robotics and Automation Letters.
[3] K. A. Theis, A. Steinweg, C. G. Helmick, E. Courtney-LLong, J. A, Bolen, and R. Lee. Which one? What kind? How many? Types, causes, and prevalence of disability among US adults. Disability and health journal, 12(3):411 421, 2019.
[4] D. Gallenberger, Bhattacharjee, Y. Kim, and S. Srinivasa. Transfer depends on acquisition: Analyzing manipulation strategies for robotic feeding. In 2019 14th ACM/IEEE International Conference on Eluman-Robor Interaction (HRI), pages 267-276. IEEE, 2019.
[5] Bhattacharjee, E. K. Gordon, R. Scalise, M. E. Cabrera, A. Caspi, Cakmak, and S. Srinivasa. more autonomy always better? exploring preferences of users with mobility impairments in robot-assisted feeding. In 2020 /5th ACM/IEEE International Conference Hamann-Robot Interaction (HRI), pages 181-190. IEEE, 2020.
[6] T. Bhattacharjee, G. Lee, H. Song, and S. Srinivasa. Towards Robotic Feeding: Role of Haptics in Fork-based Food Manipulation. IEEE Robotics and Automation Letters, 2019.
[7] R. Feng, Y. Kim, G. Lee, E. K. Gordon, M. Schmittle, S. Kumar, T. Bhattacharjee, and S. Srinivasa. Robot-Assisted Feeding: Generalizing Skewering Strategies Across Food Items Plate. In The International Symposium of Robotics Research, pages 427-442. Springer, 2019.
[8] E. K. Gordon, S. Roychowdhury, T. Bhattacharjee, K. Jamieson, and S. Srinivasa. Leveraging post hoc context for faster learning in bandit settings with applications in robot-assisted feeding. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10528-
10535. IEEE, 2021.
[9] Meal Buddy, Meal buddy, 2018. ht:ps://ww..peerfrmeeeeeeeeettte meal-
Retrieved on 25th January, 2018].
[10] Mealtime Partner. The mealtime partner dining system description, 2020, URL https:// www. www.mmalammmaaateeeemm otgpa//sccccccoooooaaaaaaaaaaaaa htm.
[11] MySpoon, 2018. URL http://ssswwww...ooooooo///////s
html.
[12] Meet Obi. Meet obi, Aug 2022. URL httpststs///stttobii
[13] R. Reviews. Winsford self-feeder, Apr 2013, http:/03......tpppe/////potttbbt
v=vYG2dVuPmos amp; wtt.g:/ww...cccaallaeaaaaaaareee
[14] Beeson. Beeson automaddak feeder, Feb 2004. URL https: /tte:///laa http//aaaaeereooeeeee/
networkofcare org/ http://gin//asssssvvee/e//// wttp:iistsiieeeeaalllleee wttp::alaappxxxd1111111111111111 &amp;
cid-797 aamp; en- we-FFFeeedesappppemm org..

---

[15] Bestic. Bestic, May 2021.
URL
2et22002200433//htggg wwww. camanic o. wttp://oo/.o.co..oeeeee comm/ts//prodduct//bettttt
[16] Neater Eater.
Neater eater robotic, 2022.
URL
weater-eatter-roott..
[17] N. C. M. Nickelsen. Imagining and tinkering with assistive robotics in care for the disabled. Paladyn, Journal of Behavioral Robotics, 10(1):128 111128199,,,00 2019.
[18] A. Campeau-Lecours, V. Maheu, S. Lepage, H. Lamontagne, S. Latour, L. Paquet, and N. Hardie. Jaco assistive robotic device: Empowering people with disabilities through innovative algo- rithms. Rehabilitation Engineering and Assistive Technology Society of North America (RESNA), 2016.
[19] C.-S. Chung, H. Wang, D. Ding, and R. A. Cooper. Feasibility analysis of daily activities using assistive robotic manipulators. In Proc. RESNA Annu. Conf., pages 1-9, 2016.
[20] P. Chua, T. Isccnee, and D. Caldwell. Robotic manipulation of food products-a review. Industrial Robot: An International Journal, 30(4):345-35, 2003.
[21] 7. Wang, R. Kanegae, and S. Hirai. Circular shell gripper for handling food products. Soft robotics, 8(5):542-554, 2021.
[22] G. Endo and N. Otomo. Development of food handling gripper considering appetizing presentation. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pages 4901-4906. IEEE, 2016.
[23] E. K. Gordon, X. Meng, M. Barnes, T. Bhattacharjee, and S. Srinivasa. Adaptive Robot- Assisted Feeding: An Online Learning Framework for Acquiring Previously-nnseen Food Items. In JEEE/RSJ International Conference Intelligent Robots and Systems, 2020.
[24] L. V. Herlant. Algorithms, Implementation, and Studies on Eating with Shared Con- trol Robot Arm. PhD thesis, Carnegie Mellon University, Mar. 2018. URL https: / //kilthub. cmu. edu http///aicllleaaaaa/////rrrriria and_Studies _nr________________________ Fatinnnniiiii htahanistttt___aare___ee Sechrrr__ootrol_ B A711A/4///6
Implementation. iom
[25] D. Park, Y. Hoshi, H. P. Mahajan, H. K. Kim, Z. Erickson, W. A. Rogers, and C. Kemp. Active robot-assisted feeding with general-purpose mobile manipulator: Design, evaluation, and lessons learned. Robotics and Autonomous Systems, 124: 2020.
[26] A. Sawhney, Lee, Zhang, M. Veloso, and O. Kroemer. Playing with food: Learning food item representations through interactive exploration. In Experimental Robotics: The 17th International Symposium, pages 309-322. Springer, 2021.
[27] M. C. Gemici and A. Saxena. Learning haptic representation for manipulating deformable food objects. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 638-645, Sept. 2014.
[28] A. Yamaguchi and C. G. Atkeson. Combining finger vision and optical tactile sensing: Reducing and handling errors while cutting vegetables. In Humanoid Robots (Humanoids), 2016 IEEE- RAS 16th International Conference on, pages 1045-1051. IEEE, 2016.
[29] P. Sundaresan, S. Belkhale, and D. Sadigh. Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding, Nov. 2022. arXiv:2211 1.6448 [cs].
[30] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-driven grasp synthesis survey. JEEE Transactions on robotics, 30(2):289-309, 2013.
[31] C. Wang, X. Zhang, X. Zang, Y. Liu, G. Ding, W. Yin, and J. Zhao. Feature sensing and robotic grasping of objects with uncertain information: A review. Sensors, 20(13):3707, 2020. 2020.
10

---

[32] P. K. Donepudi. Reinforcement learning for robotic grasping and manipulation: a review. Asia Pacific Journal of Energy and Environment, 7(2)699-78, 2020.
[33] N. Wake, R. Arakawa, I. Yanokura, I. Kiyokawa, Sasabuchi, Takamatsu, and K. Ikeuchi. Framework: One-Shot Robot Teaching for Grasp-Manipulation- Release Household Operations. In 2021 IEEE/SICE International Symposium System Integration (SII), pages 461-466 2021. doi: 10.11000/EEEE000/995550000001
A Learning-fromm
[34] C.Pérez-D Arpino and J. A. Shah. C-learn: Learning geometric constraints from demonstrations for multi-step manipulation in shared autonomy, 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 4058 4065. IEEE, 2017.
[35] R. Mao, Y. Yang, C. Fermüller, Y. Aloimonos, and J. S. Baras. Learning hand movements from markerless demonstrations for humanoid tasks. In 2014 IEEE-RAS International Conference on Humanoid Robots, pages 938-943. IEEE, 2014.
[36] B. Akgun, M. Cakmak, K. Jiang, and A. L. Thomaz. Keyframe-based learning from demonstra- tion: Method and evaluation. International Journal of Social Robotics, 4:343-355, 2012.
[37] L. Y. Chen, H. Huang, E. Novoseller, D. Seita, J. Ichnowski, M. Laskey, R. Cheng, T. Kollar, and K. Goldberg. Efficiently learning single-arm fling motions to smooth garments, 2022.
[38] S. Song, A. Zeng, J. Lee, and T. Funkhouser. Grasping in the Wild: Learning 6DoF Closed- Loop Grasping From Low-Cost Demonstrations. IEEE Robotics and Automation Letters, 5(3); 4978 4988, 2020. doi:10.1 2.0.0.RA..000004477555
[39] Y. Yang, Y. Li, C. Fermuller, and Y. Aloimonos. Robot learning manipulation action plans by "watching" unconstrained videos from the world wide web. In Proceedings of the AAAI conference on artificial intelligence, volume (9),,, 2015.
[40] P. Mandikal and K. Grauman. Dex VIP: Learning Dexterous Grasping with Human Hand Pose Priors from Video. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the Sth Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 651-661. PMLR, 08-11 Nov 2022.
[41] A. Yamaguchi, C. Atkeson, S. Nickum, and T. Ogasawara. Learning pouring skills from demonstration and practice. In 2014 TEEE-RAS International Conference Humanoid Robots, pages 908-908 905- IEEE, 2014.
[42] N. Ettehadi and A. Behal. Implementation of feeding task via learning from demonstration. In 2018 Second IEFE International Conference on Robotic Computing (IRC), pages 274-277. IEEE, 2018.
[43] A. Nanavati, R. Challa, E. K. Gordon, and S. S. Srinivasa. A Dataset of Food Manipulation Strategies for Diverse Foods, 2022. URL https::/ wrt/oorggo/ooooo 10.
[44] A. Bietti, A. Agarwal, and Langford. A Contextual Bandit Bake-off. arXiv preprint 1802.04064 Feb. 2018.
[45] C. Wei, L. Li, L. Reyzin, and R. E. Schapire. Contextual bandits with linear payoff functions. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, pages 208-2144 2011.
[46] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, P. Dollár, and R. Girshick. Segment anything, 2023.
[47] A. Nanavati, P. Alves-Oliveira T. Schrenk, E, K. Gordon, M. Cakmak, and S. S. Srinivasa. Design Principles for Robot-Assiste Feeding in Social Contexts. In 2023 18th ACM/IEEE International Conference Himan-Robou Interaction (HRI), 2023.
11

---

[48] K. JACO, 2018. robot C-armseriess
[49] A.-I F.T. Sensor, httpps//www..tti-la.cco//ppppppp aspx?id=Nandh
[50] E. Olson. AprilTag: A robust and flexible visual fiducial system. In 2011 IEEE International Conference on Robotics and Automation, pages 3400-3407, 2011. 10t10.110/1CRA.201111 5979561.
[51] T. Bhattacharjee, M. E. Cabrera, A. Caspi, M. Cakmak, and S. S. Srinivasa. A community- centered design framework for robot-assisted feeding systems. In The s international ACM SIGACCESS conference on computers and accessibility, pages 482-494, 2019.
A Human Data Collection
We ran a user study to identify which actions within this dimensional schema are effective for acquiring a diverse food items.
A.1
Study Design
The study involved participants acquiring bite-sized pieces of variety of food items with fork and feeding them to actuated mouth. The choice of food items was informed by ongoing collaboration with an end-user with C1 quadripledia', who had his caregiver take pictures of all the meals he ate in a week. One researcher then grouped similar food items (e.g., bread bun and bagel), resulting in final set of 13 diverse food items: bagel chunks, mini sub sandwiches, pizza, chicken tenders, fries, broccoli, glazed doughnut holes, mashed potatoes, lettuce, spinach mix, whole jello, instant ramen noodles, and brown rice with beans. The bagels, sub sandwich, and pizza were pre-cut into bite-sized chunks, building off of past research that found that users okay with caregivers cutting their food into bites before the robot feeds them [47]. The same brand ingredients and preparation procedure were followed for every food item.
The study space consisted of a table with plate food on it, a fork near the plate, a chair for the participant sit in, and an actuated mouth the left of the chairt. An RGB-D camera" above the plate captured visual aspects of the food. The fork, table, RGB-D camera, and actuated mouth all had motion capture markers them, which were tracked by motion capture system" in front of the table. The fork also actuated with force-torque sensor" to measure haptic aspects of the participant's 's acquisition action. An experimenter sitting behind the motion capture system, in full view of the participant, oversaw data collection. Fig. 2a shows the study setup.
When each participant arrived, they were first briefed on the study and given time to read and fill out consent form. They were then given a chance to familiarize themselves with the fork by feeding baby carrots to the actuated mouth. When participants were ready, the actual data collection began, where they were provided a plate with one of the 13 food items, in randomized order, and were asked to feed one bite at time to the actuated mouth. For each bite, participants were first asked to hold the fork in a comfortable "ready" position above the plate. When the experimented said "start," they lowered the fork to acquire the food item, moved it to the actuated mouth, and held
*CI quadriplegia refers to paralysis of all four limbs as result of an injury to the first, or top-most, cervical vertebrate.
Because all participants happened to be right-handed, this positioning allowed them easy access to feed the actuated mouth.
$ Intel RealSense D415
OptiTTaak V120 Trio
ATI Industrial Automation 6DOF Nano25 with Net F/T Interface
Duee to the force-torque sensor and motion capture dots, the fork was a different shape and weight from regular forks
12

---

there until the experimenter said "stop." For each plate of food, participants were asked to feed at least 5 bites, and possibly more the motion capture system lost tracking of the fork during the bite. In total, each study session took one hour and participants were compensated with a $10 gift card. Three researchers ran the study, and the study procedure was approved by our university's IRB. We had participants, who all happened be right-handed.
A.2 Dataset and Qualitative Observations
For each bite the participant acquired, our time series data consisted of the fork pose, force-torque sensor readings, and RGB-D images the plate. We first cleaned this data by removing mistrials (i.e., trials with missing or corrupted data) and transforming all poses to uniform frame reference. We then published this dataset [43] facilitate future research in food acquisition strategies. This dataset consists of 496 trials, totaling over 1.25 hours of food acquisition data across 13 food items and 9 participants.
Similarly to previous work [6], we observed some patterns in user interaction during data collection.
Different participants held the fork differently (thumb in front of or behind the fork), which gave rise to different acquisition actions;
Some food items could be acquired with multiple types of actions, e.g., some users scooped noodles whereas others twirled them;
Users often tilted the fork while putting downwards pressure on it, in order to get it to pierce the food (c.g., broccoli).
Some of these lead to emergent behaviors identified in Section 4.2.2
B Action Schema Point Extraction from Human Data
Exclusion Criteria We excluded any trial where tracking of the fork tip was lost for more than 0.5 seconds, where the motion capture system did read the stationary object poses (e.g., table, mouth) for the entire trial. After exclusions, had total of 410 trials.
Pre-processing To remove noise from the motion capture system, we smoothed all the fork tip poses by applying a median filter of 0.33 seconds separately the y, roll, pitch, and yaw of the pose.
Significant Timestamps We first extracted the significant timestamps of the user's acquisition action. Specifically, we defined the contact time the first timestamp when the distance from the fork tip to the camera exceeded the distance from the pixel corresponding to the fork tip to the camera in the initial depth image of the plate (i.c., the first time the fork pierced the surface of food on the plate). We then worked backwards from contact time, defining start time as the end of the 0.5 see interval where the fork tip was consistently within a sphere of radius em, was more than 5 cm from its lowest point, and was more then 35 em from the mouth. This criteria was based on our experimental design, where we asked the participant hold the fork stationary at a "ready" position before they acquired the food. In the event of multiple such periods where the fork was held stationary, we chose the one where the fork was the highest. We then defined end time to be the last timestamp when the fork was within cm from its lowest point. And finally, we worked backwards from end time, defining extraction time to be the latest time when the fork was 1 cm away from its lowest point, within 2 sec of the end time.
All-in-all, the motion the participant took between start time and contact time corresponds to their pre-grasp motion, the motion they took between contact time and extraction time corresponds to their grasp motion, and the motion they took between extraction time and end time corresponds to their extraction motion.
13

---

Food Reference Frame Although robot will know the food it is targeting before beginning its motion, with human data we have to extract the food item they were targeting from the data. We did so by segmenting the visually separate food items in the first RGB image of the plate of food. First, we detected the plate by: (a) using inpainting to remove glare; (b) using k-means clustering (k=3) to simplify the colors; and (c) finding the largest contour of the image that has least 50% blue pixels. In practice, this reliably detected the plate for every food item in our study. We then masked out all the non-plate pixels from the de-glared image, and detected the food bounding box by: (a) running k-means (with k=2 for most food items, and k=3 for broccoli since its colors were closest to blue) to simplify colors; (b) masking out all the blue colors; (c) narrowing the mask to separate touching food items; (d) computing contours; and (e) fitting rotated rectangles to every contour with an area between within a hardcoded range. In practice, this reliably segmented separate food items, like bagel pieces or chicken tenders. For food items with lot of overlap, like fries, this approach sometimes segmented multiple pieces of fries the same. However, since participants often also acquired multiple overlapping pieces of those food items, accepted those slight errors in food detection. For foods that weren't 't separated into bites like noodles or mashed potatoes, this algorithm rightly segmented it as one contiguous chunk of food.
Once we segmented separate bites of food, we defined the food reference frame to be centered at the center of the bounding box the fork tip was in contact time, rotated align with its major axis (i.e the center of the bite the user selected.)
Pre-Grasp The above preliminaries enable straightforward extraction of the pre-grasp, grasp, and extraction components of the action schema. For pre-grasp, we computed the target offset as the the target offset as the fork position at contact time in the food reference frame. We computed the initial utensil transform by taking the fork's linear velocity during a 0.5 second window before contact time and extrapolating that backwards 0.1 m, with fixed orientation. And we took the force threshold to be 50% of the max force between start time and contact time.
Grasp We defined the in-food twist to be the transformation between the fork pose at extraction time and contact time, and the duration of the twist to be the duration between extraction time and contact time, We defined the force and torque thresholds to be 50% of the max force and torque between contact time and extraction time.
Extraction We defined the out-of-food twist to be the transformation between the fork pose at end time and extraction time, and the duration of the twist be the duration between end time and extraction time.
Cleaning Three trials resulted in values of NaN or inf for at least one of the dimensions of the action schema. We eliminated these, resulting in 407 actions used for clustering.
C Experiment Setup Details
Our experimental setup is shown in Fig. 2b and was performed with a 6 DoF JACO2 robotic arm [48] with 3D-printed handle and fork-shaped end-effector. To implement the force/torque thresholding, we instrumented the fork with a 6-axis ATT Nano25 Force-Torque sensor [49]. The center of the food (and the bounding box used for visual context in the online learning experiment described in Section 6) were annotated manually from the robot's eye-in-hand vision system. This was done in order to run controlled experiment specifically focused on food acquisition, as opposed to introducing additional variance with a (possibly imperfect) food perception system. This system includes the Intel RealSense D415 RGBD camera and the NVidia Jetson Nano for wireless image transmission. Food was placed on a plate equipped with an AprilTag [50] for camera calibration and mounted on an anti-slip mat commonly found assisted living facilities [51].
In each trial, the food placed in the vicinity of the AprilTag the plate and oriented such that the major axis of the bounding ellipse is parallel to the bottom edge of the fiducial. The end-effector
14

---

moved to fixed position above the plate, and the location of the center of the food is annotated manually the RGBD camera image. After action execution, we wait at least 38 before recording success or failure. For most food items, success is defined as the entire item being removed from the plate. a homogeneous food item breaks common occurrence with banana slices), at least half of the item needs to end up on the fork. For the sandwich, success required that all layers (both pieces of bread, the lettuce, and the cheese) make it off the plate. Finally, for multi-piece and continuous items (i.e. potatoes, rice, noodles), conservative success metric was set at 200mg (~155 grains of rice, or I full noodle).
15


Reviews:
[{"review_id": "e174BhBxC_a", "reviewer": "Reviewer_W1Qt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work introduces a method for extracting a set of food acquisition primitives from human demonstration data, and shows that such a set of food acquisition actions is sufficient for acquiring a large variety of realistic real world food items. In a setting where it is assumed that there exists a single action primitive that is sufficient and optimal for acquiring a target food item, this work aims to learn the set of possible action primitives. The contributions have three components: 1) an over-parameterized 26-D action space for capturing human food acquisition techniques with three stages: approach, grasp, and extraction, 2) a 1.25 hours of food acquisition dataset from human participants, 3) a method for extracting a smaller number of food acquisition primitives from such a dataset. This work compares 11 extracted actions from the human dataset with prior works that use 3 primitives. The data-driven action space can perform well and even generalize to unseen object types.", "summary_of_recommendation": "This paper is well motivated, technically sound, and introduces non-trivial advances compared to prior works in robot-assisted feeding. Namely, expanding the flexibility and diversity of action primitives is clearly motivated from advances in other parts of robotics, and the empirical results in this work are strong. I particularly am impressed by the real world motivation with user studies in various aspects of this work, resulting in a work that will be significant within the robot-assisted feeding domain. In addition, the presentation is polished. My main considerations would be to explain the technical contribution a bit more. \n\n[Post-Rebuttal Update]\nThe reviewers have addressed my questions and I maintain my positive rating of Weak Accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "My main questions could be clarifications regarding the weaknesses above. In addition, some situation of this work with skill learning robotics papers would be helpful.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The method is very performant compared to prior art and seems tractable to scale to novel food assisted feeding scenarios\n- The overall integrated system is impressive and much closer to deployable real world impact compared to other CoRL works.\n- This work incorporates many practical motivations from realistic use cases, such as sampling training/test objects from assisted-feeding patients, extracting actions from real human feeding as opposed to robot teleoperation, and using data from in-home robot-assisted feeding user studies. This will make the work more directly applicable and impactful for practitioners.\n- The presentation is polished and very easy to understand the core contributions\n\nWeaknesses:\n- The most significant contribution is the action extraction method, but there were insufficient details in Section 4.2.2. More analysis would be helpful, such as the choice of K=11 learned actions: is the method sensitive to clustering algorithm? Would far more or far fewer learned actions affect downstream learning? Can all 410 extracted actions be used during online learning instead of 11 extracted medoids?\n- The contextual bandit setting seems overly restrictive, but I will note that my understanding of the food acquisition field is limited and I do note that the prior work [6] in the paper also make this assumption. However, in cases such as prior works that probe food as part of the action representation, it seems like the proposed framework may not be able to support such types of interactive action primitives.\n- The reliance on user-provided bounding boxes as well as RGBD are limitations not found in other types of learning based robot manipulation systems. \n- While this paper is extremely relevant and impactful for robot-assisted food acquisition, it's not clear how the technical contributions contained could extend to more general domains (this is not really a hard criticism, since the authors acknowledged this up front).", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "ZxfMC4a86i", "reviewer": "Reviewer_tE7u", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work centers on food acquisition as a task and posit that a small number of action strategies (11 parameterizations of a pre-defined action schema, including planning capabilities) are sufficient to acquire a variety (14) of food items with >=80% success rate. These 11 action strategies are identified by collecting human user food acuquisition data, mapping them to the action schema (26-dimensional) and clustering the space of actions. Finally, by treating the task as a contextual bandit (state = feature vector of food plate image + haptics), they show the optimal action out of the 11 actions can be found quickly ~in 20-30 minutes for 2-4 food items.", "summary_of_recommendation": "While I don't believe there are many generic robot learning insights that can be derived from this work, I do appreciate the experimental rigor, choice of application, and demonstration of results on a real robot and real food. The videos seemed simple at a glance, but robust and reliable execution is important to be shown. I am unsure about the applicability of CoRL as the right conference for this application-engineered paper, but I don't want to be the gatekeeper of simple but potentially impactful empirical work, and thus would recommend weak acceptance, contingent on peer reviewers and AC agreeing to the importance of the empirical contributions of this work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "- Since I am not knowledgeable in the domain of food acquisition application, I cannot fully appreciate the complexity of this task. What are the learning problems that make this task hard? Looking at the videos, it seems that main challenge is to generalize over the objects in the image space (given as state input). What are the other challenges?\n- It is not clear to me what is the role of the HapticNet feature vector in predicting the action to take. If the arm is already in the approach phase, this means one of the 11 actions has been selected and executed. And since this is just one-step task, how and when is the HapticNet feature vector utiized?\n- What is the result on the unseen food categories for the proposed method and the baselines? Does the agent primary lose performance on the unseen food items?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Strengths\n- This work presents an interesting insight: the space of the 26-dim action schema, when perceived like a skill space, can be reduced to just 11 meaningful skills for the tasks considered. Broadly, this suggests that for many practical tasks, a few discrete number of strategies are enough to solve them.\n- I appreciate the care in choosing the food items based on the meals an end-user with C1 quadripledia. This sufficiently justifies the seemingly restricted variety of food items considered (14 test items) in this work, as the benchmark's range is grounded to this real world application.\n- The results show that the use of human data to extract useful actions (or skills) is better than pre-defining the actions for this task. This justifies the role of learning from data in achieving robustness in real-world tasks as compared to pre-defined primitives.\n- The videos and extracted actions provided in the supplementary are informative and appreciated.\n\n\n# Weaknesses\n- While the insight is interesting that only a few distinct strategies are enough to solve the task of picking up food objects using forks, this could very well be a commentary on the difficulty of the particular food acquisition task itself. Naturally, as the task gets more complex, the 11 selected schema would not be enough. In a way, this paper shows that the baseline composed of 3 strategies was not enough for the task considered and one indeed needs 10-11 strategies to pick up the food items considered. As the paper suggests in limitations, as sources of error (perception, food orientation) and food varieties increase, even for the limited task definition considered here, one would expect more actions are needed to solve the task. Therefore, it is hard to take much away from the paper's key insight, other than that \"simple tasks require fewer distinct strategies to solve and harder tasks require more\" — which is not as surprising as the paper originally projects.\n- Baseline actions: Since the baseline actions also perform quite reasonably on the task, even without any human data, is it possible that if one or two manually engineered baseline policies are added, then the performance is matched with this paper's results? For exanple, since jello and spinach are the challenging tasks due to rotation or wedging, then manually modifying the action schema to incorporate these new actions would be a more efficient way to get to a working set of actions, than to collect human data. A contextual bandit can be trained on such an action space too, and thus generalize to new food.\n- Formulation as contextual bandit: Since the task of fork lifting of food is a relatively static and simple task, it can be formulated as a contextual bandit for one time action selection. But ideally, an agent should be reactive to changes in the environment dynamics. Therefore, a reactive policy is necessary to make food acquisiton more robust and generally applicable. However, for RL, it is unclear how the \"action\" or parameterized-skill space extracted in this work from user data, can be used.\n- The action schema is heavily engineered for this task and convert one trajectory into one action / skill with 26 parameters. Even the food localization in pixel space is done manually around the food center, which provides a lot of engineered information to the learning modules of this work. Would the insights of this work still be relevant when the tasks are harder or long horizon, and the action schema is not engineered?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "MXysIjA0dC", "reviewer": "Reviewer_tu76", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work proposes a novel skewering action space in bite acquisition for robot-assisted feeding. The approach partitions the skewering problem into approach, grasp, and extraction phases, and performs clustering on a dataset of skewering trajectories collected by humans to infer the continous parameters of these motions. In doing so, the dimensionality of the action space is greatly reduced to just 11 discrete actions. This manageable action space provides a small search space such that at test time, the optimal action for an unseen food item can be inferred in a bandits style approach within a small number of trials.\n\nThis approach is tested on a variety of food items ranging in visual and haptic properties, and the proposed approach achieves around 80-90% success on the difficult task, outperforming prior work which relies on discretized primitives instead.", "summary_of_recommendation": "Overall, the technical contribution is good and original compared to prior work. The paper is written very well, and the experimental evaluation is quite thorough. However, the paper should avoid making strong claims about the contributions and instead focus on the skewering subproblem, while providing more insight into the design choices in terms of the action space dimensionality. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Can you provide more insight into how 11 clusters were chosen specifically? It would be interesting to see an ablation on the success rate  as a function of k. Additionally, it would be great to better understand the rationale of the 26 continuous parameters. i.e., what happens if the force threshold is omitted, etc.\n\nHow many online interactions are required per-food item to infer the appropriate action? And is the food item replaced between interactions (otherwise, could something delicate like a banana slice keep getting damaged over time -- and the optimal action may also change?)\n\nRight now, the action clusters are a bit opaque in terms of interpretability -- could you provide some summary of the emergent behaviors in each action?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- The paper is very well written and the experimental evaluation is quite thorough. I appreciate the number of food items tested on, the diversity of physical properties, and the separation of experiments into success rates on seen food items and unseen evaluation with the bandits approach.\n- A huge problem in robot-assisted feeding is expert design of primitives which can handle a range of food items, and this paper proposes a very natural way to avoid doing so while leveraging demonstration data effectively.\n- The design of the continuous action space was thoughtful and I think provided a happy medium of covering a range of parameters without exploding the action space dimensionality.\n- The use of online bandits to infer the optimal action is a clean framework for generalizing to unseen items.\n- The supplemental video is helpful for understanding the various actions\n\n\nWeaknesses:\n- My primary concern with this work is that the paper makes strong claims about \"general food acquisition\" throughout, though the approach is largely geared towards and only tested on skewering -- I would argue that general food acquisition constitutes a range of utensils (spoon, fork, chopsticks, knife, etc.). The action space is largely tailored to skewering specifically (approach, grasp, extract) and cannot easily accommodate other non-quasistatic actions like cutting, twirling, scraping, wiggling, etc. which are all under the umbrella of general food acquisition. The 'cutting' behavior referenced with the jello, for example, is merely a linear movement in the grasping phase and does not generalize to something like cutting steak which would require much higher frequency movement. For many of the items tested (i.e. sandwich, donut, rice, mashed potatoes, noodles), a case also needs to be made for skewering, as it seems that those items would benefit from a different tool/acquisition approach altogether. I suggest reframing the motivation to focus on skewering specifically.\n- As the authors pointed out, the food perception in this work was largely simplified to a clean workspace with no distractors and classical/manual techniques to isolate the item of interest. SPANet and other works in the acquisition domain have evaluated on varied degrees of clutter and with more challenging scenarios like the food item being near a plate wall or stacked on top of other items (which would be an interesting testbed for this approach) -- this should be justified.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "Gv8KonD6T-", "reviewer": "Reviewer_kQNo", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The method proposes various ways of picking up food from a plate using a robotic end effector using online learning policies. This is an effort that can serve with people with disabilities, I love that robotics is being used for good. They propose that a small number of actions can pick up almost any type of food. This seems consistent with my personal experience. As such, it is a data constrained regime(only 33 hours). The frame definitions are defined with respect to the plate rather than the end effector, that appeared surprising. ", "summary_of_recommendation": "This paper studies the problem of food acquisition and learning of that manipulation from human actions. The paper is technically sound and well executed, also well explained. This is why the accept recommendation. There is also hardware experiments. However, the study is within a very constrained setting. The system makes assumptions that may not generalize to all homes, but it still shows incredible generalization to many different food types. There are also many hand engineered aspects that I feel will not scale, hence the weak accept. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. What could be a way to scale this?\n2. How would you inform robotics in general based on the observations of this study?\n3. Given that the title mentions human feedback, is there any user interaction observations from this study?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. Paper is well written, technique is nicely explained and details and procedures are clear.\n2. The results are also correct explained and seemed intuitive to understand. \n\nCon:\n1. There's a lot of assumptions in the system design, which works very well for the constrained setting, but its unclear how this can scale. \n2. Both experiments and results are highly domain specific, so my one suggestion to improve the paper could be, how would you inform robotics in general based on the observations of this study", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that this paper brings an insightful and worthwhile contribution to the CoRL community. The paper is well-written, the experiments tests a large number of food items, and insights around lower-dimensional action spaces are appreciated. The reviewers also brought up several points of feedback that the authors are encouraged to address, including but not limited to: (a) changing the title and the rest of the writing to scope the contribution to skewering (with a fork) rather than general food acquisition, (b) discussing the limitations around the system assumptions/task-specific engineering, and (c) including more discussion of the general takeaways of the paper.
Decision: Accept (Poster)