Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding
Authors: Priya Sundaresan
Suneel Belkhale
Dorsa Sadigh
Venue: CoRL 2022 Oral (2022)
Keywords: Assistive Feeding
Deformable Manipulation
Multisensory Learning

Abstract: Acquiring food items with a fork poses an immense challenge to a robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items on a plate remains nontrivial. Our key insight is to leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn a generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework to sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within a single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, code, and videos can be found on our website: https://sites.google.com/view/hapticvisualnet-corl22/home.

Learning Visuo-Haptic Skewering Strategies for Robot-Assisted Feeding
Priya Sundaresan Stanford University priyasun@stanf ord. edu
Suneel Belkhale Stanford University belkkhal@@taanfrdd edu
Dorsa Sadigh Stanford University
dorsa@cs.stanforre stanford......o.. edu
Abstract: Acquiring food items with fork poses immense challenge to robot-assisted feeding system, due to the wide range of material properties and visual appearances present across food groups. Deformable foods necessitate different skewering strategies than firm ones, but inferring such characteristics for several previously unseen items plate remains nontrivial. Our key insight is leverage visual and haptic observations during interaction with an item to rapidly and reactively plan skewering motions. We learn generalizable, multimodal representation for a food item from raw sensory inputs which informs the optimal skewering strategy. Given this representation, we propose a zero-shot framework sense visuo-haptic properties of a previously unseen item and reactively skewer it, all within single interaction. Real-robot experiments with foods of varying levels of visual and textural diversity demonstrate that our multimodal policy outperforms baselines which do not exploit both visual and haptic cues or do not reactively plan. Across 6 plates of different food items, our proposed framework achieves 71% success over 69 skewering attempts total. Supplementary material, datasets, code, and videos can be found on our website.
Keywords: Assistive Feeding, Deformable Manipulation, Multisensory Learning
Introduction
Realizing the full capabilities of assistive robots in the home, hospitals, or elderly care facilities remains challenging due to the dexterity required to complete many day-to-day tasks. Eating free- form meals is one such example with many nuances in perception and manipulation that can be easy to overlook. However, automating the task of feeding, one of six essential activites of daily life (ADL) [1]. has the potential to improve quality of life for over one million people in the U.S. who are unable to feed themselves due to upper-extremitt mobility impairment, their caregivers, families with young children and elders, and anyone impacted by the substantial time and effort required in meal preparation and feeding 3, 4].
In recent years, there have been significant efforts to tackle the challenging problem of robot-assisted feeding. Solutions on the market have limited traction as they rely heavily on prepprogrammed trajectories, pre-specified foods, have limited autonomy, or require manual utensil interchange [5, 6]. Meanwhile, academic research on assistive feeding largely centers around data-driven methods but has yet show widespread generalization across food groups [7, 8, 9, 10]. As necessary first step towards feeding, we focus on the problem of bite acquisition - acquiring bite-sized items from plate bowl using a robot with a fork-equipped end-effector. Developing bite acquisition strategy sensitive to differences in geometry and deformation both across and within food classes is challenging problem: Skewering position and orientation matters for items with irregular shape, such as a broccoli floret where skewering at the stem is preferable to the head for stability of the acquisition. The fragility of food also affects the optimal skewering strategy, delicate items such as thin banana slices are more likely to slip off a fork oriented vertically and instead benefit from an angled fork insertion and scooping strategy [1]]. On the other hand, hard baby carrots require a vertical insertion angle for effective and stable acquisition [1]]. In addition, instances within the same class of foods can also exhibit visual similarity but textural contrast (raw vs. boiled carrots, silken vs. extra firm tofu, cheddar vs. mozzarella cheese); choosing the optimal skewering strategy therefore depends on more than just vision.
Prior works show that food classification objectives can lead to visual features that can be used for downstream policy learning [7, 8, 9]. They also introduce action taxonomy for skewering to
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Fosition
SKEWER
Orientation
Image Encoder
Fnrce
Angled
Skewering
Time
HapticWisualNet
Primitive
Position
Orientation
Force Encoder
Vertical
Figure 1: Left: Our method learns zero-shot skewering of food items with Franka Panda robot. Given an overhead plate observation, localize food items and probe them reveal haptic and visual data. Using the multimodal data as input, VisualNee infers the optimal skewering trajectory on the fly - angled skewering for soft items like banana slices, or vertical skewering for firm textures such as a broccoli stem.
discretize the complex space of possible acquisition trajectories [7, 8, 12, 13]. Although these visy skewering strategies are able to classify food with different geometry, they lack critical information about deformation and may fail to differentiate between foods within the same food class that appear similar but have drastically different properties, such as boiled and raw carrots.
Preliminary experiments suggest that hoiled carrot requires the fork to skewer the item at an angle to avoid breakage or dropping during acquisition, whereas a rigid carrot requires more forceful vertical approach to pierce the item. Gordon et al. [10] try to address this issue by leveraging post-hoc haptic feedback to update a visual policy after skewering. This work requires multiple trials interaction per unseen food item reason about item deformation through haptic feedback. However, repeated skewering attempts can easily damage fragile items (e.g. overcooked carrots or thin slices banana) multiple
and potentially change the properties of the food, leading breakage or squishing robot interactions. These repeated interaction strategies do not scale to unseen food classes either, for the same reasons. On the other hand, open-looy strategies that do not adapt skewering plans mid-motion are also limited in their ability to handle unseen items with unknown properties.
Therefore, bite acquisition methods should be able to zero-shot generalize to new foods both within and across food classes, just like how humans skewer bites of food without the need for multiple interactions with the food.
Our key insight jointly fuse haptic and visual information during single skewering interaction to learn more robust and generalizable food item representation. We develop a bite acquisition system along with visuo-haptic skewering policy that leverages this learned representation. The proposed representation informs the skewering policy of both geometry and deformation food properties on- the-fly, thus enabling reactive policy which zero-shot generalizes unseen plates of food within and across food classes. Our experiments with wide range of food items with varying geometry and deformability demonstrate that our method outperforms those that (1) do not jointly use haptic and visual cues and (2) do not reactively plan upon contact, achieving 71 % skewering success across 21 items total. Our contributions include:
skewering system that employs coarse-to-fine visual servoing to approach a food item, sense multimodal properties upon contact, and reactively plan skewering the same continous interaction A zero-shot skewering policy that captures geometry and deformation by fusing visual and haptic information with demonstrated generalization to unseen food items
Experimental validation on diverse seen and unseen food items, with varying degrees of visual likeness and deformation
An open-source dataset for multimodal food perception and custom end-effector mount designs, which we hope expands the scope of assistive feeding research

---

2 Related Work
We build on prior works studying multisensory robot learning both within and beyond the food domain. In this section, we will discuss related work in robot-assisted feeding, food manipulation. and more generally multimodal robotic perception and manipulation.
Robot-Assisted Feeding Feeding can be split into two stages: bite acquisition and bite transfer. Previous work in bite transfer transferring acquired bite user's mouth for consumption
suggests that transfer is largely contingent upon acquisition [14, 7, 13]. To enable reliable bite
acquisition and thus transfer, recent acquisition frameworks combine image-based perception [7, 8, 9, 10, 13] - bounding boxes and food pose estimates - with an action space consisting of parameterized primitives that modulate fork roll/pitch relative to item geometry. SPANet (Skewering Position Action Network) [8] is one such forward model mapping food image observations to actions, which has been shown reasonably clear plates containing 16 types of fruits and vegetables. SPANet is trained on 2.5K fork interactions (81 hours of supervision [9, 8)) which does not readily scale to new foods. uollow-up works aim to rapidly adapt SPANet unseen food items using a contextual bandit to learn the optimal primitive selection strategy from real interactions. Approaches include updating SPANet predictions online by observing the binary outcome of acquisition attempts on unseen items [9], and additionally haptic time-series readings recorded post-hoc [10]. A key assumption [10] is that the visual context observed pre-skewer and the haptic context observed post-hoc are equivalent alternate representations for the underlying food state, which does not always hold (e.g., firm and soft tofu appear to be almost identical but yield different haptic readings). In our work, we do not restrict ourselves to this assumption and consider the more general setting where visually similar items may have different physical properties. In addition, we do not assume access to repeated interaction trials with the food, and consider a zero-shot planning setting. Bhattacharjee et al. [12]. Song et al. [15] explore classifying food compliance or skewering outcomes from haptic data from single interaction, but delegate reactive planning given these representations to future work. To address these gaps, we learn a multisensory policy that learns zero-shot skewering from pre-skewer and post-contact paired images and haptic readings.
Food Manipulation Recent simulated benchmarks for household food manipulation explore food preparation, lunch packing, and food storage [16]; pouring water [17]: pile manipulation for chopped food [18]: and drinking/feeding [19]. While these works largely abstract away the state space of food, recent work in real robotic food slicing explores multimodal food representation learning from interaction using datasets consisting of paired visual, tactile, and audio information [20, 21, 22, 23, 24]. In particular, Gemici and Saxena [20] propose to infer haptic properties ("brittleness', 'tensile strength', 'plasticity' etc.) from probing actions that can inform slicing actions, but does not consider visual properties. Likewise, Zhang et al. [25] use vibrations from probing interactions to adapt slicing motions. The learned haptic representations from these work are not directly transferable to acquisition due to hardware and task differences, but we adopt the notion of using probing actions to inform skewering in our work.
Integrating Vision and Haptics in Robotics Vision-only manipulation has proven effective in robotic domains such as semantic grasping [26, 27] and deformable manipulation [28], but contact- rich tasks such as peg insertion [29] or robotic Jenga [30] have been shown to benefit from combining vision, force, and proprioception as inputs to learned policy [31, 32]. In light of these works, we propose to learn reactive, visuo-haptic policies for bite acquisition which remains largely unexplored.
3 Method
Our goal is to learn a multisensory manipulation policy which outputs skewering actions to clear a plate of bite-sized food without any previous skewering attempts for this food item. We consider foods of varying degrees of geometric, visual, and textural similarity, all on the same plate. We first formalize the bite acqusssitinn setting in Section 3.1 and introduce our action space to tackle the problem in Section 3.2. Next, we discuss our interaction protocol for sensing visuo-haptic properties of food (Section 3.3), enabling to learn multimodal skewering policy (Sections 3.4-3.5).
3.1 Problem Formulation
3

---

Franka Panda (7DoF)
At each timestep t E I...., T, we assume access to a the current RGB-D image observation I, R RWxlxx of plate of food and an N-length history of haptic readings H, E RNN6 denoting 6-axis readings from an F/T sensor on a fork-mounted end-effector. Similar to prior work, we consider an action space parameterized by fork (z, y, position, roll y, and pitch B. We define an action at E A. visualized in Fig. 1, as follows [9, 10]:
Intel RealSense
D435i
Mini45 ATI F/T Sensor
(T. v)
B
a: = (2, U. 2, A2, 7, A,3)
(1)
Figure 2: Skewering action space: fork pitch 3, roll y, and position (x, V.
At time i, the fork starts in position (ar, y, with pitch 3 = 0"
and roll y and moves downward A < 0 while optionally tilting AB 2 O8 skewer an item (Figure 2).
We use [y[ar] E {0. to denote the binary loss of executing action ag. where 1 [aa = denotes failure, for example food failing to be picked up, slipping off the fork after skewering, breakage or damage as a result of skewering. We opt for a binary loss objective as it is difficult to quantify more nuanced notions of success such as degree of damage an item or stability of skewer.
We aim to learn a policy (a, Le, H.) that minimizes Σx 1, [a,] given no previous interactions. Our policy learns to map visual and haptic information to a discrete set of skewering primitives using small but diverse labelled dataset that can be collected offline. By conditioning the policy on haptic readings during a short initial contact period with the the food item, our method Haptic VisualNet can extract food properties that are inaccessible to vision using only single skewering interaction. First we outline how we represent the discrete set of primitives for skewering.
3.2 Skewering Action Primitive Parameterization
To successfully skewer a food item, the fork position and roll must align with the location and orientation of a food item. The fork pitch must also adapt to the compliance of food item (i.e. soft banana slice favors angled fork approach to prevent slip, while a raw carrot favors vertical approach for piercing). Thus, our action space employs two primitives, vertical skewer or angled skewer, to account for rigid or compliant items, respectively. We implement vertical skewer with AB = 0%, denoting no tilt during skewering, and angled skewer with A > 0%, where the fork gradually tilts from vertical to angled approach during insertion into a fragile food item (Figure 2). Prior work includes both angled and vertical skewering strategies amongst an even larger set of primitives, but we empirically find that simplified taxonomy reduces redundancy in this larger action space and can handle an equivalently broad range of food items, evaluated in Section 4 [8, 10J. In order decide between these strategies on-the-fly, we condition our policy not only on visual information but also on haptic information at the point of contact with the food item, discussed in the next section.
3.3 Sensing Multimodal Data Via Probing
We introduce a probing motion to obtain visuo-haptic information about a food item by bringing the fork in contact with the item surface but without actually skewering it. The multisensory information collected from probing serves as input to To which rapidly decides the skewering primitive to execute. Between the probing and skewering phases, the fork remains stationary and in constant contact with the food item, thus enabling a fluid transition between phases.
approach requires localizing an item and approaching with precision so as to not accidentally shift or topple it while making contact. To accomplish this, we first detect items from a plate image using pre-trained RetinaNet food bounding box detector from Gallenberger et al. [7]. Similar prior work, we also train a network (SkeweringPoseNee) which refines the estimated item location by predicting a keypoint for the item center within the local bounding box, and fork roll angle y with which to approach [8]. We can obtain the 3D predicted item location (át, g. 2) in robot frame by using depth information. Next, we continuously servo to the item using a learned model (ServoNet) which detects the fork-item offset as keypointss from streamed RGB images. Using this framework, we probe food item and record post-contact image observation 7, and the short window of force magnitude readings H,. Section 3.4, we discuss how these multisensory readings inform the optimal skewering strategy.
Our

---

3,4 Multimodal Representation and Policy Learning
To learn (T(a; 14, Hy), we propose HasticVisualNett, a network which takes observations captured by the probing motion and reactively outputs the appropriate skewering action. HaptieVisualNe VisualNet takes as input a post-contact image of a food item I E I x x3 and H,, the force magnitude readings recorded from the F/T sensor during the first N milliseconds of contact in the initial probing period.
Haptic maps (I+, H.) to an |Al-d vector denoting the likelihood of success for each action primitíve, in our case vertical skewer and angled skewer. The model first encodes visual information and haptic information separately. and then concatenates these features to produce joint visuo-haptic representation. The policy then predicts action success likelihood from this representation. We implement HapticVisualNet as mulli-hhedder network with a ResNet- 18 backbone for the visual encoder and a LSTM for the haptic encoder. We pass the concatenated visual and haptic encodings to a linear layer followed by softmax to obtain primitive successes, and choose the maximum likelihood predicted primitive the skewering action.
3.5 Training and Data Collection
We train Haptic VisualNet on small but diverse dataset of 300 paired post-contact images and haptic readings, augmented 8x using image affine and colorspace transforms as well as temporal scaling and shifting of the haptic readings. The dataset consists of hard items labeled vertical skewer (raw carrots/ broccoli/ / zucchini / butternut squash, grapes, cheddar cheese, and celery) and soft items labeled angled skewer (banana, kiwi, ripe mango, boiled carrots/ broccoli / zucchini butternut squash, avocado, and mozarella cheese).
In practice, we use an N 26 ms. contact window of haptic readings, from the initial probing period, which we find adequately captures force-surface interactions. Using a 20-30 ms. window of contact is also a common choice of haptic representation in other reactive, contact-rich manipulation settings [29, 10]. For each paired example, we manually assign the optimal primitive label - vertical or angled skewer based whether the annotator considers the item hard or soft. This process requires 3 hours of data collection and labeling time total (a 27x reduction from the 81 hours reported in SPANet [10]), and without the need for actual skewering attempts during data collection. We intend for the inclusion of haptic data to prevent overfitting to visual features, enabling more generalizable, food representation trained with greater sample efficiency. Both ServoNet and SkeweringPoseNer are implemented using a ResNet- 18 backbone, each trained on 200 images of the same items (2 hours of supervision) and augmented annotations. Additional training and implementation details are in Appendix C.
dataset of 3,500 paired images and
In summary, our method leverages offline datasets to learn visuo-haptic features to more robustly predict between a set of discrete skewering actions. The probing motion used to obtain haptic information is connected seamlessly to the chosen skewering motion, leading to one continuous and adaptive zero-shot skewering policy aware of haptics and vision.
4 Evaluation
Plate 1
Plate 2
Plate 3
Plate 4
Plate 5
Plate 6
Figure 3: 6 Plates for Evaluation, covering wide range of foods. From left right: 1: Raw banana, broccoli, zucchini, carrot, grapes, cucumber. 2 (Unseen); Raw pincapple, mango, dragonfruit, canteloupe, honeydew, pear. 3: Raw butternut squash, boiled butternut squash. 4: Raw broccoli florets. 5 (Unseen): Pasta, dumpling, boiled yam/sweel potato, raw yam/sweet potato. 6 (Unseen): Ice cream mochi, snow peas, canned peaches.
In this section, we seek to evaluate (1) the benefits of combining both vision and haptics for bite acquisition as opposed to only using subset of modalities, (2) the effectiveness of reactive skewering compared to open-loop strategies, and (3) the generalization capabilities of our system to previously unseen foods. We first perform classification ablations of HapticVisualNy VisualNet in (Section 4.1) and then deploy our system in the real world for trials on both seen and unseen foods (Section 4.2-4.4).
5

---

Plate
Plate 2
Plate 3
Cumulative Failures
1.0
1.0
1.0
30
Total Occureenss
(18
01.8
0U8
% Success
% Sucoss
% Surcesss
0.6
0.6
0.6
0.4
0.4
0.4
0.2
02
0.0
18ox Lnwiable Darsage Doo 3- Tiee Fillure Mode
HapticVis (OURS)
Haptic
SPANet
Figure 4: Skewering Success and Failure Modes: We visualize the number of items acquired over total acquisition attempts for all methads across Plates 1-3. Failure modes include exceeding the maximum numher of consecutive attempts (3+ tries per item), dropping after skewering, the item being unstable on the fork after skewering (affecting transfer), damage breakage to fragile items, or failure to detect item due bounding box anomalies. VisNee VisNet (ours) performs best on each plate, while causing the least failures.
4.1 Ablative Studies
We evaluate the contributions of both visual and haptic data towards primitive classification by training and evaluating Haptic VisualNet against two variants which observe exclusively the post-contact haptic readings or post-contact image after probing.
Figure 5 shows the confusion matrices for classification accuracy. Haptic NisualNet achieves the highest overall and per-class classification accuracy, and omitting either modality hurts accuracy. We hypothesize that the reduced performance of the vision- only model stems from visually similar but texturally dissimilar foods in our dataset, for which inferring the primitive is challenging without haptic context. On the other hand, the haptic-only model learns a naïve solution of mapping high magnitude contact events to vertical skewering ing and low readings angled skewer. This brittle for anomalous foods like broccoli which benefit from a verti cal skewer, yet may yield low contact readings if the fork comes contact the head instead of the stem. Similarly, the fork can easily penetrate thin banana slice and touch the plate during probing, yielding high contact readings when an angled skewer still preferable.
HapticVa (OURS)
Vis
Haptic:
Predicted
0.93
0.066
0.91
0.08
0.73
0:27
0.75
0.11
025
VSS
VS
Ground Truth
Figure 5: Confusion matrices for classification accuracy the skewering primitive (angled skewer or vertical skewer), on a held-out test dataset of 60 images, for each model. Darker off-diagonals and lighter on-diagonals indicate more accurate models.
the subsequent sections, we evaluate HapticVisualNe VisualNet (Haptic Vis) against the haptic only (Haptic), vision only (Vis), and SPANet basclines on real food acquisition trials. See Appendix D for additional ablations of the learned multimodal representation and HaptieVisualNer VisualNet sample efficiency.
4.2 Hardware Setup
Our setup consists of 7DoF Franka Emika Panda robot with the default gripper. We outfit the gripper with a custom 3D-printed mount comprised of a standard fork, Mini45 ATI F/T sensor, and D435i RealSense camera. We perform all acquisition trials on plastic dinner plate on an anti-slip surface. We instantiate each primitive, parameterized according to Equation (1) as follows, assuming a fixed Az and discretized A8:
probe (2,
SETROACHLHEIGHTT -APROACH. 9, 09)
vertical skewer = (bt 0, 2, -DT+0. 10m//mcm. 9, 0%)
angled skewer (á, 0. 2, -TT.D.T0///.e....... 9, 65%)
Here, (a, 9, 8) denotes a predicted food item location, obtained by deprojecting a predicted pixel in a depth image from ServoNet to a 3D location. EkeweringPoseNee also predicts the fork roll 9. We first probe the item starting from an APPROACH.HEITT and record haptic readings H, E 26 over the first 26 26-milliseconds of contact. Given these inputs, Haptic VisualNet infers either vertical skewer or angled skewer which we execute.
of 0.01em, observe a post-contact image It
The robot controller runs at 20Hz (DT 0.05). and primitives terminate early if the fork reaches pre-defined z-distance (the plate height) or force limit. After skewering, the end-effector scoops until the fork is nearly horizontal with 3 80", emulating the start of a feasible transfer trajectory.

---

4.3 Baselines
We deploy all methods Haptic (ours), Haptic, Vis, and SPANet on the real robot setup of Section 4.2 and classification networks trained according to Section 4.1. The Haptic Vis, Haptic, and Vis methods all perform probing- weeogg-hhnssseeeeee visual sensory information obtained from probing, respectively.
but run inference using both, only haptic, only
We also implement SPANet given the pre-trained visual models and original taxonomy of six skewering primitives reported in [8]. SPANet performs zero-shot primitive inference given an overhead image observation of a food item, without probing. SPANet still uses our ServoNet to plan, analogous to the original implementation which similarly accounted for fork-food precision error.
4.4 Real World Bite Acquisition Results
We compare all methods on the challenging task of clearing plates containing 10 bite-sized food items, evaluated according to skewering success and the distribution of skewering failure modes encountered (Table 1). We define a skewering success as one in which the fork picks up the item with least 2 tines inserted and the item remains on the fork for up to 5 seconds after scooping as in [8]. Failure modes are detailed in Figure (4), Between successful acquisitions, human operator removes the acquired item from the fork. Upon skewering failure, an attempted item remains on the plate and can be re-attempted up to 3 times before being manually removed and marked as a failure.
Plates 1-3 Full System Evaluation: We first perform a full-system evaluation of HapticVis Vis and all baselines on 3 plates. Plate I contains in-distribution fruits and vegetables that HapticVisualNe VisualNet was trained on which include both textural and visual diversity, Plate consists of unseen assorted fruits with visual diversity but similar textures, and Plate 3 contains in-distribution boiled and raw butternut squash cubes which appear similar but greatly differ in softness. When adjusted for perception failures (e-g-, bounding box false negatives) which affect all methods, Haptic Vis outperforms all methods across Plates 1-3 (Figure 4). The bulk of Haptic Vis failures center around near-misses or perception failures which are less drastic than damaging items or exceeding skewering attempts (Figure 4). Vis slightly underperforms Haptic across all plates and achieves lowest performance on Plate 3, where mispredicted vertical skewer or angled skewer actions can miss/damage soft-boiled squash, or fail to penetrate hard raw squash. SPANet achieves lowest performance, mostly due to erroneously executing vertical strategies to pick up soft items like bananas and ripe dragonfruit in many cases. SPANet's ' underperformance relative to HapticVis, Haptic, and Vis suggests the effectiveness of reactive strategies compared to open-loop acquisition, and that our simplified action space is just as expressive as SPANet's larger taxonomy. We acknowledge that the performance gap may in part be attributed to hardware differences (different robot, different fork mount) in the original SPANet compared to our re-implementation.
Plate 4 - Texturally Misleading Food: While Haptic is the most competitive baseline on Plates 1-3, we run additional experiments between Haptic Vis and Haptic (Plate 4). In cases where the stem is occluded from view or the fork servos to the leafy region to make contact, Haptic tends towards misclassifying the low readings as angled skewer, leading to frequent failures to pierce the item which Haptic is better equipped to recognize and avoid.
plate of only broccoli florets
Plates 5-6 Generalization to Distt--Dist Foods: Finally, we stress-test the generalization capabilities of Vas on two plates of unseen foods, Plate 5 and Plate 6, containing the items listed in Table 1. Across Plates 5-6, Vaptic achieves 58% success, 19% improvement over Vis, indicating the promise of multimodal representations for zero-shot food skewering. Haptic performs best on soft canned pears and boiled/raw root vegetables which are most comparable to the items in the training distribution, closely followed by pasta and ice cream mochi. The majority of failures occur due to the fragility of dumplings and thinness of snow peas which are difficult to pierce. Still, by fusing haptic and visual information, Haptic Vis is better equipped to generalize to visually and texturally diverse foods. See Appendix E for additional stress-tests of Haptic VisualNet.
Overall, HapticVisually VisualNet benefits from the use of both vision and haptics from just a single interaction, and outperforms single-modality baselines for a variety of challenging food plates. We show that skewering strategies that reactively update the strategy (Haptic Vis, Haptic, Vis) are more robust to texturally and visually diverse food items than open loop strategies (SPANet). With its multimodal representation, Haptic VisualNet can also shoo generalize to challenging unseen food classes.
7

---

Plate Type
Items Acquired / Total Attempts
Plate
Items
Visuals Haptics Category
HapticVa Vis Haptic
Vis
SPANet
Assorted fruits and vegetables
Diverse Diverse
Seen
9/10
9/12
7/10
8/20
Assorted tropical fruits
Diverse Similar
Unseen
10/11
8/13
10/14
Boiled/raw butternut squash
Similar Diverse
Seen
8/11
9/14
10/18
7/12
4
Broccoli florets
Similar Similar
Seen
8/13
7/17
Pasta, dumplings, hoiled/raw root veggies Diverse Diverse Mochi, snow peas, canned pear
Unseen
7/11
9/13
Diverse Diverse
Unseen
7/13
5/23
71%
63%
51%
54%
Table 1: Physical Results:
We evaluate the ability of all methods to clear 6 plates (Figures 4), each initially containing items. The number items acquired refers to items successfully skewered (S 10 all cases due to dropped items or early termination with bounding box false negatives). The total attempts refers to all attempted acquisitions until termination or clearance (2 10 in all cases due to failed items remaining on the plate for up 3 consecutive re-attempts). Haptic Vis outperforms baselines in 5/6 plates. The symhol denotes experiments that useful in the specific testing scenario.
5 Discussion
Summary In this work, we present a framework for zero-shot food acquisition diverse range of bite-sized items using multimodal representation learning. Our approach uses interactive probing sense complex, multisensory food properties upon contact in order to reactively plan a skewering strategy. We deploy the learned policy along with learned visual servoing controller zero-shot skewering of unseen food items. Our experiments span a wide range of food appearances and textures, and validate the need for multimodal reasoning and reactive planning to clear plates.
robot for
Limitations and Future Work One limitation our approach is the use of a small action space for acquisition. While our set of primitives can generalize to wide range of foods, our current strategies are not equipped to handle thin, flat items like finely sliced produce or leafy salad greens, which may require new techniques like positioning the fork under item to scoop, gathering multiple items together before skewering, or using plate walls for stabilization. Other food groups like noodles would benefit twirling, and foods with irregular geometries might require toppling into stable pose before skewering. Another limitation in this work is the supervision used to currently train our multimodal policy. However, since our policy learns from sparse labels, are excited by the possibility of automatically detecting skewering outcomes to self-supervise the training procedure for Haptic Net.alNet. Finally, in future work hope tackle challenging food groups such filled dumplings which easily break and extremely thin snow peas which are hard to pierce.
8

---

Acknowledgments
This work is supported by NSF Awards 2132847, 2006388, and 1941722 and the Office of Naval Research (ONR). Any opinions, findings, and conclusions recommendations expressed in this material those of the author(s) and not necessarily reflect the views the sponsors. Priya Sundaresan supported by an NSF GRFP. We thank our colleagues for the helpful discussions and feedback, especially Tapomayukh Bhattacharjee, Jennifer Grannen, Lorenzo Shaikewitz, and Yilin Wu.
References
[1] S. Katz. Assessing self-maintenance: activities of daily living, mobility, and instrumental activities of daily living. Journal of the American Geriatries Society, 1983.
[2] M. W. Brault et al. Americans with disabilities: 2010. US Department of Commerce, Economics and Statisties Administration, .0....2...
[3] S. W. Brose, D. J. Weber, B. A. Salatin, G. G. Grindle, H. Wang. J. Vazquez, and R. A. Cooper. The role assistive robotics in the lives of persons with disability, American Journal of Physical Medicine Rehabilitation, 89(6)5509 521, 2010.
[4] V. Maheu, P. S. Archambault, Frappier, and F. Routhier. Evaluation of the jaco robotic arm: Clinico- economic study for powered wheelchair users with upper-extremity disabilities. In 201 IEEE International Conference on Rehabilitation Robotics, pages 1-5. IEEE, 2011.
[5] Meet Obi
[Online: accessed 6-June-2022].
[6] Meal-Mate Made2Aid.
https://vww.zade222 co. uu/ prroductprof11eeppoouctttt company-BB//20feal hhw.aaaarrrrroooctteeeeae-1----.
[Online; accessed 6-June-2022).
[7] D. Gallenberger. T. Bhattacharjee. Y. Kim, and S. Srinivasa. Transfer depends acquisition: Analyzing manipulation strategies for robotic feeding. In 2019 14th ACM/IEEE International Conference on Human- Robot Interaction (HRI), pages 267-276. IEEE, 2019.
[8] R. Feng. Y. Kim, G. Lee, E. K. Gordon, M. Schmittle, Kumar, T. Bhattacharjee, and S. S. Srinivasa. Robot-assisted feeding: Generalizing skewering strategies across food items on plate. In The International Symposium of Roboticy Research, pages
Springer, 2019.
[9] E. K. Gordon, X. Meng. T. Bhattacharjee, M. Barnes, and S. S. Srinivasa. Adaptive robot abstssssieted feeding: An online learning framework for acquiring previously unseen food items. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9659-9666. IEFF, 2020.
[10] E. Gordon, S. Roychowdhury, T. Bhattacharjee, Jamieson, and S. S. Srinivasa. Leveraging post hoc context for faster learning bandit settings with applications in robot-assisted feeding. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 10528-10535. IEEE, 2021.
[11] T. Bhattacharjee, G. Lee, H. Song. and S. S. Srinivasa. Towards robotic feeding: fork-based food manipulation. IEEE Robotics and Automation Letters, (22::148--199) 2019. doi: 10.1 (0t.://RA..01.28944.4
Role of haptics in
[12] T. Bhattacharjec, G. Lee, H. Song, and S. S. Srinivasa. Towards robotic feeding: Role of haptics in fork-based food manipulation. IEEE Robotics and Automation Letters, 4(2):1485-1422, 2019.
[13] T. Bhattacharjee, F. K. Gordon, R. Scalise, M. F. Cabrera, A. Caspi, M. Cakmak, and S. S. Srinivasa. Is more autonomy always better? exploring preferences of users with mobility impairments robot-assisted feeding. In 2020 15th ACM/IEEE International Conference on Human-RRobl Interaction (HRI), pages 181-190. IEEE, 2020.
[14] S. Belkhale, E. K. Gordon, Y. Chen, S. Srinivasa, T. Bhattacharjee, and D. Sadigh. Balancing efficiency and comfort in robot-assisted bite transfer. In 2022 International Conference on Robotics and Automation (ICRA), pages 4757-4763. IEEE, 2022.
[15] H. Song, T. Bhattacharjee, and S. S. Srinivasa. Sensing shear forces during food manipulation: resolving the trade-off between range and sensitivity. In 2019 International Conference on Robotics und Automation
(ICRA). pages 8367-8373. IEEE, 2019.
[16] S. Srivastava, C. Li. M. Lingelbach, R. Martin-Martin, F. Xia, K. E. Vainio, Z. Lian, C. Gokmen, S. Buch, K. Liu. et al. Behavior: Benchmark for everyday household activities in virtual. interactive, and ecological environments. In Conference on Robot Learning, pages 477 490. PMLR, 2022.

---

[17] X. Lin, Y. Wang. J. Olkin, and D. Held. Softgym: Benchmarking deep reinforcement learning for deformable object manipulation. 2020.
[18] H. Suh and R. Tedrake. The surprising effectiveness of linear models for visual foresight in object pile manipulation. In International Workshop the Algorithmic Foundations Robotics, pages 347-363. Springer, 2020.
[19] Z. Erickson, V. Gangaram, A. Kapusta, C. K. Liu, and C. C. Kemp. Assistive gym: A physics simulation framework for assistive robotics. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 10169-10176. IEEE, 2020.
[20] C. Gemici and Saxena. Learning haptic representation manipulating deformable food objects. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 638-645. IEEE, 2014.
[21] Zhang, M. Sharma, M. Veloso, O. Kroemer. Leveraging multimodal haptic sensory data for robust cutting. In 2019 JEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), pages IEEE, 2019.
[22] A. Sawhney, S. Lee, K. Zhang, M. Veloso, and O. Kroemer. Playing with food: Learning food item representations through interactive exploration. In International Symposiun on Experimental Robotics, pages 309 322. Springer, 2020.
[23] M. Sharma, K. Zhang, and O. Kroemer. Learning semantic embedding spaces for slicing vegetables. arXiv preprint Xrv:: 0021/944.000 00303, 2019.
[24] E. Heiden, M. Macklin, Y. Narang, D. Fox, A. Garg, and F. Ramos. Disect: A differentiable simulator for parameter inference and control in robotic cutting. arXiv preprint arXiv:2203. 10266, 2022.
[25] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer. Leveraging multimodal haptic sensory data for robust cutting. CoRR, abs/1909.1240 12460, 2019, URL http:://axxiii wrg/abs/
abs// 19t///99.... 12460.
[26] J. Mahler. J. Liang. S. Niyaz. M. Laskey, R. Doan, X. Liu, J. A. Ojea, and K. Goldberg. Dex-net 2.0: Deep learning plan robust grasps with synthetic point clouds and analytic grasp metrics. 2017.
[27] P. R. Florence, L. Manuelli, and R. Tedrake. Dense object nets: Learning dense visual object descriptors by and for robotic manipulation. 2018.
[28] H. Ha and S. Song. Flingbot:
The unreasonable effectiveness of dynamic manipulation for cloth unfolding. In Conference Robot Leurning, pages 24-33. PMLR, 2022.
[29] M. A. Lee, Y. 7hu, K. Srinivasan, P. Shah, S. Savarese, L., Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: superviid learning of multimodal representations for contact-rich tasks. In 2019 International Conference on Robotics and Automation (ICRA), pages 8943-8950 IEEE, 2019.
[30] N. Fazeli, M. Oller, Wu, Z. Wu, J. B. Tenenbaum, and A. Rodriguez. See, feel, Hierarchical learning complex manipulation skills with multisensory fusion. Science Robotics, 4(26) 2019.
[31] Y. Li, J.-Y. Zhu, R. Tedrake, and A. Torralba. Connecting touch and vision via cross-modal prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision Puttern Recognition, pages 10609-. 10618, 2019.
[32] A. Lee, M. Tan, Zhu, and Bohg. Detect, reject, correct: Crossmodal compensation corrupted sensors. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 909-916. IEEE, 2021.
10


Reviews:
[{"review_id": "lh4_b5vSkYZ", "reviewer": "Reviewer_LSMg", "summary": "This paper proposes a new bite-acqusition framework for robot skewering food on a plate using multimodal (visual and haptic) information. The framework compose of two stages: the probing stage and the skewering stage. During the probe stage, two networks are trained to localize the food bite and approach the localized food till contact is made. During the skewering stage, the proposed VisualHapticNet fuses the haptic measurements from a force-torque sensor and images from a camera to determine a skewering primitive (vertical versus angled). Comprehensive experimental results are conducted to test the framework's performance on in-distrituion and out-of-distribution food acquisitons, with comparisons to baselines and ablations. The results show good performance on food with in training distribution and adequate generalization to unseen food. Comparison to the basline and the ablations show the importance of using both modalities and using reactive planning. ", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Robot-assisted feeding is definitely an important application so the paper is well-motivated. The idea of combining visual and haptic information makes a lot of sense for this application.\n- The experimental results are strong and pretty complete in my view. The system achieves good in-distribution and out-of-distribution performances . The failure cases are also shown which are appreciated. The comparison to baselines and abalations are well conducted and helped understand the contribution of each component of the method.\n- The paper is clearly written and I enjoyed reading it. \n\nWeakness:\nI mostly have some minor comments:\n- Can you provide more details regarding how the ServoNet is trained? Is the label just delta movement of the end-effector? \n- What does '-' mean in Table 1? Does it mean 0 success or no experiment is conducted? If the latter, why?\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think the proposed idea of combining visual and haptic information for robot assisted feeding makes a lot of sense, and the experimental results are impressive and strong and supports the proposed idea.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please see the weakness section as above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "f8oGIZ8PSas", "reviewer": "Reviewer_jNun", "summary": "This paper presents a system for food skewering by considering both visual and haptic information. Specifically, the authors propose a neural network-based framework that fuses the multimodal information during a single skewering interaction to learn food representations (via supervised learning), informing the food item's geometry and deformation properties. The model is learned by mapping visual and haptic information to a discrete set of skewering primitives using a small manually-labeled dataset collected in an offline fashion. The resulting system can then be used to skewer food items.\n\nThe authors have evaluated the proposed system on a diversified set of food items with varying geometry and deformability, achieving 71% skewering success across 21 items total. The experiment results also suggest that the consideration of multiple sensory modalities enables the proposed method to handle unseen objects and outperform those that only take single modality input.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**[Strengths]**\n\nThis paper tackles a very interesting problem of food skewering. The task is challenging as the robot has to deal with objects with different geometry and complicated physical properties, but this is also a necessary step for robot-assisted feeding. A satisfying solution to this task holds the potential to achieve great real-world impact of having robots caring for our elderly population or people with disabilities.\n\nThe authors have evaluated the proposed system on a diversified set of objects of different visual appearances and deformability. They have also provided both quantitative and qualitative evaluations demonstrating the effectiveness of the proposed method. The video demonstrations in the supplementary materials and the website are especially helpful for the readers to understand the difficulty of the task and the system's performance.\n\nThis paper also provides many very practical treatments for the design of different system components and discusses why the task is hard with concrete examples (e.g., distinguishing between raw and boiled carrots). A lot of the observations obtained through the experiments and discussions of the failure modes are also very interesting and tremendously helpful for the follow-up work to make further progress.\n\n**[Weaknesses]**\n\nMy primary concern with this work is that the action space is a bit limited. The authors mainly consider two types of pre-specified action primitives designed specifically for this task: angled skewering and vertical skewering. I understand that these two primitives may be enough for the food skewering tasks, but such constraint limits the method's applicability in other tasks. It is also hard to know how well the method can scale as the number of action primitives increases.\n\nThe authors also claim that the system learns a reactive policy. I thus hope the authors can explain in more detail how the feedback is taken during the task execution process and the frequency of the feedback loop. Correct me if I am wrong, but it seems that the robot does the task in a two-step process: probe, then skewer, where both steps are executed in an open-loop fashion. Some of the videos also show that even if the robot fails to skewer a food item, the robot continues to perform the scoop motion, indicating that no feedback is considered during this stage. Will feedback at a higher frequency help here? I'm curious to know the authors' perspectives on this and how to incorporate such visuomotor closed-loop ability in the system.\n\nOne thing missing from the experiments is a more comprehensive evaluation of how much data is needed and how the data's size and diversity influence the performance and generalization abilities. Such experiments can be essential to understanding the sample efficiency and the applicability of the method.\n\nMeanwhile, for generalization to unseen objects, are there ways to quantify the distance between the unseen objects and the training distribution (e.g., according to the shape and stiffness)? It would be helpful for the readers to understand to which extent can the learned system generalize and how the system behaves as it encounters less and less familiar objects.\n\nI could also imagine that the labeling of the objects can be ambiguous. For example, some food items with deformability between the banana and the broccoli may be successfully skewered via either angled or vertical skewering actions. Do the ambiguities in labeling matter during the learning process, and how do such ambiguities be resolved in the pipeline? Will a more continuous action representation help in this case?\n\nIn Figure 4, Cumulative Failures, why does HapticVis (the proposed method) perform worse than Haptic and Vis alone in BBox FP?\n\nThe first contribution stated in the introduction discussed a coarse-to-fine visual servoing method to approach a food item. However, no mention of it is found in the remainder of this paper. If such a coarse-to-fine visual servoing method is not the main focus but a contribution of another paper, I suggest the authors rephrase the text to highlight the unique contribution made by this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I think this is a well-executed paper tackling an important problem of food skewering. The idea of fusing multiple sensory modalities (i.e., haptic and vision) is also interesting and has shown to be effective in this specific task. The authors have also provided extensive evaluations in the real world with food items of diverse visual appearance and physical properties. I thus lean towards the acceptance side.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I have concerns regarding\n- the limited action space,\n- details about the reactive/feedback process,\n- more comprehensive evaluations of the amount/distribution/diversity/ambiguity of the training data, and\n- certain claims about the contribution of this paper.\n\nPlease see the weaknesses section for more details.\n\nIn addition, why are there missing numbers in Table 1 (slots marked with \"-\")? Is it because the specific methods are not applicable in the testing scenario, or did the authors not get the chance to run the experiments?"}}, {"review_id": "7C4R6tAyRmg", "reviewer": "Reviewer_NczR", "summary": "In the paper, the authors introduce a new network that incorporates both haptic and visual data in order to determine the best fork skewering skill parameters after first probing the item. It differs from previous work that relied solely on visual data to determine fork skewering locations. They ran many studies on both unseen and seen food on 6 different plates of food items where sometimes the food items were of the same type but some of the food was boiled so they were visually similar but had vastly differing properties. Their network improved on previous skewering networks and they tested the success results of the different ablations on the real robot.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths of the paper is that they were able to train this Haptic Visual Net using only human annotated data that required only 3 hours of data collection and labeling and didn't need to waste time to have the robot try actual skewering attempts and then a human would have to reset a scene and then clean the fork. That was actually very surprising to me. The original SPANet required 81 hours of human labeling or robot data collection according to their paper. While I have not read that paper, it seems credible in my experience of doing my own robot experiments. I also believe that sometimes if a human can give sufficient labels that are accurate enough, then trying to do the experiments with an actual robot will take more time.\n\nI only wish that the authors tried a couple more different varieties of hard and soft objects of the same category. For example, frozen fruits vs regular room temperature fruits. They seemed to only do boiled carrots. The boiled carrots and pasta as unseen was very interesting. Maybe for future work, they can try running the network on cooked items like sauteed chicken with vegetables in more cluttered plates than the much less dense plates that were used for evaluation. If the robot were to skewer multiple thing at once because they were overlapped, what would Haptic Visual Net do in case some items are not skewered well enough and they were to fall on a person and unfortunately hurt them if they were hot? Also what would be the effect of sauces be on networks like if they were charred (soft) vs. dark soy sauce on the objects. Would they be classified similarly? I hope that people that might need the robot don't just have to eat plain vegetables without any sauce haha. Also as food items cool, they might change as well so maybe there should be in between classes instead of soft and hard. For example, chewy or sticky or something.\n\nFinally, I wonder if sometime it would be more useful if the robot could also use a bandit formulation of exploration vs exploitation to figure out whether all the food items on the plate are similar and whether to continue probing or whether it is making a lot of mistakes skewering to then probe the items more to figure out its properties. I wouldn't want a robot to waste time continuously probing if every item was as predicted.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I give this paper a strong accept. I think the well-written paper and accompanying video can be useful for the robotics community. While the problem of fork skewering is not novel, they were able to develop a new network using a probing action and the haptic feedback from the action instead of only relying on vision to determine the parameters of the fork skewering actions. I see no reasons why someone would reject the paper and would argue for my recommendation. I felt like the experiments were well executed and the results are in line with what I would hope for a personal assistive feeding robot. However, there could be more experiments with food items that are the same type but different properties like frozen fruits / vegetables or slightly sauteed items vs regular food items.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "I don't really have any issues with the paper. It is well written and the video is excellent as well. Overall, if you could address maybe some of the issues I brought up in the strengths and weaknesses section above that would be ok. For example, maybe doing some more experiments on frozen fruit vs normal fruit or sauteed / wok fried objects with sauces vs normal items. In addition, the food items on the plates were very separated. If there was a normal served plate of food where lets say rice or mashed potatoes is in one corner, the vegetables are in another corner, and there are some stir fried chicken with bell peppers or something, how would the Haptic Visual Net react to more cluttered plates and what new data would be necessary to increase the robustness of your method so that it would scale for real-world adoption?"}}, {"review_id": "0Y-1aiZIIBM", "reviewer": "Reviewer_b6eX", "summary": "The paper proposes a robotic system for skewering pieces of food from a plate. Taking visual and force data as input, a learned policy controls the robot to (1) detect pieces of food, (2) move the fork via visual servoing, (3) probe the food, (4) and select a primitive based on the measured probing data. The key contribution is the NN mapping the visual and haptic probing data to one of the two primitive types; they differ in the fork pitch angle intended for either hard or soft objects. The data (300 probing examples) was human annotated. Experimental evaluation was done on various foods and compares the approach with different input modalities as well as one baseline.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is nicely written and easy to follow. We like the application as well as the idea of probing the food to get very simple closed loop feedback. Its implementation, as seen in the video, seems fast and nearly unnoticeable. Moreover, the experiments study generalization extensively with a wide range of foods. While we think that the contribution itself is not particularly large, the sound implementation and the task might be interesting for the community.\n\nHowever, we think that the quantitative results (e.g. a 71% success rate) aren’t very strong. Therefore, we would like to see a more detailed ablative study. In particular, 300 annotations (in 3h of data collection) seems very few for a data-driven approach. How do the results improve with training data? Broccoli florets (plate 4) have a 61% success rate, despite being a trained food. Is this a matter of training data? What are the success rates for using only one primitive?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "We think that the originality of this work is incremental, as probing is a common solution in (robotic) measurement tasks. Moreover, the general ideas are only applicable to the task of skewering foods from plates.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "We suggest to address following comments (in no particular order):\n\nAfter playing around with various foods, we think that both primitives could be combined into a vertical skewer motion and a subsequent angled lifting with support from the plate. In this regard, we suggest (similar to the paragraph above) adding some ablative experiments about the action space. \n\nThe action space definition could be made clearer. We suggest including only the learned action space in eq. (1), and clarify that z is via depth image, delta z is fixed, delta beta are two discrete values only.\n\nSimilarly, the reward definition seems unusual. How many timesteps does an episode have? Isn’t the episode stopped after failing, so the loss would always sum up to one? Given the description, the system seems to actually learn a binary reward from the human annotation if the whole skewing primitive is successful.\n\nCode and dataset were not published during the time of review, so please do so now or remove their references from the paper. Of course, feel free to add them to the website anytime.\n\nWhy weren't the internal force sensors used? Moreover, we suggest giving a brief (one-sentence) overview in 4.2 about the used control software for the robot.\n\nWhy is the ServoNet necessary? In general, the absolute position accuracy of the Franka robot is below 1 mm so it should be able to move to a target position without servoing.\n\nWe suggest replacing several arxiv citations with their conference/journal counterparts.\n\nIn line 222, ”only” is doubled.\n"}}]

Author rebuttal:

Decision: Accept (oral)