Learning Bimanual Scooping Policies for Food Acquisition
Authors: Jennifer Grannen
Yilin Wu
Suneel Belkhale
Dorsa Sadigh
Venue: CoRL 2022 Poster (2022)
Keywords: Bimanual Manipulation
Food Acquisition
Robot-Assisted Feeding
Deformable Object Manipulation

Abstract: A robotic feeding system must be able to acquire a variety of foods. Prior bite acquisition works consider single-arm spoon scooping or fork skewering, which do not generalize to foods with complex geometries and deformabilities. For example, when acquiring a group of peas, skewering could smoosh the peas while scooping without a barrier could result in chasing the peas on the plate. In order to acquire foods with such diverse properties, we propose stabilizing food items during scooping using a second arm, for example, by pushing peas against the spoon with a flat surface to prevent dispersion. The addition of this second stabilizing arm can lead to a new set of challenges. Critically, these strategies should stabilize the food scene without interfering with the acquisition motion, which is especially difficult for easily breakable high-risk food items, such as tofu. These high-risk foods can break between the pusher and spoon during scooping, which can lead to food waste falling onto the plate or out of the workspace. We propose a general bimanual scooping primitive and an adaptive stabilization strategy that enables successful acquisition of a diverse set of food geometries and physical properties. Our approach, CARBS: Coordinated Acquisition with Reactive Bimanual Scooping, learns to stabilize without impeding task progress by identifying high-risk foods and robustly scooping them using closed-loop visual feedback. We find that CARBS is able to generalize across food shape, size, and deformability and is additionally able to manipulate multiple food items simultaneously. CARBS achieves 87.0% success on scooping rigid foods, which is 25.8% more successful than a single-arm baseline, and reduces food breakage by 16.2% compared to an analytical baseline. Videos can be found on our website at https://sites.google.com/view/bimanualscoop-corl22/home.

Learning Bimanual Scooping Policies for Food Acquisition
Jennifer Grannen Yilin Wu*, Suneel Belkhale, Dorsa Sadigh Stanford University, Stanford, CA {jgrannen, yilinnu edu
Abstract: A robotic feeding system must be able to acquire variety of foods. Prior bite acquisition works consider single-arm spoon scooping or fork skewer- ing, which do not generalize to foods with complex geometries and deformabili- ties. For example, when acquiring a group of peas, skewering could smoosh the peas while scooping without a barrier could result in chasing the peas on the plate. In order to acquire foods with such diverse properties, we propose stabilizing food items during scooping using second arm, for example, by pushing peas against the spoon with flat surface to prevent dispersion. The added stabilizing arm can lead new challenges. Critically, this arm should stabilize the food scene with- out interfering with the acquisition motion, which is especially difficult for easily breakable high-risk food items like tofu. These high-risk foods can break between the pusher and spoon during scooping. which can lead to food waste falling out of the spoon. We propose general bimanual scooping primitive and an adaptive stabilization strategy that enables successful acquisition of diverse set of food geometries and physical properties. Our approach, CARBS: Coordinated Acqui- sition with Reactive Bimanual Scooping, learns to stabilize without impeding task progress by identifying high-risk foods and robustly scooping them using closed- loop visual feedback. We find that CARBS is able generalize across food shape, size, and deformability and is additionally able to manipulate multiple food items simultaneously. CARBS achieves 87.0% success on scooping rigid foods, which is 25.8% more successful than a single-arm baseline, and reduces food breakage by 16.2% compared an analytical baseline. Videos can be found on website.
Keywords:
Bimanual Manipulation, Food Acquisition, Robot-Assisted Feeding, Deformable Object Manipulation
Introduction
Approximately one million people in the U.S. depend on a caregiver's assistance to eat [3], which can lead to malnutrition [4, 27] and an erosion of self-worth [23]. Building robotic feeding system would enable patients to eat food independently [25]. A key component of such an assistive feeding system is bite acquisition, i.c., the act of robotic arm picking up morsels of food from a plate for the goal of transferring the food to person's 's mouth [8]. Prior strategies for bite acquisition acquire food using a single robotic arm for either skewering with fork or scooping with spoon. Fork- based bite acquisition learns to select skewering primitive parameters from large supervised dataset of food items [1, 6, 11]. However, fork-based skewering inherently limited in what foods it can acquire. For example, a fork cannot skewer brittle cashews or small peas without damaging the food item, while scooping with a spoon might more successful. Existing single-arm spoon-scoophe acquisition often uses a hard-coded single-arm scooping primitive, making generalization to varied food items difficult [21, 22]. When scooping more diverse foods, such as large, rigid-body fruit cubes and deformable cottage cheese, prior works rely on hard-coded adaptation strategies new primitives and even new tools which are not scalable. Similar to fork skewering, the single-arm scooping strategy is also inherently limited. For unstable items like broccoli or blueberries, can be hard to know where exactly to scoop without pushing it off a barrier such as a fork or bowl wall.
bite
*Equal contribution.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Segiantttton
Food Geute:
Robust
RosNerSM
Fragiity
"Roleas!"
ResNetüi
Probling
Scooplng
Transfer
T1me
Robust Grupe
Segmentaation
Food Cenner
Thee NerSi
Fragile
Classilir
Food Finglitt
"Fruglie"
ResNetIt
Talture Claesiffer
"Keep Coing
"Keep Going"
"STOP"
Figure 1: Learned Bimanual Scooping: CARBS is a bimanual scooping system for foods with varied geometries and deformabilities. CARBS uses second arm stabilize food position during scooping. To avoid breaking deformable foods between the two arms, we learn a Risk Classifier and Failure Classifier to identify high-risk, fragile foods and breakage-imminent states respectively.
A
B
Scooping objects with unstable geometries or scooping multiple objects without the help of a barrier for example, chasing peas plate is nontrivial and inefficient. To develop a gen- eralizable scooping policy that work across foods with difficult geometries, we need to use an additional robot arm stabilize the food item. For example, primary arm should scoop peas against surface that is pushing towards the spoon to stabilize the pea positions (See Fig. 1). As humans we intuitively and regu- larly use two arms to stabilize the environment in many everyday tasks: when tying our shoes, we use second arm to hold the knot; simi- larly when cutting a steak we use a second arm to hold the steak to make the act of cutting eas- ier for first arm. As result, there has been
Brittle
Fragile
Figure 2: Bimanual Failures: Adding a second stabilizing arm during scooping can lead to new breakage failures for foods of varying deformabil- ities. For both brittle and fragile food items like cashews (A), tofu (B) and cheesecake (C), food pieces can become wedged between the two arms and break due to excessive force being applied be- tween the pusher and scooper. These breakages can cause food to fall out of the scooper (C), lead- ing to food waste and scooping inefficiencies. growing focus on bimanual manipulation for tasks including rope untangling, peg insertion, food cutting, fabric manipulation, bottle opening, and bag opening, sight that additional arm act in stabilizing manner [5, 9, 12, 24, 28, 29]. In bite acquisition, we posit second arm holding a pusher (as shown Fig. 1) can support a scooper in scooping objects with difficult geometries and deformabilities.
implicitly leveraging the same
However, adding a second arm for stabilizing the environment holding or pushing the food items opens the door to new set of complications and failures (see Fig. 2): A pushing arm needs to physically make contact with the food to stabilize it. This can easily break or deform food items during scooping and thus impede helpful for the barrier follow the food into the scooper so the food does not fall out the scooper finishes scooping, but this design comes at a cost. Fragile and deformable items like tofu or jello cubes can easily break when forced into the scooper by the pusher, which can also cause the item to fall off the scooper leave residue on the plate. As result, it is difficult to define a single hard-coded primitive to that generalizes to both unstable geometry and break- age failures. We posit that many breakage-prone, scenarios when the pusher and scooper are squeezing the deformable food together. We employ this idea when scooping high-risk foods by detecting "breakage-ine iimaege-mmmienn"" states and adjusting scooping policy to anticipate and prevent food breakage and waste.
task progress. For objects with unstable geometries like snow-
peas or macaroni,
"high-risk" foods will break under predictable
Our key insight is we need second arm to effectively stabilize food environments by identifying high-risk food items, i.e. the ones that are breakable and fragile, and adapting a dynamic stabilizing strategy anticipate and prevent such failures.

---

In this work, we propose a learned bimanual scooping policy, CARBS: Coordinated Acquisition with Reactive Bimanual Scooping, that uses scooper arm and pusher arm (similar to fork or knife) to acquire food items of wide variety of properties, including shapes, sizes, and deformability. CARBS first identifies high-risk food items by classifying a visual observation of the target food item as fragile or robust. Next, CARBS prevents breakage and dropping failures by servoing to adapt the stabilizing action parameter: the distance between the scooper and pusher. This paper makes the following contributions:
A Bimanual Scooping Primitive for Bite Acquisition. To our knowledge, we are the first work to study bimanual strategies for acquiring widely varied foods. We define novel bimanual scoop- ing primitive and show that it generalizes to scooping 14 food classes with varied geometries and deformabilities. Our primitive is also the first to scoop multiple food items per acquisition action.
Learning to Avoid Bimanual Failures. We contribute framework for anticipating and preventing bimanual failures by identifying high-risk scenarios and adjusting the stabilizing parameter accord- ingly. To prevent breakage failures during bimanual scooping, this entails identifying deformable, fragile foods and adjusting the distance between arms during scooping. When scooping high-risk foods such as cheesecake and tofu, we learn detect hazardous, breakage-imminnnt scooping states for closed-loop visual feedback to adjust our dynamic stabilizing policy.
Evaluating Learned Scooping Policies. We present physical experiments with CARBS, which learns to identify fragile foods and adapt a stabilizing parameter to execute a bimanual scooping action. We find CARBS is able to successfully generalize across food geometries and deformabilities to scoop 14 food classes without breakage in 85.7% of trials. We will also open-source our food fragility and breakage datasets and the pusher and scooper CAD models.
2 Related Work
Food Acquisition. Previous works have studied single-arm food acquisition with chopsticks [17, 18], skewering with a fork [1, 6, 8, 11], and scooping with a spoon [21, 22j. Ke el al. [18] study grasping set of household objects with chopsticks, but do not consider food objects in geometric and physical properties. Past works on fork-based bite acquisition take a step towards more general food acquisition by learning an optimal skewering policy from large dataset of food items [1, 6, 11]. However, both chopsticks and forks with their accompanying acquisition primitives may struggle to generalize to many food items. In particular, it would be difficult to acquire fragile or very small foods such as jello or peas with a fork or chopsticks because the foods could break or would require very precise acquisition strategies unforgiving of slight errors. Spoon scooping is a promising acquisition alternative for these foods. Ohshima et al. [21] consider scooping portions off a large block of deformable foods - tofu and pudding - with an analytical policy. However, this an- alytical single-arm scooping method is limited to food items of relatively homogeneous geometries and deformability. Park el al. [22] also use a single arm to scoop more varied foods (i.e. fruit cubes, cottage cheese) from bowl by asking a human end-user to select between three spoon tools of differing materials and shapes per food class, which is not scalable to the large universe of potential foods. In contrast, our learned bimanual scooping approach uses one scooper and one pusher tool to acquire foods of highly variable geometries and deformabilities off a plate.
variations
Bimanual Manipulation. In recent years, bimanual manipulation has enabled robots to perform new tasks using new motion primitives such as bag opening, flinging fabrics, cutting vegetables, and opening bottles [2, 5, 9, 12, 13, 16, 20, 29]. The extra mobility provided by an additional arm can take the role of stabilizing the environment to reduce non- non-stationarity and make the task easier. For example, past works have implicitly considered utilizing stabilizing arm for tasks including peg insertion, untangling ropes, and cutting foods [5. 9, 12, 24, 29]. This idea is very relevant to the food manipulation domain. Foods can have widely varied physical properties and geometries they can be deformable, brittle, slippery, and in unstable shapes and poses. As a result, it can be difficult to manipulate these objects due to the unpredictable dynamics of food movements and interactions.
Past bimanual food manipulation works have considered food preparation tasks - cutting and peel- ing vegetables, scooping out a melon, and mixing in a bowl. Food cutting or peeling works use a stabilizing arm to hold the food place during the cutting motions, but these stabilizing strategies are largely stationary or analytical [9, 29] and do not make additional task progress [7]. On the other hand, Ureche et al. [26] use a more versatile stabilizing strategy for melon scooping, zucchini peeling, and bowl mixing by learning bimanual interaction constraints from human demonstrations.
3

---

(A) Anglod Pushing: Pusher pushes at an angle
(B) Adaptive Cupping:
(C) Pianing:
Pusher cups food snto spoon mouth
Pusher pins food spoon
f Uf
Ppash
0.5
0.9
Pushing Phase
Scooping Phase
Figure 3: Bimanual Stabilizing Strategies: CARBS uses 3 stabilizing strategies: (I) Angled Push- ing and (2) Adaptive Cupping during the Pushing Phase, and (3) Pinning during the Scooping Phase. In Angled Pushing, the pusher moves at an angle 0 15 off the vertical, which encourages food items to roll off the barrier surface and into the spoon. In Adaptive Cupping, the pusher pushes foods towards the spoon with the concave surface and cups them to be centered with the spoon mouth. This strategy is parameterized by a learned parameter a that represents the scaling of the distance travelled by the pusher. In Pinning, the pusher moves upwards to follow the spoon mouth as it scoops to prevent food items from toppling and falling out.
For melon scooping, the stabilizing arm holding the melon is able to adjust its force to brace against the scooping tool to better stabilize the melon's 's position and even progress towards task success by pushing more melon into the scooper. However, this work is only specialized for scooping a melon and does not consider a generalizable policy for the large variety of food items necessary for a scal- able robotic feeding system. Our method similarly uses a dynamic stabilizing strategy that pushes food towards the scooper to both stabilize food position and make more task progress. However, our method is able to learn to directly sense states close to bimanual constraint violations (food breakage failures) from visual feedback, and bypasses the need for expensive human demonstration collection. Our method is also able to learn these constraints in food-agnostic way and generalizes to visually varied, out-of-distribution food classes.
3 CARBS: Coordinated Acquisition with Reactive Bimanual Scooping
We consider the task of scooping a variety of food items off a plate while maximizing the integrity of the food item, i.e., the weight, after the scooping motion. Food items may vary in geometry and deformabilities. including foods that are brittle (i.e. cashews), compliant (i.e. pasta), and frag- ile/breakable (i.e. tofu). We assume access to plate workspace with standard (x,, y, =) coordinate frame (as Fig. 1). We assume full bimanual access to this plate workspace with the following two mounted tools: Scooper and Pusher (See Fig. 4). The Scooper tool is a plastic spoon mounted at an angle to the robot end effector with a camera mounted for access to angled images I E W XWxHxx HxC of the spoon and surrounding workspace. The Pusher tool is concave barrier that is mounted vertically to the robot end effector'.
We model the bimanual scooping task as Partially Observable Markov Decision Process (POMDP) M (O, S,A, A, T.R). R). We observe images I E O of the unknown food environment states S E S and define an action space A as joint 14-DoF robot actions (a", a²). The visual state space is an RGB image observation space O E IR T We assume unknown transition dynamics T A an initial state distribution of food configurations Po. and a time horizon T. We define a reward R : S x A - IR. as the weight of the food in the scooper after scooping. We aim to construct a closed loop policy π. O A to maximize the expected reward by successfully scooping set of varied foods.
S,
To scoop a large diversity of foods with two arms without dropping or breaking them, we intro- duce CARBS, reactive bimanual scooping policy learned from real food interactions (Fig. 1). We simplify the complex bimanual action space by introducing novel bimanual primitive, which is parameterized by the distance of travel for the pusher and the location of the food item, and em- ploys three bimanual stabilizing strategies: (1) Angled Pusher, (2) Cupping Motion, and (3) Pinning Motion (shown in Fig.3). We show that this parameterization generalizes to robust food items (e.g., grape) by selecting large distance of travel, as well as for breakage- prone items (e.g., tofu) which require more adaptive pusher travel distances. To handle the latter case of breakage-prone items,
This design is inspired by antique pushers that were used push foods into spoons.

---

CARBS learns to adapt the pusher travel distance by anticipating common types of failure. Our policy network learns to identify which regime we are in (robust or breakage-prone) based on just the initial plate image.
A Bimanual Scooping Primitive. We define parameterized bimanual scooping primitive that takes two inputs, pusher travel distance a and food position (r), and has three phases to be per- formed in succession: (1) Pushing, (2) Scooping, and (3) Food Transfer. We thus reduce the 14 DoF action space A to 3 dimensions: pusher travel distance a and food position (xy,3r), The scooper and pusher begin in starting positions centered around the food position along fixed pushing axis. We empirically found that the choice of pushing axis did not affect performance, so we select the rraxis to favor our robots' range.
During the Pushing phase, both the pusher and scooper move towards each other along the r-axis towards some point Ppush = (. Uy) closer to the scooper than pusher, as shown in Figure 3.A-B. The Pushing phase also utilizes two bimanual stabilization strategies: Angled Pusher and Cupping Motion (See Fig. 3.A-B). In Angled Pusher, the pusher is angled at some fixed angle 0 15 about the y-axis, rather than orthogonal to the plate. This stabilizes the food position by encouraging the food slide or roll into the scooper at the end of pushing when the two arms meet, as shown in Fig. 3.A. In Cupping Motion, the pusher pins the food against the concave surface as it moves towards the scooper, which promotes centering the food position during movement into the entrance of the spoon (see Fig. 3.B). Cupping Motion is an adaptive stabilizing strategy that takes as input the primitive imput a E [0, 1]. which determines how close the pusher and scooper get each other. = 0 implies no motion and a = 1 results in the pusher and scooper reaching each other.
After the Pushing phase, we assume CARBS has successfully manipulated the food item into the scooper and move to the Scooping phase. In this phase, we rotate the scooper up about the axis to "scoop" the food into the bowl of the spoon. During the Scooping phase, CARBS employs the Pinning Motion stabilizing strategy where the pusher moves up along the 2 axis with the scooper as it rotates (shown in Fig. 3.C). This strategy prevents foods in unstable poses within the spoon from falling out of the scooper. Lastly, CARBS finishes with the Food Transfer phase by moving the pusher away from the scooper and rotating the scooper towards an end user to prepare for feeding.
Identifying High-Risk Settings. To learn the inputs of the bimanual scooping primitive (ry, Mr) and from image observations I E O, CARBS leverages the insight that foods with similar de- formabilities encounter similar breakage failures, and posits that learning to identify high-risk set- tings, e.g. robust vs. breakage-proot high variability of food dynamics, we do not learn a dynamics model of our POMDP M and in- stead learn parameters to perform adaptive scooping with our previously-definel bimanual scooping primitive, which performs each of the three phases in succession.
foods, will help determine the optimal inputs. Due to the
Given an initial overhead image observation Lo. CARBS starts by identifying the food position in the initial environment state so- To do this, we learn a segmentation model f : RW xHXC - RW*H to obtain a food mask and food center position yy) from the initial image In. which is then passed to our bimanual scooping primitive. As illustrated in Fig. 1, to differentiate between food deformabilities, we learn a Risk Classifier r Wxxxxx - {0. 1) that identifies food image In as "Robust" or "Fragile". In practice, we instantiate the classifier with a ResNet34 model [14] trained on a hand-labelled food fragility dataset with 600 images of 14 food classes. For robust foods, we set a = 1 for maximum pushing of the food, since the food is not in danger of breaking and can benefit from the added pushing stabilization. For fragile foods, it is nontrivial to select a given only an initial observation Lo. so we propose a closed loop system for determining a.
initial
Servoing for Fragile Foods. At the beginning of each scooping rollout, we initialize cating that the pusher should travel the full distance during the Pushing phase. However, varied food dynamics and differences in deformability can dictate the need for different a values even within the same food class. For example, when scooping two pieces of tofu, the food items may deform or slide on the plate differently due to food interactions with the plate, slight robot imprecision, or food shelf life. As a result, CARBS uses closed loop visual feedback in the form of a Failure Classifier : R XXC + {0, 1} that identifies breakage-imminent states where the food item is in contact with the pusher and scooper, but not yet squeezed until breakage. This classifier is run at each state during Pushing (as in Fig. 1). When a greakage-imminent state is detected, the Pushing phase is terminated and a < 1. For example, if a wreakae-imminnen state was detected after 65% of the Pushing phase had completed, the phase would terminate early and this would correspond
1, indi-
5

---

Scooper
Pusher
Robust Food Classes
Carrot
Strawberry
Macaroni
Farfalle
Snow Peas
Broccoli
Grape
Blueberries
Goldfish
Cashews
Peas
Fragile Food Classes
Tofu
Jello
Cheesecake
The Pusher is custom concave barrier used to push food items to the mouth of the spoon. The Scooper has an RGB camera mounted above the spoon to access images of the spoon mouth and surrounding workspace during scooping. Right: We consider scooping 16 food settings: robust containing up to three food items, and fragile.
Figure 4: Experimental Setup: Left:
to 0.65 because the pusher had traveled 65% of the full pushing distance. We instantiate the Failure Classifier with a ResNet34 [14] model trained on hand-labelled images of 30 scooping trials.
4 Experiments
We validate CARBS's effectiveness on scooping foods of varied geometries and deformabilities. We design series of experiments scooping 14 food items to demonstrate the advantage of reactive bimanual strategy over hard-coded or single-arm actions. We select 14 food items to cover wide range of sizes, shapes, and deformabilities: blueberry, broccoli, carrot, cashew, cheesecake cube, farfalle pasta, goldfish, grape, jello piece, macaroni pasta, pea, snow pea, strawberry and tofu cube (See Fig. 4). See Appendix A for food property details. We assume all food groups are pregrouped, where each food item is in contact with other items, if any, in the scene. Each set of food properties comes with unique set of challenges. For example, blueberries, peas, and grapes round, which may roll around the plate, while snowpeas and macaroni are irregularly shaped and can be difficult to stabilize within the spoon. We consider three deformable foods with diverse visual and material properties (jello, tofu, and cheesecake cubes), which are critically susceptible to breakage failures.
Success Rate
Food Dataset. The Segmentation model is trained on a dataset of 600 overhead RGB images of all food items except Orange Jello in real workspace. We subtract background image of the workspace obtain masks of the food items. The Risk Classifier is trained of overhead RGB images of food items. We hand-label each image or "Fragile", and augment 8X. The Fail- Classifier trained on images from scooping rollouts of lоfu only, which we found to be sufficient for generalization other food classes as well. More train- ing details are in Appendix We collect a dataset by recording 60 image frames each of the Pushing phase of 30 rollouts with 1, meaning we push the food the maximum distance. We then hand- label when the food item breaks in each rollout, and automatically generate labels per image as "Keep Going" or "Stop" for safe and breakage-imming states respec- tively.
Food Type
Single
CARBS
Broccoli
5/5
5/5
5/5
Grapes
3/5
5/5
5/5
Blueberry
4/5
5/5
5/5
Strawberry
5/5
5/5
5/5
Carrot
4/5
5/5
5/5
the same dataset
Farfalle
5/5
5/5
5/5
Macaroni
2/5
5/5
5/5
"Robust"
Snow Pea
3/5
4/5
4/5
Cashews (2)
2/10
7/10
7/10
Goldfish (2)
6/10
8/10
8/10
Blueberries (2)
6/10
6/10
6/10
Peas (3)
3/15
15/15
14/15
Table 1: Robust Food Physical Results: We report the per food item success rate over trials of scoop- ing robust foods with the Single, strategies. As expected, we observe CARBS matches a performance across all robust foods. Both the CARBS and a = methods match or outperform the Single baseline, suggesting that the bimanual stabiliz- ing strategies (Angled Pusher, Cupping Motion, and Pinning Motion) are advantageous over static barrier.
1, and CARBS

---

Implementation Details. Our real-world environment setup consists of two 7-DoF Franka Emika Panda arms, each holding a Scooper or Pusher tool as shown in Fig. The robot bases are set to be parallel with each other with the plate workspace between two bases, and both robots are controlled with an impedance controller. The Scooper is mounted at 45 degree angle with an angled Intel Realsense D435 camera above the spoon as shown in Fig. 4. We designed a custom 3D printed pusher with a concave surface to encourage food grouping and stabilization. See Appendix C for a discussion of design choices for the pusher and the spoon. Our failure classifier to servo for breakage runs at 20 Hz frequency.
Avg.
Weight Difference (%)
Food Type
Value
Single
0.93
= 1
CARBS
Tofu
23.194
41.099
2.474
0.444
0.9477
Red Square Jello (Failure OOD)
40,700
21.376
1.076
0.420
0.9400=0.022
Cheesecake (Failure OOD)
26.873
25.937
10.867
6.639
0.923
Orange Triangle Jello (CARBS OOD)
100
62.274
9.720
0.449
0.9169=0.030
Table 2: Fragile Food Physical Results: We report the weight loss of food items after scooping as a percentage of the original food weight, averaged across trials. We also report the average values of CARBS's stabilizing parameter a, scaling value for the total pushing distance (13 em) with a 95% confidence interval. We observe that CARBS's Failure Classifier adjusts a to end pushing at different position depending on the food. This suggests that the classifier learns recognize the bimanual constraint that leads to breakage rather than a fixed ending position for all fragile foods. We also find that the triangle jello has the highest a variability, possibly due to its irregular shape relative to the cube foods. We observe CARBS's ' Failure Classifier generalizes to novel fragile food classes (Jello, Cheesecake) and its Risk Classifier generalizes to varied visual appearances and geometries within one class (Orange Jello).
Baselines. We compare against three baselines: Sin- gle, a- = 0.93 and a = 1. Single executes single-arm scooping method where the pusher is fixed and acts as a static barrier. The spoon moves towards the food and pushes the food against sta- tionary pusher during scooping. Notably, the pusher does not use any of the bimanual stabilizing strate- gies shown in Fig. 3. a = 0.93 executes a biman- ual scooping primitive without an adaptive stabiliz- ing strategy where the pushing distance is 0.93 of the total 13em pushing distance, stopping approxi- mately Icm early to prevent breakage. a = 1 exe- cutes bimanual scooping primitive instead with the full 13cm pushing distance (ax = 1), which is iden- tical to the primitive for scooping robust food item.
30
Single
25
CARDS
# Total Total Occureences
20
15
10
Not Enter
Roll Falluee Mode
Fall
Break
Figure 5: Failure Modes: We observe fail- ure modes across scooping strategies: (Not Enter) foods contact the scooper but do not enter the spoon bowl, (Roll) foods roll out of scooping range, (Fall) foods fall out of the spoon after being scooped into the spoon, and (Break) breakage. There are many (Not Enter) failures with the Single baseline be- cause irregularly shaped foods and multiple items are difficult to roll into the spoon with static pusher. We also find both basclines have a higher occurrence of breakage fail- ures, supporting the need for an adaptive sta- bilizing strategy.
Results. We compare our method, CARBS, against two basclines across 14 different food items, with 11 robust foods and fragile foods. We also include an additional a 0.93 baseline for the 4 fragile foods, which is the average a value across all CARBS frag- ile food trials. For robust foods, we report binary scooping success as whether the food ended within the spoon bowl after scooping in Table 1. We con- sider settings of single foods of varied geometries, and additional settings of up to food items. As expected, CARBS and a = 1 have similarly high performances because CARBS should learn set 1 for Robust foods. We compare to the Sin- gle baseline to observe the advantages of the three bimanual scooping strategies (Angled Pushing, Cupping, and Pinning as in Fig. 3) over a static pusher position. For round foods, the Angled Pusher stabilization (described in Sec. 3) is important for building momentum and helping the item roll into the scooper, which prevents the food from rolling away (Roll Failure). For irregularly shaped foods that often extend past the spoon edges, the food items can be in unstable poses even once in the spoon bowl, which can cause them to fall out during the Scooping phase. These items benefit
7

---

from the Pinning stabilizing strategy to pin the food in place throughout the rotation motion and prevent them from falling out (Fall Failure). See Appendix for an ablation study on the stabilizing strategies.
Lastly, we consider scooping 2-3 food items simultaneously, as inspired by the example of chasing peas around a plate. While CARBS and the a 1 baseline still outperform the Single baseline, they fail achieve as high success as scooping a single food item. more difficult stabilize multiple food items simultaneously due to the added dynamics complexity. For example, stabilizing two blueberries to ensure both roll into the mouth of the scooper is nontrivial (Not Enter Failure). The food dynamics also become more complicated as multiple foods can interact not only with the pusher and scooper tools, but also with each other.
We present experiments scooping four fragile food settings in Table 2. Two food settings are out of distribution for our Failure Classifier and one is out of distribution for both the Failure and Risk Classifier. We report the weight loss during scooping as a percentage of the original food weight to measure the breakage failure severity. CARBS is able to reduce food breakage by 16. com- pared to the a breakage-imminent states and adapt the stabilizing parameter a prevent breakage. We also find that a = 0.93 baseline, with a fixed early stop of Icm to prevent breakage, has worse performance than CARBS because the fixed distance cannot adapt to different food geometries and properties. Some smaller foods fail to fully enter the spoon bowl due to the early stop and are not scooped, resulting in 100% weight loss. We note that although the average a value for cheesecake is 0,9231, there is still large gap in performance between CARBS and the a = 0.93 baseline because CARBS is able to adjust ar for each food item. We additionally report the 95% confidence inter- vals for the CARBS a values to highlight this adaptability. We note that the weight difference for CARBS for cheesecake, while still lower than the Single and a = 1 baselines, is significantly higher than tofu and jello. This is due to the stickiness of the cheesecake and its propensity to leave food residue on the plate and tools during movement. We also report the a values learned with CARBS and show that although our Failure Classifier is only trained on one food class (tofu), CARBS is able to adjust a across novel food classes depending on their shape, size, and deformability. This supports our claim that learning to detect failures from vision generalizes across breakage-pros prone food classes. We also find CARBS's Risk Classifier generalizes within a food class to varied visual appearance and geometries, suggesting the effectiveness of CARBS for scooping novel foods.
baseline. This suggests that the Failure Classifier can effectively recognize
5 Discussion
Summary. We present CARBS, a learned bimanual scooping policy for robustly scooping food items of varied geometries and deformabilities. CARBS learns dynamic stablizing strategy to avoid breakage failures when scooping high-risk foods by identifying breakage-imment and adjusting the stabilizing action parameter: the distance between the scooper and pusher. We evaluate the generalizability of CARBS with physical experiments scooping 14 foods of varying shapes, sizes, and fragility, and compare against two baselines. We find that CARBS is able to successfully scoop 85. 3.% of foods.
states
Limitations and Future Work. CARBS struggles to scoop foods with uncommon material proper- ties and complex dynamics, and multiple food items. While our system is able to reduce cheesecake breakage compared to baselines, it does not achieve similar success compared to other fragile foods due to the cheesecake's 's stickiness. CARBS also struggles when scooping multiple blueberries with unpredictable dynamics. Their round shape and inertia allow them roll off not only the scooper and pusher, but also each other. CARBS leaves room for improvement when scooping multiple food items as well it is nontrivial to determine an optimal stabilizing policy for multiple items at once. In future work, we hope to study more dynamic stabilizing strategies for food acquisition and other bimanual tasks, such as tying knots and buttoning clothes. We plan to relax our food environment assumptions and scoop an even larger range of foods, for example by pushing to group multiple scattered peas on cluttered plate and then scooping into spoon. These cluttered food settings require longer horizon planning using potentially new primitives to group then acquire the food, which we leave to future work. We will also explore multimodal sensing strategies with new probing primitives for scooping to generalize to more unseen foods and augment our vision-only system.
8

---

Acknowledgments
This project was sponsored by NSF Awards 2132847, 2006388, and 1941722, and the Office of Naval Research (ONR). Jennifer Grannen is further grateful to be supported by an NSF GRFP. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. We additionally thank our colleagues who provided helpful feedback and suggestions, especially Priya Sundaresan.
References
[1] T. Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa, "Towards robotic feeding: Role of haptics in fork-based food manipulation," IEEE Robotics and Automation Letters, vol. 4, no. 2, pp. 1485-1492, 2019.
[2] A. Billard, S. Mirrazavi, and N. Figueroa, Learning for Adaptive and Reactive Robot Control:
A Dynamical Systems Approach. MIT Press, 2022.
[3] M. W. Brault, "Americans with disabilities: 2010," Current Population Reports, vol. 7. ppp 70-
131, 2012.
[4] D. R. Buys, K. L. Flood, K. Real, M. Chang, and J. L. Locher, "Mealtime assistance for hos- pitalized older adults: A report on the spoons volunteer program," Journal of Gerontological Nursing, vol. 39, 9, pp. 18-22, 2013.
[5] Chitnis, S. Tulsiani,
Gupta, and A. Gupta, "Efficient bimanual manipulation using Iearned task schemas," in Proc. IEEE Int. Conf. Robotics and Automation (ICRA), 2020.
[6] R. Feng, Y. Kim, G. Lee, E. Gordon, M. Schmittle, S. Kumar, T. Bhattacharjee, and S. Srinivasa, "Robot-assisted feeding: Generalizing skewering strategies across food items on a plate." Int. S. Robotics Research (ISRR), 2019.
[7] N. Figueroa and A. Billard, Learning complex manipulation tasks from heterogeneous and unstructured demonstrations, In Proceedings of Workshop on Synergies between Learning and Interaction. IEEE/RSJ International Conference Intelligent Robots and Systems, 2017.
[8] Gallenberger, T. Bhattacharjee, Y. Kim, and quisition: Analyzing manipulation strategies for robotic feeding," in 2019 14th ACM/IEEE International Conference on Human-Rone Robot Interaction (HRI), 2019, pp. 267-276
Srinivasa, "Transfer depends on ac-
[9] M. Gemici and A. Saxena, "Learning haptic representation for manipulating deformable food objects," in IROS, 2014.
[10] E. Gordon, S. Roychowdhury, T. Bhattacharjee, K. Jamieson, and S. Srinivasa, "Leverag- ing post hoc context for faster learning in bandit settings with applications in robot-assisted feeding," " in 2021 IEEE International Conference on Robotics and Automation (ICRA), 2021.
[11] E. K. Gordon, X. Meng, T. Bhattacharjee, M. Barnes, and S. S. Srinivasa, "Adaptive robot- assisted feeding: An online learning framework for acquiring previously unseen food items." in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 9659 9666.
[12] J. Grannen, Sundaresan, B. Thananjeyan, J. Ichnowski, A. Balakrishna, M. Hwang, V. Viswanath, M. Laskey, J. E. Gonzalez, and K. Goldberg, "Untangling dense knots by learning task-relevant keypoints." in Conf. on Robot Learning (CoRL), 2020.
[13] Ha and Song. "Flingbot: The unreasonable effectiveness dynamic manipulation for cloth unfolding." in Conference on Robotic Learning (CoRL), 2021.
[14] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Pro- ceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770- 778.
[15] A. B. Jung, K. Wada, J. Crall, S. Tanaka, J. Graving, C. Reinders, S. Yadav, J. Banerjee, G. Vecsei, A. Kraft, Z. Rui, J. Borovec, C. Vallentin, S. Zhydenko, K. Pfeiffer, B. Cook, I. Fernández, F.-M. De Rainville, C.-H. Weng, Ayala-Acevedo, R. Meudec, M. Laporte, er al., imgaug, atttps://gitiie httcchuu.ooomaaaaee com/aloeeaeeeeee aalaaalleggaaaau Online; accessed 01 F1-Feb-2020, 2020.
[16] S. Kataoka, S. K. S. Ghasemipour, D. Freeman, and I. Mordatch, "Bi-manual manipulation and attachment via sim-to-real reinforcement learning." arXiv preprint 2rXiv:2203.08277, 2022.

---

[17] L. Ke, A. Kamat, J. Wang, T. Bhattacharjee. C. Mavrogiannis, and S. S. Srinivasa, "Telema- nipulation with chopsticks: Analyzing human factors in user demonstrations," International Conference on Intelligent Robots and Systems (IROS), 2020.
[18] L. Ke, J. Wang. T. Bhattacharjee, B. Boots, and S. Srinivasa, "Grasping with chopsticks: Combating covariate shift in model-free imitation learning for fine manipulation." 2021 IEEE International Conference Roboties and Automation (ICRA), 2021, pp. 6185-6191.
[19] D. Kingma and J. Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.698% 2014.
[20] S. S. Mirrazavi Salehian, N. Figueroa, and A. Billard, "Coordinated multi-arm motion plan- ning: Reaching for moving objects in the face of uncertainty," in Proceedings of Robotics: Science and Systems, AnnArbor, Michigan, Jun. 2016.
[21] Y. Ohshima, Y. Kobayashi, T. Kaneko, A. Yamashita, and H. Asama, "Meal support system with spoon using laser range finder and manipulator," in 2013 IEEE Workshop on Robot Vision (WORV), 2013, pp. 82-87.
[22] Park, Y. Hoshi, P. Mahajan, H. K. Kim, Z. Erickson, W. A. Rogers, and C. C. Kemp, "Active robot-assisted feeding with a general-purpos mobile manipulator: Design, evalua- tion, and lessons learned," Robotics and Autonomous Systems, vol. 124, p. 103 344, 2020,
[23] I. Perry.. "Assisted feeding," Journal of Advanced Nursing, vol. 62, no. 5, pp. 511-511, 2008.
[24] L. Shao, T. Migimatsu, and J. Bohg, "Learning to scaffold the development of robotic manip- ulation skills," " in 2020 IEEE International Conference on Robotics and Automation (ICRA),
IEEE, 2020, pp. 5671 5671-5677.
[25] C. Stanger, C. Anglin, W. Harwin, and D. Romilly, "Devices for assisting manipulation: A summary of user task priorities," IEEE Transactions on Rehabilitation Engineering, vol. 2,
no. 4. pp. 256-265, 1994.
[26] L. P. Ureche and A. Billard, "Constraints extraction from asymmetrical bimanual tasks and their use in coordinated behavior," Robotics and Autonomous Systems, vol. 103, pp. 222-235, 2018.
[27] A. Westergren, S. Karlsson, P. Andersson, O. Ohlsson, and I. R. Hallberg, "Eating difficulties, need for assisted eating, nutritional status and pressure ulcers in patients admitted for stroke rehabilitation," J. Clin. Nurs., vol. 10, no. 2, pp. 257-269, 2001.
[28] Z. Xu, C. Chi, B. Burchfiel, E. Cousineau, S. Feng, and S. Song, "Dextairity: Deformable manipulation can be breeze," in Proceedings of Roboties: Science and Systems (RSS), 2022.
[29] Zhang, M. Sharma, M. Veloso, and Kroemer, "Leveraging multimodal haptic sensory data for robust cutting," in 2019 IEEE-RAS 19th International Conference on Humanoid Robots (Humanoids), 2019, pp. 409-416.
10


Reviews:
[{"review_id": "xhwsDKrju2b", "reviewer": "Reviewer_1gVW", "summary": "This work focuses on bimanual scooping of single instances of that would be difficult to skewer or scoop using traditional techniques due to their unique geometries, i.e. objects that are small, brittle, fragile, or those that would roll away if scooped with a single arm. To handle these problematic foods, a second arm with a pushing attachment is used to stabilize, collect and push the food onto the scooping attachment on the other arm. Three primitives are defined; angled pushing, adaptive cupping, and pinning. Adaptive cupping has a variable parameter, \\alpha, that defines how close to the scooper the pusher gets, as well as the location of the center of the food item. These parameters are learned using a convolutional network, based on images of the food. The centers and segmentation of each food are predicted as well as whether the food is robust or fragile. If it is fragile, a breakage detection prediction is run online and the food is pushed until imminent breakage is detected.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n_Novel Solution to an Important Problem:_\n\nThis is a novel solution to a previously unsolved problem. Further, this is a problem that is applicable to people who use robotics feeding systems. The solution was fully implemented into a working system that was tested on real food. \n\n_Simple and Effective Solution:_\n\nThe solution uses simple primitives that are adapted to individual food instances. The pusher design and primitives leverage funneling behaviors that reduce uncertainty in the position of the food, improving the probability of success. Further, relatively simple representation of the food allows the systems to generalize across types of food. The method shows significant improvement over the static baseline, showing that some dexterity on the second arm is necessary. Further, the closed loop stopping behavior greatly reduces food breakage.\n\n**Weaknesses**\n\n_Learning components are relatively limited:_\n\nThe learned components of this method are relatively limited. They amount to a food segmenter and centroid predictor, food robustness classifier, and a food breakage predictor. While these are sufficient to complete this task with a high degree of efficiency, they do not showcase the need for learning in robotics tasks.\n\n_Could use more complex baselines:_\n\nThe baselines were somewhat limited, with a static pusher, a constant \\alpha = 1, and the proposed method. A method that used the predicted centroid of the food, stopping when the centroid was over the spoon, could show the need for a breakage specific online policy. Additionally, a lower \\alpha baseline, or set of constant \\alpha baselines could show that no single \\alpha value works across food types.\n\n**Minor Weaknesses**\n\nThe parameter in of the angled pushing primitive is described but never defined. How is \\theta selected? Is it constant for all foods? The spoon angle is described in the supplemental material but not the pusher angle.\n\nMost food acquisition tasks are done on relatively cluttered plates. Some experiments showing robustness to this or mention of it in the limitations section should be added.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This work shows a simple application of learning to solve an important, previously unsolved problem. While the learned components are relatively simple, I believe that they are sufficient to solve the problem. Additionally, the work shows a full system, working with real word food, in a relatively realistic manner (minus the lack of clutter). If the conference would like to showcase more realistic systems, using some learning to solve relevant tasks, I believe this paper is an example of such work.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "The parameter in of the angled pushing primitive is described but never defined. How is \\theta selected? Is it constant for all foods? The spoon angle is described in the supplemental material but not the pusher angle.\n\nMost food acquisition tasks are done on relatively cluttered plates. Some experiments showing robustness to this or mention of it in the limitations section should be added.\n\nFigure captions are too close to text, particularly above line 207.\n"}}, {"review_id": "VaMcoSHlLvk_", "reviewer": "Reviewer_FpM9", "summary": "This paper introduces a bimanual scooping primitive that is capable of stably scooping bite-size food items with different geometry, color and deformation properties. In this case the stability of the scoop refers to avoiding breakage, rolling or cutting of the food item being scooped. The primitive, formulated as an MDP, takes as input an RGB image of a camera mounted on the scooping arm and selects the stabilizing strategy to perform the scoop either: angled push, cupping or pinning motions. For fragile food items an adaptive cupping strategy is introduced that leverages a “risk classifier” trained to identify high-risk breakage prone foods and a “failure classifier” trained to identify near-breakage states to adapt the cupping motion by scaling the distance traveled between the pusher and the scoop. Improved performance over single-arm scooping and baseline bimanual scooping (no active pusher) is shown. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is tackling an interesting and difficult problem, as food manipulation is not subject to a single form of deformation or failure, this paper seems to solve a range of issues that can arise from scooping food items with different properties.\n- Convincing videos and experiments.\n- Catchy acronym. \n\n**Weaknesses**\n\n- It is not entirely clear to me what is being learned in the MDP and how that data is acquired. In section 3 the MDP is introduced and states that the transition dynamics are unknown, if they are unknown then they must be learned or parametrized somehow? Also, first it is mentioned that the action space is 14-DoF but then it is mentioned that it is simplified by the bimanual primitive which has 3 phases, so is the new action space 3 dimensions? This is not really clarified in the main text or appendix. \n- The paper only mentions the learning pipeline for the risk and failure classifiers, however, shouldn’t there be a dataset where the robot learns the transitions between the different scooping phases/stabilizing strategies? For example, from angled pushing to cupping to pinning. It is unclear to me if this sequence or transitions between the phases are learned from the food interactions or if they are predefined and parametrized by the designers. How is the MDP deciding when to transition between phases?\n- Related to the previous point, the paper only gives details about the risk $r:\\mathbb{R}^{..}\\rightarrow$ \\{0,1\\} and failure classifiers  $f:\\mathbb{R}^{..}\\rightarrow${0,1} which both seem to map to a binary classification variable. It is not clear to me how the outputs of these functions are used to modulate $\\alpha$ which seems to be a continuous function based on the figures and videos. \n- It would be ideal if the authors included a baseline comparison to the work cited from [23] which learns the bimanual interaction constraints from human demonstrations. This would clarify the doubts related to what is being learned and what is assumed/predefined in this framework.\n- Is there a reason why adaptation to avoid breakage is only considered in the cupping phase? It would seem to me that breakage could be possible during the angled pushing and pinning phases as well - to avoid breakage or rolling there you would need to adapt the angle used by the pusher or modulate the speed of the rotation done by the pinning motion. \n- It would be interesting to see how the primitive performs when trying to scoop items that are slightly larger than the scoop itself, like 2 strawberries or a larger piece of jello or cheesecake, oftentimes in the kitchen we use tools that are not the appropriate size but as humans we adapt our motion to perform the task, could this form of adaptation be translated to robot through this adaptive scooping framework?\n- While the proposed primitive seems appropriate for scooping it would be great if the authors discussed how the presented strategy can be transferred to other types of tasks or if it can be adapted to handle other types of items that are not food but need a scooping mechanism like soil, sand, etc. \n- Details are lacking regarding the low-level control of the arms, is the control done purely in position mode with visual servoing? Why not use force/torque measurements? How is the simplified action space mapped back to the robot 14-DoF joint space and how are the desired end-effector commands controlled/tracked?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a novel approach to scooping food items with a bimanual arm setting including an active pusher and a scooper arm. The approach is capable of handling a variety of food items as demonstrated in the videos and shows improved performance over single-arm scooping and baselines. However, some issues need to be addressed regarding clarification of the framework, dataset generation and learning. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Comments listed in the weaknesses section should be addressed. ", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "PzrFYG7POwj", "reviewer": "Reviewer_aovx", "summary": "This paper proposes a method for scooping foods into a spoon in which both the spoon and a \"pusher\" are attached to separate robot arms. The authors show that allowing independent motion for the spoon and pusher significantly improves success rates vs a spoon-only strategy. The authors also use two neural networks as part of their control architecture: a risk classifier that decides whether to use a fixed pushing strategy for \"robust\" foods or an adaptive strategy for fragile foods, and a failure classifier that allows the robot to predict imminent breakage of fragile foods and adapt the relative usage of the pusher and spoon accordingly. The authors provide data from hardware experiments showing that their approach improves both over single-arm strategies and non-adaptive strategies (for fragile foods).", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- This paper includes very impressive hardware results; nice job!\n- I think that the architecture in this paper strikes a good balance between hand-designed and learned components, as opposed to an entirely learned policy.\n\n## Weaknesses\n\n- The paper is a bit light on the details of the scooping primitives. Particularly, it took a couple of takes to understand the role of $\\alpha$, and I'm still not quite sure how $\\alpha$ is adaptively varied. Perhaps adding a section in your supplementary material would help with this point.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper was clear and enjoyable to read, and the authors present fairly comprehensive hardware results to demonstrate the effectiveness of their method.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- As mentioned above, more detail about how the parameter $\\alpha$ is adapted would be helpful.\n- You define the image observations as the state of an MDP, but I usually think of images as observations rather than states, suggesting that a POMDP could be a better fit here. This aligns with the fact that you use a learned state estimator to get the centroid of the food from the image, suggesting that there is an underlying state that can be inferred from the image observations. Perhaps making this adjustment would help you expand to multimodal sensor feedback (as you mention in the limitations section)."}}, {"review_id": "LmCNCi-JS5-", "reviewer": "Reviewer_rkd9", "summary": "The authors proposed a general bimanual scooping primitive and an adaptive stabilization strategy that enables successful acquisition of a diverse set of food geometries and physical properties. The author's approch, CARBS learns to stabilize without impeding task progress by identifying high-risk foods and robustly scooping them using closed-loop visual feedback. The robot takes two inputs, pusher travel distance and food posiiton, and performs three stabilizing strategies. The Risk Classifier classifies food fragility into binary categories (robust or fragile), and the Failure Classifier determines whether the robot is \"Keep Going\" or \"Stop\" and adjusts the stabilizing parameter to prevent the object from being broken. With these two classifiers and the three strategies described above, CARBS generalize across food shape, size, and deformability and is additionally able to manipulate multiple food items simultaneously.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors address a very challenging task of food scooping, a subject for which there is little previous research. The supplemental video fully conveyed the appeal of the study and further helped readers understand it. The fact that the twin-armed robot performs a difficult task is also highly commendable; one can read from the paper that considerable trial and error went into finding the three stabilization strategies. From the comparison experiment, the reviewers understood the effectiveness of the proposed method as the weight loss of food was significantly lower than the baseline.\n\nHowever, the technical novelty of the proposed method is low; it is an engineering combination of existing technologies. The three stabilization strategies (1) Angled Pusher states that the Pusher was placed at a specific angle, but this is an easy thing to come up with and not a point of special claim. If the angle changes dynamically according to the shape and characteristics of the object, it can be broadly claimed as a novelty. The Planning of the Scoping Phase is also the same.\n\nThere is also a lack of specific explanation of the methodology, which confuses the reader. Since the whole system configuration is difficult to understand, it is better to show the following three in Fig. 1 to help the reader understand: (1) the name of the neural network model used, (2) the name of the information output by each module (e.g., the food position) are shown above the arrows, and (3) the data flow (steps) at runtime. In line 247, the data augmentation is used to trained the classifier, but what kind of method was used specifically?\nSince there is no description of the operation cycle during task execution, it is unclear how often the Failure Classifier performs object status determination. In line 247, \"The Failure Classifier is trained on images from scooping rollouts of tofu only, which we found to be sufficient for generalization to other food classes as well,\" requires quantitative data. The reviewer doubts that generalization can be achieved by simply learning tofu image data, since the foods have very different shapes, colors, characteristics (robust or fragile), and dynamics (e.g., blueberries with unpredictable dynamics).\n\nMinor comments are as follows.\nLine 178: Which direction does the x-axis indicate in Fig. 3? Showing each axis in the figure will help the reader understand. The text also needs to be improved, as there are many typographical errors. e.g. Appendix reference number and Figure3/Fig.3. Since the reference to Fig. 3 appears four times between lines 179 and 188, it is easier to understand by adding sub-numbers like Fig. 3 (a) and (b). How is the generalization performance of Risk Classifier? The reviewer is concerned that it may be limited because there are many similar looking but different hardnesses in the world. As the author describes in the limitation section, it is better to build inference models based on force sensors and tactile sensation. Multimodal learning allows for highly accurate estimation of contact states.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The task is challenging and the video is well presented.\t", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "I criticized this study as engineering, but it was wrong, and I understood that this work accomplished after a great deal of preliminary study. The authors' experimental results contain a many important findings for the entry of robots into the home and food industry. "}}]

Author rebuttal:

Decision: Accept (poster)