Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces a bimanual scooping primitive that is capable of stably scooping bite-size food items with different geometry, color and deformation properties. In this case the stability of the scoop refers to avoiding breakage, rolling or cutting of the food item being scooped. The primitive, formulated as an MDP, takes as input an RGB image of a camera mounted on the scooping arm and selects the stabilizing strategy to perform the scoop either: angled push, cupping or pinning motions. For fragile food items an adaptive cupping strategy is introduced that leverages a “risk classifier” trained to identify high-risk breakage prone foods and a “failure classifier” trained to identify near-breakage states to adapt the cupping motion by scaling the distance traveled between the pusher and the scoop. Improved performance over single-arm scooping and baseline bimanual scooping (no active pusher) is shown. 
strengths_and_weaknesses: **Strengths**

- The paper is tackling an interesting and difficult problem, as food manipulation is not subject to a single form of deformation or failure, this paper seems to solve a range of issues that can arise from scooping food items with different properties.
- Convincing videos and experiments.
- Catchy acronym. 

**Weaknesses**

- It is not entirely clear to me what is being learned in the MDP and how that data is acquired. In section 3 the MDP is introduced and states that the transition dynamics are unknown, if they are unknown then they must be learned or parametrized somehow? Also, first it is mentioned that the action space is 14-DoF but then it is mentioned that it is simplified by the bimanual primitive which has 3 phases, so is the new action space 3 dimensions? This is not really clarified in the main text or appendix. 
- The paper only mentions the learning pipeline for the risk and failure classifiers, however, shouldn’t there be a dataset where the robot learns the transitions between the different scooping phases/stabilizing strategies? For example, from angled pushing to cupping to pinning. It is unclear to me if this sequence or transitions between the phases are learned from the food interactions or if they are predefined and parametrized by the designers. How is the MDP deciding when to transition between phases?
- Related to the previous point, the paper only gives details about the risk $r:\mathbb{R}^{..}\rightarrow$ \{0,1\} and failure classifiers  $f:\mathbb{R}^{..}\rightarrow${0,1} which both seem to map to a binary classification variable. It is not clear to me how the outputs of these functions are used to modulate $\alpha$ which seems to be a continuous function based on the figures and videos. 
- It would be ideal if the authors included a baseline comparison to the work cited from [23] which learns the bimanual interaction constraints from human demonstrations. This would clarify the doubts related to what is being learned and what is assumed/predefined in this framework.
- Is there a reason why adaptation to avoid breakage is only considered in the cupping phase? It would seem to me that breakage could be possible during the angled pushing and pinning phases as well - to avoid breakage or rolling there you would need to adapt the angle used by the pusher or modulate the speed of the rotation done by the pinning motion. 
- It would be interesting to see how the primitive performs when trying to scoop items that are slightly larger than the scoop itself, like 2 strawberries or a larger piece of jello or cheesecake, oftentimes in the kitchen we use tools that are not the appropriate size but as humans we adapt our motion to perform the task, could this form of adaptation be translated to robot through this adaptive scooping framework?
- While the proposed primitive seems appropriate for scooping it would be great if the authors discussed how the presented strategy can be transferred to other types of tasks or if it can be adapted to handle other types of items that are not food but need a scooping mechanism like soil, sand, etc. 
- Details are lacking regarding the low-level control of the arms, is the control done purely in position mode with visual servoing? Why not use force/torque measurements? How is the simplified action space mapped back to the robot 14-DoF joint space and how are the desired end-effector commands controlled/tracked?

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper presents a novel approach to scooping food items with a bimanual arm setting including an active pusher and a scooper arm. The approach is capable of handling a variety of food items as demonstrated in the videos and shows improved performance over single-arm scooping and baselines. However, some issues need to be addressed regarding clarification of the framework, dataset generation and learning. 
issues: Comments listed in the weaknesses section should be addressed.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work focuses on bimanual scooping of single instances of that would be difficult to skewer or scoop using traditional techniques due to their unique geometries, i.e. objects that are small, brittle, fragile, or those that would roll away if scooped with a single arm. To handle these problematic foods, a second arm with a pushing attachment is used to stabilize, collect and push the food onto the scooping attachment on the other arm. Three primitives are defined; angled pushing, adaptive cupping, and pinning. Adaptive cupping has a variable parameter, \alpha, that defines how close to the scooper the pusher gets, as well as the location of the center of the food item. These parameters are learned using a convolutional network, based on images of the food. The centers and segmentation of each food are predicted as well as whether the food is robust or fragile. If it is fragile, a breakage detection prediction is run online and the food is pushed until imminent breakage is detected.
strengths_and_weaknesses: **Strengths**

_Novel Solution to an Important Problem:_

This is a novel solution to a previously unsolved problem. Further, this is a problem that is applicable to people who use robotics feeding systems. The solution was fully implemented into a working system that was tested on real food. 

_Simple and Effective Solution:_

The solution uses simple primitives that are adapted to individual food instances. The pusher design and primitives leverage funneling behaviors that reduce uncertainty in the position of the food, improving the probability of success. Further, relatively simple representation of the food allows the systems to generalize across types of food. The method shows significant improvement over the static baseline, showing that some dexterity on the second arm is necessary. Further, the closed loop stopping behavior greatly reduces food breakage.

**Weaknesses**

_Learning components are relatively limited:_

The learned components of this method are relatively limited. They amount to a food segmenter and centroid predictor, food robustness classifier, and a food breakage predictor. While these are sufficient to complete this task with a high degree of efficiency, they do not showcase the need for learning in robotics tasks.

_Could use more complex baselines:_

The baselines were somewhat limited, with a static pusher, a constant \alpha = 1, and the proposed method. A method that used the predicted centroid of the food, stopping when the centroid was over the spoon, could show the need for a breakage specific online policy. Additionally, a lower \alpha baseline, or set of constant \alpha baselines could show that no single \alpha value works across food types.

**Minor Weaknesses**

The parameter in of the angled pushing primitive is described but never defined. How is \theta selected? Is it constant for all foods? The spoon angle is described in the supplemental material but not the pusher angle.

Most food acquisition tasks are done on relatively cluttered plates. Some experiments showing robustness to this or mention of it in the limitations section should be added.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This work shows a simple application of learning to solve an important, previously unsolved problem. While the learned components are relatively simple, I believe that they are sufficient to solve the problem. Additionally, the work shows a full system, working with real word food, in a relatively realistic manner (minus the lack of clutter). If the conference would like to showcase more realistic systems, using some learning to solve relevant tasks, I believe this paper is an example of such work.
issues: The parameter in of the angled pushing primitive is described but never defined. How is \theta selected? Is it constant for all foods? The spoon angle is described in the supplemental material but not the pusher angle.

Most food acquisition tasks are done on relatively cluttered plates. Some experiments showing robustness to this or mention of it in the limitations section should be added.

Figure captions are too close to text, particularly above line 207.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a method for scooping foods into a spoon in which both the spoon and a "pusher" are attached to separate robot arms. The authors show that allowing independent motion for the spoon and pusher significantly improves success rates vs a spoon-only strategy. The authors also use two neural networks as part of their control architecture: a risk classifier that decides whether to use a fixed pushing strategy for "robust" foods or an adaptive strategy for fragile foods, and a failure classifier that allows the robot to predict imminent breakage of fragile foods and adapt the relative usage of the pusher and spoon accordingly. The authors provide data from hardware experiments showing that their approach improves both over single-arm strategies and non-adaptive strategies (for fragile foods).
strengths_and_weaknesses: ## Strengths

- This paper includes very impressive hardware results; nice job!
- I think that the architecture in this paper strikes a good balance between hand-designed and learned components, as opposed to an entirely learned policy.

## Weaknesses

- The paper is a bit light on the details of the scooping primitives. Particularly, it took a couple of takes to understand the role of $\alpha$, and I'm still not quite sure how $\alpha$ is adaptively varied. Perhaps adding a section in your supplementary material would help with this point.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper was clear and enjoyable to read, and the authors present fairly comprehensive hardware results to demonstrate the effectiveness of their method.
issues: - As mentioned above, more detail about how the parameter $\alpha$ is adapted would be helpful.
- You define the image observations as the state of an MDP, but I usually think of images as observations rather than states, suggesting that a POMDP could be a better fit here. This aligns with the fact that you use a learned state estimator to get the centroid of the food from the image, suggesting that there is an underlying state that can be inferred from the image observations. Perhaps making this adjustment would help you expand to multimodal sensor feedback (as you mention in the limitations section).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors proposed a general bimanual scooping primitive and an adaptive stabilization strategy that enables successful acquisition of a diverse set of food geometries and physical properties. The author's approch, CARBS learns to stabilize without impeding task progress by identifying high-risk foods and robustly scooping them using closed-loop visual feedback. The robot takes two inputs, pusher travel distance and food posiiton, and performs three stabilizing strategies. The Risk Classifier classifies food fragility into binary categories (robust or fragile), and the Failure Classifier determines whether the robot is "Keep Going" or "Stop" and adjusts the stabilizing parameter to prevent the object from being broken. With these two classifiers and the three strategies described above, CARBS generalize across food shape, size, and deformability and is additionally able to manipulate multiple food items simultaneously.
strengths_and_weaknesses: The authors address a very challenging task of food scooping, a subject for which there is little previous research. The supplemental video fully conveyed the appeal of the study and further helped readers understand it. The fact that the twin-armed robot performs a difficult task is also highly commendable; one can read from the paper that considerable trial and error went into finding the three stabilization strategies. From the comparison experiment, the reviewers understood the effectiveness of the proposed method as the weight loss of food was significantly lower than the baseline.

However, the technical novelty of the proposed method is low; it is an engineering combination of existing technologies. The three stabilization strategies (1) Angled Pusher states that the Pusher was placed at a specific angle, but this is an easy thing to come up with and not a point of special claim. If the angle changes dynamically according to the shape and characteristics of the object, it can be broadly claimed as a novelty. The Planning of the Scoping Phase is also the same.

There is also a lack of specific explanation of the methodology, which confuses the reader. Since the whole system configuration is difficult to understand, it is better to show the following three in Fig. 1 to help the reader understand: (1) the name of the neural network model used, (2) the name of the information output by each module (e.g., the food position) are shown above the arrows, and (3) the data flow (steps) at runtime. In line 247, the data augmentation is used to trained the classifier, but what kind of method was used specifically?
Since there is no description of the operation cycle during task execution, it is unclear how often the Failure Classifier performs object status determination. In line 247, "The Failure Classifier is trained on images from scooping rollouts of tofu only, which we found to be sufficient for generalization to other food classes as well," requires quantitative data. The reviewer doubts that generalization can be achieved by simply learning tofu image data, since the foods have very different shapes, colors, characteristics (robust or fragile), and dynamics (e.g., blueberries with unpredictable dynamics).

Minor comments are as follows.
Line 178: Which direction does the x-axis indicate in Fig. 3? Showing each axis in the figure will help the reader understand. The text also needs to be improved, as there are many typographical errors. e.g. Appendix reference number and Figure3/Fig.3. Since the reference to Fig. 3 appears four times between lines 179 and 188, it is easier to understand by adding sub-numbers like Fig. 3 (a) and (b). How is the generalization performance of Risk Classifier? The reviewer is concerned that it may be limited because there are many similar looking but different hardnesses in the world. As the author describes in the limitation section, it is better to build inference models based on force sensors and tactile sensation. Multimodal learning allows for highly accurate estimation of contact states.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The task is challenging and the video is well presented.	
issues: I criticized this study as engineering, but it was wrong, and I understood that this work accomplished after a great deal of preliminary study. The authors' experimental results contain a many important findings for the entry of robots into the home and food industry.
review_title: null

Meta-review: metareview: Strengths:

* bimanual solution to food acquisition (new)
* balancing manual design, a lot of engineering, simple learning: it's effective
* good experimental results and tests with real food

Weaknesses:

* a lack of specific explanation of the methodology (e.g., how is a primitive described/implemented? how is theta set?)
* lacking detail on the robot control (especially, where the end-effector control point is set, if it is torque controlled, impedance controlled etc.)
* lacking detail on the framework, dataset generation and learning 

Summary after rebuttal:
We thank the authors for the updates and the rebuttal. The reviewers and the AC agree that this paper provides an interesting contribution to robotics, balancing engineering and learning. Concerns over generalization remain, so it is recommended as poster.

confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)