Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a system for food skewering by considering both visual and haptic information. Specifically, the authors propose a neural network-based framework that fuses the multimodal information during a single skewering interaction to learn food representations (via supervised learning), informing the food item's geometry and deformation properties. The model is learned by mapping visual and haptic information to a discrete set of skewering primitives using a small manually-labeled dataset collected in an offline fashion. The resulting system can then be used to skewer food items.

The authors have evaluated the proposed system on a diversified set of food items with varying geometry and deformability, achieving 71% skewering success across 21 items total. The experiment results also suggest that the consideration of multiple sensory modalities enables the proposed method to handle unseen objects and outperform those that only take single modality input.
strengths_and_weaknesses: **[Strengths]**

This paper tackles a very interesting problem of food skewering. The task is challenging as the robot has to deal with objects with different geometry and complicated physical properties, but this is also a necessary step for robot-assisted feeding. A satisfying solution to this task holds the potential to achieve great real-world impact of having robots caring for our elderly population or people with disabilities.

The authors have evaluated the proposed system on a diversified set of objects of different visual appearances and deformability. They have also provided both quantitative and qualitative evaluations demonstrating the effectiveness of the proposed method. The video demonstrations in the supplementary materials and the website are especially helpful for the readers to understand the difficulty of the task and the system's performance.

This paper also provides many very practical treatments for the design of different system components and discusses why the task is hard with concrete examples (e.g., distinguishing between raw and boiled carrots). A lot of the observations obtained through the experiments and discussions of the failure modes are also very interesting and tremendously helpful for the follow-up work to make further progress.

**[Weaknesses]**

My primary concern with this work is that the action space is a bit limited. The authors mainly consider two types of pre-specified action primitives designed specifically for this task: angled skewering and vertical skewering. I understand that these two primitives may be enough for the food skewering tasks, but such constraint limits the method's applicability in other tasks. It is also hard to know how well the method can scale as the number of action primitives increases.

The authors also claim that the system learns a reactive policy. I thus hope the authors can explain in more detail how the feedback is taken during the task execution process and the frequency of the feedback loop. Correct me if I am wrong, but it seems that the robot does the task in a two-step process: probe, then skewer, where both steps are executed in an open-loop fashion. Some of the videos also show that even if the robot fails to skewer a food item, the robot continues to perform the scoop motion, indicating that no feedback is considered during this stage. Will feedback at a higher frequency help here? I'm curious to know the authors' perspectives on this and how to incorporate such visuomotor closed-loop ability in the system.

One thing missing from the experiments is a more comprehensive evaluation of how much data is needed and how the data's size and diversity influence the performance and generalization abilities. Such experiments can be essential to understanding the sample efficiency and the applicability of the method.

Meanwhile, for generalization to unseen objects, are there ways to quantify the distance between the unseen objects and the training distribution (e.g., according to the shape and stiffness)? It would be helpful for the readers to understand to which extent can the learned system generalize and how the system behaves as it encounters less and less familiar objects.

I could also imagine that the labeling of the objects can be ambiguous. For example, some food items with deformability between the banana and the broccoli may be successfully skewered via either angled or vertical skewering actions. Do the ambiguities in labeling matter during the learning process, and how do such ambiguities be resolved in the pipeline? Will a more continuous action representation help in this case?

In Figure 4, Cumulative Failures, why does HapticVis (the proposed method) perform worse than Haptic and Vis alone in BBox FP?

The first contribution stated in the introduction discussed a coarse-to-fine visual servoing method to approach a food item. However, no mention of it is found in the remainder of this paper. If such a coarse-to-fine visual servoing method is not the main focus but a contribution of another paper, I suggest the authors rephrase the text to highlight the unique contribution made by this paper.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: I think this is a well-executed paper tackling an important problem of food skewering. The idea of fusing multiple sensory modalities (i.e., haptic and vision) is also interesting and has shown to be effective in this specific task. The authors have also provided extensive evaluations in the real world with food items of diverse visual appearance and physical properties. I thus lean towards the acceptance side.
issues: I have concerns regarding
- the limited action space,
- details about the reactive/feedback process,
- more comprehensive evaluations of the amount/distribution/diversity/ambiguity of the training data, and
- certain claims about the contribution of this paper.

Please see the weaknesses section for more details.

In addition, why are there missing numbers in Table 1 (slots marked with "-")? Is it because the specific methods are not applicable in the testing scenario, or did the authors not get the chance to run the experiments?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In the paper, the authors introduce a new network that incorporates both haptic and visual data in order to determine the best fork skewering skill parameters after first probing the item. It differs from previous work that relied solely on visual data to determine fork skewering locations. They ran many studies on both unseen and seen food on 6 different plates of food items where sometimes the food items were of the same type but some of the food was boiled so they were visually similar but had vastly differing properties. Their network improved on previous skewering networks and they tested the success results of the different ablations on the real robot.
strengths_and_weaknesses: The main strengths of the paper is that they were able to train this Haptic Visual Net using only human annotated data that required only 3 hours of data collection and labeling and didn't need to waste time to have the robot try actual skewering attempts and then a human would have to reset a scene and then clean the fork. That was actually very surprising to me. The original SPANet required 81 hours of human labeling or robot data collection according to their paper. While I have not read that paper, it seems credible in my experience of doing my own robot experiments. I also believe that sometimes if a human can give sufficient labels that are accurate enough, then trying to do the experiments with an actual robot will take more time.

I only wish that the authors tried a couple more different varieties of hard and soft objects of the same category. For example, frozen fruits vs regular room temperature fruits. They seemed to only do boiled carrots. The boiled carrots and pasta as unseen was very interesting. Maybe for future work, they can try running the network on cooked items like sauteed chicken with vegetables in more cluttered plates than the much less dense plates that were used for evaluation. If the robot were to skewer multiple thing at once because they were overlapped, what would Haptic Visual Net do in case some items are not skewered well enough and they were to fall on a person and unfortunately hurt them if they were hot? Also what would be the effect of sauces be on networks like if they were charred (soft) vs. dark soy sauce on the objects. Would they be classified similarly? I hope that people that might need the robot don't just have to eat plain vegetables without any sauce haha. Also as food items cool, they might change as well so maybe there should be in between classes instead of soft and hard. For example, chewy or sticky or something.

Finally, I wonder if sometime it would be more useful if the robot could also use a bandit formulation of exploration vs exploitation to figure out whether all the food items on the plate are similar and whether to continue probing or whether it is making a lot of mistakes skewering to then probe the items more to figure out its properties. I wouldn't want a robot to waste time continuously probing if every item was as predicted.

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Overall, I give this paper a strong accept. I think the well-written paper and accompanying video can be useful for the robotics community. While the problem of fork skewering is not novel, they were able to develop a new network using a probing action and the haptic feedback from the action instead of only relying on vision to determine the parameters of the fork skewering actions. I see no reasons why someone would reject the paper and would argue for my recommendation. I felt like the experiments were well executed and the results are in line with what I would hope for a personal assistive feeding robot. However, there could be more experiments with food items that are the same type but different properties like frozen fruits / vegetables or slightly sauteed items vs regular food items.
issues: I don't really have any issues with the paper. It is well written and the video is excellent as well. Overall, if you could address maybe some of the issues I brought up in the strengths and weaknesses section above that would be ok. For example, maybe doing some more experiments on frozen fruit vs normal fruit or sauteed / wok fried objects with sauces vs normal items. In addition, the food items on the plates were very separated. If there was a normal served plate of food where lets say rice or mashed potatoes is in one corner, the vegetables are in another corner, and there are some stir fried chicken with bell peppers or something, how would the Haptic Visual Net react to more cluttered plates and what new data would be necessary to increase the robustness of your method so that it would scale for real-world adoption?
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a new bite-acqusition framework for robot skewering food on a plate using multimodal (visual and haptic) information. The framework compose of two stages: the probing stage and the skewering stage. During the probe stage, two networks are trained to localize the food bite and approach the localized food till contact is made. During the skewering stage, the proposed VisualHapticNet fuses the haptic measurements from a force-torque sensor and images from a camera to determine a skewering primitive (vertical versus angled). Comprehensive experimental results are conducted to test the framework's performance on in-distrituion and out-of-distribution food acquisitons, with comparisons to baselines and ablations. The results show good performance on food with in training distribution and adequate generalization to unseen food. Comparison to the basline and the ablations show the importance of using both modalities and using reactive planning. 
strengths_and_weaknesses: Strength:
- Robot-assisted feeding is definitely an important application so the paper is well-motivated. The idea of combining visual and haptic information makes a lot of sense for this application.
- The experimental results are strong and pretty complete in my view. The system achieves good in-distribution and out-of-distribution performances . The failure cases are also shown which are appreciated. The comparison to baselines and abalations are well conducted and helped understand the contribution of each component of the method.
- The paper is clearly written and I enjoyed reading it. 

Weakness:
I mostly have some minor comments:
- Can you provide more details regarding how the ServoNet is trained? Is the label just delta movement of the end-effector? 
- What does '-' mean in Table 1? Does it mean 0 success or no experiment is conducted? If the latter, why?



robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think the proposed idea of combining visual and haptic information for robot assisted feeding makes a lot of sense, and the experimental results are impressive and strong and supports the proposed idea.
issues: Please see the weakness section as above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a robotic system for skewering pieces of food from a plate. Taking visual and force data as input, a learned policy controls the robot to (1) detect pieces of food, (2) move the fork via visual servoing, (3) probe the food, (4) and select a primitive based on the measured probing data. The key contribution is the NN mapping the visual and haptic probing data to one of the two primitive types; they differ in the fork pitch angle intended for either hard or soft objects. The data (300 probing examples) was human annotated. Experimental evaluation was done on various foods and compares the approach with different input modalities as well as one baseline.
strengths_and_weaknesses: The paper is nicely written and easy to follow. We like the application as well as the idea of probing the food to get very simple closed loop feedback. Its implementation, as seen in the video, seems fast and nearly unnoticeable. Moreover, the experiments study generalization extensively with a wide range of foods. While we think that the contribution itself is not particularly large, the sound implementation and the task might be interesting for the community.

However, we think that the quantitative results (e.g. a 71% success rate) aren’t very strong. Therefore, we would like to see a more detailed ablative study. In particular, 300 annotations (in 3h of data collection) seems very few for a data-driven approach. How do the results improve with training data? Broccoli florets (plate 4) have a 61% success rate, despite being a trained food. Is this a matter of training data? What are the success rates for using only one primitive?
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: We think that the originality of this work is incremental, as probing is a common solution in (robotic) measurement tasks. Moreover, the general ideas are only applicable to the task of skewering foods from plates.
issues: We suggest to address following comments (in no particular order):

After playing around with various foods, we think that both primitives could be combined into a vertical skewer motion and a subsequent angled lifting with support from the plate. In this regard, we suggest (similar to the paragraph above) adding some ablative experiments about the action space. 

The action space definition could be made clearer. We suggest including only the learned action space in eq. (1), and clarify that z is via depth image, delta z is fixed, delta beta are two discrete values only.

Similarly, the reward definition seems unusual. How many timesteps does an episode have? Isn’t the episode stopped after failing, so the loss would always sum up to one? Given the description, the system seems to actually learn a binary reward from the human annotation if the whole skewing primitive is successful.

Code and dataset were not published during the time of review, so please do so now or remove their references from the paper. Of course, feel free to add them to the website anytime.

Why weren't the internal force sensors used? Moreover, we suggest giving a brief (one-sentence) overview in 4.2 about the used control software for the robot.

Why is the ServoNet necessary? In general, the absolute position accuracy of the Franka robot is below 1 mm so it should be able to move to a target position without servoing.

We suggest replacing several arxiv citations with their conference/journal counterparts.

In line 222, ”only” is doubled.
review_title: null

Meta-review: metareview: The paper proposes a system for food skewering by considering both visual and haptic information and evaluates it thoroughly, 

Strengths:
- Interesting problem of food skewering (jNun, NczR)
- relatively little data required (NczR)
- good evaluation (jNun, NczR)

Weaknesses:
- limited action space (jNun)
- limited amount of hardnesses for the same objects (NczR)
- ablations on the amount of data missing (b6eX)

This is just an excerpt of the reviews. I invite the authors to respond to the reviewers and address their concerns.

Post-rebuttal:
The authors have been able to address many of the concerns and the paper should definitely be accepted. Since the paper is round and shows great results on a real robot it might be considered for an oral (weak endorsement).
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Oral)