Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The authors of this paper explored the problem of scooping various unseen food items using a Franka with a ladle. They used active perception with some stirring actions and recorded 7 frames using a pre-specified hard coded motion. Then they ran the 7 RGBD frames into an encoder to get a latent embedding for the food item. Next, they used a closed loop policy that takes in the embedding from the active perception, 10 previous RGBD frames, and a state retrieval module that uses cropped rgb images and attention as input into a policy that outputs a delta action. Their action space is the robot's XYZ and y-axis pitch rotation of the end-effector. They compared their method with a variety of other behavior cloning approaches and templated approaches. They reported a 70% success rate using their metric of no spilling, filling up at least 1/3 of the spoon, and not damaging the food items. They also tested on black tea and pudding, which showed some fair results.
review: Writing Quality: The writing quality was pretty good. There were only a few mistakes that I noticed in the supplementary video and some places in the paper. 

Clarity: The paper was pretty clear, although some of the figures could be fixed. I think Figure 2 could be improved as it initially confused me with only arrows going into the state retrieval module and nowhere else.

Originality and Significance: This paper seems to be a combination of different previous ideas applied to the task of scooping. One would be active perception, another is an attention module, and the last is a concatenation of previous frames into an encoder. I would not consider this a very novel paper.

Strengths: The authors did do a lot of real world scooping experiments. Their method does seem to improve on their other baselines and ablations.

Weaknesses: I have a lot of concerns for this paper. I will put most of this section in the questions for Rebuttal.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: I updated my review to a weak accept because the authors did test on new liquid/viscous objects and the performance was still acceptable. In addition, they added continuous scooping results. While I still believe a well formulated baseline using some image segmentation and a good hard coded scooping action could achieve similar results. It does seem good that it works in a closed loop manner. I would still prefer if there were different containers and tools as well as the skill adapting when things change in the environment such as human interference. I will not argue for acceptance, but it has improved from its original submission.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents SCONE, a food-scooping robot learning framework that leverages active perception to learn meaningful representations of food. SCONE uses interactive encoding and state retrieval modules to capture essential food characteristics and states. The system exhibits success in real-world experiments, achieving a 71% success rate on unseen food items across varying difficulty levels, outperforming other baselines.
review: Quality: The paper is technically sound and claims well supported by real-world experiments. The idea of active perception for robot scooping is intuitive and effective.

Clarity: The paper is clearly written and well-organized.

Originality: To the best of my knowledge, this work seems to be a novel combination of methods in robot scooping.

Significance: The idea of active perception has the potential to be applied to other tasks in deformable manipulation since the interaction is a natural way to learn the dynamics of the target objects.

Strengths: 
1. The authors show that their method can generalize to an extended set of unseen materials in different settings and outperform baselines.
1. The visualizations of interactive embeddings and attention scores are very compelling.

Weaknesses:
1. I'm a little unsure about how difficult this task is since I imagine an open-loop policy will also have relatively reasonable performance. For example, if the robot inserts the ladle into the bowl, rotates around the $y$-axis by a fixed angle, pushes down, rotates back to align with the horizontal plane, and lifts, how will that end up in the task of scooping? The simplified action space (3D position and the y-axis pitch rotation) also gives the policy learning module some important priors to leverage.
1. The authors show material-wise generalization, but how about generalizing to different spatial locations of the bowls (e.g., what if the target bow is further away), different sizes/shapes/colors of the bowls and ladles?
1. It would make this submission much stronger if the authors could compare against methods beyond the paradigm of behavior cloning and active perception, such as model-based methods [[1](https://arxiv.org/abs/2306.16700)] or more up-to-date imitation learning methods [[2](https://arxiv.org/pdf/1903.01973.pdf)].
1. As the authors mentioned in the limitation section, the hard-coded actions for active perception may limit the possibilities the robot explores and inject strong human priors into the model.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper presents an effective approach to robot scooping with active perception. It's technically sound and well-supported by experiments. However, the task's difficulty is unclear. Generalization to varying spatial locations, sizes, shapes, and colors of bowls and ladles isn't addressed. Comparisons beyond behavior cloning and active perception, such as model-based or advanced imitation learning methods, would strengthen the paper. Also, the reliance on hard-coded actions for active perception might limit the robot's exploration and inject human priors. Therefore, I think more works are necessary to validate this method's robustness and applicability.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper presents a method for scooping various granular food products using a robot arm and a large spoon.  The robot first interacts with the food by stirring it with the spoon. Then the robot attempts to scoop the food and finally dump it in another container. The observations from both the interact and scoop phases are encoded into 4 latent vectors, which are then fed to a scooping policy. The results show that the robot is able to achieve a 71% success rate on average over all the tasks, and is even able to generalize to food with physical properties slightly outside the training distribution.

review: This is a novel method for solving a challenging task. Scooping and active perception are not new in robotics (as the paper points out), but using active perception to perform something akin to latent system identification for non-rigid media like beans and rice is. While the method is fairly complex, the results clearly show that it outperforms the baseline in almost every case. Furthermore, the results are reported as number of success out of total attempts, which is a very intuitive and easy to understand metric. Overall the methodology proposed by this paper shows a novel and interesting way to incorporate active feedback in a challenging manipulation task.

One thing this paper does really well is thoroughness in experimental analysis. Table 1 shows a comparison of the main method with 7 baselines spanning variations on behavioral cloning, kNN-likes (the template methods), and even an RNN. Furthermore, the methods are all evaluated on a variety of foods with very different properties, from rice to oranges! Finally, the ablation study in Table 2, even though only a small part of the paper, does a good job justifying the existence of the various parts of the model.

Overall, there’s not much to detract from in this paper. Would it be nice if there were more than 10 trials per food X algorithm combination? Yes, but as it stands there are already 800 trials evaluated in the paper, which is approaching the limit of what can be done with human supervision, so it’s not reasonable to expect more trials. However, an interesting follow-up work might investigate using automated data collection that doesn’t require human involvement (e.g., scooping back and forth between 2 containers). It would also be interesting to try this method on semi-solid and liquid foods such as jams, puddings, creams, and even water. But none of this is necessary as this is an excellent paper as is (though authors feel free to try that last idea for the rebuttal, it would really elevate the paper quite a bit).

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I recommend acceptance of this paper based on the novelty of its methodology, the challenging nature of the task, and the thoroughness of the results.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary_of_paper: This paper contributes SCONE, which is a robotic food scooping learned method that uses active perception to implicitly learn a latent space representation for food items. This is later accessed with the state retrieval module during manipulation. This paper claims that prior work is limited in that they use hard-coded primitives for food manipulation, but by implicitly learning a representation of food from human trajectories with the human manipulating the food, the model is able to learn food dynamics and alter the scooping manipulation as needed to successfully scoop. 
review: This work seems relatively original, where in the food manipulation space, it seems to be one of the few to introduce closed-loop control during food manipulation. 

The paper overall is technically sound. The only concern is there seems to be little explanation for failures and the authors seem to imply the failures in the basic setting are mainly due to unstable robot arm movements or other sources. If this is the case, it should be written more clearly. An explanation for the extended and peculiar setting failures is also needed. 

The paper is clearly written and easy to understand. The results show SCONE outperforms most of the baselines and the idea of closed-loop food manipulation with an implicitly learned food representation is a unique contribution. 

It may be interesting to see if similar foods take similar trajectories. One limitation I see is that the test food seemed similar to the train food. Chocoballs appear to be similar to macadamia nuts and red bean is similar to soy and mung bean. It would be nice to see how this would function for more varied food, such as lychee (spikey), cut banana (cylinder shape), boba (or something of a more spongy texture). 

Another limitation in my view is all experiments were done with the same ladle. In the video at least, the “spoon” also appears large, not a normal bite size spoon. It would be good to show results on a few differently shaped spoons as the way the paper presents it, the utensil used for food manipulation should not affect the performance of the method. 

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The work is original and clearly presented. The main concerns for a weak accept over a strong accept is for the following two reasons:

1) Technical quality. The train and test food were very similar so I question the generalizability of the method to unseen food types. They all embodies either a large, spherical food or grains/beans. There was also no report on the number of trials conducted in the ablation study.

2) Limitations. The biggest limitation I saw is all experiments were done on the same "spoon," but it appears to be a very large ladle which makes the problem easier. To address this limitation, the authors should either add this in the paper or test or train&test on various spoons as the method seems to have no reliance on the spoon shape.
review_title: null

Meta-review: This submission first received mixed reviews.  The authors did a good job during the rebuttal phase, and all reviewers recommended acceptance after discussion.  The AC agrees with the recommendation.  Some reviewers remain concerned about the generalization of the proposed method.  The authors are encouraged to further explore this direction based on the reviews.
Decision: Accept (Poster)