SCONE: A Food Scooping Robot Learning Framework with Active Perception
Authors: Yen-Ling Tai
Yu Chien Chiu
Yu-Wei Chao
Yi-Ting Chen
Venue: CoRL 2023 Poster (2023)
Keywords: Food Manipulation
Robot Scooping
Active Perception

Abstract: Effectively scooping food items poses a substantial challenge for current robotic systems, due to the intricate states and diverse physical properties of food. To address this challenge, we believe in the importance of encoding food items into meaningful representations for effective food scooping. However, the distinctive properties of food items, including deformability, fragility, fluidity, or granularity, pose significant challenges for existing representations. In this paper, we investigate the potential of active perception for learning meaningful food representations in an implicit manner. To this end, we present SCONE, a food-scooping robot learning framework that leverages representations gained from active perception to facilitate food scooping policy learning. SCONE comprises two crucial encoding components: the interactive encoder and the state retrieval module. Through the encoding process, SCONE is capable of capturing properties of food items and vital state characteristics. In our real-world scooping experiments, SCONE excels with a 71% success rate when tasked with 6 previously unseen food items across three different difficulty levels, surpassing state-of-theart methods. This enhanced performance underscores SCONE’s stability, as all food items consistently achieve task success rates exceeding 50%. Additionally, SCONE’s impressive capacity to accommodate diverse initial states enables it to precisely evaluate the present condition of the food, resulting in a compelling scooping success rate. For further information, please visit our website: https://sites.google.com/view/corlscone/home.

SCONE: A Food Scooping Robot Learning Framework with Active Perception
Yen-Ling Tai
Yu Chien Chiu
National Yang Ming Chiao Tung University yling.csl0@mnn 111.nycu..un..w
National Yang Ming Chiao Tung University ychiu2 cougarnet.uh..du
Wei Chao
Yi-Ting Chen
NVIDIA ychao nvm
National Yang Ming Chiao Tung University ychen @ es..ycu.edu..w
Abstract:
Effectively scooping food items poses a substantial challenge for current robotic systems, due to the intricate states and diverse physical properties of food. To address this challenge, we believe in the importance of encoding food items into meaningful representations for effective food scooping, However, the distinctive properties of food items, including deformability, fragility, fluidity, or granularity, pose significant challenges for existing representations. In this paper, inves- tigate the potential of active perception for learning meaningful food representa- tions in an implicit manner. To this end, we present SCONE, a food-scooping robot learning framework that leverages representations gained from active per- ception to facilitate food scooping policy learning- SCONE comprises two cru- cial encoding components: the interactive encoder and the state retrieval mod- ule. Through the encoding process, SCONE is capable of capturing properties of food items and vital state characteristics. In our real-world scooping exper- iments, SCONE excels with a 71% success rate when tasked with previously unseen food items across three different difficulty levels, surpassing state-of-the the- art methods. This enhanced performance underscores SCONE's stability, as all food items consistently achieve task success rates exceeding 50%. Additionally, SCONE's impressive capacity to accommodate diverse initial states enables precisely evaluate the present condition of the food, resulting in compelling scooping success rate. For further information, please visit website.
Keywords: Food Manipulation, Robot Scooping, Active Perception
Introduction
Robot scooping skills have the potential to greatly improve human life, with applications in food preparation and assistive feeding [1. 2, 3, 4, 5, 6, 7] applicable in diverse environments, including homes, hospitals, and restaurants. Nonetheless, successfully scooping various food types presents a challenging task because of the wide range of appearances and physical properties exhibited by different foods. These attributes are not easily defined or fully predicted beforehand during prepara- tion. Consequently, developing a framework that empowers robots to acquire the essential skills for handling unfamiliar food items remains substantial challenge in real-world applications.
Present methodologies [1, 2, 3, 7, 8, 9] often rely on hard-coded primitive actions to address real- world food manipulation challenges. Nonetheless, these approaches frequently hinge on predeter- mined trajectories or specialize particular food categories, constraining their adaptability to unfa-
Corresponding Author
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Scoop
Transfer
Training
Testing
Penne
Chesse Hall
Jtjibhe
Chocoha
Orange
Red Bean
Sego
Macadamia
Candy
Soy Besa
Mung Bean
Browa Rice
Figure 1: Real World Robot Food Scooping Task.
miliar food items. Current data-driven learning frameworks for assistive feeding [4, 10, 11, 12, 13] presume that food items with similar visual appearance will yield comparable success rates when adopting identical policies. However, due the diverse properties of food encountered in scooping tasks, these frameworks still face limitations in generalizing their performance. Moreover, intricate characteristics of food states pose difficulties in precisely defining segmentation masks or bounding boxes as representations [4, 11. 12, 13].
Dynamic feedback from interactions could effectively capture essential food state features without explicit definitions. Hence, active perception [14, 15, 16] show great potential in tackling real- world food manipulation's generalization challenges by recognizing object characteristics through inter- action. These frameworks consist of two stages. The first is the Interacting stage, where the robot interacts with target objects to acquire dynamic sensory feedback. Subsequently, in the Manipula- tion stage, policy model benefits from the additional information gathered during the first stage, leading to improved efficacy. The pipeline has been widely used in food classification, policy selec- tions [17, 18, 19], and dynamics inference [20, 21, 22]. Compared to static observations, interacting with food items yields higher confidence in assessing food properties.
To this end, propose SCONE: food SCOoping robot learNing framEwork with active percep- tion. We employ active perception to enhance the manipulation of target objects in a coarse plus fine manner: it involves dynamics feedback for global food properties and local state information during each Interacting frame. Building upon this intuition, our method encodes active perception data with two distinct modules. The interactive encoder encodes overall interaction observations, while the state retrieval module captures crucial state details. Subsequently, task-related embedding is obtained by fusing these encoding modules. This embedding, along with current observations, serves guidance for the closed-loop policy.
We evaluate our method on a real-world robotic food scooping platform, demonstrating its com- pelling performance compared to the benchmark methods. Overall, our method attains an impres- sive 71% task success rate across 6 previously unobserved food categories spanning three distinct levels of difficulty, thereby confirming its generalization capabilities. Furthermore, SCONE demon- strates its ability to accurately perceive current food states, ensuring continuous scooping success while minimizing spillage and damage to the food and surroundings. Additionally, leamned latent embeddings from the feature extraction modules, demonstrating that the properties of food can be extracted and the taskerelated information can be retrieved at each time step. Our main contributions are as follows:
visualize the
We propose SCONE: An active perception-baset learning framework for food scooping robots, employing Interactionn data to enhance generalization concerning food properties in real-world food scooping tasks.

---

The visualization of latent embeddings highlights the extraction of valuable knowledge and clearly illustrates how active perception aids in policy learning.
In real-world experiments, the framework effectively manipulates unseen food items across various settings, surpassing the performance of the baseline methods.
2 Related Work
Robotic Food Scooping. Recent studies have explored various robot scooping methods for handling food items [23, 24]. However, achieving effective food scooping requires careful consideration of the food's inherent characteristics, such as fragility, to enable successful scooping policy. For as- sistive feeding and food preparation tasks, several commercial meal-assistance robots [8, 9] rely on pre-defined trajectories and need human-assigner controls, which more stable but with limited autonomy. Ohshima et al. [1] introduce autonomous robot system using Laser Range Finder for non-rigid solid foods by tackling the recognition problem of remnants of foods. However, it is still limited to the geometry and deformability of food items. Park et al. [2] presented an active feeding framework where users select tool-use actions. Without considering the properties of each food, this may encounter limitations in scalability. Grannen et al. [4] proposed bimanual scooping framework with closed-loop visual feedback prevent food breakage. However, the visual detec- tion mechanism with masks in their framework still limits the range of food states. In this work, explore the potential of active perception to represent foods in an implicit manner, offering an alternative approach for generalizing to diverse states of food.
Active Perception in Closed-Loop Policy Learning. Prior works have utilized pre-interacting data for predicting object physical properties, reconstructing the object geometries, and executing open- loop control for manipulation tasks [25, 17, 26, 18, 27, 19]. However, it remains largely unexplored how the pre- pre-programmed interaction can improve glosed-loop policy learning in an implicit manner. Saito et al. [21] propose sensorimotor dynamical system model using recurrent neural network to learn the pouring task. In testing, robots observe liquid behavior with pre- re-programmedd shak- ing actions and generate pouring motions. In their subsequent work, Saito et al. [22] employ a similar architecture for tool selection and tool-use tasks, enhancing the dynamical model with a multiple-timeseee recurrent neural network. This network has updatable initial parameters, allow- ing exploration during interaction in testing. In contrast, we take different approach enhance the connection and use of interaction data in manipulation. We introduce the state retrieval module, which efficiently extracts crucial information from each step in the closed-loop process, allowing for more effective utilization of the interaction data.
3 Method
Our goal is to design learning framework that enables real-world robot scooping. We develop our method based on active perception and focus on how to utilize the interaction data to improve task success under dynamic situations. Moreover, we adopt the learning from demonstration (LiD) paradigm to enable the robot to learn using a spoon faster and more reliable manner. We assume a demonstration dataset available for model training. Figure shows an overview of our proposed framework.
3.1 Problem Formulation
Task Definition. In our work, the food scooping task is divided into three distinct stages: the interacting stage, the scooping stage, and the transferring stage. During the interacting stage, the robot employs a spoon to stir the contents of the container while observing the dynamics of the food items. the scooping stage, the robot's objective to scoop up the food items successfully. Finally, in the transferring stage, the robot transfers the scooped food to another container. In this work, we focus on learning the challenging scooping process exclusively, while keeping the interacting and transferring stages fixed with pre-defined motions. Task success is determined by specific criteria:
3

---

Ohservation (t-N t)
Zg
Global
Temporal Encoder
cropped
Crop (t)
State Retrieval Module
Robot Policy
π
Action
Zyru
Interactive
Encoder
Zg.
Pre-interacting Sequence
Figure SCONE a real-world robotic food scooping learning framework. With active perception, the generalization capability of the model can be enhanced. SCONE consists of three main feature extraction modules: (a) Global temporal encoder, which able to capture the stage information from the current observations. (b) Interactive encoder, which effectively gathers information and integrates to form a holistic understanding of physical properties of the food items. (c) State retrieval module, which explores and extracts the task rusk-related state information based the current state to facilitate manipulation.
the food items the spoon should cover over one-third area more than one entity without spillage and damage. The spoon must remain at a position higher than the container after scooping motions are performed.
Pre-processing of Interaction Data. To extract the information of food items with greater precision and efficiency, sample K frames (K 7 in case) of observed data from sequences received in the interacting stage Dirigin {o,, e;)li 1,...,N N.. We also crop the viewing area around the container. The sampling starts at the time step when the spoon reaches the lowest position and acquires the data every 0.5 seconds. The pre-processed data denotes Dinteract {(a, e 1,..., K), which has sensor readings o; (JRca, (Depta) and the robot end-effector poses e;-
SCONE Architecture (Fig. 2). At each time step, have the interaction data Dinterect and the N-length history of observation data O, {(o,, ej t ,,....1) 1). They the inputs to the three encoding modules: global temporal encoder fgt, interactive encoder fgi. and the state retrieval module farm to obtain latent vectors Zgt, Zeropped: Zga and Zern- We aim to learn the policy π (as|ze (zgt Rcroppad; zgi, 2arn) that minimizes the standard policy learning loss. In this work, we only control the 3D position and the y-axis pitch rotation of the end-effector, and the action a; E R output from the policy each time step is formulated as delta between the current and next end-effector pose, i.e., e; and e++1-
3.2 Global Temporal Encoder
The aim of the global temporal encoder capture the progressive information from the observa- tions, guiding the policy to be aware of the current phase in the task progress. We two designed convolution neural networks to extract feature vectors of certain dimension from the overall ob- servations of LaGB and Ipepth. To obtain the temporal information, we encode the past N frames of observations (N = 10 our case), so at each time step, the inputs to the observation encoding These two sequences of
module are Oobeerve = (Jии E Nx x x3 IDepta E N XNXXxXXXX different modalities pass through the corresponding encoders fACB and foepth to obtain two sets of feature embeddings. We concatenate them the last dimension. Then, flatten and downsample the feature vectors through fully- connected layer, obtaining global temporal features of current observation Zgt. This would be the overall guidance for the policy model to output actions.

---

3.3 Interactive Encoder
To extract a representation of food items through pre-programmed interaction, we introduce the interactive encoder fgi. which consists of an RGB encoder, a depth encoder, and self-attention module. The sequential data of two modalities Ointeract (JRBB RkxW x3 IDepth E RkXWxHx1) are first passed over the convolution encoder separately, and then concatenated into latent vectors Zinteract RR*D, Afterward, the concatenated embeddings fed into the self- attention module, implementing cross-modality and cross-time attention to catch the critical infor- mation, outputting final flattened latent vectors Egi- The 2gi contains the characteristics of food items in the latent space that can he treated as guidance for the policy model to recognize the similarity between seen and unseen objects.
3.4 State Retrieval Module
We assume that the state of each frame in the interacting stage has a different importance to the state of each time step in the manipulating stage, so the objective of the state retrieval module (SRM) to retrieve meaningful embeddings from the sequences of interaction data queried current observations. To obtain meaningful features for guidance, the state feature extraction layers and the cross-attention mechanism are applied.
To extract the features more precisely, we crop the current observation around the container obtain of grop, which is the same the pre-processing of the interacting data. First, the RGB-D) images in of orop are fed into designed convolution layers to output the feature embeddings separately, and then concatenated with the corresponding end-effector poses, acquiring the state pairs Zeroppad E R and Zinteract E RKXD Next, the attention mechanism is applied to generate an attention map between Acropped and Zinteract- The final feature maps are the result matrix multiplication of the attention map and each frame features in Zinteract. Furthermore, a fully connected layer is used to downsample the output latent vectors into lower dimensions. The output of the state retrieval mod- ule Zerm extracts task-related state information from the pre-interacting data, and implicitly guides the policy model to adapt the policy considering the target item and the interaction experience. Con- currently, the Pcroppad are treated as another input to the policy model.
3.5 Policy Learning
To learn the policy [aa Acroppad: Agii Parn), we adopt Behavior Cloning (BC) to train action prediction from the observation in a supervised manner. The model takes the latent vectors of the global observation, the local observation, the overall food representation, and the retrieved cue under the current state inputs pass over three-layer Multi-Layer Perceptron, and then outputs the corresponding actions. We use the mean square loss between the ground truth action and predicted action a for training the network.
4 Evaluation
4.1 Food Scooping Dataset
Hardware setting. The data collection platform consists of 7DoF Franka Emika Panda robot with the default gripper, D435 RealSense camera fixed around the platform, and two containers placed in the center of the table. During the data collection, receive sensory data of RGB-D images and the robot proprioception simultancously. The robot controller runs at 10 Hz frequency.
Food Categories. In the training stage, we select food items, i.e., brown rice, mung bean, soybean, chocolate ball, dried jujube, and cheese ball) that encompass different particle sizes, shapes, and weights. Additionally, we evaluate 6 unseen food items, i.e., orange, red bean, sago, macadamia, penne, and fruit candy. To assess the generalization ability of our proposed method, we design three difficulty levels. The basic configuration closely mimics the training data, with the container being filled with two-thirds full of food items. These primarily comprise small particles, accompanied by
5

---

smaller quantity of larger particle foods. The extended setting reduces the number of small particles and increases the number of large particles. The peculiar setting includes foods with significantly different appearances and weights compared to those in the training set. These difficulty levels are designed to have a comprehensive assessment of the model's capability to handle wide range of scenarios. Additionally, we have introduced testing scenarios that involve foods with notably different properties, such semi-solid food and liquid. See Supplementary Material for the details.
Data Collection. We employ the hand-guiding mode for intricate scooping operations. This in- volves a human kinesthetically guiding the robot while recording the end-effector's trajectory. We gather 10 demonstrations for each food category, each encompassing 6 seconds of interaction, 15 seconds of scooping, and 10 seconds of transferring. The recorded interacting motions are subse- quently replayed collect demonstrations without humans in the scene. Every trial is thoroughly reviewed to ensure trajectory viability. A trial is considered qualified demonstration if the robot successfully scoops up the food items. In total, we collect 60 real-world demonstrations.
4.2 Baselines
We conduct a comparison with seven different approaches, which encompass various BC-based methods and template-based methods. Please see Supplementary Material for the details of each baseline.
BC-based Methods:
The methods include the standard BC trained on both scooping and transfer- ring data (one stage), standard BC trained w/o interaction data, BC with interaction, and the BC conditioned on food categories represented by one-hot vector.
Template-based Methods:
The method treats the learning process template selection problem. The category-level templates obtained through the mean of all trajectories that belong to the same category. The experiments proceed with selecting templates randomly pre-defined with the trained classifier.
MTRNN [22]: The baseline strings the interacting and the scooping stage as one action sequence. The model trains a multi-timescale recurrent neural network to learn the corresponding system.
4.3 Single Scooping Experiment
In the single scooping task, the robot is required to scoop up food items and transfer them to the target container without causing spillage or damaging the food. The food items on the spoon should occupy at least one-third of its area. All tested foods are uniformly spread in the bowl, ensuring similar initial conditions for all testing trials. The experimental results are presented in Table 1. Overall, our proposed method achieves task success rate of 71% across difficulty levels.
Basic Setting. BC basclines (one stage and w/o interact) perform unsatisfactory in this setting. The incorporation of additional information from interacting data leads to improved performance, with a success rate increase of over 10% compared to the basic BC baselines. Template- methods offer improved stability over BC-based methods, yet there are cases where classifier predictions may lead to the selection of an unsuitable template, resulting in failure instances. For instance, when encountering the food item Sago, the model predicts incorrectly, leading to food spillage. The MTRNN and the SCONE achieve over 80% success rate in each category, while there are failure cases caused by unstable robot arm movements other sources of uncertainties.
Extended Setting. The overall task success rate declines as a result of disparities between the testing and training environments, posing challenges in accurately assessing the states of the target items and leading to failed attempts food scooping. Conversely, our proposed framework ex- cels leveraging the precise state information obtained during the interaction stage, resulting in improved performance. The template- based method (Classified Template) maintains stability when the appropriate template is selected.

---

Basic
Extended
Peculiar
Orange
Red Bean
Sago Macadomia
Orange
Red Bean
Sago Macadamia
Penne Candy
Success
BC (one stage)
1/10
0/10
0/10
1/10
0/10
0/10
0/10
0/10
1/10
0/10
3%
BC (w/o interact)
3/10
0/10
2/10
0/10
5/10
0/10
3/10
2/10
2/10
4/10
21%
BC (w/ food id)
7/10
2/10
0/10
2/10
0/10
0/10
3/10
0/10
4/10
3/10
21%
BC (w/ interact)
7/10
5/10
0/10
3/10
3/10
0/10
3/10
1/10
5/10
1/10
28%
Rand. Template
5/10
4/10
6/10
8/10
0/10
0/10
3/10
3/10
0/10
Classified Template
10/10
10/10
0/10
10/10
3/10
0/10
10/10
1/10
0/10
47%
MTRNN [22]
10/10
10/10
10/10
8/10
4/10
2/10
0/10
3/10
8/10
3/10
58%
SCONE (Ours)
10/10
8/10
9/10
9/10
7/10
5/10
6/10
5/10
7/10
5/10
71%
Table 1: Single Scooping Experiment Results.
Penne
Red Bean
Global Temporal Encoder
Interactive Local Encoder
Success
Latent SRM
AS1 Max/Min Penalty. AS1 Max/Min Penalty..
Rate
Classified Template
5,4
8/2
29%
21%
MTRNN [22]
2.8
42%
2.8
4/2
21%
28%
48%
SCONE (Ours)
3.2
25%
13.4
14/5
22%
71%
Table 2: Continuous Scooping Experiment Results.
Table 3: Ablation Study.
Peculiar Setting. The task success rate of template-based methods significantly decreases in com- parison to previous settings, highlighting the effectiveness and robustness of the close-loop learning- based approach. The methods with interaction outperform other baselines because of their enhanced understanding of food items. SCONE achieves the highest total success rate, which can be attributed its ability to accurately capture task-related information and establish an effective connection be- tween the pre-interaction state and the subgoal state during the manipulation stage.
4.4 Continuous Scooping Experiment
In the continuous scooping task, the robot needs to repetitively transfer food from one bowl to another until it scoops up nothing. Unlike the single scooping task, a model should precisely perceive food properties and states to achieve continuous scooping with changing initial conditions. A notable change from the single scooping experiment is the removal of the criterion involving scooping less spoon as a failure. We conduct a comparison between the proposed method
than one-third and other representative baselines in scooping tasks involving red beans and penne pasta, with five trials for each type of food. SCONE demonstrates the ability to successfully scoop food across intricate variations in food states, with lower proportion of spillage and damage to the food and surroundings. The experimental results are presented in Table 2. AS means the average number of successful scoops per trial, Max/Man represents the maximum and minimum number of scoops in a single trial, and the Penalty shows the total instances of food spillage and food/setting damage, divided by the total number of scoops.
4.5 Ablation Study
We compared four model variants in the single scooping experiment, as shown in Table 3. The base model achieves success rate of 21%. Incorporating interactive data improved performance to 28%, indicating better understanding of the target food items. Moreover, adding the current local observation, which emphasizes the model's attention the crucial stage, results in substantial

---

improvement of approximately 20% in the task success rate. Finally, including the state retrieval module significantly enhanced the success rate to an impressive 71%, showcasing its ability to ac- curately extract and utilize critical information from the interacting data.
4.6 Visualization
Interactive Embeddings. To interpret the behavior of the interactive encoder, we utilize t-SNE projections visualize the embeddings from the interactive encoder. In Figure 3, we show that food items belonging to the same category or possessing similar properties tend to cluster together. This result suggests that the interac- tive encoder effectively captures the inherent similari- ties among the food items.
Benn
Orunge
Brooa
Frait Cludy
Sago
Ming
Red Hoas
(ousad))
Attention Scores from State-retrieval Module. Fig- ure 4 depicts attention scores during manipulation. The x-axis represents the time index in the manipulation stage, and the y-axis shows attention scores for state pairs. The cross-attention mechanism effectively iden- tifies and captures state importance within the interac- dings.
Macadamia (ENELAA)
Reen
(cstead)
Orange (estend)
Figure 3: 2D t-SNE of Interactive Embed-
tion data, enhancing manipulation. Attention scores follow consistent pattern across trials, showeasing the model's ability understand task phases. Notably, even without explicit labeling, the model demonstrated an inherent ability to capture and recognize important information related to different phases of the task. Further analysis can be found the Supplementary Material.
famee]
ftame?
freme3
framno
frames5
frame6
fiame?
frame_1
X-aXis: Manipulation Time index Acteution Score
frae_22
fram_3
Scoop Up
fame_5
frame
20
60
80
100
110
140
Figure 4: Visualization of Attention Scores.
5 Conclusion
We introduced SCONE, active perception- based robot food scooping framework, utilizing in- teracting data in coarse plus fine approach: global encoding for overall physical properties, and state retrieval for critical state information. Our framework dynamically tailors the scooping policy according to real-time observations, enabling the model to excel with previously unobserved food items, thereby elevating both performance and generalization capabilities.
Limitations and Future Work. Our proposed approach places significant emphasis on extracting vital information from active perception, making the design of pre-programmed interaction motions an essential component for attaining successful outcomes. However, the current approach of human- hard-coded-defimee actions may limit the exploration of other possibilities. In our future endeavors, we envision ample room for further investigation into the design of interaction motions, aiming to enhance their exploratory nature, thereby augmenting the system's overall performance. In addition, our emphasis on food property-driven generalization imposes constraints on environmental variables such container positions and tool types. Thus, our goal is explore the integration of perception components, such object detection and pose estimation, into the proposed framework, allowing to adapt to broader range of diverse setups.

---

Acknowledgments
The work is sponsored in part by the Higher Education Sprout Project of the National Yang Ming Chiao Tung University and Ministry of Education (MOE), the Yushan Fellow Program Adminis- trative Support Grant, and the National Science and Technology Council (NSTC) under grants 110- 2222-E-A49-001-MYY, 110-2634-F-002-051,, 111-022--001111111---3------0000 111-2634-T-002--22--- and Mobile Drive Technology
Co., Ltd (MobileDrive).
References
[1] Y. Ohshima, Y. Kobayashi. T. Kaneko, A. Yamashita, and H. Asama. Meal support system with spoon using laser range finder and manipulator. In 2013 IEEE workshop on robot vision (WORV), pages 82-87. IEEE, 2013.
[2] D. Park, Y. Hoshi, H. P. Mahajan, H. K. Kim, Z. Erickson, W. A. Rogers, and C. C. Kemp- Active robot-assisted feeding with a general-purpose mobile manipulator: Design, evaluation, and lessons learned. Robotics and Autonomous Systems, 124:103344, 2020.
[3] L. P. Ureche and A. Billard. Constraints extraction from asymmetrical bimanual tasks and their use in coordinated behavior. Robotics and autonomous systems, 103:222-2355 2018.
[4] Grannen, Y. Wu, S. Belkhale, and D. Sadigh. Learning bimanual scooping policies for food acquisition. In Conference on Robot Learning (CoRL), pages 1510-1519. PMLR, 2022.
[5] H. Ochi, W. Wan, Yang, N. Yamanobe, J. Pan, and K. Harada. Deep learning scooping mo- tion using bilateral teleoperations. In 2018 3rd International Conference on Advanced Robotics and Mechatronics (ICARM), pages 1 118-123. IEEE, 2018.
[6] S. Gushi, Y. Shimabukuro, and H. Higa. self-feeding assistive robotic arm for people with physical disabilities of the extremities. 2020 5th International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBSS), pages 61-64. IEEE, 2020.
[7] D. Park, Y. Hoshi, and C. C. Kemp. A multimodal anomaly detector for robot-assisted feeding using an Istm-based variational autoencoder. IEEE Robotics and Automation Letters, 3(3): 1544 1551, 2018.
[8] Mealtime partners. URL http:://ww.eeealtimpprrreeez com/. [Online; accessed 8- J0n3-2023].
[9] Meet obi. URL https::/neettbbb com/ [Online; accessed 8-June-2023J.
[10] Bhattacharjee, G. Lee, H. Song, and S. S. Srinivasa. Towards robotic feeding: Role of hapties in fork-based food manipulation. IEEE Robotics and Automation Letters, 4(2):1485- 1492, 2019.
[11] R. Feng, Y. Kim, G. Lee, E. K. Gordon, M. Schmittle, S. Kumar, T. Bhattacharjee, and S. Srinivasa. Robot-assisted feeding: Generalizing skewering strategies across food items on plate. In Robotics Research: The 19th International Symposium ISRR, pages 427-442. Springer, 2022.
[12] E. K. Gordon, X. Meng, T. Bhattacharjee, M. Barnes, and S. Srinivasa. Adaptive robot- assisted feeding: An online learning framework for acquiring previously unseen food items. In 2020 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 9659-9666. IEEE, 2020.
[13] E. Gordon, S. Roychowdhury, T. Bhattacharjee, Jamieson, and Srinivasa. Lever- aging post hoc context for faster learning in bandit settings with applications in robot-assisted feeding. In 2021 IEEE International Conference Roboties and Automation (ICRA), pages 10528-10535 IEEE, 2021.

---

[14] R. Bajesy, Y. Aloimonos, and J. K. Tsotsos. Revisiting active perception. Autonomous Robots, 42:177-196, 2018.
[15] Q. Le, A. Saxena, and A. Y. Ng. Active perception: Interactive manipulation for improving object detection. Standford University Journal, 2008.
[16] D. Benjamin, D. Lonsdale, and D. M. Lyons. Designing a robot cognitive architecture with concurrency and active perception. In AAAI Technical Report (5), pages 1-8, 2004.
[17] P. Sundaresan, S. Belkhale, and D. Sadigh. Learning visuo-haptic skewering strategies for robot-assisted feeding. In Conference on Robot Learning (CoRL), pages 332-341. PMLR, 2022.
[18] M. C. Gemici and A. Saxena. Learning haptic representation for manipulating deformable food objects. In 2014 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 638-645. IEEE, 2014.
[19] A. Sawhney, S. Lee, K. Zhang, M. Veloso, and O. Kroemer. Playing with food: Learning food item representations through interactive exploration. In Experimental Robotics: The 17th International Symposium, pages 309-322. Springer, 2021.
[20] R. Ishikawa, M. Hamaya, F. Von Drigalski, K. Tanaka, and A. Hashimoto. Learning by break- ing: Food fracture anticipation for robotic food manipulation. IEEE Access, 10:99321-99322 2022.
[21] N. Saito, N. Dai, T. Ogata, H. Mori, and S. Sugano. Real-time liquid pouring motion generation: End-to-end sensorimotor coordination for unknown liquid dynamics trained with deep neural networks. In 2019 IEEE International Conference on Robotics and Biomimetics (ROBIO), pages 1077-1082. IEEE, 2019.
[22] N. Saito, Ogata, S. Funabashi, Mori, and Sugano. How to select and use tools?: Active perception of target objects using multimodal deep learning. IEEE Robotics and Automation Letters, 6(2):2517-25) 2524, 2021.
[23] C. Schenck, J. Tompson, S. Levine, and D. Fox. Learning robotic manipulation of granular media. Conference on Robot Learning (CoRL), pages 239-248. PMLR, 2017.
[24] S. Clarke, T. Rhodes, C. G. Atkeson, and O. Kroemer. Learning audio feedback for estimating amount and flow of granular material. Proceedings of Machine Learning Research, 87, 2018.
[25] T. Lopez-Guevara, R. Pucci, N. K. Taylor, M. U. Gutmann, S. Ramamoorthy, and K. Suhr. Stir to pour: Efficient calibration of liquid properties for pouring actions. In 2020 JEEE/RS/ International Conference Intelligent Robots and Systems (IROS), pages 2020.
IEEE,
[26] K. Zhang, M. Sharma, M. Veloso, and O. Kroemer. Leveraging multimodal haptic sensory data for robust cutting. In 2019 IEEE-RAS 19th International Conference on Humaroid Robots (Humanoids), pages 409-416. IEEE, 2019.
[27] Lu, R. Wang, Y. Miao, C. Mitash, and K. Bekris. Online object model reconstruction and reuse for lifelong improvement of robot manipulation. 2022 International Conference on Robotics and Automation (ICRA), pages 1540-1546. IEEE, 2022.
10

---

SCONE:
A Food Scooping Robot Learning Framework with Active Perception
Supplementary Material
The supplementary materials consist of:
1. Demonstration video.
2. Details of dataset.
3. Baseline approach.
4. Analysis of experiment results.
5. Details of our proposed method.
6. Visualization of attention scores.
7. Additional experiment out-of-distribution food categories.
Demonstration video
In the supplementary video, we show 1) brief introduction of our food scooping robot learning framework and 2) illustrations of scooping tasks and qualitative results.
2 Details of Dataset
In this section, we provide more details about the data collection and preparation process for training.
2.1 Food Preparation
We select a total of 12 categories of food items for our real-world experiment, with 6 categories used for training and the remaining 6 categories used for testing. To simplify the complexity, we have limited the differences between categories primarily particle size and the amount of food items.
Food in the Training Set. The food categories for training contain brown rice, mung bean, soybean, chocolate call, dried jujube, and cheese ball. For food items with small particles such as brown rice, mung bean, and soybean, we fill the bowl to approximately 2/3 of its capacity. This threshold ensures that the interaction can be carried out without the risk of spilling the food items. For food items with large particles such as chocolate balls, dried jujube, and cheese balls, we select more than one piece but fewer than certain number based on their size. This is done prevent the spoon from getting stuck breaking the food items when there insufficient space in the bowl for the spoon reach them.
Food in the Testing Set. The food categories for testing contain sago, red bean, orange, macadamia, penne, and fruit candy. the testing set, we have designed three levels of difficulty to evaluate the performance of the models. Both the Basic and Extended settings in the evaluation include sago, red bean, orange, and macadamia. and the peculiar setting includes penne and fruit candy.
Basic Setting: In this setting, the conditions kept identical to the training set. For food items with small particles, the bowl is filled to approximately 2/3 of its capacity. For food items with large particle sizes, more than one entity is included in the bowl.
Extended Setting: In this setting, we change the combination of properties related to particle size and amount food items. We want to explore different scenarios to evaluate the performance of our model under varying conditions. For food items with small particles, we fill the bowl with smaller

---

quantity of these items. The intention is to keep the height of the food in the bowl similar to that of the food items with large particles the Basic setting. Conversely, we increase the amount of food
items with large particles.
Peculiar Setting: These food items in the peculiar setting had unique features, such as different shapes, colors, or textures, that are not present in the training set. By introducing these visually distinct food items, we aim challenge the model's ' capacity to recognize and handle novel objects effectively.
2.2 Manipulation Policy
During the data collection phase, we employ two distinct manipulation policies to ensure the suc- cessful scooping of the food items. To prevent spilling, we adopt a specific policy for scooping up food items with small particles. The strategy involves positioning the spoon at a shallow depth under the height of the food items. By adopting this approach, the risk of spillage can be minimized and the food items were securely contained within the spoon during the scooping process. When dealing with food items that have large particles, we position the spoon at the lowest point within the bowl and gently push the items toward the edge. This allows the food to roll into the spoon and is able to successfully scoop up without any spillage. Figure shows the end-effector positions during manipulation in the training dataset.
Brown Rice
Mung Bean
Soy Bean
Fell bodt
Small Particle
Kgps 19A
9187
bape Jo0
PeD 200
6 1P
Cheese Rall
Chocolate Rall
Dried Jujube
H90
Large Particle
900
Figure I: Visualization of End-effector Position During Manipulation.
2.3 Implementing Detail of Overall Scooping Task
The task is divided into three stages: interacting, scooping, and transferring. Among these stages, only the scooping stage requires learning. The overall task follows predefined procedure. The interaction data is collected by replaying the recorded end-effector trajectory. The model then be- gins predicting the scooping stage. Once the scooping stage is completed, the transferring stage is initiated. The trajectory for the transferring stage is obtained by averaging all training data. Consid- ering that the ending position of the scooping stage for every testing food tends to be relatively close to initial position of the transferring stage, the Panda arm then proceeds to move to the first point using simple motion planning. Once the transition is complete, the rest of the trajectory is replayed accordingly.
3 Baseline Approach
This section provides detailed explanations of the baselines in comparison with SCONE.

---

3.1 BC-based Method
When adopting Learning from Demonstration (LfD), behavior cloning (BC) is a straightforward approach for direct learning from both observation and action. We conducted the task using different BC models, both with and without the inclusion of interacting data.
BC one stage. In BC (one stage), we consider scooping and transferring single stage. This implies that the BC (one stage) model is required to learn the long-horizon task only through obser- vation, without any additional input.
BC without interaction. The most basic method that learns the scooping task through observations.
BC with food ID. To implement BC with food ID, applied the trained classification model the interaction data. The predicted food ID was then conditioned as a one-hot vector and concatenated with the observation Zgi. serving as the input for the BC model.
BC with interaction. the case of BC with interaction, the input for the model is the concatenation of latent of observation and interaction data.
3.2 Template Policy
One of the approaches we explore utilizing interaction data is the selection of trajectories from the pre-interacting data based their similarity the food categories in the training dataset. The template trajectory is obtained by averaging the sequences of end-effector pose of 6 different food categories respectively.
Rand. Template. Random policy is selected arbitrarily.
Classified Template. The template policy selected accordingly based on the predicted category by the classifier.
3.3 Dynamical System Model
MTRNN. We utilized the multiple time scale recurrent neural network by [1] to update the initial parameter Csg using the interaction data in the testing stage.
4
Analysis of Experiment Results
4.1
Failure Cases
During our real-world evaluation, the failure cases observed included instances of spilling (SP), insufficient food on the spoon (IF), failed attempts to scoop (FA), collisions (CO), and others (OT). We select the orange, sago, and penne as examples, and their results can be found in Table and T.
Spilling (SP). Spilling happens the end of the scooping process, especially when there is an excessive amount of food on the spoon. Consequently, during the stages of transferring or when withdrawing the spoon from the bowl, the excess food spills out.
Insufficient food (IF). The failure case insufficient food occurs when the amount food on the spoon unable to cover at least one-third of surface area. This is frequently attributed to the spoon not being inserted deep enough into the bowl to effectively reach the desired food portion.
Failed Attempts (FA). Failed attempts to scoop are attributed to the same underlying reason as insufficient food. In these cases, the spoon fails to acquire any amount of food, rather than scooping up an insufficient quantity.
Collisions (CO). Relying on vision-based information can lead to incorrect decisions when facing out-of tuttof-distribution situations. In such scenarios, collisions between the spoon and the bowl may occur.

---

Basic Orange
Basic Sago
SP
IF
FA
Co
OT
Failed
SP
IF
FA
CO
OT
Failed
BC (one stage)
0
5
3
9
BC (one stage)
0
0
0
10
10
BC (w/o interact)
2
3
8
BC (w/o interact)
8
0
0
(w/ food id)
1
BC (w/ food id)
1
5
BC (w/ interact)
0
0
BC (w/ interact)
2
0
0
10
Rand. Template
0
0
0
5
Rand. Template
2
0
2
0
4
Classified Template
0
0
0
0
Classified Template
10
0
0
0
10
MTRNN [1]
0
0
0
0
MTRNN [1]
0
0
0
0
0
SCONE (Ours)
0
0
SCONE (Ours)
0
0
(a) Basic Orange
(b) Basic Sago
Table Basic Failure Cases
Extended Sago
Peculiar Penne
SP
CO
OT
Failed
SP
IF
FA
OT
Failed
BC (one stage)
0
(one stage)
0
BC (w/o interact)
5
1
0
7
BC (w/o interact)
0
6
0
BC (w/ food id)
1
1
5
BC (w/ food id)
0
0
5
0
6
BC (w/ interact)
0
3
0
7
BC (w/ interact)
0
0
4
0
5
Rand. Template
3
0
0
0
7
Rand. Template
4
0
3
0
0
7
Classified Template
0
0
0
0
0
Classified Template
2
0
7
0
0
9
MTRNN [1]
0
5
0
10
MTRNN [1]
2
0
0
0
2
SCONE (Ours)
3
0
0
4
SCONE (Ours)
2
0
0
3
(a) Extended Sago
(b) Peculiar Penne
Table 2: Extended and Peculiar Failure Cases
Others (OT), In addition to the failure cases mentioned earlier, there are instances where the task cannot be successfully completed or the food ends up being damaged.
4.2 Result Analysis
Table 1: Orange.
The BC-based baselines achieved low task success rates due to the high incidence of failed attempts because the weight of the orange used in the testing set is heavier than the foods in the training set. By contrast, the MTRNN and SCONE models exhibited stable performance due their ability learn and understand the conditions for successful scooping from the provided demonstrations. The classified template method also achieved higher performance due to its ability to select suitable templates for food items.
Table Sago. The BC-based baselines encountered challenges in providing correct predictions, causing several failure cases during the manipulation. Moreover, the template-bast based method has been limited by its reliance on the predictions from the classifier. In cases where the classifier selected template intended for food items with large particles, often resulted in spillage. The MTRNN and the proposed SCONE method demonstrated their ability to overcome this challenge, achieving higher success rate.

---

Clobal Temporal Eacader
Interactive Eacoder
RGB
2tt
Seli
Depth
Depth
State Retrieval Module
Cross Atlentoon Module
Zeroped
Current State
Sontroe
7erm
pore
Scace
Interactive
Weighted Sumn
Figure 2: Details Each Module in SCONE
Table 2: Extended Sago. The number of spillages (SP) decreases significantly compared to the basic settings because the amount of sago been reduced, but there was an increase the occurrence of insufficient food (IF) cases, due to the improper depth insertion the spoon into the bowl in most of the methods. The Select Template model classified sago as chocolate balls, resulting 10 suc- cessful trials when following the corresponding template. MTRNN demonstrated poor performance under this particular setting. During the scooping stage, it showed hovering behavior within the bowl and occasionally collided with it. However, SCONE is capable of handling the challenging setting, achieving success rate of out of 10.
Table Penne. To test the models' generalization abilities, we conducted experiments using peculiar-shaped foods such as penne and fruit candy. Though the color of penne looks familiar soy beans, the shapes of them are totally different, which led to failed attempts (FA) when em- ploying the soy bean template. While MTRNN was able to handle penne, the jittering trajectory could lead to instability and spillage (SP), which we would like to avoid in real-world evaluation. Our SCONE behaved more stable and maintained a smooth trajectory.
5 Details of Proposed Method
See Figure for detailed information on each encoding module. All the observation inputs are RGB- images, which are processed through corresponding convolutional layers to extract features.
Global temporal encoder. The global temporal encoder takes input a sequence of current obser- vations of length N. In implementation, set N to 10, which means that the model can access the previous 10 observations within time window of second. This allows the model to capture the temporal dynamics and dependencies in the input data. Then, the sequences of RGB and depth images are processed separately by their respective encoders, and the output features are concate- nated and flattened into one-dimensional embedding. further reduce the dimensionality of the features, a fully- collececnde layer is applied downsample the features to the dimension of 128.
Interactive encoder. We utilize an interactive encoder to process the sequence of observations cap- tured during the interaction stage. The number of frames K is set to 7. Similar to the encoding process in the global temporal encoder, both RGB and depth images within the sequence are pro- cessed independently through their respective convolutional layers. The output features from these layers are then concatenated form a sequence of feature maps, capturing the visual information from both modalities. Then, we introduce the multi-head self-attention mechanism, allowing the model focus relevant parts of the input sequence. The output features passed through fully- connected layer, which reduces their dimension to 128.

---

framel
frame22
frame3
framel
Cames
frame6
frame?
Tame 1
tame__
x-exis: Mimipulttiou Time index
y-axis: Amamion Score
Scoop
tame_
frame_6
BD
120
140
framel
frame22
frame3
framed
Frume5
framefi
frame?
hame_
x-uxis: Mamipalation Time index
frame_2
Attention Score
rame_3
frame
Scoup Up
fram__ 6
80
100
120
140
Figure 3: Visualization of Attention Scores. The upper the manipulation process of brown rice, and the lower is the manipulation process of chocolate ball.
State retrieval module. The state retrieval module tasks the current local observation and the se- quence of interaction as inputs, For the current local observation, which consists cropped RGB-D images captured the current time step, we pass them through an encoding module obtain latent embeddings. These embeddings are then concatenated with the current end-effector pose, result- ing in the latent state representation denoted as >cropped- Regarding the sequence of interaction, apply the same encoding layers to process the observations and obtain corresponding feature em- beddings. Similar to the current local observation, we also concatenate the end-effector poses with the encoded features, creating a sequence of states in the interaction. To retrieve the critical state information, we use Zeroppad as a query and compute relationship with sequence of states in the interaction. This is achieved by employing cross-attention mechanisms that output weighted feature embeddings. These embeddings are then flattened and downsampled to dimension of 32, obtaining the Zerz. Furthermore, also downsample the latent state representation Zeropped to dimension of 32.
6 Visualization of Attention Scores
Figure 3 shows more examples of visualization for attention scores. Based on the results, it is evident that the model can accurately capture the state information without human labeling; this demonstrated by consistent patterns observed the changing attention scores over time at each trial. Overall, our analysis reveals that frame and frame 3 exhibit higher attention scores compared other frames. We attribute this the presence of important state information related to the spoon's 's contact with the food items under specific end-effector poses. These frames can be seen as providing subgoal-like cues, guiding the model in the scooping task. Additionally, we observed that in food items with large particles, frame 3 tends to have higher attention scores than frame initially. This because the state captured frame 3 is more similar to the state prior to the scooping action in scenarios involving large particles. On the other hand, we noticed that the highest attention score for frame typically occurs close to the timing of the actual scooping action. This suggests that frame serves as general guide for determining "how" and "where" to scoop up the food items. These findings highlight the model's ability effectively identify and utilize critical state information from the interacting data to inform its scooping strategy.

---

7 Additional Experiment on Out-of-distribution Food Categories
We conducted additional tests to assess the model's adaptability to food categories with markedly different physical properties compared to the training data. Specifically, we prepared semi-solid pudding and liquid black tea. The proposed method demonstrated a success rate of over 50% for both foods, indicating a notable level of generalization our approach.
Black Tea
Pudding
SCONE (Ours)
8/10
5/10
Table 3: Out-of-DDsttribun Food Scooping Experiment
References
[1] N. Saito, T. Ogata, S. Funabashi. H. Mori, and S. Sugano. How to select and use tools?: Active perception of target objects using multimodal deep learning. IEEE Robotics and Automation Letters, 6(2):2517-25222 2021.


Reviews:
[{"review_id": "yeEDAhmtAQD", "reviewer": "Reviewer_BzGd", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a method for scooping various granular food products using a robot arm and a large spoon.  The robot first interacts with the food by stirring it with the spoon. Then the robot attempts to scoop the food and finally dump it in another container. The observations from both the interact and scoop phases are encoded into 4 latent vectors, which are then fed to a scooping policy. The results show that the robot is able to achieve a 71% success rate on average over all the tasks, and is even able to generalize to food with physical properties slightly outside the training distribution.\n", "summary_of_recommendation": "I recommend acceptance of this paper based on the novelty of its methodology, the challenging nature of the task, and the thoroughness of the results.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "As mentioned above, the paper is in excellent shape as is. Evaluating the policy on other kinds of food (cream, jam, etc.) just to see what it does would add a nice touch to the results, but isn't strictly necessary.\n\nA small nit: On line 170 z_interact is in R^{7 x D} space. Should the 7 be K? If not, can you clarify where that number comes from?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This is a novel method for solving a challenging task. Scooping and active perception are not new in robotics (as the paper points out), but using active perception to perform something akin to latent system identification for non-rigid media like beans and rice is. While the method is fairly complex, the results clearly show that it outperforms the baseline in almost every case. Furthermore, the results are reported as number of success out of total attempts, which is a very intuitive and easy to understand metric. Overall the methodology proposed by this paper shows a novel and interesting way to incorporate active feedback in a challenging manipulation task.\n\nOne thing this paper does really well is thoroughness in experimental analysis. Table 1 shows a comparison of the main method with 7 baselines spanning variations on behavioral cloning, kNN-likes (the template methods), and even an RNN. Furthermore, the methods are all evaluated on a variety of foods with very different properties, from rice to oranges! Finally, the ablation study in Table 2, even though only a small part of the paper, does a good job justifying the existence of the various parts of the model.\n\nOverall, there’s not much to detract from in this paper. Would it be nice if there were more than 10 trials per food X algorithm combination? Yes, but as it stands there are already 800 trials evaluated in the paper, which is approaching the limit of what can be done with human supervision, so it’s not reasonable to expect more trials. However, an interesting follow-up work might investigate using automated data collection that doesn’t require human involvement (e.g., scooping back and forth between 2 containers). It would also be interesting to try this method on semi-solid and liquid foods such as jams, puddings, creams, and even water. But none of this is necessary as this is an excellent paper as is (though authors feel free to try that last idea for the rebuttal, it would really elevate the paper quite a bit).\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "tuE3bQeNmef", "reviewer": "Reviewer_vCX4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents SCONE, a food-scooping robot learning framework that leverages active perception to learn meaningful representations of food. SCONE uses interactive encoding and state retrieval modules to capture essential food characteristics and states. The system exhibits success in real-world experiments, achieving a 71% success rate on unseen food items across varying difficulty levels, outperforming other baselines.", "summary_of_recommendation": "The paper presents an effective approach to robot scooping with active perception. It's technically sound and well-supported by experiments. However, the task's difficulty is unclear. Generalization to varying spatial locations, sizes, shapes, and colors of bowls and ladles isn't addressed. Comparisons beyond behavior cloning and active perception, such as model-based or advanced imitation learning methods, would strengthen the paper. Also, the reliance on hard-coded actions for active perception might limit the robot's exploration and inject human priors. Therefore, I think more works are necessary to validate this method's robustness and applicability.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "1. Have you ablated different architecture designs for each individual encoder?\n1. How would you expect this method to generalize beyond the task of scooping and transferring?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quality: The paper is technically sound and claims well supported by real-world experiments. The idea of active perception for robot scooping is intuitive and effective.\n\nClarity: The paper is clearly written and well-organized.\n\nOriginality: To the best of my knowledge, this work seems to be a novel combination of methods in robot scooping.\n\nSignificance: The idea of active perception has the potential to be applied to other tasks in deformable manipulation since the interaction is a natural way to learn the dynamics of the target objects.\n\nStrengths: \n1. The authors show that their method can generalize to an extended set of unseen materials in different settings and outperform baselines.\n1. The visualizations of interactive embeddings and attention scores are very compelling.\n\nWeaknesses:\n1. I'm a little unsure about how difficult this task is since I imagine an open-loop policy will also have relatively reasonable performance. For example, if the robot inserts the ladle into the bowl, rotates around the $y$-axis by a fixed angle, pushes down, rotates back to align with the horizontal plane, and lifts, how will that end up in the task of scooping? The simplified action space (3D position and the y-axis pitch rotation) also gives the policy learning module some important priors to leverage.\n1. The authors show material-wise generalization, but how about generalizing to different spatial locations of the bowls (e.g., what if the target bow is further away), different sizes/shapes/colors of the bowls and ladles?\n1. It would make this submission much stronger if the authors could compare against methods beyond the paradigm of behavior cloning and active perception, such as model-based methods [[1](https://arxiv.org/abs/2306.16700)] or more up-to-date imitation learning methods [[2](https://arxiv.org/pdf/1903.01973.pdf)].\n1. As the authors mentioned in the limitation section, the hard-coded actions for active perception may limit the possibilities the robot explores and inject strong human priors into the model.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "pLzr64pueO", "reviewer": "Reviewer_S1uK", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors of this paper explored the problem of scooping various unseen food items using a Franka with a ladle. They used active perception with some stirring actions and recorded 7 frames using a pre-specified hard coded motion. Then they ran the 7 RGBD frames into an encoder to get a latent embedding for the food item. Next, they used a closed loop policy that takes in the embedding from the active perception, 10 previous RGBD frames, and a state retrieval module that uses cropped rgb images and attention as input into a policy that outputs a delta action. Their action space is the robot's XYZ and y-axis pitch rotation of the end-effector. They compared their method with a variety of other behavior cloning approaches and templated approaches. They reported a 70% success rate using their metric of no spilling, filling up at least 1/3 of the spoon, and not damaging the food items. They also tested on black tea and pudding, which showed some fair results.", "summary_of_recommendation": "I updated my review to a weak accept because the authors did test on new liquid/viscous objects and the performance was still acceptable. In addition, they added continuous scooping results. While I still believe a well formulated baseline using some image segmentation and a good hard coded scooping action could achieve similar results. It does seem good that it works in a closed loop manner. I would still prefer if there were different containers and tools as well as the skill adapting when things change in the environment such as human interference. I will not argue for acceptance, but it has improved from its original submission. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "1. The limitations section only mentioned the interaction action component, but I think there are more limitations. \n2. The authors didn't use any proprioceptive feedback (force torque sensing) from the robot, which could provide many benefits during the active perception stage. Many previous papers use proprioceptive feedback during active perception.\n3. The self defined criteria for success is not great in my opinion. The authors reset after each trial to a complete starting point with one bowl with the specified amount and the other is empty. Instead, I believe a better metric would be number of scoops before 1 bowl is completely empty with extra penalties for spilling or damaging the food.\n4. I believe the task is a bit too simple. Many people have done scooping from bowls in the past and I feel like a well hard coded scoop action could get a high score on the red beans by just shaking extra beans off in the bowl before transport. Instead I think you should vary the locations of bowls and show that the closed loop control of the robot can still scoop the items from any location using your pipeline. \n5. There is a lack of citations. Some recent papers would be: (a) Excavation learning for rigid objects in clutter by Qingkai Lu and Liangjun Zhang, (b) Soil-adaptive excavation using reinforcement learning by Pascal Egli, Dominique Gaschen, Simon Kerscher, and Dominic Jud, and Marco Hutter, (c) Uncertainty-Aware Self-Supervised Target-Mass Grasping of Granular Foods by Kuniyuki Takahashi, Wilson Ko, Avinash Ummadisingu, Shin-ichi Maeda, (d) Learning robotic manipulation of granular media by Connor Schenck, Jonathan Tompson, Dieter Fox, Sergey Levine, and (e) Learning audio feedback for estimating amount and flow of granular material by Samuel Clarke, Travers Rhodes, Christopher G. Atkeson, and Oliver Kroemer\n6. There should be different container sizes and shapes like plates and larger bowls. But if it requires collecting a massive amount of data to generalize to different containers, then I think there may be an issue with the approach.\n7. There was not any water / liquid substances like jello or pudding, which I think is the most challenging. I feel like scooping of granular material like Connor Schenck's paper has already shown promise if given enough data. Sticky or liquid scooping is far more difficult with higher stakes. The insights from a paper on that would be far more impressive and novel. ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Writing Quality: The writing quality was pretty good. There were only a few mistakes that I noticed in the supplementary video and some places in the paper. \n\nClarity: The paper was pretty clear, although some of the figures could be fixed. I think Figure 2 could be improved as it initially confused me with only arrows going into the state retrieval module and nowhere else.\n\nOriginality and Significance: This paper seems to be a combination of different previous ideas applied to the task of scooping. One would be active perception, another is an attention module, and the last is a concatenation of previous frames into an encoder. I would not consider this a very novel paper.\n\nStrengths: The authors did do a lot of real world scooping experiments. Their method does seem to improve on their other baselines and ablations.\n\nWeaknesses: I have a lot of concerns for this paper. I will put most of this section in the questions for Rebuttal.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "560vpwGCPt", "reviewer": "Reviewer_VGVQ", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper contributes SCONE, which is a robotic food scooping learned method that uses active perception to implicitly learn a latent space representation for food items. This is later accessed with the state retrieval module during manipulation. This paper claims that prior work is limited in that they use hard-coded primitives for food manipulation, but by implicitly learning a representation of food from human trajectories with the human manipulating the food, the model is able to learn food dynamics and alter the scooping manipulation as needed to successfully scoop. ", "summary_of_recommendation": "The work is original and clearly presented. The main concerns for a weak accept over a strong accept is for the following two reasons:\n\n1) Technical quality. The train and test food were very similar so I question the generalizability of the method to unseen food types. They all embodies either a large, spherical food or grains/beans. There was also no report on the number of trials conducted in the ablation study.\n\n2) Limitations. The biggest limitation I saw is all experiments were done on the same \"spoon,\" but it appears to be a very large ladle which makes the problem easier. To address this limitation, the authors should either add this in the paper or test or train&test on various spoons as the method seems to have no reliance on the spoon shape. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1. During data collection, do you collect 1 human demo per food and then replay that same demo 10 times per food for data? Is there no variation, such as injected noise, in the trajectory when replayed?\n\n2. In the ablation studies, how many trials were conducted per ablation? \n\n3. In the basic setting, why does SCONE do worse than the Classified Template and MTRNN baselines?\n\n4. The generalization claim seems unstable because as mentioned above, the train food seems very similar to the test food and it would be nice to have tested food with slightly different shapes and textures. \n\n5. It would be good to have experiments with multiple different spoon shapes. \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work seems relatively original, where in the food manipulation space, it seems to be one of the few to introduce closed-loop control during food manipulation. \n\nThe paper overall is technically sound. The only concern is there seems to be little explanation for failures and the authors seem to imply the failures in the basic setting are mainly due to unstable robot arm movements or other sources. If this is the case, it should be written more clearly. An explanation for the extended and peculiar setting failures is also needed. \n\nThe paper is clearly written and easy to understand. The results show SCONE outperforms most of the baselines and the idea of closed-loop food manipulation with an implicitly learned food representation is a unique contribution. \n\nIt may be interesting to see if similar foods take similar trajectories. One limitation I see is that the test food seemed similar to the train food. Chocoballs appear to be similar to macadamia nuts and red bean is similar to soy and mung bean. It would be nice to see how this would function for more varied food, such as lychee (spikey), cut banana (cylinder shape), boba (or something of a more spongy texture). \n\nAnother limitation in my view is all experiments were done with the same ladle. In the video at least, the “spoon” also appears large, not a normal bite size spoon. It would be good to show results on a few differently shaped spoons as the way the paper presents it, the utensil used for food manipulation should not affect the performance of the method. \n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission first received mixed reviews.  The authors did a good job during the rebuttal phase, and all reviewers recommended acceptance after discussion.  The AC agrees with the recommendation.  Some reviewers remain concerned about the generalization of the proposed method.  The authors are encouraged to further explore this direction based on the reviews.
Decision: Accept (Poster)