PokeFlex: A Real-World Dataset of Deformable Objects for Robotics

Authors: Jan Obrist, Miguel Angel Zamora Mora, Hehui Zheng, Ronan Hinchet, Firat Ozdemir, Juan Jose Zarate, Robert K. Katzschmann, Stelian Coros
Primary area: datasets and benchmarks
Keywords: Deformable objects, Robotics, 3D mesh reconstruction.

Abstract:
Data-driven methods have shown great potential in solving challenging manipulation tasks, however, their application in the domain of deformable objects has been constrained, in part, by the lack of data. 
To address this, we propose PokeFlex, a dataset featuring real-world paired and annotated multimodal data that includes 3D textured meshes, point clouds, RGB images, and depth maps. Such data can be leveraged for several downstream tasks such as online 3D mesh reconstruction, and it can potentially enable underexplored applications such as the real-world deployment of traditional control methods based on mesh simulations.
To deal with the challenges posed by real-world 3D mesh reconstruction, 
we leverage a professional volumetric capture system that allows complete 360° reconstruction. PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping objects onto a flat surface or by poking the objects with a robot arm. Interaction forces and torques are also reported for the latter case. 
Using different data modalities, we demonstrated a use case for our dataset in online 3D mesh reconstruction. We refer the reader to our  [website](https://anonymized-pokeflex-dataset.github.io/) or the  [password protected supplementary material](https://drive.google.com/drive/folders/1d8iNoJZ0dUVlzP6XxP7xwGPhdVtwQ7du) for further demos and examples (password in pdf).

Decision: reject
Reviewer scores: 5, 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730563951261

thread_type:
official_review

<thread_object>
creator: reviewer_bd6s

<content>
summary:
The paper mainly introduce a deformable objects realworld datasets featuring real-world paired and annotated multimodal data. The datasets can be mainly used for online 3D mesh reconstruction. The dataset has different data modalities and is primarily collected using a professional capture system, with the collection process also incorporating cameras such as Kinect and RealSense, and simultaneously synchronizing with the robotics system. The paper further shows the application of this datasets on mesh reconstruction and has comprehensive evaluation of learning-base reconstruction.

strengths:
+ RealWorld dataset: The most prominent feature of this paper is that it presents a real-world dataset featuring real-world meshes. Benchmarks for real-world meshes in robotics are relatively scarce. Additionally, the paper demonstrates good reproducibility, providing detailed 3D printing information in the appendix.
+ Multimodal and Rich: Furthermore, this dataset is a multimodal dataset that, in addition to using high-cost acquisition systems, incorporates the commonly used RealSense camera as an eye-in-hand sensor and the Kinect as an eye-on-base sensor. The settings align well with robotic tasks and visual tasks within the field.
+ Comprehensive evaluation on reconstruction task: Finally, the paper conducts detailed experiments on reconstruction tasks, including the use of RGB data and multimodal data.

weaknesses:
+ The purpose of the dataset: One major issue with this paper is that it seems there are few other application tasks beyond online mesh reconstruction for which the dataset can be utilized.  Although online mesh reconstruction is an interesting computer vision task in itself, it appears that there are few practical applications for online mesh reconstruction. Therefore, this work seems to fall short of expectations in various fields. I will elaborate on this point from three different areas.
     + robotics: In robotics, especially in the field of deformable object manipulation, meshes are rarely used; point clouds are more commonly employed[2][3][4]. The main challenges in this area are often related to occlusion and pose estimation issues. For example, Garment Tracking [1] addresses the problem of garment point cloud completion, but this work does not seem to tackle occlusion problems, thus reducing the significance of online mesh reconstruction in this context.
     + CV: This task has limited difficulty in the computer vision field, primarily because the occlusion problem is less significant and the deformation of plush toys is minimal compared to garments. For instance, in dropping tasks, there appear to be only translational and rotational changes. Real-time 6D pose detection has already been extensively studied, so this part of mesh reconstruction can be easily addressed using mesh + pose estimation techniques.
+ Number of object: The scale of 18 objects is small, and each object providing only two trajectories (poking and dropping) results in a limited data size. This makes it difficult to serve as a large-scale evaluation and can only be used for real-world fine-tuning. Papers in similar categories  such as garmentlab[2] have scanned nearly 100 objects, including at least 20 complex plush toys.
+ Object Selection: The deformable objects chosen for this paper are somewhat too simple. There are many more complex plush toys, such as teddy bears, which could also be used for these tasks and would better reflect real-world scenarios. These more complex objects often involve occlusion issues, making the work more meaningful. Most of the objects selected in this paper are simple shapes like cubes or cylinders, which limits the scope and applicability of the task.
+ Action Selection: The actions chosen, poking and dropping, are overly simplistic and lack practical value, especially in the robotics domain. The paper could benefit from including more complex actions, such as pick-and-place, which would add significant practical value and better reflect real-world robotic tasks.
+ Simulation Evaluation: The authors mention in the paper that existing simulation methods require real-world fine-tuning. However, simulations actually perform very well in tasks such as dropping and poking, for example, using the FEM algorithm in IsaacSim. The authors need to include validation within simulators to verify whether the meshes scanned in the paper can be used for simulation and to conduct a thorough analysis of the differences between current simulation algorithms and real-world conditions. This would strengthen the setting of the paper.
+ Action Protocol: The action protocol in this paper is vague. I recommend that the authors follow established protocols from datasets like YCB or GarmentLab[2] to refine and standardize their action protocols. This would enhance the clarity and reproducibility of the experiments.

[1] GarmentTracking: https://github.com/xiaoxiaoxh/GarmentTracking

[2] GarmentLab: https://garmentlab.github.io/

[3] UniGarmentManip: https://warshallrho.github.io/unigarmentmanip/

questions:
+ Groundtruth mesh: If my understanding is correct, the ground truth mesh is obtained using an MVS (Multi-View Stereo) system. The paper's discussion of this point is somewhat unclear.
+ Multimodal: In typical robotic tasks, the position and action of the end-effector are also crucial. Here, the authors use a transparent stick for poking, but the dataset should incorporate the point cloud or mesh of the stick into the trajectory. This can be simply achieved by combining the robot's position with the mesh of the stick. Without this integration, it is difficult to effectively utilize the robot action information provided in the dataset.
+ Error of UR force and torque: The force and torque errors of the UR robot are significantly higher compared to other robots, such as the Franka Emika robot, as noted in the official product sheets.[1] The authors should discuss the impact of these errors on the experiments and results presented in the paper. 

[1] https://assets.ctfassets.net/oxcgtdo88e20/4zQbvCXiDH2RzaaiiuwWci/ca242c85669825e8c25ff3467f8774da/UR5e_Product_Factsheet_EN_WEB.pdf
</content>
</thread_object>

</thread>
<thread>
creation date:
1730566035000

thread_type:
official_review

<thread_object>
creator: reviewer_gn5g

<content>
summary:
The paper presents PokeFlex, a dataset that features real-world paired and annotated multimodal data, including 3D textured meshes, point clouds, RGB images, and depth maps. This dataset aims to enhance various downstream tasks and to demonstrate its effectiveness in online 3D mesh reconstruction.
PokeFlex consists of 18 deformable objects with varying stiffness and shapes. Deformations are generated by dropping these objects onto a surface or by interacting with them using a robot arm, with corresponding interaction forces and torques recorded. Additionally, the authors introduce a novel deep learning model to tackle 3D mesh reconstruction using diverse input data types.

strengths:
The PokeFlex dataset stands out as an original contribution to the field. It features a diverse set of objects, including 3D-printed and everyday objects. This variety enriches the dataset and makes it applicable to a wide range of research scenarios. The inclusion of different types of input data—such as 3D textured meshes, point clouds, RGB images, and depth maps—enhances its utility for various downstream tasks.
The paper clearly conveys the significant effort involved in acquiring this dataset, highlighting the meticulous process behind its compilation. Additionally, the introduction of an innovative deep learning model to address 3D mesh reconstruction challenges is particularly noteworthy, showcasing the authors' commitment to advancing the field. Furthermore, the paper is well-structured and easily readable, making it accessible to a broad audience.

weaknesses:
The paper presents results for only a limited subset of the 18 objects in the PokeFlex dataset—showing performance for just one object in some instances, five in others, and graphs for three. To better assess the dataset's overall effectiveness, it would be beneficial to provide results for all objects. This comprehensive analysis would establish a solid baseline for future comparisons.
Moreover, the choice of a transparent acrylic stick used to poke was not adequately justified or discussed. The implications of this choice on RGB and depth data need more attention, particularly regarding how reflection and refraction may distort images and how time-of-flight depth readings may be inaccurate around the stick.
To further evaluate the dataset's effectiveness, it would be helpful to have a benchmark model trained on the dataset itself to determine its robustness and construction quality. While the paper mentions the "dropping" method, the experiments reported focus solely on "poking." Including results from the dropping method would provide additional insights into the dataset's applications.
Additionally, the model that utilizes multiple data modalities only explored combinations of images and robot data. It would be interesting to see how other combinations of data types perform. 
Moreover, in Equation 3, the threshold epsilon is not defined, nor is there clarity on how values like 0.2 or 0.5 are determined. Addressing these points would enhance the clarity and rigor of the research.
Additionally, several citations in the references appear to be incorrect or improperly formatted, specifically the third, thirteenth, fourteenth, eighteenth, and twenty-ninth citations.
Another point concerns object licensing: one of the objects (Stylized pizza slice) in the dataset is reportedly subject to a non-distributable license, yet this restriction is not noted. This oversight may lead to potential misuse or distribution issues.
Finally, neither the full dataset nor the trained deep learning model is currently available, which limits reproducibility. The sample dataset provided in the supplementary materials includes only one object, a dice, rather than a representative subset or the entire collection. Making the full dataset and model accessible would not only improve transparency but also enable a more comprehensive evaluation by the research community.

questions:
1) In Equation 3, the value of epsilon is not specified. What value does epsilon take to define that the point p_i​ is close enough to the contact point t?
2) In Equation 3, how was the value of 0.2 chosen, and what is the reasoning behind this choice?
3) Why is the ROI loss weighted by 0.5 in the total loss calculation?
4) In Section 4.2, it is mentioned that the train-validation split was generated by randomly selecting one sequence per object. Are the sequences consistent across all objects? For the multi-object training split (Section "Learning from different data modalities"), how was the split handled? Did you consider a sequence in the validation set for one object that fell into the training set for another?
5) For the single-object training described in the section "Learning from RGB images of different cameras", could you provide results for additional objects? Results from only one object do not yield a statistically significant sample for generalizing findings.
6) For Figure 11, could you include data for additional objects (potentially in the supplementary materials)? In this case, what is the training setup—is the dataset composed of 20k samples, or just the 8.4k active paired poking frames? If the dataset used was the full 20k samples, how was it balanced to avoid an overrepresentation of frames without poking compared to those with poking?
7) Still regarding Figure 11, the x-axis represents distance values—how many samples are averaged at each value? Are there more samples at 0.1 than at 0.4? If so, how is this imbalance managed during training and validation?
8) Were environmental conditions, such as lighting, explicitly considered during dataset acquisition? Was any analysis performed to evaluate how variations in lighting or surrounding conditions might influence the networks used?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730567588748

thread_type:
official_review

<thread_object>
creator: reviewer_rqv9

<content>
summary:
In this paper, the authors collected a real-world multimodal dataset of 18 deformable objects. The dataset includes 3D textured meshes, point clouds, RGB images and depth maps.The authors use a volumetric capture system to build the 360degree reconstruction. It includes two types of deformation, poking and dropping. 

The author further use the dataset to train neural network models for deformed mesh reconstruction based on template meshes and different modalites included in the dataset. The neural network model architecture is reasonable and the perforamnce is good on the different trajectories of the same objects.

strengths:
The main advantage of this dataset is that it contains paired 3D meshes and contact forces. They further design a deformation mesh reconstruction method which takes the template mesh and force or image of point cloud sequece as input. By using the GT deformation meth for training, the trained model performs good on different trajectoris of the same objects.

weaknesses:
1. According to Table2, the academic value and the difference between the submsion and PLUSH(Chenetal.,2022) should be further discussed, since it is not difficult to reconstruct the meshes from point clouds. Why do we need the mesh? What other application can it enable on top of deformation reconstruction?

2. The generalizability of the network on unseen objects is not evaluated, which may be the most valuable part of the model. If the model can only work on trained objects, the usefulness of this method is very limited.

3. The technical novelty of the network model is limited and not validated through ablation studies.

questions:
1. What are the densesyntheticPoint Clouds(5kpoints) and SparsesyntheticPoint Clouds(100points)? Why are the error still large when using synthetic point clouds? 

2. What is NVP? IT should be explained briefly to make the paper self-contained.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730767311259

thread_type:
official_review

<thread_object>
creator: reviewer_m6yb

<content>
summary:
The paper introduces PokeFlex, a new dataset focused on real-world deformable objects for robotics applications. The dataset includes:

- Multimodal Data: 3D textured meshes, point clouds, RGB images, and depth maps.
- Objects: 18 deformable objects with varying stiffness and shapes, including everyday items and 3D-printed objects.
- Interaction Protocols: Deformations induced by poking with a robot arm (with force/torque data) and dropping onto a flat surface.
- Purpose: For downstream tasks like online 3D mesh reconstruction and enable applications such as real-world deployment of control methods based on mesh simulations.
- Data Acquisition: A volumetric capture system for 360 deg. reconstruction and integrates lower-cost RGB-D sensors for reproducibility.
- Baseline Models: Shows the use of PokeFlex in online 3D mesh reconstruction using different data modalities and provides evaluation criteria.

strengths:
- Rich, Real-World Dataset: PokeFlex offers a comprehensive multimodal dataset with 3D meshes, point clouds, RGB images, depth maps, and force/torque measurements, captured with a high-quality volumetric system. This real-world data bridges the sim-to-real gap, enhancing applicability in deformable object manipulation.
- Reproducibility and Accessibility: Open-source 3D-printed objects and affordable sensors (Azure Kinect, Intel RealSense) make the dataset accessible and reproducible, while baseline models and evaluation metrics support benchmarking and further robotics research.

weaknesses:
- Limited Object Diversity: While the dataset includes 18 objects, the diversity may still be limited compared to the vast range of deformable objects encountered in real-world applications. Notably, the dataset focuses on volumetric objects and excludes thin deformable items like cloth or cables, which are important in areas like garment handling.
- Controlled Interaction Protocols: The poking and dropping protocols are specific and may not capture the full spectrum of possible deformations in less controlled or more complex environments. Additional manipulation actions could enrich the dataset.
- Potential Bias in Data Collection: The use of a transparent acrylic stick for poking and specific dropping heights might introduce biases that limit the generalizability of the dataset to other types of interactions and objects.
- Related Work Coverage: The paper could benefit from a more thorough comparison with existing datasets and approaches in the literature, such as DOFS [1], CEPB [2], and other relevant works on deformable object datasets and manipulation. Discussing how PokeFlex complements or improves upon these datasets would strengthen the contribution.

References:
[1] Zhang, Zhen, et al. "DOFS: A Real-world 3D Deformable Object Dataset with Full Spatial Information for Dynamics Model Learning." arXiv preprint arXiv:2410.21758 (2024).
[2] Tripicchio, Paolo, Salvatore D’Avella, and Carlo Alberto Avizzano. "CEPB dataset: a photorealistic dataset to foster the research on bin picking in cluttered environments." Frontiers in Robotics and AI 11 (2024).

questions:
- Q1: How do you plan to address the limitations in reconstructing fine-grained details for smaller objects in future work?
- Q2: Are there plans to extend the dataset to include thin deformable objects like cloth or ropes, which are significant in many robotic applications?
- Q3: Do you intend to incorporate more complex or varied manipulation protocols beyond poking and dropping to simulate a wider range of real-world interactions?
- Q4: Is there any intention to provide additional annotations (e.g., segmentation masks, keypoints) to facilitate other types of learning tasks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732181908698

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
Dear reviewers, we are pleased that:

- Reviewer **gn5G** acknowledges PokeFlex as a novel contribution,  finds the variety of data modalities beneficial for downstream applications, and recognizes the substantial effort invested in collecting this dataset and the authors' dedication to advancing the field.
- Reviewers **m6Yb** and **BD6s** appreciate the comprehensive multimodal nature of PokeFlex and acknowledge the authors' efforts in ensuring reproducibility and accessibility.
- Reviewer **RQv9** recognizes the inclusion of 3D meshes and contact forces as a key advantage of PokeFlex.

## General concerns
In the following, we address the general concerns of the reviewers:

- **Response 1**: We decided to change the title of the paper to: "*PokeFlex: A Real-World Dataset of **Volumetric** Deformable Objects for Robotics.*", to clarify the scope of the initial version of the PokeFlex dataset wrt cloth items and ropes (**m6Yb**, **BD6s** ). This change is now reflected in the latest version of the paper. 

- **Response 2**: Below, we present a non-exhaustive list of potential applications for volumetric deformable objects that  PokeFlex enables, to highlight the usefulness of our dataset and the importance of having a textured 3D mesh representation (**BD6s** , **RQv9**):

   - Material parameter identification for FEM simulators.  
   - Real-world markerless deployment of traditional control methods for shape control using FEM simulators.
   - Learning dynamics models for volumetric deformable objects in a low data regime.
   - View-agnostic online 3D mesh reconstruction from static cameras, using virtual cameras and the 3D textured meshes.
   - Online 3D mesh reconstruction from moving cameras, using a moving virtual camera and the 3D textured meshes. 
   - Synthetic data generation for other geometry representations such as NeRFs, Gaussian Splats, SDFs.

- **Response 3**: We extended the initial experiments to include all the objects of the dataset (excluding the Pizza object for now -*Note 1). The experiments below have already been incorporated to the latest version of the paper: 
    - **Experiment 1**: Learning from RGB images of different cameras, including additional tests for lighting conditions for RGB images (Table 3) (**gn5G**). 

    - **Experiment 2**: 3D mesh reconstruction of different data modalities (Table 4 -*Note 2), including new histograms that show the data distribution according to different deformation levels (Appendix A6) and a detailed breakdown of the per-object performance for models trained from sequences of images, images + robot data, and point clouds (Appendix A5) (**gn5G**).
      -  Appendix A.5: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA5.pdf
      -  Appendix A.6: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA6.pdf

    - **Experiment 3**: Generalization of 3D mesh reconstruction to unseen objects. Training on 13 objects, testing on 4 unseen objects (Appendix A7) (**RQv9**).	

      -  Appendix A.7: https://github.com/anonymized-pokeflex-dataset/anonymized-pokeflex-dataset.github.io/blob/main/static/docs/PokeFlex_ICLR_2025_AppendixA7.pdf

- **Response 4**: Regarding the action protocols (poking and dropping),  we agree that more diverse actions could be useful for generalization and standardization (**m6Yb**, **BD6s**). However, achieving high levels of generalization for deformable objects purely based on real-world samples is challenging due to the high dimensionality of the state and action spaces associated with deformable objects.  We argue that the action protocols employed for PokeFlex are a solid starting point for future investigations of material parameter identification methods for FEM simulators to reduce the sim-to-real gap.  A possible approach to achieve generalization is then to leverage simulation models that are tuned with real-world data. Other manipulation strategies that we want to explore for future work are indeed pick-and-place interactions and shaking. The latter in particular should be able to excite most deformation modes. However, including such actions would lead to a decrease in the quality of the reconstructed meshes because of the occlusions generated by a parallel gripper, which goes against one of the initial goals of PokeFlex, which is to provide high-quality meshes. Considering such actions is therefore out of the scope of the first iteration of the dataset.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a dataset called PokeFlex, which contains real-world deformation data for 17 objects, captured through poking and dropping. The dataset includes various modalities of data, such as reconstructed 3D meshes, forces/torques, RGB images, and depth images. The authors describe how the data was collected and discuss potential applications of the dataset, such as deformation prediction.

All reviewers gave negative scores and raised similar concerns about the usability of the proposed dataset, particularly due to the limited number of objects, the diversity of objects, and the range of actions. Despite the experimental results and additional experiments presented in the rebuttal, the reviewers were not convinced about the dataset’s usability, and the AC concurs with the reviewers' consensus. As a result, the AC made the decision to reject the submission.