SyGRID: Synthetically Generated Realistic Industrial Dataset

Authors: Elena Govi, Davide Sapienza, Luca De Dominicis, Nicola Capodieci, Marko Bertogna
Primary area: datasets and benchmarks
Keywords: artificial intelligence, simulated dataset, pose estimation, industrial automation, rendering

Abstract:
Industrial automation depends on accurate object recognition and localization tasks, such as depth estimation, instance segmentation, object detection, and 6D pose estimation. 
Despite significant advancements, numerous challenges persist, especially within industrial settings. To address these challenges, we propose 
SyGRID, (Synthetically Generated Realistic Industrial Dataset), a new simulated, realistic dataset specifically designed for industrial use cases. 
Its novelty lies in several aspects: the generated frames are photo-realistic images of objects commonly used in industrial settings, capturing their unique material properties; this includes reflection and refraction under varying environmental light conditions. Moreover, SyGRID includes multi-object and multi-instance cluttered scenes accurately accounting for rigid-body physics. 
Aiming to narrow the currently existing gap between research and industrial applications, we also provide an exhaustive study on different tasks: namely 2D detection, segmentation, depth estimation and 6D pose estimation. These tasks of computer vision are essential for the integration of robotic applications such as grasping.
SyGRID can significantly contribute to industrial tasks, leading to more reliable robotic operations. By providing this dataset, we aim to accelerate advancements in robotic automation, facilitating the alignment of current progress in computer vision with the practical demands of industrial robotic applications.

Decision: reject
Reviewer scores: 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730367056111

thread_type:
official_review

<thread_object>
creator: reviewer_digx

<content>
summary:
This paper introduces SyGRID, a novel synthetically generated industrial dataset designed to enhance object recognition and localization tasks critical for industrial automation, such as 2D detection, segmentation, depth estimation, and 6D pose estimation. SyGRID’s key contributions include highly realistic, photo-realistic images that simulate industrial environments with challenging elements like material reflections, refractions, and cluttered multi-object scenes modeled with rigid-body physics. By offering a comprehensive dataset tailored for realistic industrial conditions, SyGRID bridges the gap between research and practical applications, enabling advancements in robotic tasks such as grasping, thereby supporting more robust robotic automation in industrial settings.

strengths:
Realistic industrial dataset is needed for downstream robotic tasks, while this scenario is yet to be well studied and has many challenges, such as the cluttering and transparency of objects. I believe this dataset is practical and will contribute to the development of this field.

weaknesses:
For presentation, I recommend authors to divide long paragraphs into some short paragraphs, which could largely increase the ease of reading of this paper.

For experiments, as the contribution and final goal is to facilitate downstream robotic manipulation, while this dataset is rendered in simulation, I recommend authors to demonstrate both qualitatively and quantitatively how much this simulation rendered dataset can bridge the sim2real gap of robotic perception and manipulation.

For dataset, are there diverse objects in a category? Or the diversity only exists in number of categories?

The real-world robotic manipulation video could be polished if the author presents the policy visualization, instead of only the robotic execution.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730409567982

thread_type:
official_review

<thread_object>
creator: reviewer_l67o

<content>
summary:
The paper introduces a new, synthetic dataset tailored for computer vision tasks, such as 2d object detection, segmentation, depth estimation, and 6DOF pose estimation in industrial settings. The dataset aims to bridge the gap between research datasets and real-world industrial environments by providing high-quality, photo-realistic data with challenging features such as reflections, refractions, and highly cluttered multi-object scenes. The dataset is evaluated on real and synthetic data using 2D bounding box detection, instance segmentation, depth estimation, and instance-level 6D pose estimation.

strengths:
Overall, the paper is clearly written and easy to follow. It introduces prior work on industrial datasets, their strengths and weaknesses, and presents the varying attributes visually through tables. The paper further shows that the presented dataset is more realistic overall compared to prior proposed methods (about on par to YCBV) and, therefore, partially closes the sym-to-real gap.

weaknesses:
I have multiple questions regarding the proposed dataset's motivation, technical contribution, and real-world usefulness.
1. To me, the dataset seems to be ill-motivated. The dataset focuses solely on applying industrial bin-picking with a very specific (and small) number of objects, severely limiting the applicability of the dataset to real-world scenarios. I am not aware of any application where these exact parts must be picked from an unorganized bin by a robotic manipulator. If such scenarios exist, they should be mentioned in the introduction. If the claim is that the dataset can be used for other applications, I would expect some experiments that analyze the generalization capability of algorithms trained on this dataset in real-world applications. I suggest that the paper contains a more thorough motivation of applications in industry where this dataset can be useful.
2. The motivation for introducing this dataset is to handle "occlusions and highly cluttered scenes" and "different light conditions." I would argue that industrial settings are some of the most controllable environments for robots. Not only can the lighting be fully controlled, but I also have trouble envisioning specific tasks where the robot has to pick these specific arrangements of objects. 
3. Regarding the results, the paper claims that the main goal is to reduce the sym-to-real gap in performance. However, Table 2 shows that a large gap still exists between the two data distributions. Similarly, the results show that using this data for training results in widely differing performance on real vs synthetic test data (-15 to -20% on segmentation, 10x difference on depth estimation, between 2-6x large rotation error). These differences in performance show to me that the dataset is only partially useful to generalize to real-world data and does not solve the sym-to-real problem.
4. It is also questionable how a model trained on this dataset performs on out-of-distribution data. Can these models be used on some of the other datasets (Table 1.)? If so, what is the performance? 

Overall, I do not believe that the paper contains enough motivation or technical novelty to justify publication at ICLR. The dataset is too generic in the tasks it addresses without providing real insights into real-world robotics manipulation challenges. I therefore recommend rejection of the paper.

questions:
See above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730602652368

thread_type:
official_review

<thread_object>
creator: reviewer_qksx

<content>
summary:
This paper introduces a new object dataset tailored to industrial scenarios. The dataset is generated using a physically-based renderer (PBR) on 10 distinct objects, considering various materials, lighting conditions, and background textures. The paper also conducts extensive experiments on both simulated and real data to assess the realism of the synthetic dataset.

strengths:
The paper is well-written, clear, and provides a thorough literature review of previous works related to object datasets. The experiments are comprehensive and sufficiently demonstrate the effectiveness of the proposed simulated data in real-world scenarios.

weaknesses:
1. A key concern is the limited novelty of the proposed dataset. While the authors emphasize that the dataset includes cluttered scenes, varied lighting conditions, and diverse object materials, these features are already well-supported by modern PBRs [1,2]. Previous works, such as Omni6D [3], have incorporated similar capabilities in the rendering of object datasets. It would be helpful for the authors to further clarify how their dataset differs from prior work in this domain.

2. The dataset contains only 10 object instances, which is quite limited compared to other object datasets. Industrial environments typically feature a much larger variety of objects. Expanding the dataset to include more object types would improve its diversity and applicability.

3. Although the dataset features cluttered scenes, it appears that only top-down viewpoints are used to capture the images. Incorporating images from different viewpoints would enhance the dataset’s diversity and better reflect real-world industrial scenarios.


[1] https://dlr-rm.github.io/BlenderProc/examples/datasets/bop_challenge/README.html

[2] https://docs.omniverse.nvidia.com/isaacsim/latest/replicator_tutorials/tutorial_replicator_pose_estimation.html

[3] Zhang, Jiyao, et al. "Omni6DPose: A Benchmark and Model for Universal 6D Object Pose Estimation and Tracking." arXiv preprint arXiv:2406.04316 (2024).

questions:
From a practical perspective, is it realistic to find industrial scenes where transparent objects and tools are piled together? And is it common for different tools to be stacked separately?
</content>
</thread_object>

</thread>


Metareview: