Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: The paper contributes a large dataset of simulated fractures on various mesh objects assembled from publicly available datasets. Their earlier contribution of generating more accurate fractures than procedural methods without (very computation intensive) full physical simulation was used to generate plausible fractures. Various possible use-cases of the dataset has been indicated.

Benchmarking of various architectures for shape assembly as well as the difficulty of the task has been demonstrated.  

Compared to earlier shape assembly datasets, their contribution is in generating realistic fractures which means the fractured pieces themselves may not represent semantic part individually. This presents a more challenging task whereby understanding of semantic object composition only may not be sufficient. 
strengths: The dataset is a good contribution towards benchmarking methods for shape assembly and towards geometric shape understanding. As demonstrated in the paper, the complexity of the shape assembly task especially when the fracture results in multiple pieces, has been demonstrated via benchmarking of the selected baseline architectures. 

The dataset defines a challenging task where the inductive bias and applicable priors are weaker since the fractures are geometric rather than semantic. This might help push current state of the learning algorithms. 


weaknesses: Not necessarily a weakness, but the choice of objects in the dataset which includes everyday objects seem to imply open-ended applicability/use-case rather than a focused narrow domain of subtask, for example, say a femur fracture dataset.
correctness: Yes. Although the dataset uses meshes from already available datasets, the said contribution is useful and correct.
clarity: Yes, the paper is well written and is organized such that the text flows naturally. 
relation to prior work: Yes, relevant literature is discussed. 
additional feedback: The design of the dataset seems to be open-ended in its applicability/use-case rather than a focused narrow domain of subtask. Maybe the method used to develop this dataset can potentially be useful in the medical imaging community where data acquisition can be expensive and synthetic datasets may help (for example, for generating femur fracture dataset).
review_title: Challenging Dataset for Shape Assembly and Geometric Understanding Task  

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: The paper proposes Breaking Bad, a new dataset containing over one million fractured objects. While most object part datasets use semantics to disassemble objects, a physical fracture does not follow semantics. The paper samples 3D models from Thingi10K and PartNet, simulate fracture patterns and build the whole new dataset. It brings new challenges and benchmarks for 3D geometric shape assembly.

strengths: - I think this is an interesting and real problem to trackle. While prior works only focus on assemblying 3d shapes with semantically well-defined parts, it brings a new challenge.
- This is a large-scale datasets with more than 1M 3D models. These 3D models are open-source.
- The paper is well-written with fantastic figures.

weaknesses: I don't see any obvious weakness. I agree the chosen fracture simulation method brings fundamental limitations. And the dataset is synthetic instead of real. However, as the first attempt, I still think it's valuable to our community.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: N/A
review_title: Review

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper proposes a large-scale fractured object dataset by applying an off-the-shell fracture simulation algorithm on existing 3D object datasets. The objects are assumed to be made from solid isotropic materials. For each 3D object in the dataset, 20 fracture modes are pre-computed and 80 fracture patterns are simulated. The authors show the value of the proposed dataset by benchmarking learning-based shape assembly methods.
strengths: 1. This paper successfully applies a fracture simulation algorithm to creating a large-scale fractured object dataset. The dataset is potentially helpful for studying the shape assembly task. 
2. The data collection and processing pipeline is well explained, making it possible for other researchers to expand the dataset when necessary.
3. Good dataset analyses and visualizations are provided.
4. A feasible data compression practice is used so it is much easier to download the dataset.
5. The authors develop a benchmark for the shape assembly task with appropriate data splits and evaluation protocols, which can significantly facilitate the research on this problem.
weaknesses: 1. As the authors point out in the paper, this dataset is strongly biased by the underlying fracture simulation method. Although the authors deliberately select object categories that fit the simulator’s characteristics, there is still some domain gap between the simulation and the reality. Given a shape assembly method that works well on the proposed dataset, it is not clear how it can generalize to real-world data. It would be nice if the authors can discuss this domain gap and provide insights on what use cases this dataset is not suitable for.
2. The authors may need to justify the usage of the specific simulation method. Computational efficiency does not seem to be a valid advantage for dataset generation, as everything can be processed offline. If there is a computational expensive method that can simulate more realistic fracture, it should be the preferable one. Moreover, it might be even better if multiple simulation methods are used for data generation, as it may be able to mitigate the data bias.
correctness: Yes. The claims made in the submission are correct. The dataset is constructed in a sound way. The evaluation methods and experiment design are appropriate and performed correctly.
clarity: Yes. The paper is well written and easy to follow.
relation to prior work: Yes. The difference of this work from previous works is clearly discussed.
additional feedback: See "Weaknesses".
review_title: A simulated dataset for benchmarking geometric fracture and reassembly

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a large-scale dataset of fractured objects in a physically realistic scenario. The dataset consists of 20 daily object categories and artifacts with a series of fractures at various scale levels. The simulation of fracture modes is fast and produces impact-dependent fracture patterns. Three SOTA deep learning based shape assembly methods for semantic assembly, are benchmarked on the proposed dataset. These methods are hard to assemble fractures produced by impacts. The dataset facilitates several research works and is useful in different scenarios.
strengths: * The proposed geometric fracture dataset is large-scale, physical realistic and considers diverse breaking patterns in different level of scales. Breaking Bad consists of 10,474 shapes, with 100 breakdown patterns in each shape. The decomposition is geometric-based, which is different from the semantic-based datasets, e.g. PartNet. The produced fractures are closer to the realistic scenario.
* The generation of the dataset is fast. Besides, the dataset is compressed for efficient storage, which can be easily accessed. 
* The benchmark for 3D geometric shape assembly is interesting. The results show that current SOTA semantic assembly methods fail to assemble well geometrically. There is still a large space for improvement. Besides, the benchmark analyzes the influence brought by the number of fractured pieces. It shows that a larger number of pieces leads to a more challenging  situation. The dataset is worthwhile to further explore.
weaknesses: * The case study only includes 3D geometric shape assembly. I hope to see some possible benchmarks for downstream tasks to validate its value, such as tasks like missing fractured pieces study mentioned in the future work.
* The validation of the dataset should be included. I believe that there exists some fractures generated are not truly realistic. Please explain how to deal with these pieces and provide failure case analysis.
correctness: The claims are correct. The dataset is constructed well. The benchmark for fracture assembly is appropriate and performed correctly.
clarity: Yes, the paper is well written.
relation to prior work: Yes, the relation to prior work is discussed in the related work.
additional feedback: The Breaking Bad dataset is a valuable dataset for 3D shape assembly and can facilitate the 3D shape geometric analysis. I will consider improving the rating after receiving the responses for the weaknesses.
review_title: The introduced Breaking Bad dataset is interesting and valuable for 3D community.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper proposed a dataset for geometric fracture and reassembly, based on 3D geometry. Different from some dataset that considers the assembly as a graph relationship for different semantic parts, this dataset provides the fragments that are generated from a destruction process for an object. 

For the generation of these fragments, the authors use an existing algorithm to simulate the destruction process. Also, the authors benchmark three deep learning baseline methods, showing further effort is needed in this direction.

I think the motivation of this dataset is strong, as it could help many potential domains like manufacturing, archaeology, and computer vision.  However, some questions should be answered and some issues should be addressed before the paper can be accepted.
strengths: 1. It is the first large-scale dataset for reassembly based on physical simulation, which could be helpful for domains like manufacturing, archaeology, and computer vision.

2. In the meanwhile, the efficiency of generating desctructed compenets is high, which provides the possibility to further increase the amount of the data. Also, the authors proposed a lossless compression for the data access, which could help the use and spread of the dataset in practice.


weaknesses: 1. My major concern about this paper is the baseline adaptation, considering the poor performance when the number of fractures is more than two. The authors should have more analysis for the baseline methods.
Other questions:
First, can the author consider using other 3D data types, like signed distance function (SDF) for the learning purpose? As it may have a better ability than just using point cloud to learn complex shapes. Second, though using the Euler angle to parameterize the rotation is intuitive, it might be problematic when the rotation angle is large. Using 6DOF representation[1] or Bingham rotation[2] might help improve the performance. And RMSE and MAE for the Euler angle might also be problematic since the Euler anger is discontinuous. 

2. The authors should mention more about the physical simulation assumption to help the user understand under what assumptions and conditions are these data generated. I suggest the authors condense Line 115-128 and refer them to the original paper for replacement.

3. The author should prove the simulation is a good approximation to the real-world destruction process, so it is meaningful for the users to use it.

4. It would be better if the authors can consider different materials for the objects.

5. For the simulation, should the balancing weight omega be varied during the data generation process? Why did the author fix 0.001 here?

Last, I am not sure if the contribution of this paper is substantial, since, for the data generation, the authors are using an existing algorithm.

[1] Zhou, Yi, et al. "On the continuity of rotation representations in neural networks." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.
[2] Peretroukhin, Valentin, et al. "A Smooth Representation of SO(3) for Deep Rotation Learning with Uncertainty." Proceedings of Robotics: Science and Systems. 2020
correctness: The claims are all correct to the best of my knowledge. 
clarity: The paper is overall well written, and the logic of this paper is also sound to me.
I have one question regarding the writing:
First, for Tab.2, if 25/50/75-th is representing the data sample at the 25/50/75 percentile, why do the authors divide it by the number of objects?

relation to prior work: The authors are using an existing simulation algorithm to generate a dataset. The related work part is clear.
additional feedback: Can the authors consider a loss that can constrain the connection of each fragment? From Fig. 5 we can see many parts are not connected.
review_title: A nice dataset, but several improvements could be make, especially for the baseline design

Meta-review: metareview: The paper contributes towards generating realistic fractures where the fractured pieces themselves may not represent semantic part individually. This presents a challenging task which requires understanding beyond semantic object composition. The reviews are aligned in the largely positive reviews (4 positive, 1 negative) for the paper and the authors have successfully addressed the weaknesses raised by the reviewers.
I believe that this is an important contribution towards the study of geometric fracture and reassmebly and should be accepted.

confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept