Venue: NeurIPS 2021 Track Datasets_and_Benchmarks Round2

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This work propose to adapt the simulator parameters to one specifically design structured light scanner so that the rendered scans are close to the captured scan images. The tuned simulator can be used to generate 10K photorealistic scanning images for 1000 CAD objects from ABC dataset under this hardware setup, which can be used to train and benchmark post-processing models/algorithms for 3D scanning, particularly depth denoising, depth completion and surface reconstruction from point cloud. Besides those synthetic data, this work also provides real captured scans for 4 calibration objects and 7 color textured objects (3D printed from models on Sketchfab).
strengths: - a solid setting for benchmark 3d recon for structure light scanning, including both hardware and simulator, and plenty of details on the hardware, renderer, etc in the supp materials
- rigorous data gathering pipeline, and detailed data generation process, presented in the supp material, including preprocessing, data source, code to reproduce
- really like the docker image and try out the jupyter notebook for rendering scan images, decoding and reconstruction

weaknesses: - the title is a bit overstating the scope of this paper, it seems the current capture, rendering and datasets are all for structured light scanning, but not 'general' 3d reconstruction, also the abstract does not explicitly point it out, but tell the story from a very abstract level
- as pointed out in the supp material, the 3 benchmark problems using this dataset/setup are all postprocessing algorithms/models for 3D scanning, which are decoupled from the specific scanning mechanism, e.g. ToF, LIDAR, etc. But it is not clear how useful the datasets generated from SLS can be transfer to other scanners? e.g. what if I have a room level scanning data from iPad pro, point cloud of a outdoor driving scene?
- this is stated in the limitation section, there is not much variation of difficulty or diversity of lighting and material

correctness: I am not familiar with SLS, can not comment on the correctness of hardware setup and rendering (w/ mitsuba)
clarity: well written
relation to prior work: I think there are lots of sim-to-real works in robotics / self-driving, so the claim in L94 is a bit bold
additional feedback: - L49: difference between 3 precisely-machined calibration objects and 7 color-textured 3D printed objects? any considerations for choosing those particular objects as the testset? are they diverse enough as testset to evaluate the generalization of new reconstruction models?

- L103: "We are not aware of any existing work able to achieve a faithfulness comparable to our approach, especially on geometrically complex objects"  --- this is a bold claim

- L250: what's the resolution of 250 samples? bitdepth? Is it too small as training set? what the distribution it can represent? could it too restrictive to generalize only to the CAD models from ABC dataset?

- Table 1: CNN performs too bad on normal direction, have you consider surface normal loss additional to the depth loss? 

- Fig 11: it looks like denoised depth introduces large positive error (red points). is it expected?

- L254: can I understand the Shape completion task considered here is grayscale image (diffusely illuminated rendering) guided depth completion?

- L269 & Fig 14: the generalization is shown from synthetic to real on CAD models, how about to the real objects?

- I run the ipynb for simulating scans, decoding and reconstruction. It’s a very nice and smooth experience. Any idea for why the recon for vase misses point clouds near the neck?
review_title: solid work on structured light scanning hardware and simulator co-design for data capture and generation, but also restrictive to this scanning setting

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper introduces a novel hardware system along with a rendering pipeline to collect a benchmark dataset for 3D reconstruction and preprocessing tasks. Overall, three calibration objects and seven textured scanned objects are provided. Three benchmark challenges are designed to advance data-driven models for 3D reconstruction and geometry processing: 1) surface reconstruction from point clouds. 2) denoised 3D geometry reconstruction from noisy geometry. 3) 3D shape completion from partial geometry. For each task, several existing methods are evaluated as initial baselines.
strengths: + The 3D dataset (10 objects) that includes both scanned results and CAD models are useful to quantify the performance of existing 3D reconstruction and postprocessing algorithms.

+ A novel system and associated calibration script can be used to simulate the scan of any 3D object. This allows a user to customize their dataset with their own 3D models.
weaknesses: - Weak justification for the novelty of the proposed dataset. The novelty of this data comes from the fact that it provides a 3D CAD model and the paired real scan. However, it is not fully justified why we indeed need re-scan of the 3D CAD model. Why not simply reconstruct a 3D geometry from a set of synthesized 2D images (e.g., multiview stereo from synthesized multiview images) from a 3D CAD model? 

- Weak scalability in data capturing pipeline and lack of dataset. The data capturing pipeline is not practical to collect a large amount of data. It requires printing out a physical object and re-scanning the 3D-printed objects. This leads to collecting a small amount of textured dataset (ten objects). Compared to the previous 3D model dataset [1,2,3,4,5], this paper shows weakness in the size and diversity of the dataset.

[1] 3D human body dataset: https://dfaust.is.tue.mpg.de/, 
[2] Non rigid object dataset: http://tosca.cs.technion.ac.il/book/shrec.html
[3] 3D object dataset: http://modelnet.cs.princeton.edu/
[4] 3D chair and vehicle dataset: https://shapenet.org/
[5] Many textured 3D cad model dataset: https://github.com/timzhang642/3D-Machine-Learning

- Insufficient discussion on the previous 3D dataset and weak comparison with them. The current related work and experiment are not sufficient to localize this paper to the state of the art. A deeper discussion and analysis on why existing datasets are not sufficient, what their limitations are, and how this paper addressed those limitations are required to highlight the novelty of the proposed dataset. For example, 1) the number of dataset and their types can be summarized in a table; 2) a paragraph that discuss why the proposed dataset is better than others in terms of what perspective is needed; 3) a cross dataset evaluation can be made using multiple 3D model datasets (e.g., training a model on the previous dataset, and testing on the proposed dataset). 

- Weak formulation on benchmark challenges. The suggested three benchmark challenges do not effectively demonstrate the quality and novelty of the proposed dataset. Most importantly, it seems possible to enable the suggested challenges even without the proposed data. The current benchmark formulation does not answer the following questions. 1) Denoising tasks: why not simply add synthetic noise on existing CAD models? More importantly, how meaningful is it to handle submillimeter-scale noise (i.e., the quality of the reconstructed surface already looks good)? 2) Shape completion task: why not simply add synthetic holes on the CAD model?; why can the previous 3D model dataset not be used for this task? 3) 3D reconstruction tasks: it is not clear what the authors wanted to deliver from this task. Why is it challenging to reconstruct 3D models from clean depth maps and calibrations?; why can the previous 3D model dataset not be used for this task?


correctness: The proposed dataset is constructed in a sound way.
clarity: The paper is clearly written.
relation to prior work: It is clear how the system differs from previous works, but not clear the dataset. It would be nice if the authors provide a clear summary of the number of dataset and its category in the context of previous works.
additional feedback: Please see the weakness section for the main comments.
review_title: Review for ID 75

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper proposes a technique to jointly scan an object and render realistic images. A dataset is gathered using the technique consisting of 3 calibration objects and 7 3D printed objects, of which CAD models are available. Scans are also simulated for 1000 objects from the ABC dataset. Careful setup along with optimization of calibration parameters and materials enables high rendering accuracy of the simulated environment, enabling supervised 3D models to train in on realistic images.
strengths: The broad theme of optimizing calibration and material in synthetic environments to improve rendering is exciting to the field of 3D vision, as fully-supervised 3D is challenging to obtain on realistic-looking images. This paper provides a small step towards this end.

This setup enables the unique situation of data-driven scan denoising, and qualitative results are impressive. Similarly, data-driven shape completion enables results such as highly detailed hole-fillings in Figure 14.

Hardware setup and data gathering are meticulous. They yield a large set of HDR images for which precise 3D data is available.

weaknesses: Limitations in scope: while I acknowledge collecting data at such precision is difficult and scaling up may not be feasible, the following two points are nevertheless limitations of the paper.
1. Minimal real data: this dataset consists of only 10 real objects. I acknowledge the scans have high detail and therefore memory, but from a practical standpoint 10 objects is a very small sample to draw any “real-world” data conclusions. 
2. Highly-controlled, limited data setting: the paper refers several times to scenes – in 3D this is where we most need realistic and annotated data. However, in addition to real objects being limited in number, they are indeed single objects, not scenes, are of approximately uniform size, and have existing CAD data available. This limits the method from being applicable beyond a small domain.

Evaluation of setups has no baselines or ablations: although reconstructions look good, it is difficult to assess the performance without comparisons to prior work. More explanation is needed. See Correctness.

The paper explains the proposed data-driven method underperforms simple baselines due to lack of normal orientations. However, this does not give evidence in favor of data-driven scan denoising.

correctness: While the data collection is well-described, its benchmarks are not. Instead of presenting an experiment to compare different methods on an important task, the benchmarks typically run a out-of-the-box method and show one or two qualitative results. This seems unlikely to be broadly useful to the scientific community, and does not help back the paper’s claims of facilitating data-driven training on high-resolution 3D.

For instance, line 268-271: “The preliminary results suggest that the model trained purely on synthetic data can generalize to our real world 3D scans. The resulting depth maps have less holes from occlusion and are better suitable, for example, for surface reconstruction tasks.” Yet, these results refer to a single figure, which is a qualitative example and contains no comparison or ablation.

clarity: In general, the paper is well written and easy to follow. Please see my main criticism of comparison to related work below, which confused me at first read.
relation to prior work: Fair. The paper overviews relevant work across a few subfields, which is helpful as the paper is somewhat cross-disciplinary. However, limited reference to prior methods used during evaluation e.g. Image-To-Image Translation or Shape Completion. One additional drawback is that I am unable to find citation or detail for “ABC” dataset contained in the paper. This is a critical component of the paper; please include in final version.
additional feedback: None at this time.
review_title: Exciting direction of work, while benchmarks make assessing its benefit difficult.

Meta-review: The paper develops a 3D scanning setup and rendering pipeline that allows for near pixel-perfect matches between photographs of the scene and renders of the scene. The paper argues that this enables researchers to develop models for various 3D tasks on simulated data generated from their pipeline, and introduces three benchmarks based on simulated data. All reviewers agreed that the task is important, and that the careful co-design of the scanning hardware and rendering pipeline are technically impressive. Reviewers were initially put-off by the small number of actual scans provided, but after discussion agreed that the emphasis and true value of the work are not the scans of actual objects, but instead the data generation pipeline that allows for generating highly realistic synthetic data. In the end all reviewers were marginally in favor of accepting the paper. The AC agrees with the conclusion reached by the reviewers, and thinks that the synthetic data generated from the proposed pipeline is interesting and likely to be useful. Congratulations on having your paper accepted to the NeurIPS 2021 Datasets & Benchmarks Track!

When preparing the camera-ready version of the paper, authors are strongly encouraged to take into consideration the feedback from reviewers. In particular, the AC agrees with reviewers that the title of the paper should be changed (as the authors have already agreed to do); the authors should also consider revising the paper to better emphasize the importance of the synthetic data generated by the pipeline.
Decision: Accept