Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper presents MultiScan, an RGBD dataset for indoor scenes with semantically annotated 3D objects. The acquisition approach is similar to ScanNet [7] which used many users with commodity iOS and Android devices with active LiDAR sensors. Compared to existing 3D datasets, this paper achieved dense textured meshes, multi-level hierarchical annotations (needs elaboration) and object parts correspondences with respect to time and motion changes. 

While the paper presented an interesting extension to existing large scale datasets, I think this paper is more suited for the NeurIPS Datasets and Benchmark track. 
strengths_and_weaknesses: The strength of this paper is that it presented a large scale dataset with a comprehensive pipeline for annotations. It also has advantages for recording scenes at multiple timestamps with object motions such as opening a drawer/window. 

For the weaknesses, the paper does not present the details for the hierarchical part labels and how it can be evaluated/benchmarked. It is also not clear how the hierarchical annotation is different from ScanNet [7].
questions: None.
limitations: I think the limitations have been addressed.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper proposed a new dataset and annotation methodology for 3D scans with articulated objects. Unlike previous datasets such as ScanNet or ARKit scenes, this dataset includes part-level semantic segmentations part mobility parameters.  The authors create a specialized annotation tool for labels
strengths_and_weaknesses: Strengths:
* This is the largest dynamic scans dataset to date. The authors propose an annotation pipeline which allows scans to be labeled and collected at scale.
* Scans are captured using common devices which reflect real use cases.
* I think this dataset could be useful to researchers working on dynamic reconstruction and part mobility prediction.

Weaknesses:
* It's difficult to assess the quality of the 3D scans in the paper and supplemental pdf.  From the visual examples provided in the paper, there appear to be of lower quality than other similar datasets such as ARKit scenes. Given that this is a dataset paper, some example meshes should have been included in the supplemental to better assess the quality of the reconstructions.
* The main body of paper should clearly state what annotations are provided with the dataset. All descriptions are of the mobility annotations are quite vague. The most informative descriptions I could are:
Ln 43: a dataset of densely annotated 3D interiors with object, part, and part mobility annotations
Ln 159: Object and part instances are correlated across scans of the same scene with consistent object and part IDs
Ln. 172: We define an articulated object to be an object consisting of rigid parts that are connected by joints

From these descriptions I am still not exactly what form of annotations are provided (i.e. object poses, axes of rotation, motion constraints?). Also what if the scene contains multiple instances of the same object (for example multiple instances of the same chair). In that case, it doesn't seem possible to match instances. I believe Rescan [9] solved this problem using a permutation matrix between object instances.

Basically, the Dataset (Sec 4) of the paper really needs to be expanded whereas some of the details in the acquisition and processing stages can be moved to the supplemental.

* Most of the related work section is focused on static datasets (e.g. ScanNet, ARKitScenes) whereas the overall goal of this dataset is more similar to Rescan [9]. I think more discussion of dynamic datasets is warranted. 
* The motion parameter annotation methodology is not really a novel contribution as the authors note in the supplementary material that it is adopted from Xu et al.[47] with the addition of defined open/closed states. 
* I wasn't able to completely understand the problem statement in 5.2 (Mobility prediction). From my understanding, the input is a point cloud Q at time t and the goal is to predict point cloud Q' at time t`. Is this problem even possible? If a door is opened, wouldn't it be equally probable that it (1) opens (2) remains half-open or (3) becomes fully open? I feel like I am missing something here if the authors could clarify that would be helpful.
* I am not really sure if there are any particular benefits of this dataset that could not be achieved by scaling up Rescan [9]. What sort of articulated objects captured by this approach would not be possible with the rescan methodology?
questions: * Are the instance segmentations provided consistent across scans? What if there are multiple instances of the same object?
* Regarding the last point above, could the authors provide some clarification regarding the mobility prediction task?
limitations: Limitations are adequately discussed
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents a new RGBD dataset called Multiscan focusing on articulated objects. When it is released, this dataset would provide raw RGBD data, camera poses, geometry reconstruction, and semantic annotation at the object level for each scan. The paper provides detailed explanation on data acquisition and annotation. It also provides benchmark on instance/part segmentation, and mobility prediction in the experiment section. 
strengths_and_weaknesses: Strengths:
- The dataset especially with the articulate annotation is useful for the community.

- The code and data will be publicly available under permissive MIT and CC-BY-NC licenses. Since the capturing and annotation is quite efficient, other researchers could build upon the code to expand the dataset.

Weaknesses:
- Depth map resolution:
Compared to other RGBD datasets with a relatively high-resolution (e.g. at least 480*640) depth maps, the resolution in this dataset is much lower since it used mobile device for scanning. I suspect this could cause problems like losing details of object geometry when doing TSDF Fusion using low-res depth maps. Can the authors comment on this? 
Another related comment: Although using mobile device for scanning can certainly help the scalability of the data acquisition, they often provides worse depth maps than dedicated depth sensors, such as Microsoft Azure. Did the authors compare the depth map quality between depth sensors when deciding the scanning device? 

- Demo in supplementary material
It would be helpful if the author could provide an example of the scan and annotations.

- missing explanation on some details
At line 132, " We also filter out pixels where depth changes by more than 5cm between adjacent frames" Why is this useful and correct?
At line 135-136, how the voxel size and truncated distance are selected?
questions: - Scanning device and depth sensor consistency
The authors mentioned they developed both iOS and Android app for scanning, but in the paper, seems like only iOS devices are used. Is it correct? If both iOS and Android are used, presumably they use different depth sensors, then should make sure the same scene scanned by different devices should produce the consistent reconstruction.
limitations: N/A
review_title: null

Meta-review: metareview: The reviewers tend to agree on the value of this 3D dataset, but point to some questions about labelling and accuracy.  The rebuttal very convincingly addresses these points, clarifying the novelty and value of this new dataset.

I agree with the authors that datasets are clearly in scope for the main NeurIPS program and that the datasets track explicitly includes as a FAQ:   "My work is in scope for this track but possibly also for the main conference. Where should I submit it?" with the answer "This is ultimately your choice".   

confidence: Certain
Decision: Accept