Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: - [C1] Open Set Learning, which is a task of classificing data over the known categories but that are OOD, is widely studied in 2D. In contrast, there were no 3D dataset or benchmarks that attempt to solve the Open Set problems.
- [C2] They forumalize Open Set Learning problem and benchmark typical methods.  They evaluate two scores: AUROC(Area Under the Receiver Operating Characteristic) and FPR95(False Positive Rate of OOD examples when choosing a normality threshold for which the true positive rate of in-distribution example is 95%).
- [C3] They organize their data from weel known 3D objects datasets: ShapeNerCore, ModelNet40, and ScanObjectNN. On top of the proposed data, they also organize two benchmarks: synthetic to synthetic(S2S) and synthetic to real(S2R) benchmark.
- [C4] They evaluate various approahces that are feasible to handle challenges of Open Set Learning. They compare for 1) discriminative models, 2) density and reconstruction based models, 3) outlite exposure with OOD generated data, and 4) representation and distance based models.
strengths: - [S1] Although I’m not an expert in this area, I did not have problems of reading this article. The paper is well-written and claims are followed by supporting evidence.
- [S2] Well-designed experiments and tasks. Motivation is clear and paper is well organized.
- [S3] They experimentally demonstrate that traditional methods on Open Set 2D task do not fit on Open Set 3D task, indicating there are many challenges that researchers should solve in the proposed task.
weaknesses: - [W1] Changing the number of points can also affect the performance. In other words,  both train and test set share the number of points on data so that we cannot clearly say that the splited sets have different domains. Considering real-world situation such as autonomous driving, the authors should also handle this issue.
- [W2] According to Table 1, the reported score does not show clear tendency. For me, it was hard to interpret the message from the Table 1. It would be better to change the table to graph and put the table on the supplementary materials.
- [W3] For Section 3.3, many contents here are repeated from Section 2.2. It would be better to put the details on the supplementary materials.
correctness: No problems at all.
clarity: Well written. (Mentioned on the Strength part)
relation to prior work: Yes.
additional feedback: Well-orgnaized paper. 
It would be better to add experiments regarding the number of points.
review_title: Great motivation and supporting experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: The authors introduce a novel open set 3d learning testbed based on 3 existing 3D datasets, with several experiment settings in terms of category semantic shift and also covers both in-domain and cross-domain scenarios. They also provide extensive comparisons on these benchmarks and give detailed analysis over them.
strengths: 1. This paper proposes a first testbed for open set 3d learning, which will facilitate the future research and applications on robots and autonomous systems;
2. This paper investigates several algorithms extensively and revealing their strengths and limitations for further research on open set 3d models;
weaknesses: 1. The PointNet++ backbone always outperforms DGCNN on the synthetic to real benchmark while DGCNN outperforms the PointNet++ on the synthetic benchmark, could the authors provide more deep analysis on it;
2. The synth → real benchmark seems to be very sensitive to data augmentation tricks, I doubt whether the dataset settings are proper and consistent to verify the capability of the models.
3. The overall novelty is limited since it only re-organizes three existing 3D datasets and conducts experiments with several existing methods;
4. Results on real → real benchmark is desired;
5. Some typos should be fixed, e.g., L115: Y_t should be Y^t, L328: Tab. 4.3 should be Tab. 4.
correctness: Yes, the experiment design are appropriate and performed correctly.
clarity: Yes, the paper is easy to follow. Though some typos should be fixed.
relation to prior work: Yes.
additional feedback: No additional feedbacks.
review_title: Propose a open set 3D learning benchmark with extensive evaluations.

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: This paper focus on building a benchmark on OOD detection task for point cloud. It is the first benchmark for open set 3D learning. The proposed benchmark considers two tracks: synthetic -> synthetic and synthetic -> real, which is reasonable for the 3D scenario. To better analyze the situation on OOD detection, authors adopt series of 2D OOD detection methods and directly apply them on point cloud classification methods.  The experiments discover the SOTA method for open set 3D learning, and analyze the advantages and limitations of existing methods.  
strengths: * The paper presents the point cloud open set learning benchmark firstly and considers two benchmark tracks for synthetic and real point clouds. 
* The problem definition is clear, the metrics for evaluating the performance are fairly correct and datasets are sufficient for point cloud OOD detection.
* A thorough analysis of 2D OOD detection methods in 3D situation is explicitly demonstrated in experiments. The results show that 3D open set learning has space for improvement, since SOTA 2D methods are not good enough in  3D situation. 
weaknesses: 1) This paper builds a new benchmark for 3D open set learning, which is necessary for 3D literature. However, there is little discussion about the difference between the 2D and 3D open set learning. One obvious difference is the input data structure. Besides, the benchmark almost depends on the 2D counterpart. I think the discussion about the challenges for building 3D compared to 2D open set learning is essential.
2) The performance metrics need more detailed definitions with formulations and citations, since the paper build a new benchmark for a new task.
3) The baseline approach MSP [18] also uses the AUPR metric for evaluation and the mean Pred. Prob as a comparison for the advantages of new metrics (AUROC, AUPR). But this paper only use the AUROC. I think authors need an explanation for metric selection, since a reasonable benchmark should include comprehensive metrics.
4) For the synthetic to real benchmark, it introduces the domain shift in the task. However, the benchmark is for evaluating OOD detection and focusing on discriminating OOD examples. The domain gap is another variable influencing the results in Tab. 3. A real to real track may better analyze the 3D open set learning for comparison.
correctness: The evaluation methods and experiments design are correct but not enough, as mentioned in the weakness 4.
clarity: The paper is written clearly.
relation to prior work: The differences from 2D open set learning need more discussion.
additional feedback: I recommend a borderline-accept for this paper due to the weaknesses. Therefore, the judgement is subject to change according to the other reviews and the rebuttal.
review_title: This paper presents a new benchmark of OOD detection for 3D point cloud. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The paper presents a benchmark for out-of-distribution detection on 3D point clouds. Their benchmark comprises two tracks. The first one is purely synthetic and based on the ShapeNetCore dataset. In the second one, the training set is also synthetic and based on ModelNet40, while the test set contains real point clouds from ScanObjectNN. The authors benchmark multiple OOD methods and reveal room for future improvement.
strengths: - The paper tackles an important problem that has been largely ignored by the research community so far, namely the detection of OOD samples in point cloud classification. The provided benchmark takes a good first step into filling this gap.

- The authors evaluate numerous existing methods that are well-selected. Their benchmark indicates that existing OOD methods from the 2D domain do not transfer well to 3D. This leaves room for future research to improve in this area.
weaknesses: - My main concern is that all methods are trained on exclusively synthetic samples. The benchmark lacks an evaluation of methods on point cloud data obtained from real sensors. Furthermore, all samples are of relatively low resolution (1024 3D points). Hence, an evaluation on this benchmark leaves the question how well the results transfer to more realistic real-world settings.

- The synthetic benchmark consists of three subsets that are described as increasingly difficult in the paper (l.152). However, I did not find an explanation as to why one subset should be more difficult than another?
correctness:  
clarity: The paper is well written and easy to follow.
relation to prior work: Prior work is sufficiently described.
additional feedback: - The FPR95 values for almost all evaluated methods seem extremely high. In the Synthetic-to-Real benchmark, the top performing methods report a false positive rate of around 70% to 80%. Is there an explanation for why the performance is this poor?
review_title: Review: Towards Open Set 3D Learning: Benchmarking and Understanding Semantic Novelty Detection on Pointclouds

Meta-review: metareview: This work for the first time proposes to study the task of Open Set 3D Learning for 3D point cloud data. The authors have conducted extensive experiments under different settings with varied category semantic shifts and provided comprehensive experiments benchmarking the popular methods from 2D open set learning which leads to some important conclusions about the transferability of the 2D methods to the 3D settings. The contribution of the work is clear and novel. The AC believes the paper provides important findings for the community to be aware of.

During the rebuttal, reviewers raised up concerns regarding the lack of real-to-real setting and the missing of some evaluations/metrics. The authors have added the requested materials in the revised paper, which addressed most of the raised issues. While there are some minor questions asked by the reviewers, the authors have carefully addressed them and the AC does not think they are major issues preventing me from accepting this paper. The final scores are 3 accepts and 1 reject, and two reviewers confirmed their final decisions after rebuttal. The reviewer who gave the reject review didn't come back for responding to the authors' rebuttal and the AC think the authors have addressed his/her questions well.

Therefore, the AC is very confident in recommending an acceptance of this work to the track. But, please carefully revise the final paper for the camera-ready submission, based on the reviewers' suggestions. Congratulations! 
confidence: 5: The area chair is absolutely certain
Decision: Accept