Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a method for unsupervised pretraining of models for large-scale point cloud processing. The learned representations can be used for finetuning across datasets for various tasks such as point cloud classification or segmentation. The experiments show that pretraining the representation with local and global context improves performance compared to training a model from scratch.

strengths_and_weaknesses: Strengths:
The idea of transfer learning has been shown to be highly effective for learning in the 2D image domain. Adapting this concept to 3D data is an exciting and promising research direction. The experiments also seem to validate that unsupervised pretraining can lead to better-separated feature spaces and improve performance on downstream tasks. 

* The paper is clearly written and easy to follow, well polished, and does not contain many errors. The paper also contains supplemental material with additional details for reproducibility and additional results. 

* The paper motivates the proposed approach by showing a gap in the literature, focusing mainly on data containing a single prominent object. In contrast, the paper proposes to study the problem of contrastive pretraining of large-scale point clouds containing multiple objects. 

* The provided ablations and visualizations of the t-SNE embeddings are very helpful in solidifying the point that the learned features are more informative and better separated in feature space than the features learned from scratch or by the baselines.

Weaknesses:
* As I am not intrinsically familiar with the literature on contrastive learning on point clouds, it was unclear what the intuition was behind global features. Here are some pointers towards making this clear: 
1. In Figure 2. what is the dimension of the global feature queue? 
2. In Section 3.3: What is the intuition behind global features? What do they represent? How is the global feature queue constructed and updated? Are random f2 features from the momentum encoder added to this queue or do some criteria choose them?

* One point the paper should address is how the proposed method compares against methods that use diverse, labeled datasets for pretraining. I think it would be interesting to see how the proposed method performs against a method that, for example, uses SUN and ScanNet training sets for pretraining. I am especially interested if the proposed method's resulting feature space is still more informative (I think it might be).

* Code and data are not released with this paper. Is there a reason why the paper does not contain this or will this be released upon acceptance?

Minor issues:
* Algorithm 1: "Computer local contrastive" -> Compute local contrastive
* Line 275: "the the" -> the
* Line 278: "This is hypothesis" -> This hypothesis
questions: * In the experiments, all datasets contain similar scenes, so the global features encode some common context between scenes. What happens if the context is changed drastically? For example, from urban outdoor environments to more rural ones with less man-made features and more natural ones? Would this method suffer in these cases?

* Line 265: "On the other hand, we observe that the gains are reduced when additional annotated data is provided, as seen in the case with 5% and 10% annotations, as expected." Why is this expected? Please elaborate.

limitations: As far as I can tell limitations of the approach are not discussed in the paper. Consider discussing limitations in the conclusion section.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper proposes a self-supervised pretraining method for large-scale point clouds (SSPL). SSPL is motivated by the observation that most 3D scenes don’t consist of a single, large, foreground object but instead contain many objects of equal scale and importance. To account for this, SSPL introduces a contrastive term over local crops in the same scene and couples this to a standard global contrastive loss. SSPL implements this algorithm using standard techniques from the SSL literature. The resulting algorithm outperforms previous SSL algorithms on several 3D datasets for object detection and semantic segmentation.

strengths_and_weaknesses: **Strengths**

- The paper addresses an important problem: developing SSL methods beyond the simple, highly stereotyped settings (ImageNet, single-frame ScanNet) for which standard global SSL techniques are engineered.

- The proposed method is straightforward and elegantly reuses features for both local and global contrastive losses, which in principle (and at adequate scale) should allow it to learn features that are strictly more discriminative than local- or global-only contrastive methods.

- The approach is well-motivated both in terms of the literature and in terms of the limitations of existing SSL methods for representation learning on point-clouds.

- The approach produces compellingly strong results against fair SSL baselines on several competitive benchmarks in the point cloud domain.


**Weaknesses**
- Table 1: only SSL methods are presented. These are problems that are well-addressed by existing supervised learning methods, so baseline and SotA supervised results are stronger baselines: SotA supervised on ScanNet is >70% AP25. These numbers should be reported to make it clear that while SSPL narrows the gap, it still performs significantly worse than supervised methods.

- The proposed method makes few assumptions that are specific to point clouds (other than the augmentation strategy). It would be fantastic to see this work applied to other domains, such as images (where most SSL methods are currently evaluated), audio, text, multimodal, or other domains. Is there an algorithmic reason to focus on point clouds exclusively? If not, why is the evaluation limited to point clouds, which are less competitive than other SSL domains?

- While the method is evaluated on two tasks (*detection* on ScanNet/SUN and semantic *segmentation* on SemanticKITTI and Waymo Open Dataset) it is not evaluated on multiple tasks on a single dataset. A key reason to investigate SSL at all (rather than directly optimize a task of interest, which works on this data) is that it allows direct application to multiple downstream tasks. The case for this method would be much stronger if it led to simultaneous improvements on several of the tasks supported by ScanNet, rather than just one.

questions: - How dependent is the performance of the method on the augmentation used? Augmentation is the trick by which contrastive SSL methods of this kind are given information about the structure of the target domain. So additional ablations or analysis would be helpful for interpretation of how general this method is. In particular, it would be very useful to see an error analysis as a function of the augmentation used: what specific examples in the dataset benefit from using e.g. RandomScaling vs. RandomLocalDrop?

- L273: “We see that the embeddings produced using DepthContrast and our approach are both able to separate the three classes in feature space to a much stronger degree than the BYOL baseline,...” The t-SNE plots shown in Figure 3 aren’t sufficient to support this claim. The separability exhibited in t-SNE is well-known to depend on details of the hyperparameters in unpredictable and unreliable ways (https://distill.pub/2016/misread-tsne/). I suggest reporting quantitative numbers on clustering or removing this claim. If using t-SNE plots to compare methods, it is better practice to sweep best hypers individually for each plot to avoid confirmation bias.

- L137: “pre-text” -> “pretext”

- The use of “SK” as an abbreviation for SemanticKITTI is a bit distracting and makes the paper harder to read. It makes complete sense to use this when space is severely constrained (e.g.  Table 2), but please write out SemanticKITTI in the main text if at all possible.

- Table 1: mAP@0.25 -> AP25 to match the text (L219)

limitations: n/a
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed an unsupervised pertaining technique to improve representation learning of point clouds that can be applied to popular models for various downstream tasks.
The key idea of this paper, compared to the baseline method (DepthContrast [67]), is to learn the features from local patch level.
strengths_and_weaknesses: **Strengths**
- the main idea, implemented through two different losses, i.e. local volume PointInfoNCE and global volume ClusterNCE, makes sense to me, since patch-based approach allows the feature encoding to learn from repetitive structures in large-scale point cloud scenes.
- the method is evaluated on different types of data, including outdoor dataset, which is not done in [67].
- the pretrained model can be valuable to the community.

**Weakness**
- The method seems to work well. But I have some concerns about the principles of this approach. 
  - As mentioned by the authors, the information in outdoor lidar scans is very sparse, "important foreground often comprise few points". Local crops may contain zero foreground. In this case, $f^1$ does not capture any unique information that can be sufficiently associated to its corresponding global crop. 
  - The pretraining takes place on local crop level and as suggested by Figure 4 the crop size matters. What happens when at test-time the scale of the input point cloud changes? In this case the receptive field of the network (e.g. the spatial span of KNN in PointNet++) also changes. How can the proposed method address such scale variation?
- The paper is quite difficult to read for anyone not up-to-date with unsupervised learning. I find Section 3.3 particularly difficult to understand. IMHO, some basic concepts of unsupervised feature clustering should be given to make the paper complete.
- Similarly, at several places the authors mentioned the advantage of local feature learning in terms of computational cost, e.g. L127-128, L141 and L146. However, from the paper itself, it's not obvious what the baseline is, why it won't scale well with the large number of occupied volumes. The authors should briefly summarise what would be the computational cost for standard practice.
- The comparison with [67] was not very clear to me. In L227-229, the author mentioned they had to reproduce [67] using a different training set, which seems to result in a slightly worse result for [67] compared to the what's originally reported (Table 2 in [67]). The reason for this isn't very clearly conveyed.
questions: - PointNet++ gradually downsamples the point cloud by grouping local neighbourhood together. How do you get per-point feature?
- L151, should the reference be [3] instead?

limitations: I didn't find discussions about limitations.
review_title: null

Meta-review: metareview: All three reviewers recommend acceptance after rebuttal, although they were not very confident. I also read the paper and agree it is well written, has good results and addresses a gap in the large-scale point cloud learning literature. However I also see that the paper selectively builds upon ideas from other domains, e.g. vision [19,3] while ignoring closest related work that also does self-supervised learning in images from large scenes, using also local or local+global contrastive learning. Papers that are highly related include:

- Contrastive learning of global and local features for medical image segmentation with limited annotations. Chaitanya et al.
- Efficient Visual Pretraining with Contrastive Detection, Henaff et al.
- Point-Level Region Contrast for Object Detection Pre-Training. Bai et al.
- SegContrast: 3D Point Cloud Feature Representation Learning Through Self-Supervised Segment Discrimination. Nunes et al.

I strongly suggest explaining the differences to these papers. In an ideal world the first one should be a baseline to really show if there's something about the fit of the proposed method and point cloud data.
confidence: Less certain
Decision: Accept