Title: Distribution Guidance Network for Weakly Supervised Point Cloud Semantic Segmentation
Venue: NeurIPS 2024
Accepted: yes
Keywords: distribution guidance

weak supervision

point cloud

semantic segmentation
Abstract: Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from inadequate supervision signals. In response to this challenge, we introduce a novel perspective that imparts auxiliary constraints by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we analyze the superiority of the mixture of von Mises-Fisher distributions (moVMF) among several common distribution candidates. Accordingly, we develop a Distribution Guidance Network (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined latent space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings.

Distribution Guidance Network for Weakly Supervised Point Cloud Semantic Segmentation

Abstract

Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from inadequate supervision signals. In response to this challenge, we introduce a novel perspective that imparts auxiliary constraints by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we analyze the superiority of the mixture of von Mises-Fisher distributions (moVMF) among several common distribution candidates. Accordingly, we develop a D istribution G uidance Net work (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined latent space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings.

1 Introduction

As a fundamental task in 3D scene understanding, point cloud semantic segmentation [42, 32, 29] is widely entrenched in 3D applications, such as 3D reconstruction [15, 33], autonomous driving [20], and embodied intelligence [14, 50]. Despite significant accomplishments in tackling the disorder and disorganization, point cloud semantic segmentation remains annotation-intensive, hindering its expansion in big datasets and large models. For this reason, the academic community explores achieving point cloud semantic segmentation in a weakly supervised manner. However, due to the lack of supervision signals, learning point cloud segmentation on sparse annotations is nontrivial.

In recent years, considerable effort has been made to pursue additional constraints in weak supervision. As shown in Fig. 1, representative work can be broadly categorized into several paradigms: 1) Contrastive Learning / Perturbation Consistency imposes contrastive loss or consistency constraint between network embeddings of original and perturbed point clouds, respectively. 2) Self-training progressively enhances segmentation quality by treating reliable predictions as pseudo-labels, with

Backbone

Embedding

Backbone

Embedding

Perturbation

Backbone

Embedding

Sparse Label

Similarity Map

Backbone

Embedding

Sparse Label

Clustering Result

Normalization

Previous Stage Embedding

(a) Contrastive Learning / Perturbation Consistency

(b) Self-training

(d) DGNet (Ours)

W

oo

口

↑

A

...

O

C

口

>

业

业

....

非

oo

>

oo

口

....

given sparse annotations as initial labels. 3) Similarity Metric transfers supervision signals from annotated points to unlabeled regions via leveraging the low-level features or the embedding similarity. Nevertheless, most constraints stem from heuristic assumptions and ignore the inherent distribution of network embedding, resulting in ambiguous interpretations of point-level predictions. In contrast to existing paradigms, in this paper, we re-examine two fundamental issues: How to characterize the semantic segmentation feature space for weak supervision and how to intensify this intrinsic distribution under weakly supervised learning?

For the first issue about oughtness , we expect to provide a mathematically describable distribution for weakly supervised features. Consequently, we attempt to precisely describe the feature space in terms of two dimensions: distance metric and distribution modeling. In distance metric, we compare the Euclidean norm and cosine similarity between representations, and in distribution modeling, the category prototype model and the mixture model are considered. Among our candidate combinations, the mixture of von Mises-Fisher distributions (moVMF) with cosine similarity is finalized, due to its powerful fitting capability to segment head and insensitivity to the Curse of Dimensionality [44]. We believe that a superior weakly supervised feature space should adhere to this distribution.

For the second issue about practice , we dynamically align the embedding distribution in the hidden space to moVMF during weakly supervised learning. Accordingly, we propose a D istribution G uidance Net work (DGNet), comprising a weakly supervised learning branch and a distribution alignment branch. Specifically, the weakly supervised learning branch learns semantic embeddings under sparse annotations, while the distribution alignment branch constrains the distribution of the network embeddings. Via a Nested Expectation-Maximum Algorithm, the semantic features are dynamically refined. Therefore, restricting and fitting is a mutually reinforcing, iterative optimization process. To curtail the pattern of feature distribution, we derive the vMF loss based on the maximum likelihood estimation and the discriminative loss inspired by metric learning [22]. For joint optimization, consistency loss is imposed between the segmentation predictions and the posterior probabilities. During the inference phase, only the weakly supervised learning branch is activated to maintain inference consistency with fully supervised learning.

We validate DGNet on three prevailing point cloud datasets, i.e. , S3DIS [1], ScanNetV2 [11], and SemanticKITTI [5]. After the constraints of feature distribution, DGNet provides significant performance improvements over multiple baselines. Across various label rates, our method achieves state-of-the-art weakly supervised semantic segmentation performance. Extensive ablation studies also confirm the effectiveness of each loss term we proposed. In addition, posterior probabilities under the moVMF provide a plausible interpretation for predictions on unlabeled points.

2 Related Work

Weakly Supervised Point Cloud Semantic Segmentation. Weakly supervised point cloud semantic segmentation methods aim to provide reliable additional supervision with sparse annotations. Four paradigms have been successively proposed in recent years, i.e. , perturbation consistency, contrastive learning, self-training, and similarity metric. Perturbation consistency methods are based on the assumption of perturbation invariance of network features, imposing diverse perturbations (such affine transforms with point jitter [59, 54], downsampling [56], masking [35], etc.) to construct pairs of point clouds. Several methods [31, 34] introduce contrastive learning in weak supervision to encourage the discriminability of hidden layer features. Additionally, pre-training methods [55, 17] with contrastive learning similarly demonstrate the ability to bias induction in the face of downstream semantic segmentation tasks with sparse annotations. Self-training methods generate reliable dynamic pseudo-labels based on previous stage predictions for subsequent training stages. Via CAMs [62], MPRM [51] and J2D3D [25] dynamically generate point-wise pseudo-labels from subcloud-level annotations and image-level annotations, respectively. Recently, REAL [24] integrate SAM [23] to self-training. Similarity metric methods measure the similarity between labeled and unlabeled points to propagate supervision information, in which the similarity is elaborated on low-level features [49, 52], network embedding [36, 18, 40] or category prototypes [58, 46]. Most similar to our method is the similarity metric strategy. However, the distinction is that DGNet focuses on describing network embeddings holistically, rather than constructing pair relationships between features.

Feature Distribution Constraints. The constraints of feature distribution are always presented in the form of feature clustering. DeepClustering [6], which integrates clustering and unsupervised feature learning, utilizes the clustering results as pseudo-labels to extract visual features dynamically. Following this groundbreaking work, a series of subsequent studies [7, 2, 30] apply feature clustering in unsupervised learning to obtain discriminative visual features for pretraining. For point cloud semantic segmentation, PointDC [10] delineates semantic objects by aligning the features on the same super-voxel in an unsupervised manner. Feng et al . [12] imposes a clustering-based representation learning to enhance the discrimination of embeddings under full supervision. In contrast, our DGNet is oriented towards weakly supervised semantic segmentation by restricting the feature distribution.

Mixture of von Mises-Fisher Distributions. The moVMF [3] describes the embeddings of multiple categories on the unit hypersphere in feature space, where the parameters are jointly optimized with the clustering results by Expectation-Maximum algorithm [3, 4]. Some work attempts to combine neural networks with the moVMF in the deep learning era. For example, [16] view face verification as a direct application of clustering, introducing a vMF loss to align the distribution of face features. Segsort [21], on the other hand, utilizes the prior of the moVMF to over-segment images. DINOVMF [13] achieve a more stable pre-trained method by precisely describing DINO [8] as a moVMF. In this work, we discuss the superiority of moVMF in characterizing semantic embeddings and trust it as a priori to guide weakly supervised learning.

3 Methodology

3.1 Preliminaries

Task Definition. Without loss of generality, a point cloud for weakly supervised learning is denoted as { ( X l , Y ) , ( X u , ∅ ) } = { ( x 1 , y 1 ) , · · · , ( x m , y m ) , x m +1 , · · · , x n } , where X l and X u are the point sets with and without annotations, respectively. Y is the corresponding annotations on X l , in which y i ∈ C and C is the set of category indices. n and m are the point numbers of the point cloud and labeled set, respectively. Fed into the segment head, the network embedding f i is projected into the category probability vector p i . The partial cross-entropy loss is employed in conventional weakly supervised semantic segmentation:

where p y i i represents the probability of y i -th category in p i .

von Mises-Fisher Distribution (vMF). The vMF has demonstrated strong data fitting and generalization capabilities in the fields of self-supervised learning [9, 13], classification [44], variational inference [47], and online continual learning [38]. This distribution describes the distribution of normalized embedding v i = norm( f i ) on the unit hypersphere, with the probability density function:

where u represents the mean vector of vMF and κ ≥ 0 is a concentration parameter that controls the probability concentration around µ . C d ( κ ) is the normalization constant.

Mixture of vMF (moVMF). Similar to other mixture models, moVMF treats vMF as a subdistribution to describe the overall distribution of multiple categories. Over the entire set of categories C , the probability density function of moVMF is formulated as:

where Θ = { α c , κ c , u c | c ∈ C } is the parameters of moVMF. α c denotes the proportion of the von Mises-Fisher distribution for the c -th category and ∑ α c = 1 .

3.2 Feature Space Description

We intend to provide additional supervision signals for weakly supervised learning by portraying and enhancing its inherent distribution. Specifically, We explore it in two dimensions, i.e. , the distance metric and the distribution modeling:

Distance metric. Distance metric influences the similarity relationship between features. We consider the two most commonly used distance measures in clustering, i.e. , Euclidean norm and cosine similarity. For given vectors u and v , the Euclidean norm is defined as ∥ u -v ∥ 2 and the cosine similarity is defined as u ⊤ v ∥ u ∥ 2 ∥ v ∥ 2 . Cosine similarity can be viewed as the inner product of normalized u and v .

Distribution modeling. Distribution modeling determines the clustering results of features. A straightforward model is the Category Prototype [45]. In this model, clusters are assigned by comparing the distance between the features and each category prototype. In addition, we incorporate mixture models into the comparison. Depending on the distance measure, we categorize it into the Gaussian Mixture Model (GMM) with Euclidean norm and the mixture of von Mises-Fisher distributions (moVMF) with cosine similarity, respectively.

Describing the feature space of a neural network remains an open problem. Various factors influence the feature space, including network architecture, training data, parameter configurations, and the optimization (loss) function. Given this intractability, deriving a universally optimal mathematical description is impractical. Therefore, we discuss the merits and demerits of these candidate distributions from the following perspectives:

̸

Segment head. To facilitate the analysis, we simplify the structure of the segment head as SegHead ( f ) = argmax ( softmax ( wf ⊤ )) , where f is the semantic feature extracted by the decoder and w is the parameter of the output layer. Consider a group of feature vectors { k f | k ≥ 0 & f = 0 } . For any two feature vectors k 1 f and k 2 f within this group, the segmentation predictions are identical, i.e. , SegHead ( k 1 f ) = argmax ( softmax ( k 1 wf ⊤ )) = argmax ( softmax ( k 2 wf ⊤ )) = SegHead ( k 2 f ) . If the general case of using an activation function is taken into account, it does not change the result after argmax since the activation function is usually monotonically nondecreasing. Therefore, the segment head is a radial classifier with a more pronounced classification performance on the angles, so cosine similarity describes the feature space better than the Euclidean norm.

Curse of dimensionality. Another advantage of cosine similarity can be explained in terms of the Curse of Dimensionality. Most high-dimensional features are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors. Besides, the Euclidean norm is sensitive to scale while cosine similarity is not affected by the length of the vectors.

Weakly Supervised Learning

Distribution Alignment

Normalized Embedding

Embedding

E step

M step

Soft-moVMF

Segment Head

ℒtCE

ℒCON

ℒvMF

ℒDIS

Prediction

Sparse Label

Embedding on

Unlabeled Point

Embedding on

Labeled Point

Semantic Embedding

Posterior Probability

Category Probability

Feature Space

Unit

Hypersphere

Fitting ability. Despite its computational simplicity, the Category Prototype Model clusters the features by comparing the distance between features and category prototypes. This means that the Category Prototype Model ignores the distribution within categories and the variability between categories. In contrast, the Mixture Model possesses intra-category fitting and inter-category perception capabilities.

Based on the above analysis and experimental validation in Sec. 4.3, we characterize the feature space as moVMF and propose the Distribution Guidance Network to enhance this distribution.

3.3 Distribution Guidance Network

To enhance the intrinsic distributions discussed in Section 3.2, we propose a D istribution G uidance Net work (DGNet). The structure of DGNet is shown in Fig. 2, which comprises the weakly supervised learning branch and the distribution alignment branch.

3.3.1 Weakly Supervised Learning Branch

Sparse annotations pose two challenges for the point cloud semantic segmentation. The first is underfitting the entire dataset, as the supervision signals are insufficient for complex structured point clouds. To address underfitting, we introduce additional signals by reinforcing the inherent distribution of the feature space. The second challenge is overfitting within the labeled set X l , as the model capacity is more than adequate to fit the labeled points. To mitigate overfitting, we replace the conventional partial cross-entropy loss Eq. 1 with the truncated cross-entropy loss [60] in the weakly supervised learning branch, which is defined as:

where β ∈ [0 , 1] represents the threshold for truncating the cross-entropy loss. With L tCE , the gradient of the cross-entropy loss is curtailed when the predicted class probability for an annotated point exceeds β . Over-optimization for that point is halted, thereby preventing overfitting.

The weakly supervised learning branch also provides robust initialization for the distribution alignment branch by utilizing the average feature vector of the labeled points. According to the Central Limit Theorem [27], the difference between the initialization vector from the weakly supervised

learning branch and the theoretical optimal average vector conforms to a Gaussian distribution with a mean of 0. The initialized mean vector in DGNet has a high probability of appearing in the vicinity of the optimal solution, which facilitates the clustering algorithm in achieving rapid and stable convergence.

3.3.2 Distribution Alignment Branch

The feature space descriptor moVMF, investigated in Sec. 3.2, is employed to regulate the feature space under weakly supervised learning. Initially, the network embeddings are normalized and projected onto the unit hyperspherical surface of the feature space, i.e. , v i = norm ( f i ) . Following this, the optimization objective function is defined utilizing maximum likelihood estimation 1 :

where ϕ , V = { v i } , Z = { z i } and Θ = { α c , κ, u c } denote the learnable network parameters, the normalized network embeddings, the corresponding clustering results and the parameters of moVMF, respectively. To avoid the long-tail problem and simplify computations, the concentration parameter κ is fixed as a constant in our implementation. Since the clustering initialization is category-aware, the clustering results z i ∈ C are with category labels. While the primary optimization goal is the learning of network parameters ϕ , we dynamically resolve Z and Θ to furnish a more precise feature space description. Consequently, we develop a Nested Expectation-Maximum Algorithm to manage the challenge associated with the three optimization variables delineated in Eq. 5.

E Step (Optimize Θ and Z ): Regarding network embeddings as input conditions, we integrate the soft-moVMF EM algorithm [3] into the network to alternately optimize Θ and Z . To enhance the stability and computational efficiency of the algorithm, we utilize the average features of labeled points from the weakly supervised branch to initialize u . The posterior probability set Q = { q i } serves as the soft assignment for the clustering results Z , where q i is defined as

Compared to other algorithms in [3], the soft-moVMF algorithm updates Θ by weighting all features according to their posterior probabilities, considering inter-cluster similarities, thereby achieving more accurate parameter updates. The posterior probability q i ∈ [0 , 1] | C |× 1 is employed not only as a weighting factor for updates but also in the calculation of the loss function for joint optimization. Furthermore, Q provides a probabilistic explanation for the predictions during the inference phase.

The complexity of the soft-voVMF is O ( tn | C | ) , where t is the iteration number, n is the point number of the point cloud, and | C | is the number of semantic categories. Since t , n , and | C | are all set to constant values during network training, the extra computation introduced by the distribution alignment branch is trivial.

M Step (Optimize ϕ ): With the converged parameters Θ and Z fixed, we optimize ϕ by the backpropagation process. Consistent with the philosophy of the soft-moVMF, we incorporate the posterior probability Q into Eq. 5, and reformulate it into the loss function as follows:

Additionally, acknowledging the significance of distinct decision boundaries within the mixture model, we incorporate a discriminative loss derived from metric learning [22] which is defined as:

̸

The interpretation of Bayesian posterior probabilities for predictions based on the moVMF is an attractive property of DGNet. Fig. 3 visualizes the posterior probabilities for some categories. Taking the floor as an example, according to the Bayesian theorem, those points with relatively high posterior probabilities are more likely to be floor , which explains the prediction results.

Prediction

Floor

Table

Chair

DGNet

Posterior Probability

Distribution Alignment

Weakly Supervised

Learning

Probabilistic

Explanation

3.3.3 Loss Function

In addition to the previously mentioned initialization, we introduce a consistency loss to fortify the exchange of information between the two branches. This consistency loss is imposed on the class probability map p from the weakly supervised learning branch and the posterior probability q from the distribution alignment branch, in the form of cross-entropy:

If regard the posterior probability q as pseudo-labels, the consistency loss is proved to diminish prediction uncertainty and alleviate distribution discrepancies in [48].

Without laborious adjustments to the weights 2 , the overall loss function is defined as follows:

4 Experimental Analysis

4.1 Experiment Settings

Datasets. S3DIS [1] encompasses six indoor areas, constituting a total of 271 rooms with 13 categories. Area 5 within S3DIS serves as the validation set, while the remaining areas are allocated for network training. ScanNetV2 [11] offers a substantial collection of 1,513 scanned scenes originating from 707 indoor environments with 21 indoor categories. Adhering to the official ScanNetV2 partition, we utilize 1,201 scenes for training and 312 scenes for validation. SemanticKITTI [5] with 19 classes is also considered. Point cloud sequences 00 to 10 are used in training, with sequence 08 as the validation set. To simulate sparse annotations, we randomly discard the dense annotations proportionally.

Implementation details. ResGCN-28 in DeepGCN [29] and PointNeXt-l [43] are reimplemented as the segment backbones with OpenPoints library [43]. We discard the last activation layer of the decoder to extract orientation-completed feature space. We maintain a memory bank [53] to store class prototypes across the entire dataset. In cases where class annotations are absent from the scene, the class prototypes from the memory bank are employed as supplementary initialization. We employ the LaDS [39] to maintain a higher rate of training supervision after point cloud sampling. For truncated cross-entropy loss, β = 0 . 8 . The concentration constant κ = 10 and the iteration number t = 10 . The distribution alignment branch is not activated in the first 50 epochs to stabilize the feature learning. In our implementation, the DGNet is trained with one NVIDIA V100 GPU on S3DIS, eight NVIDIA TESLA T4 GPUs on ScanNetV2, and one NVIDIA V100 GPU on SemanticKITTI. In the inference stage, only the weakly supervised learning branch is activated to produce predictions.

4.2 Comparative Analysis

Results on S3DIS. We detail the segmentation performance at 0.1% and 0.01% label rates on S3DIS Area 5. DGNet boosts performance for each baseline, which is evenly distributed across categories.

The lower the label rate, the more distinct the enhancement brought by DGNet, which suggests that the guidance on feature distribution is more valuable with extremely sparse annotations. Specifically, DGNet achieves more than 97% performance of fullysupervision with only 0.1% points labeled. The 0.02% label rate denotes a sparse labeling form of "one-thingone-click". DGNet outperforms these methods without introducing super-voxel information. In addition, Fig. 4 visualizes a qualitative comparison. DGNet provides a holistic enhancement to the baseline. Although the photos on the wall are misclassified, DGNet captures consistent objects more accurately than the baseline.

Results on ScanNetV2. Compared to S3DIS, ScanNetV2 involves diverse categories and versatile scenes. Therefore, following the super-voxel setting in OTOC [36], we report the segmentation performances with 20 labeled points per scene and 1% points labeled. The cross-entropy loss term generates relatively sufficient supervised information to train the network due to introducing pseudo-labeling, resulting in a less pronounced DGNet improvement than S3DIS. However, DGNet is still slightly superior to the latest SOTA methods.

Results on SemanticKITTI. DGNet performs excellently on indoor datasets and demonstrates strong weakly supervised learning efficiency on outdoor SemanticKITTI. For a fair comparison, we replace the DGNet backbone with RandLA-Net [19]. DGNet outperforms SQN [18] with 1.0% and 2.1% mIoU on 0.1% and 0.01% label rates, respectively, demonstrating the necessity of supervision on feature space.

Ground Truth

PointNeXt

DGNet (PointNeXt)

3

4.3 Ablations and Analysis

All ablation studies are performed on S3DIS with PointNeXt-l as baseline.

Distribution comparison. We impose a comparison experiment in the distribution alignment branch of DGNet for distribution selection. The relevant experimental results are reported in Tab. 4. For category prototype models, we discard the vMF loss L vMF and consistency loss L CON due to the lack of corresponding forms. For the mixture model with Euclidean Norm (GMM), we replace the maximum likelihood estimation in GMM form with the L vMF in DGNet. In terms of distance metrics, cosine similarity trumps Euclidean norm. In terms of distribution modeling, mixture models have a significant performance advantage over the category prototype models. Integrating these two aspects, the stronger fitting ability of moVMF leads to more accurate and effective supervised signals for weakly supervised learning in DGNet.

Ablation study for loss terms. Tab. 5 demonstrates the validity of each loss term in DGNet. Compared with partial cross-entropy loss, the truncated cross-entropy loss improves segmentation performance due to its avoidance of overfitting. Performance improvements are obtained by imposing L vMF with soft assignment form, L DIS and L CON individually, and optimal performance is achieved by using these loss terms simultaneously. In contrast to the soft assignment, the hard assignment does not take into account the inter-cluster similarity and is mismatched with the soft-moVMF algorithm. Therefore, L vMF with hard assignment form in hard-moVMF algorithm and KNN-moVMF algorithm undermines the segmentation efficiency.

Ablation study for Nested EM Algorithm. We ablate the proposed Nested Expectation-Maximum Algorithm in two respects. First, we optimize certain parameters on moVMF and fix other parameters with initialized values. The first, second, third, and last rows in Tab. 6 reveal that individually optimizing parts of the parameters impairs the segmentation performance. Secondly, we ablate how the parameters of moVMF are updated. Compared with kNN-moVMF (fourth raw) and hard-moVMF (fifth raw) in [3], the soft assignment strategy delivers 2.6% and 2.2% mIoU improvements, respectively. This shows that the optimization of moVMF parameters benefits from the soft assignment strategy.

Hyperparameter selection. In Tab. 7, we search the parameter space for suitable κ , t , and β . We observe that (a) the segmentation performance shows an increasing and then decreasing trend as the

concentration constant κ increases. Our analysis suggests that too small κ leads to a dispersion of features within the class, which can be easily confused with other classes. And too large κ forces overconcentration of features within the class and overfits the network. (b) As the iteration number t increases, the segmentation performance gradually rises and then stabilizes. We believe that the soft-moVMF algorithm gradually converges as t increases, and increasing t after convergence will no longer bring further gains to the network. (c) As the truncated threshold β decreases, the segmentation performance shows a tendency to first increase and then decrease. The conventional cross-entropy loss function is the truncated cross-entropy loss function with β = 1 . When β decreases, the overfitting on sparse annotations is alleviated, but when β is too small, it weakens the supervised signal on sparse labeling leading to performance degradation.

5 Limitations and Future Work

Despite the promising performance achieved by DGNet, exploring the distribution of embeddings is preliminary. Feng et al . [12] proposes a more sophisticated distribution to restrict feature learning with full supervision. However, such refinement restrictions will lead to overfitting under sparse annotations. Therefore, how to prevent weakly-supervised learning overfitting with enhanced feature description is a promising research topic.

6 Conclusion

In this paper, we propose a novel perspective by regulating the feature space for weakly supervised point cloud semantic segmentation and develop a distribution guidance network to verify the superiority of this perspective. Based on the investigation of the distribution of semantic embeddings, we choose moVMF to describe the intrinsic distribution. In DGNet, we alleviate the underfitting across the entire dataset and overfitting within the labeled points. Extensive experimental results demonstrate that DGNet rivals or even surpasses the recent SOTA methods on S3DIS, ScanNetV2, and SemanticKITTI. Moreover, DGNet demonstrates the interpretability of network predictions and scalability to various label rates. We expect our work to inspire the point cloud community to strengthen the inherent properties of weakly supervised learning.

References

Iro Armeni, Ozan Sener, Amir R Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1534-1543, 2016.

Yuki Markus Asano, Christian Rupprecht, and Andrea Vedaldi. Self-labelling via simultaneous clustering and representation learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net, 2020.

Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research , 6(9), 2005.

Florian Barbaro and Fabrice Rossi. Sparse mixture of von mises-fisher distribution. In ESANN , 2021.

Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9297-9307, 2019.

Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. In Proceedings of the European conference on computer vision (ECCV) , pages 132-149, 2018.

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems , 33:9912-9924, 2020.

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 9650-9660, 2021.

Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 15750-15758, 2021.

Zisheng Chen, Hongbin Xu, Weitao Chen, Zhipeng Zhou, Haihong Xiao, Baigui Sun, Xuansong Xie, et al. Pointdc: Unsupervised semantic segmentation of 3d point clouds via cross-modal distillation and super-voxel clustering. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14290-14299, 2023.

Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 5828-5839, 2017.

Tuo Feng, Wenguan Wang, Xiaohan Wang, Yi Yang, and Qinghua Zheng. Clustering based point cloud representation learning for 3d analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8283-8294, 2023.

Hariprasath Govindarajan, Per Sidén, Jacob Roll, and Fredrik Lindsten. DINO as a von mises-fisher mixture model. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023 . OpenReview.net, 2023.

Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications , 12(1):5721, 2021.

Christian Hane, Christopher Zach, Andrea Cohen, Roland Angst, and Marc Pollefeys. Joint 3d scene reconstruction and class segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 97-104, 2013.

Md Abul Hasnat, Julien Bohné, Jonathan Milgram, Stéphane Gentric, and Liming Chen. von mises-fisher mixture model-based deep learning: Application to face verification. arXiv preprint arXiv:1706.04264 , 2017.

Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-efficient 3d scene understanding with contrastive scene contexts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15587-15597, 2021.

Qingyong Hu, Bo Yang, Guangchi Fang, Yulan Guo, Aleš Leonardis, Niki Trigoni, and Andrew Markham. Sqn: Weakly-supervised semantic segmentation of large-scale 3d point clouds. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXVII , pages 600-619. Springer, 2022.

Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11108-11117, 2020.

Yihan Hu, Jiazhi Yang, Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tianwei Lin, Wenhai Wang, et al. Planning-oriented autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 17853-17862, 2023.

Jyh-Jing Hwang, Stella X Yu, Jianbo Shi, Maxwell D Collins, Tien-Ju Yang, Xiao Zhang, and Liang-Chieh Chen. Segsort: Segmentation by discriminative sorting of segments. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7334-7344, 2019.

Mahmut Kaya and Hasan ¸ Sakir Bilge. Deep metric learning: A survey. Symmetry , 11(9):1066, 2019.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 4015-4026, 2023.

Hyeokjun Kweon, Jihun Kim, and Kuk-Jin Yoon. Weakly supervised point cloud semantic segmentation via artificial oracle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3721-3731, 2024.

Hyeokjun Kweon and Kuk-Jin Yoon. Joint learning of 2d-3d weakly supervised semantic segmentation. Advances in Neural Information Processing Systems , 35:30499-30511, 2022.

Yuxiang Lan, Yachao Zhang, Yanyun Qu, Cong Wang, Chengyang Li, Jia Cai, Yuan Xie, and Zongze Wu. Weakly supervised 3d segmentation via receptive-driven pseudo label consistency and structural consistency. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 1222-1230, 2023.

Pierre Simon Laplace. Théorie analytique des probabilités . Courcier, 1820.

Min Seok Lee, Seok Woo Yang, and Sung Won Han. Gaia: Graphical information gain based attention network for weakly supervised point cloud semantic segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 582-591, 2023.

Guohao Li, Matthias Müller, Guocheng Qian, Itzel C Delgadillo, Abdulellah Abualshour, Ali Thabet, and Bernard Ghanem. Deepgcns: Making gcns go as deep as cnns. IEEE transactions on pattern analysis and machine intelligence , 45(6):6923-6939, 2021.

Junnan Li, Pan Zhou, Caiming Xiong, and Steven C. H. Hoi. Prototypical contrastive learning of unsupervised representations. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 . OpenReview.net, 2021.

Mengtian Li, Yuan Xie, Yunhang Shen, Bo Ke, Ruizhi Qiao, Bo Ren, Shaohui Lin, and Lizhuang Ma. Hybridcr: Weakly-supervised 3d point cloud semantic segmentation via hybrid contrastive regularization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1493014939, 2022.

Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. Advances in neural information processing systems , 31, 2018.

Zhuangzi Li, Ge Li, Thomas H Li, Shan Liu, and Wei Gao. Semantic point cloud upsampling. IEEE Transactions on Multimedia , 25:3432-3442, 2022.

Kangcheng Liu, Yuzhi Zhao, Qiang Nie, Zhi Gao, and Ben M Chen. Weakly supervised 3d scene segmentation with region-level boundary awareness and instance discrimination. In European conference on computer vision , pages 37-55. Springer, 2022.

Lizhao Liu, Zhuangwei Zhuang, Shangxin Huang, Xunlong Xiao, Tianhang Xiang, Cen Chen, Jingdong Wang, and Mingkui Tan. Cpcm: Contextual point cloud modeling for weakly-supervised point cloud semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 18413-18422, 2023.

Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. One thing one click: A self-training approach for weakly supervised 3d semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1726-1736, 2021.

Pascal Mettes, Elise Van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in neural information processing systems , 32, 2019.

Nicolas Michel, Giovanni Chierchia, Romain Negrel, and Jean-François Bercher. Learning representations on the unit sphere: Investigating angular gaussian and von mises-fisher distributions for online continual learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 14350-14358, 2024.

Zhiyi Pan, Nan Zhang, Wei Gao, Shan Liu, and Ge Li. Point cloud semantic segmentation with sparse and inhomogeneous annotations. arXiv preprint arXiv:2312.06259 , 2023.

Zhiyi Pan, Nan Zhang, Wei Gao, Shan Liu, and Ge Li. Less is more: label recommendation for weakly supervised point cloud semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 38, pages 4397-4405, 2024.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems , 30, 2017.

Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. Advances in Neural Information Processing Systems , 35:23192-23204, 2022.

Tyler R Scott, Andrew C Gallagher, and Michael C Mozer. von mises-fisher loss: An exploration of embedding geometries for supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 10612-10622, 2021.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Yongyi Su, Xun Xu, and Kui Jia. Weakly supervised 3d point cloud segmentation via multi-prototype learning. IEEE Transactions on Circuits and Systems for Video Technology , 2023.

Jalil Taghia, Zhanyu Ma, and Arne Leijon. Bayesian estimation of the von-mises fisher mixture model with variational inference. IEEE transactions on pattern analysis and machine intelligence , 36(9):1701-1715, 2014.

Liyao Tang, Zhe Chen, Shanshan Zhao, Chaoyue Wang, and Dacheng Tao. All points matter: Entropyregularized distribution alignment for weakly-supervised 3d segmentation. Advances in Neural Information Processing Systems , 36, 2024.

An Tao, Yueqi Duan, Yi Wei, Jiwen Lu, and Jie Zhou. Seggroup: Seg-level supervision for 3d instance and semantic segmentation. IEEE Transactions on Image Processing , 31:4952-4965, 2022.

Weikang Wan, Haoran Geng, Yun Liu, Zikang Shan, Yaodong Yang, Li Yi, and He Wang. Unidexgrasp++: Improving dexterous grasping policy learning via geometry-aware curriculum and iterative generalistspecialist learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3891-3902, 2023.

Jiacheng Wei, Guosheng Lin, Kim-Hui Yap, Tzu-Yi Hung, and Lihua Xie. Multi-path region mining for weakly supervised 3d semantic segmentation on point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4384-4393, 2020.

Yushuang Wu, Zizheng Yan, Shengcai Cai, Guanbin Li, Xiaoguang Han, and Shuguang Cui. Pointmatch: A consistency training framework for weakly supervised semantic segmentation of 3d point clouds. Computers & Graphics , 116:427-436, 2023.

Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3733-3742, 2018.

Zhonghua Wu, Yicheng Wu, Guosheng Lin, Jianfei Cai, and Chen Qian. Dual adaptive transformations for weakly supervised point cloud segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI , pages 78-96. Springer, 2022.

Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsupervised pre-training for 3d point cloud understanding. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16 , pages 574-591. Springer, 2020.

Cheng-Kun Yang, Ji-Jia Wu, Kai-Syun Chen, Yung-Yu Chuang, and Yen-Yu Lin. An mil-derived transformer for weakly supervised point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 11830-11839, 2022.

Baochen Yao, Hui Xiao, Jiayan Zhuang, and Chengbin Peng. Weakly supervised learning for point cloud semantic segmentation with dual teacher. IEEE Robotics and Automation Letters , 2023.

Yachao Zhang, Zonghao Li, Yuan Xie, Yanyun Qu, Cuihua Li, and Tao Mei. Weakly supervised semantic segmentation for large-scale point cloud. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3421-3429, 2021.

Yachao Zhang, Yanyun Qu, Yuan Xie, Zonghao Li, Shanshan Zheng, and Cuihua Li. Perturbed selfdistillation: Weakly supervised large-scale point cloud semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15520-15528, 2021.

Zhilu Zhang and Mert Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. Advances in neural information processing systems , 31, 2018.

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision , pages 16259-16268, 2021.

Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2921-2929, 2016.

Appendix A Derivation of Optimization Objective Function

According to maximum likelihood estimation, the optimization objective function is defined as:

To avoid the long-tail problem and simplify computations, we set a constant value for κ on each class c . Therefore, the objective optimization function Eq. 11 can be further simplified as:

Algorithm 1: soft-moVMF Algorithm

Appendix B The soft-moVMF Algorithm

Initialization. Due to the sparsity of the annotations, some classes in the scene may lack any labeled points, thereby hindering proper initialization. Consequently, we maintain a memory bank [53] to store class prototypes across the entire dataset. Specifically, the mean embedding directions on labeled points are set as the initial vectors for categories with labeled points in the point cloud scene.

For the missing categories of this scene, we retrieve the category prototypes ρ from the memory bank as supplementary initialization. Consequently, the initial vector h c is formulated as:

Pseudo Code. As presented in Algorithm 1, we incorporate prior knowledge about the semantics during the optimization process based on soft-moVMF [3]. The weights α = 1 / | C | are initialized uniformly. Subsequently, based on the cosine similarity between features on each point and mean directions of each category, the prior probability f and the posterior probability q i are estimated. Finally, we determine the clustering result z i for each point by argmax( q i ) . After updating the mean directions u and weights α , the process is repeated until the clustering results converge.

Appendix C More Experimental Results

Impact of label rates. To demonstrate the capability of DGNet on extreme label rates, we compare the segmentation performance on sparse annotations over a larger range of rates. Tab. 8 reports the mIoU performance of the DGNet and baseline at 10%, 1%, 0.1%, 0.01%, and 0.001% label rates. It can be observed that at 100,000 times less sparse annotations, the baseline fails to learn accurate semantic embedding from it. At the same time, our DGNet still maintains acceptable segmentation performance since it can be conducted unsupervised.

Varying labeled points. Following SQN [18], we verified the sensitivity of DGNet (PointNeXt) to different labeled points at the same label rate. We repeated the experiment five times for each label setting, keeping the network and label rate unchanged and changing only the labeled points' locations. In Tab. 9, we observe a slight performance fluctuation within a reasonable range.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claim of this paper is using mathematically definable feature distributions to promote the learning of point cloud semantic segmentation under weak supervision.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitations of our work can be found in Sec. 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: The complete derivation of Eq. 5 is given in Appendix.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The relevant code and data will be open-sourced upon acceptance of the paper. Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The relevant code and data will be open-sourced upon acceptance of the paper. Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The main details are shown in Sec. 4.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Following other weak supervision methods, we do not provide error bars.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We only provide information on the computer resources for the main experiment.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We conduct in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: There is no societal impact of this work.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cite the original papers that produce the codes and datasets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
In this work, the authors use a selected feature space distribution as supplementary supervision signals, combining with cosine similarity as the distance metric, and implement a dual-branch weakly supervised point cloud semantic segmentation network. Experiments show that using feature space distribution as supervision information can improve the performance of existing point cloud semantic segmentation networks.

STRENGTHS:
This work proposes to use moVMF distribution to regulate the feature space for weakly supervised point cloud semantic segmentation. Experimental results show that this approach can improve the performances of backbone networks.

WEAKNESSES:
As the feature space distribution is selected in a fully supervised manner, it is questionable whether it is suitable for weakly supervised scenarios where labeled samples are extremely sparse. Directly comparing different distributions in the distribution alignment branch of DGNet is more convincing.

Eq. 1 in supplementary material, the last term should be log(a_zi) + k_zi * u^T_zi * vi, instead of log(a_zi) * k_zi * u^T_zi * vi.

In Eq. 9, why use the same weight for each loss term? Considering that these losses are defined from very different perspectives, their values can be of different orders of magnitude.

There are some minor flaws in the writing: some tables and figures are not referenced in the text, and some figures are not numbered in the order referenced.

QUESTIONS:
Please respond to my comments written above.

LIMITATIONS:
Evaluation on outdoor datasets is insufficient.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA weakly-supervised approach has been studied to relieve the annotation burden for point cloud semantic segmentation. Unlike conventional works that mainly use priors (such as similarity or augmentation-based regularization) to overcome the lack of information in weak labels, this paper introduces a novel approach directly aligning the features on the latent space of a neural network. Specifically, the proposed method employs a mixture of von Mises-Fisher distribution to model the feature distribution. This brings meaningful gains on multiple distinct architectures across various benchmarks.\n\nSTRENGTHS:\n1. The feature distribution is a core problem but has not yet been deeply addressed, at least in the field of weakly supervised point cloud semantic segmentation.\n2. The methodology and presentation for it are pretty good.\n3. The performance is impressive.\n\nWEAKNESSES:\nI think this paper has no major flaws in methodological aspects.\nTherefore, I would like to ask some questions for further discussion.\n\n1. Table 1 shows that a mixture of vMF distributions represents the distribution of normalized features better than the other choices. But why? Can you discuss the \"oughtness\" in more detail, at least at the level of the hypothesis? For example, why is using a mixture of vMFs better than that of Gaussians? \n\n2. Intuitively, using a hypersphere can represent each feature as an angle, resulting in a more concise distribution. However, if the number of categories increases, this kind of approach may not effectively use the latent space, as all the points on a line map to one single point on the hypersphere (due to normalization). I wonder if the proposed method faces difficulty when handling the dataset including more categories.\n\n3. It will be interesting if the authors check the feature distribution of the neural network trained by fully-supervised learning. Do the features still follow the mixture of vMFs well or not? Can the proposed method even enhance the performance in a fully-supervised setting?\n\n4. Related works.\nAs far as I know, MPRM does not use 2D images (Line 82). And it would be better if the authors could cite recent papers in CVPR 2024.\n\nQUESTIONS:\nSee the weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of weakly supervised point cloud semantic segmentation. The authors propose imparting supplementary supervision signals by regulating the feature space under weak supervision. The initial investigation identifies which distributions accurately characterize the feature space in fully supervised learning, subsequently leveraging this prior knowledge to guide the alignment of weakly supervised embeddings. The authors first investigate different feature space characterizations and find that a mixture of von Mises-Fisher (moVMF) distributions with cosine similarity best describes the fully supervised embedding space. Based on this finding, DGNet is proposed with two branches: a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined feature space. Experimental results demonstrate that DGNet achieves state-of-the-art performance on multiple datasets.\n\nSTRENGTHS:\n1.\tThe proposed framework presents a novel and intriguing approach, distinguishing itself from consistency training, self-training, and similarity metric methods.\n\n2.\tThe authors perform a comprehensive study to identify the most appropriate model for characterizing the feature space in fully supervised learning.\n\n\n3.\tDGNet is designed with two complementary branches: one dedicated to weakly supervised learning and another focused on distribution alignment. This dual-branch structure facilitates the effective integration of the distribution prior into the learning process.\n\n4.\tDGNet demonstrates its efficacy by achieving superior results across multiple datasets and various weakly supervised settings.\n\nWEAKNESSES:\n1. Computational Complexity: The paper omits a crucial discussion on the computational complexity of the distribution alignment branch, particularly regarding the EM-like algorithm. This is important, given that EM-based clustering methods traditionally tend to be computationally intensive.\n\n2. Ablation Study: The current ablation study would benefit from exploring a broader range of loss term combinations. For instance, examining the performance of the complete loss function without $L_{DIS}$ or without $L_{CON}$ would provide more comprehensive insights into the contribution of each component.\n\n3. Discussion and Analysis of moVMF: The paper lacks a deep analysis or discussion explaining why the mixture of von Mises-Fisher (moVMF) distribution outperforms other distributions. Additionally, it fails to provide formal guarantees on convergence or optimality. A more thorough discussion or justification for the choice of moVMF over alternative distributions would be better.\n\nQUESTIONS:\nSee weakness\n\nLIMITATIONS:\nA main limitation is the computational complexity of the training.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper outlines a study focusing on distance metric and distribution modeling, highlighting the effectiveness of combining mixture of von Mises-Fisher distributions (moVMF) with cosine similarity. A Distribution Guidance Network (DGNet) is introduced, featuring two main branches: weakly supervised learning and distribution alignment. The network utilizes reliable clustering initialization from the weakly supervised branch to iteratively update moVMF parameters and network parameters, ensuring alignment with moVMF-defined feature spaces.\n\nSTRENGTHS:\nOriginality: DGNet is a distribution guidance network that combines a weakly supervised learning branch and a distribution alignment branch. The weakly supervised learning branch makes use of reliable cluster initialization for learning. This means that the network is able to perform preliminary learning and classification of data through clustering methods without detailed labeling. This approach helps to reduce the dependence on large amounts of labeled data and improves the robustness and scalability of the model. The distribution alignment branch alternately updates the parameters of the moVMF (von Mises-Fisher distribution) and the parameters of the network to ensure that the network parameters are aligned with the feature space defined by the moVMF. This method helps to guide the network to learn the structure and characteristics of data distribution, so as to improve the generalization ability and adaptability of the model.\n\nQuality: DGNet explains the prediction probability from the perspective of Bayes' theorem and provides the confidence of each prediction, which proves the rationality of DGNet selection distribution and network design. In addition, extensive experiments confirm the soundness and efficacy of the chosen distribution and network design.\n\nWEAKNESSES:\n1. The section of 4.2 and 4.3 can be written in more detail. In particular, the explanation of changes in model performance in ablation experiments should be added. Specifically, a thorough explanation of the changes in model performance observed during ablation experiments should be incorporated. Ablation experiments involve systematically removing components or features from the model to analyze their impact on performance. By detailing how each modification affects the overall performance metrics, readers gain insights into the critical elements contributing to the model's effectiveness.\n2. In Abstract, there is no a clear problem to address from existing methods of weakly supervised learning for 3D point clouds. Similarly, in Introduction, it’s also not clear about the motivation of this work and existing problems of this topic.\n3. There is no main contributions about this work in Introduction.\n4. How to balance weights of each loss function in Equation (9)? \n5. There is no any qualitative comparison between this work with others.\n6. Some quantitative results from this work are also incremental improvements compared to others within one percentage point.\n7. Compared with previous work, the performance improvement of DGNet model may not be substantial enough to justify its superiority over existing methods, especially on the ScanNet and SemanticKITTI datasets.\n8. In the section of 4.3, why does the performance of the model degrade when optimizing some parameters? What is the impact of parameter optimization on resource consumption?\n\nQUESTIONS:\nPlease answer the questions in Weaknesses.\n\nLIMITATIONS:\nThe authors have thoroughly discussed the limitations of DGNet, and there are no potential negative societal impacts associated with their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}