Distribution Guidance Network for Weakly Supervised Point Cloud Semantic Segmentation

Authors: zhiyi  pan
wei  gao
shan  liu
ge  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 09:47:50

Abstract: Despite alleviating the dependence on dense annotations inherent to fully supervised methods, weakly supervised point cloud semantic segmentation suffers from inadequate supervision signals. In response to this challenge, we introduce a novel perspective that imparts auxiliary constraints by regulating the feature space under weak supervision. Our initial investigation identifies which distributions accurately characterize the feature space, subsequently leveraging this priori to guide the alignment of the weakly supervised embeddings. Specifically, we analyze the superiority of the mixture of von Mises-Fisher distributions (moVMF) among several common distribution candidates. Accordingly, we develop a Distribution Guidance Network (DGNet), which comprises a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined latent space. Extensive experiments validate the rationality and effectiveness of our distribution choice and network design. Consequently, DGNet achieves state-of-the-art performance under multiple datasets and various weakly supervised settings.

Main paper: 
 introduction: 
 As a fundamental task in 3D scene understanding, point cloud semantic segmentation [42, 32, 29] is widely entrenched in 3D applications, such as 3D reconstruction [15, 33] , autonomous driving [20] , and embodied intelligence [14, 50] . Despite significant accomplishments in tackling the disorder and disorganization, point cloud semantic segmentation remains annotation-intensive, hindering its expansion in big datasets and large models. For this reason, the academic community explores achieving point cloud semantic segmentation in a weakly supervised manner. However, due to the lack of supervision signals, learning point cloud segmentation on sparse annotations is nontrivial.In recent years, considerable effort has been made to pursue additional constraints in weak supervision. As shown in Fig. 1 , representative work can be broadly categorized into several paradigms: 1) Contrastive Learning / Perturbation Consistency imposes contrastive loss or consistency constraint between network embeddings of original and perturbed point clouds, respectively. 2) Self-training progressively enhances segmentation quality by treating reliable predictions as pseudo-labels, with given sparse annotations as initial labels. 3) Similarity Metric transfers supervision signals from annotated points to unlabeled regions via leveraging the low-level features or the embedding similarity. Nevertheless, most constraints stem from heuristic assumptions and ignore the inherent distribution of network embedding, resulting in ambiguous interpretations of point-level predictions. In contrast to existing paradigms, in this paper, we re-examine two fundamental issues: How to characterize the semantic segmentation feature space for weak supervision and how to intensify this intrinsic distribution under weakly supervised learning?For the first issue about oughtness, we expect to provide a mathematically describable distribution for weakly supervised features. Consequently, we attempt to precisely describe the feature space in terms of two dimensions: distance metric and distribution modeling. In distance metric, we compare the Euclidean norm and cosine similarity between representations, and in distribution modeling, the category prototype model and the mixture model are considered. Among our candidate combinations, the mixture of von Mises-Fisher distributions (moVMF) with cosine similarity is finalized, due to its powerful fitting capability to segment head and insensitivity to the Curse of Dimensionality [44] . We believe that a superior weakly supervised feature space should adhere to this distribution.For the second issue about practice, we dynamically align the embedding distribution in the hidden space to moVMF during weakly supervised learning. Accordingly, we propose a Distribution Guidance Network (DGNet), comprising a weakly supervised learning branch and a distribution alignment branch. Specifically, the weakly supervised learning branch learns semantic embeddings under sparse annotations, while the distribution alignment branch constrains the distribution of the network embeddings. Via a Nested Expectation-Maximum Algorithm, the semantic features are dynamically refined. Therefore, restricting and fitting is a mutually reinforcing, iterative optimization process. To curtail the pattern of feature distribution, we derive the vMF loss based on the maximum likelihood estimation and the discriminative loss inspired by metric learning [22] . For joint optimization, consistency loss is imposed between the segmentation predictions and the posterior probabilities. During the inference phase, only the weakly supervised learning branch is activated to maintain inference consistency with fully supervised learning.We validate DGNet on three prevailing point cloud datasets, i.e., S3DIS [1] , ScanNetV2 [11] ,and SemanticKITTI [5] . After the constraints of feature distribution, DGNet provides significant performance improvements over multiple baselines. Across various label rates, our method achieves state-of-the-art weakly supervised semantic segmentation performance. Extensive ablation studies also confirm the effectiveness of each loss term we proposed. In addition, posterior probabilities under the moVMF provide a plausible interpretation for predictions on unlabeled points. 
related work: 
 Weakly Supervised Point Cloud Semantic Segmentation. Weakly supervised point cloud semantic segmentation methods aim to provide reliable additional supervision with sparse annotations. Four paradigms have been successively proposed in recent years, i.e., perturbation consistency, contrastive learning, self-training, and similarity metric. Perturbation consistency methods are based on the assumption of perturbation invariance of network features, imposing diverse perturbations (such affine transforms with point jitter [59, 54] , downsampling [56] , masking [35] , etc.) to construct pairs of point clouds. Several methods [31, 34] introduce contrastive learning in weak supervision to encourage the discriminability of hidden layer features. Additionally, pre-training methods [55, 17] with contrastive learning similarly demonstrate the ability to bias induction in the face of downstream semantic segmentation tasks with sparse annotations. Self-training methods generate reliable dynamic pseudo-labels based on previous stage predictions for subsequent training stages. Via CAMs [62] , MPRM [51] and J2D3D [25] dynamically generate point-wise pseudo-labels from subcloud-level annotations and image-level annotations, respectively. Recently, REAL [24] integrate SAM [23] to self-training. Similarity metric methods measure the similarity between labeled and unlabeled points to propagate supervision information, in which the similarity is elaborated on low-level features [49, 52] , network embedding [36, 18, 40] or category prototypes [58, 46] . Most similar to our method is the similarity metric strategy. However, the distinction is that DGNet focuses on describing network embeddings holistically, rather than constructing pair relationships between features.Feature Distribution Constraints. The constraints of feature distribution are always presented in the form of feature clustering. DeepClustering [6] , which integrates clustering and unsupervised feature learning, utilizes the clustering results as pseudo-labels to extract visual features dynamically. Following this groundbreaking work, a series of subsequent studies [7, 2, 30] apply feature clustering in unsupervised learning to obtain discriminative visual features for pretraining. For point cloud semantic segmentation, PointDC [10] delineates semantic objects by aligning the features on the same super-voxel in an unsupervised manner. Feng et al. [12] imposes a clustering-based representation learning to enhance the discrimination of embeddings under full supervision. In contrast, our DGNet is oriented towards weakly supervised semantic segmentation by restricting the feature distribution.Mixture of von Mises-Fisher Distributions. The moVMF [3] describes the embeddings of multiple categories on the unit hypersphere in feature space, where the parameters are jointly optimized with the clustering results by Expectation-Maximum algorithm [3, 4] . Some work attempts to combine neural networks with the moVMF in the deep learning era. For example, [16] view face verification as a direct application of clustering, introducing a vMF loss to align the distribution of face features.Segsort [21] , on the other hand, utilizes the prior of the moVMF to over-segment images. DINO-VMF [13] achieve a more stable pre-trained method by precisely describing DINO [8] as a moVMF.In this work, we discuss the superiority of moVMF in characterizing semantic embeddings and trust it as a priori to guide weakly supervised learning.3 MethodologyQuestion: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: The limitations of our work can be found in Sec. 5. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
preliminaries: 
 Task Definition. Without loss of generality, a point cloud for weakly supervised learning is denoted as{(X l , Y), (X u , ∅)} = {(x 1 , y 1 ), • • • , (x m , y m ), x m+1 , • • • , x n }, where X l and X u are the point sets with and without annotations, respectively. Y is the corresponding annotations on X l , in which y i ∈ C and C is the set of category indices. n and m are the point numbers of the point cloud and labeled set, respectively. Fed into the segment head, the network embedding f i is projected into the category probability vector p i . The partial cross-entropy loss is employed in conventional weakly supervised semantic segmentation:EQUATIONwhere p yi i represents the probability of y i -th category in p i .von Mises-Fisher Distribution (vMF). The vMF has demonstrated strong data fitting and generalization capabilities in the fields of self-supervised learning [9, 13] , classification [44] , variational inference [47] , and online continual learning [38] . This distribution describes the distribution of normalized embedding v i = norm(f i ) on the unit hypersphere, with the probability density function:EQUATIONwhere u represents the mean vector of vMF and κ ≥ 0 is a concentration parameter that controls the probability concentration around µ. C d (κ) is the normalization constant.Mixture of vMF (moVMF). Similar to other mixture models, moVMF treats vMF as a subdistribution to describe the overall distribution of multiple categories. Over the entire set of categories C, the probability density function of moVMF is formulated as:EQUATIONwhere Θ = {α c , κ c , u c |c ∈ C} is the parameters of moVMF. α c denotes the proportion of the von Mises-Fisher distribution for the c-th category and α c = 1.We intend to provide additional supervision signals for weakly supervised learning by portraying and enhancing its inherent distribution. Specifically, We explore it in two dimensions, i.e., the distance metric and the distribution modeling:• Distance metric. Distance metric influences the similarity relationship between features. We consider the two most commonly used distance measures in clustering, i.e., Euclidean norm and cosine similarity. For given vectors u and v, the Euclidean norm is defined as ∥u -v∥ 2 and the cosine similarity is defined as u ⊤ v ∥u∥2∥v∥2 . Cosine similarity can be viewed as the inner product of normalized u and v.• Distribution modeling. Distribution modeling determines the clustering results of features. A straightforward model is the Category Prototype [45] . In this model, clusters are assigned by comparing the distance between the features and each category prototype. In addition, we incorporate mixture models into the comparison. Depending on the distance measure, we categorize it into the Gaussian Mixture Model (GMM) with Euclidean norm and the mixture of von Mises-Fisher distributions (moVMF) with cosine similarity, respectively.Describing the feature space of a neural network remains an open problem. Various factors influence the feature space, including network architecture, training data, parameter configurations, and the optimization (loss) function. Given this intractability, deriving a universally optimal mathematical description is impractical. Therefore, we discuss the merits and demerits of these candidate distributions from the following perspectives:• Segment head. To facilitate the analysis, we simplify the structure of the segment head as SegHead(f ) = argmax(softmax(wf ⊤ )), where f is the semantic feature extracted by the decoder and w is the parameter of the output layer. Consider a group of feature vectors {kf |k ≥ 0 & f ̸ = 0}. For any two feature vectors k 1 f and k 2 f within this group, the segmentation predictions are identical, i.e., SegHead(k1 f ) = argmax(softmax(k 1 wf ⊤ )) = argmax(softmax(k 2 wf ⊤ )) = SegHead(k 2 f ).If the general case of using an activation function is taken into account, it does not change the result after argmax since the activation function is usually monotonically nondecreasing. Therefore, the segment head is a radial classifier with a more pronounced classification performance on the angles, so cosine similarity describes the feature space better than the Euclidean norm. • Curse of dimensionality. Another advantage of cosine similarity can be explained in terms of the Curse of Dimensionality. Most high-dimensional features are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors. Besides, the Euclidean norm is sensitive to scale while cosine similarity is not affected by the length of the vectors. • Fitting ability. Despite its computational simplicity, the Category Prototype Model clusters the features by comparing the distance between features and category prototypes. This means that the Category Prototype Model ignores the distribution within categories and the variability between categories. In contrast, the Mixture Model possesses intra-category fitting and inter-category perception capabilities.Based on the above analysis and experimental validation in Sec. 4.3, we characterize the feature space as moVMF and propose the Distribution Guidance Network to enhance this distribution.To enhance the intrinsic distributions discussed in Section 3.2, we propose a Distribution Guidance Network (DGNet). The structure of DGNet is shown in Fig. 2 , which comprises the weakly supervised learning branch and the distribution alignment branch.Sparse annotations pose two challenges for the point cloud semantic segmentation. The first is underfitting the entire dataset, as the supervision signals are insufficient for complex structured point clouds. To address underfitting, we introduce additional signals by reinforcing the inherent distribution of the feature space. The second challenge is overfitting within the labeled set X l , as the model capacity is more than adequate to fit the labeled points. To mitigate overfitting, we replace the conventional partial cross-entropy loss Eq. 1 with the truncated cross-entropy loss [60] in the weakly supervised learning branch, which is defined as:EQUATIONwhere β ∈ [0, 1] represents the threshold for truncating the cross-entropy loss. With L tCE , the gradient of the cross-entropy loss is curtailed when the predicted class probability for an annotated point exceeds β. Over-optimization for that point is halted, thereby preventing overfitting.The weakly supervised learning branch also provides robust initialization for the distribution alignment branch by utilizing the average feature vector of the labeled points. According to the Central Limit Theorem [27] , the difference between the initialization vector from the weakly supervised learning branch and the theoretical optimal average vector conforms to a Gaussian distribution with a mean of 0. The initialized mean vector in DGNet has a high probability of appearing in the vicinity of the optimal solution, which facilitates the clustering algorithm in achieving rapid and stable convergence.The feature space descriptor moVMF, investigated in Sec. 3.2, is employed to regulate the feature space under weakly supervised learning. Initially, the network embeddings are normalized and projected onto the unit hyperspherical surface of the feature space, i.e., v i = norm(f i ). Following this, the optimization objective function is defined utilizing maximum likelihood estimation1 :EQUATIONwhere ϕ, V = {v i }, Z = {z i } and Θ = {α c , κ, u c } denote the learnable network parameters, the normalized network embeddings, the corresponding clustering results and the parameters of moVMF, respectively. To avoid the long-tail problem and simplify computations, the concentration parameter κ is fixed as a constant in our implementation. Since the clustering initialization is category-aware, the clustering results z i ∈ C are with category labels. While the primary optimization goal is the learning of network parameters ϕ, we dynamically resolve Z and Θ to furnish a more precise feature space description. Consequently, we develop a Nested Expectation-Maximum Algorithm to manage the challenge associated with the three optimization variables delineated in Eq. 5.• E Step (Optimize Θ and Z): Regarding network embeddings as input conditions, we integrate the soft-moVMF EM algorithm [3] into the network to alternately optimize Θ and Z. To enhance the stability and computational efficiency of the algorithm, we utilize the average features of labeled points from the weakly supervised branch to initialize u. The posterior probability set Q = {q i } serves as the soft assignment for the clustering results Z, where q i is defined asEQUATIONCompared to other algorithms in [3] , the soft-moVMF algorithm updates Θ by weighting all features according to their posterior probabilities, considering inter-cluster similarities, thereby achieving more accurate parameter updates. The posterior probability q i ∈ [0, 1] |C|×1 is employed not only as a weighting factor for updates but also in the calculation of the loss function for joint optimization. Furthermore, Q provides a probabilistic explanation for the predictions during the inference phase. EQUATIONAdditionally, acknowledging the significance of distinct decision boundaries within the mixture model, we incorporate a discriminative loss derived from metric learning [22] which is defined as:EQUATIONThe interpretation of Bayesian posterior probabilities for predictions based on the moVMF is an attractive property of DGNet. Fig. 3 visualizes the posterior probabilities for some categories. Taking the floor as an example, according to the Bayesian theorem, those points with relatively high posterior probabilities are more likely to be floor, which explains the prediction results. In addition to the previously mentioned initialization, we introduce a consistency loss to fortify the exchange of information between the two branches. This consistency loss is imposed on the class probability map p from the weakly supervised learning branch and the posterior probability q from the distribution alignment branch, in the form of cross-entropy:EQUATIONIf regard the posterior probability q as pseudo-labels, the consistency loss is proved to diminish prediction uncertainty and alleviate distribution discrepancies in [48] .Without laborious adjustments to the weights2 , the overall loss function is defined as follows:EQUATION4 Experimental Analysis Implementation details. ResGCN-28 in DeepGCN [29] and PointNeXt-l [43] are reimplemented as the segment backbones with OpenPoints library [43] . We discard the last activation layer of the decoder to extract orientation-completed feature space. We maintain a memory bank [53] to store class prototypes across the entire dataset. In cases where class annotations are absent from the scene, the class prototypes from the memory bank are employed as supplementary initialization. We employ the LaDS [39] to maintain a higher rate of training supervision after point cloud sampling. For truncated cross-entropy loss, β = 0.8. The concentration constant κ = 10 and the iteration number t = 10. The distribution alignment branch is not activated in the first 50 epochs to stabilize the feature learning. In our implementation, the DGNet is trained with one NVIDIA V100 GPU on S3DIS, eight NVIDIA TESLA T4 GPUs on ScanNetV2, and one NVIDIA V100 GPU on SemanticKITTI. In the inference stage, only the weakly supervised learning branch is activated to produce predictions.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: We only provide information on the computer resources for the main experiment.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
comparative analysis: 
 Results on S3DIS. We detail the segmentation performance at 0.1% and 0.01% label rates on S3DIS Area 5. DGNet boosts performance for each baseline, which is evenly distributed across categories. 54.7 HybirdCR [31] 56.8 DCL [57] 59.6 GaIA [28] 65.2 EDRA [48] 63.0 AADNet [39] 66.8 DGNet (PointNeXt) 67.4 20pts Hou et al. [17] 55.5 OTOC [36] 59.4 MILTrans [56] 54.4 DAT [54] 55.2 PointMatch [52] 62.4 EDRA [48] 57.0 AADNet [39] 62.5 DGNet (PointNeXt) 62.9 The lower the label rate, the more distinct the enhancement brought by DGNet, which suggests that the guidance on feature distribution is more valuable with extremely sparse annotations. Specifically, DGNet achieves more than 97% performance of fullysupervision with only 0.1% points labeled. The 0.02% label rate denotes a sparse labeling form of "one-thingone-click". DGNet outperforms these methods without introducing super-voxel information. In addition, Fig. 4 visualizes a qualitative comparison. DGNet provides a holistic enhancement to the baseline. Although the photos on the wall are misclassified, DGNet captures consistent objects more accurately than the baseline.Compared to S3DIS, ScanNetV2 involves diverse categories and versatile scenes. Therefore, following the super-voxel setting in OTOC [36] , we report the segmentation performances with 20 labeled points per scene and 1% points labeled. The cross-entropy loss term generates relatively sufficient supervised information to train the network due to introducing pseudo-labeling, resulting in a less pronounced DGNet improvement than S3DIS. However, DGNet is still slightly superior to the latest SOTA methods.Results on SemanticKITTI. DGNet performs excellently on indoor datasets and demonstrates strong weakly supervised learning efficiency on outdoor Se-manticKITTI. For a fair comparison, we replace the DGNet backbone with RandLA-Net [19] . DGNet outperforms SQN [18] with 1.0% and 2.1% mIoU on 0.1% and 0.01% label rates, respectively, demonstrating the necessity of supervision on feature space. All ablation studies are performed on S3DIS with PointNeXt-l as baseline.Distribution comparison. We impose a comparison experiment in the distribution alignment branch of DGNet for distribution selection. The relevant experimental results are reported in Tab. 4. For category prototype models, we discard the vMF loss L vMF and consistency loss L CON due to the lack of corresponding forms. For the mixture model with Euclidean Norm (GMM), we replace the maximum likelihood estimation in GMM form with the L vMF in DGNet. In terms of distance metrics, cosine similarity trumps Euclidean norm. In terms of distribution modeling, mixture models have a significant performance advantage over the category prototype models. Integrating these two aspects, the stronger fitting ability of moVMF leads to more accurate and effective supervised signals for weakly supervised learning in DGNet.Ablation study for loss terms. Tab. 5 demonstrates the validity of each loss term in DGNet.Compared with partial cross-entropy loss, the truncated cross-entropy loss improves segmentation performance due to its avoidance of overfitting. Performance improvements are obtained by imposing L vMF with soft assignment form, L DIS and L CON individually, and optimal performance is achieved by using these loss terms simultaneously. In contrast to the soft assignment, the hard assignment does not take into account the inter-cluster similarity and is mismatched with the soft-moVMF algorithm. Therefore, L vMF with hard assignment form in hard-moVMF algorithm and KNN-moVMF algorithm undermines the segmentation efficiency.Ablation study for Nested EM Algorithm. We ablate the proposed Nested Expectation-Maximum Algorithm in two respects. First, we optimize certain parameters on moVMF and fix other parameters with initialized values. The first, second, third, and last rows in Tab. 6 reveal that individually optimizing parts of the parameters impairs the segmentation performance. Secondly, we ablate how the parameters of moVMF are updated. Compared with kNN-moVMF (fourth raw) and hard-moVMF (fifth raw) in [3] , the soft assignment strategy delivers 2.6% and 2.2% mIoU improvements, respectively. This shows that the optimization of moVMF parameters benefits from the soft assignment strategy.Hyperparameter selection. In Tab. 7, we search the parameter space for suitable κ, t, and β. We observe that (a) the segmentation performance shows an increasing and then decreasing trend as the concentration constant κ increases. Our analysis suggests that too small κ leads to a dispersion of features within the class, which can be easily confused with other classes. And too large κ forces overconcentration of features within the class and overfits the network. (b) As the iteration number t increases, the segmentation performance gradually rises and then stabilizes. We believe that the soft-moVMF algorithm gradually converges as t increases, and increasing t after convergence will no longer bring further gains to the network. (c) As the truncated threshold β decreases, the segmentation performance shows a tendency to first increase and then decrease. The conventional cross-entropy loss function is the truncated cross-entropy loss function with β = 1. When β decreases, the overfitting on sparse annotations is alleviated, but when β is too small, it weakens the supervised signal on sparse labeling leading to performance degradation.  
limitations and future work: 
 Despite the promising performance achieved by DGNet, exploring the distribution of embeddings is preliminary. Feng et al. [12] proposes a more sophisticated distribution to restrict feature learning with full supervision. However, such refinement restrictions will lead to overfitting under sparse annotations. Therefore, how to prevent weakly-supervised learning overfitting with enhanced feature description is a promising research topic. 
conclusion: 
 In this paper, we propose a novel perspective by regulating the feature space for weakly supervised point cloud semantic segmentation and develop a distribution guidance network to verify the superiority of this perspective. Based on the investigation of the distribution of semantic embeddings, we choose moVMF to describe the intrinsic distribution. In DGNet, we alleviate the underfitting across the entire dataset and overfitting within the labeled points. Extensive experimental results demonstrate that DGNet rivals or even surpasses the recent SOTA methods on S3DIS, ScanNetV2, and SemanticKITTI. Moreover, DGNet demonstrates the interpretability of network predictions and scalability to various label rates. We expect our work to inspire the point cloud community to strengthen the inherent properties of weakly supervised learning.For the missing categories of this scene, we retrieve the category prototypes ρ from the memory bank as supplementary initialization. Consequently, the initial vector h c is formulated as:EQUATION)Pseudo Code. As presented in Algorithm 1, we incorporate prior knowledge about the semantics during the optimization process based on soft-moVMF [3] . The weights α = 1 /|C| are initialized uniformly. Subsequently, based on the cosine similarity between features on each point and mean directions of each category, the prior probability f and the posterior probability q i are estimated. Finally, we determine the clustering result z i for each point by argmax(q i ). After updating the mean directions u and weights α, the process is repeated until the clustering results converge. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We cite the original papers that produce the codes and datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: Impact of label rates. To demonstrate the capability of DGNet on extreme label rates, we compare the segmentation performance on sparse annotations over a larger range of rates. Tab. 8 reports the mIoU performance of the DGNet and baseline at 10%, 1%, 0.1%, 0.01%, and 0.001% label rates. It can be observed that at 100,000 times less sparse annotations, the baseline fails to learn accurate semantic embedding from it. At the same time, our DGNet still maintains acceptable segmentation performance since it can be conducted unsupervised. Varying labeled points. Following SQN [18] , we verified the sensitivity of DGNet (PointNeXt) to different labeled points at the same label rate. We repeated the experiment five times for each label setting, keeping the network and label rate unchanged and changing only the labeled points' locations. In Tab. 9, we observe a slight performance fluctuation within a reasonable range. 1. Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: The main claim of this paper is using mathematically definable feature distributions to promote the learning of point cloud semantic segmentation under weak supervision. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Justification: We conduct in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no societal impact of this work.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).A complete reasoning process can be found in the Appendix.The experiments demonstrate that DGNet is not sensitive to loss term weights. 


Decision: This paper receives bipolar ratings of 1 accept, 1 borderline accept, and 2 borderline reject. Most reviewers agree the core problem of investigating feature distribution raised by the paper is important and interesting, and the authors did sufficient experiments on the benchmark datasets. However, there are few concerns raised regarding the oughtness of the selected moVMF distributions, incremental improvements against existing methods, and lack of evaluation on large-scale outdoor datasets. Some reviewers' concerns are addressed by the rebuttal but not all of them.

After carefully reading the paper, reviews, and the rebuttal discussion, AC finds that the paper is well-written, logically sound with adequate evaluation. Therefore, AC thinks the paper is ready and suitable for publication in NeurIPS. The authors are encouraged to add more justification on the oughtness of moVMF, given cosine similarity simply outperforms euclidean distance in high dimensional space and moVMF is the only distribution candidate in the design space. We congratulate the authors on the acceptance of their paper!

Reviews: review: paper_summary: paper_summary: This paper outlines a study focusing on distance metric and distribution modeling, highlighting the effectiveness of combining mixture of von Mises-Fisher distributions (moVMF) with cosine similarity. A Distribution Guidance Network (DGNet) is introduced, featuring two main branches: weakly supervised learning and distribution alignment. The network utilizes reliable clustering initialization from the weakly supervised branch to iteratively update moVMF parameters and network parameters, ensuring alignment with moVMF-defined feature spaces.
strength_weakness: strength_weakness: 1. The section of 4.2 and 4.3 can be written in more detail. In particular, the explanation of changes in model performance in ablation experiments should be added. Specifically, a thorough explanation of the changes in model performance observed during ablation experiments should be incorporated. Ablation experiments involve systematically removing components or features from the model to analyze their impact on performance. By detailing how each modification affects the overall performance metrics, readers gain insights into the critical elements contributing to the model's effectiveness.
2. In Abstract, there is no a clear problem to address from existing methods of weakly supervised learning for 3D point clouds. Similarly, in Introduction, it’s also not clear about the motivation of this work and existing problems of this topic.
3. There is no main contributions about this work in Introduction.
4. How to balance weights of each loss function in Equation (9)? 
5. There is no any qualitative comparison between this work with others.
6. Some quantitative results from this work are also incremental improvements compared to others within one percentage point.
7. Compared with previous work, the performance improvement of DGNet model may not be substantial enough to justify its superiority over existing methods, especially on the ScanNet and SemanticKITTI datasets.
8. In the section of 4.3, why does the performance of the model degrade when optimizing some parameters? What is the impact of parameter optimization on resource consumption?
questions: questions: Please answer the questions in Weaknesses.
limitations: limitations: The authors have thoroughly discussed the limitations of DGNet, and there are no potential negative societal impacts associated with their work.
paper_summary: paper_summary: This paper addresses the problem of weakly supervised point cloud semantic segmentation. The authors propose imparting supplementary supervision signals by regulating the feature space under weak supervision. The initial investigation identifies which distributions accurately characterize the feature space in fully supervised learning, subsequently leveraging this prior knowledge to guide the alignment of weakly supervised embeddings. The authors first investigate different feature space characterizations and find that a mixture of von Mises-Fisher (moVMF) distributions with cosine similarity best describes the fully supervised embedding space. Based on this finding, DGNet is proposed with two branches: a weakly supervised learning branch and a distribution alignment branch. Leveraging reliable clustering initialization derived from the weakly supervised learning branch, the distribution alignment branch alternately updates the parameters of the moVMF and the network, ensuring alignment with the moVMF-defined feature space. Experimental results demonstrate that DGNet achieves state-of-the-art performance on multiple datasets.
strength_weakness: strength_weakness: 1. Computational Complexity: The paper omits a crucial discussion on the computational complexity of the distribution alignment branch, particularly regarding the EM-like algorithm. This is important, given that EM-based clustering methods traditionally tend to be computationally intensive.

2. Ablation Study: The current ablation study would benefit from exploring a broader range of loss term combinations. For instance, examining the performance of the complete loss function without $L_{DIS}$ or without $L_{CON}$ would provide more comprehensive insights into the contribution of each component.

3. Discussion and Analysis of moVMF: The paper lacks a deep analysis or discussion explaining why the mixture of von Mises-Fisher (moVMF) distribution outperforms other distributions. Additionally, it fails to provide formal guarantees on convergence or optimality. A more thorough discussion or justification for the choice of moVMF over alternative distributions would be better.
questions: questions: See weakness
limitations: limitations: A main limitation is the computational complexity of the training.
paper_summary: paper_summary: A weakly-supervised approach has been studied to relieve the annotation burden for point cloud semantic segmentation. Unlike conventional works that mainly use priors (such as similarity or augmentation-based regularization) to overcome the lack of information in weak labels, this paper introduces a novel approach directly aligning the features on the latent space of a neural network. Specifically, the proposed method employs a mixture of von Mises-Fisher distribution to model the feature distribution. This brings meaningful gains on multiple distinct architectures across various benchmarks.
strength_weakness: strength_weakness: I think this paper has no major flaws in methodological aspects.
Therefore, I would like to ask some questions for further discussion.

1. Table 1 shows that a mixture of vMF distributions represents the distribution of normalized features better than the other choices. But why? Can you discuss the "oughtness" in more detail, at least at the level of the hypothesis? For example, why is using a mixture of vMFs better than that of Gaussians? 

2. Intuitively, using a hypersphere can represent each feature as an angle, resulting in a more concise distribution. However, if the number of categories increases, this kind of approach may not effectively use the latent space, as all the points on a line map to one single point on the hypersphere (due to normalization). I wonder if the proposed method faces difficulty when handling the dataset including more categories.

3. It will be interesting if the authors check the feature distribution of the neural network trained by fully-supervised learning. Do the features still follow the mixture of vMFs well or not? Can the proposed method even enhance the performance in a fully-supervised setting?

4. Related works.
As far as I know, MPRM does not use 2D images (Line 82). And it would be better if the authors could cite recent papers in CVPR 2024.
questions: questions: See the weaknesses.
limitations: limitations: Yes.
paper_summary: paper_summary: In this work, the authors use a selected feature space distribution as supplementary supervision signals, combining with cosine similarity as the distance metric, and implement a dual-branch weakly supervised point cloud semantic segmentation network. Experiments show that using feature space distribution as supervision information can improve the performance of existing point cloud semantic segmentation networks.
strength_weakness: strength_weakness: As the feature space distribution is selected in a fully supervised manner, it is questionable whether it is suitable for weakly supervised scenarios where labeled samples are extremely sparse. Directly comparing different distributions in the distribution alignment branch of DGNet is more convincing.

Eq. 1 in supplementary material, the last term should be log(a_zi) + k_zi * u^T_zi * vi, instead of log(a_zi) * k_zi * u^T_zi * vi.

In Eq. 9, why use the same weight for each loss term? Considering that these losses are defined from very different perspectives, their values can be of different orders of magnitude.

There are some minor flaws in the writing: some tables and figures are not referenced in the text, and some figures are not numbered in the order referenced.
questions: questions: Please respond to my comments written above.
limitations: limitations: Evaluation on outdoor datasets is insufficient.

Comments: title: The analysis of moVMF from the perspective of the Curse of Dimensionality
A detailed explanation for R1
comment: Thanks for your response. We believe that your final comments are basically from other reviewers and have already been responded to by us. Even though you did not change your final score, we would still like to clarify these points of concern you listed as a matter of courtesy and respect:

* About improvement and certificates. The category prototype model with cosine similarity and the final adopted moVMF are compared under our proposed distributional alignment framework. **There is a logical problem with taking two different distributional choices under the same framework to show that the framework is not valid.** In addition, we do not agree that there is no theoretical proof for the method you claim. We devote considerable space throughout the rebuttal phase to theorizing the superiority of moVMF, please refer to R3 to reviewer KHTz and R1 to reviewer ruvS.

* About comparison. We believe that your concern stems from reviewer fVFW's question 1. However, this concern has been eliminated based on our experiments and the response of reviewer fVFW. Please refer to R1 to reviewer fVFW.

* About qualitative comparison. We think that some representative samples of visualization comparisons are sufficient to demonstrate the validity of the methodology under the current page limitations. What additional conclusions would you like to draw from more qualitative results?

* About typos: We guarantee that it is a transcription error without affecting the correctness of the method. **We would be regretful if our months of work were to be dismissed simply because of a typo that could be corrected immediately.** Moreover, the $\kappa_c$ is ignored in Eq. 5 because the $\kappa_c$ is a constant value for different catergories.
After reading the authors' responses and comments from other reviewers, I do not change my final score due to the following:
1. As shown in Table 1, compared with the Category Prototype (CP) model plus cosine similarity, the proposed method only shows an incremental improvement. And the method does not have any theoretical certificates. 
2. Further, the comparison is insufficient because the authors do not directly compare their method with CP and other distributions under different weak label ratios. 
3. Regarding the Qualitative comparison, this paper does not give insufficient results (only a few in the manuscript), and they also can not be found in the Appendix. 
4. Some Symbols do not have definitions and are even wrong, e.g., there is no Kc in Equation (5).
Thanks for your response! We are pleased to see that three of your original five concerns have been addressed. Although you still maintain your original scores, we feel it is necessary to provide further clarification for the only two remaining.

* About outdoor datasets: We sincerely disagree with your assertion that the evaluation of DGNet on outdoor scenarios is inadequate. To illustrate that our evaluation of outdoor datasets is relatively adequate in the field of weakly supervised learning, we show whether the methods are validated on these outdoor scenes by the following table:

  | Method       | SemanticKITTI | Semantic3D   |
  | ------------ | ------------- | ------------ |
  | SQN          | $\checkmark$  | $\checkmark$ |
  | OTOC         | $\times$      | $\times$     |
  | CPCM         | $\times$      | $\times$     |
  | PointMatch   | $\times$      | $\times$     |
  | PSD          | $\times$      | $\checkmark$ |
  | HybridCR     | $\checkmark$  | $\checkmark$ |
  | DCL          | $\checkmark$  | $\times$     |
  | MILTrans     | $\times$      | $\times$     |
  | ERDA         | $\times$      | $\times$     |
  | MulPro       | $\times$      | $\times$     |
  | **DGNet (Ours)** | $\checkmark$  | $\checkmark$ |

  This table reveals that **DGNet reaches an adequate level of outdoor validation compared with a bunch of methods.**

* About typos: We guarantee that it is a transcription error without affecting the correctness of the method. **We would be regretful if our months of work were to be dismissed simply because of a typo that could be corrected immediately.**
I keep my rating as Borderline reject, due to the insufficient evaluation on large-scale outdoor dataset, and the serious error in the objective function.
Dear Reviewer KHTz:

Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows:

* A crucial discussion on the computational complexity.

* An ablation study for removing the loss terms individually.

* The discussion and analysis of the superiority of moVMF.

We understand your time is valuable, but your opinion updates are vital to our work.
Dear Reviewer bWaK:

Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows:

* Further explanation of the experimental results.

* One weight balance experiment for tuning the weights of Eq. 9.

* Detailed clarification of the article writing.

We understand your time is valuable, but your opinion updates are vital to our work.
Dear Reviewer fVFW:

Given that the deadline for the reviewer-author discussion period is less than two days away, we are eager to discuss our work with you to clear up any concerns you may have. Overall, we have made the main efforts based on your comments as follows:

* One comparison experiment in the distribution alignment branch of DGNet for several distributions.

* One weight balance experiment for tuning the weights of Eq. 9.

* One detailed discussion of DGNet in outdoor scenarios.

We understand your time is valuable, but your opinion updates are vital to our work.
Dear Review KHTz:

The discussion and analysis of moVMF are further enriched during the conversation with the Review ruvS. Specifically, we consider further analyzing the difference between the Euclidean norm (Gaussian mixture model) and cosine similarity (moVMF) from the perspective of the **Curse of Dimensionality**.

* The data becomes extremely sparse in feature space as the feature dimension increases. Most feature vectors are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors.

* Euclidean norm is very sensitive to scale. In a high-dimensional feature space, if the feature changes at different scales in different dimensions, it may result in the contribution of some dimensions to the Euclidean norm being magnified while the contribution of other dimensions is ignored. Cosine similarity is not affected by the length of the vectors, and the main measure is the angle between the features. Therefore, changes in the scale of different dimensions do not significantly affect the cosine similarity calculation.
Thanks for your prompt response! We are pleased that our response addresses most of your concerns. We also respect the importance you place on von Mises-Fisher distribution, and we will further elaborate on its superiority in solving the weakly supervised point cloud semantic segmentation task from multiple perspectives in our next release.

We consider further analyzing the difference between the Euclidean norm (Gaussian mixture model) and cosine similarity (moVMF) from the perspective of the **Curse of Dimensionality**.

* The data becomes extremely sparse in feature space as the feature dimension increases. Most feature vectors are far from each other, causing the Euclidean distance to become ineffective in distinguishing differences between feature vectors. Cosine similarity, on the other hand, is more effective in distinguishing differences between features by measuring the angle between the vectors.

* Euclidean norm is very sensitive to scale. In a high-dimensional feature space, if the feature changes at different scales in different dimensions, it may result in the contribution of some dimensions to the Euclidean norm being magnified while the contribution of other dimensions is ignored. Cosine similarity is not affected by the length of the vectors, and the main measure is the angle between the features. Therefore, changes in the scale of different dimensions do not significantly affect the cosine similarity calculation.
Thank you for the rebuttal.
Most of my concerns are addressed.
However, I still do not think that the explanation about the oughtness of using vMF is sufficient.
The main contribution of this paper is introducing vMF distribution to tackle weakly supervised learning of point cloud semantic segmentation.
Hence, from an academic perspective, this part should be more justified and discussed in more detail.
Although I would like to keep my rating, it would be better if the authors could elaborate on this further.
Next, we provide a detailed elaboration of the analysis of the experimental results in Sec. 4.2 and 4.3. **Please note that these additions of details do not detract from the main conclusions in the paper, and we believe that some details are so burdensome that may dilute the focus on key experiments and make the paper lengthy.**

* **Results on S3DIS.** The gain that DGNet brings to the baseline method is more pronounced at 0.01% label rate compared to 0.1% label rate, since the guidance on feature distribution is more valuable with extremely sparse annotations. The 0.02% label rate denotes a sparse labeling form of "one-thing-one-click", and unlike methods that use this labeling form, we did not introduce super-voxel information to enrich the original sparse labeling. However, DGNet outperforms these methods at a smaller label rate. In addition, we would like to re-emphasize that DGNet demonstrates strong generalization capabilities, both in terms of different annotation scales and various baseline methods.

* **Results on ScanNetV2.** Confronted with the diverse categories and versatile scenes of ScanNetV2, we follow the other methods and report on the segmentation performance with the sparse annotations processed by super-voxels. The cross-entropy loss term can generate relatively sufficient supervised information to train the network due to the introduction of pseudo-labelings, resulting in a less pronounced improvement of DGNet than that of S3DIS. However, DGNet is still slightly superior to the latest SOTA methods.

* **Results on SemanticKITTI.** In the field of weakly supervised point-cloud semantic segmentation, only a few works provide segment performance on outdoor SemanticKITTI.  With the same baseline method RandLa-Net, DGNet improves 1.0% and 2.1% mIoU over SQN at 0.1% label rate and 0.01% label rate, respectively. Considering the inherent density inhomogeneity of outdoor scenes, the excellent performance with such sparse annotations validates the effectiveness of DGNet with the outdoor dataset.

* **Interpretability of prediction results.** The interpretation of Bayesian posterior probabilities for trained networks based on the distribution function of the moVMF is an attractive advantage of DGNet. Figure 3 visualizes the posterior probabilities for some categories. Taking the “floor” as an example, according to the Bayesian theorem, those points with relatively high posterior probabilities have high probabilities of belonging to the floor, which is also consistent with the network predictions.

* **Hyperparameter selection.** In Table 5, we search the parameter space for suitable $\kappa$, $t$, and $\beta$. We observe that (1) the segmentation performance shows an increasing and then decreasing trend as the concentration constant $\kappa$ increases. Our analysis suggests that too small $\kappa$ leads to a dispersion of features within the class, which can be easily confused with other classes. And too large $\kappa$ forces overconcentration of features within the class and overfits the network. (2) As the iteration number $t$ increases, the segmentation performance gradually rises and then stabilizes. We believe that the soft-moVMF algorithm gradually converges as $t$ increases, and increasing $t$ after convergence will no longer bring further gains to the network. (3) As the truncated threshold $\beta$ decreases, the segmentation performance shows a tendency to first increase and then decrease. The conventional cross-entropy loss function is the truncated cross-entropy loss function with $\beta=1$. When $\beta$  decreases, the overfitting on sparse annotations is alleviated, but when $\beta$  is too small, it weakens the supervised signal on sparse labeling leading to performance degradation. 

* **Ablation study for loss terms.** Table 6 demonstrates the validity of each loss term in DGNet. It is worth noting that $\mathcal{L}\_\text{vMF}$ with hard assignment form undermines the segmentation efficiency, on the contrary, $\mathcal{L}\_\text{vMF}$ with soft assignment form has a clear positive effect. In contrast to the soft assignment, the hard assignment does not take into account the inter-cluster similarity and is mismatched with the soft-moVMF algorithm. In addition, according to suggestions from another reviewer, we supplement this ablation experiment with more ablative forms. Please refer to the response to the Reviewer KHTz.

* **Ablation study for soft-moVMF.**  The experimental results in the first, second, third, and last rows of Table 7 show that it is optimal to optimize both $\alpha$ and $\mathbf{u}$ with the soft-moVMF algorithm. Optimizing $\alpha$ and $\mathbf{u}$ individually impairs the segmentation performance, for which the fixed parameters affect the accurate update of the optimizable parameters. The forth, fifth and last rows of Table 7 show that the soft-moVMF outperforms hard-moVMF and KNN-moVMF due to more accurate parameter updates.