Title: All Points Matter: Entropy-Regularized Distribution Alignment for Weakly-supervised 3D Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: point cloud segmentation

weak supervision
Abstract: Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning.
Existing methods often rely on empirical label selection strategies, such as confidence thresholding, to generate beneficial pseudo-labels for model training.
This approach may, however, hinder the comprehensive exploitation of unlabeled data points.
We hypothesize that this selective usage arises from the noise in pseudo-labels generated on unlabeled data. The noise in pseudo-labels may result in significant discrepancies between pseudo-labels and model predictions, thus confusing and affecting the model training greatly.
To address this issue, we propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions.
More specifically, our method introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks, resulting in an ERDA learning strategy.
Interestingly, by using KL distance to formulate the distribution alignment loss, it reduces to a deceptively simple cross-entropy-based loss which optimizes both the pseudo-label generation network and the 3D segmentation network simultaneously.
Despite the simplicity, our method promisingly improves the performance.
We validate the effectiveness through extensive experiments on various baselines and large-scale datasets.
Results show that ERDA effectively enables the effective usage of all unlabeled data points for learning and achieves state-of-the-art performance under different settings.
Remarkably, our method can outperform fully-supervised baselines using only 1\% of true annotations.
Code and model will be made publicly available at https://github.com/LiyaoTang/ERDA.

All Points Matter: Entropy-Regularized Distribution Alignment for Weakly-supervised 3D Segmentation

Abstract

Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning. Existing methods often rely on empirical label selection strategies, such as confidence thresholding, to generate beneficial pseudo-labels for model training. This approach may, however, hinder the comprehensive exploitation of unlabeled data points. We hypothesize that this selective usage arises from the noise in pseudo-labels generated on unlabeled data. The noise in pseudo-labels may result in significant discrepancies between pseudo-labels and model predictions, thus confusing and affecting the model training greatly. To address this issue, we propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions. More specifically, our method introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks, resulting in an ERDA learning strategy. Interestingly, by using KL distance to formulate the distribution alignment loss, it reduces to a deceptively simple cross-entropy-based loss which optimizes both the pseudo-label generation network and the 3D segmentation network simultaneously. Despite the simplicity, our method promisingly improves the performance. We validate the effectiveness through extensive experiments on various baselines and large-scale datasets. Results show that ERDA effectively enables the effective usage of all unlabeled data points for learning and achieves state-of-the-art performance under different settings. Remarkably, our method can outperform fully-supervised baselines using only 1% of true annotations. Code and model will be made publicly available at https://github.com/LiyaoTang/ERDA.

1 Introduction

Point cloud semantic segmentation is a crucial task for 3D scene understanding and has various applications [27, 83], such as autonomous driving, unmanned aerial vehicles, and augmented reality. Current state-of-the-art approaches heavily rely on large-scale and densely annotated 3D datasets, which are costly to obtain [21, 33]. To avoid demanding exhaustive annotation, weakly-supervised point cloud segmentation has emerged as a promising alternative. It aims to leverage a small set of annotated points while leaving the majority of points unlabeled in a large point cloud dataset for learning. Although current weakly supervised methods can offer a practical and cost-effective way to perform point cloud segmentation, their performance is still sub-optimal compared to fully-supervised approaches.

During the exploration of weak supervision [85], a significant challenge is the insufficient training signals provided by the highly sparse labels. To tackle this issue, pseudo-labeling methods [85, 94, 32] have been proposed, which leverage predictions on unlabeled points as labels to facilitate the learning of the segmentation network. Despite some promising results, these pseudo-label methods have been outperformed by some recent methods based on consistency regularization [47, 87]. We tend to attribute this to the use of label selection on pseudo-labels, such as confidence thresholding, which

37th Conference on Neural Information Processing Systems (NeurIPS 2023).

NetworkPrediction q

mloU

ERDA

ERDA

68

one-hot (existing)

★

w/ full anno.

Stop

67

·

othermethods

Pseudo-Label

Label

Grad.

<H

H(p, q)

☆

baseline w/1% anno.

Generation

Selection

66

★

P

·HybridCR [47]

65

(a) Existing: Sparse Pseudo-Labels

★

64

SQN [32]

63

Network Prediction q

62

zhang. [94]

61

Pseudo-Label

Lp = LER + LDA

Generation

60

p

= H(p,q)

★

0(none) 64 500103

104

all

#pseudo labels (per-scene)

gradients direction

(b) ERDA: Dense Pseudo-Labels

(c)Performance Comparison

could lead to unlabeled points being wasted and under-explored. We hypothesize that the need for label selection arises from the low-confidence pseudo-labels assigned to unlabeled points, which are known for their noises [21] and potential unintended bias [60, 100]. These less reliable and noisy pseudo-labels could contribute to discrepancies between the pseudo-labels and the model predictions, which might confuse and impede the learning process to a great extent.

By addressing the above problem for label selection in weakly supervised 3D segmentation, we propose a novel learning-based approach in this study. Our method aims to leverage the information from all unlabeled points by mitigating the negative effects of the noisy pseudo-labels and the distributional discrepancy.

Specifically, we introduce two learning objectives for the pseudo-label generation process. Firstly, we introduce an entropy regularization (ER) objective to reduce the noise and uncertainty in the pseudo-labels. This regularization promotes more informative, reliable, and confident pseudo-labels, which helps alleviate the limitations of noisy and uncertain pseudo-labels. Secondly, we propose a distribution alignment (DA) loss that minimizes statistical distances between pseudo-labels and model predictions. This ensures that the distribution of generated pseudo-labels remains close to the distribution of model predictions when regularizing their entropy.

In particular, we discover that formulating the distribution alignment loss using KL distance enables a simplification of our method into a cross-entropy-style learning objective that optimizes both the pseudo-label generator and the 3D segmentation network simultaneously. This makes our method straightforward to implement and apply. By integrating the entropy regularization and distribution alignment, we achieve the ERDA learning strategy, as shown in Fig. 1.

Empirically, we comprehensively experiment with three baselines and different weak supervision settings, including 0.02% (1-point, or 1pt), 1%, and 10%. Despite its concise design, our ERDA outperforms existing weakly-supervised methods on large-scale point cloud datasets such as S3DIS [2], ScanNet [16], and SensatUrban [33]. Notably, our ERDA can surpass the fully-supervised baselines using only 1% labels, demonstrating its significant effectiveness in leveraging pseudo-labels. Furthermore, we validate the scalability of our method by successfully generalizing it to more other settings, which illustrates the benefits of utilizing dense pseudo-label supervision with ERDA.

2 Related Work

Point cloud segmentation. Point cloud semantic segmentation aims to assign semantic labels to 3D points. The cutting-edge methods are deep-learning-based and can be classified into projection-based and point-based approaches. Projection-based methods project 3D points to grid-like structures, such as 2D image [84, 55, 39, 12, 4, 45] or 3D voxels [15, 71, 67, 28, 22, 23, 76]. Alternatively, point-based

methods directly operate on 3D points [56, 57]. Recent efforts have focused on novel modules and backbones to enhance point features, such as 3D convolution [3, 48, 78, 68, 51, 58], attentions [34, 26, 97, 79, 42, 59], graph-based methods [74, 44], and other modules such as sampling [18, 86, 88, 7] and post-processing [54, 35, 66]. Although these methods have made significant progress, they rely on large-scale datasets with point-wise annotation and struggle with few labels [85]. To address the demanding requirement of point-wise annotation, our work explores weakly-supervised learning for 3D point cloud segmentation.

Weakly-supervised point cloud segmentation. Compared to weakly-supervised 2D image segmentation [99, 49, 75, 1, 64], weakly-supervised 3D point cloud segmentation is less explored. In general, weakly-supervised 3D segmentation task focus on highly sparse labels: only a few scattered points are annotated in large point cloud scenes. Xu and Lee [85] first propose to use 10x fewer labels to achieve performance on par with a fully-supervised point cloud segmentation model. Later studies have explored more advanced ways to exploit different forms of weak supervision [77, 14, 40] and human annotations [53, 69]. Recent methods tend to introduce perturbed self-distillation [95], consistency regularization [85, 62, 80, 81, 43], and leverage self-supervised learning [62, 37, 47, 87] based on contrastive learning [29, 10]. Pseudo-labels are another approach to leverage unlabeled data, with methods such as pre-training networks on colorization tasks [94], using iterative training [32], employing separate networks to iterate between learning pseudo-labels and training 3D segmentation networks [53], or using super-point graph [44] with graph attentional module to propagate the limited labels over super-points [13]. However, these existing methods often require expensive training due to hand-crafted 3D data augmentations [95, 87, 80, 81], iterative training [53, 32], or additional modules [87, 32], complicating the adaptation of backbone models from fully-supervised to weakly-supervised learning. In contrast, our work aims to achieve effective weakly supervised learning for the 3D segmentation task with straightforward motivations and simple implementation.

Pseudo-label refinement. Pseudo-labeling [46], a versatile method for entropy minimization [24], has been extensively studied in various tasks, including semi-supervised 2D classification [82, 60], segmentation [64, 89], and domain adaptation [98, 73]. To generate high-quality supervision, various label selection strategies have been proposed based on learning status [72, 91, 20], label uncertainty [60, 98, 73, 50], class balancing [100], and data augmentations [64, 89, 100]. Our method is most closely related to the works addressing bias in supervision, where mutual learning [20, 70, 92] and distribution alignment [100, 31, 41] have been discussed. However, these works typically focus on class imbalance [100, 31] and rely on iterative training [70, 20, 92, 41], label selection [20, 31], and strong data augmentations [100, 31], which might not be directly applicable to 3D point clouds. For instance, common image augmentations [64] like cropping and resizing may translate to point cloud upsampling [96], which remains an open question in the related research area. Rather than introducing complicated mechanisms, we argue that proper regularization on pseudo-labels and its alignment with model prediction can provide significant benefits using a very concise learning approach designed for the weakly supervised 3D point cloud segmentation task.

Besides, it is shown that the data augmentations and repeated training in mutual learning [70, 38] are important to avoid the feature collapse, i.e., the resulting pseudo-labels being uniform or the same as model predictions. We suspect the cause may originate from the entropy term in their use of raw statistical distance by empirical results, which potentially matches the pseudo-labels to noisy and confusing model prediction, as would be discussed in Sec. 3.2. Moreover, in self-supervised learning based on clustering [5] and distillation [6], it has also been shown that it would lead to feature collapse if matching to a cluster assignment or teacher output of a close-uniform distribution with high entropy, which agrees with the intuition in our ER term.

3 Methodology

3.1 Formulation of ERDA

As previously mentioned, we propose the ERDA approach to alleviate noise in the generated pseudolabels and reduce the distribution gaps between them and the segmentation network predictions. In general, our ERDA introduces two loss functions, including the entropy regularization loss and the distribution alignment loss for the learning on pseudo-labels. We denote the two loss functions as L ER and L DA , respectively. Then, we have the overall loss of ERDA as follows:

where the λ > 0 modulates the entropy regularization which is similar to the studies [46, 24].

Before detailing the formulation of L ER and L DA , we first introduce the notation. While the losses are calculated over all unlabeled points, we focus on one single unlabeled point for ease of discussion. We denote the pseudo-label assigned to this unlabeled point as p and the corresponding segmentation network prediction as q . Each p and q is a 1D vector representing the probability over classes.

Entropy Regularization loss. We hypothesize that the quality of pseudo-labels can be hindered by noise, which in turn affects model learning. Specifically, we consider that the pseudo-label could be more susceptible to containing noise when it fails to provide a confident pseudo-labeling result, which leads to the presence of a high-entropy distribution in p .

To mitigate this, for the p , we propose to reduce its noise level by minimizing its Shannon entropy, which also encourages a more informative labeling result [61]. Therefore, we have:

where H ( p ) = ∑ i -p i log p i and i iterates over the vector. By minimizing the entropy of the pseudo-label as defined above, we promote more confident labeling results to help resist noise in the labeling process 1 .

Distribution Alignment loss. In addition to the noise in pseudo-labels, we propose that significant discrepancies between the pseudo-labels and the segmentation network predictions could also confuse the learning process and lead to unreliable segmentation results. In general, the discrepancies can stem from multiple sources, including the noise-induced unreliability of pseudo-labels, differences between labeled and unlabeled data [100], and variations in pseudo-labeling methods and segmentation methods [92, 20]. Although entropy regularization could mitigate the impact of noise in pseudolabels, significant discrepancies may still persist between the pseudo-labels and the predictions of the segmentation network. To mitigate this issue, we propose that the pseudo-labels and network can be jointly optimized to narrow such discrepancies, making generated pseudo-labels not diverge too far from the segmentation predictions. Therefore, we introduce the distribution alignment loss.

To properly define the distribution alignment loss ( L DA ), we measure the KL divergence between the pseudo-labels ( p ) and the segmentation network predictions ( q ) and aim to minimize this divergence. Specifically, we define the distribution alignment loss as follows:

where KL ( p || q ) refers to the KL divergence. Using the above formulation has several benefits. For example, the KL divergence can simplify the overall loss L p into a deceptively simple form that demonstrates desirable properties and also performs better than other distance measurements. More details will be presented in the following sections.

Simplified ERDA. With the L ER and L DA formulated as above, given that KL ( p || q ) = H ( p , q ) -H ( p ) where H ( p , q ) is the cross entropy between p and q , we can have a simplified ERDA formulation as:

In particular, when λ = 1 , we obtain the final ERDA loss 2 :

The above simplified ERDA loss describes that the entropy regularization loss and distribution alignment loss can be represented by a single cross-entropy-based loss that optimizes both p and q .

We would like to emphasize that Eq. (5) is distinct from the conventional cross-entropy loss. The conventional cross-entropy loss utilizes a fixed label and only optimizes the term within the logarithm function, whereas the proposed loss in Eq. (5) optimizes both p and q simultaneously.

̸

3.2 Delving into the Benefits of ERDA

To formulate the distribution alignment loss, different functions can be employed to measure the differences between p and q . In addition to the KL divergence, there are other distance measurements like mean squared error (MSE) or Jensen-Shannon (JS) divergence for replacement. Although many mutual learning methods [20, 92, 41, 38] have proven the effectiveness of KL divergence, a detailed comparison of KL divergence against other measurements is currently lacking in the literature. In this section, under the proposed ERDA learning framework, we show by comparison that KL ( p || q ) is a better choice and ER is necessary for weakly-supervised 3D segmentation.

To examine the characteristics of different distance measurements, including KL ( p || q ) , KL ( q || p ) , JS ( p || q ) , and MSE ( p || q ) , we investigate the form of our ERDA loss L p and its impact on the learning for pseudo-label generation network given two situations during training.

More formally, we shall assume a total of K classes and define that a pseudo-label p = [ p 1 , ..., p K ] is based on the confidence scores s = [ s 1 , ..., s K ] , and that p = softmax ( s ) . Similarly, we have a segmentation network prediction q = [ q 1 , ..., q K ] for the same point. We re-write the ERDA loss L p in various forms and investigate the learning from the perspective of gradient update, as in Tab. 1.

Situation 1: Gradient update given confident pseudo-label p . We first specifically study the case when p is very certain and confident, i.e., p approaching a one-hot vector. As in Tab. 1, most distances yield the desired zero gradients, which thus retain the information of a confident and reliable p . In this situation, however, the KL ( q || p ) , rather than KL ( p || q ) in our method, produces non-zero gradients that would actually increase the noise among pseudo-labels during its learning, which is not favorable according to our motivation.

Situation 2: Gradient update given confusing prediction q . In addition, we are also interested in how different choices of distance and λ would impact the learning on pseudo-label if the segmentation model produces confusing outputs, i.e., q tends to be uniform. In line with the motivation of ERDA learning, we aim to regularize the pseudo-labels to mitigate potential noise and bias, while discouraging uncertain labels with little information. However, as in Tab. 1, most implementations yield non-zero gradient updates to the pseudo-label generation network. This update would make p closer to the confused q , thus increasing the noise and degrading the training performance. Conversely, only KL ( p || q ) can produce a zero gradient when integrated with the entropy regularization with λ = 1 . That is, only ERDA in Eq. (5) would not update the pseudo-label generation network when q is not reliable, which avoids confusing the p . Furthermore, when q is less noisy but still close to a uniform vector, it is indicated that there is a large close-zero plateau on the gradient surface of ERDA, which benefits the learning on p by resisting the influence of noise in q .

In addition to the above cases, the gradients of ERDA in Eq. (5) could be generally regarded as being aware of the noise level and the confidence of both pseudo-label p and the corresponding prediction q . Especially, ERDA produces larger gradient updates on noisy pseudo-labels, while smaller updates on confident and reliable pseudo-labels or given noisy segmentation prediction. Therefore, our formulation demonstrates its superiority in fulfilling our motivation of simultaneous noise reduction and distribution alignment, where both L ER and KL-based L DA are necessary. We provide more empirical studies in ablation (Sec. 4.3) and detailed analysis in the supplementary.

3.3 Implementation Details on Pseudo-Labels

In our study, we use a prototypical pseudo-label generation process due to its popularity as well as simplicity [94]. Specifically, prototypes [63] denote the class centroids in the feature space, which

Network

b

Prediction

Inputs

Features

Projected

Score

DensePseudo-Labels

X

MLPs

unlabeled

softmax

= LEr + LDA

S

= H(p, q)

labeled

Projection

Network

g

Momentum

update

→

connectionw/gradients

ERDA

→connectionw/ogradients

w/Pseudo-Label Generation

Class Centroids

→

gradients direction

are calculated based on labeled data, and pseudo-labels are estimated based on the feature distances between unlabeled points and class centroids.

As shown in Fig. 2, we use a momentum-based prototypical pseudo-label generation process due to its popularity as well as simplicity [94, 85, 93]. Specifically, prototypes [63] denote the class centroids in the feature space, which are calculated based on labeled data, and pseudo-labels are estimated based on the feature distances between unlabeled points and class centroids. To avoid expensive computational costs and compromised representations for each semantic class [94, 87, 47], momentum update is utilized as an approximation for global class centroids.

Based on the momentum-updated prototypes, we attach an MLP-based projection network to help generate pseudo-labels and learn with our method. Aligned with our motivation, we do not introduce thresholding-based label selection or one-hot conversion [46, 94] to process generated pseudo-labels. More details are in the supplementary.

More formally, we take as input a point cloud X , where the labeled points are X l and the unlabeled points are X u . For a labeled point x ∈ X l , we denote its label by y . The pseudo-label generation process can be described as follows:

where C k denotes the global class centroid for the k -th class, N l k is the number of labeled points of the k -th class, g ◦ f = g ( f ( · )) is the transformation through the backbone network f and the projection network g , m is the momentum coefficient, and we use cosine similarity for d ( · , · ) to generate the score s . By default, we use 2-layer MLPs for the projection network g and set m = 0 . 999 .

Besides, due to the simplicity of ERDA, we are able to follow the setup of the baselines for training, which enables straightforward implementation and easy adaptation on various backbone models with little overhead.

Overall objective. Finally, with ERDA learning in Eq. (5), we maximize the same loss for both labeled and unlabeled points, segmentation task, and pseudo-label generation, where we allow the gradient to back-propagate through the (pseudo-)labels. The final loss is given as

where L p ( q , p ) = L ce ( q , p ) = H ( q , p ) is the typical cross-entropy loss used for point cloud segmentation, N l and N u are the numbers of labeled and unlabeled points, and α is the loss weight.

4 Experiments

We present the benefits of our proposed ERDA by experimenting with multiple large-scale datasets, including S3DIS [2], ScanNet [16], SensatUrban [33] and Pascal [19]. We also provide ablation studies for better investigation.

4.1 Experimental Setup

We choose RandLA-Net [34] and CloserLook3D [51] as our primary baselines following previous works. Additionally, while transformer models [17, 52] have revolutionized the field of computer vision as well as 3D point cloud segmentation [97, 42], none of the existing works have addressed the training of transformer for point cloud segmentation with weak supervision, even though these models are known to be data-hungry [17]. We thus further incorporate the PointTransformer (PT) [97] as our baseline to study the amount of supervision demanded for effective training of transformer.

For training, we follow the setup of the baselines and set the loss weight α = 0 . 1 . For a fair comparison, we follow previous works [94, 95, 32] and experiment with different settings, including the 0.02% (1pt), 1% and 10% settings, where the available labels are randomly sampled according to the ratio 3 . More details are given in the supplementary.

4.2 Performance Comparison

Results on S3DIS. S3DIS [2] is a large-scale point cloud segmentation dataset that covers 6 large indoor areas with 272 rooms and 13 semantic categories. As shown in Tab. 2, ERDA significantly improves over different baselines on all settings and almost all classes. In particular, for confusing classes such as column, window, door, and board, our method provides noticeable and consistent improvements in all weak supervision settings. We also note that PT suffers from severe over-fitting and feature collapsing under the supervision of extremely sparse labels of '1pt' setting; whereas it is alleviated with ERDA, though not achieving a satisfactory performance. Such observation agrees with the understanding that transformer is data-hungry [17].

Ground Truth

Baseline

ERDA

Improvement

Input

Table 4. Results on ScanNet test.

Table 6. Results on Pascal dataset.

Impressively, ERDA yields competitive performance against most supervised methods. For instance, with only 1% of labels, it achieves performance better than its stand-alone baselines with full supervision. Such result indicates that the ERDA is more successful than expected in alleviating the lack of training signals, as also demonstrated qualitatively in Fig. 3.

Therefore, we further extend the proposed method to fully-supervised training, i.e., in setting 'Fully' in Tab. 2. More specifically, we generate pseudo-labels for all points and regard the ERDA as an auxiliary loss for fully-supervised learning. Surprisingly, we observe non-trivial improvements (+3.7 for RandLA-Net and +3.4 for CloserLook3D) and achieve the state-of-the-art performance of 72.6 (+2.2) in mIoU with PT. We suggest that the improvements are due to the noise-aware learning from ERDA, which gradually reduces the noise during the model learning and demonstrates to be generally effective. Moreover, considering that the ground-truth labels could suffer from the problem of label noise [38, 90, 65], we also hypothesize that pseudo-labels from ERDA learning could stabilize fully-supervised learning and provide unexpected benefits.

We also conduct the 6-fold cross-validation, as reported in Tab. 3. We find our method achieves a leading performance among both weakly and fully-supervised methods, which further validates the effectiveness of our method.

Results on ScanNet. ScanNet [16] is an indoor point cloud dataset that covers 1513 training scenes and 100 test scenes with 20 classes. In addition to the common settings, e.g., 1% labels, it also provides official data efficient settings, such as 20 points, where for each scene there are a pre-defined

set of 20 points with the ground truth label. We evaluate on both settings and report the results in Tab. 4. We largely improve the performance under 0 . 1% and 1% labels. In 20pts setting, we also employ a convolutional baseline (CloserLook3D) for a fair comparison. With no modification on the model, we surpass MIL-transformer [87] that additionally augments the backbone with transformer modules and multi-scale inference. Besides, we apply ERDA to baseline under fully-supervised setting and achieve competitive performance. These results also validate the ability of ERDA in providing effective supervision signals.

Results on SensatUrban. SensatUrban [33] is an urban-scale outdoor point cloud dataset that covers the landscape from three UK cities. In Tab. 5, ERDA surpasses SQN [32] under the same 0.1% setting as well as its fully-supervised baseline, and also largely improves under full supervision. It suggests that our method can be robust to different types of datasets and effectively exploits the limited annotations as well as the unlabeled points.

Generalizing to 2D Pascal. As our ERDA does not make specific assumptions on the 3D data, we explore its potential in generalizing to similar 2D settings. Specifically, we study an important task of semi-supervised segmentation on image [72, 89, 8] and implement our method to the popular baseline, FixMatch [64], which combines the pseudo-labels with weak-to-strong consistency and is shown to benefit from stronger augmentation [89]. We use DeepLabv3+ [9] with ResNet-101 [30].

As in Tab. 6, we show that ERDA brings consistent improvement from low to high data regime, despite the existence of strong data augmentation and the very different data as well as setting 4 . It thus indicates the strong generalization of our method. We also see that the improvement is less significant than the 3D cases. It might be because 2D data are more structured ( e.g., pixels on a 2D grid) and are thus less noisy than the 3D point cloud.

4.3 Ablations and Analysis

We mainly consider the 1% setting and ablates in Tab. 7 to better investigate ERDA and make a more thorough comparison with the current pseudo-label generation paradigm. For more studies on hyper-parameters, please refer to the supplementary.

Individual effectiveness of ER and DA. To validate our initial hypothesis, we study the individual effectiveness of L ER and L DA in Tab. 7a. While the pseudo-labels essentially improve the baseline performance, we remove its label selection and one-hot conversion when adding the ER or DA term. We find that using ER alone can already be superior to the common pseudo-labels and largely reduce the entropy of pseudo-labels (ent.) as expected, which verifies that the pseudo-labels are noisy, and reducing these noises could be beneficial. The improvement with the DA term alone is even more significant, indicating that a large discrepancy is indeed existing between the pseudo-labels and model prediction and is hindering the model training. Lastly, by combining the two terms, we obtain the ERDA that reaches the best performance but with the entropy of its pseudo-labels larger than ER only and smaller than DA only. It thus also verifies that the DA term could be biased to uniform distribution and that the ER term is necessary.

Different choices of ER and DA. Aside from the analysis in Sec. 3.2, we empirically compare the results under different choices of distance for L DA and λ for L ER . As in Tab. 7b, the outstanding result justifies the choice of KL ( q || p ) with λ = 1 . Additionally, all different choices and combina-

tions of ER and DA terms improve over the common pseudo-labels (63.3), which also validates the general motivation for ERDA.

Ablating label selection. We explore in more detail how the model performance is influenced by the amount of exploitation on unlabeled points, as ERDA learning aims to enable full utilization of the unlabeled points. In particular, we consider three pseudo-labels types: common one-hot pseudo-labels, soft pseudo-labels ( p ), and soft pseudo-labels with ERDA learning. To reduce the number of pseudo-labels, we select sparse but high-confidence pseudo-labels following a common topk strategy [94, 53] with various values of k to study its influence. As in Tab. 7c, ERDA learning significantly improves the model performance under all cases, enables the model to consistently benefit from more pseudo-labels, and thus neglects the need for label selection such as topk strategy, as also revealed in Fig. 1. Besides, using soft pseudo-labels alone can not improve but generally hinders the model performance, as one-hot conversion may also reduce the noise in pseudo-labels, which is also not required with ERDA.

5 Limitation and Future Work

While we mostly focus on weak supervision in this paper, our method also brings improvements under fully-supervised setting. We would then like to further explore the effectiveness and relationship of L ER and L DA under full supervision as a future work. Besides, despite promising results, our method, like other weak supervision approaches, assumes complete coverage of semantic classes in the available labels, which may not always hold in real-world cases. Point cloud segmentation with missing or novel classes should be explored as an important future direction.

6 Conclusion

In this paper, we study the weakly-supervised 3D point cloud semantic segmentation, which imposes the challenge of highly sparse labels. Though pseudo-labels are widely used, label selection is commonly employed to overcome the noise, but it also prevents the full utilization of unlabeled data. By addressing this, we propose a new learning scheme on pseudo-labels, ERDA, that reduces the noise, aligns to the model prediction, and thus enables comprehensive exploitation of unlabeled data for effective training. Experimental results show that ERDA outperforms previous methods in various settings and datasets. Notably, it surpasses its fully-supervised baselines and can be further generalized to full supervision as well as 2D images.

Acknowledgement. This project is supported in part by ARC FL-170100117, and IC-190100031.

References

Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4981-4990, 2018. 3

Iro Armeni, Sasha Sax, Amir R Zamir, and Silvio Savarese. Joint 2d-3d-semantic data for indoor scene understanding. arXiv preprint arXiv:1702.01105 , 2017. 2, 6, 7, 15

Alexandre Boulch, Gilles Puy, and Renaud Marlet. Fkaconv: Feature-kernel alignment for point cloud convolution, 2020. 3

A. Boulch, B. Le Saux, and N. Audebert. Unstructured point cloud semantic labeling using deep segmentation networks. In Proceedings of the Workshop on 3D Object Retrieval , 3Dor '17, page 17-24, Goslar, DEU, 2017. Eurographics Association. 2

Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. ArXiv , abs/2006.09882, 2020. 3

Mathilde Caron, Hugo Touvron, Ishan Misra, Herve Jegou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , Oct 2021. 3

Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Semantics-augmented set abstraction for point-based 3d object detection. In AAAI , 2022. 3

Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning. 2023. 9

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoderdecoder with atrous separable convolution for semantic image segmentation. In European Conference on Computer Vision , 2018. 9

Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709 , 2020. 3, 17

Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big selfsupervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029 , 2020. 17

Zhe Chen, Jing Zhang, and Dacheng Tao. Progressive lidar adaptation for road detection. IEEE/CAA Journal of Automatica Sinica , 6:693-702, 05 2019. 2

Mingmei Cheng, Le Hui, Jin Xie, and Jian Yang. Sspc-net: Semi-supervised semantic 3d point cloud segmentation network. 2021. 3

Julian Chibane, Francis Engelmann, Tuan Anh Tran, and Gerard Pons-Moll. Box2mask: Weakly supervised 3d semantic instance segmentation using bounding boxes. Computer Vision - ECCV 2022 , pages 681-699, 2022. 3

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 3075-3084, 2019. 2, 7

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proc. Computer Vision and Pattern Recognition (CVPR), IEEE , 2017. 2, 6, 8, 15, 17

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR , 2021. 7

Oren Dovrat, Itai Lang, and Shai Avidan. Learning to sample. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2019. 3

M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision , 111(1):98-136, Jan. 2015. 6

Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Dmt: Dynamic mutual training for semi-supervised learning. Pattern Recognition , 130:108777, Oct 2022. 3, 4, 5

Biao Gao, Yancheng Pan, Chengkun Li, Sibo Geng, and Huijing Zhao. Are we hungry for 3d lidar data for semantic segmentation? a survey of datasets and methods. IEEE Transactions on Intelligent Transportation Systems , 23(7):6063-6081, Jul 2022. 1, 2

Benjamin Graham, Martin Engelcke, and Laurens van der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , Jun 2018. 2

Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. ArXiv , abs/1706.01307, 2017. 2

Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems , volume 17. MIT Press, 2004. 3, 4

Jean-Bastien Grill, Florian Strub, Florent Altch'e, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Ávila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, Bilal Piot, Koray Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. ArXiv , abs/2006.07733, 2020. 17

Meng-Hao Guo, Junxiong Cai, Zheng-Ning Liu, Tai-Jiang Mu, Ralph R. Martin, and Shi-Min Hu. PCT: point cloud transformer. CoRR , abs/2012.09688, 2020. 3

Yulan Guo, Hanyun Wang, Qingyong Hu, Hao Liu, Li Liu, and Mohammed Bennamoun. Deep learning for 3d point clouds: A survey. IEEE transactions on pattern analysis and machine intelligence , 2020. 1

Lei Han, Tian Zheng, Lan Xu, and Lu Fang. Occuseg: Occupancy-aware 3d instance segmentation. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2937-2946, 2020. 2

Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. arXiv preprint arXiv:1911.05722 , 2019. 3

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 770-778, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society. 9

Ruifei He, Jihan Yang, and Xiaojuan Qi. Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6930-6940, 2021. 3

Qingyong Hu, Bo Yang, Guangchi Fang, Yulan Guo, Ales Leonardis, Niki Trigoni, and Andrew Markham. Sqn: Weakly-supervised semantic segmentation of large-scale 3d point clouds. In European Conference on Computer Vision , 2022. 1, 3, 7, 8, 9

Qingyong Hu, Bo Yang, Sheikh Khalid, Wen Xiao, Niki Trigoni, and Andrew Markham. Sensaturban: Learning semantics from urban-scale photogrammetric point clouds. International Journal of Computer Vision , 130(2):316-343, 2022. 1, 2, 6, 9, 15, 17

Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. CoRR , abs/1911.11236, 2019. 3, 7, 8, 15

Zeyu Hu, Mingmin Zhen, Xuyang Bai, Hongbo Fu, and Chiew-Lan Tai. Jsenet: Joint semantic segmentation and edge detection network for 3d point clouds. CoRR , abs/2007.06888, 2020. 3

Zhuo Huang, Zhiyou Zhao, Banghuai Li, and Jungong Han. Lcpformer: Towards effective 3d point cloud analysis via local context propagation in transformers. ArXiv , abs/2210.12755, 2022. 8

Li Jiang, Shaoshuai Shi, Zhuotao Tian, Xin Lai, Shu Liu, Chi-Wing Fu, and Jiaya Jia. Guided point contrastive learning for semi-supervised point cloud semantic segmentation. In ICCV , pages 6403-6412, 10 2021. 3, 7

Yi Kun and Wu Jianxin. Probabilistic End-to-end Noise Correction for Learning with Noisy Labels. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 3, 5, 8

Abhijit Kundu, Xiaoqi Yin, Alireza Fathi, David A. Ross, Brian Brewington, Thomas A. Funkhouser, and Caroline Pantofaru. Virtual multi-view fusion for 3d semantic segmentation. In ECCV , 2020. 2

Hyeokjun Kweon and Kuk-Jin Yoon. Joint learning of 2d-3d weakly supervised semantic segmentation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. 3

Donghyeon Kwon and Suha Kwak. Semi-supervised semantic segmentation with error localization network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 9957-9967, June 2022. 3, 5

Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8490-8499, 2022. 3, 7

Yuxiang Lan, Yachao Zhang, Yanyun Qu, Cong Wang, Chengyang Li, Jia Cai, Yuan Xie, and Zongze Wu. Weakly supervised 3d segmentation via receptive-driven pseudo label consistency and structural consistency. Proceedings of the AAAI Conference on Artificial Intelligence , 37(1):1222-1230, Jun. 2023. 3

L. Landrieu and M. Simonovsky. Large-scale point cloud semantic segmentation with superpoint graphs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4558-4567, 2018. 3

Felix Järemo Lawin, Martin Danelljan, Patrik Tosteberg, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. Deep projective 3d semantic segmentation. Lecture Notes in Computer Science , page 95-107, 2017. 2

Dong-Hyun Lee. Pseudo-label : The simple and efficient semi-supervised learning method for deep neural networks. 2013. 3, 4, 6

Mengtian Li, Yuan Xie, Yunhang Shen, Bo Ke, Ruizhi Qiao, Bo Ren, Shaohui Lin, and Lizhuang Ma. Hybridcr: Weakly-supervised 3d point cloud semantic segmentation via hybrid contrastive regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 14930-14939, June 2022. 1, 3, 6, 7, 8, 17

Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointcnn: Convolution on x-transformed points. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 31. Curran Associates, Inc., 2018. 3, 8

Di Lin, Jifeng Dai, Jiaya Jia, Kaiming He, and Jian Sun. Scribblesup: Scribble-supervised convolutional networks for semantic segmentation. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 3159-3167, 2016. 3

Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro. Perturbed and strict mean teachers for semi-supervised semantic segmentation. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , Jun 2022. 3

Ze Liu, Han Hu, Yue Cao, Zheng Zhang, and Xin Tong. A closer look at local aggregation operators in point cloud analysis. ECCV , 2020. 3, 7, 15

Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103.14030 , 2021. 7

Zhengzhe Liu, Xiaojuan Qi, and Chi-Wing Fu. One thing one click: A self-training approach for weakly supervised 3d semantic segmentation. CoRR , abs/2104.02246, 2021. 3, 7, 10

Tao Lu, Limin Wang, and Gangshan Wu. Cga-net: Category guided aggregation for point cloud semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11693-11702, June 2021. 3

A. Milioto, I. Vizzo, J. Behley, and C. Stachniss. Rangenet++: Fast and accurate lidar semantic segmentation. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 4213-4220, 2019. 2

Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. CoRR , abs/1612.00593, 2016. 3, 7, 8

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. CoRR , abs/1706.02413, 2017. 3, 8

Guocheng Qian, Yuchen Li, Houwen Peng, Jinjie Mai, Hasan Hammoud, Mohamed Elhoseiny, and Bernard Ghanem. Pointnext: Revisiting pointnet++ with improved training and scaling strategies. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. 3, 8

Haibo Qiu, Baosheng Yu, and Dacheng Tao. Collect-and-distribute transformer for 3d point cloud analysis. arXiv preprint arXiv:2306.01257 , 2023. 3

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudolabeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. In International Conference on Learning Representations , 2021. 2, 3

Claude Elwood Shannon. A mathematical theory of communication. The Bell System Technical Journal , 27:379-423, 1948. 4

Hanyu Shi, Jiacheng Wei, Ruibo Li, Fayao Liu, and Guosheng Lin. Weakly supervised segmentation on outdoor 4d point clouds with temporal matching and spatial graph propagation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11830-11839, 2022. 3

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. In Advances in Neural Information Processing Systems , 2017. 5, 6

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 596-608. Curran Associates, Inc., 2020. 3, 8, 9, 15

Hwanjun Song, Minseok Kim, Dongmin Park, Yooju Shin, and Jae-Gil Lee. Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learning Systems , pages 1-19, 2022. 8

Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8479-8489, 2022. 3

Lyne Tchapmi, Christopher Choy, Iro Armeni, JunYoung Gwak, and Silvio Savarese. Segcloud: Semantic segmentation of 3d point clouds. 2017 International Conference on 3D Vision (3DV) , Oct 2017. 2

Hugues Thomas, Charles R. Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas J. Guibas. Kpconv: Flexible and deformable convolution for point clouds. CoRR , abs/1904.08889, 2019. 3, 7, 8, 15

Ozan Unal, Dengxin Dai, and Luc Van Gool. Scribble-supervised lidar semantic segmentation. ArXiv , abs/2203.08537, 2022. 3

Guo-Hua Wang and Jianxin Wu. Repetitive reprediction deep decipher for semi-supervised learning. Proceedings of the AAAI Conference on Artificial Intelligence , 34(04):6170-6177, Apr. 2020. 3

Peng-Shuai Wang, Yang Liu, Yu-Xiao Guo, Chun-Yu Sun, and Xin Tong. O-CNN: Octree-based Convolutional Neural Networks for 3D Shape Analysis. ACM Transactions on Graphics (SIGGRAPH) , 36(4), 2017. 2

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Yue Fan, , Zhen Wu, Jindong Wang, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Bernt Schiele, and Xing Xie. Freematch: Self-adaptive thresholding for semi-supervised learning. 2023. 3, 9

Yuxi Wang, Junran Peng, and Zhaoxiang Zhang. Uncertainty-aware pseudo label refinery for domain adaptive semantic segmentation. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 9072-9081, 2021. 3

Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph CNN for learning on point clouds. CoRR , abs/1801.07829, 2018. 3

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. ArXiv , abs/2203.03884, 2022. 3

Zongji Wang and Feng Lu. Voxsegnet: Volumetric cnns for semantic part segmentation of 3d shapes. IEEE transactions on visualization and computer graphics , 2018. 2

Jiacheng Wei, Guosheng Lin, Kim-Hui Yap, Tzu-Yi Hung, and Lihua Xie. Multi-path region mining for weakly supervised 3d semantic segmentation on point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4384-4393, 2020. 3

Wenxuan Wu, Zhongang Qi, and Fuxin Li. Pointconv: Deep convolutional networks on 3d point clouds. CoRR , abs/1811.07246, 2018. 3

Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition-based pooling. In NeurIPS , 2022. 3

Zhonghua Wu, Yicheng Wu, Guosheng Lin, and Jianfei Cai. Reliability-adaptive consistency regularization for weakly-supervised point cloud segmentation, 2023. 3

Zhonghua Wu, Yicheng Wu, Guosheng Lin, Jianfei Cai, and Chen Qian. Dual adaptive transformations for weakly supervised point cloud segmentation. arXiv preprint arXiv:2207.09084 , 2022. 3

Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. arXiv preprint arXiv:1911.04252 , 2019. 3

Y. Xie, J. Tian, and X. X. Zhu. Linking points with labels in 3d: A review of point cloud semantic segmentation. IEEE Geoscience and Remote Sensing Magazine , 8(4):38-59, 2020. 1

Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. ArXiv , abs/2004.01803, 2020. 2

Xun Xu and Gim Hee Lee. Weakly supervised semantic point cloud segmentation: Towards 10x fewer labels. In CVPR , 2020. 1, 3, 6, 7

Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. CoRR , abs/2003.00492, 2020. 3

Cheng-Kun Yang, Ji-Jia Wu, Kai-Syun Chen, Yung-Yu Chuang, and Yen-Yu Lin. An mil-derived transformer for weakly supervised point cloud segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 11830-11839, June 2022. 1, 3, 6, 7, 8, 9

Jiancheng Yang, Qiang Zhang, Bingbing Ni, Linguo Li, Jinxian Liu, Mengdie Zhou, and Qi Tian. Modeling point clouds with self-attention and gumbel subset sampling. CoRR , abs/1904.03375, 2019. 3

Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. In CVPR , 2023. 3, 9

Shuquan Ye, Dongdong Chen, Songfang Han, and Jing Liao. Learning with noisy labels for robust point cloud segmentation. International Conference on Computer Vision , 2021. 8

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. 2021. 3

Pan Zhang, Bo Zhang, Ting Zhang, Dong Chen, and Fang Wen. Robust mutual learning for semisupervised semantic segmentation, 2021. 3, 4, 5

Xiaolong Zhang, Zuqiang Su, Xiaolin Hu, Yan Han, and Shuxian Wang. Semisupervised momentum prototype network for gearbox fault diagnosis under limited labeled samples. IEEE Transactions on Industrial Informatics , 18(9):6203-6213, 2022. 6

Yachao Zhang, Zonghao Li, Yuan Xie, Yanyun Qu, Cuihua Li, and Tao Mei. Weakly supervised semantic segmentation for large-scale point cloud. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 3421-3429, 2021. 1, 3, 5, 6, 7, 8, 10

Yachao Zhang, Yanyun Qu, Yuan Xie, Zonghao Li, Shanshan Zheng, and Cuihua Li. Perturbed selfdistillation: Weakly supervised large-scale point cloud semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 15520-15528, 2021. 3, 7, 8

Yan Zhang, Wenhan Zhao, Bo Sun, Ying Zhang, and Wen Wen. Point cloud upsampling algorithm: A systematic review. Algorithms , 15(4), 2022. 3

Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun. Point transformer, 2021. 3, 7, 8, 15

Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision , 129(4):1106-1120, Jan 2021. 3

Bolei Zhou, Aditya Khosla, Àgata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2921-2929, 2016. 3

Yanning Zhou, Hang Xu, Wei Zhang, Bin Gao, and Pheng-Ann Heng. C3-semiseg: Contrastive semisupervised segmentation via cross-set learning and dynamic class-balancing. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) , pages 7016-7025, 2021. 2, 3, 4

A Supplementary: Introduction

In this supplementary material, we provide more details regarding implementation details in Appendix B, more analysis of ERDA in Appendix C, full experimental results in Appendix D, and studies on parameters in Appendix E.

B Supplementary: Implementation and Training Details

For the RandLA-Net [34] and CloserLook3D [51] baselines, we follow the instructions in their released code for training and evaluation, which are here (RandLA-Net) and here (CloserLook3D), respectively. Especially, in CloserLook3D[51], there are several local aggregation operations and we use the 'Pseudo Grid' (KPConv-like) one, which provides a neat re-implementation of the popular KPConv [68] network (rigid version). For point transformer (PT) [97], we follow their paper and the instructions in the code base that claims to have the official code (here). For FixMatch [64], we use the publicly available implementation here.

Our code and pre-trained models will be released.

C Supplementary: Delving into ERDA with More Analysis

Following the discussion in Sec. 3, we study the impact of entropy regularization as well as different distance measurements from the perspective of gradient updates.

.

In particular, we study the gradient on the score of the i -th class i.e., s i , and denote it as g i = ∂L p ∂s i Given that ∂p j ∂s i = 1 ( i = j ) p i -p i p j , we have g i = p i ∑ j p j ( ∂L p ∂p i -∂L p ∂p j ) . As shown in Tab. 8, we demonstrate the gradient update ∆ = -g i under different situations.

In addition to the analysis in Sec. 3.2, we find that, when q is certain, i.e., q approaching a one-hot vector, the update of our choice KL ( p || q ) would approach the infinity. We note that this could be hardly encountered since q is typically also the output of a softmax function. Instead, we would rather regard it as a benefit because it would generate effective supervision on those model predictions with high certainty, and the problem of gradient explosion could also be prevented by common operations such as gradient clipping.

In Fig. 4, we provide visualization for a more intuitive understanding on the impact of different formulations for L DA as well as their combination with L ER . Specifically, we consider a simplified case of binary classification and visualize their gradient updates when λ takes different values. We also visualize the gradient updates of L ER . By comparing the gradient updates, we observe that only KL ( p || q ) with λ = 1 can achieve small updates when q is close to uniform ( q = 1 2 under the binary case), and that there is a close-0 plateau as indicated by the sparse contour lines.

Additionally, we also find that, when increasing the λ , all distances, except the KL ( p || q ) , are modulated to be similar to the updates of having L ER alone; whereas KL ( p || q ) can still produce effective updates, which may indicate that KL ( p || q ) is more robust to the λ .

D Supplementary: Full Results

We provide full results for the experiments reported in the main paper. For S3DIS [2], we provide the full results of S3DIS with 6-fold cross-validation in Tab. 10. For ScanNet [16] and SensatUrban [33], we evaluate on their online test servers, which are here and here, and provide the full results in Tab. 11 and Tab. 12, respectively.

̸

̸

𝑲𝑳(𝐩||𝐪)

𝑱𝑺(𝐩, 𝐪)

𝑲𝑳(𝐪||𝐩)

𝑴𝑺𝑬(𝐩,𝐪)































































































































































(a)

λ

= 0



































































































































































(b)

λ

= 1



































































































































































(c)

λ

= 2















































(d) Gradient updates of

L

ER

̸

̸

E Supplementary: Ablations and Parameter Study

We further study the hyper-parameters involved in the implementation of ERDA with the prototypical pseudo-label generation, including loss weight α , momentum coefficient m , and the use of projection network. As shown in Tab. 9, the proposed method acquires decent performance (mIoUs are all > 65 and mostly > 66 ) in a wide range of different hyper-parameter settings, compared with its fully-supervised baseline (64.7 mIoU) and previous state-of-the-art performance (65.3 mIoU by HybridCR [47]).

Additionally, we suggest that the projection network could be effective in facilitating the ERDA learning, which can be learned to specialize in the pseudo-label generation task. This could also be related to the advances in contrastive learning. Many works [10, 11, 25] suggest that a further projection on feature representation can largely boost the performance because such projection decouples the learned features from the pretext task. We share a similar motivation in decoupling the features for ERDA learning on the pseudo-label generation task from the features for the segmentation task.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission proposes a semi-supervized pseudo-label-based method for 3D point cloud segmentation method with sparse label annotations. Instead of thresholding on the confidence of pseudo labels, the authors propose to use all unlabelled points and encourage high-confidence pseudo labels by regularizing with their entropy. The authors propose a theoretical analysis of their approach and experiments where they show that their method improves the low-label regime for several backbone and several datasets.\n\nSTRENGTHS:\n- the authors propose a theoretical analysis with some insight\n\n- The experimental results are convincing\n\n- The experiments are extensive: 4 datasets and 4 baselines\n\nWEAKNESSES:\n- The entire ER+DA analysis leads to using a loss which is the cross-entropy between the pseudo labels and the prediction, which is already sensible and does not need to be seen as a special case of a more general setting that is never explored anyway. The fact that KL(p,q)+H(q)=CE(p,q) was known and did not need two full pages of motivation and equations (some parts are interesting such as the gradient limits, but would be better suited in the appendix). Equation (6) is completely logical and can be used directly.\n\n- On the other hand, the most interesting part of the paper is the pseudo-label generation hidden in Appendix B, which, as far as the reviewer knows, is novel. The reviewer thinks that the non-backpropagable momentum update of the class prototypes used for pseudo labels makes this method work. Without this, if p could be learned along q, then equation (6) would be meaningless as p=q is a trivial solution, and the pseudo labels would not help at all. The paper would need to be rewritten to highlight this hidden mechanism, brushed in the main paper in a single sentence, and at the method's core. \n\n- Equation (6) reduces to classic cross entropy when all points are labelled, and yet the proposed methods improve the backbones in the fully supervised setting?\n\n- The writing is subpar and lacks rigour. This leads to imprecise or even outright false statements at the core of the motivation. For example, the authors state that regularizing by the entropy decreases the \"noise\" in pseudo-labels when it actually encourages confident distributions and does not affect noise. There are many vague and uninformative sentences, some dealing with critical aspects of the paper. Some variables are also not introduced. The article remains overall understandable with the suppmat open in another tab.\n\nQUESTIONS:\nOverall, the reviewer believes a very good idea is hidden in the paper; but, it takes a lot of effort to see through the subpar writing, imprecise statements, core ideas hidden in the appendix, and confusing notations. \nThe reviewer also believes the authors focused on the wrong part of their contribution.\nAs of now, the paper is not publishable. But if the authors put the work into improving their clarity and rigour, and focused on the important part of their contribution, this could become an impactful paper. The amount of effort might be too much for a rebuttal. \n\nQ1) Equation (6) reduces to classic cross entropy when all points are labelled, and yet the proposed methods improve the backbones in the fully supervised setting?\n\nS1) Swap the derivation of (6) and the pseudo-label generation between the appendix and the main paper.\n\nS2) Add detailed proofs of Table 1 in the appendix\n\nS3) Remove the notion that lower entropy = less noise\n\nLIMITATIONS:\nok", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes two losses for point cloud semantic segmentation. The first one is Entropy Regularization (ER) loss, which makes the pseudo-labels have low entropy and thereby be confident (like one-hot vectors). The other one is Distribution Alignment (DA) loss, which is a KL divergence between the pseudo-labels and the predictions. Extensive experiments show that the combination of these two losses is not only beneficial in a weakly supervised setting but also in a fully supervised setting.\n\nSTRENGTHS:\n1. The proposed method is extremely simple, but theoretically sounding and effective in practice.\n2. Extensive experiments strongly support the benefit of the proposed method.\n3. Overall presentation is great. Especially, section 3.2 is notably helpful to understand the working logic behind the proposed method.\n\nWEAKNESSES:\n1. Could the authors provide a visualization of the (1) entropy map, (2) pseudo-label, and (3) KD loss map of the point cloud, and how they change as training proceeds? The figure would be extremely helpful to elucidate how the proposed method works.\n\n2. Actually, it was quite surprising for me that the proposed method brings performance gain even in a fully-supervised setting. I think more discussion is required for this. Also, it would be great if the authors could provide an ablation study of Table 7(a) in a fully-supervised setting. I expect that DA would be much more beneficial compared to ER in this case, but not sure of course.\n\n3. I understand why the authors refrained from comparing with some super-voxel-based approaches for a fair comparison. However, using a super-voxel partition is one of the widely used settings in 3D semantic segmentation. Hence, to clearly demonstrate the superiority of the proposed method, it would be much better if the authors could provide the performance of ERDA using a super-voxel partition.\n\n4. Some related works are missing. \n\nJoint Learning of 2D-3D Weakly Supervised Semantic Segmentation (NeurIPS 2022)\n\nBox2Mask: Weakly Supervised 3D Semantic Instance Segmentation Using Bounding Boxes (ECCV 2022)\n\nWeakly Supervised 3D Segmentation via Receptive-Driven Pseudo Label Consistency and Structural Consistency (AAAI 2023)\n\nQUESTIONS:\nThe results of the experiments on 2D semi-supervised semantic segmentation (SSSS) are interesting. I think the less significant improvement is mainly due to the gap between the weakly-supervised 3D semantic segmentation task and the SSSS task. Maybe the proposed ERDA can make benefit from 2D WSSS (using some points or scribbles).\n\nLIMITATIONS:\nThe limitations are properly mentioned in the paper (section 5).", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper proposes a novel learning strategy to regularize the generated pseudo-labels and narrow the gaps between pseudo-labels and model predictions. It introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks. The approach can better leverage unlabeled data points and achieves state-of-the-art performance under different settings.

STRENGTHS:
1. The proposed losses can work with other frameworks to consistently boost their performances. It can potentially achieve better performance with future stronger approaches. 
2. The approach can improve many existing approaches by a significant margin.

WEAKNESSES:
I think the motivation and design of the proposed losses needs further analysis. 
1. Entropy Regularization loss: does it filter out some high-frequency predictions? How to balance the effects of filtering high-frequency components and removing noise? 
2. Distribution Alignment loss: 
(a) Does it always improve the performance to encourage the consistency of prediction and pseudo labels? 
(b) Are the pseudo labels derived from network predictions? If so, they should be the similar thing. In what cases they tend to be similar and in what cases they are different? This part seems not very clearly presented.

QUESTIONS:
See above.

LIMITATIONS:
See above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers the task of weakly supervised 3D scene semantic segmentation, where only a limited number of points in each training scene are given labels. Assuming a baseline system that operates within a pseudo-label paradigm, the paper proposes a new set of regularizing loss terms, that aim to (1) reduce pseudo-label entropy and (2) align the distribution of the pseudo-labels and network predictions. Under a default weighting strategy, these terms simplify into cross entropy from the pseudo-labels to the network predictions. Under a wide variety of experimental settings, the paper shows that incorporating this term leads to improved performance for 3d scene semantic segmentation, regardless of the level of supervision.\n\nSTRENGTHS:\nThis is a well-written, clear, and compelling paper on an important topic of interest to the community. While the introduced technique is not terribly complex, its benefits are well-justified, and the paper provides substantial analysis to support its inclusion: investigating how gradients from this loss term behave under different prediction settings, and why those gradients align with desirable properties. Further, the paper provides extensive ablation experiments that experimentally support this analysis, and show that all of its components lead to improved performance on the domains under investigation.\n\nThe strongest point for the paper is in its thorough and overwhelmingly positive experimental results. For multiple datasets of 3d scenes (all standard), under multiple levels of supervision, adding this loss to an array of baseline models always improves performance, and outperforms previous state-of-the-art models on competitive benchmarks. Substantial improvements are observed when labels are severely limited, and even under fully supervised settings, including this loss term is helpful. From the presented evidence, it seems likely this term should be widely useful for this task and domain in future work, as it presents robustly strong performance under a myriad of framings and settings.\n\nWEAKNESSES:\nMy biggest outstanding question is to what extent this technique can offer benefits for other domains? In its formulation, there is nothing specific to 3D scene segmentation, so ostensibly it could be generally useful for other weakly supervised domains that employ pseudo-labels. Some evidence is provided that it can transfer to image segmentation, but it would also be interesting to consider domains like 3D shape segmentation. The initial results (on domains other than 3d scenes) provided by the paper are encouraging, but a more thorough analysis would of course strengthen the paper, and likely dramatically improve the reach/impact of the contribution.\nRelatedly, I would like to see more analysis / discussion about under what situations this term is helpful? Is it always beneficial to include such a term (no matter the domain / task). For instance, I could imagine that when the initial pseudo-labeling mechanism is highly inaccurate, this term might actually be harmful for learning. For 3D scene segmentation, my prior is that pseudo-labeling techniques are largely successful because strong locality cues in this domain can often be used to propagate labels to nearby unlabeled points, with a relative high degree of confidence; so the quality of initial pseudo-labels for 3D scene segmentation might be higher than would be expected for other domains of interest. It would be interesting to consider the effect that the “goodness” of the pseudo-labeling mechanism has on the final model performance, which could potentially be evaluated in a synthetically designed experiment that introduces “corruption” (at varying levels) into the distributions produced by the pseudo-labeling network. A deeper understanding of how the various components of the system interact with the added loss terms would be beneficial, and may give insight into what other domains and systems may benefit from this insights this paper provides. \n\nMinor:\n\nThe formatting of table 2 can be improved. The read highlights are distracting, and largely unneeded as they overstate information. Consider replacing the red text coloring with italics, or better yet, marking only the columns that the baseline does not get improved with the added loss term.\n\nQUESTIONS:\nPerhaps the most surprising result in the paper is that the method improves baselines, even under full label supervision. While I don’t doubt that trend is “real”, as the experimentation seems robust and well-designed, I was not quite satisfied by the explanation given to explain the result on lines 250-254. Is this explanation claiming that the “gt” labels have noise, so using ERD, which is “noise-aware”, can help regulate and remove the noise present in them? If so, this seems like a testable hypothesis (e.g. analyzing differences between pseudo-label predictions and gt label predictions). While I don’t think its required to have a compelte explanation for this phenomenon, the paper should either clarify the explanation here, or simply say that it is unknown why ERDA offers benefits in this paradigm, and to fully understand it would require further study.\n\nLIMITATIONS:\n\"Limited\" limitations are given, see weaknesses section as to other potential limitations that should be explored.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions, which introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks, resulting in an ERDA learning strategy.\n\nSTRENGTHS:\nThis paper solve an interesting problem, and the experimental results support the conclusions.\n\nWEAKNESSES:\n1.There exists some confusions in Figure 1:\na)For (a), there are generally two ways of generating pseudo-labels based on self-training weakly supervised methods. One way is to input both the sample and its augmented version into two different or shared-weight networks (with gradient updating); the other way is to input them into the student network and the teacher network updated via EMA (with only the student network being updated). I think the author's general description of sparse pseudo-labels in (a) is inadequate, which makes it difficult to establish a connection with (b) and understand the essential differences between them.\nb)I think the author's naming of (b) is inappropriate. It only optimizes the pseudo-labels p and predictions q simultaneously, without reflecting the \"dense\" aspect.\n2.I appreciate the author's theoretical analysis of entropy regularization and distribution alignment, as well as the evaluation of different loss combinations via formula derivation. However, when I saw Line 160 and Table 7, I wondered if when lambda=1, the best result is achieved, and at this point, the ERDA loss simplifies to a single cross-entropy-based loss that optimizes both p and q. My question is, to my knowledge, I don't think only optimizing p for gradient update at the same time can bring such a high performance gain, and I speculate that the diversity of different perturbations as input enables the model to learn the geometric invariance of features, and the ERDA loss is just the icing on the cake.\n3.In Line 207, the author uses a prototypical pseudo-label generation process due to its popularity and simplicity. However, in Table 7(a), the baseline with pseudo-labels already reaches 63.3%. I would like to see the difference between the results of prototypical pseudo-labels and plainest pseudo-labels.\n4.I am a little confused about the author's approach under the setting of “fully”. Generally, the PL strategy is applied to unlabeled data, and then leverage prototypes, perturbation or contrastive learning to improve the model's robustness to the unlabeled data. However, in the fully experiment, the author applies ERDA to labeled data, so how can we talk about pseudo-labeling? There is also no gradient update for p.\n5.Table 1 is difficult to understand, and I think this part is the core of the method. Therefore, the author needs to provide a detailed description of it.\n\nQUESTIONS:\nSee section weakness.\n\nLIMITATIONS:\nThere are some issues not explained clearly,  see Section Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}