SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Liyao Tang $^{1}$ , Zhe Chen $^{2}$ , Shanshan Zhao $^{1}$ , Chaoyue Wang $^{1}$ , Dacheng Tao $^{1}$
<sup>1</sup> The University of Sydney, Australia <sup>2</sup> La Trobe University, Australia
ltan9687@uni.sydney.edu.au, zhe.chen@latrobe.edu.au
chaoyue.wang@outlook.com, {sshan.zhao00, dacheng.tao}@gmail.com
Pseudo-labels are widely employed in weakly supervised 3D segmentation tasks where only sparse ground-truth labels are available for learning. Existing methods often rely on empirical label selection strategies, such as confidence thresholding, to generate beneficial pseudo-labels for model training. This approach may, however, hinder the comprehensive exploitation of unlabeled data points. We hypothesize that this selective usage arises from the noise in pseudo-labels generated on unlabeled data. The noise in pseudo-labels may result in significant discrepancies between pseudo-labels and model predictions, thus confusing and affecting the model training greatly. To address this issue, we propose a novel learning strategy to regularize the generated pseudo-labels and effectively narrow the gaps between pseudo-labels and model predictions. More specifically, our method introduces an Entropy Regularization loss and a Distribution Alignment loss for weakly supervised learning in 3D segmentation tasks, resulting in an ERDA learning strategy. Interestingly, by using KL distance to formulate the distribution alignment loss, it reduces to a deceptively simple cross-entropy-based loss which optimizes both the pseudo-label generation network and the 3D segmentation network simultaneously. Despite the simplicity, our method promisingly improves the performance. We validate the effectiveness through extensive experiments on various baselines and large-scale datasets. Results show that ERDA effectively enables the effective usage of all unlabeled data points for learning and achieves state-of-the-art performance under different settings. Remarkably, our method can outperform fully-supervised baselines using only  $1\%$  of true annotations. Code and model will be made publicly available at https://github.com/LiyaoTang/ERDA.
Point cloud semantic segmentation is a crucial task for 3D scene understanding and has various applications [27, 83], such as autonomous driving, unmanned aerial vehicles, and augmented reality. Current state-of-the-art approaches heavily rely on large-scale and densely annotated 3D datasets, which are costly to obtain [21, 33]. To avoid demanding exhaustive annotation, weakly-supervised point cloud segmentation has emerged as a promising alternative. It aims to leverage a small set of annotated points while leaving the majority of points unlabeled in a large point cloud dataset for learning. Although current weakly supervised methods can offer a practical and cost-effective way to perform point cloud segmentation, their performance is still sub-optimal compared to fully-supervised approaches.
During the exploration of weak supervision [85], a significant challenge is the insufficient training signals provided by the highly sparse labels. To tackle this issue, pseudo-labeling methods [85, 94, 32] have been proposed, which leverage predictions on unlabeled points as labels to facilitate the learning of the segmentation network. Despite some promising results, these pseudo-label methods have been outperformed by some recent methods based on consistency regularization [47, 87]. We tend to attribute this to the use of label selection on pseudo-labels, such as confidence thresholding, which
Figure 1. While existing pseudo-labels (a) are limited in the exploitation of unlabeled points, ERDA (b) simultaneously optimizes the pseudo-labels  $\mathbf{p}$  and predictions  $\mathbf{q}$  taking the same and simple form of cross-entropy. By reducing the noise via entropy regularization and bridging their distributional discrepancies, ERDA produces informative pseudo-labels that neglect the need for label selection. As in (c), it thus enables the model to consistently benefit from more pseudo-labels, surpasses other methods and its fully-supervised baseline, and can be extended to advance the fully-supervised performance.
could lead to unlabeled points being wasted and under-explored. We hypothesize that the need for label selection arises from the low-confidence pseudo-labels assigned to unlabeled points, which are known for their noises [21] and potential unintended bias [60, 100]. These less reliable and noisy pseudo-labels could contribute to discrepancies between the pseudo-labels and the model predictions, which might confuse and impede the learning process to a great extent.
By addressing the above problem for label selection in weakly supervised 3D segmentation, we propose a novel learning-based approach in this study. Our method aims to leverage the information from all unlabeled points by mitigating the negative effects of the noisy pseudo-labels and the distributional discrepancy.
Specifically, we introduce two learning objectives for the pseudo-label generation process. Firstly, we introduce an entropy regularization (ER) objective to reduce the noise and uncertainty in the pseudo-labels. This regularization promotes more informative, reliable, and confident pseudo-labels, which helps alleviate the limitations of noisy and uncertain pseudo-labels. Secondly, we propose a distribution alignment (DA) loss that minimizes statistical distances between pseudo-labels and model predictions. This ensures that the distribution of generated pseudo-labels remains close to the distribution of model predictions when regularizing their entropy.
In particular, we discover that formulating the distribution alignment loss using KL distance enables a simplification of our method into a cross-entropy-style learning objective that optimizes both the pseudo-label generator and the 3D segmentation network simultaneously. This makes our method straightforward to implement and apply. By integrating the entropy regularization and distribution alignment, we achieve the ERDA learning strategy, as shown in Fig. 1.
Empirically, we comprehensively experiment with three baselines and different weak supervision settings, including  $0.02\%$  (1-point, or 1pt),  $1\%$ , and  $10\%$ . Despite its concise design, our ERDA outperforms existing weakly-supervised methods on large-scale point cloud datasets such as S3DIS [2], ScanNet [16], and SensatUrban [33]. Notably, our ERDA can surpass the fully-supervised baselines using only  $1\%$  labels, demonstrating its significant effectiveness in leveraging pseudo-labels. Furthermore, we validate the scalability of our method by successfully generalizing it to more other settings, which illustrates the benefits of utilizing dense pseudo-label supervision with ERDA.
Point cloud segmentation. Point cloud semantic segmentation aims to assign semantic labels to 3D points. The cutting-edge methods are deep-learning-based and can be classified into projection-based and point-based approaches. Projection-based methods project 3D points to grid-like structures, such as 2D image [84, 55, 39, 12, 4, 45] or 3D voxels [15, 71, 67, 28, 22, 23, 76]. Alternatively, point-based
methods directly operate on 3D points [56, 57]. Recent efforts have focused on novel modules and backbones to enhance point features, such as 3D convolution [3, 48, 78, 68, 51, 58], attentions [34, 26, 97, 79, 42, 59], graph-based methods [74, 44], and other modules such as sampling [18, 86, 88, 7] and post-processing [54, 35, 66]. Although these methods have made significant progress, they rely on large-scale datasets with point-wise annotation and struggle with few labels [85]. To address the demanding requirement of point-wise annotation, our work explores weakly-supervised learning for 3D point cloud segmentation.
Weakly-supervised point cloud segmentation. Compared to weakly-supervised 2D image segmentation [99, 49, 75, 1, 64], weakly-supervised 3D point cloud segmentation is less explored. In general, weakly-supervised 3D segmentation task focuses on highly sparse labels: only a few scattered points are annotated in large point cloud scenes. Xu and Lee [85] first propose to use  $10x$  fewer labels to achieve performance on par with a fully-supervised point cloud segmentation model. Later studies have explored more advanced ways to exploit different forms of weak supervision [77, 14, 40] and human annotations [53, 69]. Recent methods tend to introduce perturbed self-distillation [95], consistency regularization [85, 62, 80, 81, 43], and leverage self-supervised learning [62, 37, 47, 87] based on contrastive learning [29, 10]. Pseudo-labels are another approach to leverage unlabeled data, with methods such as pre-training networks on colorization tasks [94], using iterative training [32], employing separate networks to iterate between learning pseudo-labels and training 3D segmentation networks [53], or using super-point graph [44] with graph attentional module to propagate the limited labels over super-points [13]. However, these existing methods often require expensive training due to hand-crafted 3D data augmentations [95, 87, 80, 81], iterative training [53, 32], or additional modules [87, 32], complicating the adaptation of backbone models from fully-supervised to weakly-supervised learning. In contrast, our work aims to achieve effective weakly supervised learning for the 3D segmentation task with straightforward motivations and simple implementation.
Pseudo-label refinement. Pseudo-labeling [46], a versatile method for entropy minimization [24], has been extensively studied in various tasks, including semi-supervised 2D classification [82, 60], segmentation [64, 89], and domain adaptation [98, 73]. To generate high-quality supervision, various label selection strategies have been proposed based on learning status [72, 91, 20], label uncertainty [60, 98, 73, 50], class balancing [100], and data augmentations [64, 89, 100]. Our method is most closely related to the works addressing bias in supervision, where mutual learning [20, 70, 92] and distribution alignment [100, 31, 41] have been discussed. However, these works typically focus on class imbalance [100, 31] and rely on iterative training [70, 20, 92, 41], label selection [20, 31], and strong data augmentations [100, 31], which might not be directly applicable to 3D point clouds. For instance, common image augmentations [64] like cropping and resizing may translate to point cloud upsampling [96], which remains an open question in the related research area. Rather than introducing complicated mechanisms, we argue that proper regularization on pseudo-labels and its alignment with model prediction can provide significant benefits using a very concise learning approach designed for the weakly supervised 3D point cloud segmentation task.
Besides, it is shown that the data augmentations and repeated training in mutual learning [70, 38] are important to avoid the feature collapse, i.e., the resulting pseudo-labels being uniform or the same as model predictions. We suspect the cause may originate from the entropy term in their use of raw statistical distance by empirical results, which potentially matches the pseudo-labels to noisy and confusing model prediction, as would be discussed in Sec. 3.2. Moreover, in self-supervised learning based on clustering [5] and distillation [6], it has also been shown that it would lead to feature collapse if matching to a cluster assignment or teacher output of a close-uniform distribution with high entropy, which agrees with the intuition in our ER term.
3.1 Formulation of ERDA
As previously mentioned, we propose the ERDA approach to alleviate noise in the generated pseudo-labels and reduce the distribution gaps between them and the segmentation network predictions. In general, our ERDA introduces two loss functions, including the entropy regularization loss and the distribution alignment loss for the learning on pseudo-labels. We denote the two loss functions as  $L_{ER}$  and  $L_{DA}$ , respectively. Then, we have the overall loss of ERDA as follows:
$$
L _ {p} = \lambda L _ {E R} + L _ {D A}, \tag {1}
$$
where the  $\lambda > 0$  modulates the entropy regularization which is similar to the studies [46, 24].
Before detailing the formulation of  $L_{ER}$  and  $L_{DA}$ , we first introduce the notation. While the losses are calculated over all unlabeled points, we focus on one single unlabeled point for ease of discussion. We denote the pseudo-label assigned to this unlabeled point as  $\mathbf{p}$  and the corresponding segmentation network prediction as  $\mathbf{q}$ . Each  $\mathbf{p}$  and  $\mathbf{q}$  is a 1D vector representing the probability over classes.
Entropy Regularization loss. We hypothesize that the quality of pseudo-labels can be hindered by noise, which in turn affects model learning. Specifically, we consider that the pseudo-label could be more susceptible to containing noise when it fails to provide a confident pseudo-labeling result, which leads to the presence of a high-entropy distribution in  $\mathbf{p}$ .
To mitigate this, for the  $\mathbf{p}$ , we propose to reduce its noise level by minimizing its Shannon entropy, which also encourages a more informative labeling result [61]. Therefore, we have:
$$
L _ {E R} = H (\mathbf {p}), \tag {2}
$$
where  $H(\mathbf{p}) = \sum_{i} - p_{i}\log p_{i}$  and  $i$  iterates over the vector. By minimizing the entropy of the pseudo-label as defined above, we promote more confident labeling results to help resist noise in the labeling process<sup>1</sup>.
Distribution Alignment loss. In addition to the noise in pseudo-labels, we propose that significant discrepancies between the pseudo-labels and the segmentation network predictions could also confuse the learning process and lead to unreliable segmentation results. In general, the discrepancies can stem from multiple sources, including the noise-induced unreliability of pseudo-labels, differences between labeled and unlabeled data [100], and variations in pseudo-labeling methods and segmentation methods [92, 20]. Although entropy regularization could mitigate the impact of noise in pseudolabels, significant discrepancies may still persist between the pseudo-labels and the predictions of the segmentation network. To mitigate this issue, we propose that the pseudo-labels and network can be jointly optimized to narrow such discrepancies, making generated pseudo-labels not diverge too far from the segmentation predictions. Therefore, we introduce the distribution alignment loss.
To properly define the distribution alignment loss  $(L_{DA})$ , we measure the KL divergence between the pseudo-labels  $(\mathbf{p})$  and the segmentation network predictions  $(\mathbf{q})$  and aim to minimize this divergence. Specifically, we define the distribution alignment loss as follows:
$$
L _ {D A} = K L (\mathbf {p} | | \mathbf {q}), \tag {3}
$$
where  $KL(\mathbf{p}||\mathbf{q})$  refers to the KL divergence. Using the above formulation has several benefits. For example, the KL divergence can simplify the overall loss  $L_{p}$  into a deceptively simple form that demonstrates desirable properties and also performs better than other distance measurements. More details will be presented in the following sections.
Simplified ERDA. With the  $L_{ER}$  and  $L_{DA}$  formulated as above, given that  $KL(\mathbf{p}||\mathbf{q}) = H(\mathbf{p},\mathbf{q}) - H(\mathbf{p})$  where  $H(\mathbf{p},\mathbf{q})$  is the cross entropy between  $\mathbf{p}$  and  $\mathbf{q}$ , we can have a simplified ERDA formulation as:
$$
L _ {p} = H (\mathbf {p}, \mathbf {q}) + (\lambda - 1) H (\mathbf {p}). \tag {4}
$$
In particular, when  $\lambda = 1$ , we obtain the final ERDA loss:
$$
L _ {p} = H (\mathbf {p}, \mathbf {q}) = \sum_ {i} - p _ {i} \log q _ {i} \tag {5}
$$
The above simplified ERDA loss describes that the entropy regularization loss and distribution alignment loss can be represented by a single cross-entropy-based loss that optimizes both  $\mathbf{p}$  and  $\mathbf{q}$ .
We would like to emphasize that Eq. (5) is distinct from the conventional cross-entropy loss. The conventional cross-entropy loss utilizes a fixed label and only optimizes the term within the logarithm function, whereas the proposed loss in Eq. (5) optimizes both  $\mathbf{p}$  and  $\mathbf{q}$  simultaneously.
<sup>1</sup>We note that our entropy regularization aims for entropy minimization on pseudo-labels; and we consider noise as the uncertain predictions by the pseudo-labels, instead of incorrect predictions.
2We would justify the choice of  $\lambda$  in the following Sec. 3.2 as well as Sec. 4.3
LDA	KL(p||q)	KL(q||p)	JS(p,q)	MSE(p,q)
Lp	H(p,q) - (1 - λ)H(p)	H(q,p) - H(q) + λH(p)	H( p+ q/2) - (1/2 - λ)H(p) - 1/2H(q)	1/2 ∑i (pi - qi)^2 + λH(p)
S1	0	qi - 1k=i	0	0
S2	(λ - 1)p_i ∑j pj log pi/p_j	1/K - pi + λpi ∑j pj log pi/p_j	pi ∑j≠i pj( 1/2 log Kpj + 1 / Kpj + 1 + (λ - 1/2) log pi/p_j )	- pi^2 + pi ∑j pj^2 + λpi ∑j pj log pi/p_j
Table 1. The formulation of  $L_{p}$  using different functions to formulate  $L_{DA}$ . We study the gradient update on  $s_i$ , i.e.,  $-\frac{\partial L_p}{\partial s_i}$  under different situations. S1: update given confident pseudo-label,  $\mathbf{p}$  being one-hot with  $\exists p_k \in \mathbf{p}, p_k \to 1$ . S2: update given confusing prediction,  $\mathbf{q}$  being uniform with  $q_1 = \ldots = q_K = \frac{1}{K}$ . More analysis as well as visualization can be found in the Sec. 3.2 and the supplementary Appendix C.
3.2 Delving into the Benefits of ERDA
To formulate the distribution alignment loss, different functions can be employed to measure the differences between  $\mathbf{p}$  and  $\mathbf{q}$ . In addition to the KL divergence, there are other distance measurements like mean squared error (MSE) or Jensen-Shannon (JS) divergence for replacement. Although many mutual learning methods [20, 92, 41, 38] have proven the effectiveness of KL divergence, a detailed comparison of KL divergence against other measurements is currently lacking in the literature. In this section, under the proposed ERDA learning framework, we show by comparison that  $KL(\mathbf{p}||\mathbf{q})$  is a better choice and ER is necessary for weakly-supervised 3D segmentation.
To examine the characteristics of different distance measurements, including  $KL(\mathbf{p}||\mathbf{q})$ ,  $KL(\mathbf{q}||\mathbf{p})$ ,  $JS(\mathbf{p}||\mathbf{q})$ , and  $MSE(\mathbf{p}||\mathbf{q})$ , we investigate the form of our ERDA loss  $L_{p}$  and its impact on the learning for pseudo-label generation network given two situations during training.
More formally, we shall assume a total of  $K$  classes and define that a pseudo-label  $\mathbf{p} = [p_1,\dots,p_K]$  is based on the confidence scores  $\mathbf{s} = [s_1,\dots,s_K]$ , and that  $\mathbf{p} = \mathrm{softmax}(\mathbf{s})$ . Similarly, we have a segmentation network prediction  $\mathbf{q} = [q_1,\dots,q_K]$  for the same point. We re-write the ERDA loss  $L_{p}$  in various forms and investigate the learning from the perspective of gradient update, as in Tab. 1.
Situation 1: Gradient update given confident pseudo-label  $\mathbf{p}$ . We first specifically study the case when  $\mathbf{p}$  is very certain and confident, i.e.,  $\mathbf{p}$  approaching a one-hot vector. As in Tab. 1, most distances yield the desired zero gradients, which thus retain the information of a confident and reliable  $\mathbf{p}$ . In this situation, however, the  $KL(\mathbf{q}||\mathbf{p})$ , rather than  $KL(\mathbf{p}||\mathbf{q})$  in our method, produces non-zero gradients that would actually increase the noise among pseudo-labels during its learning, which is not favorable according to our motivation.
Situation 2: Gradient update given confusing prediction  $\mathbf{q}$ . In addition, we are also interested in how different choices of distance and  $\lambda$  would impact the learning on pseudo-label if the segmentation model produces confusing outputs, i.e.,  $\mathbf{q}$  tends to be uniform. In line with the motivation of ERDA learning, we aim to regularize the pseudo-labels to mitigate potential noise and bias, while discouraging uncertain labels with little information. However, as in Tab. 1, most implementations yield non-zero gradient updates to the pseudo-label generation network. This update would make  $\mathbf{p}$  closer to the confused  $\mathbf{q}$ , thus increasing the noise and degrading the training performance. Conversely, only  $KL(\mathbf{p}||\mathbf{q})$  can produce a zero gradient when integrated with the entropy regularization with  $\lambda = 1$ . That is, only ERDA in Eq. (5) would not update the pseudo-label generation network when  $\mathbf{q}$  is not reliable, which avoids confusing the  $\mathbf{p}$ . Furthermore, when  $\mathbf{q}$  is less noisy but still close to a uniform vector, it is indicated that there is a large close-zero plateau on the gradient surface of ERDA, which benefits the learning on  $\mathbf{p}$  by resisting the influence of noise in  $\mathbf{q}$ .
In addition to the above cases, the gradients of ERDA in Eq. (5) could be generally regarded as being aware of the noise level and the confidence of both pseudo-label  $\mathbf{p}$  and the corresponding prediction  $\mathbf{q}$ . Especially, ERDA produces larger gradient updates on noisy pseudo-labels, while smaller updates on confident and reliable pseudo-labels or given noisy segmentation prediction. Therefore, our formulation demonstrates its superiority in fulfilling our motivation of simultaneous noise reduction and distribution alignment, where both  $L_{ER}$  and KL-based  $L_{DA}$  are necessary. We provide more empirical studies in ablation (Sec. 4.3) and detailed analysis in the supplementary.
3.3 Implementation Details on Pseudo-Labels
In our study, we use a prototypical pseudo-label generation process due to its popularity as well as simplicity [94]. Specifically, prototypes [63] denote the class centroids in the feature space, which
Figure 2. Detailed illustration of our ERDA with the prototypical pseudo-label generation process, which is shared for both (a) and (b) in Fig. 1.
are calculated based on labeled data, and pseudo-labels are estimated based on the feature distances between unlabeled points and class centroids.
As shown in Fig. 2, we use a momentum-based prototypical pseudo-label generation process due to its popularity as well as simplicity [94, 85, 93]. Specifically, prototypes [63] denote the class centroids in the feature space, which are calculated based on labeled data, and pseudo-labels are estimated based on the feature distances between unlabeled points and class centroids. To avoid expensive computational costs and compromised representations for each semantic class [94, 87, 47], momentum update is utilized as an approximation for global class centroids.
Based on the momentum-updated prototypes, we attach an MLP-based projection network to help generate pseudo-labels and learn with our method. Aligned with our motivation, we do not introduce thresholding-based label selection or one-hot conversion [46, 94] to process generated pseudo-labels. More details are in the supplementary.
More formally, we take as input a point cloud  $\mathcal{X}$ , where the labeled points are  $\mathcal{X}^l$  and the unlabeled points are  $\mathcal{X}^u$ . For a labeled point  $\mathbf{x} \in \mathcal{X}^l$ , we denote its label by  $y$ . The pseudo-label generation process can be described as follows:
$$
\hat {C} _ {k} = \frac {1}{N _ {k} ^ {l}} \sum_ {\mathbf {x} \in \mathcal {X} ^ {l} \wedge y = k} g \circ f (\mathbf {x}), C _ {k} \gets m C _ {k} + (1 - m) \hat {C} _ {k},
$$
$$
\forall \mathbf {x} \in \mathcal {X} ^ {u}, s _ {k} = d (g \circ f (\mathbf {x}), C _ {k}), \mathbf {p} = \operatorname {s o f t m a x} (\mathbf {s}),
$$
where  $C_k$  denotes the global class centroid for the  $k$ -th class,  $N_k^l$  is the number of labeled points of the  $k$ -th class,  $g \circ f = g(f(\cdot))$  is the transformation through the backbone network  $f$  and the projection network  $g$ ,  $m$  is the momentum coefficient, and we use cosine similarity for  $d(\cdot, \cdot)$  to generate the score  $s$ . By default, we use 2-layer MLPs for the projection network  $g$  and set  $m = 0.999$ .