SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Rui Sun $^{1*}$
Huayu Mai $^{1*}$
Tianzhu Zhang $^{1,2\dagger}$
Feng Wu $^{1,2}$
<sup>1</sup>Deep Space Exploration Laboratory/School of Information Science and Technology, University of Science and Technology of China
$^{2}$ Institute of Artificial Intelligence, Hefei Comprehensive National Science Center {issunrui, mai556}@mail.ustc.edu.cn, {tzzhang, fengwu}@ustc.edu.cn
The critical challenge of semi-supervised semantic segmentation lies how to fully exploit a large volume of unlabeled data to improve the model's generalization performance for robust segmentation. Existing methods tend to employ certain criteria (weighting function) to select pixel-level pseudo labels. However, the trade-off exists between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels in these methods when handling pseudo-labels without thoughtful consideration of the weighting function, hindering the generalization ability of the model. In this paper, we systematically analyze the trade-off in previous methods when dealing with pseudo-labels. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudolabels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. To this end, we propose Distribution-Aware Weighting (DAW) to strive to minimize the negative equivalence impact raised by the trade-off. We find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions. Besides, we devise distribution alignment to mitigate the issue of the discrepancy between the prediction distributions of labeled and unlabeled data. Extensive experimental results on multiple benchmarks including mitochondria segmentation demonstrate that DAW performs favorably against state-of-the-art methods. Code is available at https://github.com/yuisuen/DAW.
Semantic segmentation is a fundamental task that has achieved conspicuous achievements credited to the recent advances in deep neural networks [1]. However, its data-driven nature makes it heavily dependent on massive pixel-level annotations, which are laborious and time-consuming to gather. To alleviate the data-hunger issue, considerable works [2-7] have turned their attention to semi-supervised semantic segmentation, which has demonstrated great potential in practical applications [8, 9]. Since only limited labeled data is accessible, how to fully exploit a large volume of unlabeled data to improve the model's generalization performance for robust segmentation is thus extremely challenging.
In previous literature, pseudo-labeling [10-12] and consistency regularization [13-15] have emerged as mainstream paradigms to leverage unlabeled data for semi-supervised segmentation. In specific,
(a) Pos. & Neg. Dist.
(b) E1-E2 Trade-off
(c) E1
Figure 1: Illustration of our motivation. (a) shows the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels. (b) illustrates the negative equivalence impact on generalization performance raised by the trade-off. (c) (d) (e) summarize the models inevitably face a trade-off when dealing with pseudo-labels. Our method can guarantee the theoretical optimal solution by minimizing the negative impact.
(d) E2
(e)  $\mathbf{E1} + \mathbf{E2}$
the pseudo-labeling methods train the model on unlabeled samples with pseudo labels derived from the up-to-date model's own predictions. And the consistency regularization methods encourage the model to produce consistent predictions for the same sample with different perturbation views, following the smoothness assumption [16]. Recently, these two paradigms are often intertwined in the form of a teacher-student scheme [17-20, 3]. The critical idea involves updating the weights of the teacher model using the exponential moving average (EMA) of the student model, and the teacher model generates corresponding pseudo labels of the perturbed samples to instruct the learning of the student model.
Despite yielding promising results, these methods tend to employ certain criteria (referred to as weighting function) to select pixel-level pseudo labels, considering that the quality of the chosen pseudo-labels determines the upper bound of performance. On the one hand, naive pseudo-labeling methods such as Pseudo-Label [10] recruit all pseudo labels into training, assuming that each pseudo label is equally correct (i.e., weighting function can be regarded as a constant function). However, as training progresses, maximizing the utilization of pseudo-labels tends to lead to confirmation bias [21], which is a corollary raised by erroneous pseudo-labels (i.e., inaccurate yet utilized pseudo-labels). On the other hand, a series of threshold-based pseudo-labeling methods [22] such as FixMatch [18] attempt to set a high threshold (e.g., 0.95) to filter out pixel-level pseudo-labels with low confidence (i.e., the weighting function can be considered as a step function that jumps at 0.95). Although tangling the quality of pseudo-labels can alleviate noise, the strict criteria inevitably lead to the contempt of numerous unconfident yet correct pseudo-labels (i.e., correct yet discarded pseudolabels), hindering the learning process. To make matters worse, the negative impact is inevitably amplified by inbuilt low-data regimes of semi-supervised segmentation, leading to sub-optimal results. As a compromise, AEL [19] ad hoc defines the weighting function as a power function, which assigns weights conditioned on the confidence of pseudo-labels, that is, convincing pseudo-labels will be allocated more weights. However, the lack of sophisticated consideration and the arbitrary control of hyperparameters (i.e., tunable power) for the weighting function inevitably compromise its capability. In a nutshell, the trade-off exists between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels in these methods when handling pseudo-labels without thoughtful consideration of the weighting function, hindering the generalization ability of the model. Then, the question naturally arises: How to explore the better weighting function to effectively alleviate the negative impact raised by the trade-off?
In this work, we systematically analyze the trade-off in previous methods that hinder the model's learning when dealing with pseudo-labels in semi-supervised semantic segmentation. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. In specific, two Gaussian functions excelling at the maximum entropy property are devised to fit the confidence distribution of correct (positive distribution in Figure 1 (a)) and inaccurate (negative distribution in Figure 1 (a)) pseudo-labels using maximum likelihood estimation, respectively. The parameters of the Gaussian functions are updated via the exponential moving average (EMA) in pursuit of perceiving the learning status of the model. Then the trade-off can be naturally derived by calculating the expectations of inaccurate yet utilized pseudo-labels (depicted as  $E_{1}$  in Figure 1 (a)), and correct yet discarded pseudo-labels (displayed as
Table 1: We analyze the learning process of the mainstream methods for semi-supervised semantic segmentation systematically and uniformly abstract the criteria they adopt to select pseudo labels as weighting function  $f(p)$  conditioned on the confidence  $p$  of pseudo-labels. There are inherent distributions  $g(p)$  for positive and negative pseudo-labels (Gaussian distribution is employed as an approximation).
Method	Pseudo-Label	Fixmatch	AEL	Ours
f(p)	Weighting function of Pseudo Label	Weighting function of Feature	Weighting function of AEL	Optimal Weighting Function (Ours)
f(p)=u(p)	f(p)=u(p-t)	f(p)=p2	f(p)=u(p-t')
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used	Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
Negative Used	Negative Used
$E_{2}$  in Figure 1 (a)) respectively, on top of the weighting function and the corresponding confidence distribution. Now, we are prepared to propose the Distribution-Aware Weighting (DAW) function striving to minimize the negative equivalence impact on generalization performance raised by the trade-off, i.e., minimizing  $E1 + E2$ . By leveraging functional analysis, we find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions. Note that the dedicated weighting function is theoretically guaranteed by reconciling the intrinsic tension between  $E1$  and  $E2$  (see Figure 1 (b)), and is free of setting thresholds manually compared to previous methods. Besides, considering the imbalance issue caused by the discrepancy between the prediction distributions of labeled and unlabeled data, we propose distribution alignment to further unlock the potential of the weighting function and enjoy the synergy. In practice, the weighting function generated by DAW determines the criteria for selecting pseudo-labels to minimize the negative equivalence impact of the trade-off, minimizing  $E1 + E2$  (see Figure 1 (e)), which is conducive to model training. In this way, the model improves, benefiting from the effective probe of reliable pseudo-labels. And in turn, the positive distribution will be maximally separated from the negative one, leading to a simultaneous decrease in both  $E1$  and  $E2$  (see Figure 1 (c) and Figure 1 (d)), which is conducive to the generation of the weighting function.
Extensive experiments on mainstream benchmarks demonstrate that our method performs favorably against state-of-the-art semi-supervised semantic segmentation methods, proving that it can better exploit unlabeled data. Besides, we further validate the robustness of DAW on the electron microscopy mitochondria segmentation task, which involves images with dense foreground objects and cluttered backgrounds, making it more challenging to discriminate the reliability of the pseudo-labels.
2 Distribution-Aware Weighting
In this section, we first formulate the semi-supervised semantic segmentation problem as preliminaries (Section 2.1), and then formally define the trade-off between inaccurate yet utilized pseudo-labels  $(E1)$ , and correct yet discarded pseudo-labels  $(E2)$  by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function (Section 2.2).
Based on the analysis, we propose the distribution-aware weighting function (DAWF) to avoid performance degradation raised by the trade-off (Section 2.3). Finally, distribution alignment (DA) is devised to alleviate the discrepancies between the confidence distributions of labeled and unlabeled data. (Section 2.4).
In semi-supervised semantic segmentation, given a set of labeled training images  $\mathcal{D}_l = \{\mathbf{x}_i^l,\mathbf{y}_i^l\}_{i = 1}^{N_l}$  and a large amount of unlabeled images  $\mathcal{D}_u = \{\mathbf{x}_i^u\}_{i = 1}^{N_u}$ , where  $N_{l}$  and  $N_{u}$  denote the number of labeled and unlabeled images, respectively, and  $N_{u}\gg N_{l}$ . Let  $\mathbf{q}(\mathbf{x}_{ij}^{*})\in \mathbb{R}^{C}$  denotes the prediction of the  $j$ -th pixel in the  $i$ -th labeled (or unlabeled) image, and  $*\in \{l,u\}$ ,  $C$  is the number of categories. Then the supervised loss  $\mathcal{L}_s$  can be formulated as,
$$
\mathcal {L} _ {s} = \frac {1}{N _ {l}} \sum_ {i = 1} ^ {N _ {l}} \frac {1}{W H} \sum_ {j = 1} ^ {W H} \ell_ {c e} \left(\mathbf {y} _ {i j} ^ {l}, \mathbf {q} \left(\mathbf {x} _ {i j} ^ {l}\right)\right), \tag {1}
$$
where  $W$  and  $H$  represent the width and height of the input image,  $\ell_{ce}$  denotes the standard pixel-wise cross-entropy loss, and  $\mathbf{y}_{ij}^{l}$  denotes the ground-truth label from  $\mathcal{D}_l$ . Considering most methods [22, 18, 19, 10, 3, 23] tend to employ certain criteria (weighting function) to attempt to select reliable pseudo labels, we formulate the unsupervised loss  $\mathcal{L}_u$  as weighted cross-entropy for the convenience of introducing the weighting function  $f(p_{ij})$ ,
$$
\mathcal {L} _ {u} = \frac {1}{N _ {u}} \sum_ {i = 1} ^ {N _ {u}} \frac {1}{W H} \sum_ {j = 1} ^ {W H} f \left(p _ {i j}\right) \cdot \ell_ {c e} \left(\hat {\mathbf {y}} _ {i j} ^ {u}, \mathbf {q} \left(\mathcal {A} ^ {s} \left(\mathcal {A} ^ {w} \left(\mathbf {x} _ {i j} ^ {u}\right)\right)\right)\right), \tag {2}
$$
where  $\mathcal{A}^w / \mathcal{A}^s$  denotes weak/strong perturbation to encourage the model to produce consistent predictions,  $\mathbf{y}_{ij}^u$  denotes  $\mathbf{q}(\mathcal{A}^w(\mathbf{x}_{ij}^u))$ , i.e., prediction under the weak perturbation view. And  $\hat{\mathbf{y}}_{ij}^u = \operatorname{argmax}(\mathbf{y}_{ij}^u)$  is the one-hot pseudo-label,  $f(p_{ij})$  is a weighting function conditioned on  $p_{ij}$ , and  $p_{ij} = \max(\mathbf{y}_{ij}^u)$ , denotes the maximum confidence of the prediction. Then we define the overall loss function as  $\mathcal{L} = \mathcal{L}_s + \mathcal{L}_u$ .
2.2 E1-E2 Trade-off from Unified Weighting Function
We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. We instantiate the different biases inherent in the trade-off of previous methods and reveal their tight connection with the capability of the model. We start by defining the pseudo-label confidence distribution.
Orthogonal to other previous models, we assume that the confidence distribution of correct  $(g^{+}(p)$ , positive distribution) and inaccurate  $(g^{-}(p)$ , negative distribution) pseudo-labels follows a truncated Gaussian distribution with mean  $\mu^{+} / \mu^{-}$  and standard deviation  $\sigma^{+} / \sigma^{-}$ , formulated as,
$$
g ^ {+} (p) = \left\{ \begin{array}{l l} \frac {\alpha}{\sqrt {2 \pi} \sigma^ {+}} \exp \left[ - \frac {\left(p - \mu^ {+}\right) ^ {2}}{2 \left(\sigma^ {+}\right) ^ {2}} \right], & \frac {1}{C} \leqslant p \leqslant 1 \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {3}
$$
where  $1 / \alpha = \Phi \left(\frac{1 - \mu^{+}}{\sigma^{+}}\right) - \Phi \left(\frac{1 / C - \mu^{+}}{\sigma^{+}}\right)$  denotes the normalization factor,  $\Phi$  is the cumulative distribution function of the standard normal distribution. Note that  $p = \max (\mathbf{q}(\mathbf{x}_{ij}^l))$ , denotes the maximum confidence of the prediction from labeled data, so it must meet the condition of  $p \geqslant \frac{1}{C}$ . Similarly,
$$
g ^ {-} (p) = \left\{ \begin{array}{l l} \frac {\beta}{\sqrt {2 \pi} \sigma^ {-}} \exp \left[ - \frac {\left(p - \mu^ {-}\right) ^ {2}}{2 \left(\sigma^ {-}\right) ^ {2}} \right], & \frac {1}{C} \leqslant p \leqslant 1 \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {4}
$$
where  $1 / \beta = \Phi \left(\frac{1 - \mu^{-}}{\sigma^{-}}\right) - \Phi \left(\frac{1 / C - \mu^{-}}{\sigma^{-}}\right)$ . The reason we choose Gaussian is its valuable maximum entropy property, please refer to the supplementary material for more details. Then we estimate
the mean and standard deviation of the positive  $g^{+}(p)$  and negative distribution  $g^{-}(p)$ , respectively, resorting to maximum likelihood estimation,
$$
\hat {\mu} ^ {+} = \frac {1}{N _ {l}} \sum_ {i = 1} ^ {N _ {l}} \frac {1}{N _ {i} ^ {+}} \sum_ {j = 1} ^ {N _ {i} ^ {+}} p _ {i j} ^ {+}, \quad (\hat {\sigma} ^ {+}) ^ {2} = \frac {1}{N _ {l}} \sum_ {i = 1} ^ {N _ {l}} \frac {1}{N _ {i} ^ {+}} \sum_ {j = 1} ^ {N _ {i} ^ {+}} \left(p _ {i j} ^ {+} - \hat {\mu} ^ {+}\right) ^ {2}, \tag {5}
$$
where  $p_{ij}^{+}$  denotes the prediction confidence, where  $\mathrm{argmax}(\mathbf{q}(\mathbf{x}_{ij}^{l}))$  on the labeled data equals the ground truth  $\mathbf{y}_{ij}^{l}$ , and  $N_{i}^{+}$  is the number of  $p_{ij}^{+}$  in the  $i$ -th image. For the negative distribution  $g^{-}(p)$ , the parameters are evaluated in the same way, except that the predictions involved in the calculation are not equal to the ground truth. Note that we only consider predictions from labeled data to evaluate the Gaussian distribution equipped with ground truth, rather than estimating biases raised from unlabeled data with noisy pseudo-labels. Then the parameters of the Gaussian functions are updated via the exponential moving average (EMA) in pursuit of perceiving the learning status of the model in a dynamic manner,
$$
\hat {\mu} _ {t} ^ {+} = m \hat {\mu} _ {t - 1} ^ {+} + (1 - m) \hat {\mu} ^ {+}, \qquad (\hat {\sigma} _ {t} ^ {+}) ^ {2} = m (\hat {\sigma} _ {t - 1} ^ {+}) ^ {2} + (1 - m) \frac {\sum_ {i} N _ {i}}{\sum_ {i} N _ {i} - 1} (\hat {\sigma} ^ {+}) ^ {2}, \qquad (6)
$$
where unbiased variance is adopted for EMA,  $\hat{\mu}_0^+$  and  $(\hat{\sigma}_0^+)^2$  are initialized as  $1 / C$  and 1.0 respectively. A similar way also works for the negative distribution  $g^{-}(p)$ .
Then the trade-off can be naturally derived by calculating the expectations of inaccurate yet utilized pseudo-labels  $(E1)$ , and correct yet discarded pseudo-labels  $(E2)$  respectively, on top of the weighting function  $f(p)$  and the corresponding confidence distribution  $g^{-}(p) / g^{+}(p)$ .
Definition 3.1 Inaccurate yet utilized pseudo-labels,  $E1$
$$
E 1 = \mathbb {E} _ {g ^ {-}} [ f (p) ] = \int_ {\frac {1}{C}} ^ {1} f (p) \cdot g ^ {-} (p) d p. \tag {7}
$$
Definition 3.2 Correct yet discarded pseudo-labels,  $E2$
$$
E 2 = 1 - \mathbb {E} _ {g ^ {+}} [ f (p) ] = 1 - \int_ {\frac {1}{C}} ^ {1} f (p) \cdot g ^ {+} (p) d p. \tag {8}
$$
After formally defining the trade-off between  $E1$  and  $E2$ , it is natural to measure the impact of negative equivalence effects (i.e.,  $E1 + E2$ ), considering the trade-off between  $E1$  and  $E2$ , where an increase in one necessitates a decrease in the other.
Definition 3.3 Negative equivalence effect of the trade-off,  $E1 + E2$ .
$$
E 1 + E 2 = 1 + \int_ {\frac {1}{C}} ^ {1} f (p) \cdot \left[ g ^ {-} (p) - g ^ {+} (p) \right] d p, \tag {9}
$$
Then, we systematically analyze the trade-off in previous methods as tabulated in Table 1. For more detailed derivations, please refer to the supplementary material. (1) For example, naive pseudolabeling methods such as Pseudo-Label [10] enroll all pseudo labels  $(E2 = 0)$  into training. However, as training progresses, maximizing the utilization of pseudo-labels tends to a confirmation bias raised by erroneous pseudo-labels  $(E1 = 1)$ . (2) And for threshold-based pseudo-labeling methods such as FixMatch [18], which attempts to set a high threshold (0.95) to filter out pixel-level pseudo-labels with low confidence (small value of  $E1$  caused by the proximity of  $t = 0.95$  to 1). However, the strict criteria inevitably lead to the contempt of numerous unconfident yet correct pseudo-labels (large value of  $E2$  caused by the proximity of  $t = 0.95$  to 1). (3) As a compromise, AEL [19] ad hoc defines the weighting function as a power function, which assigns weights conditioned on confidence. That is, convincing pseudo-labels will be allocated more weight. However, the lack of sophisticated consideration and the arbitrary control of hyperparameters (tunable power) for the weighting function inevitably compromise its capability (not guaranteeing the lowest negative equivalence effect).
2.3 Distribution-Aware Weighting Function
Then we seek to explore a better weighting function equipped with the formal trade-off definition, aiming at minimizing the negative equivalence impact raised by the trade-off, that is, minimizing
$$
E 1 + E 2,
$$
$$
\min  _ {f (p)} \quad E 1 + E 2 = 1 + \int_ {\frac {1}{C}} ^ {1} f (p) \cdot \left[ g ^ {-} (p) - g ^ {+} (p) \right] d p, \tag {10}
$$
$$
\begin{array}{l} \text {s . t .} \quad 0 \leq f (p) \leq 1, \end{array}
$$
By leveraging functional analysis, we find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions, formulated as,
$$
f ^ {*} (p) = \left\{ \begin{array}{l l} 1, & t ^ {*} \leqslant p \leqslant 1 \\ 0, & \text {o t h e r w i s e} \end{array} , \quad t ^ {*} = \left(\left(\beta_ {2} ^ {2} - 4 \beta_ {1} \beta_ {3}\right) ^ {\frac {1}{2}} - \beta_ {2}\right) / \left(2 \beta_ {1}\right), \right. \tag {11}
$$
where  $\beta_{1} = (\sigma^{+})^{2} - (\sigma^{-})^{2},\beta_{2} = 2[\mu^{+}(\sigma^{-})^{2} - \mu^{-}(\sigma^{+})^{2}],\beta_{3} = (\sigma^{+}\mu^{-})^{2} - (\sigma^{-}\mu^{+})^{2} +$ $2(\sigma^{+}\sigma^{-})^{2}\ln [(\alpha \sigma^{-}) / (\beta \sigma^{+})]$  and  $p = \max (\mathbf{y}_{ij}^u)$  denotes the confidence of the prediction from unlabeled data. Please refer to the supplementary material for more detailed derivations. Note that the dedicated weighting function  $f^{*}(p)$  is theoretically guaranteed by reconciling the intrinsic tension between  $E1$  and  $E2$  (see Table 1) and is free of setting thresholds manually compared to previous methods.
2.4 Distribution Alignment
Furthermore, considering the imbalance issue caused by the discrepancy between the prediction distributions of labeled and unlabeled data, we propose distribution alignment (DA) to further unlock the potential of the distribution-aware weighting function. In specific, we define the confidence distributions from labeled data and unlabeled data as expectations  $\mathbb{E}_{\mathcal{D}_l}\left[\mathbf{q}(\mathbf{x}_{ij}^l)\right]$  and  $\mathbb{E}_{\mathcal{D}_u}\left[\mathbf{q}(\mathbf{x}_{ij}^u)\right]$ , respectively. Both of these are estimated in the form of EMA in each batch as the training progresses, denoted as  $\hat{\mathbb{E}}_{\mathcal{D}_l}\left[\mathbf{q}(\mathbf{x}_{ij}^l)\right]$  and  $\hat{\mathbb{E}}_{\mathcal{D}_u}\left[\mathbf{q}(\mathbf{x}_{ij}^u)\right]$ . Then we use the ratio between the expectations of labeled and unlabeled to normalize the each prediction  $\mathbf{y}_{ij}^{u} = q(\mathbf{x}_{ij}^{u})$  on unlabeled data, formulated as,
$$
\mathrm {D A} \left(\mathbf {y} _ {i j} ^ {u}\right) = \operatorname {N o r m} \left(\mathbf {y} _ {i j} ^ {u} \cdot \frac {\hat {\mathbb {E}} _ {\mathcal {D} _ {l}} \left[ \mathbf {q} \left(\mathbf {x} _ {i j} ^ {l}\right) \right]}{\hat {\mathbb {E}} _ {\mathcal {D} _ {u}} \left[ \mathbf {q} \left(\mathbf {x} _ {i j} ^ {u}\right) \right]}\right), \tag {12}
$$
where  $\mathrm{Norm}(\cdot)$  denotes the normalization operation used to constrain the probabilities to sum up to 1. Then we bring the normalized probability back to Equation 2 to calculate the loss weight of each pseudo-label after alignment,
$$
\mathcal {L} _ {u} = \frac {1}{N _ {u}} \sum_ {i = 1} ^ {N _ {u}} \frac {1}{W H} \sum_ {j = 1} ^ {W H} f ^ {*} \left(\max  \left(\mathrm {D A} \left(\mathbf {y} _ {i j} ^ {u}\right)\right)\right) \cdot \ell_ {c e} \left(\hat {\mathbf {y}} _ {i j} ^ {u}, \mathbf {q} \left(\mathcal {A} ^ {s} \left(\mathcal {A} ^ {w} \left(\mathbf {x} _ {i j} ^ {u}\right)\right)\right)\right), \tag {13}
$$