Title: DAW: Exploring the Better Weighting Function for Semi-supervised Semantic Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: semi-supervised semantic segmentation
Abstract: The critical challenge of semi-supervised semantic segmentation lies in how to fully exploit a large volume of unlabeled data to improve the model’s generalization performance for robust segmentation.  Existing methods tend to employ certain criteria (weighting function) to select pixel-level pseudo labels. However, the trade-off exists between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels in these methods when handling pseudo-labels without thoughtful consideration of the weighting function, hindering the generalization ability of the model. In this paper, we systematically analyze the trade-off in previous methods when dealing with pseudo-labels. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. To this end, we propose Distribution-Aware Weighting (DAW) to strive to minimize the negative equivalence impact raised by the trade-off. We find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions. Besides, we devise distribution alignment to mitigate the issue of the discrepancy between the prediction distributions of labeled and unlabeled data. Extensive experimental results on multiple benchmarks including mitochondria segmentation demonstrate that DAW performs favorably against state-of-the-art methods.

DAW: Exploring the Better Weighting Function for Semi-supervised Semantic Segmentation

Abstract

The critical challenge of semi-supervised semantic segmentation lies how to fully exploit a large volume of unlabeled data to improve the model's generalization performance for robust segmentation. Existing methods tend to employ certain criteria (weighting function) to select pixel-level pseudo labels. However, the trade-off exists between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels in these methods when handling pseudo-labels without thoughtful consideration of the weighting function, hindering the generalization ability of the model. In this paper, we systematically analyze the trade-off in previous methods when dealing with pseudo-labels. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudolabels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. To this end, we propose Distribution-Aware Weighting (DAW) to strive to minimize the negative equivalence impact raised by the trade-off. We find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions. Besides, we devise distribution alignment to mitigate the issue of the discrepancy between the prediction distributions of labeled and unlabeled data. Extensive experimental results on multiple benchmarks including mitochondria segmentation demonstrate that DAW performs favorably against state-of-the-art methods. Code is available at https://github.com/yuisuen/DAW .

1 Introduction

Semantic segmentation is a fundamental task that has achieved conspicuous achievements credited to the recent advances in deep neural networks [1]. However, its data-driven nature makes it heavily dependent on massive pixel-level annotations, which are laborious and time-consuming to gather. To alleviate the data-hunger issue, considerable works [2-7] have turned their attention to semi-supervised semantic segmentation, which has demonstrated great potential in practical applications [8, 9]. Since only limited labeled data is accessible, how to fully exploit a large volume of unlabeled data to improve the model's generalization performance for robust segmentation is thus extremely challenging.

In previous literature, pseudo-labeling [10-12] and consistency regularization [13-15] have emerged as mainstream paradigms to leverage unlabeled data for semi-supervised segmentation. In specific,

(a)

Pos. & Neg. Dist.

(b)

E1-E2 Trade-off

(c)

E1

(d)

E2

(e)

E1

+

E2

Positive Distribution

Positive and Negative Distribution

Trade-off between E1 and E2

Fixmatch

1.0

Fixmatch

0.25

Fixmatch

1.0

Fixmatch

Negative Distribution

AEL

Ours

Pseudo-Label

0.9

0.8

Ours

Pseudo-Label

AEL

0.20

Ours

AEL

Pseudo-Label

0.9-

Ours

Pseudo-Label

AEL

0.8

0.6

0.7

0.15

0.6

0.5

0.10

0.7

0.6

E1

0.6

0.05

0.5

E2

0.0

0.3

00'0

0.0

0.4

0.6

9.8

1.0

0.0

0.4

2

0.6

.0

100000

Taining teratiors

200000

300000

400000

100000

Training iterations

200000

300000

400000

100000

Training Iteratiors

200000

390000

400000

the pseudo-labeling methods train the model on unlabeled samples with pseudo labels derived from the up-to-date model's own predictions. And the consistency regularization methods encourage the model to produce consistent predictions for the same sample with different perturbation views, following the smoothness assumption [16]. Recently, these two paradigms are often intertwined in the form of a teacher-student scheme [17-20, 3]. The critical idea involves updating the weights of the teacher model using the exponential moving average (EMA) of the student model, and the teacher model generates corresponding pseudo labels of the perturbed samples to instruct the learning of the student model.

Despite yielding promising results, these methods tend to employ certain criteria (referred to as weighting function ) to select pixel-level pseudo labels, considering that the quality of the chosen pseudo-labels determines the upper bound of performance. On the one hand, naive pseudo-labeling methods such as Pseudo-Label [10] recruit all pseudo labels into training, assuming that each pseudo label is equally correct (i.e., weighting function can be regarded as a constant function). However, as training progresses, maximizing the utilization of pseudo-labels tends to lead to confirmation bias [21], which is a corollary raised by erroneous pseudo-labels (i.e., inaccurate yet utilized pseudo-labels ). On the other hand, a series of threshold-based pseudo-labeling methods [22] such as FixMatch [18] attempt to set a high threshold (e.g., 0.95) to filter out pixel-level pseudo-labels with low confidence (i.e., the weighting function can be considered as a step function that jumps at 0.95). Although tangling the quality of pseudo-labels can alleviate noise, the strict criteria inevitably lead to the contempt of numerous unconfident yet correct pseudo-labels (i.e., correct yet discarded pseudolabels ), hindering the learning process. To make matters worse, the negative impact is inevitably amplified by inbuilt low-data regimes of semi-supervised segmentation, leading to sub-optimal results. As a compromise, AEL [19] ad hoc defines the weighting function as a power function, which assigns weights conditioned on the confidence of pseudo-labels, that is, convincing pseudo-labels will be allocated more weights. However, the lack of sophisticated consideration and the arbitrary control of hyperparameters (i.e., tunable power) for the weighting function inevitably compromise its capability. In a nutshell, the trade-off exists between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels in these methods when handling pseudo-labels without thoughtful consideration of the weighting function, hindering the generalization ability of the model. Then, the question naturally arises: How to explore the better weighting function to effectively alleviate the negative impact raised by the trade-off?

In this work, we systematically analyze the trade-off in previous methods that hinder the model's learning when dealing with pseudo-labels in semi-supervised semantic segmentation. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudolabels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. In specific, two Gaussian functions excelling at the maximum entropy property are devised to fit the confidence distribution of correct (positive distribution in Figure 1 (a)) and inaccurate (negative distribution in Figure 1 (a)) pseudo-labels using maximum likelihood estimation, respectively. The parameters of the Gaussian functions are updated via the exponential moving average (EMA) in pursuit of perceiving the learning status of the model. Then the trade-off can be naturally derived by calculating the expectations of inaccurate yet utilized pseudo-labels (depicted as E 1 in Figure 1 (a)), and correct yet discarded pseudo-labels (displayed as

Method

Pseudo-Label

Fixmatch

AEL

Ours

f

(

p

)

f

(

p

)

·

g

(

p

)

E

1

1

β

[Φ(

1

-

µ

-

σ

-

)

-

Φ(

t

-

µ

-

σ

-

)]

(

µ

-

)

2

+(

σ

-

)

2

+

µ

-

(

σ

-

)

2

g

-

(

1

C

)

-

(1 +

µ

-

)(

σ

-

)

2

g

-

(1)

β

[Φ(

1

-

µ

-

σ

-

)

-

Φ(

t

∗

-

µ

-

σ

-

)]

E

2

0

1

-

α

[Φ(

1

-

µ

+

σ

+

)

-

Φ(

t

-

µ

+

σ

+

)]

1

-

(

µ

+

)

2

-

(

σ

+

)

2

-

µ

+

(

σ

+

)

2

g

+

(

1

C

)

+(1

-

µ

+

)(

σ

+

)

2

g

+

(1)

1

-

α

[Φ(

1

-

µ

+

σ

+

)

-

Φ(

t

∗

-

µ

+

σ

+

)]

E

1 +

E

2

1 +

β

[Φ(

1

-

µ

-

σ

-

)

-

Φ(

t

∗

-

µ

-

σ

-

)]

-

α

[Φ(

1

-

µ

+

σ

+

)

-

Φ(

t

∗

-

µ

+

σ

+

)]

+

∫

t

∗

1

C

[

g

-

(

p

)

-

g

+

(

p

)]

dp

1 +

β

[Φ(

1

-

µ

-

σ

-

)

-

Φ(

t

∗

-

µ

-

σ

-

)]

-

α

[Φ(

1

-

µ

+

σ

+

)

-

Φ(

t

∗

-

µ

+

σ

+

)]

-

∫

t

t

∗

[

g

-

(

p

)

-

g

+

(

p

)]

dp

1 +

β

[Φ(

1

-

µ

-

σ

-

)

-

Φ(

t

∗ -

µ

-

σ

-

)]

-

α

[Φ(

1

-

µ

+

σ

+

)

-

Φ(

t

∗ -

µ

+

σ

+

)]

+

∫

t

∗

1

C

x

2

[

g

-

(

p

)

-

g

+

(

p

)]

dp

+

∫

1

t

∗

(

x

2

-

1)[

g

-

(

p

)

-

g

+

(

p

)]

dp

1 +

β

[Φ(

1

-

µ

-

σ

-

)

-

Φ(

t

∗

-

µ

-

σ

-

)]

-

α

[Φ(

1

-

µ

+

σ

+

)

-

Φ(

t

∗

-

µ

+

σ

+

)]

Note

∫

t

∗

1

C

[

g

-

(

p

)

-

g

+

(

p

)]

dp

>

0

-

∫

t

t

∗

[

g

-

(

p

)

-

g

+

(

p

)]

dp

⩾

0

∫

t

∗

1

C

x

2

[

g

-

(

p

)

-

g

+

(

p

)]

dp

+

∫

1

t

∗

(

x

2

-

1)[

g

-

(

p

)

-

g

+

(

p

)]

dp

>

0

Smaller than all of them!

Wieighting function (f Pseudo-Label

1.9

(d)n = (d)y

0.8

0.6

0.4-

0.2-

0.0-

9.2

0.4

F

9.8

1.0

Weightedl Distribution of Pseude-Label

Negative Discarded

Negative Used

Positive Discarded

Positive Used

E1

8.0

2

0.4

0.6

0.8

Weighting function of fiematch

0.8

90

f(p) = u(p  t)

co

0.2

9.4

B

0.6

9.8

1.0

Weightedl Distribution of Fixmatch

1

Positive Discarded

Positive Used

Negative Discarded

Negative Used

F2

0.0

9.2

0.4

0.6

0.8

feighting Function of AEL

1.9

0.8

0.6

f(p)=p²

0.4-

0.2

0.0-

0.2

9.4

9.6

0.8

1.0

Weightee Distrilbution of AEL

Original Negative Distribution

Negative Used

Original Positive Distribution

Positive Used

E1

9.9

9.2

0.4

0.6

0.8

1.0

Optimal Weighting function (Ours)

1.9

0.8

0.6

(,2  d)n = (d)

0.4-

0.2

9.9

0.0

9.2

0.4

F

9.8

1.0

Weighted Distribution of Ours

Negative Discarded

Negative Used

Positive Discarded

Positive Used

E2

E1

0.0

0.2

0.4

P

0.6

0.8

E 2 in Figure 1 (a)) respectively, on top of the weighting function and the corresponding confidence distribution. Now, we are prepared to propose the Distribution-Aware Weighting ( DAW ) function striving to minimize the negative equivalence impact on generalization performance raised by the trade-off, i.e., minimizing E 1+ E 2 . By leveraging functional analysis, we find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions. Note that the dedicated weighting function is theoretically guaranteed by reconciling the intrinsic tension between E 1 and E 2 (see Figure 1 (b)), and is free of setting thresholds manually compared to previous methods. Besides, considering the imbalance issue caused by the discrepancy between the prediction distributions of labeled and unlabeled data, we propose distribution alignment to further unlock the potential of the weighting function and enjoy the synergy. In practice, the weighting function generated by DAW determines the criteria for selecting pseudo-labels to minimize the negative equivalence impact of the trade-off, minimizing E 1 + E 2 (see Figure 1 (e)), which is conducive to model training. In this way, the model improves, benefiting from the effective probe of reliable pseudo-labels. And in turn, the positive distribution will be maximally separated from the negative one, leading to a simultaneous decrease in both E 1 and E 2 (see Figure 1 (c) and Figure 1 (d)), which is conducive to the generation of the weighting function.

Extensive experiments on mainstream benchmarks demonstrate that our method performs favorably against state-of-the-art semi-supervised semantic segmentation methods, proving that it can better exploit unlabeled data. Besides, we further validate the robustness of DAW on the electron microscopy mitochondria segmentation task, which involves images with dense foreground objects and cluttered backgrounds, making it more challenging to discriminate the reliability of the pseudo-labels.

2 Distribution-Aware Weighting

In this section, we first formulate the semi-supervised semantic segmentation problem as preliminaries (Section 2.1), and then formally define the trade-off between inaccurate yet utilized pseudo-labels ( E 1 ), and correct yet discarded pseudo-labels ( E 2 ) by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function (Section 2.2).

Based on the analysis, we propose the distribution-aware weighting function (DAWF) to avoid performance degradation raised by the trade-off (Section 2.3). Finally, distribution alignment (DA) is devised to alleviate the discrepancies between the confidence distributions of labeled and unlabeled data. (Section 2.4).

2.1 Preliminaries

In semi-supervised semantic segmentation, given a set of labeled training images D l = { x l i , y l i } N l i =1 and a large amount of unlabeled images D u = { x u i } N u i =1 , where N l and N u denote the number of labeled and unlabeled images, respectively, and N u ≫ N l . Let q ( x ∗ ij ) ∈ R C denotes the prediction of the j -th pixel in the i -th labeled (or unlabeled) image, and ∗ ∈ { l, u } , C is the number of categories. Then the supervised loss L s can be formulated as,

where W and H represent the width and height of the input image, ℓ ce denotes the standard pixel-wise cross-entropy loss, and y l ij denotes the ground-truth label from D l . Considering most methods [22, 18, 19, 10, 3, 23] tend to employ certain criteria (weighting function) to attempt to select reliable pseudo labels, we formulate the unsupervised loss L u as weighted cross-entropy for the convenience of introducing the weighting function f ( p ij ) ,

where A w / A s denotes weak/strong perturbation to encourage the model to produce consistent predictions, y u ij denotes q ( A w ( x u ij )) , i.e., prediction under the weak perturbation view. And ˆ y u ij = argmax( y u ij ) is the one-hot pseudo-label , f ( p ij ) is a weighting function conditioned on p ij , and p ij = max( y u ij ) , denotes the maximum confidence of the prediction. Then we define the overall loss function as L = L s + L u .

2.2 E1 -E2 Trade-off from Unified Weighting Function

We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. We instantiate the different biases inherent in the trade-off of previous methods and reveal their tight connection with the capability of the model. We start by defining the pseudo-label confidence distribution.

Orthogonal to other previous models, we assume that the confidence distribution of correct ( g + ( p ) , positive distribution) and inaccurate ( g -( p ) , negative distribution) pseudo-labels follows a truncated Gaussian distribution with mean µ + /µ -and standard deviation σ + /σ -, formulated as,

where 1/ α = Φ ( 1 -µ + σ + ) -Φ ( 1 /C -µ + σ + ) denotes the normalization factor, Φ is the cumulative distribution function of the standard normal distribution. Note that p = max( q ( x l ij )) , denotes the maximum confidence of the prediction from labeled data, so it must meet the condition of p ⩾ 1 C . Similarly,

where 1 /β = Φ ( 1 -µ -σ -) -Φ ( 1 /C -µ -σ -) . The reason we choose Gaussian is its valuable maximum entropy property, please refer to the supplementary material for more details. Then we estimate

the mean and standard deviation of the positive g + ( p ) and negative distribution g -( p ) , respectively, resorting to maximum likelihood estimation,

where p + ij denotes the prediction confidence, where argmax( q ( x l ij )) on the labeled data equals the ground truth y l ij , and N + i is the number of p + ij in the i -th image. For the negative distribution g -( p ) , the parameters are evaluated in the same way, except that the predictions involved in the calculation are not equal to the ground truth. Note that we only consider predictions from labeled data to evaluate the Gaussian distribution equipped with ground truth, rather than estimating biases raised from unlabeled data with noisy pseudo-labels. Then the parameters of the Gaussian functions are updated via the exponential moving average (EMA) in pursuit of perceiving the learning status of the model in a dynamic manner,

where unbiased variance is adopted for EMA, ˆ µ + 0 and (ˆ σ + 0 ) 2 are initialized as 1 /C and 1.0 respectively. A similar way also works for the negative distribution g -( p ) .

Then the trade-off can be naturally derived by calculating the expectations of inaccurate yet utilized pseudo-labels ( E 1 ), and correct yet discarded pseudo-labels ( E 2 ) respectively, on top of the weighting function f ( p ) and the corresponding confidence distribution g -( p ) / g + ( p ) .

Definition 3.1 Inaccurate yet utilized pseudo-labels, E 1 .

Definition 3.2 Correct yet discarded pseudo-labels, E 2 .

After formally defining the trade-off between E 1 and E 2 , it is natural to measure the impact of negative equivalence effects (i.e., E 1 + E 2 ), considering the trade-off between E 1 and E 2 , where an increase in one necessitates a decrease in the other.

Definition 3.3 Negative equivalence effect of the trade-off, E 1 + E 2 .

Then, we systematically analyze the trade-off in previous methods as tabulated in Table 1. For more detailed derivations, please refer to the supplementary material. (1) For example, naive pseudolabeling methods such as Pseudo-Label [10] enroll all pseudo labels ( E 2 = 0 ) into training. However, as training progresses, maximizing the utilization of pseudo-labels tends to a confirmation bias raised by erroneous pseudo-labels ( E 1 = 1 ). (2) And for threshold-based pseudo-labeling methods such as FixMatch [18], which attempts to set a high threshold (0.95) to filter out pixel-level pseudo-labels with low confidence ( small value of E 1 caused by the proximity of t = 0 . 95 to 1). However, the strict criteria inevitably lead to the contempt of numerous unconfident yet correct pseudo-labels ( large value of E 2 caused by the proximity of t = 0 . 95 to 1). (3) As a compromise, AEL [19] ad hoc defines the weighting function as a power function, which assigns weights conditioned on confidence. That is, convincing pseudo-labels will be allocated more weight. However, the lack of sophisticated consideration and the arbitrary control of hyperparameters (tunable power) for the weighting function inevitably compromise its capability (not guaranteeing the lowest negative equivalence effect).

2.3 Distribution-Aware Weighting Function

Then we seek to explore a better weighting function equipped with the formal trade-off definition, aiming at minimizing the negative equivalence impact raised by the trade-off, that is, minimizing

E 1 + E 2 ,

By leveraging functional analysis, we find an interesting fact that the optimal solution for the weighting function is a hard step function, with the jump point located at the intersection of the two confidence distributions, formulated as,

where β 1 = ( σ + ) 2 -( σ -) 2 , β 2 = 2[ µ + ( σ -) 2 -µ -( σ + ) 2 ] , β 3 = ( σ + µ -) 2 -( σ -µ + ) 2 + 2 ( σ + σ -) 2 ln[( ασ -) / ( βσ + )] and p = max( y u ij ) denotes the confidence of the prediction from unlabeled data. Please refer to the supplementary material for more detailed derivations. Note that the dedicated weighting function f ∗ ( p ) is theoretically guaranteed by reconciling the intrinsic tension between E 1 and E 2 (see Table 1) and is free of setting thresholds manually compared to previous methods.

2.4 Distribution Alignment

Furthermore, considering the imbalance issue caused by the discrepancy between the prediction distributions of labeled and unlabeled data, we propose distribution alignment (DA) to further unlock the potential of the distribution-aware weighting function. In specific, we define the confidence distributions from labeled data and unlabeled data as expectations E D l [ q ( x l ij ) ] and E D u [ q ( x u ij ) ] , respectively. Both of these are estimated in the form of EMA in each batch as the training progresses, denoted as ˆ E D l [ q ( x l ij ) ] and ˆ E D u [ q ( x u ij ) ] . Then we use the ratio between the expectations of labeled and unlabeled to normalize the each prediction y u ij = q ( x u ij ) on unlabeled data, formulated as,

where Norm( · ) denotes the normalization operation used to constrain the probabilities to sum up to 1. Then we bring the normalized probability back to Equation 2 to calculate the loss weight of each pseudo-label after alignment,

where ˆ y u ij = argmax(DA( y u ij )) . In this way, the distribution-aware weighting function is rewarded with better generalization, benefiting from more equal learning of labeled and unlabeled data, mitigating the issue of distribution imbalance, and enjoying the synergy. The algorithm flow is shown in the supplementary material.

3 Experiments

3.1 Experimental Setup

Datasets: (1) PASCAL VOC 2012 [29] is an object-centric semantic segmentation dataset, containing 21 classes with 1,464 and 1,449 finely annotated images for training and validation, respectively. Some researches [30, 19] augment the original training set (e.g., classic ) by incorporating the coarsely annotated images in SBD [31], obtaining a training set (e.g., blender ) with 10,582 labeled samples. (2) Cityscapes [32]is an urban scene understanding dataset with 2,975 images for training and 500 images for validation.

Implementation Details: For a fair comparison, we follow the common practice and use ResNet [33] as our backbone and DeepLabv3+[34] as the decoder. We set the crop size as 513 × 513 for PASCAL and 801 × 801 for Cityscapes, respectively. For both datasets, we adopt SGD as the optimizer with the same batch size of 16 and different initial learing rate, which is set as 0.001 and 0.005 for PASCAL and Cityscapes. We use the polynomial policy to dynamically decay the learning rate along the whole

Method

Classic

Blender

1/16(92)

1/8(183)

1/4(366)

1/2(732)

Full(1464)

1/16(662)

1/8(1323)

1/4(2646)

ResNet-50

Sup.-only

44

.

0

52

.

3

61

.

7

66

.

7

72

.

9

62

.

4

68

.

2

72

.

3

Pseudo-Label[ICML'13] [10]

55

.

7

60

.

2

65

.

6

69

.

7

74

.

8

66

.

3

70

.

8

74

.

5

FixMatch[NeurIPS'20] [18]

60

.

1

67

.

3

71

.

4

73

.

7

76

.

9

70

.

6

73

.

9

75

.

1

iMAS[CVPR'23] [24]

-

-

-

-

-

74

.

8

76

.

5

77

.

0

AugSeg[CVPR'23] [25]

64

.

2

72

.

2

76

.

2

77

.

4

78

.

8

74

.

7

76

.

0

77

.

2

DAW(Ours)

68

.

5

73

.

1

76

.

3

78

.

6

79

.

7

76

.

2

77

.

6

77

.

4

∆

↑

+

24

.

5

+

20

.

8

+

14

.

6

+

11

.

9

+

6

.

8

+

13

.

8

+

9

.

4

+

5

.

1

Sup.-only

45

.

1

55

.

3

64

.

8

69

.

7

73

.

5

67

.

5

71

.

1

74

.

2

Pseudo-Label[ICML'13] [10]

57

.

3

64

.

1

69

.

4

73

.

3

77

.

2

69

.

1

73

.

8

76

.

7

FixMatch[NeurIPS'20] [18]

63

.

9

73

.

0

75

.

5

77

.

8

79

.

2

74

.

3

76

.

3

76

.

9

CPS[CVPR'21] [26]

64

.

1

67

.

4

71

.

7

75

.

9

-

74

.

5

76

.

4

77

.

7

AEL[NeurIPS'21] [27]

-

-

-

-

-

77

.

2

77

.

6

78

.

1

iMAS[CVPR'23] [24]

68

.

8

74

.

4

78

.

5

79

.

5

81

.

2

76

.

5

77

.

9

78

.

1

AugSeg[CVPR'23] [25]

71

.

1

75

.

5

78

.

8

80

.

3

81

.

4

77

.

0

77

.

3

78

.

8

CCVC[CVPR'23] [28]

70

.

2

74

.

4

77

.

4

79

.

1

80

.

5

77

.

2

78

.

4

79

.

0

DAW(Ours)

74

.

8

77

.

4

79

.

5

80

.

6

81

.

5

78

.

5

78

.

9

79

.

6

∆

↑

+

29

.

7

+

22

.

1

+

14

.

7

+

10

.

9

+

8

.

0

+

11

.

0

+

7

.

8

+

5

.

4

training and assemble the channel dropout perturbation [22] to improve the generalization ability of the model. We train the model for 80 epochs on PASCAL and 240 epochs on Cityscapes, using 8 × NVIDIA GeForce RTX 3090 GPUs.

3.2 Comparison with State-of-the-art Methods

We conduct experiments on two popular benchmarks including PASCAL VOC 2012 and Cityscapes and make a fair comparison with SOTA semi-supervised semantic segmentation methods. We consistently observe that our DAW outperforms all other methods under all partition protocols on all datasets with different backbones, which strongly proves the effectiveness of our method.

Results on PASCAL VOC 2012 Dataset. Table 2 shows the comparison of our method with the SOTA methods on PASCAL classic and blender set. Specifically, on the PASCAL classic set, our method outperforms the supervised-only ( Sup.-only ) model by 29.7%, 22.1%, 14.7%, 10.9% under the partition protocols of 1/16, 1/8, 1/4 and 1/2, respectively with ResNet-101. Our method also significantly outperforms the existing semi-supervised SOTA methods under all data partition protocols. Taking the recently proposed method AugSeg [25] as an example, the performance gain of our approach reaches to +4.3% under 1/16 partition protocol with ResNet-50. The same superiority of our method can also be observed on the PASCAL blender set.

Results on Cityscapes Dataset. Table 3 compares DAW with SOTA methods on the Cityscapes dataset. DAW achieves consistent performance gains over the Sup.-only baseline, obtaining improvements of 11.9%, 7.3%, 6.0% and 2.9% under 1/16, 1/8, 1/4 and 1/2 partition protocols with ResNet-50, respectively. We can also see that over all protocols, DAW outperforms the SOTA methods, e.g., DAW excels to iMAS [24] by 1.1% under the 1/16 partition with ResNet-101.

None

Fixed

DA

mIoU

✓

62

.

7

✓

66

.

9

✓

68

.

0

✓

✓

68

.

5

m

mIoU

0

.

99

68

.

5

0.999

68

.

1

0

.

9999

67

.

9

Pseudo-Label

Image

Ground Truth

FixMatch

AEL

Ours

μ± 3o+

μ ±30*

t'

25000 50000 75000 10000 125000 150000 175000200000

Training Iterations

Qualitative Results. We compare qualitative results of our DAW with different SOTA methods. As shown in Figure 3 , DAW also shows more powerful segmentation performance in fine-grained details (see the first and second row in Figure 3). With the help of the optimal weighting function, DAW demonstrates superior abilities in most scenarios.

3.3 Ablation Study and Analysis

To look deeper into our method, we perform a series of ablation studies on PASCAL classic set under 92 partition protocol with ResNet-50 to analyze each component of our DAW, including the D istributionA ware W eighting F unction (DAWF) and the D istribution A lignment (DA). The baseline method is UniMatch [22].

Effectiveness of Components. In Table 4, 'None' denotes there is no threshold for pseudo-label during the training (i.e., Pseudo-Label [10]) while 'Fixed" indicates that a fixed threshold is set (i.e., UiMatch [22]). A certain performance lift compared with the baseline can be observed owing to

Fixmatch

Pseudo-Label

AEL

Ours

Epoch 79

w DA

w/o DA

Positive Distribution

Negative Distribution

μ*=0.93

0*=0.056

Negative Distribution

Positive Distribution

°=04

Positive Distribution

Negative Distribution

°=04

Positive Distribution

Negative Distribution

μ²=0.938

o°=0.057

v~=0.692

0° =0.091

Positive Distribution

Negative Distribution

Negative Distribution

Positive Distribution

Positive Distribution

Negative Distribution

8.83

Positive Distribution

Negative Distribution

.

=0.752

8.3

Negative Distriltxtien beNind Label Data

Positive Distributien behind Unlabel Data

Negative Distributien behind Unlabel Data

Positive Distribution behind Label Data

0.0

0.4

0.6

0.8

Negative Distritution behind Label Data

Pesitive Distribution behind Unlabel Data

Positive Distributign behind Label Data

0.0

0.2

0.4

0.6

0.8

1L0

Ground Truth

the introduction of Distribution-Aware Weighting Function and Distribution Alignment. (1) The utilization of DAWF brings a 1.1% improvement of mIoU, demonstrating that the negative impact raised by the E 1 + E 2 trade-off is effectively alleviated. (2) DA brings further accuracy gains, indicating that the existence of the discrepancy between the distributions of labeled and unlabeled data may cause a bottleneck in learning. For better visualization, we employ the unused annotations of 'unlabeled data' to calculate the ground-truth distribution on the unlabeled data. And as shown in Figure 5, there is a relatively large gap between the distributions, and DA can effectively relieve it, further unlocking the potential of the weighting function and enjoy the synergy.

Hyperparameter Evaluations. As shown in Table 5, it can be observed that the performance is optimal with m = 0 . 99 .

Scalability for Other Scenarios. We further conduct extra experiments on Lucchi [37-42] to evaluate the scalability of our method. Figure 6 shows the image and ground-truth of Lucchi dataset, presenting a common problem in electron microscope images that the instances are very small and scattered. This calls for more reliable supervision in training under a semi-supervised setting. As shown in Table 6, DAW outperforms other competitive methods in the electron microscopy domain, indicating that our method can provide more reliable supervision.

Comparison of Distribution. As shown in Figure 4, the distributions of different methods are almost the same. As the learning goes on, the discrepancy between the positive and negative distributions of ours becomes larger (simultaneously shown in Figure 2) while the others almost no change. A large discrepancy between the positive and negative distributions means that we can filter out as many negative samples while recruiting as many positive samples as possible, which is conducive to model training. This is the fundamental reason behind why our method outperforms other methods.

4 Related Work

Semi-Supervised Learning. Semantic segmentation is a fundamental task that has achieved conspicuous achievements credited to the recent advances in deep neural networks [43-48]. However, its data-driven nature makes it heavily dependent on massive pixel-level annotations, which are laborious and time-consuming to gather. To alleviate the data-hunger issue, considerable works have turned their attention to semi-supervised learning. Designing appropriate and effective supervision signals for unlabelled data is the core problem of semi-supervised learning. Previous research can be summarized into two learning schemes: self-training and consistency regularization. Self-training based methods [49, 12, 10, 50] aim to train the model based on the pseudo-labels generated by the up-to-date optimized model for the unlabelled data. Consistency regularization-based methods aim to obtain prediction invariance under various perturbations, including input perturbation [51, 52], feature perturbation [53] and network perturbation [54-57]. The recently semi-supervised methods, MixMathch [58] and FixMatch [18] combine these two techniques together and achieve state-of-theart performance. Based on the paradigm of existing semi-supervised learning methods, our method explores a better weighting function for the pseudo-label scheme during training.

Semi-Supervised Semantic Segmentation. Semi-supervised semantic segmentation aims at pixellevel classification with limited labeled data. Recently, following the paradigm of semi-supervised learning, many semi-supervised semantic segmentation methods also focus on the design of selftraining [26, 27, 5] and consistency regularization [59, 60, 53, 61] strategies. U 2 PL [5] proposes to

make sufficient use of unreliable pseudo-labeled data. CCT [53] adopts a feature-level perturbation and enforces consistency among the predictions from different decoders. More recently, SOTA semi-supervised segmentation methods also integrate both technologies for better performance. PseudoSeg [7], AEL [27] and UCC [62] propose to use the pseudo-labels generated from weak augmented images to constrain the predictions of strong augmented images. In this paper, we shed light on semi-supervised semantic segmentation based on pseudo-labeling and strive to explore better strategies for using pseudo-labels.

5 Conclusion

In this paper, we propose DAW to systematically analyze the trade-off in previous methods that hinder the model's learning. We formally define the trade-off between inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels by explicitly modeling the confidence distribution of correct and inaccurate pseudo-labels, equipped with a unified weighting function. Experiments show the effectiveness.

6 Acknowledgments

This work was partially supported by the National Defense Basic Scientific Research Program (Grant JCKY2020903B002).

References

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation with prototype-based consistency regularization. Advances in Neural Information Processing Systems , 35:26007-26020, 2022.

Ying Jin, Jiaqi Wang, and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. Advances in Neural Information Processing Systems , 35:2803-2816, 2022.

Lihe Yang, Wei Zhuo, Lei Qi, Yinghuan Shi, and Yang Gao. St++: Make self-training work better for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4268-4277, 2022.

Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4248-4257, 2022.

Jie Qin, Jie Wu, Ming Li, Xuefeng Xiao, Min Zheng, and Xingang Wang. Multi-granularity distillation scheme towards lightweight semi-supervised semantic segmentation. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXX , pages 481-498. Springer, 2022.

Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv preprint arXiv:2010.09713 , 2020.

Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driving. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops , pages 587-597, 2018.

Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review , 54:137-178, 2021.

Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML , volume 3, page 896, 2013.

Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo-labeling: An uncertainty-aware pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06329 , 2021.

Eric Arazo, Diego Ortego, Paul Albert, Noel E O'Connor, and Kevin McGuinness. Pseudolabeling and confirmation bias in deep semi-supervised learning. In 2020 International Joint Conference on Neural Networks (IJCNN) , pages 1-8. IEEE, 2020.

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transformations and perturbations for deep semi-supervised learning. Advances in neural information processing systems , 29, 2016.

Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with pseudo-ensembles. Advances in neural information processing systems , 27, 2014.

Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02242 , 2016.

Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Semi-supervised learning (chapelle, o. et al., eds.; 2006)[book reviews]. IEEE Transactions on Neural Networks , 20(3):542-542, 2009.

Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang. Freematch: Self-adaptive thresholding for semisupervised learning. arXiv preprint arXiv:2205.07246 , 2022.

Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying semisupervised learning with consistency and confidence. Advances in neural information processing systems , 33:596-608, 2020.

Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems , 34:22106-22118, 2021.

Yuyuan Liu, Yu Tian, Yuanhong Chen, Fengbei Liu, Vasileios Belagiannis, and Gustavo Carneiro. Perturbed and strict mean teachers for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4258-4267, 2022.

Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning , pages 1321-1330. PMLR, 2017.

Lihe Yang, Lei Qi, Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation. arXiv preprint arXiv:2208.09910 , 2022.

Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie, Bhiksha Raj, and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semisupervised learning. arXiv preprint arXiv:2301.10921 , 2023.

Zhen Zhao, Sifan Long, Jimin Pi, Jingdong Wang, and Luping Zhou. Instance-specific and model-adaptive supervision for semi-supervised semantic segmentation. arXiv preprint arXiv:2211.11335 , 2022.

Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation matters: A simple-yet-effective approach to semi-supervised semantic segmentation. arXiv preprint arXiv:2212.04976 , 2022.

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2613-2622, 2021.

Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised semantic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems , 34:22106-22118, 2021.

Zicheng Wang, Zhen Zhao, Luping Zhou, Dong Xu, Xiaoxia Xing, and Xiangyu Kong. Conflictbased cross-view consistency for semi-supervised semantic segmentation. arXiv preprint arXiv:2303.01276 , 2023.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-338, 2010.

Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmentation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2613-2622, 2021.

Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Semantic contours from inverse detectors. In 2011 international conference on computer vision , pages 991-998. IEEE, 2011.

Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3213-3223, 2016.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV) , pages 801-818, 2018.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12674-12684, 2020.

Aurélien Lucchi, Kevin Smith, Radhakrishna Achanta, Graham Knott, and Pascal Fua. Supervoxel-based segmentation of mitochondria in em image stacks with learned shape features. IEEE transactions on medical imaging , 31(2):474-486, 2011.

Huayu Mai, Rui Sun, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Dualrel: Semi-supervised mitochondria segmentation from a prototype perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 19617-19626, 2023.

Rui Sun, Naisong Luo, Yuwen Pan, Huayu Mai, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Appearance prompt vision transformer for connectome reconstruction. In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCAI-23 , pages 1423-1431. International Joint Conferences on Artificial Intelligence Organization, 8 2023. Main Track.

Yuwen Pan, Naisong Luo, Rui Sun, Meng Meng, Tianzhu Zhang, Zhiwei Xiong, and Yongdong Zhang. Adaptive template transformer for mitochondria segmentation in electron microscopy images. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21474-21484, 2023.

Rui Sun, Huayu Mai, Naisong Luo, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Structuredecoupled adaptive part alignment network for domain adaptive mitochondria segmentation. In International Conference on Medical Image Computing and Computer-Assisted Intervention , pages 523-533. Springer, 2023.

Xiaoyu Liu, Wei Huang, Zhiwei Xiong, Shenglong Zhou, Yueyi Zhang, Xuejin Chen, Zheng-Jun Zha, and Feng Wu. Learning cross-representation affinity consistency for sparsely supervised biomedical instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 21107-21117, 2023.

Rui Sun, Yuan Wang, Huayu Mai, Tianzhu Zhang, and Feng Wu. Alignment before aggregation: trajectory memory retrieval network for video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1218-1228, 2023.

Yuan Wang, Rui Sun, Zhe Zhang, and Tianzhu Zhang. Adaptive agent transformer for few-shot segmentation. In European Conference on Computer Vision , pages 36-52. Springer, 2022.

Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, and Yongdong Zhang. Lesionaware transformers for diabetic retinopathy grading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10938-10947, 2021.

Huayu Mai, Rui Sun, Yuan Wang, Tianzhu Zhang, and Feng Wu. Pay attention to target: Relation-aware temporal consistency for domain adaptive video semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence , 2024.

Yuan Wang, Rui Sun, and Tianzhu Zhang. Rethinking the correlation in few-shot segmentation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7183-7192, 2023.

Yuan Wang, Naisong Luo, and Tianzhu Zhang. Focus on query: Adversarial mining transformer for few-shot segmentation. In Advances in Neural Information Processing Systems , 2023.

Paola Cascante-Bonilla, Fuwen Tan, Yanjun Qi, and Vicente Ordonez. Curriculum labeling: Revisiting pseudo-labeling for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 6912-6920, 2021.

Bowen Zhang, Yidong Wang, Wenxin Hou, Hao Wu, Jindong Wang, Manabu Okumura, and Takahiro Shinozaki. Flexmatch: Boosting semi-supervised learning with curriculum pseudo labeling. Advances in Neural Information Processing Systems , 34:18408-18419, 2021.

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: a regularization method for supervised and semi-supervised learning. IEEE transactions on pattern analysis and machine intelligence , 41(8):1979-1993, 2018.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. Advances in neural information processing systems , 33:6256-6268, 2020.

Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12674-12684, 2020.

Zhengyang Feng, Qianyu Zhou, Qiqi Gu, Xin Tan, Guangliang Cheng, Xuequan Lu, Jianping Shi, and Lizhuang Ma. Dmt: Dynamic mutual training for semi-supervised learning. Pattern Recognition , 130:108777, 2022.

Zhanghan Ke, Daoye Wang, Qiong Yan, Jimmy Ren, and Rynson WH Lau. Dual student: Breaking the limits of the teacher in semi-supervised learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 6728-6736, 2019.

Hieu Pham, Zihang Dai, Qizhe Xie, and Quoc V Le. Meta pseudo labels. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11557-11568, 2021.

Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. Advances in neural information processing systems , 30, 2017.

David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. Mixmatch: A holistic approach to semi-supervised learning. Advances in neural information processing systems , 32, 2019.

Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semisupervised semantic segmentation with directional context-aware consistency. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 1205-1214, 2021.

Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consistent semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7273-7282, 2021.

Zenggui Chen and Zhouhui Lian. Semi-supervised semantic segmentation via prototypical contrastive learning. In Proceedings of the 30th ACM International Conference on Multimedia , pages 6696-6705, 2022.

Jiashuo Fan, Bin Gao, Huan Jin, and Lihui Jiang. Ucc: Uncertainty guided cross-head co-training for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9947-9956, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies semi-supervised semantic segmentation, where the key challenge is how to select high-quality pseudo-labels. This paper systematically analyzes the trade-off between correct unreliable predictions and wrong confident predictions and theoretically finds the optimal weighting function. Interestingly, by explicitly modeling the confidence distributions of both correct and wrong predictions, the optimal weighting function is simply the hard step function with the jump point located at the intersection of the estimated two distributions. Experiments under various evaluation protocols demonstrate the efficacy of the proposed method.\n\nSTRENGTHS:\n1. This paper is well-written and very easy to follow.\n2. Sufficient details are provided in the supplementary material.\n3. The idea of minimizing $E_1+E_2$ is quite novel.\n4. The proposed method surpasses state-of-the-art alternatives on various benchmarks, especially when the labeled images are extremely limited (e.g., 1/16 protocol on classic VOC).\n\nWEAKNESSES:\nThere are some minor questions.\n1. The detailed operation of the proposed Distribution Alignment is relatively confusing. Does \"$\\cdot$\" in Eq. (12) indicate dot-product? It is better to provide a more detailed explanation of how Eq. (12) operates.\n2. Will more precise estimations contribute to better performances? For instance, will re-computing $(\\mu^{+}, \\sigma^{+})$ and $(\\mu^{-}, \\sigma^{-})$ over the *whole* labeled set after each iteration bring improvements? Or, is there any other technique better than the simple EMA when estimating the parameters of distributions?\n3. I am curious about the performance when applying DAW to other related fields, such as domain adaptive segmentation and semi-supervised classification/detection.\n\nQUESTIONS:\nI do not have any questions. Please refer to the weaknesses section.\n\nLIMITATIONS:\nThe authors **did not** discuss limitations and broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors thoroughly analyze the pixel-level pseudo label selection criteria and the trade-off associated with weighting functions in semi-supervised semantic segmentation. The authors propose a novel method called distribution-aware weighting (DAW) based on their analysis and confidence distribution modeling. The authors present sophisticated mathematical derivations and provide extensive experimental results to demonstrate the effectiveness of their approach. This research makes a valuable contribution to the field of semi-supervised semantic segmentation.\n\nSTRENGTHS:\n1. The authors establish a well-thought-out confidence modeling framework and conduct a comprehensive analysis of the trade-off involved in pseudo-labeling. The mathematical derivations presented in this paper are sophisticated and thoughtful. The theoretical analysis provides valuable insights and guidance for future research.\n2. The theoretical analysis of the trade-off effectively demonstrates the effectiveness of the distribution-aware weighting (DAW) method and explains the reasons behind its superiority over competitor approaches. This mathematical tool not only validates the performance of DAW but also enables subsequent researchers to assess the theoretical performance of semi-supervised learning (SSL) techniques.\n3. In addition to evaluating the effectiveness of this work on commonly used semi-supervised semantic segmentation benchmarks, the authors extend their analysis to include other benchmarks, such as Lucchi.\n\nWEAKNESSES:\n1.\tThe notation in L107, L108 and L109 are not well matched. The authors use N_L and N_U in L107, L108, but write this notation as N_l and N_u in the following.\n\nQUESTIONS:\n1. In the confidence distribution modeling approach proposed by the authors, the normalization factor (\\alpha) for the truncated Gaussian distribution modeling the confidence distribution of pseudo-labels is not explicitly specified in the provided information. How is the value of the normalization factor obtained? \n2. The specific implementation details regarding the semi-supervised semantic segmentation methods employed in the paper have not been mentioned. Which kind of semi-supervised semantic segmentation methods are used in your paper? Teacher-Student scheme? Or other kinds? Please point out detailed implementation.\n3. The authors justify their choice of the Gaussian distribution for confidence distribution modeling based on the principle of maximum entropy and provide a detailed mathematical derivation to support their decision. However, it is worth considering if there are alternative explanations to support their opinion, such as experimental observations or more general assumptions.\n\nLIMITATIONS:\nThe authors have not stated the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper discusses the challenge of utilizing unlabeled data to improve the generalization performance of semantic segmentation models. Existing methods use weighting functions to select pseudo-labels, but face a trade-off between accuracy and utilization. The proposed method, DAW, models the confidence distribution of pseudo-labels and uses a unified weighting function to define the trade-off problem. The authors find the optimal solution to be a hard step function and propose distribution alignment to mitigate prediction distribution discrepancies.\n\nSTRENGTHS:\n1. A new method called DAW is proposed, which models the confidence distribution of correct and inaccurate pseudo-labels and uses a unified weighting function to define the trade-off problem. This method achieves better results in handling pseudo-labels compared to existing methods.\n2. The hard step function and distribution alignment are used to solve the trade-off problem and prediction distribution discrepancy problem.\n3. The writing and presentation of the paper are well-done.\n\nWEAKNESSES:\n1. The premise for the proposed method to work is to correctly estimate the distribution of correct and incorrect pseudo-labels, which is a difficult problem. Firstly, for unlabeled data, it is very challenging to distinguish which pseudo-labels are correct and which are incorrect because there is no ground truth. Secondly, estimating the distribution based on labeled data may not guarantee the effectiveness of all data because in most scenarios, the proportion of labeled data is low.\n2. The paper uses a hard step function as the weighting function. What are the advantages of using a hard step function over a regular step function with a threshold set as a hyperparameter?\n3. From the results of Table 2, 3, it can be seen that the improvement in the final performance in most scenarios is relatively minimal compared to previous works.\n\nQUESTIONS:\nWhat kind of results would we get if we combine the DA method with a manual approach of setting the most appropriate hyperparameters while using the FixMatch method?\n\nLIMITATIONS:\nThe paper does not illustrate any limitations or any potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a systematic analysis of the trade-off involved in previous methods that deal with pseudo-labels. The authors define the trade-off between inaccurate yet utilized pseudo-labels and correct yet discarded pseudo-labels by modeling the confidence distribution of both types of pseudo-labels. They propose a unified weighting function called DAW to minimize the negative impact caused by this trade-off. Interestingly, they find that the optimal solution for the weighting function is a hard step function, located at the intersection of the confidence distributions. Additionally, the authors introduce distribution alignment to address the discrepancy between the prediction distributions of labeled and unlabeled data. The experimental results on various benchmarks, including mitochondria segmentation, demonstrate that DAW outperforms state-of-the-art methods.\n\nSTRENGTHS:\n+ The paper is presented in a clear and concise manner, effectively communicating the motivation and ideas behind the proposed approach. The authors thoroughly analyze the trade-off involved in handling pseudo-labels in previous methods, defining it as the balance between utilizing inaccurate pseudo-labels and discarding correct ones.\n+ The paper provides a rigorous evaluation of the proposed approach through extensive experiments conducted on multiple benchmarks, ensuring a comprehensive assessment of its performance. The results obtained from the experiments showcase the superiority of the proposed approach against state-of-the-art methods.\n\nWEAKNESSES:\n- It is not accurate to state that existingmethods tend to employ certain criteria (weighting function) to select pixel-level pseudo labels. For instance, FlexMatch [40] dynamically adjusts thresholds for different classes at each time step to allow informative unlabeled data and their corresponding pseudo labels to be included, and FreeMatch [17] employs a self-adaptive approach to adjust the confidence threshold based on the learning status of the model.\n- The current state-of-the-art method [a] is not included for comparison in Tables 2-3.\n[a] \"NP-SemiSeg: When Neural Processes meet Semi-Supervised Semantic Segmentation.\" ICML. 2023.\n\nQUESTIONS:\nPlease refer to the Weaknesses.\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper introduces DAW, a method for semi-supervised semantic segmentation that addresses the trade-off between false-positive and false-negative pseudo-labels. DAW utilizes Gaussian functions to model confidence distributions and derives an optimal weighting function based on their inter-section. It also incorporates distribution alignment to reduce the discrepancy between labeled and unlabeled data. Experimental results demonstrate the superiority of DAW over state-of-the-art methods on two benchmarks.

STRENGTHS:
- The author's viewpoint offers valuable insights into the problem of semi-supervised semantic segmentation.
- The technical approach proposed in the paper is described adequately.

WEAKNESSES:
- While the paper demonstrates the efficacy of DAW on certain benchmarks, it would be beneficial to investigate the method's performance on larger datasets such as COCO-Stuff, ADE20K, and COCO to provide a comprehensive evaluation across different scales.
- Table 1 is challenging to interpret without additional illustration of the symbols used in both the table and figure. Further clarification and explanation are needed to enhance readability.
- The English writing in the paper requires further improvement. For example:
    + In line 1, revise "lies how to fully" to "lies in how to fully." 
    + In line 2, rephrase "inaccurate yet utilized pseudo-labels, and correct yet discarded pseudo-labels" to "false positive pseudo-labels" and "false negative pseudo-labels" respectively.

---
Post rebuttal:  
After carefully reviewing the authors' feedback, I have made the decision to maintain my rating, which falls within the realm of borderline acceptance.

QUESTIONS:
Please refer to the Weaknesses.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08