Towards Unbiased Learning in Semi-Supervised Semantic Segmentation
Authors: Rui Sun
Huayu Mai
Wangkai Li
Tianzhu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Semi-Supervised Semantic Segmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Semi-supervised semantic segmentation aims to learn from a limited amount of labeled data and a large volume of unlabeled data, which has witnessed impressive progress with the recent advancement of deep neural networks. However, existing methods tend to neglect the fact of class imbalance issues, leading to the Matthew effect, that is, the poorly calibrated model’s predictions can be biased towards the majority classes and away from minority classes with fewer samples. In this work, we analyze the Matthew effect present in previous methods that hinder model learning from a discriminative perspective. In light of this background, we integrate generative models into semi-supervised learning, taking advantage of their better class-imbalance tolerance. To this end, we propose DiffMatch to formulate the semi-supervised semantic segmentation task as a conditional discrete data generation problem to alleviate the Matthew effect of discriminative solutions from a generative perspective. Plus, to further reduce the risk of overfitting to the head classes and to increase coverage of the tail class distribution, we mathematically derive a debiased adjustment to adjust the conditional reverse probability towards unbiased predictions during each sampling step. Extensive experimental results across multiple benchmarks, especially in the most limited label scenarios with the most serious class imbalance issues, demonstrate that DiffMatch performs favorably against state-of-the-art methods.

Published as a conference paper at ICLR 2025
TOWARDS UNBIASED LEARNING IN SEMI-SUPERVISED SEMANTIC SEGMENTATION
Rui Sun 1*
Huayu Mai Wangkai Li
Tianzhu Zhang
University of Science and Technology of China National Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory (issunrui, mai556, 1wklwk] @maail il ustu. edu. en, tzzhang@ustc edu..ececddudu...... cn
ABSTRACT
Semi-supervised semantic segmentation aims to learn from limited amount of labeled data and a large volume of unlabeled data, which has witnessed impres- sive progress with the recent advancement of deep neural networks. However, existing methods tend neglect the fact class imbalance issues, leading to the Matthew effect, that is, the poorly calibrated model's predictions can be biased towards the majority classes and away from minority classes with fewer samples. In this work, we analyze the Matthew effect present in previous methods that hin- der model Iearning from a discriminative perspective. In light of this background. we integrate generative models into semi-supervised learning, taking advantage their better class-imbalance tolerance. To this end, we propose DiffMatch to formulate the semi-supervised semantic segmentation task conditional dis- crete data generation problem to alleviate the Matthew effect of discriminative solutions from generative perspective. Plus, to further reduce the risk of over- fitting to the head classes and to increase coverage of the tail class distribution. we mathematically derive a debiased adjustment to adjust the conditional reverse probability towards unbiased predictions during each sampling step. Extensive experimental results across multiple benchmarks, especially in the most limited label scenarios with the most serious class imbalance issues, demonstrate that DiffMatch performs favorably against sfate-of-the-art methods. Code available at http////hhhb..co//usssuuuu////// EiMMtt
INTRODUCTION
Machine learning, especially deep learning, has been consistently reported achieve competitive or even superior performance compared to human beings in certain supervised learning tasks (LeCun) jtt al. 2013). In real-world scenarios, however, its data-driven nature makes it heavily dependent on massive annotations, especially at the dense pixel level, which is laborious and time-consuming to gather (taking semantic segmentation as a case study). To alleviate the data-hunger issue, con- siderable works (Wang et al., 2023b Na et al. 2023 Wang et al.) 2023a Liang et al. 2023 Mai jet al., 2023) 2025) have turned their attention to supeevied semantic segmentation in pursuit of bypassing the labeling cost, demonstrating great potential in widespread applications (Siam al. 2018 Asgari Taghanak et al. 202T). Since only limited labeled data is accessible, how to fully exploit large volume of unlabeled data to improve the model's 's generalization performance for robust segmentation is thus extremely challenging. To leverage unlabeled data effectively, pseudo- labeling (Lee et al. 2013; Rizve al., 2222]) and consistency regularization (Sajjadi et 2016 Laine & Aila 2016 have emerged as mainstream paradigms for semi-supervised segmentation. Re- cently, these two paradigms
often assembled in the form of teacher-student scheme (Wang ct al. 2222a; Chen et al. 2023a). In this scheme, the teacher network, with a weakly augmented view, generates pseudo labels guide the counterparts from the student network in the presence of strongly augmented view, following the smoothness assumption (Chapelle et al.) 2009).
"Equal contribution
1 "orresponniig author

---

Published as a conference paper at ICLR 2025
Tue FA
172
133
11.2
13
balldig viplidic
Bo Spa
araroda
cld
01:: 1/0
Heod Pseaaia Count
Classss/ Peradl Labai Count
60 Tall
Figure 1: We count the training samples of each class on Cityscapes (Cordts et al. 2016) under 1/16 partition protocols and compare the proposed DiffMatch with the highly competitive Uni- Match (Yang et al. 2022) in terms of Pseudo Label Count and loU, assuming that the ground truth for unlabeled data is available solely for theoretical analysis purposes. (a) Prediction distribution of head classes. (b) Prediction distribution of tail classes. (c) Performance of head classes. (d) Performance of tail classes. Our DiffMatch strives to mitigate the Matthew effect raised by the class imbalance issue and stands out for head/tail classes.
From the perspective of probabilistic modeling, almost all de facto methods can be unified as dis- criminative models, which directly model the conditional probability of discriminating different values across classes for given pixels of an image (i.e. maximizing posterior probability). Despite yielding promising results, these methods tend to neglect the fact of class imbalance issue (i.e.. long-tain tailed distribution). For example, the pixel count of head class road can be hundreds of times larger than that of tailed class motorcycle in the widely used Cityscapes dataset (Cordts et al.) 2016) as shown in Figure [ This highly skewed distribution can lead to the Matthew effect; that is, the poorly calibrated model's predictions can be biased towards the majority classes and away from mi- nority classes with fewer samples. This is a corollary raised by discriminative models, which only learn decision boundaries between classes while disregarding the underlying distribution. In other words, these methods, by minimizing empirical risk under the assumption of low-density separa- tion, are highly fragile to the number of pixels per class (i.e. class imbalance), leading decision boundaries that can be drastically altered by the majority classes (i.e., confirmation bias (Guo et al.) 2017). This affects the quality of pseudo labels, and then aggressively training with erroneous pseudo labels, in turn, exacerbates the model's 's bias in a self-reinforcing manner, compromising per- formance. For example, UniMatch (Yang et al.) (022) tends to prioritize the head classes in Figure [ (a) over tail classes in Figure [ (b) in terms of pseudo label count compared to real distribution. To make matters worse, the negative impact is inevitably amplified by inbuilt low-data regimes of semi-supe remi-supervised segmentation, hindering the learning process. Then, the question naturally arises: How to effectively alleviate the negative impact raised by class imbalance issue and move towards unbiased learning?
In this work, we analyze the Matthew effect present previous methods that hinder the model's learning when dealing with class imbalance issues from discriminative perspective. Compared
with the discriminative models, the generative models conceptually exhibit better class-imbalance tolerance, attributed to their better asymptotic error approaching rate (Ng & Jordan| 2001) (detailed in Appendix A.. In light of this background, we turn to formulate the semi-supervised semantic segmentation task as conditional discrete data generation problem to model the underlying distri- bution, alleviating the Matthew effect of discriminative solutions from a generative perspective. To this end, we propose DiffMatch learn series of state transitions under the guidance of the input image, transforming noise from a known noise distribution into a prediction that better matches the real distribution, maximizing the mutual information between the learned distribution and the under- lying real one. A heuristic explanation of the transition process is that it can be viewed as the human process of discriminating objects, gradually scrutinizing them closer after an initial glance with the naked eye. By formulating the pseudo-label generation of the teacher-student scheme as an opti- mization problem progressively solved by the denoising diffusion process, DiffMatch favors a better capacity to tackle the severe class imbalance issue in semi-supervised learning. Plus, to further re- duce the risk of overfittiing to the head classes and to increase coverage of the tail class distribution, mathematically derive debiased adjustment based on the state transition function of the diffu- sion process to adjust the conditional reverse probability towards unbiased predictions during each sampling step. This adjustment, formalized an additional regularization term, further unlocks the
2

---

Published as a conference paper at ICLR 2025
potential of DiffMatch mitigate the Matthew effect effectively and is in line with the step-by-step sampling nature of the diffusion model. In practice, tackling class imbalance issue appropriately enables well-calibrated models to generate high-quality pseudo labels (see Figure I (c) & (d)), and
in turn, improved quality of pseudo labels favorably manifests the mitigation of Matthew effect (see Figure [ (a) & (b)), moving the learning toward unbiased.
Extensive experiments across diverse benchmarks spanning different backbones demonstrate that our method performs favorably against state-of-the-art semi-supervised semantic segmentation methods, especially in the most limited label scenarios with the most severe class imbalance issues (e.g., +2.6%/42.0%% compared to DDFP (Wang et al., (024a) and RankMatch (Mai et al. 2024b) respectively on PASCAL classic under 1/16 protocol with the ResNet- 101), evidencing the merits of modeling underlying distribution in the challenging dense pixel-level classification task.
2 RELATED WORK
Class-Imbalanee Semi-Supervised Segmentation. Real-world datasets usually yield a class- imbalanced distribution, especially in dense prediction tasks (e.g., semantic segmentationn) mak- ing the standard training of machine learning models harder to generalize. Existing methods to
re-balance the training objective can be roughly categorized into two paradigms: (1) re-sampling
based methods (Chawla la al.] 2002) [He & Garcial 2009 (yyd & Lipton| 2019) Chang al. 2021
Shi et al.] [023]] Wei et al. 2022) to adjust prediction labels by over-sampling the minority class
or under-sampling the majority class. (2) re-weighting based methods (Cao al. 2019: Cui al.)
2019 Huang et al. 2019; Ren et al., 2018 Hu el al. 2019; Chen el al.j 2023d to influence the loss function conditioned on specific criteria (e.g., imposing the weights by strictly inverse the class frequency). However, all these methods assume all labels are accessible to alleviate the class imbal- ance issue and thus inapplicable to the unlabelled data in semi-supervised semantic segmentation. Recently, several studies have attempted to transfer these techniques on top of pseudo labels such as
re-sampling (Wei et al.| (202I). re-weighting (Wang et al. 2022a Sun et al., 2023c Xu et al. 2021
He et al. 2021 Wang et al. 2022c Peng et al. 2023) (c.g., Adsh (Guo & Li 2022) utilizes adaptive thresholding that can be considered as binary weighting for semi-supervised learning, U'PL (Wang
et al. 2022c) adjusts the threshold adaptively to determine the reliability of pixels and constructs the extra supervised signal based on the negative classes of unreliable pixels, paying more attention to the tail classes), or combination of both for semi-supervised learning (e.g., AEL (Hu et al.] 202T) adaptively balances the training of different categories). Nevertheless, these pseudo labels are often noisy as they are generated from poorly calibrated models. Furthermore, USRN (Guan al. 2022) explores unbiased subclass regularization for alleviating the class imbalance issue. However, these discriminative methods are still confined to learning decision boundaries, which are brittle to the class imbalance issue, and the inherent nature of contempt for the underlying distribution re- mains unchanged. As a significant departure from the status quo, we formulate the semi-super semantic segmentation task as conditional discrete data generation problem to model underlying distribution to overcome the shortcomings of discriminative solutions from a generative perspective.
Diffusion Models for Visual Perception. In addition to the significant progress in content gen- cration, diffusion models have demonstrated potential for perception tasks (Gu et al. 2022 Chen
ct al. 2023c Brempong et al./ 2022). Earlier studies primarily investigate latent representations of diffusion models for zero-shot image segmentation (Baranchuk et al., 2021 Burgert et al. 2022) or
(Wolleb et al., 2022 Wu et al. 2022). Despite substantial progress, the outcomes of these efforts remain limited to specific local designs. DiffusionDet (Chen et al.] 2023b) and DiffusionInst (Gu et al., 2022) explore diffusion models for query-based object detec- tion (Carion et al. 2020) and instance segmentation (Zhang et al. (022]. Recently, several works have introduced diffusion into various semi-supervised tasks, such as classification, federated learn- ing, time-series classification, and 3d object detection. Among them, both DPT (You et al., (024) and FedDISC (Yang et al. 2024) aim to introduce an external diffusion model to generate data and utilize these data in a multi-staee training manner. DiffShape (Liu et al.) 2024) utilizes diffusion in a self-supervised manner to improve representation capability, and Diffusion-ss3d (Ho et al. 2023) exploits the denoising ability of the diffusion to improve the quality of the pseudo label. How- ever, these methods differ from ours both from motivation to implementation. We comprehensively compare DiffMatch with these diffusion based semi-supervised methods in Appendix In gen- eral, DiffMatch completely utilizes the characteristics of the diffusion process for semi cemi-supervised semantic segmentation, aiming provide new perspective to alleviate the Matthew effect.
medical image segmentation
3

---

Published as a conference paper at ICLR 2025
(9) Encodligg
Frewnd Datheioo
Feature Extractor
condiliinn
Foap
ZB
condiiionn
Stadent
Mark Derriiorr
Insubbo
P=Dn-
Figure 3: Conditional discrete data genera- tion pipeline for semi-supervised semantic seg- mentation. Specifically, a conditional diffusion model is employed, where q is the forward dif- fusion process and po is the inverse process.
Figure 2: Our DiflMatch framework, which includes feature extractor g(-) and a mask denoiser f(-). The diffusion process con- ducted progressively in mask denoiser, aiming for lightweightness.
METHOD
In this section, we first formulate the semi-supervised semantic segmentation problem as prelimi- naries (Section [.1].. specify the core framework of DiffMatch (Section 3.2 and Section 3.3) and elaborate the training (Section 3.4) and inference (Section[3.3 details.
3.1 PROBLEM DEFINITION
Given a labeled set D' = {<r), y M==1 denote the number of labeled and unlabeled images, respectively. N > N semi-supervised se- mantic segmentation aims to train a segmentation model with limited labeled data and fully exploit a large volume of unlabeled data. As shown in Figure 2 the popular teacher-studen scheme consists of teacher network and student network. The student network guided by two sources of super- vision, including the ground truth y for the labeled data (yielding supervised loss Leup) and the pseudo labels generated by the teacher network for the unlabeled data (constituting the unsupervised loss ) In specific, for the unlabeled data, the unsupervised loss Luneup is constructed in the form of consistency regularization, that is, the teacher network with a weakly augmented perturba- tion view xx generates pseudo labels 3hth to instruct the counterparts y from the student network under the presence of a strongly augmented perturbation view x."
1 and an unlabeled set D {x;" Ti=1 where N' and N
N"
The teacher network can either be the same the student network average (EMA) version of the student network. Note that in this paper, the teacher and student networks are identical, following UniMatch (Yang al. 2022), ensure simplicity and efficiency. The overall objective is the combination of supervised and unsupervised losses L = Leup + Lunsup-
exponentially moving
In this work, we integrate generative models into semi-supervised learning, taking advantage of its better iaaalai tolerance. In the next section, we elaborate on the modeling of our DiffMatch in detail, that is, how to realize closer collaboration between the diffusion process and the teacher- student paradigm.
3.2
THE DIFEMATCH FRAMEWORK
Figure 2 sheds light on the architecture of generation modeling for proposed DiftMatch. In specific, during training, the Gaussian noise e controlled by noise schedule (Ho et al. 2020) is added to the ground truth y' (from labeled data) or pseudo labels V. (from unlabeled data) to construct the noisy masks. Then, the noisy mask is fused with the pixel embeddings z (acts as the condition) from the feature extractor g(-), and the resulting fused features are fed into lightweight mask denoiser f()) to generate the prediction without noise. At the inference phase, DiffMatch generates predictions by reversing the learned diffusion process, which transforms a known Gaussian distribution into a prediction that better matches the real distribution under the guidance of the images, maximizing the mutual information between the learned distribution and the underlying real one.
Due to the iterative nature of the diffusion sampling process, it requires multiple runs of the model during the inference phase. To minimize computational cost, we separate the entire network into two parts: the feature extractor and the mask denoiser following Chen et al. (2023c): et 2023)
4

---

Published as a conference paper at ICLR 2025
The former forward only to extract the pixel embedding, and then the mask denoiser employs it as the condition rather than the raw image to iteratively reads out the prediction mask.
3.2.1 THE FEATURE EXTRACTOR
The feature extractor g(:) aims to extract the semantic features of the image a and upsample to a high-resolution pixel embedding
# xXxD xD with an FPN-style structure (Lin et al. 2017) for sufficient representations. In our experiments, we adopt DeepLabv3+ (Chen et al. 2018) without the last layer classifier for fair comparison (Na al. 2023 Sun et al. 2023c), where D 256.
3.2.2 THE MASK DENOISER
Denoiser Network. The input of the mask decoder f(-) is the concatenation of the noisy mask yo, which is obtained by adding Gaussian noise E to the ground truth from labeled data (y') or pseudo labels from unlabeled data (WW), and the pixel embedding z from the feature extractor. To further minimize computational cost, we simply stack L layers of deformable attention (Zhu et al., 2020: Ji et al. (202)) the mask denoiser (The number of layers L is set by default. Its effect can be referred to in Table 77 This lightweight design enables efficient reuse shared parameters during multi-step denosing diffusion processes (i.e., after running the feature extractor only once, reuse the efficient denoiser in several iterative steps), while maintaining highly competitive performance. More sophisticated mask denoiser are possible, to leverage recent advances in architecture designs (e.g., TrensUNet (Chen et al. 2021a)), but this is not our main focus so we opt for simplicity.
Forward and Reverse Process. Inspired by non-equilibrium thermodynamies, the optimization goal of the diffusion model is to maximize the likelihood to favor the alignment of the learned dis- tribution and underlying real one. To this end, the diffusion model learns series of state transitions (as shown in Figure 3 to transform noise e (i.e., yr 6) from a known noise distribution into data sample yo from the data distribution p(yo). To learn this mapping. we first define forward transition q (3/: 3/-1) from state yt-1 to more noisy state yo. which is defined as:
y/: vo.yt-1 + v1 agE = q (3: Dt-1) = -N (yts 1o 1, (1 (1-a.) a.)
(I)
where is from uniform density on [0. 1] and is drawn from standard normal density. as denotes the noise schedule (Ho et al. 2020; Song et al. 2020), meaning that the larger the time step the more the noise dominates and finally converges to pure Gaussian noise. Denoting the conditional reverse process p 11+1, 2), the straightforward objective is:
Laifs Σ DKL lg (3/ 31-1) llpo (yx ye+1,
Benefiting from the reparameterization technique, the forward process can be simplified that directly obtaining y, from yo. as:
v a.yo + V1 ā,e = (3: yo) = N (yis v Va.yo. (1 a))
(3)
where 0₁ = IT's Cs.. Similarly, we can learn mask denoiser f(-)) to predict yo directly from y: under the guidance of i.e., f(y,, 2) po (yo ye, 2). The objective can be simplified to:
Lasf = If (ye. 2) yo 12
4)
Note that in our semi-supervised setting, the data samples are either the ground truth mask from labeled data (yo y') or pseudo labels from unlabeled data (yo- )²) In specific, deriving from Equation the supervised loss Loup for labeled data can be formulated as:
Coup ||s (()') -
(5)
In the same way, for the unlabeled data, the unsupervised loss Lunmup can be formulated as:
Lunsup ||f (i", g(x")) D
where Dö = f(e, g(xxt)))) denotes the pseudo labels and s/w means the strong/weak augmenta- tion. Intuitively, the unsupervised loss fits with the consistency regularization of a standard teacher- student paradigm in semi semi-supervised semantic segmentation. In Algorithm we present the pseudo algorithm of DiffMatch clearly summarize method. At this point, we have explored the in- tegration of the diffusion process and the teacher-student paradigm to alleviate the class imbalance issue from generative perspective.
5

---

Published as a conference paper at ICLR 2025
3.3 DEBIASED ADIUSTMENT
Given the long-tailed nature of the class distribution p(yo) in practice, the learned conditional inverse probability Po( Yt: z) is inevitably biased. To further improve the tolerance of the diffusion model to class imbalance, we propose the debiased adjustment. First, we represent the conditional inverse probability under ideal condition * (i.e., when the class distribution is uniform, p" (yo)
where C' is the number of classes) as (y0 Yr. 2). With the Bayesian formula, we deduce the relation between po
Yt. 2) and pa (yo y/. =) (refer to Appendix C for detailed derivation):
Po (y:) 9" (36+1)
pa (y: 9c+1: 2) = Pa (ye yc+1,
Py (y:) á (1++1)
Intuitively, can obtain the ideal conditional inverse probability P (ye y)+1, by modulate Po (ye 3,++1 z) by a factor po(ve) 4" (w+++)) However, directly estimating this modulation factor at each time step t is highly challenging. Instead, we incorporate it into the training loss function to achieve an equivalent objective. Replacing the po( yt, x) in Equation 2 with (3 yt, 2):
p3(3)) o(M..1))
Casss = Σ DKL. lq (ye 11-1) lipt (ye 30+1,2)
Σ { DKL la (z1 11-1) ) Ilpo (Be 13-11, =] + tDKL
Pe (Yt-1 ye) llpt (vi-1 mn)
Ope (Bo)
-Lasss + Σ { tDKL
po (yt-1 y1)
ll2b (y:-1 11)) }
Cpo (Bo)
(8)
In practice, we approximate the po ye) with Monte-Carlo sampling from po yts and the loss reduces to:
[(y Cp (3o)
Cass = If (3/., =) 3011 rt f (y), 2)
where r the trade-off weight for the regularization term, set to 0.1 by default, and C' is the number of classes. Please refer to Appendix foo detailed derivation. Intuitively, the second term imposes constraint directly between the prediction of mask denoiser and its roughly debiased version, reduc- ing the risk of overfitting to the head classes and increasing coverage of the tail class distribution. Based on Equation 9 the supervised loss and unsupervised loss are updated as:
f [3/ 9(x))())))
Coup ||f g(x")) -v6ll + f (sh.
(10)
Cp (vr)
f (Fit. g(x2)))
Lunsup = ||5 (yr g (xx")) Vo. 1|? + 7l (Di g(x)))
(11)
Cp (8%
Note that, in our implementation, p(v() is statistically derived from the ground truth of labeled data while the p(Hö,w) is initialized as p(3() and updated based on its own pseudo label in an exponential moving average (EMA) manner to progressively align the class prior on unlabeled data. By formulating the pseudo label generation of consistency regularization as optimization problem progressively solved by the denoising diffusion process, DiffMatch bridges the gap by drifting biased prediction towards unbiased learning.
3.4 TRAINING
Our main training objective to learn a series of state transitions under the guidance of input image to transform noise from a known noise distribution into prediction that better matches real class distribution. We adopt analog bits encoding strategy (Chen et al., 2022) to first convert discrete integers from ground truth pseudo labels into bit strings, and then cast them real number. When constructing the analog bits, we can shift and scale them into {-b, b} (The scaling factor b follow the
is by default set to 0.1. Its impact can be referred to in Table B) To draw samples, same procedure as sampling in a continuous diffusion model, except that we apply a quantization operation at the end by simply thresholding the generated analog bits. In our implementation, we replace the I.2 loss with the standard cross-entropy loss to hetter suit the segmentation task. The training procedure for the diffusion process is provided in Algorithm

---

Published as a conference paper at ICLR 2025
Table Quantitative results of different SSL methods on PASCAL classic set. We report mloU (%) under various partition protocols and show the improvements over Sup.-only baseline. The best is highlighted in bold.
ResNet-50
ResNet-101
Method
1/16
1/8
1/4
1/2
Full
1/16
1/8
1/4
1/2
Full
(92)
(183)
(366)
(732)
(1464)
(92)
(183)
(366)
(732)
(1464)
Sup.-only
44.0
52.3
61.7
66.7
45.1
55.3
64.8
69.7
73.5
FixMatch
67.3
71.4
73.7
73.0
75.5
77.8
79.2
PCRessdts
70.0
74.7
77.1
78.5
80.7
GTA-Segnnwwttsn
70.0
73.2
75.6
78.4
80.5
ReCojicur22)
64.8
72.0
73.1
74.7
AugSegicveeng
64.2
72.1
76.1
77.4
78.8
71.0
75.4
78.8
80.3
81.3
UniMatchrevees
67.4
71.9
75.3
78.0
79.3
73.5
75.4
78.7
80.2
81.9
NP- Pe-SemiSegnource
72.3
75.7
77.4
DAW
68.5
73.1
76.3
78.6
79.7
74.8
77.4
79.5
80.6
81.5
DDFP
74.9
78.0
79.5
81.2
81.9
RankMatchicwees
71.6
74.6
76.7
78.8
80.0
75.5
77.6
79.8
80.7
82.2
PRCL
71.2
72.2
75.2
76.2
78.3
DiffMateh (Ours)
73.3
75.7
77.9
79.6
81.6
77.5
78.3
80.6
81.5
83.3
A
+29.3
+23.4
+16.2
+12.9
4-8.7
+33.4
+-2300
+11.8
+1188
4-9.8
Table 2: Quantitative results of different SSL methods PASCAL blender set. We report mloU (%) under various partition protocols and show the improvements over Suplonnly baseline.
ResNet-50 1/8 (1323)
ResNet- 101 1/8 (1323)
Method
1/16 (662)
1/4 (2646)
1/16 (662)
1/4 (2646)
Sup.-onfy
62.4
68.2
72.3
67.5
71.1
74.2
FixMutchiNaaPPS
70.6
73.9
75.1
74.3
76.3
76.9
AEL/Nuat
77.2
77.6
78.1
PCRnieotS
78.6
80.7
80.8
GTA Seg Segiowars
77.8
80.5
80.6
Augeegienwazy
74.7
76.0
77.2
77.0
77.3
78.8
UniMatchueses.co
75.8
76.9
78.1
78.4
79.2
CHCG
75.0
77.1
77.7
76.8
79.1
79.9
NP- NP-SemiSSguuster
73.4
76.5
76.7
DAW (onatterr1)
76.2
77.6
77.4
78.5
78.9
79.6
DDFP,s
78.3
78.8
79.8
RankMutcheessea
76.6
77.8
78.3
78.9
79,2
80.0
77.9
79.1
79.9
DiffMatch (Ours)
77.9
78.7
79.0
80.3
814
A
+15.5
4.10.5
46.77
412.8
410.3
47.4
3.5 INFERENCE
At the inference phase, the target data sample Yo reconstructed from noise yr with the mask denoiser (()) and an updating rule (Song al. 2020 Ho et al. (020)) in an iterative Markovian way. We choose the DDIM update rule (Song al. 2020) for the sampling process. We also represent the trade-off between performance and computation by different sampling steps for multi-step inference in Table Please refer to Algorithm3]Jor details about the sampling procedure for diffusion process. Note that reduce inference overhead, do not employ any post-processing techniques, such as self-condition (Chen et al. 2022), and sampling drift (Ji et al. 2023), etc.
EXPERIMENTS
In this section, we give comprehensive evaluations of various class-imbalanced datasets. We first describe the experimental setups in Section 4.1 Then, present the empirical results of our Diff- Match and other compared competitors under extensive setups in Section 4.2 Finally, we present detailed analyses help understand our method in Section 4.
4.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments three datasets with severe class- dlss-imbalanced issues. (1) PAS- CAL VOC 2012 (Everingham et al. 2010)) contains 21 classes with 1,464 and 1,449 finely annotated images for training and validation, respectively. We augment the original training set (i.e, classic)

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative results of different SSL methods on Cityscapes. We report mIoU (%) under various partition protocols and show the improvements over Sup.-only baseline.
ResNet-50
ResNet-101
Method
1/16 (186)
1/8 (372)
1/4 (744)
(1488)
1/16 (186)
(372)
1/4 (744)
1/2 (1488)
Sup.-only
63.3
70.2
73.1
76.6
66.3
72.8
75.0
78.0
FixMatch
72.6
75.7
76.8
78.2
74.2
76.2
77.2
78.4
AEL (Neat19311
74.0
75.8
76.2
75.8
77.9
79.0
80.3
PCR CiNaaas 22]
73.4
76.3
78.4
79.1
GTA-Seu (210
09.4
72.0
76.1
AugSegrover CVP98223
73.7
76.3
78.8
79.3
75.2
79.5
80.4
UniMatch
75.0
76.8
77.5
78.6
76.6
79.2
79.5
Co- To--Triinovee
76.3
77.1
75.0
77.3
78.7
NP
UP-SSeiiSeeiiolee
73.0
77.1
78.8
78.7
DAW (1c2
75.2
77.5
79.1
79.5
76.6
78.4
79.8
80.6
DDFP (40
77.1
79.8
80.8
RankMatchieveess
75.4
77.7
79.2
79.5
77.1
PRCLUCY)34
73.4
77.0
77.9
80.0
DiffMatch (Ours)
76.5
78.3
79.8
80.0
77.8
79.1
80.5
81.3
1
+13.2
+8..1
+.6.7
+3.4
+11.5
+66.3
+55.5
+.33
with additional 9,118 coarsely annotated images in SBD Hariharan et al.
2011 to get blender
training set following other researches (Chen al. 2021b Hu et 2021] According to statistics, the pixel number of the head class background is Cityscapes (Cordts et al., 2016) consists of 2,975 images for training and 500 images for validation with 19 classes. The ratio of head class road to tail class motorcycle reaches 400. (3) COCO (Lin jet al. 2014), composed of 1 18k/5k training/vale training/validation images, is more severe mass-immallacee dataset, containing 81 classes predict, with over 10, 000 head-to-tail ratio.
than x that of the tail class bicycle. (2)
(a) UniMatch
(b) DiflMatch
(a) UniMatch
(b) DiffMatch
10
Gips
0.0
0.6
Thuhhtale
0.4
B4
ECE: 0.20
n2
ECE: 0.10
0.0
02
04
1.0
CarGdenne
Conidence
Lood
ProketLadd
Figure Calibration on unlabeled data pro- duced by UniMatch (left) and DiffMatch (right).
Figure 5: Confusion matrix on unlabeled data of UniMatch (left) and DiffMatch (right).
Implementation Details. For a fair and exhaustive comparison, we use ResNet-50/101 (He al., 2016) pretrained on ImageNet (Krizhevsky et al. 2012) and Xception-65 (Chollet) (017) as the backbones and DeepLabv3+ (Chen al.) 2018) as the decoder. We set the sampling step as inference, the number of layers in mask denoiser L experiments. During training, we randomly crop 513 x 513 for PASCAL and COCO datasets, and train 80 and 30 epochs, respectively. For Cityscapes, the cropsize set 801 801 and the training epoch is 240. The batch size the three datasets set to 8. Polynomial Decay learning rate policy applied throughout the whole training. The strong augmentation contains feature dropout, random color jitter, grayscale and Gaussian blur. The weak augmentation consists of random crop. resize and horizontal flip. All experiments are conducted 8x RTX 3090 GPUs (memory 24G/GPU),
and the scaling factor b as 0.1 for all
4.2 EMPIRICAL RESULTS
We evaluate our method on PASCAL (classic and blender), Cityscapes datasets with ResNet-50/101, and COCO dataset with Xception-65 under different semi-supervised learning settings (i.e., parti- tion protocols). The partition protocol (e.g., 1/16) indicates the ratio of labeled data used in training to the entire training set. It is worth noting that the smaller the partition protocol, the less labeled data used for training, and the more biased the training may be. The consistently dominant perfor- mance under all partition protocols with different backbones all datasets against other competi-
8

---

Published as a conference paper at ICLR 2025
tors (FixMatch (Sohn et al. 2020), PseudoSeg (Zou et al. 2020)
AEL (Hu et al. 2021). ReCo (Liu
et al.
2021). PC2Seg (Zhong et al. 2021 PCR (Xu et al. 2022). GTA-Seg (Jin et al.
2022
UniMatch
Yang et al., 2022), AugSeg (Zhao et al. 2023c). SemiSee (Wang et al. 2023c
DAW (Sun et al., 2023c). CFCG (Li et al., 2023a). Co- Train (Li et al. 2023b), MKD Yuan et al.,
2023). DDFP (Wang et al.) 2024a), MankMatch (Mai et al. 2024b), PRCL (Xie et al. (224 proves
the effectiveness of our DiftMatch against the class imbalance issue, evidencing the merits of mod-
eling underlying distribution in the challenging dense pixel-level classification task.
Results on PASCAL. Table I and Table the comparison of our method with the SOTA meth- ods on PASCAL classic and blender set. Compared with the supervised-only (Sup.-only) model, our method achieves considerable performance improvements, demonstrating that the information in un- labeled data is effectively utilized in our method. Moreover, in the label-scarce scenario, egg. 1/16 (92) in PASCAL classic, our approach achieves 73.3% and 77.5% mloU with the backbone ResNet- 50 and ResNet- 101, boosting the SOTA DAW (Sun al. 2023cb by 4.8% and 2.7%, respectively. These superior results prove that our training is more unbiased.
Results on Cityscapes. Table summarizes the performance of our DiffMatch and compared meth- ods on the Cityscapes dataset. For the more class-imbalanced dataset (the ratio of head class road to tail class motorcycle reaches 400), our method still achieves SOTA performance. Specifically, com- pared with the leading methods DAW (Sun al. 2023c), DiffMatch improves up to 1.3% 1..% at absolute mloU gain under 1/16 partition protocols with ResNet-50/ResNet-100 respectively, show- ing the superiority of our method over discriminative methods.
Results on COCO. COCO is a large-scale dataset where the class imbalance issue is most severe (the number of head-to-tail ratio is more than 10,000). In Table 4 DiffMatch achieves surprising performance lift compared with the discriminative model. For example, under the 1/512 partition protocol, the performance of DiffMatch is superior that of UniMatch (Yang et al. 2022) (34.6% vs. 31.9%), this is in line with the goal of DiffMatch against class imbalance issue.
4.3 DETAILED ANALYSES
Performance in Head& Tail Classes. Considering that the Matthew effect refers to the bias in model predictions, can also be viewed measure of model calibration. This directly impacts the quality of pseudo-labels for unlabeled data, thereby affecting the model's performance across different classes. Therefore, we compare DiffMatch to other competitive methods to analyze the effectiveness of addressing class imbalance by examining the performance of head/tail classes. To show the source of our absolute performance gain, we present the mloU of the top-5 classes (mloU,,) and the bottom-5 classes (mloU,) under PASCAL classic 1/16 (92) with ResNet-50. To make com- prehensive comparison, we reproduce several methods under the same experiment setting, including class-imbalanced learning methods + SSL methods A. and recently proposed class-imbalanced SSL methods & (1) Specifically, for class imsallncea learning, we consider the two most popular
paradigms: a) Re- Ra--annling (Byrd & Lipton| 2019) and b) Re- de-weiihting (Cui al. 2019); (2) for
SSL methods, we take Fixmatch (Sohn et al. 2020). ReCo (Liu et al. 2021) and NP-SemiSeg (Wang et al., 2023c) into consideration. (3) To further show the efficacy of our proposal, also compare with the recently proposed algorithms that consider SSL and class imbalance issues simultancously,
including DARP (Kim et al. 2020). CReST (Wii al. 2021). FreeMatch (Wang et al. (022a))
DARS He et al. 2021). AEL
(Hu et al., 2021 U (Wang et al. 2022c) and USRN (Guan et al.
2022
Please refer to Section
2 for more details.
As depicted in Table 6 we have the following findings: (1) It is not desirable to directly apply the dlas--mbalanced learning method to SSL tasks because it does not utilize unlabeled data. (2) SSL methods achieve certain performance gains, but still underperform in the tail classes. (3) Thanks to the modeling of distributions and the derived debiased adjustment, DiftMatch yields favorable performance especially in the tail classes, effectively alleviating the Matthew effect. To better un- derstand the prediction bias of each class, as Figure @illustrates. DiffMatch achieves more unbiased predictions on all 21 classes. Moreover, we provide training curves for the number of pseudo labels in the head (road) and tail (motorcycle) classes in the Appendix D demonstrating the effectiveness of our DiffMatch in mitigating the Matthew effect.
Accuracy vs. Efficiency. We show the dynamic trade-off of DiffMatch between accuracy and ef- ficiency in Table 5 To begin with, construct discriminative baseline (Dis. Baseline) with the

---

Published as a conference paper at ICLR 2025
Table 4:
Quantitative results of different SSL methods on COCO.
Table 5: Accuracy vs. Efficiency.
Method
mloU (92)
mIoU (1464)
FPS
#Puram
Mothod
1/512
1/256
1/128
1/64
1/32
(t)
Sup--nny
22.9
28.0
33.6
37.8
42.2
UniMatch
67.4
79.3
24.9
40.5M
Dis. Baseline
67.9
79.5
44.9M
PseudoSeg
29.8
97.1
39.1
41.8
43.6
DiffMatch w/o adj.
72.2
81.3
44.9M
PC2Seg
29.9
37.5
10.1
13.7
46.1
MKD
30.2
38.0
42.3
45.5
47.3
step1
68.7
79.9
23.3
step2
71.2
8U.7
21.2
UniMatch
31.9
35.9
41.4
48.2
49.8
DiffMatch
step3
73.3
81.6
19.6
44.9M
DiffMatch (Ours)
34.6
41.9
47.2
19.8
52.4
step4
73.3
81.4
18.2
+11.7
-3.0
13.8
+12.0
+10.2
step5
73.4
81.7
16.9
cDPMab -AC ofweeen
Table 6: Performance of head & tail classes.
lodeweal
tstaaoo
PASCAL classic 1/16 (92) mloU,
ResNet-50
mloU
mloU,
Sup.-onty
44.0
66.5
28.1
Re-Sampling
45.6
67.8
29.3
Re-weighting
46.2
68.3
30.1
FixMatch
60.1
78.4
48.4
ReCo
61.88
81.2
49.6
NP-SemiSeg
65.8
82.7
50.2
+ DARP
61.5
79.9
49.0
CReST
62.2
80.6
49.1
FreeMatch
62.3
80:2
49.1
DARS
62.7
82.5
50.3
AEL
66.3
84.2
51.1
P
67.4
85.3
53.7
USRN
66.8
83.9
51.8
Figure 6: Normalized performance on PASCAL classic 92 for per class.
DiffMatch (Ours)
73.3
66.8
same extra deformable attention layers. (1) Comparing the 1st and 2nd rows, we can see that simply increasing the number of parameters in the model does not lead to an effective performance im- provement. Then, 2nd vs. 3rd indicates that the performance improvement of DiffMatch primarily stems from modeling the underlying distribution, as opposed to discriminative models (Dis. Base- line). (2) Comparing the 3nd row (DiffMatch w/o adj.) and the final DiffMatch, we can observe clear performance lift credited to debiased adjustment. This suggests the effectiveness of debiased adjustment adjust the conditional reverse probability, reducing the risk of overfitting the head classes and increasing coverage of the tail class distribution.
(3) With the sampling step increase, the performance gets better (same result can also be observed in Figure 1O). When adopting 3 sampling steps, the performance is further boosted while maintaining comparable FPS. These results show that DiffMatch can iteratively infer multiple times with reasonable time cost.
Quality of Pseudo Label. To take close look at DiffMatch, we showcase the confusion matrix (Figure and expected calibration error (Figure on unlabeled data to directly measure the perfor- mance of different models in the Matthew effect and model calibration respectively, on Cityscapes 1/16 partition. The results show that the raw pseudo-labels generated by UniMatch are biased to- ward the major classes. For example, there are more than 20% examples that belong to class wall are predicted wrongly as class building. On the contrary, our DiffMatch can achieve a more unbi- ased confusion matrix, striving to mitigate the Matthew effect. These results indicate that the quality of pseudo-labels is actually improved, which can help to improve the generalization performance. Similarly, better-calibrated model is obtained thanks the modeling of the underlying distribution by DiffMatch (Figure 4. Based on this, well-calibrated models will generate high-quality pseudo labels, and in turn, improved quality of pseudo labels could result in a better distribution estimation.
CONCLUSION
In this paper, we analyze the Matthew effect in previous methods that hinder model learning when dealing with class imbalance issues from discriminative view. we propose Math to formulate the semi-supervised semantic segmentation task as conditional discrete data generation problem to model underlying distribution against the Matthew effect. DiffMatch offers fresh generative perspective to alleviating class imbalance, and we believe it has the potential to complement other semi-supervised learning strategies to facilitate future advancements.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was partially supported by the National Defense Basic Scientific Research Program (Grant (CCamJJCKY02110018811))))
REFERENCES
Saeid Asgari Taghanaki, Kumar Abhishek, Joseph Paul Cohen, Julien Cohen-Adad, and Ghassan Hamarneh. Deep semantic segmentation of natural and medical images: a review. Artificial Intelligence Review, 54:137-178, 2021.
Yunhao Bai, Duowen Chen, Qingli Li, Wei Shen, and Yan Wang. Bidirectional copy-paste for semi-supervised medical image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11514-11524, 2023.
Wele Gedara Chaminda Bandara and Vishal M Patel. Revisiting consistency regularization for semi- supervised change detection in remote sensing images. arXiv preprint urXiv:2204.0888 2022.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint 22.0:322122.3//222 2021.
Emmanuel Asiedu Brempong, Simon Kornblith, Ting Chen, Niki Parmar, Mathhis Minderer, and Mohammad Norouzi. Denoising pretraining for semantic segmentation. Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pp. 4175-4186, 2022.
Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peekaboo: Text to image diffusion models are zero-shot segmentors. arXiv preprint arXiv:2211. 2022.
Jonathon Byrd and Zachary Lipton. What is the effect of importance weighting in deep learning? In International conference on machine learning, pp. 872-881. PMLR, 2019.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distribution--aar margin loss. Advances in neural information processing systems, 32, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference computer vision, pp. 213-229. Springer, 2020.
Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M Alvarez. Image-level or object-level? a tale of two resampling strategies for long-tailed detection. In International conference on machine learning, pp. 1463-1472, PMLR, 2021.
Olivier Chapelle, Bernhard Scholkopf, and Alexander Zien. Sepi-super learning (chapelle, o. et al., eds.; 2book reviews). IEEE Transactions on Neural Networks, 20(3):542-542, 2009.
Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357. 2002.
Hao Chen, Ran Tao, Yue Fan, Yidong Wang, Jindong Wang, Bernt Schiele, Xing Xie. Bhiksha Raj. and Marios Savvides. Softmatch: Addressing the quantity-quality trade-off in semi-supervised learning. arXiv preprint arXiv:2301.10992. 2023a.
Jieneng Chen, Yongyi Lu, Qihang Yu, Xiangde Luo, Ehsan Adeli, Yan Wang, Le Lu, Alan L Yuille, and Yuyin Zhou. Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.043000 2021a.
I Liang-Chie Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference computer vision (ECCV), pp. 801-818, 2018. 2018.
Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 19830-19843, 2023b.

---

Published as a conference paper at ICLR 2025
Ting Chen, Ruixiang Zhang, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with self-conditioning. arXiv preprint 0rXiv:2208.04202 2022.
Ting Chen, Lala Li, Saurabh Saxena, Geoffrey Hinton, and David Fleet. A generalist framework for panoptic segmentation of images and videos. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 909-919, 2023c.
Xiaohua Chen, Yucan Zhou, Dayan Wu, Chule Yang, Bo Li, Qinghua Hu, and Weiping Wang- Area: adaptive reweighting via effective area for long-tailed classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19277-19287, 2023d.
Xiaokang Chen, Yuhui Yuan, Gang Zeng, and Jingdong Wang. Semi-supervised semantic segmen- tation with cross pseudo supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2-222, 2021b.
Yujia Chen, Rui Sun, Wangkai Li, Huayu Mai, Naisong Luo, Yuwen Pan, and Tianzhu Zhang. Alleviate and mining: Rethinking unsupervised domain adaptation for mitochondria segmentation from pseudo-labe perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025.
Zenggui Chen and Zhouhui Lian. Semi-supervised semantic segmentation via prototypical con- trastive learning. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 6696-6705, 2022.
Hanyang Chi, Jian Pang. Bingfeng Zhang, and Weifeng Liu. Adaptive bidirectional displacement for semi-supervised medical image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4070-4080. 2024.
François Chollet. Xception:
Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3213 2213-3223, 2016.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 702-703, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268 0088-9977,27 2019.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88: 303-338, 2010.
Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. Diffu- sioninst: Diffusion model for instance segmentation. arXiv preprint arXiv:2212.00277., 2022.
Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Unbiased subclass regularization for semi- supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9968-9978, 2022.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger. On calibration of modern neural networks. In International conference on machine learning, pp. 1321-1330. PMIR, 2017.
Lan-Zhe Guo and Yu-Feng Li. Class innalimmbled semi-supervised learning with adaptive threshold- ing. In International Conference Machine Learning, pp. 8082-8094, PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Bharath Hariharan, Pablo Arbeláez, Lubomir Bourdev, Subhransu Maji, and Jitendra Malik. Se- mantic contours from inverse detectors. In 2011 international conference on computer vision, pp- 991-998. IEEE, 2011.
Haibo He and Edwardo A Garcia. Learning from imbalanced data, IEEE Transactions on knowledge and data engineering, 21(9): (19):1263-1284 2009.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ruifei He, Jihan Yang. and Xiaojuan Qi. Re-distributing biased pseudo labels for semi-supervised semantic segmentation: A baseline investigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6930-6940, 2021.
Cheng-Ju Ho, Chen-Hsuan Tai, Yen-Y Yu Lin, Ming-Hsuan Yang, and Yi-Hsuan Tsai. Diffusion- ss3d: Diffusion model for semi-supervised 3d object detection. Advances in Neural Information Processing Systems, 36:49100 119910004911 12, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6855 2020.
Prantik Howlader, Srijan Das, Hieu Le, and Dimitris Samaras. Beyond pixels: Semi-supervised
semantic segmentation with a multi-scale patch-based multi-label classifier. In European Confer- ence on Computer Vision, pp. 342-360. Springer, 2025a.
Prantik Howlader, Hieu Le, and Dimitris Samaras. Weighting pseudo-labels via high-activation feature index similarity and object detection for semi-supervised segmentation. In European Conference on Computer Vision, pp. 456 4744. Springer, 2025b.
Lukas Hoyer, David Joseph Tan, Muhammad Ferjad Nacem, Luc Van Gool, and Federico Tombari. Semivl: semi-supervised semantic segmentation with vision-language guidance. In European Conference on Computer Vision, pp. 257-275. Springer, 2025.
Hanzhe Hu, Fangyun Wei, Han Hu, Qiwei Ye, Jinshi Cui, and Liwei Wang. Semi-supervised seman- tic segmentation via adaptive equalization learning. Advances in Neural Information Processing Systems, 34:22106-2211 2021.
Zhiting Hu, Bowen Tan, Russ R Salakhutdinov, Tom M Mitchell, and Eric P Xing. Learning data manipulation for augmentation and weighting. Advances in Neural Information Processing Sys- tems, 32, 2019.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Deep imbalanced learning for face recognition and attribute prediction. IEEE transactions on pattern analysis and machine intelli- gence, 42(1 (((11):27812777)
2019.
Yuanfeng Ji, Zhe Chen, Enze Xie, Lanqing Hong, Xihui Liu, Zhaoqiang Liu, Tong Lu, Zhenguo Li, and Ping Luo. Ddp: Diffusion model for dense visual prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21741-21752, 2023.
Ying Jin, Jiaqi Wang, and Dahua Lin. Semi-supervised semantic segmentation via gentle teaching assistant. Advances in Neural Information Processing Systems, 35:2803-2816, 2022.
Jachyung Kim, Youngbum Hur, Sejun Park, Eunho Yang, Sung Ju Hwang, and Jinwoo Shin. Dis- tribution aligning refinery of pscudo-label for imbalanced uui-suuereised learning. Advances in neural information processing systems, 33: 13:14677-4557 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Xin Lai, Zhuotao Tian, Li Jiang, Shu Liu, Hengshuang Zhao, Liwei Wang, and Jiaya Jia. Semi- supervised semantic segmentation with directional context-aware consistency. In Proceedings of the IEFE/CVF Conference Computer Vision and Pattern Recognition, pp. 1205-1214, 2021.
13

---

Published as a conference paper at ICLR 2025
Samuli Laine and Timo Aila. Temporal ensembling for semi-supervised learning. arXiv preprint arXiv:1610.02222, 2016.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. 2015.
Deep learning. nature, 521((553):436 444,
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Shuo Li, Yue He, Weiming Zhang, Wei Zhang. Xiao Tan, Junyu Han, Errui Ding, and Jingdong Wang. Cfeg: Semi-supervised semantic segmentation via cross-fusion and contour guidance supervision. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023a.
Wangkai Li, Rui Sun, and Tianzhu Zhang. A universal degradation-based bridging technique for domain adaptive semantic segmentation. arXiv preprint arXiv:2412.10339, 2024a.
Yijiang Li, Xinjiang Wang, Lihe Yang, Litong Feng, Wayne Zhang, and Ying Gao. Diverse cotrain- ing makes strong supi-supe segmentor. arXiv preprint arXiv:2308. 092811,0..90.00.. 2023b.
Zhaoyang Li, Yuan Wang, Wangkai Li, Rui Sun, and Tianzhu Zhang. Localization and expansion: A decoupled framework for point cloud few-shot semantic segmentation. In European Conference on Computer Vision, pp. 18-34. Springer, 2024b.
Chen Liang, Wenguan Wang, Jiaxu Miao, and Yi Yang. Logic-induced diagnostic reasoning for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pp. 16197-16208, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Shikun Liu, Shuaifeng Zhi, Edward Johns, and Andrew Davison. Bootstrapping semantic segmen- tation with regional contrast. arXiv preprint 9rXiv:2104.04465. 2021.
Zhen Liu, Wenbin Pei. Disen Lan, and Qianli Ma. Diffusion Language-shapelee for semi semi-supervised time-series classification. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 38, pp. 1479--14077 2024.
Naisong Luo, Rui Sun, Yuwen Pan, Tianzhu Zhang, and Feng Wu. Electron microscopy images as set of fragments for mitochondrial segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 3981-3989, 2024.
Huayu Mai, Rui Sun, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Dualrel: Semi-supervised mito- chondria segmentation from a prototype perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19617-19626, 2023.
Huayu Mai, Rui Sun, Yuan Wang, Tianzhu Zhang, and Feng Wu. Pay attention to target: Relation- aware temporal consistency for domain adaptive video semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024a.
Huayu Mai, Rui Sun, Tianzhu Zhang, and Feng Wu. Rankmatch: Exploring the better consistency regularization for remi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 3391-3401, 2024b.
Huayu Mai, Rui Sun, and Feng Wu. Relaxed class-consensus consistency for semi-supervised se- mantic segmentation. In Proceedings of the AAAI Conference Artificial Intelligence, 2025.
14

---

Published as a conference paper at ICLR 2025
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314 2020.
Sudhanshu Mittal, Maxim Tatarchenko, and Thomas Brox. Semi-supervised semantic segmenta- tion with high-and low-level consistency. IEEE transactions pattern analysis and machine intelligence, 43(4):1339-1133 2019.
Jaemin Na, Jung-Jnnnnnn Woo Ha, Hyung Jin Chang, Dongyoon Han, and Wonjun Hwang. Switching tem- porary teachers for semi-supervised semantic segmentation. arXiv preprint arXiv:2310.18644 2023.
Andrew Ng and Michael Jordan. On discriminative vs. generative classifiers: A comparison of logistic regression and naive bayes. Advances in neural information processing systems, 14, 2001.
Yassine Ouali, Céline Hudelot, and Myriam Tami. Semi-supervised semantic segmentation with cross-consistency training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12674-12684, 2020.
Yuwen Pan, Naisong Luo, Rui Sun, Meng Meng. Tianzhu Zhang, Zhiwei Xiong, and Yongdong Zhang. Adaptive template transformer for mitochondria segmentation in electron mieroscopy images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp-
21474-21484, 2023.
Yuwen Pan, Rui Sun, Naisong Luo, Tianzhu Zhang, and Yongdong Zhang. Exploring reli- able matching with phase enhancement for night-time semantic segmentation. arXiv preprint arXiv:2408. 13838, 2024.
Hanyu Peng, Weiguo Pian, Mingming Sun, and Ping Li. Dynamic re-weighting for long-tailed semi-supervised learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 6464 6474, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748 8763. PMLR, 2021.
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp. 4334 4343. PMLR, 2018.
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An uncertainty-awwr pseudo-label selection framework for semi-supervised learning. arXiv preprint arXiv:2101.06322 2021.
Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with stochastic transfor- mations and perturbations for deep semi-supervised learning. Advances in neural information processing systems, 29, 2016.
Min Shi, Tong Wei, Yuke Xiang, and Yu-Feng Li. How re-sampling helps for long-tail learn- ing? Advances in Neural Information Processing Systems, 36, 2023.
Mennatullah Siam, Mostafa Gamal, Moemen Abdel-Razek, Senthil Yogamani, Martin Jagersand, and Hong Zhang. A comparative study of real-time semantic segmentation for autonomous driv- ing. In Proceedings of the IEEE conference on computer vision and pattern recognition work- shops, pp. 587-597, 2018.
Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying cemi-supervised learning with consistency and confidence. Advances in neural information processing systems, 33:596-608, 2020.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
15

---

Published as a conference paper at ICLR 2025
Boyuan Sun, Yuqi Yang, Le Zhang, Ming-Ming Cheng, and Qibin Hou. Cormatch: Label propaga- tion via correlation matching for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 3097-3107, 2024.
Rui Sun, Yihao Li, Tianzhu Zhang, Zhendong Mao, Feng Wu, and Yongdong Zhang. Lesion-aware transformers for diabetic retinopathy grading. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10938-10947, 2021.
Rui Sun, Naisong Luo, Yuwen Pan, Huayu Mai, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Appearance prompt vision transformer for connectome reconstruction. In ICCAI, pp. 1423-1431, 2023a.
Rui Sun, Huayu Mai, Naisong Luo, Tianzhu Zhang, Zhiwei Xiong, and Feng Wu. Structure- decoupled adaptive part alignment network for domain adaptive mitochondria segmentation. In International Conference on Medical Image Computing and Computer-Assise Intervention, pp. 523-533. Springer, 2023b.
Rui Sun, Huayu Mai, Tianzhu Zhang, and Feng Wu. Daw: Exploring the better weighting function for semi-supervised semantic segmentation. Shirty-seventh Conference on Neural Information Processing Systems, 2023c.
Rui Sun, Yuan Wang, Huayu Mai, Tianzhu Zhang, and Feng Wu. Alignment before aggregation: tra- jectory memory retrieval network for video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1218-1228, 2023d.
Changqi Wang, Haoyu Xie, Yuhui Yuan, Chong Fu, and Xiangyu Yue. Space engage: Collaborative space supervision for contrastive-base semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 931-942, 2023a.
Jia-Xin Wang, Si-Bao Chen, Chris HQ Ding, Jin Tang, and Bin Luo. Ranpaste: Paste consistency and pseudo label for semisupervised remote sensing image semantic segmentation. IEEE Trans- actions on Geoscience and Remote Sensing, 60:1-16, 2021.
Jianfeng Wang, Daniela Massiceti, Xiaolin Hu, Vladimir Pavlovic, and Thomas Lukasiewicz. Np- semiseg: when neural processes meet semi-supervised semantic segmentation. In International Conference on Machine Learning, pp.
PMLR, 2023b.
Jianfeng Wang, Daniela Massiceti, Xiaolin Hu, Vladimir Pavlovic, and Thomas Lukasiewicz. Np- semiseg: when neural processes meet supersuper semantic segmentation. In International Conference on Machine Learning, pp. (6118-361155 PMLR, 2023c.
Xiaoyang Wang, Huihui Bai, Limin Yu, Yao Zhao, and Jimin Xiao. Towards the uncharted: Density- descending feature perturbation for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3303-3312, 2024a.
Yidong Wang, Hao Chen, Qiang Heng, Wenxin Hou, Marios Savvides, Takahiro Shinozaki, Bhiksha Raj, Zhen Wu, and Jindong Wang- Freematch: Self-adaptive thresholding for semi-supervised learning. arXiv preprint arXiv:2005.07246, 2022a.
Yuan Wang, Rui Sun, Zhe Zhang, and Tianzhu Zhang. Adaptive agent transformer for few-shot segmentation. In European Conference on Computer Vision, pp. 36-52. Springer, 2022b.
Yuan Wang, Naisong Luo, and Tianzhu Zhang. Focus on query: Adversarial mining transformer for few-shot segmentation. Advances in Neural Information Processing Systems, 36:31524-31542, 2023d.
Yuan Wang, Rui Sun, and Tianzhu Zhang. Rethinking the correlation in few-shot segmentation: A buoys view. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7183-7192, 2023e.
Yuan Wang, Rui Sun, Naisong Luo, Yuwen Pan, and Tianzhu Zhang. Image-to-ime matching via foundation models: A new perspective for open-vocabulay semantic segmentation. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3952-3963, 2024b.
16

---

Published as a conference paper at ICLR 2025
Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei. Wei Li, Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Le. Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 4248 4257, 2022c.
Chen Wei, Kihyuk Sohn, Clayton Mellina, Alan Yuille, and Fan Yang. Crest: A class- rebalancing self-training framework for imbalanced semi-supervised learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pp. 10857-10866, 2021.
Hongxin Wei, Lue Tao, Renchunzi Xie, Lei Feng, and Bo An. Open-sampling: Exploring out-of- distribution data for re-balancing long-tailed datasets. In International Conference on Machine Learning, pp. 23615-23630. PMLR, 2022.
Julia Wolleb, Robin Sandkühler, Florentin Bieder, Philippe Valmaggia, and Philippe C Cattin. Diffu- sion models for implicit image segmentation ensembles. In International Conference Medical Imaging with Deep Learning, pp. 1336-1348. PMLR, 2022.
Junde Wu, Rao Fu, Huihui Fang. Yu Zhang, Yehui Yang, Haoyi Xiong, Huiying Liu, and Yanwu Xu. Medsegdiff: Medical image segmentation with diffusion probabilistic model. arXiv preprint arXiv:2211.066 2022.
Linshan Wu, Leyuan Fang, Xingxin He, Min He, Jiayi Ma, and Zhun Zhong. Querying labeled for unlabeled: Cross-image semantic consistency guided semi-supervised semantic segmentation. IEEE Transactions Pultern Analysis and Machine Intelligence, 2023.
Haoyu Xie, Changqi Wang, Jian Zhao, Yang Liu, Jun Dan, Chong Fu, and Baigui Sun. Prel: Prob- abilistic representation contrastive learning for semi-supervised semantic segmentation. Interna- tional Journal of Computer Vision, pp. 1-19, 2024.
Haiming Xu, Lingqiao Liu, Qiuchen Bian, and Zhen Yang. Semi-supervised semantic segmentation with prototype-based consistency regularization. Advances in Neural Information Processing Systems, 35:26007-26022 2022.
Yi Xu, Lei Shang, Jinxing Ye, Qi Qian, Yu-Feng Li, Baigui Sun, Hao Li, and Rong Jin. Dash: Semi-supervised learning with dynamic thresholding. In International Conference on Machine Learning. pp. 11525-15 11536. PMLR, 2021.
Lihe Yang, Lei Qi. Litong Feng, Wayne Zhang, and Yinghuan Shi. Revisiting weak-to-strong con- sistency in semi-supervised semantic segmentation. arXiv preprint arXiv:2208. 09910, 2022.
Mingzhao Yang, Shangchao Su, Bin Li. and Xiangyang Xue. Exploring one-shot semi-supervised federated learning with pre-trained diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 16325-16333, 2024.
Zebin You, Yong Zhong, Fan Bao, Jiacheng Sun, Chongxuan Li, and Jun Zhu. Diffusion models and semi-supervised learners benefit mutually with few labels. Advances in Neural Information Processing Systems, 36, 2024.
Lequan Yu, Shujun Wang. Xiaomeng Li, Chi-Wing Fu, and Pheng- Heng. Uncertainty-awer self-ensembling model for semi semi-supervised 3d left atrium segmentation. In Medical image com- puting and computer assisted Aiervention-MICCA 2019: 22nd international conference, Shen- zhen, China, October 13-17, 2019, proceedings, part 11 22, pp. 605-613. Springer, 2019.
Jianlong Yuan, Jinchao Ge. Zhibin Wang, and Yifan Liu. Semi-supervised semantic segmentation with mutual knowledge distillation. In Proceedings of the 3 31st ACM International Conference on Multimedia, pp. 5436-5444, 2023.
Shiying Yuan, Ruofei Zhong, Cankun Yang, Qingyang Li, and YaXin Dong. Dynamically up- dated semi-supervised change detection network combining cross-supervision and screening al- gorithms. IEEE Transactions on Geoscience and Remote Sensing, 2024.
Wenwei Zhang, Jiangmiao Pang, Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. Advances in Neural Information Processing Systems, 1113601003333 2021.
17

---

Published as a conference paper at ICLR 2025
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5729-5739, 2023a.
Zhen Zhao, Sifan Long, Jimin Pi, Jingdong Wang, and Luping Zhou. Instance-specific and model-adaptive supervision for semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 23705-23714 2023b.
Zhen Zhao, Lihe Yang, Sifan Long, Jimin Pi, Luping Zhou, and Jingdong Wang. Augmentation mat- ters: A simple-yet-effective approach to semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1350--1339, 2023c.
Zhen Zhao, Zicheng Wang, Longyue Wang, Dian Yu, Yixuan Yuan, and Luping Zhou. Alternaee diverse teaching for semi-supervised medical image segmentation. In European Conference on Computer Vision, pp. 227-243. Springer, 2025.
Shuhong Zheng. Zhipeng Bao, Ruoyu Zhao, Martial Hebert, and Yu- Yu-Xiong Wang. Diff- 2-in-1: Bridging generation and dense perception with diffusion models. arXiv preprint arXiv:2411.05005 2024.
Yuanyi Zhong, Bodi Yuan, Hong Wu, Zhiqiang Yuan, Jian Peng, and Yu-Xiong Wang. Pixel contrastive-consistenn semi-supervised semantic segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7273-7282, 2021.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
Yuliang Zou, Zizhao Zhang, Han Zhang, Chun-Liang Li, Xiao Bian, Jia-Bin Huang, and Tomas Pfister. Pseudoseg: Designing pseudo labels for semantic segmentation. arXiv preprint arXiv:2010.09713, 2020.
18

---

Published as a conference paper at ICLR 2025
A MOTIVATION FOR GENERATIVE MODELS TO ALLEVIATE CLASS IMBALANCE ISSUE
Previous research Ng & Jordan 2001 theoretically derived the differences in generalization error e(.) between discriminative (Dis) and generative models (Gen) under ideal conditions (i.e., with an infinite number of samples oo), where denotes the number of samples, is the number of model parameters, and G(.) represents a small meaningful bound.
(hps) S (oDDis.o) +0
log
(12)
n
e (hiton) S e (ee +G
log
(13)
The above theory demonstrates that the axymptotic ervor approaching rate of generative models is O(logn), which is better than the discriminative model's (O(n)). In other words, under the same number of model parameters, generative models can approach the optimal form under the ideal condition (i.e., infinite training sample) with fewer training samples (logarithmic number, i.e., (log n)), compared the discriminative model, which requires linear number samples (O(n)). This provides special bonus for the inherent class imbalance problem in semi-supervised semantic segmentation, particularly for tail classes. Specifically, generative models have better potential to enable tail classes with extremely limited sample quantity to converge the form assumed under sufficient sample conditions, conceptually bridging the gap with the ample samples of head classes, tolerance.
i.e., better
True Hyperplane
Blasd
Different Classes
Unlabeled
A
(a) Discriminative Model
(b) Diffusion Model
Figure 7: Illustration on discriminative model vs. diffusion model.
On the other hand, from the perspective of optimization objectives, diffusion-based generative mod- els and discriminative models have fundamentally different optimization objectives. Specifically, discriminative models are typically trained by minimizing empirical risk, aiming to minimize the prediction eтror or loss function of the model solely on the training data. In this case, these meth- ods, only learning decision boundaries between classes, are highly fragile to the number pixels class (i.e,, class imbalance), leading to decision boundaries that can be drastically altered by the majority classes (as shown in the left part of Figure 7).. In contrast, diffusion-basel generative models use log-likelihood as their optimization objective, maximizing the log-likelihood between the explicitly modeled class distribution and the underlying real one (as shown the right part of Figure 77). Benefiting from modeling probabilistic density, diffusion- based generative models pay more attention to the class distribution rather than the boundaries across classes. Therefore, they conceptually exhibit better tolerance to class imbalance.
19

---

Published as a conference paper at ICLR 2025
B PSEUDO ALGORITHM
In this section, we summarize the pseudo algorithm of DiffMatch Algorithm The inputs consist of labeled set DD = {(ar. u!)} and an unlabeled set D { where N > N The feature extractor g(-), mask denoiser f(-), weak augmentation w, strong augmentation 8, and time step f are defined. The algorithm iterates over each batch of labeled data (æ', yy) and unlabeled data x". For labeled data, the pixel embedding z' is extracted using g(.), noise is injected into y' to obtain y via the forward process (Equation } and the noisy mask y/ denoised conditioned on 2 and using f() in the reverse process. The supervised loss Sup is calculated by Equation 10 For unlabeled data, weak and strong augmentations are applied x: obtain æ and x respectively. Their pixel embeddings z and 2 are extracted using g(-). The pseudo label Fo.w is obtained by denoising e conditioned on z in the reverse process. Noise is injected into Vö, to obtain Di, via the forward process, and the noisy mask Vi is denoised conditioned on 2 using f(-) in the reverse process. The unsupervised loss Lunsup is calculated by Equation Finally, the model is updated by performing gradient backward on Loup + Lunsup-
Algorithm Pseudo algorithms of DiffMatch.
1: Inputs: Labeled Set D = {(ac, b Unlabeled Set DD {ar JE (N" >> N
2: Define: Feature Extractor g(-), Mask Denoiser f()), Weak Augmentation w, Strong Augmen-
tation s, time step
3: Output: Feature Extractor g()), Mask Denoiser f(-)
4: for each batch of (x', ,'), x in D,, D., do
5: Labeled Data:
6: Extract pixel embedding x' for x' using g(-)
7: Inject noise into y' and obtain y by Equation
> Forward Process
8: Denoise the noisy mask y conditioned z and using f())
Reverse Process
9: Calculate Cuup by Equation 10
b Supervised Loss
10:
Unlabeled Data:
11: Obtain x and x by applying weak and strong augmentation x", respectively
Extract pixel embedding = and z using g(.)
13: Obtain the pseudo label yö. by denoising e conditioned on z using f(.)
14:
Reverse Process
Inject noise into Fe. and obtain V. by Equation 3
Forward Process
16: Denoise the noisy mask V. conditioned on z using f(-)
b Reverse Process
17:
Calculate Lunsup by Equation 11
Unsupervised Loss
Gradient backward Leup Lunsap
Update Model
19:
end for
20

---

Published as a conference paper at ICLR 2025
Algorithm 2 Diffusion Training Process
def clpha__cuprod (t,
ts=0.00025.
"""cosine noise schedule"". """
torch.cos ( (t ns) / (1 ds) math.pi / 2) ** -2
return -torch. log (n 1, eps-le-5)
def train train (images, masks)
"""images: [b, w), masks: [b, 1, h, "]"" img_ene foaturo_cxtractor (images) encede image
mask_enc enc encoding (masks) # encode gt pseudo labels
mask_enc (sigmoid (mask_eng) - 1) scale corrupt gt or pseudo
labels
eps uniform (0, 1), normal (mean=0, std=1)
mask_crpt sgrt phappccrro (t)) mask_enc sqrt clpha_cumprod
)) eps
# predict and backward
mask_ dew.keednceeeeeeeeee (mask_crpp) mask_enc t)
loss Cosl (mask_pred, masks) calculate the loss after deblased
adjustment
return loss
21

---

Published as a conference paper at ICLR 2025
Algorithm 3 Diffusion Sampling Process
def ddim (mask_t,
t_now, t_next):
""" estimate t_next with DDIM update rule""" ansa (t a alpha_cumprod (t_next.)
mask_ enc encod in (mask_pred)
mask enc (sigmoid (mask enc) * 1). scale
eps -m-a (mask_ Vinaw mask enc)
mask next Vinnxt x pred vI asox eps
return mask next
def sample (images, steps, td=1);
"""steps: sample steps, td: time difference"""
img_enc Feature_extractor/ti (imracttoriimagee
mask_t = normal (0, 1) # [b, 256, h/4, w/4] for step range (steps)
time intervals n_oow = step / steps
next = max (1 (step 1 td) / steps, 0)
predict mask_0 from mask_t
pask_pred mask_donoiser (mask_t, img_ene, t_now)
estimate mask_t next
mask_t ddim (mask_t, mask_pred, t_now, t_next)
return mask pred
22

---

Published as a conference paper at ICLR 2025
C DERIVATION OF Casss
Here, we present the detailed derivation of Cass from the learning of the diffusion model. Denot- ing the underlying conditional distribution as q. we can rewrite the conditional reverse probability
q (yt y/++, 2) according to Bayes' formula following Dhariwal & Nichol (2021):
9 (y/, 3/++1) ) 9 (z yy))
② (ye V0+1,
(z M0++)
9 (y. M++1) ④ (3): 22) (2) (36+1)
(14)
(yc+1 z) q (z) (3r)
9 (y/ 3i++1) á (3t 2)) (31-1)
i (Vr+1
z) q (9r)
Since the conditional diffusion model is trained to fit prior distribution with known conditions by definition, we can approximate ((ye) with (g)) and have:
Pe (ye 2)+ (3/ 3041) g (3t x) (3(+1)
(15)
i (ye+1 po (yr)
Given the long tailed nature of the class distribution p(yo) in practice, the learned conditional inverse probability Po (3: y x) is inevitably biased. To further reduce the risk of overfitting to the head classes and to increase coverage of the tail class distribution, we propose the debiased adjustment. First, represent the conditional inverse probability under ideal condition (i.e., when the class distribution is uniform, p" (yo) = t. where C' is the number of classes) as PT (yo same way:
y/., z). In the
Pd (y/ 3141,2 4 (2v 144) (ye 12) q" (y++1).
(16)
q" (11+1 z) P3 (36)
Since yo is uniquely determined by 2, we have:
@ ④ (y): yo) = 4 (yr z)) i' (yt, 2)) = q" (yt, yo)
(17)
where the equality ① holds because q" (yr, yo (3/: yoo is conditioned on you i.e., unrelated to p(yo). In the same way:
q" (yt-1 2) 9 (vt+1 2).
(18)
In other words, (3: | z) and (Be+1 z) are not affected by the class distribution. Combining the above equations, have:
pa (ye 9t+1, = Po (ye y1+1, 2) Pe (yr) q" (y++1) Pe (yc) á (B142)
(19)
be seen that there is only a factor of difference (i.e., (o(ue) 6" (we11)
p((M)) (1) between the ideal condi-
tional inverse process P (y/, 2)+ and the actual conditional inverse process (y:: yt+1, =). However, the factor is difficult obtain directly. Therefore, We convert it into the training loss
and gradually remove this difference during training. Since ("(M+11 is independent of the model d(v++1) parameters, it follows from Menon al. (2020) that the sign should be reversed when converting the post-hoc adjustment factors into the training loss, giving us:
Pd (y:)
pe (v B1+1, 2) po (v: 31+1,22) 2)
(20)
Pe (yt)
Then we get the unbiased loss for the conditional diffusion model by replacing the po (No 3/: x)) in Equation 2 with p 2);
23

---

Published as a conference paper at ICLR 2025
Cass
Σ DEL (1re 1 11--1) llp@ (yre yit1, =)]
Pú (3/ 91-1, 2)
Σ
Byg -
log
(3/ Vt-1)
(21)
Σ
Bag
log
Pa (yr,
y1+1, 2)
+ Bng log
pb (3/x)
9 (y/ 91-1))
po (3ra)
log Pi (3/.)
Σ DKL. lg (yr. 11 llpo (3r,
ye+1, 2]] + Eng
po (3/e)
Focus on the second item of the above equation:
Po (3:)
Bg
log
po (3r))
pb (Yu) IT [1=1 Pe (ye 11
Bg
log Ep
Po (3/0
11 -1 Pa (ye (3(--1)
Pg (3/0)
II P0 (y (y
PC (3r
P5( (1)
Bg
log Eps
pe (yo)
IT: (yr-11
3/
Pol Po(W.r)
po (e1
p (30)
I PT (yr-1
Pb (Brr)
p5 (1)
Bq
Epp
log
pe (yo) IT 1'=1 (yr
yer)
Po (M41)
po (B
(22)
Pe (yo) pā (yrr-1
En
Eap
Σ
log
P7 (Mo) Pa (g/r-11 y/r
log Po (yo) Pă (ye
y/r)
E2g
Exp
pe (yo) (ye ye
1==1
Σ t.E.g
log
Pô (3rr 1 Ylr)
po (MM 1 :lMee)
Cpe(Do)
Σ 1DRL
Po (y)- y1)
||p (y1-1 v1))
Cpo (yo)
where the inequality @ holds due to Jensen's Inequality, and the equality @ is holds because p" (y(() = 1. In practice, we approximate the (0(31-11 ye) with Monte-Carlo samples from Po(Ui-1 yt, 2) and the loss reduce to:
f (yr. x) Cp (2/)
Cais - (yr, 2) -yo))l2 =tt 7t [g(,))
(23)
[11,, 2
-Laiff + Tl 5 (1112)
(yo)
24

---

Published as a conference paper at ICLR 2025
D TRAINING CURVE FOR HEAD&TAIL CLASSES
Figure provides comparative analysis of the training samples of the head class road and the tail class motorcycle on the Cityscape Cordts et al. (2016) under the 1/16 partition protocol as the train- ing progresses. The proposed DiffMatch compared with the highly competitive UniMatch Yang et al. (2022) in terms of pseudo label count, assuming that the ground truth for unlabeled data is available solely for theoretical analysis purposes.
33
Road
31
DiffMatch
UniMatch
Real Distribution
29
27
25
23
www.
21
19
17
15
20
40
60
80
100
120
140
160
180
200
220
Epoch
185
4
Motorcyde
3.5
2.5
2
1.5
DiffMatch
UniMatch
Real Distribution
0.5
0
20
40
60
80
100
120
140
160
180
200
220
Epoch
Figure 8: We count the training samples head class road and tail class motorcycle on Cityscapes (Cordts et al. 2016) under 1/16 partition protocols the training processes, and com- pare the proposed DiffMatch with the highly competitive UniMatch (Yang et al.) 2022) in terms of Pseudo Label Count, assuming that the ground truth for unlabeled data is available solely for theo- retical analysis purposes. Our DiffMatch strives to mitigate the Matthew effect raised by the class imbalance issue and stands out for head/tail classes.
The top plot in Figure S illustrates the prediction distribution the head class road. UniMatch generates significantly higher pseudo label count compared to the real distribution, indicating its tendency to over- over-predict the dominant class. In contrast, DiffMatch exhibits a pseudo label count that more aligned with the real distribution, demonstrating its ability to mitigate the bias towards the head class. The bottom plot in deepictss the prediction distribution of the tail class motorcycle. UniMatch generates substantially fewer pseudo labels compared to the real distribution, highlighting
25

---

Published as a conference paper at ICLR 2025
its under-predde onder-preiittin of the minority class. Conversely, DiftMatch demonstrates pseudo label count that much closer to the real distribution, showcasing its effectiveness addressing the under- representation of the tail class.
The comparative analysis in Figure substantiates the effectiveness of DiffMatch in leveraging the advantages of generative models to alleviate the Matthew effect. By incorporating diffusion model and theoretically deriving a debiased adjustment (3.3). DiffMatch effectively mitigates the bias towards head classes and the under-prediction of tail classes, promoting unbiased learning in semi-supervised semantic segmentation. This finding aligns with the quantitative results analyzed in the " PPrffrrance in Head&Tat Classes" of Section 4.3 please refer to it for more details.
E DETAILED ANALYSES OF HYPER-PARAMETERS
Decoder Depth. Table investigates the effect the decoder depth, i.e., the number of layers in the mask denoiser ʃ(). The results demonstrate that increasing the depth initially improves the model accuracy, with the optimal performance achieved at 4 layers (73.3% mIoU on PASCAL. classic 1/16(92)). However, further increasing the depth beyond 4 layers leads to the saturation of performance. This observation aligns with the goal behind the lightweight design of the mask denoiser, which enables efficient reuse of shared parameters during multi-step denoising diffusion processes while maintaining highly competitive performance. The chosen architecture with 4 layers strikes balance between accuracy and efficiency, with a parameter count of 44.9M.
Table 7: Evaluation of number of layers
#Layer L
ml2
mloU((164)
#Param
70.1
79.5
42.4M
71.2
80.6
43.3M
4
73.3
81.6
44.9M
6
72.6
80.8
45.8M
12
71.9
81.1
49.9M
Scaling Factor, Table explores the impact of the scaling factor used in the analog bits encod- ing strategy (Section 3.4). The scaling factor determines the range {-b,b] b} into which the analog bits are shifted and scaled. The results show that suitable scaling factor necessary for opti- mal performance. As the scaling factor increases, the model accuracy improves until reaching peak at = 0.1 (73.3% mloU on PASCAL classic 1/16(92) and 81.6% mloU on PASCAL classic Full(1 (4464)). Further increasing the scaling factor leads decline in performance. We hypothesize that a larger scaling factor retains more easy cases with the same time step, potentially affecting the balance between easy and hard cases during training.
Table 8: Evaluation of scaling factor.
Scale mloU(92)
mloU(1464)
0.01
71.7
80.9
0.05
72.2
81.2
0.1
73.3
81.6
0.2
70.7
80.8
0.5
70.6
80.5
Regularization Term r. Table 9 examines the influence of the trade-off weight T for the regular- ization term in the debiased adjustment. The regularization term imposes constraint between the prediction of mask denoiser and its roughly debiased version, reducing the risk of overfitting to head classes and increasing coverage of tail class distribution. The results indicate that setting 0.1 yields the optimal performance, that is, 73.3% mloU on PASCAL classic 1/16(92) and 81.6% mloU on PASCAL classic Full(1 1464).
26

---

Published as a conference paper at ICLR 2025
Table 9: Evaluation of trade-off weight for the regularization term 7.
mloU(92)
mloU(1464)
0.01
71.8
80.0
0.02
72.2
80.4
0.05
72.7
80.9
0.1
73.3
81.6
0.2
71.9
80.2
F COMPARISON WITH OTHER DIFFUSION-BASED SEMI-SUPERVISED METHODS
As diffusion gains popularity in visual perception, researchers have introduced it into various semi- supervised tasks (You et al., 2024] Yang et al. 2024| Liu et al. 2024, Ho et al. 2023), such as classification, federated learning, time-series classification and 3d object detection. In the following, we will comprehensively and meticulously compare our DiffMatch with these diffusion-basee semi- supervised methods and summarize in Table 10 to highlight the originality of our work.
Different from our DiffMatch, both DPT You et al. 2024) and FedDISC (Yang et al. 2024) uti-
lize external diffasion model generate additional data and demonstrate their effectiveness in facilitating the original model training. Specifically, DPT introduces a from-scratch diffusion- -bsed conditional generative model address the scarcity of labeled data in semi-supervised classifica- tion task in three stages: train the original classifier on limited labeled data to predict pseudo-labele train the conditional generative model using these pseudo-labele to generate labeled data; retrain the classifier with a combination of limited real and vast generated labeled data. FedDISC addresses the challenge of semi-supervised federated learning by introducing a well-trained diffusion model. To alleviate the communication burden between the server and clients, the diffusion model generates rich client-style data for the server, conditioned on the cluster centroid of client data representations, thereby facilitating model training on the server.
Regarding DiffShape (Liu et al. 2024), although explores integrating the diffusion process into sumi-super time-series classification, it does through self-supervised mechanism rather than incorporating it into the sracher-studer network paradigm. Specifically, DiffShape employs large amounts of unlabeled instance subsequences conditions in the diffusion process to gener- ate the subsequences themselves, enhancing similarity in the generated sequences compared to the original ones, thereby improving representation capability in a self-supervised manner.
For Diffusion-ss3d (Ho et al. 2023), although it integrates the diffusion process into the teacher- student network paradigm in semi-supervised 3D object detection, we categorize it as a noise-to- filter paradigm, leveraging the denoising capability of diffusion models to generate more accurate 3D bounding boxes as pseudo labels. Specifically, Diffusion-ssā3 first predicts coarse bounding boxes (fixed bounding box candidate points) with detection model, which can be considered as intermediate states in the diffusion process, and then employs the diffusion model as a denoising process to obtain other parameters of the bounding box (e.g., bounding box size). Overall, this paradigm partially exploits the characteristics of the diffusion process, that is, the denoising ability, to improve the quality of the bounding boxes prediction.
Distinguished from these methods, Our DiffMatch integrates the diffusion process into the teacher- student network for semi-supervised semantic segmentation, which can be viewed as a noise-to- prediction paradigm. Motivated by the potential of generative models with better tolerance to class imbalance, DiffMatch learns the complete process of transforming noise from known distribu- tion to class predictions (all states from time 0 to time T). Additionally, we mathematically derive a debiased adjustment based the state transition function encapsulated in the diffusion process to further mitigate the Matthew effect. This mathematical formulation translates into strong empirical performance on real-world datasets, particularly in scenarios with the most limited labeled data and the most severe class imbalance. In general, DiffMatch completely utilizes the characteristics of the diffusion process in a different problem for comi-supervised semantic segmentation, aiming to provide a new perspective to alleviate the Matthew effect.
27

---

Published as a conference paper at ICLR 2025
semantic segmentation
integrating the diffusion process into the teacher-sture stur--tdeentt framework in a noise-to- (o-pprdccctoon paradigm
(1) learning complete diffusion process (2)
leveraging the well class- pes-imballacee tolerance of Diffusion to alleviate the Matthew effect
DiflMatch (Ours)
deriving debiased
ad adjustment based on the state transitionn function
mathematically
exploiting the denoising ability of Diffusion to improve the quality of pseudo label
integrating the diffusion process into Pracher-studen
leaming an incomplete diffusion process
3d object detection
Diffusion-ss22
framework. in a noise-to-filte to-to-- filter paradigm
Table 10: Comparison with other diffusiion iitton-basedde semi- superviseeeeeddd methods.
using diffusion in self-supervisee manner improve representation capability
integrating the diffusion process through self- self-supervised mechanism
time-series classification
DifIShape
harnessing the data generation capability of Diffusion to alleviate data scareity
federated learning
introducing well-trained external diffusion model
FedDISC
harnessing the data generation capability Diffusion to alleviate data scareity
introducing Tront-scracch external diffusion model
classification
DPT
Implementation
Motivation
Task
Note
28

---

Published as a conference paper at ICLR 2025
G LIMITATION AND SOCIETY IMPACT
DiffMatch may face a potential limitation in terms of increased computational cost during multi-step inference. And how to adapt the number of inference steps to the degree of change in the generation state is a feasible direction. Within this paper, we present an approach for semi-supervised semantic segmentation, pivotal research area in the realm of computer vision, with no apparent negative societal implications known thus far.
H EXTENDED DISCUSSION ON RELATED WORK
Semi-Supervised Segmentation. Semantic segmentation has achieved conspicuous achievements
attributed to the recent advances in the deep neural network (Mai et al. 2024a] Sun et al. (023d).
Among them, semi-supervised semantic segmentation is a fundamental task with extensive appli-
cations in scene understanding (Mittal et al., 2019; Wu et al. [023]] Wang et al.,
2024b] 2023d e
2022b), medical image analysis Yu et al. 2019 Bai et al. 20233 /haoo et al. 2025
Chi et al. [2024]
Sun et al. 2021), brain neuroscience (Sun al. 2203b Chen et al. [2025] Luo et al.) 2024 Pan
et al. 2023), autonomous driving (Li et al. 2024b] Pan et al.) 2024 Li et al. 2024a) and remote
sensing interpretation (Wang et al. [202]] Bandara & Patel 2022: Yuan al.) 2024). These al-
gorithms leverage the mature combination of pseudo-labeling and consistency regularization (Lai
al. 2021 Zhong et al., 2021 Ouali et al. 2020 Chen & Lian 2022) improve performance.
More recently, UniMatch Yang et all.]22222 acknowledges the characteristics of semantic segmen-
tation and incorporates appropriate data augmentations into Fix.Match (Sohn et al. 2020). resulting in a concise yet powerful semi- supervised semantic segmentation baseline. Subsequently, series of works aim to improve segmentation performance mainly in the following aspects. (1) Employ reasonable augmentation strategies to expand the augmentation space. For example, AugSeg (Zhao) ct al., 2023c) increases the randomness in RandAugment (Cubuk et al. 2020) for richer data aug- mentation space. iMAS (Zhao et al.) (023b) employs adaptive augmentations and supervisions conditioned on the model state. (2) Design effective teacher networks for better guidance. For example, Switch (Na et al. 2023) targets the coupling problem in the exponentially moving aver- age (EMA) update process of teacher-student network and proposes a dual-teacher structure in an ensemble manner. (3) Utilize external knowledge enhance the quality of pseudo labels. For exam- ple, LOGIC (Liang et al. 2023) integrates symbolic reasoning derived from symbolic knowledge to mitigate erroneous pseudo labels. SemiVL (Hoyer et al., (025) incorporates CLIP encoder (Rad- ford et al. 2021). pre-trained on large-scale data, into semi-supervised semantic segmentation and employs a language-aware decoder introduce text modality priors. (4) Enhance consistency regu- larization (Sun et al., 2024, Howlader et al. 2025b) to effectively exploit the information contained in unlabeled data. For example, RankMatch (Mai et al.) (024b) utilizes inter-pixel correlations to construct more safe and effective supervision signals, which are in line with the nature of semantic segmentation. MPMC (Howlader et al., 2025a) identifies the classes present in an image region to incorporate pixel-level contextual information, thereby exploring more supervision signals. Despite yielding promising results, these methods tend to neglect the fact of class imbalance issue. this paper, strive alleviate the negative impact (Matthew effect) raised by class imbalance issue and move towards unbiased semi-supervised learning.
Semi-Supervised Segmentation. Real-world datasets usually yield class- imbalanced distribution, especially in dense prediction tasks (e.g., semantic segmentation), mak- ing the standard training of machine learning models harder generalize. Existing methods re- balance the training objective can be roughly categorized into two paradigms: (1) Re-sampling based methods (Chawla al.) 2002; He & Garcial 2009, [yrd & Lipton| 2019 2 al., 2021 Shi
et al. 2023| Wei et al.) 2022) attempt to artificially balance the training data distribution. These ap- proaches either employ over-samplin techniques to increase the representation of minority classes or utilize under-sampling strategies to reduce the dominance of majority classes. While effective in certain scenarios, these methods often struggle with the trade-off between maintaining data diver- sity and achieving balanced class distributions. (2) Re-weighting based methods (Cao et al. 2019
Cui et al.) 2019 Huang et al., 2019 Ren et al.] 2018 Hu et al. 2019 Chen et al., 2023d) focus on modifying the loss function to prioritize learning from under- represeepde classes. These approaches typically assign importance weights to different classes based on various criteria, such as inverse class frequency or dynamic class-wise difficulty measures. Although these methods have shown
29

---

Published as a conference paper at ICLR 2025
promising results, they often require careful tuning of weighting schemes to prevent instability dur- ing training. However, all these methods assume all labels are accessible to alleviate the class imbal- ance issue and thus inapplicable to the unlabelled data in semi-supervised semantic segmentation.
Recently, several studies have attempted to transfer these techniques on top of pseudo labels such as
re-sampling (Wei et al.] (02T). re-weighting (Wang al. 2022a Sun et al., 2023c Xu et al. 202T
He et al., [021] Wang et al. [022c] Peng et al. 20023 (e.g,, Adsh (Guo & Li 2022) utilizes adaptive
thresholding that can be considered as binary weighting for semi-supervised learning, U Waang
et al. 2022c) adjusts the threshold adaptively to determine the reliability of pixels and constructs the extra supervised signal based on the negative classes of unreliable pixels, paying more attention to the tail classes), or a combination of both for semi-supprvised learning (e.g. AEL (Hu et al.) 2021] adaptively balances the training of different categories). Nevertheless, these pseudo labels are often noisy as they are generated from poorly calibrated models. Furthermore, USRN (Guan et al.) 2022) explores unbiased subclass regularization for alleviating the class imbalance issuc. However, these discriminative methods are still confined to learning decision boundaries, which are brittle to the class imbalance issue, and the inherent nature of contempt for the underlying distribution re- mains unchanged. As significant departure from the status quo, we formulate the semi-supervised semantic segmentation task as a conditional discrete data generation problem to model underlying distribution to overcome the shortcomings of discriminative solutions from generative perspective.
Diffusion Models for Visual Perception. In addition to the significant progress in content genera- tion, diffusion models have demonstrated incremental potential in the domain of perception (Chen) Chen et al. [203cc] Brempong et al. 2022). Earlier studies primarily delve into investigating latent representations of diffusion models for zero-shot image segmenta-
iet al. 2202bb Gu et al. 2022
2022) or applied diffusion models to medical image segmentation (Wolleb et al.] 2022] Wu et al.
tion (Baranchuk et al. 2021 Burgert et al.
(022). Despite substantial progress, the outcomes of
designs. The recent Pix2Seq-D (Chen et al.) (023c) extends the bit-diffusion (Chen et al. 2022) to panoptic segmentation, marking the first work of such expansion in a broader context. Additionally, DiffusionDet (Chen et al.) (2233)) and Diffusion- Inst (Gu et al.] (2022) explore diffusion models for query-based object detection (Carion et al.] 2020)
these efforts remain limited to specific local
and instance segmentation (Zhang et al., 2021). Most recently, groundbreaking work has extended
the application of diffusion models to a comprehensive range of dense visual perception tasks (Ji
eet al. 2023] Zhao et al., 2023a] Zheng al. 2024). These latest developments have achieved promising results across multiple challenging scenarios, further solidifying the position of diffusion models as versatile and powerful tool in the visual perception domain. Recently, several works have introduced diffusion into various semi-supervised tasks, such as classification, federated learn- ing, time-series classification, and 3d object detection. Among them, both DPT (You et al., 2024) and FedDISC (Yang et al. 2024) aim to introduce an external diffusion model to generate data and utilize these data in a multi-stage -ttage training manner. DiffShape (Liu et al.) (024) utilizes diffusion in self-supervised manner to improve representation capability, and Diffusion-ss3d (Ho et al.) 2023) exploits the denoising ability of the diffusion to improve the quality of the pseudo label. How- ever, these methods differ from ours both from motivation to implementation. We comprehensively and meticulously compare our DiffMatch with these biffusion-basee semi-supervised methods in Appendix F In general, DiffMatch completely utilizes the characteristies of the diffusion process for cepirsupe semantic segmentation, aiming to provide a new perspective to alleviate the Matthew effect.
I MORE VISUALIZATION
Here, we provide additional visualizations to qualitatively assess the performance of DiffMatch in comparison to other methods. Figure 9 showcases the segmentation results on the PASCAL VOC dataset, highlighting the effectiveness of DiffMatch in obtaining more accurate semantic segmenta- tion, particularly for pixels that are incorrectly segmented as the most dominant class by other meth- ods. For example, in the 2nd row, FreeMatch, UniMatch, and RankMatch encounter difficulties in accurately segmenting the person pixels. They misclassify a considerable portion of the person pix- els as the horse class. These misclassifications can be attributed to the class imbalance issue, where the models are inclined to favor the majority classes, resulting in subpar segmentation performance for the less represented classes like person. In contrast, DiffMatch demonstrates a notable ability to overcome these challenges and generate more precise segmentations. By incorporating genera-
30

---

Published as a conference paper at ICLR 2025
tive perspective and employing debiased adjustment, DiffMatch effectively mitigates the Matthew effect stemming from class imbalance. As a result, accurately segments the person pixels.
Furthermore, Figure 10 offers additional insights into the inference trajectory of DiffMatch across different diffusion sampling steps. The ground truth segmentation is provided as a reference, and the segmentation results steps I, and 3 visualized. As the number of sampling steps increases, the segmentation quality progressively improves, with finer details and more accurate boundary delineation.
Image
Ground Truth
FreeMatch
UniMatch
RankMatch
DiffMatch (Ours)
Figure 9: Qualitative results PASCAL VOC dataset. DiffMatch obtain more accurate seg- mentation for pixels that are inaccurately segmented as the most dominant class.
Ground Truth
Step
Step 2
Step 3
Figure 10: Inference trajectory with diffusion sampling steps. The model gradually refines the prediction, starting from coarse estimation in Step and progressively improving the results in Step The final output Step closely resembles the ground truth, demonstrating the effectiveness of DiffMatch in capturing fine-graines details and accurately delineating the boundaries of the changed buildings.
31

---

Published as a conference paper at ICLR 2025
J DETAILED ILLUSTRATION OF DIFFMATCH FRAMEWORK
In this section, we provide detailed illustrations for clearer understanding of the DiffMatch frame- work and the conditional discrete data generation pipeline using the diffusion process strategy. Fig- TreeTppreent a comprehensive overview of the key components in DiffMatch, including the feature extractor, mask denoiser, and the supervised and unsupervised loss calculations. Figure [2 further illustrates the forward and reverse diffusion processes employed in the conditional discrete data generation pipeline for superiupeeeieed semantic segmentation.
(a) Encoding
(c) Lumsup
Teacher Network
H
yu
Feaurre Extractor g
condition
(b) sup
z!
y
zw
Z'S
condition
yd
T
Student Network
+
Student Network
Y
Mask Denoiser
iterative
Mask Denoiser f(.)
iterative
Figure 11: Illustration of the DiffMatch framework. (a) The encoding process. The feature extractor g(-) takes an input image æ and outputs the pixel embedding z. (b) Supervised loss calculation. The ground truth mask y' is corrupted with noise sampled from the Gaussian distribution to obtain the noisy mask y!. The mask denoiser f(-) takes yi and z' as inputs to predict the denoised mask y'. The supervised loss Lsup is computed between ÿ and y'. (c) Unsupervised loss calculation. Weak and strong augmentations are applied to the unlabeled image x" to obtain x and a The teacher network generates pseudo labels Vö,.- by denoising conditioned z Noise is injected into Vo obtain V The student network denoises y conditioned 2 predict y.. The unsupervised loss Cunsu calculated between y and Vo.w.
Forward Diffusion
yo
Yt-1
Yt
yr
Po(yi-1lb02)
Reverse Diffusion
Figure 12: Illustration of the conditional discrete data generation pipeline for semi-supervised se- mantic segmentation using the diffusion process strategy. The forward diffusion process ((v.lVr-) progressively corrupts the input mask Yo by adding Gaussian noise at each time step f, resulting in the noisy mask y:- The reverse diffusion process pa p conditioned on the pixel embedding x to recover the mask ft-1 at previous time step. The denoising is performed iteratively, with the mask denoiser f(.) predicting the denoised mask at each step.
10c-1/10 | z) learns to denoise the noisy mask y:
32


Reviews:
[{"review_id": "ZnpO4wufg0", "reviewer": "Reviewer_iDj4", "summary": "This paper analyzes the issue of class imbalance, which leads to model predictions being biased towards head classes and away from tail classes, summarizing it as the Matthew effect. To address this problem, the authors propose a new method called DiffMatch, which formulates the semi-supervised semantic segmentation task as a conditional discrete data generation problem, alleviating the Matthew effect from a generative perspective. Furthermore, based on the mathematical derivations, the authors introduce a debiased adjustment to adjust the conditional reverse probability, reducing the risk of overfitting to head classes during the generation process. Experimental results demonstrate the effectiveness of their proposed method in the challenging dense pixel-level classification task.", "questions": "There are also some minor questions.\n\n1.\tIn the experiments, the authors use a fixed number of diffusion sampling steps, and the results demonstrate that the proposed method achieves competitive performance with reasonable computational cost. However, it is worth considering if a dynamic adjustment of sampling steps could potentially lead to further reductions in computational cost. Please discuss this possibility, as I think it will be interesting and feasible.\n\n2.\tI am curious about the effectiveness of DiffMatch when applied to other related domains, such as semi-supervised classification.\n\nOverall, I think now this paper meets the bar for ICLR, and I will consider raising my score if my questions are addressed during the rebuttal.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1.\tThe authors establish a well-thought-out framework to integrate generative models into semi-supervised learning (SSL) for the challenging semi-supervised semantic segmentation task, mitigating the class imbalance issue from an interesting generative perspective, which is quite novel.\n\n2.\tThe mathematical derivation of the debiased adjustment is ingenious and thoughtful. The theoretical analysis not only explains the reasons behind the proposed method's effectiveness in addressing the class imbalance issue in generative modeling but also provides valuable insights and guidance for future research in SSL techniques.\n\n3.\tThe paper is presented in a clear and concise manner, effectively conveying the intuitive motivation and ideas behind the proposed method. The authors thoroughly analyze the Matthew effect present in previous methods and leverage mathematical tools to alleviate the issue from a generative perspective.\n\n4.\tThe evaluation in this paper is sufficient and comprehensive. In addition to evaluating the effectiveness of the proposed method on common semi-supervised semantic segmentation benchmarks, the authors extend their analysis to other domains, including remote sensing and medical images.", "weaknesses": "1.\tAlthough the authors report the trade-off between performance and computational cost of the proposed method during inference, the computational cost during the *training* stage compared to the baseline should also be reported to demonstrate the method's effectiveness.\n\n2.\tThe specific implementation details of the scheme adopted in this work have not been mentioned. Is it a teacher-student network scheme or another type of scheme?\n\n3.\tIn Appendix C, the factor t is introduced in Equality ③ without sufficient description. Please provide more details on the reasoning behind this step and how t is derived. What is the impact of t on the overall debiased adjustment? Discuss whether removing or simplifying t would impact the method's performance.\n\n4.\tWhile I understand that the ICLR submission deadline is earlier than the release phase for ECCV papers, I encourage the authors to include comparisons with some recent works from ECCV 2024, such as MPMC [a], which could serve as a recent example. [a] Beyond Pixels: Semi-Supervised Semantic Segmentation with a Multi-scale Patch-based Multi-Label Classifier. ECCV 2024.\n\n5.\tI encourage the authors to discuss the limitations of the proposed method.\n\n6.\tMinor typos: The “its” in “taking advantage of its their better class-imbalance tolerance” in L19 should be deleted.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SR2Yp32Al4", "reviewer": "Reviewer_w9jN", "summary": "The paper introduces a new method called DiffMatch that aims to address the Matthew effect in semi-supervised semantic segmentation. The approach reframes semi-supervised semantic segmentation as a conditional discrete data generation problem, with a debiased adjustment. This adjustment mathematically modifies the conditional reverse probability at each sampling step to produce unbiased results. The experimental results show that DiffMatch outperforms current state-of-the-art methods, especially in scenarios with limited labeling and severe class imbalance issues.", "questions": "Please refer to the weakness section for detailed suggestions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "DiffMatch is technically innovative, particularly in terms of incorporating debiased adjustments and conditional discrete data generation problems. It is not only theoretically appealing, but also performs well in experiments. Your analysis and explanation of how DiffMatch performs well in both head and tail classes are clear and convincing, which improves the model's ability to recognize minority classes and has potential applications in a variety of computer vision tasks.", "weaknesses": "1. MORE VISUALIZATION: I have noticed that Remote Sensing Interpretation and Medical Image segmentation are mentioned in the scalability for other scenarios, I suggest that the authors further provide information on the visualization of the results of the two datasets mentioned below to help readers more intuitively understand DiffMatch's performance.\n\n2. SCALABILITY FOR OTHER SCENARIOS: The experimental part is extensive and in-depth, covering the fields of natural images, remote sensing images and medical images. I suggest that the authors further explore how the model performs on different datasets, and whether there are specific categories or scenarios that have a greater impact on model performance. \n\n3. Writing quality: The overall writing quality of the paper is good, with clear logic and a well-structured layout. However, some sections, such as the discussion of related work, could be further expanded to better position the study within the existing literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYnMdFhyyj", "reviewer": "Reviewer_nyBx", "summary": "This paper presents DiffMatch, which aims to use Diffution models to solve the data imbalance in SSS. Specifically, Diffmatch employs conditional discrete data generation to align the predictions with real class distributions. The authors conduct extensive experiments to demonstrate their effectiveness. While the paper tells a good story and offers thorough formulations as evidence, there remain certain concerns that could potentially impact the overall quality.", "questions": "Please refer to the weakness for the questions.\n\nMinor Suggestion (not vital):\nFigs 2 and 3 are not very illustrative and too small. Could they be enlarged, considering their significance in explaining the method? And the caption can be more illustrative since there are numerous mathematical symbols in the figures. It was only after delving into Section 3 that I grasped the workings of DiffMatch. The current presentation may be challenging to follow.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and easy to follow.\n2. The authors provide extensive experimental results to demonstrate the effectivness of their method.", "weaknesses": "Method:\n1. Using diffusion models to solve segmentation is not very sound to me, as diffusion models are not yet mainstream in visual perception solutions. Despite mentioning some papers in related works, these approaches do not rank as top performers in segmentation.\n2. The primary challenge in SSS is noisy pseudo labels. Even though the DiffMatch can align the predictions of unlabeled data, there is no guarantee they are correct.\n3. I think the learning of unlabeled images is still based on the knowledge learned from labeled images. The process of adding noise and denoise did not bring new information to improve the quality of pseudo labels. \n4. How to measure the effectiveness of DiffMatch in prediction alignment? Instead of final segmentation results, can we analyze them by distribution visualization or other methods?\n5. Intuitively, the diffusion process will increase computation costs. However, as in Table 5, DiffMatch is more efficient than other methods. In addition, it seems that only a few sampling steps can already achieve good performances. Why? In addition, how are the effects of the new loss functions in efficiency (Secs 3.2 and 3.3)?\n\nExperiments:\n1. The authors claim that DiffMatch can extend to remote sensing and medical. However, DiffMatch is designed for the long-tailed problem with many categories. WHU-CD is a binary change detection dataset. ACDC is for heart segmentation. Both datasets do not have long-tail problems. Why DiffMatch still outperforms other methods by a clear margin? I think if the authors would like to prove the effectiveness of DiffMatch in long-tail, it would be better to select other datasets (e.g., containing different organs and tumors). Otherwise, such statements may be over-claimed.\n2. Some recent works employ transformers and push the SSS to another level. Can DiffMatch apply to transformers?\n2. The authors claimed that \"DiffMatch will serve as a solid baseline and facilitate future research\", but I don't think Diffusion models would be an extendable design for future research in SSS. \n3. DiffMatch compares a lot with UniMatch and follows its settings. UniMatch released a strong codebase with checkpoints and training logs for following research. I think this part is important for DiffMatch to be a solid baseline.\n\nOverall, I am incline to the negative aspect currently. I am afraid that potential readers may consider this paper as a story but not a practical solution. However, I would be very glad to improve my ratings if the authors could solve my concerns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)