Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation
Authors: Haonan Wang
Xiaomeng Li
Venue: NeurIPS 2023 poster (2023)
Keywords: Volumetric Medical Image Segmentation
Semi-supervised Learning
Unsupervised Domain Adaptation
Semi-supervised Domain Generalization

Abstract: Volume-wise labeling in 3D medical images is a time-consuming task that requires expertise. As a result, there is growing interest in using semi-supervised learning (SSL) techniques to train models with limited labeled data. However, the challenges and practical applications extend beyond SSL to settings such as unsupervised domain adaptation (UDA) and semi-supervised domain generalization (SemiDG). This work aims to develop a generic SSL framework that can handle all three settings. We identify two main obstacles to achieving this goal in the existing SSL framework: 1) the weakness of capturing distribution-invariant features; and 2) the tendency for unlabeled data to be overwhelmed by labeled data, leading to over-fitting to the labeled data during training. To address these issues, we propose an Aggregating & Decoupling framework. The aggregating part consists of a Diffusion encoder that constructs a "common knowledge set" by extracting distribution-invariant features from aggregated information from multiple distributions/domains. The decoupling part consists of three decoders that decouple the training process with labeled and unlabeled data, thus avoiding over-fitting to labeled data, specific domains and classes. We evaluate our proposed framework on four benchmark datasets for SSL, Class-imbalanced SSL, UDA and SemiDG. The results showcase notable improvements compared to state-of-the-art methods across all four settings, indicating the potential of our framework to tackle more challenging SSL scenarios. Code and models are available at: https://github.com/xmed-lab/GenericSSL.

Towards Generic Semi-Supervised Framework for Volumetric Medical Image Segmentation
Haonan Wang', Xiaomeng Li
'The Hong Kong University of Science and Technology
HKUST Shenzhen-Hong Kong Collaborative Innovation Research Institute, Futian, Shenzhen hwanggr&connect. ust.kk, mermli@ust.hee
Abstract
Volume-wise labeling in 3D medical images is time-consuming task that quires expertise. As a result, there growing interest in using semi-supervised learning (SSL) techniques to train models with limited labeled data. However, the challenges and practical applications extend beyond SSL to settings such as unsupervised domain adaptation (UDA) and semi-supervin tion (SemiDG). This work aims to develop generic SSL framework that can handle all three settings. We identify two main obstacles to achieving this goal in the existing SSL framework: 1) the weakness of capturing distribution-invariii features; and the tendency for unlabeled data to be overwhelmed by labeled data, leading to over-fitting to the labeled data during training. To address these issues, we propose an Aggregating & Decoupling framework. The aggregat- ing part consists of Diffusion encoder that constructs a common knowledge set by extracting distribution-invarian features from aggregated information from multiple distributions/tomainss. The decoupling part consists of three decoders that decouple the training process with labeled and unlabeled data, thus avoiding over-fitting to labeled data, specific domains and classes. We evaluate our proposed framework on four benchmark datasets for SSL, Class-imbllanced SSL, UDA and SemiDG. The results showcase notable improvements compared to state-of- the-art methods across all four settings, indicating the potential of our framework to tackle more challenging SSL scenarios. Code and models are available at: http:/gggguubooooeceeeeaaaaaaaea
domain generaliza-
Introduction
Labeling volumetric medical images requires expertise and is time-consuming process. Therefore, the use semi-supervised learning (SSL) is highly desirable for training models with limited labeled have been proposed, particularly in the field
data. Various SSL techniques [1 of semi-supervised volumetric medical image segmentation (SSVMIS), to leverage both labeled
2
3 4
6 7
14 15 [6] [7 [8] assume that the labeled and unlabeled data are from the same domain, implying they share the same distribution. In practice, medical images often collected from different clinical centers using various scanners, resulting in significant domain shifts. These shifts arise due to differences in patient populations, scanners, and scan acquisition settings. As consequence, these SSVMIS methods have limitations in real-world application scenarios and frequently encounter overfitting issues, leading to suboptimal results.
and unlabeled data. However, current SSVMIS methods [8] 9 IO
II 12 I3
To address this limitation, researchers have increasingly focused on Unsupervised Domain Adaptation (UDA) techniques. These techniques leverage both labeled (source domain) and unlabeled data (target
Conresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Labeled data from different domains
Data from wiseon domasins
Training
Testing
Doman shifts
Uniabeled data from different domains
Real-world Application Domain
Reul-world Application Domain
Real-world
Application Domain
Sumple more dala
Sample more
x
X
daly
x
(a) SSL
(b) UDA
(c) SemiDG
Figure 1: From (a), (b) to (c): generalizing SSL UDA and SemiDG settings by sampling more diverse data to form the training and testing sets.
domain) for training, but the data originate from different domains. Furthermore, Semi-supervised Domain Generalization (SemiDG), more stringent scenario, has garnered significant interest. SemiDG utilizes labeled and unlabeled data from multiple domains during training and evaluated on an unseen domain. Currently, methods for these three scenarios are optimized separately, and there no existing approach that addresses all three scenarios within a unified framework. However, given that all training stages involve labeled and unlabeled data, it is intuitive to explore generic SSL-based framework that can handle all settings and climinate the need for complex task-specific designs. Therefore, this paper aims to develop generic framework that can handle existing challenges in world scenarios, including:
Scenario 1: SSL (Figure a))): The sample data used for both training and testing are from the same domain, representing the standard SSL setting.
Scenario 2: UDA (Figure M bj): The sampled data originate from two domains, with the labels of the target domain being inaccessible, representing the UDA setting.
Scenario 3: SemiDG (Figure (Figure[cc)): The sampled data encompasses multiple domains, with only limited number of them being labeled, representing the SemiDG setting.
Potential similarities can be found and summarized as follows: (1) in the training stage, both labeled data and unlabeled data are used; (2) in the scenario of the real-world application domain, whether the distribution shifts in SSL or the domain shifts in UDA and SemiDG can all be regarded sampling bias, i.e., the main difference is how we sample the data in Figure
Now wonder whether the existing SSVMIS methods powerful enough to handle this general task. Experimental results show that the existing SSL methods do not work well on UDA and SemiDG settings, shown in Table 3 & 4 and vice versa (Table One of the main obstacles lies in the severe over-filting of these models, which is caused by the dominance of labeled data during training. Specifically, the state-offthe-art SSVMIS methods Teacher-student framework [1], where a student model is first trained with the labeled data, and a teacher model obtained from the EMA of the student model generates the pseudo label re-train the student model with labeled data, see Figure a): (2) CPS (Cross Pseudo Supervision) (3) framework, which leverages the consistency between two perturbed models and the pscudo label generated by one of the networks will be used to train the other network, see Figure bib). The predicting modules in these two main frameworks are trained with both labeled and unlabeled data; however, the labeled data, with precise ground truths as supervision, converges more rapidly compared with the unlabeled data. Thus, the training process will easily get overwhelmed by the supervised training task, as shown in Figure 3. Another challenge lies in the fact that existing SSVMIS methods fail to address the issue of distribution shifts, let alone domain shifts, resulting in a limitation in capturing features that are invariant to changes in distribution.
mainly based on two frameworks: (1)
2

---

UA.MT Sracche-SStudet-e Student-based)
Und for prodection
Labeled data flow
Unlobeled data flowe
Studow
(
ylu
J.M.A
Teacher
y
x
Mean Tencher
spoch
CLD (CPS.based)
D(9)
ylccp
(
(E)
X
y
((6)
Figure 3: The training loss curves UA- MT and CLD on MMWHS dataset. The overall losses (black) are dominated by the supervised losses.
Figure 2: Our proposed A&D framework differs in that the labeled and unlabeled data flows are separate, and only the unlabeled flow is used for predicting.
Based on the similarities and the main weaknesses of the mainstream SSVMIS methods, we argue that a generic framework is possible if we can solve the over-fitting issue and design a powerful methods to capture the distribution invarriant features. To tackle the above issues and design generic SSVMIS methods for the real- wolld application scenarios, this work proposes novel Aggregating & Decoupling (A&D) framework. Specifically, A&D consists of an Aggregating stage and a Decoupling stage. In the Aggregating stage, based the recent success of the diffusion model [19 20
propose a VNet to aggregate the multi-domain features into one shared encoder to construct a common knowledge set to improve the capacity of capturing the distribution-invariie solve the over-fitting issue, in the Decoupling stage, we decouple the decoding process to (1) a labeled data training flow which mainly updates a Diff-VNet decoder and a difficulty-wwaee V-Net decoder to generate high-quality pseudo labels; (2) an unlabeled data training flow which mainly updates another vanilla V-Net decoder with the supervision of the pseudo labels. The denoising process of the VNet decoder provides the unbnaseedbbised pseudo labels while the difficulty-await V-Net decoder -abaased pseudo labels. We also propose a re-parameterizing & smoothing combination strategy to further improve the quality of the pseudo labels.
features. To
The key contributions of work can be summarized as follows: (1) unify the SSL., Class Imbalanced SSL, UDA, and SemiDG for volumetric medical image segmentation with one generic framework; (2) state the over-fitting issues of the current SSL methods and propose to solve it by an efficient data augmentation strategy and decoupling the decoders for labeled data and unlabeled, respectively: (3) we introduce the Diffusion V-Net learn the underlying feature distribution from different domains to generalize the SSL methods to more realistic application scenarios; (4) The proposed Aggregating & Decoupling framework achieves state-of-ther of-the-art on representative datasets of SSL, class imbalance SSL, UDA. and SemiDG tasks. Notably, our method achieves significant improvements on the Synapse dataset (12.3 Dice) and the MMWHS dataset the MR to CT setting (8.5 in Dice). Extensive ablation studies are conducted to validate the effectiveness of the proposed methods.
Related Work
2.1 Semi-supervised Segmentation & the Class Imbalance Issue
Semi-supervised segmentation aims to explore tremendous unlabeled data with supervision from limited labeled data. Recently, self-training based methods [ 4 22 have become the mainstream
3

---

of this domain. Approaches with consistency regularization strategies 22
3 21 achieved good
performance by encouraging high similarity between the predictions of two perturbed networks for the same input image, which highly improved the generalization ability. In the medical image domain, the 17 16 25
data limitation issue is more natural and serious. Existing approaches [23 T0 to combat the limited data have achieved great success but are bottlenecked by the application scenarios and cannot handle more challenging but practical settings such as UDA and SemiDG.
Class Imbalance Issue Class imbalance issue is significant problem to extend the existing SSL- based methods to more practical setting, since medical datasets have some classes with notably higher instances in training samples than others. In natural image domain. different means
proposed to solve this issue, including leveraging unlabeled data 26 27 28 30 re-balancing
data distributions in loss B1 30 32. debiased learning 6 5 etc. In medical image domain, this issue
more severe but only few work 15 33 25 noticed this problem. Incorporating the class-imbalance
awareness crucial for the generalization of the SSL methods.
2.2 Unsupervised Domain Adaptation & Semi-supervised Domain Generalization
Domain adaptation (DA) aims to solve the domain shifts by jointly training the model with source domain data and target domain data. Unsupervised Domain Adaptation (UDA) 34 35 36
37] is the
most challenging and practical setting among all the DA setting, since no target labels are required. In this context, UDA becoming increasingly important in the medical image segmentation field, and as a result, a myriad of UDA approaches have been developed for cross-domain medical image segmentation by leveraging: generative adversarial -aassarra-bbeed based methods [3] 39 40
45 46).
semi- sumi-super learning techniques [47] 48 49), and self relf-training as well as contrastive learning techniques [50] [5] etc. Though with promising adaptation results, these methods highly rely on the unlabeled target domain information, which hinders the generalizability.
Domain generalization (DG) is more strict setting, where the difference with DA is that the model does not use any information from the target domain. Unlike unsupervised domain adaptation, semi-supervised domain generalization DG)miDG) does not assume access to labeled data from the target domains. Existing SemiDG methods 552 53 leverage various unusual strategies to solve
the domain shifts, e.g., meta-learning (52), Fourier Transformation [54] compositionality (55) etc., which are not general and have unsatisfactory performance on the tasks such as UDA and SSL.
Compared to these prior works, work is the first to unify SSL, Imbalanced SSL, UDA, and SemiDG settings. This extension not only amplifies the scope and versatility of SSL-based frame- works in medical image segmentation but also stands in stark contrast to earlier approaches that remained confined to singular domains such as SSL or UDA.
2.3 Diffusion Model
Denoising diffusion models [56 19 57 58 have shown significant success in various generative
tasks [59 60 61 62 63), due to the powerful ability of modeling the underlying distribution of
the data, conceptually having a greater capacity to handle challenging tasks. Noticing this property, there has been rise in interest to incorporate them into segmentation tasks, including both natural image segmentation [64 665 (66), and medical image segmentation (67 68 20 Given the notable achievements of diffusion models in these respective domains, leveraging such models to develop
generation- based perceptual models would prove to be highly promising avenue push the boundaries of perceptual tasks to newer heights.
3
Method
3.1
Overview of the Aggregating & Decoupling Framework
In this section, we will introduce our Aggregating & Decoupling (A&D) framework, shown in Figure which consists of an Aggregating stage and a Decoupling stage. The training pipeline is illustrated in Algorithm
The Aggregating stage aims to construct a common knowledge set across domains based on the idea that all data share common underlying high-level knowledge, such as texture information.

---

Aggregating Stage
Decoupling Stuge
Difficult--wware Supervised Training
x
Ground truth
plo
hp
KoNer
Ne-wolghnng
plu
p
Class-nise Difl
D (p))
pli
Diffirdonn
ViNer
p
Pseudo Labeling
Ploaddo latel
SVD4
Exx,x*
plop
Softmax
Apematx
4
Knowledge drnilaation
p
Gumbel Sofamax
y&u
& Smooth
V-Ner
(x B
Forward & back propagation
Stop gradient
Loss sapervision
Figure 4: Overview of the proposed Aggregating & Decoupling framework. Blue and orange regions denote the training process with labeled data and unlabeled data, respectively. We separate the training the decoders using labeled data and unlabeled data, and only use the decoder trained with unlabeled data for prediction.
By aggregating information from multiple domains and jointly trained, the encoder can capture the underlying Sampling-basee based Volumetric Data Augmentation (SVDA) strategy enlarge the distribution diversity and leverage the diffusion model to capture the invariant features of the diversified data.
features. To achieve this, we introduce a powerful yet efficient
In the decoding of the existing SSL methods, the decoders simultaneously trained with both labeled and unlabeled data, which leads to coupling and over-fitting issues and further hinder the extending the general SSI.. The Decoupling stage aims to solve these issues by decoupling the labeled and unlabeled data training flows. Concretely, for the labeled data flow, (I) a diffusion decoder mainly used guide the diffusion encoder learn the distribution invarrian representations through the diffusion backward process, and thus produces the Jomain--ubiasee pseudo labels; (2) a vanilla V-Net decoder with the proposed difficulty-awar re-weighting strategy is mainly to avoid the model over-fit fit to the easy and majority classes, and thus produces the cabss-d pseudo labels. Then, for the unlabeled data flow, the domain- and class-unbiased pseudo labels are ensembled through proposed Reparameterize & Smooth (RS) strategy to generate high quality pseudo labels. Finally, the pseudo labels are used to supervise the training of an additional V-Net decoder for prediction only.
3.2 Aggregating Stage
Assume that the entire dataset comprises of N,, labeled samples { u:) and Nv unlabeled Ny samples (x J==1- where v; E xD the input volume and y/s E RKxDxHXW is the ground- truth annotation with K classes. The goal of the aggregating stage is augment the data with SVDA and encode the labeled (x', y) and unlabeled data x" to high-level distribution-invarian features for denoising labeled data flow h difficulty-awee aware labeled data flow hiw,, and unlabeled data flow h..
Sampling-based Volumetric Data Augmentation (SVDA) Instead of the tone-consuming tra- ditional data augmentation used in (69) which cascaded all the augmentations, SVDA build upon an augmentation set and Naug operations are randomly sampled to apply to both the labeled and unlabeled data. The augmentation set consists of 3D spatial-wise transforms (random crop, random rotation, random scaling) and voxel-wise transforms (Gaussian blur, brightness, contrast, gamma). Naug is empirically set to 3.
5

---

Algorithm 1: Training Pipeline of A&D.
Input:
Labeled samples {(rl, Mi ) unlabeled samples {a No T==1
Output: Diffusion encoder Exx', z=; { with V-Net decoder D(x"; 0) for inference
Initialization:
for batched data (z); y.). x. do
Add SVDA on x' and x" to obtain s' and "";
Add step noise on Y/1 to obtain the noisy label ye by Eq.
Train the diffusion model F((x: and D(a'; {) with (i yt) by Eq. 2
Train the diffusion encoder E(x";E) and difficulty-aware decoder D ((r'; v) by Eq.
7
Generate domain-unbiann
probability map pas with E(x', x""; Da" E) by DDIM (77
Generate class-unbiased probability map pa: with D(x"; 1)) with forward pass;
Ensemble the two maps and obtain the pseudo label yf: by Eq.
Train Ex" E) and V-Net decoder (xx"; 0) with unlabeled data (*", y and by Eq. 8
10
end
Diffusion for Capturing Invariant Features We follow Diff-UNet (20 to use diffusion model for perception but modify to a V-Net version and remove the additional image encoder. Given the labeled volume data x' E R RDxxx and its label y E IR Pxxx x first convert the label to the one-hot format yo E R x and add successive step noise e to obtain the noisy label ye E R which the diffusion forward process:
Or Vi, (0,
V/
Then, the noisy label is concatenated with the image as the input of the Diff- Diet-VNet. Concretely, the high-level features different for different data flows. For the denoising flow, i.e., () as decoder, the diffusion encoder takes concatenation concat( ædI) and time step t as input to generate the time-step-embeddded multi-scale features ! E TR 1x 12 x where is the stage and F is the basic feature size. ii are further used by D)= to predict the clear label yo- For the difficulty-awe aware training flow and the unlabeled data flow, i.e,, ; and Dzx";;) as decoders, the encoder only takes rı and Fn as input obtain the multi-scale features h/1 hy, respectively. 1:st Note that It 1 are with same shapes with 1.
3.3 Decoupling Stage
The decoupling stage consists of four steps: supervised denoising training to generate domain- unbiased pseudo labels, supervised difficulty- training to generate class-unbiased pseudo labels, pseudo labeling to ensemble the two pseudo labels and unsupervised training to get the final predictor.
Supervised Denoising Training with the Diffusion Decoder D(+':E) Taking n as inputs, ()= decodes the features to predict the clear label yo as domain-unbiased pseudo label. The objective function is defined as follow:
Ldeno
1g y.)
Nr.
where (2, 1 (CCE(H,)) v) + C (x,, w) is the combined dice and cross entropy loss.
Supervised Difficulty-aware Training with D(r e)) To alleviate the common class imbalance issue in SSVMIS, we design a Difficulty-wware Re-weighting Strategy (DRS) to force the model to focus on the most difficult classes (i.e. the classes learned slower and with worse performances). The difficulty is modeled in two ways with the probability map p/s produced by diffusion decoder D(x:: E): learning speed and performance. We use Population Stability Index [70] measure the

---

learning speed of each class after the ex iteration:
Akie
duk.c Σ min(4, 0)nn(
dla
Σ max 0)1n(
AR.e-1
2
where d denotes the Dice score of pi of t class in e iteration and = AR.e Ak,e-1- duk,e and dl denote classes not learned and learned after eh iteration. T is the number accumulation iterations and set to 50 empirically. Then, we define the difficulty of w class after e iteration as:
duk.e
dR.e
dlk,c
where the classes learned faster have smaller dk.e. the corresponding weights in the loss function will be smaller to slow down the learn speed. We also accumulate λk.e for iterations obtain the reversed dice weight WA and weight dg,e. In this case, classes with lower dice scores will have larger weights in the loss function, which forces the model to pay more attention to these classes. The overall difficulty-wware weight of htth class is defined as:
diff = wxe,- (dr,e) w
(5)
where is empirically set to _ in the experiments alleviate outliers. The objective function of the supervised difficult-awann training is defined as follow:
N₂
aiff LDioCEE(PER
Casf Nr. K
Wk
(6)
Pseudo Labeling with Reparameterize & Smooth (RS) Strategy The domain-unbiased pas probability map is generated by iterating the diffusion model (E(z', <)) times with the Denoising Diffusion Implicit Models (DDIM) method [57]. The class- p"iti probability map can be obtained by D(a'; v)) with stopped gradient forward pass. We ensemble p and pd4 to generate high-quality pseudo labels. However, when combining these two maps, we found that denoised probability map p pdw is too sparse, i.e, with very high confidence of each class. This property is benefit for the fully-supervised tasks, but in this situation, it will suppress paw and is not robust to noise and error. Thus, we re- re-parameterize na with the Gumbel-Softmax to add some randomness and using Gaussian blur kernel to remove the noise brought by this operation. The final pseudo label is:
y F.v. argmax -argmaxGGumbel--Soan GumbellSSffmxxpp55 Sof Sof tmax(p
(7)
Unsupervised Training with (xx": 0) Finally, an unsupervised manner. The objective function of the unsupervised training is defined as:
the pseudo label of train D)=";
No
C.
DDiceC ..).
(8)
NU
==0
To better leverage the domain- and class-unbiased features, also transmit with knowledge distilla- tion strategy: 0 Wema x 0 + Werna) x (E + v)) Wema = 0.99. The overall training function of the A&G framework is:
L= L + Leiff + PCC,
(9)
where ja empirically set as 10 and follow 15] to use the epoch-dependent Gaussian ramp-up strategy to gradually enlarge the ratio of unsupervised loss. In the inference stage, only the diffusion encoder Flx'', ,=;; and Dxx"; 0) are used to generate the predictions.
Experiments
4.1 Datasets and Implementation Details
We evaluate our proposed A&D framework on four datasets for four tasks, i.e., LASeg dataset 72] for SSL, Synapse dataset [73) for class imbalanced SSL, MMWHS dataset (74) for UDA, and M&Ms

---

Table I: Results on Synapse dataset with 20% labeled data for class imbalanced SSL task. 'Common' or 'Imbalance' indicates whether the methods consider the imbalance issue or not. Notably, our method does not introduce many additional parameters compared with the existing SSL methods. Results of 3-times repeated experiments are reported in 'mean=std" format. Best results are boldfaced.
Avg- Dice
Avg. ASD
Dice of Each Class
Methods
LK
IVC
PSV
RAG
LAG
V-Net (fully)
62.09x1.2
10.28&3.9
84.6
77.2
73.8
73.3
38.2
94.6
68.4
72.1
71.2
58.2
48.5
17.9
29.0
UA-MT
8
20.26+2.2 2
71.67+7.4
48.2
31.7
22.2
0.0
0.0
81.2
29.1
23.3
27.5
0.0
0.0
0.0
0.0
Geneni
URPC
10
25.68+5.1
72.74=15.5
38.2
56.8
0.0
0.0
85.3
33.9
33.1
14.8
0.0
0,0
CPS
33.55+3.7
41.21+9.1
55.2
45.4
35.9
91.1
31.3
41.9
49.2
14.5
0.0
SS-Net
1
35.0842.8
50.8126.5
62.7
67.9
60.9
34.3
0.0
89.9
20.9
61.7
44.8
0.0
8.7
4.2
0.0
34.47+1.6
37.69+2.9
46.9
0.0
DePL
36.27+0.9
36,02+0.8
62.8
61.0
48.2
54,8
0.0
90.2
36.0
42.5
48.2
10.7
17.0
0.0
0.0
Adsh 77
35.29+0.5
+4.6
0.0
Imbalance
CRcST 20
38.33+3.4
22. 85=9.0
62.1
64.7
53.8
43.8
8.1
85.9
27.2
54.4
47.7
14.4
13.0
18.7
4.6
SimiS 28
40.07=006
32. 98+0.5
62.3
69
50.7
614
0.0
87.0
33.0
59.0
57.2
29.2
11.8
0.0
0.0
Basak
33.24#0.6
43.78+2.5
57.4
87.8
45.4
6.3
15.0
0.0
CLD
16
41,07+1.2
32. 15+3.3
62.0
66.0
59.3
61.5
0.0
89.0
31.7
62.8
49.4
28.6
18.5
0.0
5.0
DHC
A5
48.61+0.9
10.71+2.6
36.9
61.5
31.4
A&D (ours)
60.88+0.7
2.52400.4
85.2
66.9
67.0
52.7
62.9
89.6
52.1
83.0
74.9
41.8
43.4
44.8
27.2
Table 2: Results on two settings of LASeg dataset for SSL task.
Table Results on two settings of MMWHS
dataset for UDA task.
5% labeled data (labolsd.mlaaeelldd477)))
CT
Mothod
Dice
Jaccard
Metries
95HD
ASD
Method
LAC
LVC
Dice
MYO
Average
Average
ASD
V-Net (fully)
91.47
84.36
5.48
1.51
Supervised Training
92.7
91.1
91.9
87.8
90.9
2.2
V-Nett (5%)
52.55
39.60
47.05
9.87
PnP.AdaNet
74.0
68.9
61.9
50.8
63.9
12.8
UA.MT
13.71
AdaOutput
37
65.2
76.6
54.4
43.6
59.9
SASSNeL
81.60
68.63
16.16
3.58
CycleGAN
34
73.8
75.7
52.3
28.7
57.6
10.8
11
CyCADA
62.4
URPC [T0
SS-Net 1
MC-Net 12
83.59
82.48
86.33
71.35
72.36
76.15
14.65
14.07
9.97
3.68
2,70
2,31
SÍFA A1
DSAN [5]
DSPN
81.3
79.9
84.8
76.9
79.5
79.1
73.8
82.8
61.6
66.5
74.1
78.5
75.8
5.9
7.0
26
78.
LMISA-3D
84.5
82.8
88.6
70.1
81.5
A&D (ours)
89.93
81.82
5.25
1,86
A&D (ours)
93.2
89.5
91.7
86.2
90.1
1.7
10% labeled data Innellddanlahelldde53
CT MR
Metrics
Method
Dice
Jaccard
9511D
ASD
Method
Dice
ASD
LAC
LVC MYO
Average
Average
V-Net (10%)
82.74
71.72
13.35
3.26
UA-MT g
87.79
78.39
8.68
2.12
Supervised Training
80.5
92.4
SASSNet
PdPeed 38
43.7
68.9
61.9
50.8
63.9
8.9
DTC 11
87.51
78.17
8.23
2.36
AdaOutput
39.8
71.5
URPC
86.92
77.03
11.13
2.28
CycleGAN
34
64.3
30.7
65.0
43.0
50.7
6.6
LMISA-3D
22
86.06
76.53
12.99
2,41
CyCADA
36
60.5
44.0
77.6
47.9
57.5
XMINet 55
73.88
62.56
16.81
5.04
SIFA 41
65.3
62.3
78.9
47.3
63.4
5.7
MC-Net
12
87.62
78.25
10.03
1.82
DSAN 43
71.3
66.2
76.2
52.1
66.5
Net 17
88.55
79.62
7.49
1,90
LMISA-3D
60.7
72.4
86.2
64.1
BCP 76
89.62
81.31
6.81
1.76
SS-Net 17
62.1
58.4
68.9
51.4
60.2
5,9
A&D tours)
90.31
82.40
5.55
1.64
76
70.9
A&D (ours)
62.8
87.4
61.3
74.1
71.4
7.9
use set for validation, we labeled instead.
SOTA methods tailored for UDA and SemiDG, respectively.
SOTA meihods supervised segmentation.
dataset (75] for SemiDG. For more details, please refer to the Appendix. We implement the proposed framework with PyTorch, using a single NVIDIA A100 GPU. The network parameters are optimized by SGD with Nesterov and momentum of 0.9. We employ "poly" decay strategy follow [69]. For more implementation details, e.g., data preprocessing, learning rates, batch sizes, etc., please refer to the Appendix. We evaluate the prediction of the network with two metrics, including Dice and the average surface distance (ASD). For the LASeg dataset, we employ additional metrics, Jaccard and HD95, following [8) Final segmentation results are obtained using sliding window strategy.
4.2 Experiment results
Our proposed A&D framework achieves state-of-the-art on the four settings, i.e., SSL. (Table imbalanced SSL (Table UDA (Table B. and SemiDG (Table 4). In particular, method achieves

---

Table 4: Results on two settings of M&Ms dataset for SemiDG task.
2% Labeled data
5% Domain
Labeled data
Method
Domain
Domain C Domain
Average
Domain
Domain
Average
52.87
64.63
72.97
73.27
65.30
79.73
78.06
81.25
76.09
SDNet Aug
35
54,48
67.81
76.46
74.35
68.28
71.21
77.31
81.40
79.95
77.47
LDDG
59.47
56.16
68.21
68.56
63.16
69.49
73.40
75.66
71.29
SAML 79
56.31
56.32
75.70
69.94
64.57
67.11
76.35
77.43
78.64
74.88
BCP 76
71.57
76,20
76.87
77.94
75.65
73.66
79.04
77.01
78.49
77.05
DGNet 52
66.01
72.72
77.54
75.14
72.85
80.30
82.51
83.77
79.75
vMENee 35
73.13
77.01
81.57
82.02
78.43
77.06
82.29
84.01
85.13
82.12
(ours)
83.31
85.44
82.18
SOTA method un semi supervised segmentation.
FW
CINT
0169
2093
Figure 5: Ablation study on different architectures. (c) is the final framework
significant improvements over the existing fftte-of-the-art approach the Synapse dataset with 20% labeled data and the MMWHS dataset in the MR to CT setting. demonstrating a substantial increase of 12.3 in Dice score for the Synapse dataset and 8.5 for the MMWHS dataset.
4.3 Analyses
Architecture Analysis As shown in Figure we test dif- ferent architectures for the framework, the corresponding results are in Table When using three separate networks (a)(b), the performance drop significantly, the reason is that this structure can not learn the domain-invariant fea- tures. Using pure V-Net (c) or diffusion- (d) also leads to inferior results, especially for the pure diffusion model, it is hard to train due the limited labeled data, high-quality pseudo labels are hard to obtain, which further hinder the training the unsupervised branch.
Table 5: Ablation study on the MMWHS MR CT setting of different architec- tures with respect to those in Figure
Arch.
networks
Dice
76.14
50.53
82.19
45.63
90.14
Ablation on the components We analyze the effectiveness of different components in our method. According to the results in Ta- blel6] on MMWHS MR to CT setting (UDA), when removing the diffusion model, perfor- mance decreases the most, which indicated the importance ability of the diffusion model for capturing the instributioniinariinn fea- tures. The result of removing the SVDA also indicates that when the data is diverse enough, the diffusion model cannot capture effective underlying features. On Synapse dataset for IBSSL, the results are slightly different with those in the UDA setting, the DRS plays more important role than the Diffusion. In 5% M&MS dataset for SemiDG, the results are quite aligned with results in the UDA setting.
Table 6: Ablation study of the components in our framework on 20% labeled Synapse setting (IBSSL), MMWHS MR to CT setting (UDA) and 2% labeled 2% labeled M&Ms setting (SemiDG).
Methods
A&D
w/o SVDA
w/o Diflusion
weo DRS
wlo
IBSSI.
60.9
55.3
56.7
52.0
58.7
UDA
90.1
84.6
79.2
85.8
87.0
SemiDG
80.6
77.9
74.9
78.2
78.6
Selection We eval- uate the performance our method un- der different time step / of the diffusion model and the number of sampled aug- mentations, shown in Table
Table 7: Ablation study of the noise step and the number of sampled augmentations on MMWHS CT to MR setting.
Time step
100
500
1000
Aug
2
Dice
69.7
70.2
71.4
Dice
68.9
71.4
70.2

---

View
View
MR
GT
Pred
GT
Pred
Figure Visualization of the UDA task 2D and 3D views.
Visualization We also present the visualization results to further analyze method. As shown in F for the UDA task, our framework can generate smoother volumetric objects. Our framework also has detect minority classes, be seen in Appendix, which indicates the effectiveness of incorporating the class-imbalance awareness with the proposed difficulty-awan re-weighing strategy.
Limitations The diffusion process introduces additional training costs; however, the inference efficiency remains unaffected only the decoder trained using unlabeled data is utilized during infer- ence. Moreover, the failure cases are mainly in the M&Ms dataset for SemiDG setting. Specifically, our method usually fails on the first and the last slices along the depth axis. Due to the restricted depth dimension (less than 10), the 2D slices and the corresponding masks vary significantly. In such a case, hard for our volumetric framework to capture depth-wise information for the first or last slice with only one neighboring slice as a reference, and thus leads to false positive results.
Conclusion
In this paper, we propose generic framework for semi-supervised learning in volumetric medical image segmentation, called Aggregating Decoupling. This framework addresses four related settings, namely SSL, class imbalanced SSL, UDA, and SemiDG. Specifically, the aggregating part of our framework utilizes Diffusion encoder to construct "common knowledge set" by extracting cestrbutiooniivariiae features from aggregated information across multiple distributions/domainss the other hand, the decoupling part involves three decoders that facilitate the training process by decoupling labeled and unlabeled data, thus mitigating overfitting to labeled data, specific domains, and classes. Experimental results validate the effectiveness our proposed method under four settings.
The significance of this work lies in its ability to encourage supervise medical image segmen- tation methods to address more complex real-world application scenarios, rather than just developing frameworks ideal experimental environments. Furthermore, we have consolidated all four settings within single codebase, enabling the execution of any task using single bash file by merely adjusting the arguments. We believe that this consolidated codebase will convenient for further research and beneficial for the community.
Acknowledgement
We thank the anonymous NeurIPS reviewers for providing us with valuable feedback that greatly improved the quality of this paper! This work was supported part by the Hong Kong Innovation and Technology Fund under Project ITS//30/21 1 and in part by Foshan HKUST Projects under Grants FSUST2I-HKUSTI and FSUST21- HKUSTI IE and in part by the Project of Hetao Shenzhen- Hong Kong Science and Technology Innovation Cooperation Zone (HZQB-KCZYB
10

---

References
[1] A. Tarvainen and H. Valpola, "Mcan teachers are better role models: Weight-averagod consistency targets improve semi-supervised deep learning results," Advances in neural information processing systems, vol. 30, 2017.
[2] K. Sohn, D. Berthelot, N. Carlini, Z. Zhang, H. Zhang, C. A. Raffel, E. D. Cubuk, A. Kurakin, and C.-L. Li, "Fixmatch: Simplifying semi- supervised learning with consistency and confidence," " Advances in neural information processing systems, vol. 33, pp. 596 0766-608222 2020.
[3] X. Chen, Y. Yuan, G. Zeng, and J. Wang. "Semiisuuerree"
semantic segmentation with cross pseudo
supervision." in CVPR, pp. 2-12-2622. 2021.
[4] Y. Wang. H. Wang, Y. Shen, J. Fei, W. Li, G. Jin, I.. Wu, R. Zhao, and X. Le, "Semi-supervised semantic segmentation using unreliable pscudo-lahels," in CVPR, pp. 4248-4257, 2022.
[5] X. Wang. Z. Wu, L. Lian, and S. X. Yu, "Debiased learning from naturally imbalanced pseudo-labels." in
CVPR. pp. 14647-14657, 2022.
[6] B. Chen, J. Jiang, X. Wang. P. Wan, J. Wang, and M. Long, "Debiased self-training for semi-supervised
lcarning." in Advances in Neural Information Processing Systems, 2022.
[7] H. Wu, X. Li, and K.-T. Cheng, "Exploring feature representation learning for cuni--upereieed medical image segmentation." IEEE Transactions on Neural Networks and Learning Systems, 2023.
[8] L. Yu, S. Wang, X. Li, C.-W. Fu, and P.-A. Heng. "Uncertainty-awar self-ensembling model for semi- supervised 3d left atrium segmentation." in MICCAI, pp. 605-613, 2019.
[9] S. Li, C. Zhang. and X. He, "Shape-aware semi-supervised 3d semantic segmentation for medical images," in MICCAI, pp. 552-561, Springer, 2020.
[10] X. Luo. W. Liao, J. Chen, T. Song, Y. Chen. S. Zhang. N. Chen, G. Wang. and S. Zhang, "Efficient
semi supervuee gross target volume of nasopharyngeal carcinoma segmentation via uncertainty rectified pyramid consistency." in MICCAI. pp. 318-329, 2021.
[11] X. Luo, J. Chen, Song, and G. Wang, "Semi-supervised medical image segmentation through dual-task consistency," in Proceedings of the AAA/ Conference on Artificial Intelligence, vol. 35, pp. 8801-8809,
2021.
[12] Y. Wu, M. Xu, Z. Ge. J. Cai. and L. Zhang, "Semi-supervised left atrium segmentation with mutual consistency training." " in MICCAI. pp. 297-306, Springer. 2021.
[13] J. Wang and T. Lukasiewicz. "Rethinking bayesian deep learning methods for semi-supervised volumetric medical image segmentation," in CVPR, pp. 182-190, 2022.
[14] H. Wu, Z. Wang. Y. Song. L. Yang, and J. Qin, "Cross-patch dense contrastive learning for semi-supervised segmentation of cellular nuclei histopathologic images," in CVPR. pp. 11666-11675. 2022. [15] Y. Lin, H. Yao, Z. Li, G. Zheng. and X. Li, "Calibrating label distribution for class-imbalanced barely-
supervised knee segmentation." in MICCAI, pp. 109-118, 18, 2022.
[16] C. You, R. Zhao, L. H. Staib, and J. S. Duncan, "Momentum contrastive voxel-wise representation learning for sumi-supe volumetric medical image segmentation." " in MICCAI, pp. 633-652. 2022. [17] Y. Wu. Z. Wu, Q. Wu, Z. Ge, and J. Cai. "Exploring smoothness and daass-separttion for remi-supervised medical image segmentation." in MICCAI, pp. 34-43, 2022.
[18] H. Wu, X. Li, Y. Lin, and K.T Cheng, "Compete to win: Enhancing pseudo labels for barely-supervised
medical image segmentation," IEEE Transactions Medical Imaging. 2023.
[19] Ho, A. Jain, and P. Abbecl, "Denoising diffusion probabilistic models," " Advances Neural Information
Processing Systems, vol. 33. pp. 1840-6851, 2020.
[20] Z. Xing, L. Wan, H. Fu, G. Yang. and L. Zhu, "Diff-unet: A diffusion embedded network for volumetric
segmentation." arXiv preprint arXiv:2303. 2303.1326., 2023.
[21] J. Fan, B. Gao, H. Jin, and L. Jiang, "Ucc: Uncertainty guided cross-head co-training for semi-supervised
semantic segmentation," in CVPR. pp. 9947-9997999 995699566222 2022.
[22] Y. Ouali, C. Hudelot, and M. Tami, -emi-suuprvised semantic segmentation with cross- cosssconnsst training," in CVPR, pp. 12674-12684, 2020.
[23] X. Li, I. Yu, H. Chen, W. Fu, and P.-A. Heng, "Semi-supervised skin lesion segmentation via transfor- mation consistent self-ensembling model," BMVC, 2018.
[24] X. Li, L. Yu, H. Chen. C.WW. Fu, Xing, and P.-A. Heng, "Transformation-cossstt self-ensembling model for semisupervised medical image segmentation." JEEE Transactions on Neural Networks and
Learning Systems, vol. 32, no. 2. pp. 523-534, 2020.
[25] H. Wang and X. Li, "Dhe: Dual-debiased heterogencous co-training framework for class- imbalanced semi-supervised medical image segmentation," in MICCAI pp. 582-591, Springer, 2023.
[26] C. Wei, K. Sohn, C. Mellina, A. Yuille, and F. Yang, "Crest: A claas-ncbi self-training framework for imbalanced semi-supereised learning." in CVPR, pp. 10857-10866, 2021.
[27] Y. Fan, D. Dai, A. Kukleva, and B. Schiele, "Cossl: Co-learning of representation and classifier for imbalanced semi-supervised learning." " in CVPR, pp. 14574-14584, 2022.
[28] H. Chen, Y. Fan, Y. Wang, Wang. B. Schiele, X. Xie, M. Savvides, and B. Raj. "An embarrassingly
simple baseline for imbalanced semi-supervised learning." arXiv preprint arXiv:221 1.11086, 2022.
[29] Y. Oh, D.JJ Kim, and I. S. Kweon, "Daso: Distribution aware semantics-oriente pseudo-label for imbalanced remi-supervised learning." in CVPR, pp.
2022.
11

---

[30] Z. Lai, C. Wang. S.-c. Cheung, and C.-N. Chuah, "Sar: Self-adaptive refinement on pseudo labels for multiclas-imballnced sumi-supervised learning," in CVPR, pp. 4091-4100, 2022.
[31] Y. Hong. S. Han, K. Choi, S. Seo, B. Kim, and Chang, "Disentangling label distribution for long-tailed visual recognition." in CVPR, pp. 6626-66666 6636, 2021.
[32] S. Yu, Guo, R. Zhang. Y. Fan, Z. Wang, and X. Cheng, "A re- belanalnnigg strategy for class laas-immalanced classification based instance difficulty." in CVPR, pp. 70-79, 2022.
[33] H. Basak, S. Ghosal, and R. Sarkar, "Addressing class imbalance in semi-supervised image segmentation:
A study on cardiac mri," in MICCAI, pp. 224-233, 2022.
[34] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired image-to-image translation using cycle-consistent adversarial networks," in Proceedings of the IEEE international conference on computer vision, pp. 2223-
2232. 2017.
[35] Y. Huo, Z. Xu, Moon, S. Bao. A. Assad, K. Moyo, M. R. Savona, R. G. Abramson, and B. A. Landman, "Synssggnet: Synthetic segmentation without target modality ground truth," IEEE transactions on medical imaging, vol. 38, no. 4. pp. 1016-1025, 2018.
[36] J. Hoffman, E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Sacnko, A. Efros, and T. Darrell "Cycada: Cycle- consistent adversarial domain adaptation." " in International conference on machine learning, pp. 1989-1998,
Pmlr, 2018
[37] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker, "Learning to adapt structured output space for semantic segmentation," in CVPR, pp. 7472-7481, 2018.
[38] Q. Dou, C. Ouyang. C. Chen, H. Chen, B. Glocker, X. Zhuang, and P.-A. Heng. "Pup-adance" Plug- aagg-ndd-lay adversarial domain adaptation network at unpaired cross modality cardiac segmentation." IEEE Access.
vol. 7. pp. 99065-99077, 2019.
[39] C. Chen, Q. Dou, H. Chen, J. Qin, and P.-A. Heng, "Synergistic image and feature adaptation: Towards cross-modality domain adaptation for medical image segmentation," in Proceedings of the AAAI conference on artificial intelligence, vol. 33, pp. 865-872, 2019.
[40] Bateson, H. Kervadec, Dolz, H. Lombaert, and I. Ben Ayed. "Source-relaxed domain adaptation for image segmentation." " in MICCAI, pp. 490-499, Springer, 2020.
[41] C. Chen, Q Dou, H. Chen, J. Qin, and P. A. Heng, "Unsupervised bidirectional cross-modality adaptation via dceply syncrgistic image and feature alignment for medical image segmentation." IEEE transactions on medical imaging, vol. 39, no. 7, pp. 2494-2505, 2020.
[42] D. Zou, Q. Zhu, and P. Yan, "Unsupervised domain adaptation with dual-scheme fusion network for medical image segmentation.." in L/CAI, pp. 3291-3298, 2020.
[43] X. Han, L. Qi Q. Yu, Z. Zhou, Y. Zheng, Y. Shi, and Y. Gao, "Deep symmetric adaptation network
for cross-modality medical image segmentation." " IEEE transactions on medical imaging, vol. 41, no. 1,
pp. 121-132. 2021.
[44] K. Yao, Z. Su, K. Huang, X. Yang. J. Sun, A. Hussain, and F. Coenen, "A novel 3d unsupervised domain adaptation framework for cross-modality medical image segmentation," IEFE Journal of Biomedical and Health Informatics, vol. 26, no. 10, pp. 4976-4986, 2022.
[45] X. Bian, X. Luo, C. Wang, W. Liu, and X. Lin, "Dda-net: Unsupervised cross-modality medical image segmentation via dual domain adaptation," Computer Methods and Programs in Biomedicine, vol. 213,
p. 106531, 2022.
[46] H. Shin, H. Kim, S. Kim, Y. Jun, T. Eo, and D. Hwang, "Cosmos: Cross-modality unsupervised domain adaptation for 3d medical image segmentation based target-aware domain translation and iterative self-training," " arXiv preprint arXiv:2203. 16557, 2022.
[47] C. S. Perone, P. Ballester, R. C. Barros, and J. Cohen-Adad, "Unsupervised domain adaptation for medical Neurolmage, vol. 194, pp. 1-11, 2019. [48] Y. Xia, D. Yang. Z. Yu, F. Liu, J. Cai, L. Yu, Z. Zhu, D. Xu, A. Yuille, and H. Roth, "Uncertainty-wware multi-view co-training for semi-supervised medical image segmentation and domain adaptation," Medical
imaging segmentation with self-ensembling
image analysis, vol. 65, p. 101766, 2020.
[49] X. Liu, F. Xing. N. Shusharina, R. Lim, C.-C. Jay Kuo, G. El Fakhri, and J. Woo, "Act: Semi-supervised domain- adaptive medical image segmentation with asymmetric co-training." in MICCAI. pp. 66-76.
Springer. 2022.
[50] L. Liu, Z. Zhang, S. Li, K. Ma, and Y. Zheng, "S-cuda: Self-clcansing unsupervised domain adaptation for medical image segmentation," " Medical Image Analysis, vol. 74, p. 102214, 2021.
[51] M. Gu, S. Vesal, M. Thics, Z. Pan, F. Wagner, M. Rusu, A. Maier, and R. Kosti, "Confuda: Contrastive few-
shot unsupervised domain adaptation for medical image segmentation," arXiv preprint urXiv:2206.03888, 2022.
[52] X. Liu, S. Thermos, A. O'Neil, and S. A. Tsaftaris, 'Semi-supervin" meta-learning with disentanglement for domain-genenee gennralgeee medical image segmentation." in MICCAI, pp. 3177311, Springer, 2021.
[53] X. Liu, S. Thermos, A. Chartsias, A. O'Neil, and S. A. Tsaftaris, "Disentangled representations for domain-generalize generalized cardiac segmentation." in Statistical Atlases and Computational Models of the Heart. M&Ms and EMIDEC Challenges: 1th International Workshop, STACOM 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Revised Selected Papers II, pp. 187-195. Springer, 2021. [54] H. Yao, X. Hu, and X. Li, "Enhancing pseudo lahel quality for semi-supervised domain-generalle medical image segmentation," in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36.
12

---

pp. 3099-9907,
[55] X. Liu, S. Thermos. P. Sanchez, A. Q. O'Neil, and S. A. Tsaftaris, "vmfnee: Compasitionalliy meets
Somain--enerallied segmentation." " MICCAL. pp. 704-714, Springer, 2022.
[56] J. Sohl- Doch-Diictttin, E. Weiss, N. Maheswaranathan, and S. Ganguli, "Deep unsupervised learning using
nonequilibrium thermodynamies," in International Conference on Machine Learning, pp. 2256-2265,
PMLR, 2015.
[57] J. Song, C. Meng, and S. Ermon, "Denoising diffusion implicit models," arXiv preprint arXiv:2010002500,
2020.
[58] A. Q. Nichol and P. Dhariwal, "Improved denoising diffusion probabilistic models," in International Conference on Machine Learning, pp. 8162- 8171. PMLR, 2021.
[59] A. Nichol, P. Dhariwal, A. Ramesh, Shyam, P. Mishkin, B. McGrew, I. Sutskever, and M. Chen, "Glide: Towards photorealistic image generation and editing with text-guided diffusion models." arXiv preprint ArXiv:21 10741, 2021.
[60] P. Dhariwal and A. Nichol, "Diffusion models beat gans on image synthesis," " Advances in Neural Informa- tion Processing Systems, vol. 34, pp. 7780-8794, 2021,
[61] C. Saharia, J. Ho, W. Chan, T. Salimans, D. J. Fleet, and M. Norouzi, "Image super-resolution via iterative refinement," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[62] J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, "Video diffusion models," arXiv
preprint 2204:22044 03455, 2022.
[63] B. L. Trippe, J. Yim, D. Tischer, T. Broderick, D. Baker, R. Barzilay, and T. Jaakkola, "Diffusion
probabilistic modeling of protein backbones in 3d for the scaffoliaa problem." " arXiv preprint arXiv:2206.0411
2022.
[64] T. Amit, T. Shaharbany, F. Nachmani, and L.. Wolf, "Segdiff: Image segmentation with diffusion proba-
bilistic models," arXiv preprint arXiv:2112 12.00390, 2021.
[65] T. Chen, L. Li, S. Saxena, G. Hinton, and D. J. Fleet, "A generalist framework for panoptic segmentation of images and videos," arXiv preprint arXiv:2210.00 063606 2022.
[66] Y. Ji, Z. Chen, E. Xie, L. Hong. X. Liu, Z. Liu, T. Lu, Z. Li, and P. Luo, "Ddp: Diffusion model for dense
visual prediction," arXiv preprint ar arXiv:2303.
2023.
[67] J. Wu, H. Fang. Y. Zhang. Y. Yang, and Y. Xu, "Medsegdiff:: Medical image segmentation with diffusion probabilistic model," arXiv preprint arXiv:2211.0061 [68] J. Wolleb, R. Sandkühler, F. Bieder, P. Valmaggia, and P. C. Cattin, "Diffusion models for implicit
2022.
image segmentation ensembles," in International Conference on Medical Imaging with Deep Learning,
pp. 1336-1348, PMLR, 2022.
[69] F. Isensee, P. F. Jacger, S. A. Kohl, J. Petersen, and K. H. Meinr "nnu-net: configunng method for deep tearning-based biomedical image segmentation," Nature methods, vol. 18, no. 2. pp. 203-20 211, 2021.
[70] H. Jeffreys, "An invariant form for the prior probability in estimation problems," " Proceedings of the Royal Society of London. Series A. Mathematical and Physical Sciences, vol. 186, no. 1007, pp. 453-461, 1946.
[71] I..-.. Guo and Y.-F. Li, "Class-imbalanced semi-supervised learning with adaptive thresholding." in ICMI.,
pp. 2082-8094, PMLR, 2022.
[72] Z. Xiong, Q. Xia, Z. Hu, N. Huang, C. Bian, Y. Zheng, S. Vesal, N. Ravikumar, A. Maier, X. Yang, et al.,
"A global benchmark of algorithms for segmenting the left atrium from late gadolinimm-enhanee cardiac magnetic resonance imaging," Medical image analysis, vol. 67, p. 101832, 2021.
[73] B. Landman, Z. Xu, J. Igelsias, M. Styner, T. Langerak, and A. Klein, "2015 miccai multi-atlas labeling beyond the cranial vault- workshop and challenge," 2015.
[74] X. Zhuang and J. Shen, MMulti-scal patch and molti-moddaliy atlases for whole heart segmentation of mri," Medical image analysis, vol. 31, pp. 77-87, 2016.
[75] V. M. Campello, P. Gkontra, C. Izquierdo, C. Martin-Isla, A. Sojoudi, P. M. Full, K. Maier-Hein, Y. Zhang,
Z. He, Ma, et al., "Multi-centre, multi-vendor and multi-disease cardiac segmentation: the m&ms challenge." IEEE Transactions on Medical Imaging, vol. 40, no. 12, pp. 311333543333 3554,, 2021.
[76] Y. Bai. D. Chen, Q Li, W. Shen, and Y. Wang. "Bidirectional copy-pase for semi- semi-supeviised medical image segmentation." " in CVPR. pp. 11514-11524, 2023.
[77] M. Jafari, S. Francis, M. Garibaldi, and X. Chen, "Lmisa: A lightweight multi-modality image segmen- tation network via domain adaptation using gradicnt magnitude and shape constraint," Medical Image Analysis, vol. 81, 102536, 2022.
[78] H. Li, Y. Wang. R. Wan. S. Wang, T.-Q. Li, and A. Kot, "Domain generalization for medical imaging classification with linear-dependency regularization." Advances in Neural Information Processing Systems, vol. 33, pp. 31 3118-3122 129, 2020.
[79] Q Liu, Q. Dou, and P.-A. Heng, "Shape-aware meta-lcarning for generalizing prostate mri segmentation to unseen domains," in MICCAI, pp. 475-40-5 485, Springer, 2020.
13

---

A
Appendix
3D View
2D View
GT
A&D (ours)
CLD
Figure 7: Visualization of the class imbalance SSL task in 2D and 3D views. The yellow arrows denote minority classes detected.
A.1 More Details of Datasets and Implementation
The hyper-parameters for different datasets shown in Table
Table 8:
for different datasets.
Datasets
patch size
learning rate
batch size
feature size F
LASeg
112 x 112 x 80
le-2
4
32
Synapse
64 x 128 128
3e-2
4
32
MMWHS
128 128 128
5c-3
2
32
M&Ms
x 128
le-2
The details of the datasets and the pre-processing operations are as follows.
LASeg Dataset for SSL The Atrial Segmentation Challenge (LASeg) dataset 72 provides 100 3D gadolinium-enhanee MR imaging scans (GE-MRIs) and LA segmentation masks for training and validation. Following previous work 8 split the 100 scans into 80 for training and 20 for evaluation. We use the processed datasets from S where all the scans were cropped centering the heart region for better comparison of the segmentation performance of different methods and normalized as zero mean and unit variance. In the training stage, SS-Net [17] and BCP (76) use test for validation select the best model, which unreasonable. We use labeled data instead and achieve better performances.
Synapse Dataset for Class Imbalanced SSL The Synapse 773 dataset has 13 foreground classes, including spleen (Sp), right kidney (RK), left kidney (LK), gallbladder (Ga), esophagus (Es). liver(Li), stomach(St), aorta (Ao), inferior vena cava (IVC), portal splenic veins (PSV), pancreas (Pa), right adrenal gland (RAG), left adrenal gland (LAG) with one background and 30 axial contrast-enhanced abdominal CT scans. We randomly split them 20,4 and 6 seans for training, validation, and testing, respectively. Following DHC 25
we run the experiments 3 times with different random seeds.
MMWHS Dataset for UDA Multi-Modality Whole Heart Segmentation Challenge 2017 dataset (MMWHS) [74] is cardiac segmentation dataset including two modality images (MR and CT). Each modality contains 20 volumes collected from different sites, and no pair relationship exists between modalities. Following the previous work [41, we choose four classes of cardiac structures. They are the ascending aorta (AA), the left atrium blood cavity (LAA)), the left ventricle blood cavity (LVC), and the myocardium of the left ventricle (MYO), For the pre-processing, follow 41) (1) all the scans
14

---

were cropped centering at the heart region, with four cardiac substructures selected for segmentation; (2) for each 3D cropped image top 2% of its intensity histogram was off for alleviating artifacts; (3) each 3D image was then normalized to [0, l]. To make a fair comparison, we keep the test set the same with prior arts
M&Ms Dataset for SemiDG The multi-center, multi-vendor & multi-disease cardiac image segmentation (M&Ms) dataset [75] contains 320 subjects, which are scanned at six clinical centers in three different countries by using four different magnetic resonance scanner vendors, i.e., Siemens, Philips, GE, and Canon. We consider the subjects scanned from different vendors are from different domains (95 in domain A. 125 in domain B, 50 in domain C, and another 50 in domain D). We use each three of them as the source domain for training and the rest as the unseen domain for testing. For the pre- proceescesing (1) all the scans were cropped with four cardiac substructures selected for segmentation; (2) for each 3D cropped image top and bottom 0.5% of its intensity histogram was cut then normalized [0, 1]. Since the data has very
off for alleviating artifacts; (3) each 3D image few slices on the z-axis (less than 16), the previous work used 3D-based solutions. this work, since aim to design a generic framework for volumetric medical image segmentation, stacked the z-axis to 32 to meet the minor requirement for our encoder with four down-sampling layers. This dataset also be considered the extreme case of 3D segmentation tasks.
Table 9: Comparison of computational costs of state-of-the-art methods different settings with the performances in terms Dice score.
Methods
Param.
Inference time (s/iter)
SSL
UDA
SS-Net [17
75.672
21.20
88.55
78.2
CLD [1]]
75.554
39.58
85.37
75.4
vMFNet [55]
20.423
4.89
73.88
72.3
EPL [54]
80.939
5,52
76.49
71.9
A&D (ours)
57.894
3.69
90.31
90.1
RS
nug
Sum
Argmax
Pseudo label (red) on original image
Sofumax
Figure 8: Visualization of the RS process the foreground class on LASeg dataset. The probability map nrv of the difficulty-wware decoder may have low confidence in the inner region (red box), whereas the probability map p of the diffusion decoder may have inaccurate boundaries but with very high confidence (red arrows).
A.2 More Analyses
Comparison Computational Costs We compared the parameters and the inference time of our method with the most SOTA methods in different settings: SS-Net [17] on LASeg dataset for SSL, CLD 15 on Synapse dataset for IBSSL, and vMFNet [55] as well as EPL
54 on M&MSs dataset
for SemiDG, as shown in Table 9 From the table we can observe that our method has the fewest parameters except for vMFNet. Although vMFNet has only parameters, their performances (73,88% and 72.3%) SSL and UDA tasks are significantly lower than ours (89.40% and 90.0%).
15

---

RS
plet
Sum
Argmax
Pseudo label MYO on original image
Softmax
p lep
Figure 9: Visualization of the RS process of the myocardium of the left ventricle (MYO) class which the class with worst performance MR CT setting of MMWHS dataset. In this case, the probability map p"t of the diffusion decoder contains more error regions due to the ambiguous boundaries and noise but also with very high confidence.
As for the inference time, we can observe that our A&D is the fastest. This can be attributed to effective aggregating and decoupling strategies, enabling our method to exclusively utilize the unlabeled branch for inference.
Visualization of the Re- Re-parameterize & Smooth (RS) As shown in Figure the output probability map n of diffusion with DDIM D(E) is with very high confidence with its prediction, however, the results not stable since the unlabeled data is unseen during the training process of the diffusion decoder, especially for some problematic classes (MYO of MMWHS, Figure 9) with ambiguous boundaries and noise. Thus, if sum with the map p generated the error regions (e.g., upper right corner) with high confidence will surpass some correct regions of p wa with lower confidence and further harm the quality of the final pseudo label. Moreover, in some cases, the two output probability maps have complementary properties (Figure 8 indicating the effectiveness of ensembling them for the high-quality pseudo labels.
the V-Net decoder D(v),
Ablation on the Effectiveness of Decoupling the Labeled and Unlabeled Data Training Flows Based on our final framework, we add an additional training process with labeled data on the decoder (x 0) trained with unlabeled data to verify the effectiveness of the decoupling idea. Compared with the final A&D framework, when adding additional labeled data training branch, the performance in terms of Dice drops from 90.03% to 86.94% on the MR to CT setting of the MMWHS dataset. The result indicates that when the predictor is trained with labeled and unlabeled data, it may get over- fitted to the easier labeled data flow, which verifies the effectiveness of the key idea of our decoupling stage.
16


Reviews:
[{"review_id": "3PAbk5DEro", "reviewer": "Reviewer_suEd", "summary": "The paper presents a novel framework for semi-supervised medical segmentation that seeks to address three different scenarios simultaneously: lack of labelled data (standard semi-supervised learning -- SSL), domain shifts (unsupervised domain adaptation -- UDA) and generalization to new, unseen domains (semi-supervised domain generalization -- SemiDG). Toward this goal, they combine several strategies in their framework: sampling-based data augmentation, diffusion V-Net for learning domain invariant representations, difficulty-aware supervised training to generate class-unbiased pseudo-labels, and a reparameterize and smooth (RS) technique based on on the Gumbel softmax & ensembling strategy to generate high-quality pseudo-labels. The proposed approach is tested on four different datasets and tasks: LASeg for the SS taskL, Synapse for class imbalanced SSL, MMWHS for UDA, and M&Ms for SemiDG. Results show the method to achieve a similar or better performance on all tasks compared to SOTA methods.\n\n\n", "questions": "* p2: \"Currently, methods for these three scenarios are optimized separately, and there is no existing approach that addresses all three scenarios within a unified framework.\" As I understand, SSL and UDA are subcases of SemiDG, hence methods designed for the latter should be able to handle all three. I would relevant to compare the proposed approach against those in all three settings.\n\n* p2: \"Thus, the training process will easily get overwhelmed by the supervised training task.\" It is unclear to me why this is the case since unlabelled data is typically used to regularize the model trained with a supervised loss, and one of the main goals of regularization is to avoid overfitting. Authors should better validate this claim in their experiments.\n\n* p5: \" SVDA build upon an augmentation set and Naug operations are randomly sampled to apply to both the labelled and unlabelled data.\" This is in fact a very common strategy in SSL.\n\n* Section 3.3: Normally, diffusion models require a lot of data to train. How is it possible to achieve this using only a few labelled samples ?\n \n* Section 3.3: In the diffusion model, why do you add noise on the ground-truth segmentation maps? Assuming that the domain shift occurs in the image space (the anatomy should not vary from one modality or site to another), what can this achieve?\n\n* Figure 4: I am confused about the diffusion encoder taking both a labelled and an unlabelled image as input, since they have different shapes. Can authors explain?\n\n* Eq (2): the notation is confusing. Is 'e' the index of the sum or not? \n\n* Eq (2): Technically, in the standard definition of PSI, you should have a Delta in the sum (as it is similar to a KL)\n\n* Eq (2)-(3): As I understand, du_{k,e} is always negative while dl_{k,e} is always positive, hence w_diff should be negative?\n\n* Eq (4): Why not use a standard focal loss to address this issue?\n\n* p6: \"we found that the denoised probability map pu;ψ is too sparse, i.e., with very high confidence of each class.\" Why not use a different softmax temperature or label smoothing to avoid this ?\n\n* p7: For the class imbalanced scenario, why not used a class prior as in Bateson et al, since these priors are known from anatomical references\n\nBateson, Mathilde, Hoel Kervadec, Jose Dolz, Hervé Lombaert, and Ismail Ben Ayed. \"Source-free domain adaptation for image segmentation.\" Medical Image Analysis 82 (2022): 102617.\n\n* Table 3: the results for UDA on MMWHS are almost too good. It seems unlikely that the proposed method can achieve a Dice score similar to that of supervised trainining and a better ASD. Have authors checked for data leakage? Can they comment on this?\n\n\n\n\n\n\n \n\n\n\n \n ", "limitations": "The discussion of limitations could be expanded (only the training time of the diffusion model is mentioned). Any failure cases?\n\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The proposed framework brings together three different medical image segmentation scenarios involving unlabelled data: SSL, UDA and SemiDG. This bridges a gap in the literature, as existing approaches typically focus on a single of these scenarios.\n\n* Authors do a good job in the Introduction (Figs 1-3) to situate their method with respect to existing approaches and highlight the limitations of these approaches. \n\n* Experiments to evaluate the proposed method are comprehensive. The proposed method is evaluated in four different scenarios and, fr each one, compared against the SOTA. The experimental validation also includes several relevant ablation studies that demonstrate the usefulness of the method's different components.\n\n* Results show the method to achieve SOTA performance on all tasks. These results are particularly impressive for the UDA task on the MMWHS (MR to CT) where the method achieves a performance on par with supervised training (more on this later).", "weaknesses": "* From my understanding, the technical novelty of the work comes from the combination of several existing techniques to address different problems in SSL. While some results are good, the proposed method feels heavily-engineered and complex.\n\n* Experiments do not fully support the main claims of the paper. For example, current experiments employ different baselines to compare the method in the SSL, UDA and SemiDG settings, instead of testing the same approaches (or some of them) across these settings. Hence, the claim that current approaches can only handle one setting is not truly validated. Likewise, the claim that having separate branches for labelled and unlabelled is not well validated in the ablation study on architectures since all tested models implement this idea (as I understand)  \n\n*  In terms of results, the method yields more limited improvements for the SSL and SemiDG settings. It is unclear if these are statistically significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere gratitude to reviewer suEd for the recognition of our idea. We hope our answers can address all your concerns. We are looking forward to having further discussions with you.\n\n**Q1: p2: \"Currently, methods for …\"**\n\nWhile current Semi-DG methods can be evaluated within SSL and UDA, they have yet to furnish results within these specific scenarios. To address this gap, we evaluate the state-of-the-art SemiDG methods in the SSL and UDA tasks, as shown below.\n\nFor SSL on 10% labeled LASeg dataset:\n|Method| Dice|\n|--| --|\n|SS-Net (SOTA in SSL)|88.55|\n|vMFNet|73.88 |\n|EPL_SemiDG|76.49|\n|A\\&D (ours)|89.40|\n\nFor UDA of MR to CT on MMWHS dataset:\n|Method|Dice|\n|--|--|\n|LMISA-3D (SOTA in UDA)|81.5|\n|vMFNet|72.3|\n|EPL_SemiDG|71.9|\n|A\\&D (ours)|90.0|\n\nThis superiority of our approach over SemiDG methods in SSL and UDA settings can be attributed to the efficacy of our proposed SVDA, A&D strategy with diffusion, and DRS, which adeptly addresses scarcity of labeled data, domain shifts, and imbalanced class distributions.\n\n[1] Liu, et al. \"vMFNet: Compositionality Meets Domain-Generalised Segmentation.\" MICCAI, 2022.\n\n[2] Yao et al.. \"Enhancing pseudo label quality for semi-supervised domain-generalized medical image segmentation.\" AAAI, 2022.\n\n**Q2: p2: \"Thus, the training process …**\n\nIn theory, unlabeled data serves to mitigate overfitting. However, Figure 3 indicates that even when there is a significantly larger amount of unlabeled data compared to labeled data, the impact of unlabeled data on regularization is limited. This is due to the fact that labeled data, guided by precise masks for supervision, converges more rapidly.\n\n**Q3: p5: \" SVDA build upon …**\n\nWe agree that augmentation is a very common strategy. However, our SVDA, comprises a combination of 3D transformations, is designed to empower Diffusion-vnet in generating a more diverse dataset. It enhances feature diversity within the limited medical images and aids in fully harnessing the capabilities of our diffusion-based A&D framework across various scenarios.\n\n**Q4: Normally, diffusion models …**\n\nWhile it's true that generative diffusion models often demand a substantial amount of data for training, the scenario changes when employing them for perceptual tasks in volumetric medical image segmentation. In such cases, even a limited number of samples can yield favorable performance [3], which is due to the inherent nature of volumetric samples, wherein a single sample typically includes over 200 2D slices. \n\n[3] Xing et al. \"Diff-UNet: A Diffusion Embedded Network for Volumetric Segmentation.\" arXiv, 2023.\n\n**Q5: In the diffusion model …**\n\nAdding noise to the label can perform better in perceptual tasks [3,4]. Following [3, 4], we concatenate the noisy label with image embeddings, the diffusion model needs to denoise these concatenated noisy features and recover the clear label. Thus, the diffusion model can jointly learn the image-level feature.\n\n[4] Ji et al. DDP: diffusion model for dense visual prediction. ICCV, 2023\n\n**Q6: Figure 4: I am confused …**\n\nThe training processes of the labeled and unlabeled data are sequential. In each training iteration, the model first takes the labeled data to train the modules highlighted in blue within Figure 4, then generates the pseudo labels. Finally, the model takes the unlabeled data to train the encoder again with the orange-colored decoder. We employ if-else statements and different embedding layers to handle variations in shapes.\n\n**Q7: Eq (2): the notation …**\n\nYes, 'e' is the index of the sum to indicate sum from $e-\\tau$ iteration to e iteration.\n\n**Q8: Eq (2): Technically …**\n\nSorry, it is a mistake, $\\mathbb{I}(\\cdot)$ function should be $\\mathbb{I}_u(x)=min(x,0)$ and  $\\mathbb{I}_l(x)= max(x,0)$ .\n\n**Q9: Eq (2)-(3): As I understand …**\n\nActually, $du_{k,e}$ and $dl_{k,e}$ are both always positive since in $du_{k,e}$, $\\Delta<0$, $ln(\\cdot)<0$, so $du_{k,e}>0$; in $dl_{k,e}$, when $\\Delta>0$, $ln(\\cdot)>0$, so $dl_{k,e}>0$. Hence, $w_{diff}$ is always positive.\n\n**Q10: Eq (4): Why not use …**\n\nThe focal loss relies on a predefined class distribution. However, in SSL tasks, only the class distribution of labeled data is known and constant, while the distribution of unlabeled data remains uncertain and evolves dynamically throughout training. Consequently, employing a standard focal loss directly is unsuitable for addressing imbalanced SSL tasks.\n\n**Q11: p6: \"we found that …**\n\nWe thank the reviewer for the suggestion. We add new ablation experiments by replacing the our RS with softmax temperature and label smoothing. The results are shown below:\n\n| | Dice|\n|--|--|\n|A\\&D w/ RS (ours)|90.0|\n|A\\&D w/o $p^{u;\\xi}$(lower bound)|87.3|\n|A\\&D w/ softmax temperature 2|88.0|\n|A\\&D w/ softmax temperature 4|87.7|\n|A\\&D w/ label smoothing|86.5|\n\nThe degradations of the performance are due to the fact that these smoothing strategies lead to a reduction in the confidence of true positive regions. As a result, when combined with $p^{u;\\psi}$ to obtain the pseudo label, $p^{u;\\xi}$ will be overwhelmed.\n\n\n**Q12: p7: For the class imbalanced scenario ...**\n\nThe class prior approach requires a substantial amount of prior knowledge, demanding task-specific ratio computations. While it might lead to improved performance, it lacks generality, as obtaining the class prior for certain tasks could be challenging. In contrast, our A&D framework yields favorable results in a wider range of practical scenarios.\n\n**Q13: Table 3: the results …**\n\nWe have reviewed the code carefully and are confident that no data leakage issues are present. Upon acceptance, we will release both our code to the public.\n\n**Limitations**\n\nOne limitation lies in the modest enhancements over the SemiDG benchmarks. This limitation arises from the MRI data in the SemiDG dataset, which possesses restricted depth dimension (less than 10). Consequently, the performance of our model, founded upon the 3D V-net architecture, is limited. \n"}, {"review_id": "ZtrzJfyzrS", "reviewer": "Reviewer_yd6u", "summary": "This paper proposes an Aggregating & Decoupling framework, which unifies the SSL, Class Imbalanced SSL, UDA, and SemiDG for volumetric medical image segmentation with one generic framework.", "questions": "1. How are the running time and parameters of the proposed method compared to other methods?\n2. The proposed approach appears to require separate training for different tasks, and whether it is possible to train only once for the SSL, Class Imbalanced SSL, UDA, and SemiDG, and truly achieve a generic framework.\n3. Do ablation experiments on other datasets also demonstrate the performance of the proposed architecture and components?", "limitations": "The authors have analyzed the limitations of the proposed method.\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper explores a generic framework for the SSL, Class Imbalanced SSL, UDA, and SemiDG of volumetric medical image segmentation.", "weaknesses": "The description of the method is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere gratitude to reviewer yd6u for the recognition of our idea. We hope our answers can address all your concerns. We are looking forward to having further discussions with you.\n\n**Q1: How are the running time** **and parameters of the proposed method compared to other methods?**\n\nWe compared the parameters and the inference time of our method with the most SOTA methods in different settings: SS-Net on LASeg dataset for SSL, CLD on Synapse dataset for IBSSL, and vMFNet as well as EPL_SemiDG on M&Ms dataset for SemiDG:\n\n| Methods | Param.(M) | Inference time (s/iter) | SSL| UDA|\n| --| -- | --| --| -- |\n| SS-Net | 75.672 |21.20| 88.55 | 78.2 |\n| CLD | 75.554 | 39.58 | 85.37| 75.4|\n| vMFNet | 20.423 | 4.89 | 73.88 | 72.3 |\n| EPL_SemiDG | 80.939 | 5.52 | 76.49 | 71.9|\n| A\\&D (ours) | 57.894 | 3.69 | 89.40 | 90.0 |\n\nFrom the table we can observe that our method has the fewest parameters except for vMFNet. Although vMFNet has only 20 M parameters, their performances (73.88% and 72.3%) on SSL and UDA tasks are significantly lower than ours (89.40% and 90.0%).\n\nAs for the inference time, we can observe that our A&D is the fastest. This can be attributed to our effective aggregating and decoupling strategies, enabling our method to exclusively utilize the unlabeled branch for inference.\n\n\n**Q2: The proposed approach appears to require separate training for different tasks, and whether it is possible to train only once for the SSL, Class Imbalanced SSL, UDA, and SemiDG, and truly achieve a generic framework.**\n\nWe appreciate the reviewer's suggestion regarding training the model once and testing it across SSL, UDA, and SemiDG settings. Undoubtedly, this concept holds great value. However, its viability hinges on the presence of a uniform segmentation task across these settings, implying that the same target must be segmented in SSL, UDA, and SemiDG scenarios. However, our present paper employs widely-used benchmark datasets, which were chosen to ensure a fair comparison with existing state-of-the-art SSL, class-imbalanced SSL, UDA, and SemiDG methods, respectively. It's crucial to know that these benchmark datasets do not share a consistent segmentation objective. Specifically, prior best-performing SSL methods employs the LASeg dataset for benchmarking left atrium segmentation, the Class Imbalanced SSL methods employs the Synapse dataset to segment 13 abdominal organs, and prior best-performing UDA methods employ the MMWHS dataset to segment 3 cardiac structures. Consequently, the inconsistent segmentation objectives within these datasets render the implementation of the aforementioned idea unfeasible.\n\nTo achieve the aforementioned idea, we will endeavor to collect an extensive dataset from various hospitals and conduct a comprehensive and equitable assessment of the top-performing SSL, Imbalanced SSL, UDA, and SemiDG methods in the new benchmark.\n\n**Q3: Do ablation experiments on other datasets also demonstrate the performance of the proposed architecture and components?**\n\nWe add additional ablation experiments on 20% labeled Synapse and 5% labeled M&Ms datasets, the results are as follows:\n\n| Methods | IBSSL (20% labeled Synapse) | SemiDG (2% labeled M&Ms) |\n| --| -- | -- |\n| A&D | 60.2 | 78.7 |\n| w/o SVDA | 55.3 | 76.5|\n| w/o Diffusion | 56.7 | 72.9|\n| w/o DRS | 52.0 | 76.8|\n| w/o RS | 58.7 | 77.1|\n\nOn Synapse dataset for IBSSL, the results are slightly different with those in the UDA setting, the DRS plays more important role than the Diffusion. In 5% M&Ms dataset for SemiDG, the results are quite aligned with results in the UDA setting."}, {"review_id": "LiZlcXJYRw", "reviewer": "Reviewer_GWTw", "summary": "This work proposes a generic framework for semi-supervised learning (SSL) in medical image segmentation. They identify two obstacles that prevent the applicability of SSL in real-world scenarios. Then they propose Aggregating & Decoupling to alleviate the issues under several settings, including class imbalanced SSL, UDA, and SemiDG. They show the proposed method can mitigate overfitting to labeled data, specific domains, and classes across four settings and many datasets. ", "questions": "See weakness. ", "limitations": "Yes. ", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-structured and easy to follow. The proposed method is generalizable to four different settings, enhancing the applicability of SSL in real-world scenarios. The presentation is very clear, especially the introduction, where the obstacles, and motivations are clearly stated. ", "weaknesses": "My biggest concern about this work is the effectiveness in more generic applications, e.g., standard computer vision benchmarks. I appreciate the authors identifying obstacles that hurt the performance of SSL in real-world applications. However, what is the relationship between the proposed approach and medical image segmentation? Is any proposed component tailored to deal with the issues with medical images? In my eye, the proposed framework is generic and should be working widely well in many applications instead of only focusing on medical images. It would be great if the authors could verify the effectiveness of the proposed methods in CV benchmarks instead of only medical imaging applications if the proposed method is not designed or tailored to medical settings. I would consider re-rating if the authors response to my concern. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere gratitude to reviewer GWTw for the recognition of our idea and suggestion for extending the application scenarios to natural images. We hope our answers can address all your concerns. We are looking forward to having further discussions with you.\n\n**Weaknesses: My biggest concern about this work is the effectiveness in more generic applications ...**\n\nWe thank the reviewer for this insightful comment.\n\n**Regarding the relationship of our method with medical image segmentation:**\n\n(1) Our DRS is tailored for the imbalance issue in medical images. In volumetric medical image segmentation, the MRI/CT scans are obtained by scanning the same region of the patients, hence the anatomical structure and segmentation targets within each dataset the similar character but vary in size and texture. **The imbalance issue lies in organs with tiny sizes, which can be regarded as local imbalance.** In contrast, natural images are diverse and from complex scenarios, the objects with each image, **the imbalance issue lies in the objects with scarce samples, which can be regarded as global imbalance**. DRS computes the imbalance ratio within **each training sample**, thus, is tailored for the local imbalance issue in medical images.\n\n(2) The evaluation of our method is in a patient-by-patient manner, the failure to segment a target has larger impacts on the overall results, unlike the class-by-class approach commonly applied to natural images. Therefore, our method places a stronger emphasis on accurately handling small objects. This design is tailored for the clinical settings, where even minor mis-segmentation of organs and objects can exert a significant impact on the overall results.\n\n(3) Our framework, particularly the aggregating stage of the diffusion model, serves as a robust backbone for the volumetric medical image segmentation. Powerful pre-trained feature extractors is unavailable in SSVMIS domain due to the limited data. Observed the failure of existing 3D encoders for extracting domain-invariant features, we propose to aggregate the multi-domain information within both scarce labeled and unlabeled data into one Diffusion-VNet encoder to solve this issue.\n\n(4) Our SVDA comprises a combination of 3D spatial- and voxel-wise transformations which are tailored for the gray-scale MRI/CT volumetric data, in contrast with the 2D augmentations for RGB natural images.\n\n(5) Our aggregating and decoupling strategies for diffusion are tailored for the limited medial data compared with large amounts of natural images. In natural image datasets, both the labeled and unlabeled data are abundant, hence the observation in Figure 3 is uncommon. However, in medical image datasets, even the unlabeled data is scarce, leading to over-fitting problem of the labeled data. We decouple the training of labeled and unlabeled data to solve this issue.\n\n**Regarding the reason why our method can’t be applied to CV:**\n\n(1) Our DRS computes the imbalance ratio in a **sample-by-sample manner** to solve the aforementioned local imbalance issue in medical image. In the natural image domain, the classes may be quite balanced in one sample but seriously imbalanced across the whole training set. Thus, applying our DRS to natural images will lead to incorrect representations of the class-wise difficulty.\n\n(2) When applied to CV, our perception-based diffusion model is hard to pre-train since it also takes the masks as inputs. Thus, it is hard to align with the SOTA approaches equipped with powerful pre-trained backbones in the natural image domain.\n\n(3) Our SVDA is tailored for the gray-scale volumetric data, which is unsuitable for the RGB 2D data in natural images.\n\n(4) Notably, we have evaluated several SOTA CV semi-supervised semantic segmentation methods (CPS and GTA [3]) on our tasks but obtained inferior results, as shown in the table below.\n\n\n| Method | SSL | IBSSL |\n| --| -- | --|\n| CPS (CVPR'21) | 87.91 | 37.4 |\n| GTA (NeurIPS'22) | 88.25 | 39.3|\n| A\\&D (ours) | 89.40| 60.2 |\n\nMoreover, directly extending existing CV semi-supervised classification methods to segmentation with CPS as the baseline is also not effective, as shown in Table 1 of the paper.\n\n[1] Fan, Jiashuo, et al. \"Ucc: Uncertainty guided cross-head co-training for semi-supervised semantic segmentation.\" CVPR, 2022.\n[2] Jin, Ying, Jiaqi Wang, and Dahua Lin. \"Semi-supervised semantic segmentation via gentle teaching assistant.\" Advances in Neural Information Processing Systems 35 (2022): 2803-2816."}, {"review_id": "yUYcbaLmBB", "reviewer": "Reviewer_HPgc", "summary": "This paper addressed both the semi-supervised issue and the domain adaptation issue for medical image segmentation. The proposed method incorporated the diffusion model and used a consistency-based regularization for training. Performance on several public datasets is good for the segmentation tasks.", "questions": "See the above weaknesses", "limitations": "See the above weaknesses", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Applying a diffusion model to reduce the domain gaps is reasonable and achieves performance gains according to the results.\n2. Extensive experiments and sufficient comparisons.\n3. The motivation is sound. Considering both tasks together is now the research attention.", "weaknesses": "1. There are some recent papers considering the two tasks as well. See:\n[1] Bai Y, Chen D, Li Q, et al. Bidirectional Copy-Paste for Semi-Supervised Medical Image Segmentation[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 11514-11524.\n[2] Wu H, Li X, Lin Y, et al. Compete to Win: Enhancing Pseudo Labels for Barely-supervised Medical Image Segmentation[J]. IEEE Transactions on Medical Imaging, 2023.\nTherefore, the problem definition is new.\n\n2. It's a bit over-claimed to unify four tasks together. For example, this paper does not provide unique designs for imbalanced learning. Do not make the claims too broad.\n3. The performance is not SOTA, especially on the popular LA dataset, see [1]. Moreover, about the comparison, it is weird that there are different compared methods for different tasks.\n4. Regarding the main task (semi-supervised learning), the technical insight is not clear. The domain gap should be a critical issue here. Please elaborate more on the take-home insights rather than tricks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere gratitude to reviewer HPgc. We hope our answers can address all your concerns. We are looking forward to having further discussions with you.\n\n**Q1: There are some recent papers considering the two tasks as well ...**\n\nWe believe there might be some misunderstandings regarding the differences between the mentioned papers and our own work. The papers the reviewer mentioned utilized datasets like the ACDC dataset, Pancreas-NIH dataset, Colon cancer segmentation dataset, and LA dataset—all originating from a consistent source domain. As a result, their focus was confined to situations without any domain shift, i.e., \"semi-supervised setting.\" However, our work is distinctive in that it can achieve the best performance on four distinct settings simultaneously: SSL, Imbalanced SSL, UDA, and SemiDG. This expansive coverage underscores our ability to address a broader range of challenges. As such, our problem formulation and methodology stand as a novel contribution.\n\n**Q2: It's a bit over-claimed to unify four tasks together. For example, this paper does not provide unique designs for imbalanced learning. Do not make the claims too broad.**\n\nIndeed, we introduce a novel approach to address imbalanced learning - our Difficulty-aware Re-weighting Strategy (DRS). Moreover, our proposed method demonstrates a notable superiority over prior state-of-the-art solutions within the class-imbalanced SSL settings, as clearly illustrated in Table 1. Based on these achievements, we are confident that our approach effectively unifies the Imbalanced SSL setting.\n\n**Q3: The performance is not SOTA, especially on the popular LA dataset, see [1]. Moreover, about the comparison, it is weird that there are different compared methods for different tasks.**\n\n(1) The results in [1] reveal Dice values of 88.02 and 89.62 for LA datasets with 5% and 10% labeled data, respectively. While these values do show a slight improvement over our method's performance, which records scores of 86.95 and 89.40 in the same scenarios, it's important to note that our approach is designed to cater to broader scenarios. Specifically, our method addresses not only the SSL setting, as [1] does, but also encompasses imbalanced SSL, UDA, and SemiDG settings, making it more versatile and applicable across various challenges.\n\n(2) Different compared methods for different tasks is due to that we conducted fair comparisons with existing state-of-the-art methods in the SSL, UDA, and SemiDG benchmarks. Currently, these three settings and their corresponding methods were individually formulated and designed, leading to the diversity in the compared approaches. However, it's worth noting that our method consistently outperforms the best-performing methods tailored to these three settings. Our approach consistently achieves superior performance across all three settings, demonstrating its efficacy and generalization than other existing STOA methods.\n\n**Q4: Regarding the main task (semi-supervised learning), the technical insight is not clear. The domain gap should be a critical issue here. Please elaborate more on the take-home insights rather than tricks.**\n\nRegarding the semi-supervised learning task, our key technical contribution pertains to the identification of the limited impact of unlabeled data in the training process of the existing SSL frameworks, which is due to the unlabeled data easily getting overwhelmed by the labeled data. Aiming to solve this issue, we design the Aggregating & Decoupling framework to decouple the labeled data and the unlabeled data training branches. The motivation is mainly illustrated in Figure 3, and the validation of the decoupling process is demonstrated in the Appendix."}, {"review_id": "xLsQIum3hP", "reviewer": "Reviewer_GWMf", "summary": "This paper tackles semi-supervised medical image segmentation over multiple domains, generalizing unsupervised domain adaptation and semi-supervised learning. A multiple branch encoder-decoder architecture with a shared encoder is proposed. The first branch is a diffusion model with V-Net architecture trained over labeled images and their noisy labels, where the encoder aims to extract domain-invariant features. The second decoder is trained over labeled images to combat class-imbalance by reweighting the loss with respect to Population Stability Index. The third branch is trained over unlabeled images with pseudolabels, where pseudolabels are ensembles of probability maps predicted by the other two decoders. To account for the sparse predictions of diffusion model decoder, its probability predictions are reparamatrized via Gumbel-Softmax prior to ensembling into pseudolabels. Each branch is trained over a combination of cross-entropy and dice loss, with input images and labels varying with respect to the branch. ", "questions": "None", "limitations": "Limitations are discussed", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Adopting diffusion models for generalized semi-supervised segmentation and domain adaptation is novel. Abundant experiments over multiple tasks and datasets demonstrate the benefit of the proposed method, particularly for unsupervised domain adaptation. \n", "weaknesses": "Unsupervised/semi-supervised segmentation and domain adaptation in medical imaging are well-explored domains with several missing related works:\n\nGu et al. “ConFUDA: Contrastive Fewshot Unsupervised Domain Adaptation for Medical Image Segmentation”, 2022\n\nXia et al. “Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation”, 2020\n\nBian et al. “DDA-Net: Unsupervised cross-modality medical image segmentation via dual domain adaptation”, 2021\n\nPerone et al. “Unsupervised domain adaptation for medical imaging segmentation with self-ensembling”, 2019\n\nShin et al. “COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training”, 2022\n\nBateson et al. “Source-Relaxed Domain Adaptation for Image Segmentation”, 2021\n\nLiu et al. “S-CUDA: Self-cleansing unsupervised domain adaptation for medical image segmentation”, 2021\n\nLiu et al. “ACT: Semi-supervised Domain-adaptive Medical Image Segmentation with Asymmetric Co-Training”, 2022\n\nParticularly, Liu et al. 2022 propose decoupled semi-supervised training and domain adaptation branches, a similar motivation to the proposed method. The authors should highlight their novelty against these methods that are not discussed. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincere gratitude to reviewer GWMf for the recognition of our idea and suggestion for supplementing the related work. We hope our answers can address all your concerns. We are looking forward to having further discussions with you.\n\nRegarding the differences with ACT:\n\n(1) **The settings are different.** ACT is tailored specifically for Semi-supervised Domain Adaptation, which is a simpler setting than Unsupervised Domain Adaptation since some labeled data from the target domain is available for training. Our goal is to build a generic framework that unifies four settings, including UDA. Furthermore, our A&D can also solve the SemiDA setting.\n\n(2) **The motivation for decoupling and the data we want to decouple are different.** ACT decouples the semi-supervised training and domain adaptation branches based on the observation that **abundant labeled source data** can easily dominate training when directly combined with **limited labeled target data**. However, we found that the **limited labeled data** can dominate training when combined with **abundant unlabeled data**, hence we decouple them to avoid the unsupervised branch overfit with the labeled data. Experiments in Figure 3 and Appendix A.2 support and verify of our motivation.\n\nIn terms of the **novelty** compared to previous works, our work is the first to unify SSL, Imbalanced SSL, UDA, and SemiDG. This extension not only amplifies the scope and versatility of SSL-based frameworks in medical image segmentation but also stands in stark contrast to earlier approaches that remained confined to singular domains such as SSL or UDA.\n\nThe significance of this work lies in its ability to encourage semi-supervised medical image segmentation methods to address more complex real-world application scenarios, rather than just developing frameworks in ideal experimental environments. Furthermore, we have consolidated all four settings within a single codebase, enabling the execution of any task using a single bash file by merely adjusting the arguments. We believe that this consolidated codebase will be convenient for further research and beneficial for the community."}]

Author rebuttal:

Meta-review: The paper introduces a semi-supervised learning (SSL) for 3D medical image segmentation to work in three scenarios: 1) SSL and imbalanced SSL; 2) unsupervised domain adaptation (UDA); and 3) semi-supervised domain generalisation. The proposed method combines data augmentation, diffusion model with V-Net architecture for domain invariant representations, difficulty-aware supervised training to produce class-balanced pseudo-labels, and an approach to generate high-quality pseudo-labels. Four datasets have been used in the experiments, namely: LASeg for the SS taskL, Synapse for class imbalanced SSL, MMWHS for UDA, and M&Ms for SemiDG. Results on these datasets demonstrate that the proposed approach has competitive or better performance than the SOTA methods. The paper received the following scores: 7,5,5,4,4. The main strengths of the paper are: 1) interesting unification of the SSL, UDA and SemiDG tasks; 2) nice motivation for the paper; 3) comprehensive evaluation; 4) SOTA performance on all tasks; and 5) well-written. The main weaknesses found by reviewers were: 1) method looks heavily-engineered and complex; 2) unclear description of the method; 3) unclear translation to tasks outside medical image analysis; 4) unclear if claims are too broad; 5) experiments missing SOTA results on the LA dataset; 6) unclear take-home message to explain the SSL SOTA results; 7) several related works are missing [1,2,3,4,5,6,7,8]. The authors did a good job to clarify the issues in the rebuttal. In general, the paper shows a solid approach, with good results. However, the technique seems to be incremental without a clear contribution.

[1] Gu et al. “ConFUDA: Contrastive Fewshot Unsupervised Domain Adaptation for Medical Image Segmentation”, 2022 [2] Xia et al. “Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation”, 2020 [3] Bian et al. “DDA-Net: Unsupervised cross-modality medical image segmentation via dual domain adaptation”, 2021 [4] Perone et al. “Unsupervised domain adaptation for medical imaging segmentation with self-ensembling”, 2019 [5] Shin et al. “COSMOS: Cross-Modality Unsupervised Domain Adaptation for 3D Medical Image Segmentation based on Target-aware Domain Translation and Iterative Self-Training”, 2022 [6] Bateson et al. “Source-Relaxed Domain Adaptation for Image Segmentation”, 2021 [7] Liu et al. “S-CUDA: Self-cleansing unsupervised domain adaptation for medical image segmentation”, 2021 [8] Liu et al. “ACT: Semi-supervised Domain-adaptive Medical Image Segmentation with Asymmetric Co-Training”, 2022
Decision: Accept (poster)