VQ-Seg: Vector-Quantized Token Perturbation for Semi-Supervised Medical Image Segmentation
Authors: Sicheng Yang
Zhaohu Xing
Lei Zhu
Venue: NeurIPS 2025 poster ()
Keywords: Vector Quantization
Semi-Supervised Learning
Medical Image Segmentation
Primary area: machine_learning_for_sciences
TL;DR: VQ-Seg uses QPM and a dual-branch design with foundation model-guided alignment to enable controllable perturbations for consistency learning, achieving strong performance on two datasets for medical semi-supervised segmentation.

Abstract: Consistency learning with feature perturbation is a widely used strategy in semi-supervised medical image segmentation. However, many existing perturbation methods rely on dropout, and thus require a careful manual tuning of the dropout rate, which is a sensitive hyperparameter and often difficult to optimize and may lead to suboptimal regularization. To overcome this limitation, we propose VQ-Seg, the first approach to employ vector quantization (VQ) to discretize the feature space and introduce a novel and controllable Quantized Perturbation Module (QPM) that replaces dropout. Our QPM perturbs discrete representations by shuffling the spatial locations of codebook indices, enabling effective and controllable regularization. To mitigate potential information loss caused by quantization, we design a dual-branch architecture where the post-quantization feature space is shared by both image reconstruction and segmentation tasks. Moreover, we introduce a Post-VQ Feature Adapter (PFA) to incorporate guidance from a foundation model (FM), supplementing the high-level semantic information lost during quantization. Furthermore, we collect a large-scale Lung Cancer (LC) dataset comprising 828 CT scans annotated for central-type lung carcinoma. Extensive experiments on the LC dataset and other public benchmarks demonstrate the effectiveness of our method, which outperforms state-of-the-art approaches. Codes will be released.

VQ-Seg: Vector-Quantized Token Perturbation for Semi-Supervised Medical Image Segmentation
Sicheng Yang!* Zhaohu Xing Lei Zhu
The Hong Kong University of Science and Technology (Guangzhou) The Hong Kong University of Science and Technology
Abstract
Consistency learning with feature perturbation is widely used strategy in semi- supervised medical image segmentation. However, many existing perturbation methods rely on dropout, and thus require careful manual tuning of the dropout rate, which a sensitive hyperparameter and often difficult to optimize and may lead to suboptimal regularization. To overcome this limitation, we propose VQ-Seg, the first approach to employ vector quantization (VQ) to discretize the feature space and introduce novel and controllable Quantized Perturbation Module (QPM) that replaces dropout. Our QPM perturbs discrete representations by shuffling the spatial locations of codebook indices, enabling effective and controllable regularization. To mitigate potential information loss caused by quantization, we design a dual- branch architecture where the cost-quantization feature space is shared by both image reconstruction and segmentation tasks. Moreover, we introduce a Post- VQ Feature Adapter (PFA) to incorporate guidance from foundation model (FM), supplementing the high-level semantic information lost during quantization. Furthermore, we collect large-scale Lung Cancer (L.C) dataset comprising 828 CT scans annotated for central-type lung carcinoma. Extensive experiments on the LC dataset and other public benchmarks demonstrate the effectiveness our method, which outperforms state-of-the -ftt--f--tee-rt approaches. Codes will be released
Introduction
Medical image segmentation serves as a fundamental step in numerous clinical applications, such as disease diagnosis 1 anatomical structure delineation 64 lesion localization 77, and surgical planning [10 In recent years, supervised deep learning methods have demonstrated outstanding performance segmentation tasks, significantly surpassing traditional approaches both accuracy and robustness [55.15) However, these methods typically require large amounts of finely annotated medical data, the collection of which not only expensive and labor-intensive but also demands substantial domain expertise. To address this limitation, semi-supervised learning, which leverages small set of labeled data alongside larger pool unlabeled samples, has emerged as promising direction and is receiving growing attention in the medical imaging community.
Consistency learning represents a widely adopted paradigm in semi-supervised medical image segmentation, designed to enforce prediction invariance under various perturbations [16]. Feature- level dropout [17] 16] commonly employed technique within this framework, introducing perturbation into intermediate representations to enhance model robustness. However, its effectiveness critically dependent on the selection of the dropout rate (DR), hyperparameter.
"Equal contribution.
!Lei Zhu (leizhu@ust.hk) is the corresponding author.
https://githhb..oo/// wwwptoo//////ooooo......oooooott Yane-/nnn///SSee
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

As depicted in Fig. our experiments reveal two primary challenges associated with this methodology. Firstly, the adoption of lower dropout rates (e.g., DR-0.3, DR=0.5) yields negligible impact on segmentation performance. This suggests that the induced perturbation is insufficient to provide meaningful regularization. Secondly, elevating the dropout rate (e.g., DR 2 0.7) leads to a rapid decline in performance metrics. Specifically, Dice and Jaccard scores exhibit sharp decrease, while HD95 and ASD values significantly increase, indicating a substantial degradation in both structural accuracy and boundary delineation. Qualitative analyses further corroborate these findings, demonstrating that segmentation outputs under high dropout rates frequently fail to yield meaningful segmentation results, rendering them practically unusable.
These observations underscore the inherent difficulty in identifying an optimal dropout rate that consistently enhances performance while mitigating the risk of model collapse. Consequently, there a pressing need for a more stable perturbation strategy, thereby achieving the desired regularization effect in predictable and robust manner.
In this paper, we introduce VQ-Seg, a novel semi- supervised framework for medical image segmentation. The core innovations of our approach include: first, the Quantized Perturbation Module (QPM) designed to re- place traditional dropout by enabling controlled and struc- tured perturbations of encoded features within a discrete VQ space. Unlike dropout, which relies on hyperparam- eters such as the dropout rate, QPM leverages distances between codebook codewords to define perturbation strate- gies, thereby offering enhanced interpretability and stabil- ity. To address potential visual information loss arising from vector quantization, we tackle this issue from two perspectives. Initially, we construct a dual-branch archi- tecture that shares a post-quantized space, unifying image reconstruction and semantic segmentation tasks within a joint optimization framework in discrete representation space. This design facilitates the preservation of critical structural information from images while also utilizing the reconstruction task as a self-supprvisory signal to en- courage the VQ encoder to learn improved representations. Furthermore, to enhance semantic consistency and miti- gate the loss of high-level semantic information during quantization, we incorporate a foundation model-guided alignment strategy. Specifically, we develop a Post- VQ Feature Adapter (PFA) that employs contrastive learning to align quantized features with semantic features derived from a pre- rre-traied visual foundation model. This approach effectively enriches the semantic content and spatial consistency of discrete representations. Extensive experiments conducted on our collected Lung Cancer (LC) dataset (comprising 828 annotated cases) and the open-source ACDC dataset demonstrate that VQ-Seg significantly outperforms state-of-the art semi-supervised segmentation methods across key evaluation metrics, including Dice, Jaccard, HD95, and ASD. Detailed ablation studies further validate the efficacy and synergistic effects of the individual components of VQ-Seg.
Soymentation Metiks Dropout
250
0.8
200
0.4
0.0
0.0
0.2
0.4
0.5
3R-00
DR-055
DR-0.7
D8:000
Figure 1: Effect of dropout rate on segmentation performance in a fully supervised setting on the LC dataset. Low dropout rates show negligible impact, whereas a high dropout rate (DR 2 0.7) severely degrades both quan- titative metrics and visual outputs. No- tably, DR = 0.9 leads to unusable pre- dictions, highlighting the challenge of selecting an optimal dropout rate.
In summary, our contributions are five-fold:
We collected a new large-scale dataset, the Lung Cancer (LC) dataset, comprising 828 chest CT scans with annotations of central-type carcinoma of the lung.
We propose novel Quantized Perturbation Module (QPM) perturb features within discrete vector quantization (VQ) space. QPM enables a more structured and controllable mechanism for representation perturbation by shuffling the spatial locations of codebook indices, offering enhanced interpretability and stability compared to traditional dropout.
We introduce a dual-branch architecture where the post-quantized space is concurrently utilized for both image reconstruction and the downstream segmentation task. This design

---

uses reconstruction self-suuprvisory signal, encouraging the VQ encoder to learn better representations and preserving essential visual information.
We develop a foundation model-guided alignment strategy, where a frozen foundation model (FM) serves as an external semantic prior to guide and regularize the Post-VQ representation. A Post-VQ Feature Adapter (PFA) is introduced to transform the quantized codebook embeddings into semantically aligned space using contrastive learning, mitigating the loss high-level semantic features.
We compare our method with multiple cutting-edge methods on the LC dataset and open- source datasets, demonstrating that our approach achieves state-of- of-the-art performance, which verifies its effectiveness.
2 Related work
2.1 Semi-supervised Medical Image Segmentation
Obtaining large-scale, high-quality manual annotations for medical images is both time-consuming and labor-intensive [18] [9].. Semii-spprrvieed learning (SSL) methods have thus gained attention as
promising solution for medical image segmentation under limited annotation settings [20]. Among
based approaches are widely adopted due to their simplicity typically involve training an initial model on the labeled dataset and then generating pseudo-labels. However, the use of pseudo-labels can introduce noise and lead to training instability, especially when incorrect labels are propagated [26]. To address this, subsequent research (27-29 has incorporated consistency regularization to enforce prior constraints on the learned representations. Consistency regularization is grounded in the smoothness assumption, which posits that small perturbations the input data should not significantly alter the model's predictions (30). In this context, the design and implementation of perturbations play critical role in determining the effectiveness of the model.
various SSL strategies,
and ease of implementation [21 These methods [22 25
2.2 Feature-level Dropout for Consistency Regularization
We focus on feature-level Dropout strategies. Leveraging Dropout- based regularization for con- sistency learning has become a widespread approach in semi-supervised medical image segmen- tation 331. Specifically, techniques such as Monte Carlo Dropout (MC-Dropout) [16] have been employed to model uncertainty and enforce prediction consistency under random feature pertur- bations. These perturbations help improve the generalization ability of the model by encouraging it to make stable and robust predictions despite the uncertainty inherent in the data. Several stud- ies [66 32 [29] have shown the effeectiveness of such methods, demonstrating improved performance in semi-supervised medical image Segmentation tasks. However, while these approaches have shown promise, they typically require the careful tuning of typprparameter-tthe Dropout rate. The appro- priate setting of the Dropout rate is critical. Both peer observations [29] and our own empirical studies (see Fig. TD suggest that this process is often challenging in practice, as the optimal Dropout rate may vary depending on the specific dataset, task, and network architecture. In particular, inappropriate Dropout configurations prevents the model from learning effectively [33 31). This highlights a significant limitation of traditional Dropout-based methods. This situation motivates the need for alternative strategies that can provide controlled and effective feature-level perturbations.
lead to unstable training dynamics, such as excessive regularization that
3 Methodology
3.1 Overview
Fig. provides an overview of our Vector Quantization-baat based Semi cemi-supervised Segmentation frame- work (VQ-Seg). The input image is first encoded into continuous latent features, which are then quantized into a discrete codebook space via the Vector Quantization (VQ) module. introduce structured perturbations for consistency learning, we propose the Quantized Perturbation Module (QPM), which shuffles codebook indices based on their learned distances, offering a stable and interpretable alternative to dropout. To compensate for potential visual information loss, VQ-Seg
3

---

Calign
FM
PRA
fm
m t m
1
D2 *
Query
2
Senc
Sample
3
a
Sdec
2g
Codebook
Argeno
N-3
*
Sobe cnd
EMA
x
Update
N-2
QPM
EMA
D
N-1
Tenc
Tdec
STF
EMA
Sanc
Penc
Unlabeled
Augmented
Labeled
Codebook token
FM token.
Tdee
Figure 2: Overview of the VQ-Seg framework. The input image is encoded into continuous features 2, which are then quantized into a discrete codebook space via vector quantization (VQ). Quantized Perturbation Module (QPM) introduces controllable perturbations for consistency learning. The dual-branch architecture jointly optimizes image reconstruction and segmentation using the shared Post-VQ features. Additionally, Post-VQ Feature Adapter (PFA) aligns the quantized features with semantic embeddings from foundation model (FM).
adopts dual-branch architecture that jointly optimizes reconstruction and segmentation tasks using the shared Post-VQ feature space. Furthermore, the Post-V Vo Feature Adapter (PFA) aligns quan- tized features with semantic embeddings from pre-trained foundation model through patch-wise contrastive learning, enriching representation semantics and reducing drift.
3.2 Theoretical Motivation
To analyze the sensitivity of the model to perturbations in the feature space, interpret the KL divergence (L(PI[Q) as a measure of the perturbation radius from the original distribution P to the perturbed one Q. This is inspired by distributionally robust optimization (DRO) B34 33). where the worst-case risk evaluated over an uncertainty set defined by a bounded KL divergence. Thus, the divergence reflects the extent to which the input perturbation has structurally shifted the feature representation. Supported by recent studies (36), for Dropout, the KI. divergence between the posterior distribution (induced by dropout) and prior distribution P can be approximated
DRI(PIIQ) R
1(_ 1-p
log(1 12)
where p E (0, 1) denotes the dropout rate. As increases, the approximation indicates a growing perturbation radius, with the KL divergence rising sharply (see Appendix A for full derivation). Such behavior reveals the inherent instability of dropout from a theoretical standpoint: large dropout rate causes the posterior distribution to deviate significantly from the prior, potentially leading to over-regularization and degraded learning performance, as supported by our empirical results (see Fig. To mitigate this issue, we propose the Quantized Perturbation Module (QPM), which perturbs features within discrete vector quantization (VQ) space, enabling a more structured and controllable mechanism for representation perturbation.

---

30%
*
FM
3%
Resize
1x1
4)G <(211) 0.69
PFA
d(G,,6)) x(311)
Lalign
dCG, x(4]1) 0.03
fpla
<(11))
Figure 3: concrete example of the Quantized Perturbation Mechanism (QPM) with a code- book size of K = 4 and a perturbation strength 0.7. It illustrates the probabilistic transi- tions from the original codeword (index 1) itself and other codewords (02, C3, Cạ) with their respective probabilities π (i]l), where the transition to 02 (49%) exhibits the highest prob- ability of replacement.
Figure 4: Architecture of the Post- Post- cest-Quantization
Feature Adapter (PFA) designed for aligning post- quantization features with a frozen Foundation Model (FM) via a patch-wise contrastive loss, Calign. The PFA initially employs a resizing opera- tion followed by a x convolution match the spatial resolution and channel dimensionality of FM features, thereby facilitating subsequent semantic alignment.
3.3 Quantized Perturbation Module (QPM)
In our method, the encoder output fenc (x) represents a continuous feature embedding of the input medical image This embedding is then projected into discrete latent space by selecting the nearest codeword from learnable codebook {c1, C22 .... Cg }, where K denotes the total number of codewords, i.e., the size of the codebook. The quantized codeword index is obtained by
arg
min jE je(I...,k)
||
cyll
After the encoding step, we apply perturbation strategy x(j i). For each original codeword c; (with index i), a new codeword сy (with index j) is sampled based on the conditional probability T( i), and Çj replaces the original codeword as the input to the decoder. This procedure introduces structured perturbation within discrete latent space, resulting form of regularization. To implement this perturbation, we first define prior distribution P(c+) over the codebook Z. We assume a uniform distribution over all codewords:
more controlled and interpretable
1 K'
P(c)
vi (1.2, K}.
(3)
The perturbation strategy defines the conditional probability of transitioning from the current code- word e; to another codeword ej:
11-e, rexpí rexp(-d(e,.s) d(c,e;))
if if ن
(j ()
%
where [0, 1] is control term controlling the perturbation strength, d(c;, is distance metric between codewords &; and Cj. and Z; = Σxze exp(-d(cs, cx)) is normalization factor. The resulting perturbed distribution Q(cyle) over the codewords is then given by:
K Q(cyke) = Σ P(c))(i
1- É K
8.
K
Σ igj
Enys
kiyi exp( d(c;, cx))
exp(-d(c,
(5)
1=1
The KL. divergence between the prior distribution P and the perturbed distribution Q is:
DkI(P E P(c,) log
P(()) (Q(c,ee) )
1
Σ
K
log (KQ(c,le))
(6)
Compared Dropout, our QPM offers several advantages. The perturbed distribution Q(cyle) is always well-defined and bounded, ensuring numerical stability (Detailed proof can be found
5

---

in Appendix B). QPM is directly controlled via a single control term and is influenced by the learned codebook structure. Moreover. QPM introduces structured perturbations by probabilistically transitioning between learned codebook entries based on their distances, yielding a potentially more interpretable and controllable form of regularization compared to the stochastic perturbations inherent in Dropout (refer to an example presented in Fig. B) The perturbation remains within the learned discrete latent space, offering a distinct approach to regularization.
3.4 Dual-Branch Architecture with Shared Post-VQ Space
Visual information inherently exists in a continuous space. The process of quantization, while enabling discrete representations, can lead loss of fine-grained details and potentially reduce the representational capacity for modeling intricate visual structures [37] 38 To address this, we introduce dual-branch architecture where the custtquantization feature space (Post-VO Space), is concurrently utilized for both image reconstruction and the downstream segmentation task. By jointly optimizing the quantized feature space with respect to these two objectives, we encourage to encode fundamental structural information, as well as segmentation-relleaat representations, This dual utilization facilitates the preservation of essential visual information within the Post-VC Space, without compromising the performance of the segmentation task.
As depicted in Fig. 2 both the teacher (T) and student (S) decoder networks adopt this design. Following the encoding stages and vector quantization, we obtain the discrete representations ((2))
for æı, q(22) ) for Za. ((=]) for Iu processed by the student, and q(="") for I. processed by the
teacher. The QPM-perturbed quantized representation of the unlabeled data is denoted as g' (x(2))
QPM (g(2)) )) which is used by the student network. These discrete representations, where ( E
(. (1(2)) serve as the input to the image decoder (D,) and the segmentation decoder (D,), as described by:
{g(=)) ((2))))))))))))))))))))))))))q q(²")) (=TT).
$ = Di(a())), Ds((()))
The loss function for the labeled data æι with its associated ground truth segmentation y is defined as:
Li Cre + Ceeg G
(8)
For the unlabeled data Fas we employ a pseudo-labeling strategy. Initially, a predicted segmentation map is generated by the teacher network Tace as y = DJ )), with a corresponding recon- structed image o = D (.. Subsequently, a pseudo-label V/ is derived from Y by selecting the class with the maximum probability (argmax operation). This pseudo-label Th. is then used as the target segmentation for the QPM-perturbed quantized representation g'zzu) of the unlabeled data in the student network Sace. The loss for the unlabeled data in the student network Saee integrates both the reconstruction loss of the original unlabeled data and a segmentation loss based on the pscudo-label applied to the perturbed representation:
L, Cree (wu a5) + Laeg (Wu:: 15) + Loeg(in 12).
(9)
The overall dual- bual-branch loss is defined as:
Lab Li + Xuh
(10)
where A, is a hyperparameter that balances the contribution of the unlabeled loss Cu relative to the labeled loss L. Lree denotes the L1 loss, and Leeg represents the Cross-Entropy loss. By jointly minimizing Lab. we encourage the shared quantized feature space to encode information beneficial for both reconstructing the input image and segmenting relevant structures. This synergistic optimization leverages supervision from labeled data and self-supervisory signals from unlabeled data via labelin,,,,,ing, where the teacher network generates pseudo-labels to guide the student network's 's learning on the QPM-perturbed quantized representations of the unlabeled data.
3.5 Foundation Model-Guided Alignment for Post-VQ Space
Although vector quantization (VQ) compacts the feature space, its discretization process introduce semantic bias and loss of fine details (39], which particularly detrimental in nigh-precision tasks such as medical image segmentation. To mitigate these issues, we propose a foundation model-guided

---

aligmment strategy, where frozen foundation model (FM) serves as an external semantic prior guide and regularize the Post- VQ representation. Specifically, we introduce Post- VQ Feature Adapter (PFA) that transforms the quantized codebook embeddings into semantically aligned space, as illustrated in Fig. The PFA first resizes the VQ features and then applies match the spatial resolution and channel dimension of the FM features. Denoting the output of the PFA fela R xW" and the corresponding FM features as fTD We then apply patch-wise contrastive learning objective [40] 41 to minimize the semantic discrepancy between the adapted VQ features and the FM representations:
convolution to
1 HW
HW
exp
(iim[SPth )/T)
Calign
log
(11)
HW 2j=1 exp ffth f3mm)/ //T
where sim(a, | [ and is the temperature parameter. f. and f. represent the feature vectors at spatial location i (flattened from the 2D grid) extracted from the adapted VQ features and the FM features, respectively, both with dimensionality C". The indices i E (1..... H'W" correspond to all patch locations on the H' x W" spatial grid. The patch-wise formulation enables localized semantic supervision, allowing the model to align not just global representations but also fine-graine spatial semantics. By minimizing Calign the discretized features are encouraged to retain rich and spatially semantic information that are consistent with the external FM prior, effectively mitigating the loss of detail and semantic drift introduced during quantization. In practice, we adopt as the foundation model to provide semantic supervision. DINOv2 42
3.6 Total Optimization Objective
Drawing upon the loss formulations in equations TO and 11 framework is defined as follows:
the overall optimization objective of our
L Lub + Àa Calign:
(12)
where A is a hyperparameter that balances the two loss terms. A decrease in Las indicates that the model is learning more effective representations for reconstructing images and segmenting relevant lesion regions. Furthermore, the optimization of Calign suggests that stronger consistency is achieved between quantized features and external foundation model priors, thereby mitigating detail loss and semantic shift introduced during the quantization process. Detailed discussions on the hyperparameter can be found in the ablation study part 4.5
Experiments
4.1 Datasets.
Lung Cancer (LC) Dataset. We collect a multi-center dataset including 828 chest CT scans of central-type lung carcinoma, which reveals the inherent challenges associated with detecting and analyzing such cases, presenting subtle anomalies within the imaging data. There exists one segmentation target per volume, and the dominant lesion annotated precisely for each case.
This dataset 43 is a cardiac MRI collection comprising 100 short-axis cine-MRI scans, acquired using both 3T and 1.5T scanners.
ACDC dataset.
Following previous studies, we applied the 70-10-20 split ratio for training, validation, and testing on the LC dataset. To ensure a fair comparison, all experiments were conducted 2D slices.
4.2 Implementation Details.
Our model is implemented using Py PoTch 2.0.1 1 with CUDA 11.8 and MONAL 1.3.0. All slices are cropped to 128 128 and used as input, with a batch size of 4 per GPU. The training process runs for 100 epochs, employing the cross-entropy loss and an SGD optimizer with polynomial learning rate scheduler (initial learning rate of 10-4 and decay of 50-5). As shown in Fig. 2 several data augmentation strategies are applied, including random rotation, color jittering, Gaussian noise, and
7

---

Table I: Quantitative comparison the LC dataset with two labeled ratio settings (5%, 10%) using four metrics: Dice and Jaccard (f), HD95 and ASD ()). Best results are in bold, second best are underlined.
5% Labeled
10% Labeled
Method
Dice
Jaccardf
HD995,
ASD1
Dicef
Jaccard
HD95J
ASD
UNet-F
0.8345
0.7386
6.9634
2.2913
0.8345
0.7386
6,9634
2.2913
UNet-S +7
0.4343
0.3118
26.0498
12.6188
0,6490
0.5175
21.4063
7.3382
nnUNet-F
0.8259
0.7236
4.2533
1.4216
0.8259
0.7236
4.2533
1.4216
nnUNet-S
0,4590
0.3438
13.2746
8.8636
0.6538
0,5194
25.2100
8,9332
UA-MT 16
0.6029
0.4647
48.6681
24.6020
0.7222
0.5989
11.6724
5,4939
MCNet
0.6378
0.4970
15.2759
4.9231
0.7555
0.6414
16.1903
9.9647
SSNet 49
0.6328
0.4886
25.1005
9.3180
0.7480
0.6278
14.9581
7,3399
BCP
50
0.6243
0.4854
26.9303
10.4789
0.7252
0.5994
18.9768
6,5105
ARCO
0.6162
0.4778
36.2256
14.6243
0.7246
0.5945
14,4803
4,3660
ABD
52
0.6414
0.5024
12.5608
5.9661
0.7468
0.6244
12.6570
6.7437
Unimatch
53
0.6493
0.5071
17,8700
5,4526
0.7511
0.6333
17,0178
5.7388
Ours
0.6643
0.5257
12.2525
4.2276
0,7852
0.6731
11.6179
4.2094
papperr WE
poparr
MCNet
BCP
ARCO
ABD
Unimalch
Ous
GT
Figure 5: Visual results on LC with 5% and 10% labeled data show that VQ-Seg consistently yields more accurate predictions of anatomical structures and boundaries than all other compared methods.
blurring. To enable gradient backpropagation through the non-differentiable operations in our model, we employ the Straight-hhrough Estimator (STE) technique 44 which approximates gradients during the backward pass. In our main experiments, we set the VQ codebook size to 16,384 and further conduct corresponding ablation studies (see Sec. 4.5 to evaluate its influence. Moreover, the codebook mapping mechanism 45 46 is incorporated to accelerate the learning dynamics the codebook. An Exponential Moving Average (EMA) strategy is applied to update the teacher network, where the parameters of the teacher encoder and decoder are updated from the student's parameters as follows: O, t- Og + a)- O., where 0, and 0, represent the parameters of the teacher and student networks, respectively, and a 0.99 denotes the EMA decay rate. All experiments are conducted on cloud computing platform equipped with four NVIDIA GeForce RTX 4090 GPUs.
4.3 Evaluation and Metrics.
We compare our method with other state-of-the-art (SOTA) approaches, including UNet 47 UA-
MT [16], MCNet (48), SSNet [49], BCP [50]. ARCO [5]] ABD [2). and Unimatch 53 Note that methods labeled with "F" denote fully supervised models trained using all labeled data, while those labeled with "S" indicate semi- sumissuper models trained with limited annotations. To ensure fair comparison, our VQ-Seg model adopts the same encoder and decoder architecture as Unimatch (53] The main difference lies in the introduction of quantization and alignment process after the encoder output. Segmentation performance is evaluated using four common metrics: Dice and Jaccard for region overlap, and HD95 and ASD for boundary accuracy.

---

Table 2:
A comprehensive ablation study evalu- ating the contributions of the Quantized Pertur- bation Module (QPM). the dual-branch architec- ture with shared Post-VQ space (DB), and the Post- VQ Feature Adapter (PFA) on segmentation
Table 3: Impact of hyperparameters metrics.
Param.
Value
Dicef
Jaccard
HD95.
ASD]
0.3
0.7741
0.6612
12.3821
4.2983
0.5
0.7803
0.6685
11.9042
4.2150
0.7
0.7852
11.6179
4,2094
0.9
0,7418
0.6142
14.8054
5.1328
performance using the I.C dataset (10% labeled).
0.7720
0.6591
11,5080
4.2672
0.7852
0.6731
11.6179
4.2094
Bose
QPM
DB
PEA
0.711
Dicct
Jaccardf
0.6258
142153
HDWS,
5.2301
ASD1
10
0.7765
0.6640
11,9235
4.1926
0.6559
1340266
1
0,7852
0.6731
11.6179
4.2094
0.7784
0.6620
1244288
4.6013
A
5
0.7670
0.6553
11.7421
4.1652
0.7761
0.6597
12.7381
4.7005
0.7843 10
0.6724
118852
4.1013
0.7852
0.6731
11,6179
4.2094
4.4 Comparison Results
Quantitative Comparisons. Tables [ and 7 summarize the quantitative comparisons for each method on the LC and ACDC datasets across two labeled ratio settings (5% and 10%). On the LC dataset, our method demonstrates superior performance. At 5% labeled data, VQ-Seg achieves the highest Dice (0.6643) and Jaccard (0.5257), outperforming the second-best Unimatch (Dice: 0.6493, Jaccard: 0,5071) by 1.5% and 1.86%, respectively. It also yields the best HD95 (12.2525) and ASD (4.2276), improving upon the second-best ABD (HD95: 12.5608) and MCNet (ASD: 4.9231) by 0.3083 and 0.6955. With 10% labeled data, our method maintains the lead, achieving the highest Dice (0.7852) and Jaccard (0.6731), surpassing the second-bess MCNet (Dice: 0.7555, Jaccard: 0.6414) by 2.97% and 3. 3.17%. It also yields the best HD95 (11 16.699 and ASD (4.20% (4.2094), improving upon the second-best UA-MT (HD95: 11.6724) and ARCO (ASD: 4. 4.3660) by 0.0545 and 0. 1566. Results on the ACDC dataset (see Appendix C and Table exhibit a similar trend, confirming the robustness and generalizability of our approach across diverse datasets.
Visual Comparisons. As depicted in Fig. 5 our VQ-Seg effectively identifies the cancerous areas with high precision. Segmentation outcomes exhibit improved consistency and clearer boundary delineation when compared other state-of-the oortte--rttttttt techniques. Notably, our method better preserves the structural integrity of cancer regions. These visual advantages further demonstrate the superior performance our model across various cases. For more comprehensive understanding, please refer to Appendix D for the statistical visualization analysis and Appendix for the t-SNE visualization of the codebook update dynamics.
4.5 Ablation Studies
Module Analysis. As shown in Table 2 we conduct step-by-step ablation evaluate the con- tribution of each proposed component. The baseline model is a VQ-embedded Unimatch with student-teacher framework. Based on this, incorporating the Quantized Perturbation Module (QPM) leads to a notable improvement in Dice from 0.7443 to 0.7701. Adding the dual-branch (DB) archi- tecture further enhances performance to 0.7784, while using the Post-VQ Feature Adapter (PFA) alone yields a Dice of 0. 0.7761. When all three modules are combined, the full model achieves the best performance across all metrics, including a Dice of 0.7852, Jaccard of 0.6731, HD95 of 1 11.6179, and ASD of 4.2094, confirming the complementary benefits of the proposed components.
Hyper-parameter Experiment. Table 3 reports the impact of key hyperparameters. For the perturbation strength c, performance improves as increases to 0.7, achieving the best Dice score of 0.7852. Regarding the loss weights, setting Xa 5 and X, leads to the best overall results.
Effect of Foundation Models. We choose DINOv2 as our backbone because it has demonstrated remarkable generalization across diverse downstream tasks [55] [5);; despite being trained on natural images, it transfers strongly to medical domains and has been widely used in medical segmentation and registration. To further validate this design choice, we conducted an ablation replacing DINOv2 with alternative foundation models, including those pretrained on medical data. CLIP [57] and BiomedCLIIP (58) are -asion-language models trained on large-scale image-text pairs (natural and medical, respectively), MAE [59 is trained via masked autoencoding, and Rad-DINO [60] adopts DINO-style architecture on radiology images. As summarized in Table DINOv2 consistently outperforms all alternatives under both 5% and 10% labeled regimes, including models specialized

---

Table 4: Ablation on foundation models.
Table 5: Ablation on the codebook size.
Foundation Model
Dice (5%
Dice (10%)
Codebook Size
Dice (5%)
(10%)
Uti. (%)
CLIP [57]
0.6421
0.7483
1024
0.6531
0.7608
100
BiomedCLIP 58
0,6507
0.7629
2048
0.6582
0.7748
100
0.7775
MAE 59
0.6386
0.7541
16384
0,6643
0.7852
98
Rad-DINO 60
0.6535
0.7793
32768
0,6595
0.7764
95
DINOv2
42
0.6643
0.7852
65536
0,6415
0.7638
92
Table 6: Comparison performance under different labeled data ratios.
Method
5%
10%
20%
50%
100%
UNet-S 47
0.4343
0.6490
0.7205
0.7880
0.8345
MCNet
48
0.6378
0.7555
0.7812
0.8203
0,8751
ABD [52]
0.6414
0.7468
0.7780
0.8235
0,8824
Unimatch S3
0.6493
0.7511
0.7855
0.8279
0.8871
VQ-Seg (Ours)
0.6643
0.7852
0.8100
0.8507
0.9102
for medical domains, empirically supporting DINOv2 as robust and effective semantic prior for semi-supervised medical image segmentation.
Effect of Codebook Size. We further investigate the impact of codebook size on model performance, as shown in Table [ The codebook size determines the granularity of the discrete latent space: smaller codebook provides limited representational capacity, while an excessively large one may lead code redundancy and unstable optimization. Our results show that the Dice score steadily increases as the codebook size grows from 1,024 to 16,384, indicating that moderately large codebook enables richer and more discriminative representations. However, further enlargement (e.g., 32,768 or 65,536) slightly degrades performance due to decreased code utilization and overfitting. Here, "Uti." denotes codebook utilization, which measures the proportion code vectors actively used during training.
Ablation on Labeled Ratio Settings. As shown in Table we evaluate VQ-Seg under different labeled ratios to examine its scalability in semi-supervised learning. The results show a consistent per- formance gain with more labeled data, demonstrating the model's s strong ability to exploit supervision. Remarkably, VQ-Seg achieves significant improvements in low-label regimes, indicating that the discrete representation learned via vector quantization provides effective regularization and semantic consistency. Even with increasing supervision, the performance gain remains stable, highlighting VQ-Seg's robustness and scalability across varying annotation levels.
5
Conclusion and Limitations
present VQ-Seg, novel semi-supervised medical image segmentation framework. VQ-Seg introduces Quantized Perturbation Module (QPM) that performs controlled perturbations in the vector-quantized (VQ) feature space, enhancing the robustness representation learning. addition, dual-branch architecture with Post-VQ Feature Adapter (PFA) is designed to refine the quantized features and integrate high-level semantic information. Extensive experiments on the Lung Cancer (LC) and ACDC datasets demonstrate that VQ-Seg achieves state-of-the-art performance, substantially improving segmentation accuracy under limited supervision.
However, the current perturbation operates solely in the discrete VQ space, making it difficult to extend to continuous feature representations commonly used in existing semi-supervised frameworks. Moreover, while the adoption of foundation model introduces richer semantic priors, it also brings additional computational overhead. Future work will focus on developing controllable perturbation mechanisms in continuous spaces and exploring more efficient foundation model integration.
Acknowledgments and Disclosure of Funding
This work is supported by the Guangdong Science and Technology Department (2024ZDZX2004) and the Guangzhou Industrial Information and Intelligent Key Laboratory Project (No. 2024A03J0628),
10

---

References
[1] Arnaud Arindra Adiyoso Setio, Alberto Traverso, Thomas De Bel, Moira SN Berens, Cas Van Den Bogaard, Piergiorgio Cerello, Hao Chen, Qi Dou, Maria Evelina Fantacci, Bram Geurts, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed lomography images: the lunal6 challenge. Medical image analysis, 42:1-13, 2017.
[2] Mandong Hu, Yi Zhong, Shuxuan Xie, Haibin Lv, and Zhihan Lv. Fuzzy system based medical image processing for brain disease prediction. Fronfiers in Neuroscience, 15:714318, 2021.
[3] Zhaohu Xing, Lequan Yu, Liang Wan, Tong Han, and Lei Zhu. Nestedformer: Nested modality- aware transformer for brain tumor segmentation. International conference on medical image computing and computer-assister intervention, pages 140-150. Springer, 2022.
[4] Dinesh D Patil and Sonal G Deore. Medical image segmentation: a review. International Journal of Computer Science and Mobile Computing, 2(1):22-27, 2013.
[5] Zhaohu Xing, Tian Ye, Yijun Yang. Guang Liu, and Lei Zhu. Segmamba: Long-range sequential modeling mamba for 3d medical image segmentation. International conference on medical image computing and computer-assiste intervention, pages 578-588. Springer, 2024.
[6] Sicheng Yang. Hongqiu Wang, Zhaohu Xing, Sixiang Chen, and Lei Zhu. Segdino: An efficient design for medical and natural image segmentation with dino-v3. arXiv preprint arXiv:2509.00833, 2025.
[7] Huiyan Jiang, Zhaoshuo Diao, Tianyu Shi, Yang Zhou, Feiyu Wang, Wenrui Hu, Xiaolin Zhu, Shijie Luo, Guoyu Tong, and Yu-Dong Yao. A review of deep learning- ceaning-baseed based multiple-lesion recognition from medical images: classification, detection and segmentation. Computers in Biology and Medicine, 157:106726, 2023.
[8] Hongqiu Wang, Xiangde Luo, Wu Chen, Qingqing Tang, Mei Xin, Qiong Wang, and Lei Zhu. Advancing uwf-slo vessel segmentation with source-free free active domain adaptation and novel multi- dataset. In International Conference on Medical Image Computing and Cosiuter-Assisted Intervention, pages 75-85. Springer, 2024.
[9] Hongqiu Wang, Yixian Chen, Wu Chen, Huihui Xu, Haoyu Zhao, Bin Sheng, Huazhu Fu, Guang Yang, and Lei Zhu. Serp-mamba: Advancing high-resolution retinal vessel segmentation with selective state-space model. IEEE Transactions Medical Imaging, 2025.
[10] Dzung L Pham, Chenyang Xu, and Jerry L Prince. Current methods in medical image segmen- tation. Annual review of biomedical engineering, 2(1):315-337, 2000.
[11] Hongqiu Wang, Guang Yang. Shichen Zhang, Jing Qin, Yike Guo, Bo Xu. Yueming Jin, and Lei Zhu. Video-instrument synergistic network for referring video instrument segmentation in robotic surgery. IEEE Transactions on Medical Imaging, 2024.
[12] Yijun Yang, Zhaohu Xing, Lequan Yu, Chunwang Huang, Huazhu Fu, and Lei Zhu. Vivim: A video vision mamba for medical video segmentation. arXiv preprint arXiv:2401. 14168, 2024.
[13] Muhammad Imran Razzak, Saeeda Naz, and Ahmad Zaib. Deep learning for medical image processing:
Overview, challenges and the future. Classification in BioApps: Automation of decision making, pages 323-350, 2017.
[14] Mohammad Hesam Hesamian, Wenjing Jia, Xiangjian He, and Paul Kennedy. Deep learning techniques for medical image segmentation: achievements and challenges. Journal of digital imaging, 32:582-596. 2019.
[15] Risheng Wang, Tao Lei, Ruixia Cui, Bingtao Zhang, Hongying Meng, and Asoke K Nandi. Medical image segmentation using deep learning: survey. IET image processing, 16(5): 1243-1267, 2022.

---

[16] Lequan Yu, Shujun Wang, Xiaomeng Li, Chi- Whigiing& Fu, and Pheng-Am Ann Heng. Uncertainty- aware self-ensembling model for semi-supervised 3d left atrium segmentation. In Medical image computing and computer assisted MMTerVention-MICCA/ 2019: 22nd international conference, Shenzhen, China, October 13-17, 2019, proceedings, part 22, pages 605-613. Springer, 2019.
[17] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1): (111)::999-99999 1958, 2014.
[18] Rushi Jiao, Yichi Zhang, Le Ding, Bingsen Xue, Jicong Zhang, Rong Cai, and Cheng Jin. Learning with limited annotations: a survey on deep semi-supervised learning for medical image segmentation. Computers in Biology and Medicine, 169:107840, 2024.
[19] Jun Ma, Yao Zhang, Song Gu, Cheng Ge, Shihao Mac, Adamo Young, Cheng Zhu, Xin Yang. Kangkang Meng, Ziyan Huang, et al. Unleashing the strengths of unlabelled data in deep learning-assisted pan-cancer abdominal organ quantification: the flare22 challenge. The Lancet Digital Health, 10xx)e88822666 2024.
[20] Veronika Cheplygina, Marleen De Bruijne, and Josien PW Pluim. Not-so-supervised: survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical image analysis, 54:280-296, 2019.
[21] Isaac Triguero, Salvador García, and Francisco Herrera. Self-labeled techniques for semi- supervised learning: taxonomy, software and empirical study. Knowledge and Information systems, 42:245-284, 2015.
[22] Bethany H Thompson, Gaetano Di Caterina, and Jeremy P Voisey. Pseudo-label refinement using superpixels for semi-supervised brain tumour segmentation. 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1-5, IEEE, 2022.
[23] Liang Qiu, Jierong Cheng, Huxin Gao, Wei Xiong, and Hongliang Ren. Federated semi- supervised learning for medical image segmentation via pseudo-label denoising. IEEE journal of biomedical and health informatics, 27(10)4662-4468, 2023.
[24] Moucheng Xu, Yukun Zhou, Chen Jin, Marius de Groot, Daniel C Alexander, Neil P Oxtoby, Yipeng Hu, and Joseph Jacob. Expectation maximisation pseudo labels. Medical Image Analysis, 94:103125, 2024.
[25] Wenji Wang, Qing Xia, Zhiqiang Hu, Zhennan Yan, Zhuowei Li, Yang Wu, Ning Huang. Yue Gao, Dimitris Metaxas, and Shaoting Zhang. Few-shot learning by a cascaded framework with shape-constrained pseudo label assessment for whole heart segmentation. IEEE Transactions on Medical Imaging, 40( 40(10):629-26411 2021.
[26] Kai Han, Victor S Sheng, Yuqing Song, Yi Liu, Chengjian Qiu, Siqi Ma, and Zhe Liu. Deep semi-supervised learning for medical image segmentation: A review. Expert Systems with Applications, 245: 1..012.322.000 2024.
[27] Cheng Chen, Kangneng Zhou, Zhiliang Wang, and Ruoxiu Xiao. Generative consistency for semi-supervised cerebrovascular segmentation from tof-mra. IEEE Transactions on Medical Imaging, 42(2))346-33) -53, 2022.
[28] Ling-Li Zeng, Kai Gao, Dewen Hu, Zhichao Feng, Chenping Hou, Pengfei Rong, and Wei Wang. Ss-tbn: A semi-supervised tri-branch network for covid-19 19 screening and lesion segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(8):10477-04442 2023.
[29] Liyun Lu, Mengxiao Yin, Liyao Fu, and Feng Yang. Uncertainty- pseudo-label and consistency for semi-supervised medical image segmentation. Biomedical Signal Processing and Control, 19:104203, 2023.
[30] Jianfeng Wang and Thomas Lukasiewicz. Rethinking bayesian deep learning methods for semi-supervised volumetric medical image segmentation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 182-190, 2022.
12

---

[31] Vikas Verma, Kenji Kawaguchi, Alex Lamb, Juho Kannala, Arno Solin, Yoshua Bengio, and David Lopez-Paz. Interpolation consistency training for cumi-sueee learning. Neural Networks, 145:90-106, 2022.
[32] Xinyue Huo, Lingxi Xie, Jianzhong He, Zijie Yang, Wengang Zhou, Houqiang Li, and Qi Tian. Atso: Asynchronous teacher-student optimization for semi-supervised image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1235-1244, 2021.
[33] Yixin Wang, Yao Zhang, Jiang Tian, Cheng Zhong, Zhongchao Shi, Yang Zhang, and Zhiqiang He. Double-uncertainty weighted method for semi-supervised learning. In Medical Image Computing and Computer Assisted InterventionMMICCAl 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part 23, pages 542-551. Springer, 2020.
[34] Hamed Rahimian and Sanjay Mehrotra. Frameworks and results in distributionally robust optimization. Open Journal of Mathematical Optimization, 3:1-85, 2022.
[35] Rui Gao and Anton Kleywegt. Distributionally robust stochastic optimization with wasserstein distance. Mathematics of Operations Research, 48(2):603-655, 2023.
[36] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
[37] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[38] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12873-12883 2021.
[39] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 37: 777.84839-84888 2024.
[40] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PmLR, 2020.
[41] Taesung Park, Alexei A Efros, Richard Zhang, and Jun-Yan Zhu. Contrastive learning for unpaired image-to-image translation. In Computer Vision-ECCV 2020: 16th Earopean Confer- ence, Glasgow, UK, August 23-28, 2020, Proceedings, Part IX 16, pages 319-345. Springer, 2020.
[42] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[43] Olivier Bernard, Alain Lalande, Clement Zotti, Frederick Cervenansky, Xin Yang, Pheng-Ann Heng, Irem Cetin, Karim Lekadir, Oscar Camara, Miguel Angel Gonzalez Ballester, al. Deep learning techniques for automatic mri cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE transactions on medical imaging, 37(11):2514-252 2018.
[44] Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 1308.4388. 2013,
[45] Yongxin Zhu, Bocheng Li, Yifei Xin, and Linli Xu. Addressing representation collapse in vector quantized models with one linear layer. arXiv preprini urXiv:2411.02038 2024.
[46] Lei Zhu, Fangyun Wei, Yanye Lu, and Dong Chen. Scaling the codebook size of vggan to 100,000 with a utilization rate of 99%. arXiv preprint arXiv:2406. 1837, 2024.
[47] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCAA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
13

---

[48] Yicheng Wu, Zongyuan Ge, Donghao Zhang, Minfeng Xu, Lei Zhang, Yong Xia, and Jianfei Cai. Mutual consistency learning for sume-supereised medical image segmentation. Medical Image Analysis, 81: 81:102530, 2022.
[49] Yicheng Wu, Zhonghua Wu, Qianyi Wu, Zongyuan Ge, and Jianfei Cai. Exploring smoothness and separatton for semi seni-supervised medical image segmentation. In International confer- ence medical image computing and computer-assian asssiste intervention, pages 34 44-4.. Springer, 2022.
[50] Yunhao Bai, Duowen Chen, Qingli Li, Wei Shen, and Yan Wang. Bidirectional copy-paste for semi-supervised medical image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11514-15224, 2023.
[51] Chenyu You, Weicheng Dai, Yifei Min, Fenglin Liu, David Clifton, S Kevin Zhou, Lawrence Staib, and James Duncan. Rethinking semi-supervised medical image segmentation: variance- reduction perspective. Advances in neural information processing systems, 36:9984-1000 10021, 2023.
[52] Hanyang Chi, Jian Pang, Bingfeng Zhang, and Weifeng Liu. Adaptive bidirectional displacement for semi-supervised medical image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1070-4080, 2024.
[53] Lihe Yang, Zhen Zhao, and Hengshuang Zhao. Unimatch v2: Pushing the limit of semi- supervised semantic segmentation. IEEE Transactions Pattern Analysis and Machine Intelligence, 2025.
[54] Fabian Isensee, Paul F Jaeger, Simon AA Kohl, Jens Petersen, and Klaus H Maier-Hein. nnu-net: self-configuring method for deep learning- based biomedical image segmentation. Nature methods, 18(2):203-2) 2021.
[55] Mohammed Baharoon, Waseem Qureshi, Jiahong Ouyang, Yanwu Xu, Abdulrhman Aljouie, and Wei Peng. Evaluating general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks. arXiv preprint arXiv:2312.02366. 2023.
[56] Xinrui Song, Xuanang Xu, and Pingkun Yan. Dino-reg: General purpose image encoder for training-free multi-modal deformable medical image registration. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 608-617. Springer, 2024.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PmLR, 2021.
[58] Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedelip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint urXiv:2303.00915 2023.
[59] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEFF/CVE conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[60] Fernando Perez-Garcia, Harshita Sharma, Sam Bond-Taylor, Kenza Bouzid, Valentina Salvatelli, Maximilian Ilse, Shruthi Bannur, Daniel C Castro, Anton Schwaighofer. Matthew P Lungren, et al. Exploring scalable medical image encoders beyond text supervision. Nature Machine Intelligence, 7(1):119-130) 2025.
14

---

A KL Divergence Approximation under Dropout Perturbation
To support the theoretical analysis in Section 3.2 derive approximation of the KI. divergence between a prior distribution P(h) and perturbed distribution ((h) induced by applying dropout to intermediate feature. This divergence used to quantify the perturbation radius caused by dropout in the feature space.
A.1 Dropout-Indueed Feature Distribution as a Mixture
Consider simplified setting where feature activation h follows Gaussian prior:
P(h) (0,
(13)
Under dropout with rate p. the feature zeroed out with probability p. or retained with probability 1 - p. This results in the perturbed distribution:
((h) (1 - p) -N(0, +p-
(14)
where 8(h) is the Dirac delta function centered zero. This mixture captures the effect of randomly dropping activations.
A.2 Intractability of Exact KL Divergence
The KL divergence between P(h) and Q(h) is:
DkI. = [
P(h)) Q(h)
P(h) log
dh.
(15)
Substituting the forms of P(h) and (h), we obtain:
W(h (1-p) o3) + pō(h)
DkL(P)lQ) = (h; 0, oه)) log
dh.
(16)
Due to the singular nature of 5(h) at h 0, this expression is intractable in closed form. To make progress, we adopt a moment- moment-matching approximation.
A.3 Moment-Matching Approximation
We approximate Q(h) with Gaussian distribution Qupprox (h) that matches the first and second moments of the dropout-perturhed activations. Let h ~ Qhh) denote the post-dropout feature. Then:
EThh) = 0, Var(h') 0/ p).
(17)
Therefore, define:
Qupprox (h) (0, of p))- The KL divergence between the original and approximated feature distributions becomes:
(18)
1 oj
1 log (0B(1-p) pp)
:o(1 oj
DKL. (PllQ
(19)
2
[g p)
Simplifying gives:
( 1 (PIlQQQa 2 -
log(1 -p) p)
1 p ( + log(1 - -p))
(20)
2
1-
A.4
Interpretation and Connection Perturbation Radius
This result provides a clean, analytical expression for the perturbation radius induced by dropout, interpreted KL divergence. Notably, as p-
+ 1, the divergence grows rapidly and even diverges (.e., becomes unbounded), indicating severe deviation from the original distribution. This supports our theoretical claim: Dropout induces increasingly unstable perturbations when the dropout rate is high, as observed empirically (see Fig. D potentially leading to over-regularization, distorted representations, and degraded model performance. This motivates our Quantized Perturbation Module (QPM), which constrains perturbations to structured, discrete space and avoids such instability.
15

---

B Proof of Numerical Stability of QPM
In this appendix, we prove that the perturbed distribution Q(c, ct) used in the Quantized Perturbation Module (QPM) is always well-defined and bounded, thereby ensuring the numerical stability of the associated KL divergence, even in the extreme case of
B.1 Definition of Q(cy
Let the prior distribution over codewords uniform:
1 P(c)
(1. KJ.
(21)
K'
The perturbation mechanism π( i) is defined as:
1-c, cexp(-d(c,.))
jf = if *
( 1
where Zi = Σ expp(-d(cs,cx))
(22)
k
Then, the overall perturbed distribution is:
K
Qie (6) - Σ P(a)T(i] 11)
Σ
(23)
K
1=1
t=1
B.2 Basic Properties of Q(ej (c)
(1) Non-negativity and Positivity: Since x(j i) 2 0 for all i, follows that Q(e, (6) 2
Furthermore, under mild assumptions (e.g., finite distances and 0), we have x( i) > for some i, so ((e, c) > for all j.
(2) Normalization: We verify that Q is valid probability distribution:
K Σ
K K 1 T
K K
K Σ 1
ΣΣ E=
=
1=1.
(24)
K
K ==1
B.3 KL Divergence Between P Pand Q
The KL divergence between P and Q is defined as:
DKL (Q) = Σ P(c,) log ( P(e))
K
K Σ
1/K
log
(25)
Q(G
K
Q(e;
(e)
1=1
j=1
log (KQ(c) e)).
(26)
This expression numerically stable as long as Q(e, c) > 0 and bounded away from 0, which we prove next for the extreme case
1.
B.4
QPM Perturbation Distribution at =
Recall that when 1, the transition distribution becomes:
0,
if =
イ ()) = expl d(c..);))
where Z;
Σ ck)).
(27)
if i
kif
Thus, the resulting perturbed distribution is given by:
K
exp( exp(-d(cn,c)))
Q(c E= 1) Σ
P(e)T
(28)
K
kiti exp( (uspp-dlcii (4))
16

---

B.5 Lower Bound of Q(c;) ==
Assume the codebook C {ci, CK } fixed and that the pairwise distance is bounded: < Dmin I d(c,,c)) I Dmax 00 for all i + j. Then for any + j. the transition term is lower bounded:
exp(-d(ct,G)
exp( sep(--Dmxx -max)
exp(Dmin D Dmax)
(29)
Expeexpl exp(- -d((,,
(K 1) exp( Dmin)
K
Hence,
Σ exp(-d(a,,ci)
Q(o E= 1)
(30)
Σ kel exp( -d((c,
inf
cxp( Sap(Duin min Dinax)
exp(Dmin D Dmax)
[K 1)
K 0.
(31)
K
K-1
B.6 Upper Bound of Q(G₂ e = 1)
Likewise, for any + j:
exp(-d(c,,c))
exp(-1) -min))
exp(Dman max Dmmin)
(32)
Cxstexpl d(cs,
(K exp(- (-Dmax)
K-1
Thus,
exp(Dmax Mmax - -Dmin) exp(Dmax D max - Dmin)
Olglc=1)5 K -K-
(33)
K-1
K
B.7
Implication for KI. Divergence
Since Q(e, c 1) is strictly positive and bounded above by 1, the term divergence remains finite:
in the KL
K log KQ(c)) < oo.
DKL (1I = K
1
Σ
(34)
J=1
Therefore, the QPM perturbation strategy ensures numerical stability for all valid E [0, 1].
C Performance Analysis on the ACDC Dataset
Table 7: Quantitative comparison on the ACDC dataset with two labeled ratio settings (5%, 10%)
using Dice and Jaccard (t). Best results are in bold.
5 5% Labeled
10 % Labeled
Method
Dicef
Jaccard1
Dicef
Jaccard
UNet-F
47
0.9130
0.8427
0.9130
0.8427
UNet-S
0.4674
0.3698
0.7952
0.6882
UN
54
0.9185
0.8491
0.9185
0.8491
UNet-
54
0.4892
0.3876
0.8113
0.7040
UA-MT
16
0.6123
0.5324
0.8423
0,7382
MCNet
48
0.6485
0.5338
0.8621
0.7701
SSNet
0.6542
0,5568
0.8689
0.7753
BCP S0
0.8621
0.7846
0.8827
0.8032
ARCO ST
0,8879
0.8021
0.9026
0,8252
ABD
52
0.8874
0.7924
0.8992
0,8213
Unimatch [53]
0.8915
0.7983
0.8978
0.8290
Ours
0.9057
0.8173
0,9103
0.8327
Table 7 compares our method with leading baselines on the ACDC dataset under 5% and 10% labeled data settings. Our model consistently outperforms others in both Dice and Jaccard metrics.
17

---

With 5% labeled data, our method achieves Dice of 0.9057, exceeding Unimatch (0.8915), ABD (0.8874), and ARCO (0.8879), and approaching the fully supervised nnUNet-F (0.9185). This highlights its strong representation ability under limited supervision.
At 10% labeled data, the advantage becomes Jaccard (0.8327), surpassing ABD (0.8992) and ARCO (0. (0.9026). These consistent gains demonstrate the robustness and scalability of our approach more labeled data available.
evident, reaching the best Dice (0.9103) and
D Statistical Analysis
0.665
Dice Score
0.655
0.650
0.645
0.649
0.630
Figure 6: Comparison under 5% labeled LC dataset.
To evaluate the statistical reliability of method, Unimatch [53], and our VQ-Seg ten times under the 5% labeled LC dataset using different random seeds. The averaged Dice scores together with their standard deviations visualized in Fig. 6 where the error bars denote the standard deviation across repeated runs, reflecting the robustness and stability of each method. The noticeably smaller error bar of VQ-Seg indicates lower performance variance, demonstrating its stronger training stability across random seeds.
the competing models MCNet [48].
To confirm the statistical significance of the observed improvements, we performed paired two-tailed I-tests between VQ-Seg and each baseline method over the ten independent trials. The results show that the differences statistically significant with p 0.05.
E
Codebook Evolution
IraciNG
frective
rocthe Acietted
Uti =3.4%
Ut. =07%
Uti. 99%%
Figure 7: T-SNE visualization of codebook evolution.
We visualize in Fig. the t-SNE projections
codebook vectors across three training stages. Each point represents codeword, where orange and blue indicate activated and inactive entries, respectively. Initially (left), only a few codewords (3.4%) are activated, showing a compact cluster and limited diversity. As training proceeds (middle), activation increases 67%, and the distribution becomes more uniform. By convergence (right), nearly all codewords (98%) are active and evenly dispersed, forming stable and well-structured embedding space.
These results demonstrate that our quantization strategy progressively enhances codebook utilization and representation diversity.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, we clearly outline five main contributions: (1) the collection of new large-scale Lung Cancer (I.C) dataset; (2) the proposal of the Quantized Perturbation Module (QPM), which replaces dropout for structured feature perturbation; (3) the design of a dual-branch architecture that jointly optimizes segmentation and image reconstruction in the post- space; (4) the development of the Post- VQ Feature Adapter (PFA) to align quantized features with semantic priors from a foundation model; and (5) extensive experiments demonstrating state-of-the-art performance on both the LC dataset and public benchmarks.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
Justification:
We discuss the limitations of the work in Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
19

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Theoretical derivations, including KL divergence approximation under dropout and numerical stability of QPM, are provided in Appendix and with detailed assumptions and mathematical proofs supporting the proposed method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We disclose all implementation details in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
also be provided via detailed
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility.
20

---

In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification:
We will provide code and the data downloading link.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (nttps://niis.cc)
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 //nips. /twpps.ccppublli/// httpcc/pccccccceeeeeeeeeeeeeeeee gwtcc//guieesee/eee SaweeeeeeoeeSeeeeeiieiii..
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We disclose all implementation details in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification:
We provide detailed statistical analysis in Appendix D
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In Section 4.2 provide the information about the computer resources use.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't itdn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttpp///eeuriss.ccppplcl..cc.e ctcuiiieeeeenn
Answer: [Yes]
Justification:
The research does not raise ethical concerns related to privacy, bias, misuse.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction)
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive societal impacts in Section 5 As research work in the field of medical image segmentation, we believe this paper will also not have negative impacts society.
Guidelines:
22

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have cited all the models used in the paper and stated their version.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The paper does not employ large language models (LLMs) in any capacity.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /tapp/:////urps..... /0tp:///eccccooeeeeee/////////// for what should or should be described.
25


Reviews:
[{"review_id": "XSFJJwxGgm", "reviewer": "Reviewer_4i9r", "summary": "This paper presents VQ-seg, a novel semi-supervised medical image segmentation framework. It uses Quantized Perturbation Module (QPM) for robust representations in VQ feature space and a dual-branch architecture with Post-VQ Feature Adapter (PFA) for high-level semantic integration. Experiments on LC and ACDC datasets show the performance compared to previous work.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe proposed dataset large-scale Lung Cancer (LC) dataset is valuable.\n\n2.\tThe paper provides theoretical analysis which is good.\n\n3.\tThe experimental results show the advantages of the proposed method compared to previous work.\n\nWeaknesses:\n\n1.\tTable 1 only shows the results on the LC dataset with two labeled ratio settings 5% and 10%. More labeled ratio settings are suggested such that it can show the trend of the performance with increasing ratio of labeled data.\n\n2.\tIn Sec. 5, it mentions that “However, the effectiveness of VQ-seg depends on the quality and relevance of the foundation model, which may introduce bias if not carefully selected and validated.” Can the authors provide more details about this limitation? What is the impact of different choices of foundation models? Please provide more discussion about the limitations. \n\n3.\tIt claims that “QPM enables a more structured and controllable mechanism for representation perturbation by shuffling the spatial locations of codebook indices, offering enhanced interpretability and stability compared to traditional dropout.” However, there is no evidence, such as the visualization of the perturbation generated by QPM versus traditional dropout. Please add visualization results.", "was_revised": true, "final_justification": "I appreciate the efforts by the authors for rebuttal. The rebuttal partially addresses my concerns, but there is still room for this paper to improve. After reading other reviews and authors' response, I choose to keep my original rating.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly. \n***\n> [W1] Limited labeled ratio settings.\n\n[A1] We agree that incorporating additional labeled ratio settings can better illustrate the performance trend under varying levels of supervision. In response, we have extended our experiments on the LC dataset to include additional labeled ratios of 20%, 50%, and 100%. The updated results, evaluated using the Dice coefficient, are summarized below:\n\n| Method         | 5%     | 10%    | 20%    | 50%    | 100%   |\n|----------------|--------|--------|--------|--------|--------|\n| UNet-S    | 0.4343 | 0.6490 | 0.7205 | 0.7880 | 0.8345 |\n| MCNet          | 0.6378 | 0.7555 | 0.7812 | 0.8203 | 0.8751 |\n| ABD            | 0.6414 | 0.7468 | 0.7780 | 0.8235 | 0.8824 |\n| Unimatch       | 0.6493 | 0.7511 | 0.7855 | 0.8279 | 0.8871 |\n| VQ-Seg (Ours) | **0.6643** | **0.7852** | **0.8100** | **0.8507** | **0.9102** |\n\nAs shown above, VQ-Seg consistently outperforms all baselines across different supervision levels. Notably, with only 50% labeled data, our method already surpasses the fully supervised UNet baseline (0.8345). The consistent improvements as more annotations are available further highlight the scalability and effectiveness of our approach.\n\n\n> [W2] Lack of discussion on the impact of different foundation models.\n\n[A2] Thank you for the helpful comment. We provide further clarification below.\n\nWe assess the impact of different foundation models by replacing DINOv2 in VQ-Seg and reporting Dice scores under the 5% and 10% labeled settings:\n\n| Foundation Model  | Dice (5% Labeled) | Dice (10% Labeled) |\n| ----------------- | ----------------- | ------------------ |\n| CLIP              | 0.6421            | 0.7483             |\n| BiomedCLIP        | 0.6507            | 0.7629             |\n| MAE               | 0.6386            | 0.7541             |\n| rad-DINO          | 0.6535            | 0.7793             |\n| DINOv2 (Ours) | **0.6643**        | **0.7852**         |\n\nCLIP [1] and BiomedCLIP [2] are vision-language models pre-trained on large-scale natural and medical image–text pairs, respectively. MAE [3] is a vision model pre-trained via masked image modeling, while rad-DINO [4] employs a DINO-style architecture trained on radiology-specific datasets.\n\nAs shown above, DINOv2 consistently achieves the best performance across both supervision levels. It surpasses CLIP and MAE by more than 2% in absolute Dice score and outperforms BiomedCLIP and rad-DINO by a non-trivial margin.\n\nThese results reveal that foundation models pretrained with self-supervised, vision-only objectives (e.g., DINOv2, rad-DINO) produce more structure-aware representations and outperform vision-language models (e.g., CLIP, BiomedCLIP) on segmentation tasks. Among all candidates, DINOv2, which is used in our method, achieves the highest overall performance.\n\n> [W3] Lack of evidence supporting the benefits of QPM over dropout. \n\n[A3] Thank you for the insightful comment. Due to NeurIPS rebuttal policy, we are unable to modify the main paper or include additional figures at this stage. However, we would like to clarify that the perturbations introduced by QPM are anatomically meaningful. For example, in lung cancer segmentation, QPM-perturbed predictions remain focused on cancer-related regions. In contrast, dropout often introduces noisy outputs, including false positives in the background or mis-segmentation of adjacent structures such as bronchi or the chest wall. This behavior reflects the controlled nature of QPM, which samples from semantically similar codewords and thereby preserves anatomical semantics in the predictions. Although visualizations cannot be included at this stage, we will provide detailed qualitative and quantitative comparisons in the final version to support this claim.\n\n***\n> References:\n\n[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PmLR, 2021: 8748-8763.\n\n[2] Zhang S, Xu Y, Usuyama N, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs[J]. arXiv preprint arXiv:2303.00915, 2023.\n\n[3] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.\n\n[4] Perez-Garcia F, Sharma H, Bond-Taylor S, et al. Exploring scalable medical image encoders beyond text supervision[J]. Nature Machine Intelligence, 2025, 7(1): 119-130.\n\n***\nWe have incorporated additional results and detailed analyses addressing your concerns into the revised version of the manuscript. We hope these supplements adequately address your questions and improve the clarity and robustness of our work. Please do not hesitate to let us know if you require further discussion, additional experiments, or any clarifications—we are happy to provide more details to address your concerns comprehensively."}, {"review_id": "GZaQlU0VYt", "reviewer": "Reviewer_snt1", "summary": "This paper introduces VQ-Seg, a novel semi-supervised framework for medical image segmentation that leverages vector quantization (VQ) to address the limitations of traditional dropout-based consistency learning approaches. The authors propose three key innovations: (1) a Quantized Perturbation Module (QPM) that replaces dropout by shuffling codebook indices in the discrete VQ space, providing more stable and controllable perturbations; (2) a dual-branch architecture that jointly optimizes image reconstruction and segmentation tasks using shared post-VQ features to mitigate information loss from quantization; and (3) a Post-VQ Feature Adapter (PFA) that aligns quantized features with semantic embeddings from a foundation model to preserve high-level semantic information. The method is evaluated on a newly collected Lung Cancer (LC) dataset comprising 828 CT scans and the public ACDC dataset, demonstrating state-of-the-art performance across multiple metrics.", "questions": "1. How does the method prevent codebook collapse where only a subset of codewords are utilized?\n2. What is the actual utilization rate of the 16384 codewords, and how does underutilization affect the perturbation space?\n3. How sensitive is performance to the choice of foundation model？\n4. How can you verify that shuffling spatial locations of codebook indices produces semantically valid perturbations for medical images, where spatial context is crucial for anatomical structures?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The replacement for dropout-based perturbation through vector quantization is well motivated, and supported by theoretical analysis showing that QPM offers bounded and numerically stable perturbations compared to the potentially unbounded KL divergence of high dropout rates.\n2. The collection and annotation of 828 CT scans represents a significant contribution to the medical imaging community.\n\nWeaknesses:\n1. The relience on pre-trained foundation models for semantic guidance lacks novelty and the authors do not analyze the impact of different foundation models. \n2. The experimental results lack error bars or statistical significance tests.\n3. The paper sets codebook size following prior work, without exploring how this parameter affects the performance.\n4. The choice of spatial shuffling over other perturbation strategies in the discrete space, e.g., nearest neighbor substitution, probabilistic replacement, lacks empirical justification.\n5. The quality of the learned codebook is not analyzed. And how the learned codebook structure influences perturbation effectiveness is not explored.\n6. The ablation study on the individual components should be provided beyond the incremental contributions.", "was_revised": true, "final_justification": "Considering that most of my concerns have been addressed, I will raise my score accordingly.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly. \n***\n> [W1] Limited novelty in using foundation models; lack of comparison.\n\n[A1] Rather than simply using a foundation model, we introduce a novel strategy that leverages it as a semantic anchor to calibrate quantized representations for semi-supervised medical image segmentation. The Post-VQ Feature Adapter (PFA) aligns discrete VQ features with high-level semantics, effectively mitigating quantization-induced degradation. As shown in the ablation study (Table 2), removing the PFA results in a performance drop: Dice decreases from 0.7852 to 0.7784, and Jaccard from 0.6731 to 0.6620.\n\nTo evaluate the impact of different foundation models, we replace DINOv2 in the PFA module with several representative alternatives and report Dice scores under 5% and 10% labeled settings:\n\n| Foundation Model  | Dice (5% Labeled) | Dice (10% Labeled) |\n| ----------------- | ----------------- | ------------------ |\n| CLIP              | 0.6421            | 0.7483             |\n| BiomedCLIP        | 0.6507            | 0.7629             |\n| MAE               | 0.6386            | 0.7541             |\n| rad-DINO          | 0.6535            | 0.7793             |\n| DINOv2 (Ours) | **0.6643**        | **0.7852**         |\n\nCLIP \\[1] and BiomedCLIP \\[2] are vision-language models pre-trained on large-scale natural and medical image–text pairs, respectively. MAE \\[3] is a vision model trained via masked image modeling. rad-DINO \\[4] adopts a DINO-like architecture trained on radiology-specific datasets. As shown above, DINOv2 achieves the best performance across both settings, confirming its strong semantic capacity as the backbone for our PFA module.\n\n> [W2] No error bars or significance testing in experiments.\n\n[A2] Thank you for the constructive suggestion. To assess statistical robustness, we ran both Unimatch and VQ-SEG 10 times under the 5% labeled setting using different random seeds. The results are summarized below:\n\n| Method        | Dice (mean ± std) |\n| ------------- | ----------------- |\n| Unimatch      | 0.6491 ± 0.0124   |\n| VQ-SEG (Ours) | **0.6644 ± 0.0107**   |\n\nA paired t-test (p = 0.0013) and a Wilcoxon signed-rank test (p = 0.0021) confirm that the performance gain of VQ-SEG over Unimatch is statistically significant.\n\n\n> [W3] Codebook size not studied.\n\n[A3] Thank you for the helpful comment regarding codebook size.\nWe conducted an ablation study to examine how the codebook size $K$ affects segmentation performance under 5% and 10% labeled settings:\n\n| Codebook Size | Dice (5%)  | Dice (10%) | Codebook Utilization (%) |\n| ------------- | ---------- | ---------- | --------------- |\n| 1024          | 0.6531     | 0.7608     | 100%            |\n| 2048          | 0.6582     | 0.7748     | 100%            |\n| 4096          | 0.6627     | 0.7775     | 99%             |\n| 16384         | **0.6643** | **0.7852** | 98%             |\n| 32768         | 0.6595     | 0.7764     | 95%             |\n| 65536         | 0.6415     | 0.7638     | 92%             |\n\nThe results show that segmentation performance improves steadily as the codebook size increases, peaking at $K = 16384$, where Dice scores are highest and utilization remains high. Beyond this point, performance degrades and codebook utilization drops. This confirms that a codebook size of 16384 offers the best trade-off between representational capacity and generalization. \n\n> [W4] QPM lacks empirical justification vs. other strategies.\n\n[A4] Thank you for the insightful comment. Our Quantized Perturbation Module (QPM) adopts a probabilistic replacement strategy based on pairwise codeword distances (Eq. 4), rather than simple spatial shuffling.\n\nTo justify this design, we compare QPM with two deterministic baselines under both 5% and 10% labeled settings:\n\n* Nearest Neighbor Replacement: each codeword is replaced with its closest neighbor.\n* Farthest Neighbor Replacement: each codeword is replaced with the most distant one.\n\n| Strategy          | Dice (5%)  | Dice (10%) |\n| ----------------- | ---------- | ---------- |\n| Nearest Neighbor  | 0.6591     | 0.7662     |\n| Farthest Neighbor | 0.6112     | 0.6931     |\n| QPM (proposed)    | **0.6643** | **0.7852** |\n\nQPM consistently achieves better performance than both baselines across different labeled settings. \n\n> [W5] No analysis of codebook quality or its impact on perturbation.\n\n[A5] Thank you for your comment. The quality of the learned codebook is analyzed in **[A3]**, where we examine how codebook size affects both segmentation performance and codeword utilization. We find that performance peaks when the codebook is sufficiently expressive and efficiently utilized (98% utilization at $K = 16384$). In contrast, excessively large codebooks suffer from underutilization and slightly reduced performance, while overly small codebooks, though fully utilized, lack diversity and representational capacity, leading to suboptimal results.\n\nThis level of utilization is critical for QPM: a well-utilized codebook provides dense coverage of the latent space, ensuring that each replacement remains within a semantically meaningful neighborhood. Conversely, a sparse or underutilized codebook results in unstable and potentially harmful perturbations.\n\n> [W6] Missing ablation beyond incremental contributions.\n\n[A6] Thank you for your attention to the ablation analysis.  We would like to clarify that an ablation study on the individual components of our method is already included in the main paper (Table 2, Section 4.5). This table presents a step-by-step evaluation of the Quantized Perturbation Module (QPM), the dual-branch architecture (DB), and the Post-VQ Feature Adapter (PFA).\n\nThe results show that each component contributes clear performance gains over the baseline, and their combination achieves the best overall performance. These findings confirm the effectiveness and complementarity of the proposed modules beyond mere incremental contributions. We hope this addresses the reviewer’s concern.\n\n> [Q1] Prevention of codebook collapse.\n\n[A7]  To clarify how our method prevents codebook collapse, we include an additional ablation study demonstrating that each component contributes to improved codebook utilization. The codebook size is set to 16,384.\n\n| Setting                | Dice    | Codebook Utilization (%) |\n|------------------------|---------|-----------------|\n| Base                   | 0.7443  | 11.6%           |\n| + QPM                  | 0.7701  | 87.2%           |\n| + QPM + DB             | 0.7784  | 90.1%           |\n| + QPM + DB + PFA       | **0.7852**  | **98.0%**       |\n\nEach component plays a distinct role in enhancing utilization. QPM introduces structured perturbations that activate a broader set of codewords. DB promotes representational diversity through joint reconstruction supervision. PFA encourages consistent and semantically meaningful codeword activation by aligning with high-level features. Together, these modules effectively mitigate codebook collapse.\n\n> [Q2] Utilization rate of 16384 codewords and its impact.\n\n[A8] Please refer to **[A3]** for detailed analysis. A codebook size of 16,384 provides the best trade-off between performance and utilization (98%).\n\n> [Q3] Sensitivity to foundation model choice.\n\n[A9] Please refer to **[A1]** for detailed analysis on the impact of different foundation models. DINOv2 consistently outperforms other alternatives, confirming its effectiveness.\n\n> [Q4] Validity of spatial shuffling in medical images.\n\n[A10] We appreciate the reviewer’s concern regarding the validity of spatial shuffling in medical images. Recent studies \\[5] have shown that distances between codewords in the VQ codebook reflect semantic similarity. We observed a similar property in our experiments. Perturbations introduced by the QPM, which rely on controlled sampling of nearby codewords, help preserve anatomical semantics in the predicted segmentation. For instance, in lung cancer segmentation, QPM-perturbed predictions remain focused on cancer-relevant regions, maintaining their location and shape. In contrast, dropout often produces noisy outputs, including false positives in the background or mis-segmentation of surrounding structures such as the bronchi or chest wall. Although visualizations cannot be included at this stage due to NeurIPS policy, we will provide detailed qualitative and quantitative comparisons in the final version to substantiate this claim.\n\n***\n\n> References:\n\n[1] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PmLR, 2021: 8748-8763.\n\n[2] Zhang S, Xu Y, Usuyama N, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs[J]. arXiv preprint arXiv:2303.00915, 2023.\n\n[3] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.\n\n[4] Perez-Garcia F, Sharma H, Bond-Taylor S, et al. Exploring scalable medical image encoders beyond text supervision[J]. Nature Machine Intelligence, 2025, 7(1): 119-130.\n\n[5] Hu T, Zhang J, Yi R, et al. Improving autoregressive visual generation with cluster-oriented token prediction[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 9351-9360.\n\n***\n\nAdditional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details."}, {"review_id": "3pLrjv05eG", "reviewer": "Reviewer_jSKd", "summary": "This paper introduces VQ-seg, a semi-supervised learning framework for medical image segmentation. It replaces conventional dropout-based consistency regularization with a more stable mechanism based on vector quantization (VQ). The motivation is that dropout is sensitive to its rate and can cause either negligible effect or severe model collapse if not carefully tuned. One key contribution is Quantized Perturbation Module (QPM), which leverages distances between codebooks to define perturbations. The paper also proposes \nconcurrently reconstructing the image and performing segmentation tasks using a dual-branch architecture. To address the loss of high-level semantic features, the authors introduce a module that aligns quantized features with semantic features from a pre-trained foundation model. This paper also contributes a new multi-center CT dataset of 828 scans with annotations for central-type lung carcinoma.", "questions": "Please see weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The proposed framework is well-motivated, addressing known limitations of dropout-based perturbation in semi-supervised segmentation. The authors offer a rigorous theoretical analysis contrasting the instability of dropout with the stability of their Quantized Perturbation Module (QPM).\n2. The ablation study confirms the effect of QPM, the dual-branch architecture, and the foundation model guidance, demonstrating that each component provides improvements.\n\n\nWeaknesses:\n\n1. The experiments are still quite limited in scale. While the proposed LC dataset has a decent sample size, overall the authors only conducted studies on 2 datasets. The ACDC dataset only has 100 samples. It is questionable whether the proposed method can generalize to larger scale datasets such as FLARE 22, which also tackles the semi-supervised segmentation.\n2. The experiments focus on 5% and 10% labeled data. However, existing works of semi-supervised learning often includes a broader range of label ratios (e.g., 20%, 50%, and 100%) to demonstrate both the lower and upper bounds of the method. \n3. There is no evaluation of domain shift or external generalization. The authors should provide an analysis of the segmentation performance on external datasets. \n4. The reliance on DINOv2 as the foundation model raises concerns regarding domain alignment of the proposed method. Since DINOv2 is pretrained on natural image domain, why did the authors choose this model as the semantic teacher for the segmentation model? If the foundation model is poorly matched to the medical domain, the guidance may introduce bias. The authors acknowledge this in their limitations, but empirical evaluation with medical-specific foundation models is missing.", "was_revised": true, "final_justification": "The author's rebuttal addressed my concerns, so I am willing to increase my score.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly.\n***\n\n> [W1] The experiments are limited to small-scale datasets.\n\n[A1] To further demonstrate the scalability of our approach, we conducted additional experiments on the large-scale FLARE22 benchmark. We compared our method, VQ-Seg, with a supervised UNet baseline (UNet-S) and the state-of-the-art semi-supervised method Unimatch, using average Dice score as the evaluation metric.\n\n| Method           | Average Dice   |\n|------------------|--------|\n| UNet-S           | 0.818  |\n| Unimatch         | 0.917  |\n| VQ-Seg (Ours)    | **0.923** |\n\nOur method achieves the highest average Dice score, demonstrating strong generalization and effectiveness on large-scale datasets. We will include these results and cite the FLARE22 dataset and its related publication in the final version of the paper.\n\n> [W2] More labeled data ratios should be evaluated.\n\n[A2] Thank you for the suggestion. We fully agree that evaluating broader label ratios offers a more comprehensive view of the model’s scalability. Accordingly, we have extended our experiments on the LC dataset to include 20%, 50%, and 100% labeled settings. The updated Dice scores are presented below:\n\n| Method           | 5%     | 10%    | 20%    | 50%    | 100%   |\n|------------------|--------|--------|--------|--------|--------|\n| UNet-S           | 0.4343 | 0.6490 | 0.7205 | 0.7880 | 0.8345 |\n| MCNet            | 0.6378 | 0.7555 | 0.7812 | 0.8203 | 0.8751 |\n| ABD              | 0.6414 | 0.7468 | 0.7780 | 0.8235 | 0.8824 |\n| Unimatch         | 0.6493 | 0.7511 | 0.7855 | 0.8279 | 0.8871 |\n| VQ-Seg (Ours)| **0.6643** | **0.7852** | **0.8100** | **0.8507** | **0.9102** |\n\nVQ-Seg consistently outperforms all semi-supervised baselines across all settings, and even surpasses the fully supervised UNet using only 50% of the labeled data. These results demonstrate the strong scalability and efficiency of our method.\n\n> [W3] No external dataset is used to evaluate domain generalization.\n\n[A3] To evaluate external generalization, we conducted cross-domain experiments using the publicly available LUNA16 dataset [1], which also focuses on lung cancer segmentation. We compare our method and baselines trained on LC (5% labeled) and tested on LUNA16, along with a UNet trained directly on LUNA16 as an in-domain upper bound:\n\n| Method               | Training Data         | Testing Data | Dice↑  |\n|----------------------|------------------------|---------------|--------|\n| UNet (Full Sup.)     | LUNA16 (100% labeled)  | LUNA16        | 0.937  |\n| UNet (Full Sup.)     | LC (100% labeled)      | LUNA16        | 0.845  |\n| ABD (Semi-Sup.)      | LC (5% labeled)        | LUNA16        | 0.906  |\n| Unimatch (Semi-Sup.) | LC (5% labeled)        | LUNA16        | 0.913  |\n| VQ-Seg (Ours)    | LC (5% labeled)        | LUNA16        | **0.928** |\n\nOur method achieves the best generalization performance among all semi-supervised models and closely approaches the in-domain upper bound, demonstrating strong robustness to domain shift.\n\n> [W4] Concern over using DINOv2.\n\n[A4] We appreciate the reviewer’s insightful concern regarding the potential domain mismatch between DINOv2 and medical imaging tasks. While it is true that DINOv2 is pretrained on natural images, we selected it as our semantic teacher for several reasons:\n\n(1) Strong Generalization and Empirical Success in Medical Imaging:\nDINOv2 has demonstrated remarkable generalization capabilities across diverse downstream tasks. Despite being trained on natural images, it has shown strong transfer performance in medical domains. For example, [2] reports that DINOv2 achieves the best average segmentation accuracy among a range of pretraining methods. Additionally, [3] shows that directly using frozen DINOv2 features yields state-of-the-art results in medical image registration. These findings suggest that DINOv2 encodes rich and transferable semantic representations, even under domain shift.\n\n(2) Separation of Roles: DINOv2 for Quantization Alignment, DB for Domain Knowledge:\nDINOv2 is specifically used to align semantic representations before and after quantization, addressing the semantic degradation and information loss introduced by vector quantization. It serves as a stable, high-level anchor that preserves representation consistency across the quantization boundary. Crucially, medical domain knowledge is not derived from DINOv2; instead, it is injected via our Dual-Branch (DB) module, which incorporates task-specific features and medical semantics. This separation ensures that our model benefits from both robust general-purpose semantics (via DINOv2) and precise domain-specific guidance (via DB).\n\n(3) Comparison with Medical Foundation Models:\nTo further validate our design choice, we conducted an ablation study replacing DINOv2 with several alternative foundation models, including ones pretrained on medical data. The Dice scores under 5% and 10% labeled data are shown below:\n\n| Foundation Model  | Dice (5% Labeled) | Dice (10% Labeled) |\n| ----------------- | ----------------- | ------------------ |\n| CLIP              | 0.6421            | 0.7483             |\n| BiomedCLIP        | 0.6507            | 0.7629             |\n| MAE               | 0.6386            | 0.7541             |\n| rad-DINO          | 0.6535            | 0.7793             |\n| DINOv2 (Ours) | **0.6643**        | **0.7852**         |\n\nHere, CLIP \\[4] and BiomedCLIP \\[5] are vision-language models trained on large-scale image–text datasets (natural and medical, respectively). MAE \\[6] is trained via masked autoencoding, and rad-DINO \\[7] follows a DINO-style architecture trained on radiology-specific images. As shown, DINOv2 consistently outperforms all alternatives across both labeling regimes, including models specialized for medical domains. These results empirically validate DINOv2 as a robust and effective semantic prior for semi-supervised medical image segmentation.\n\n> References:\n\n[1] Setio A A A, Traverso A, De Bel T, et al. Validation, comparison, and combination of algorithms for automatic detection of pulmonary nodules in computed tomography images: the LUNA16 challenge[J]. Medical image analysis, 2017, 42: 1-13.\n\n[2] Baharoon M, Qureshi W, Ouyang J, et al. Evaluating general purpose vision foundation models for medical image analysis: An experimental study of dinov2 on radiology benchmarks[J]. \narXiv preprint arXiv:2312.02366, 2023.\n\n[3] Song X, Xu X, Yan P. Dino-reg: General purpose image encoder for training-free multi-modal deformable medical image registration[C]//International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2024: 608-617.\n\n[4] Radford A, Kim J W, Hallacy C, et al. Learning transferable visual models from natural language supervision[C]//International conference on machine learning. PmLR, 2021: 8748-8763.\n\n[5] Zhang S, Xu Y, Usuyama N, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs[J]. arXiv preprint arXiv:2303.00915, 2023.\n\n[6] He K, Chen X, Xie S, et al. Masked autoencoders are scalable vision learners[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022: 16000-16009.\n\n[7] Perez-Garcia F, Sharma H, Bond-Taylor S, et al. Exploring scalable medical image encoders beyond text supervision[J]. Nature Machine Intelligence, 2025, 7(1): 119-130.\n\n***\nAdditional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details."}, {"review_id": "matV95aXX0", "reviewer": "Reviewer_Nzxa", "summary": "The paper presents a vector-quantized token perturbation method for feature perturbation in semi-supervised medical image segmentation. The continuous output features from the encoder network are quantized and subsequently perturbed using the proposed Quantized Perturbation Module (QPM). The decoder network is designed to both reconstruct the input image and produce the output segmentation map. To preserve semantic priors, a Post-VQ Feature Adapter is used to align the quantized features with embeddings from a foundation model. Empirical results on the Lung Cancer (LC) and ACDC datasets demonstrate the effectiveness of the proposed methodology compared to baseline methods.", "questions": "1. During training, did you observe any signs of codebook collapse, such as a small subset of codewords being used disproportionately while others remained inactive?\n2. In Table 1, two sets of results are presented, and the results in the upper rows(above the dividing horizontal line) are consistently better than those of the proposed method. Could you clarify this discrepancy?", "limitations": "Addressed", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The proposed QPM provides a novel alternative to the dropout based methods for feature perturbation.\n2. The methodology is supported with theoretical proofs.\n3. The experiments highlights the effectiveness of proposed framework against baselines.\n4. Ablation studies highlight the importance of each component in the proposed module.\n\n## Weaknesses\n\n1. The design of the codebook and the choice of codebook size=16384 are not explained.  \n2. The dual branch pipeline introduces unnceessary computational overhead. \n3. The performance of the student network on the unlabeled data is dependent on the pseudo-labels generated from the teacher network. The reliablilty of the pseudo labes is not discussed.\n4. The choice and design of encoder and decoder networks are not explained.", "was_revised": true, "final_justification": "The authors performed additional experiments to validate the utilization of the learned codewords and the convergence of the codebook. These results further validated the effectiveness of the proposed framework.", "rebuttal": "We sincerely appreciate your constructive feedback. Below, we provide point-by-point responses, with all revisions incorporated accordingly. \n***\n\n> [W1] The choice of codebook size.\n\n[A1] Thank you for the helpful comment regarding codebook size.\nWe conducted an ablation study to investigate how the codebook size $K$ influences segmentation performance under 5% and 10% labeled settings:\n\n| Codebook Size | Dice (5%)  | Dice (10%) | Codebook Utilization (%) |\n| ------------- | ---------- | ---------- | --------------- |\n| 1024          | 0.6531     | 0.7608     | 100%            |\n| 2048          | 0.6582     | 0.7748     | 100%            |\n| 4096          | 0.6627     | 0.7775     | 99%             |\n| 16384         | **0.6643** | **0.7852** | 98%             |\n| 32768         | 0.6595     | 0.7764     | 95%             |\n| 65536         | 0.6415     | 0.7638     | 92%             |\n\nThe results show that segmentation performance consistently improves as the codebook size increases, peaking at $K = 16384$, where Dice scores are highest and utilization remains high. Beyond this point, performance degrades and utilization drops, suggesting that excessively large codebooks may lead to underutilization. These findings indicate that a codebook size of 16384 strikes the best balance between representational capacity and generalization performance.\n\n> [W2] Dual-branch design incurs unnecessary overhead.\n\n[A2] We thank the reviewer for raising the concern regarding the computational overhead introduced by the dual-branch (DB) architecture. While the DB module adds one additional image decoder, it reuses the same post-VQ latent space. More importantly, it plays a crucial role in improving both segmentation performance and codebook utilization, especially under low-label regimes. We justify this design choice from two perspectives:\n\n* Segmentation Performance Improvement: As shown in Table 2 of the main paper, adding the DB module to a model already equipped with QPM yields a clear performance gain: Dice increases from 0.7701 to 0.7784. This suggests that the reconstruction branch helps preserve more structural details in the post-VQ representation, directly benefiting segmentation accuracy.\n\n* Improved Codebook Utilization: We further conduct an ablation study to evaluate the impact of each module on codebook utilization (with codebook size set to 16,384). Introducing the DB module increases utilization from 87.2% to 90.1%, indicating that the auxiliary reconstruction task provides an additional learning signal that encourages broader and more stable codeword activation.\n\n| Setting                | Dice    | Codebook Utilization (%) |\n|------------------------|---------|-----------------|\n| Base                   | 0.7443  | 11.6%           |\n| + QPM                  | 0.7701  | 87.2%           |\n| + QPM + DB             | 0.7784  | 90.1%           |\n| + QPM + DB + PFA       | **0.7852**  | **98.0%**       |\n\nAlthough the DB module introduces a slight computational overhead, it significantly enhances codebook utilization and helps maintain a semantically meaningful perturbation space. This results in clear improvements in both segmentation performance and representation quality, which is especially valuable in semi-supervised settings.\n\n> [W3] The reliability of pseudo-labels is not discussed.\n\n[A3] Thank you for the comment. We would like to clarify that our pseudo-labeling strategy is consistent with those adopted in prior teacher-student-based semi-supervised segmentation methods, including the baselines \\[1] used for comparison. Specifically, we employ an Exponential Moving Average (EMA) strategy to update the teacher network—a widely accepted practice that improves the stability and reliability of pseudo labels by smoothing parameter updates over time.\n\nWhile NeurIPS does not allow modifications to the original submission during the rebuttal period, we will include visualizations of the generated pseudo labels in a future revision to better illustrate their quality and consistency.\n\n> [W4] The choice and design of encoder and decoder networks.\n\n[A4] To ensure a fair comparison, our VQ-Seg model adopts the same encoder and decoder architecture as Unimatch \\[2], which achieves the best average performance among all comparison methods. Detailed architectural specifications can be found in the original Unimatch paper.\n\n> [Q1] Signs of codebook collapse.\n\n[A5] Thank you for raising this important point. To clarify, our VQ-Seg framework achieves high codebook utilization, reaching 98% with a codebook size of $K = 16384$, as shown in the ablation study below:\n\n| Setting          | Dice       | Codebook Utilization (%) |\n| ---------------- | ---------- | ------------------------ |\n| Base             | 0.7443     | 11.6                     |\n| + QPM            | 0.7701     | 87.2                     |\n| + QPM + DB       | 0.7784     | 90.1                     |\n| + QPM + DB + PFA | **0.7852** | **98.0**                 |\n\nThis table demonstrates that each proposed component—QPM, DB, and PFA—contributes substantially to alleviating codebook collapse and enhancing utilization. In contrast, the base setting suffers from severe underutilization (only 11.6%), which clearly indicates codebook collapse.\n\nWe further examined the impact of varying codebook sizes on both segmentation performance and utilization:\n\n| Codebook Size | Dice (5%)  | Dice (10%) | Codebook Utilization (%) |\n| ------------- | ---------- | ---------- | ------------------------ |\n| 1024          | 0.6531     | 0.7608     | 100                      |\n| 2048          | 0.6582     | 0.7748     | 100                      |\n| 4096          | 0.6627     | 0.7775     | 99                       |\n| 16384         | **0.6643** | **0.7852** | **98**                   |\n| 32768         | 0.6595     | 0.7764     | 95                       |\n| 65536         | 0.6415     | 0.7638     | 92                       |\n\nThese results confirm that codebook utilization remains consistently high (>90%) across different codebook sizes. Notably, performance peaks at $K = 16384$, suggesting it provides the best trade-off between representational capacity and generalization. Beyond this point, increasing the size yields diminishing returns and slightly reduced utilization.\n\nTaken together, these findings strongly suggest that our framework effectively avoids codebook collapse during training.\n\n> [Q2] Superior results in upper rows of Table 1.\n\n[A6] Thank you for pointing this out. The upper portion of Table 1 reports results from fully supervised models (denoted as “F”), which are trained using 100% of the labeled data. In contrast, our method and others listed below the horizontal line are trained using only 5% or 10% of the labeled data under a semi-supervised setting.\n\nThe apparent performance gap stems from this difference in supervision level. However, one of the core strengths of VQ-Seg lies in its ability to deliver competitive performance with minimal annotations. As shown in Table 1, VQ-Seg achieves Dice scores of 0.6643 (5%) and 0.7852 (10%), already comparable to those of fully supervised methods.\n\nTo further assess scalability, we gradually increased the proportion of labeled data and report the results below:\n\n| Method             | 5%     | 10%    | 20%    | 50%        | 100%       |\n| ------------------ | ------ | ------ | ------ | ---------- | ---------- |\n| UNet-F (Full Sup.) | –      | –      | –      | –          | 0.8345     |\n| VQ-Seg (Ours)      | 0.6643 | 0.7852 | 0.8100 | **0.8507** | **0.9102** |\n\nRemarkably, VQ-Seg outperforms the fully supervised UNet (trained with 100% labels) even when using only 50% of the labeled data. This demonstrates that VQ-Seg performs robustly under limited supervision and continues to improve with more annotations, ultimately surpassing fully supervised baselines.\n\nThese findings underscore the effectiveness and scalability of our approach, clarifying that the superior results in the upper rows of Table 1 are due to supervision level differences rather than performance inconsistencies.\n\n\n***\n> References:\n\n[1] Yu L, Wang S, Li X, et al. Uncertainty-aware self-ensembling model for semi-supervised 3D left atrium segmentation[C]//International conference on medical image computing and computer-assisted intervention. Cham: Springer International Publishing, 2019: 605-613.\n\n[2] Yang L, Qi L, Feng L, et al. Revisiting weak-to-strong consistency in semi-supervised semantic segmentation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 7236-7246.\n\n***\nAdditional results/analyses on your concerns are in the revised manuscript. We hope these address your questions. Let us know if you need further discussion, experiments, or clarifications—happy to provide more details."}]

Author rebuttal:

Meta-review: This paper introduces VQ-Seg, a semi-supervised segmentation framework that replaces dropout-based consistency learning with a vector-quantization-based Quantized Perturbation Module. The framework further includes a dual-branch design and Post-VQ Feature Adapter (PFA) guided by a foundation model. The paper also contributes a new annotated LC dataset.

Reviewers initially expressed concerns about dataset scale, limited labeled ratios, foundation model dependence, codebook justification, and lack of error bars. However, the authors provided extensive new experiments in their rebuttal addressing many concerns with quantitative evidence. Three of the four reviewers raised their scores to weak accept and all reviewers now give positive ratings indicating unanimous support for acceptance.

Reviews and AC encourage the authors to include their responses in the camera-ready version, including but not limited to:

1. Include all additional experimental results 
2. Add error bars and statistical significance tests to main results
3. Add t-SNE visualizations of codebook evolution
4. Expand discussion on foundation model selection
Decision: Accept (poster)