PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization
Authors: Honglin Li
Zhongyi Shui
Yunlong Zhang
Chenglu Zhu
Lin Yang
Venue: NeurIPS 2025 poster ()
Keywords: Computational Pathology
Foundation Model
Whole Slide Image
Vector Quantization
Primary area: machine_learning_for_sciences
TL;DR: We improve the scalability of pathology foundation models through multi-scale vector quantization (MSVQ) to keep more  patch tokens with compression, achieving performance comparable to other SOTA methods with significantly less WSI data.

Abstract: Pathology whole slide image (WSI) analysis is vital for disease diagnosis and understanding. While foundation models (FMs) have driven recent advances, their scalability in pathology remains a key challenge. In particular, vision-language (VL) pathology FMs align visual features with language annotation for downstream tasks, but they rely heavily on large-scale image-text paired data, which is scarce thus limiting generalization. On the other hand, vision-only pathology FMs can leverage abundant unlabeled data via self-supervised learning (SSL). However, current approaches often use the [CLS] token from tile-level ViTs as slide-level input for efficiency (a tile with 224×224 pixels composed of 196 patches with 16×16 pixels). This SSL pretrained [CLS] token lacks alignment with downstream objectives, limiting effectiveness. We find that spatial patch tokens retain a wealth of informative features beneficial for downstream tasks, but utilizing all of them incurs up to 200× higher computation and storage costs compared [CLS] token only (e.g., 196 tokens per ViT$_{224}$). This highlights a fundamental trade-off between efficiency and representational richness to build scalable pathology FMs. To address this, we propose a feature distillation framework via vector-quantization (VQ) that compresses patch tokens into discrete indices and reconstructs them via a decoder, achieving 64× compression (1024 → 16 dimensions) while preserving fidelity. We further introduce a multi-scale VQ (MSVQ) strategy, enhancing both reconstruction and providing SSL supervision for slide-level pretraining. Built upon MSVQ features and supervision signals, we design a progressive convolutional module and a slide-level SSL objective to learn spatially rich representations for downstream WSI tasks. Extensive experiments across multiple datasets demonstrate that our approach achieves state-of-the-art performance, offering a scalable and effective solution for high-performing pathology FMs in WSI analysis.

PathVQ: Reforming Computational Pathology Foundation Model for Whole Slide Image Analysis via Vector Quantization
Honglin Li Zhongyi Shui Yunlong Zhang!. 1.2
Chenglu Zhu2t Lin Yang2.3,4f
College of Computer Science and Technology. Zhejiang University
2 School of Engineering, Westlake University
The Institute of Advanced Technology, Westlake Institute for Advanced Study
Center for Interdisciplinary Research and Innovation, MuyuanL.aborate (lihonglin in, zhuchenglu, yangl in @westlake edu. cn
Abstract
Pathology whole slide image (WSI) analysis is vital for disease diagnosis and understanding. While foundation models (FMs) have driven recent advances, their scalability in pathology remains a key challenge. In particular, vision-language (VL) pathology FMs align visual features with language annotation for downstream tasks, but they rely heavily on large-scale image-text paired data, which is scarce thus limiting generalization. On the other hand, vision-only pathology FMs leverage abundant unlabeled data via self-supervised learning (SSL). However, current approaches often the [CLS] token from tile-level ViTs as slide-level input for efficiency (a tile with 224x224 pixels composed of 196 patches with 16x16 pixels). This SSL pretrained [CLS] token lacks alignment with downstream objectives, limiting effectiveness. We find that spatial patch tokens retain a wealth of informative features beneficial for downstream tasks, but utilizing all of them incurs up to 200x higher computation and storage costs compared [CLS] token only (e.g., 196 tokens per 224). This highlights fundamental trade-off between efficiency and representational richness to build scalable pathology FMs. To address this, we propose a feature distillation framework via quattr-uaanizzation (VQ) that compresses patch tokens into discrete indices and reconstructs them via a decoder, achieving 64x compression (1024 + 16 dimensions) while preserving fidelity. We further introduce multi-scuu scale VQ (MSVQ) strategy, enhancing both reconstruction and providing SSL supervision for slide-level pretraining. Built upon MSVQ features and supervision signals, design progressive convolutional module and a slide-level SSL objective to learn spatially rich representations for downstream WSI tasks. Extensive experiments across multiple datasets demonstrate that approach achieves state-of-the the-art performance, offering a scalable and effective solution for high-performing pathology FMs in WSI analysis.
1 Introduction
Cancer remains one of the most challenging diseases to diagnose and prognosticate, with pathology playing pivotal role in understanding its complexities (28]. Traditional histopathological analysis relies heavily on manual examination tissue samples by pathologists, process that not only time-
Equual Contribution
Corresponding Author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

consuming but also prone to inter-observer variability 24]. In recent years, computational pathology has emerged as transformative method, leveraging whole-slide images (WSIs) to enable automated WSIs, which are high-resolution digital scans of entire tissue slides, provide wealth of information that can be harnessed for cancer diagnosis, prognosis, and treatment planning. However, the ultra-high resolution of WSIs, often exceeding billions of pixels, presents significant challenges for effective computational modeling [61[41].
and quantitative analysis of tissue samples [4
79 48
70 have shown strong potential in com- putational pathology. Studies have demonstrated the effectiveness of self-supervised learning (75L) TO 20 [5] and vision-language (VL) pretraining [17 48 57 in extracting semantic features on pathology images. The FMs typically process WSIs by dividing them into smaller tiles (e.g., x24 pixels as a tile), extracting features from each tile, and aggregating these features to make seide-level predictions. VI.-FMs S7
Recent advances in foundation models (FMs) 3
48 excel in downstream tasks (e.g. zero-/few-shot ROI classification (65), but their scalability is limited by the scarcity of large-scale image-text text contrast, vision-only FMs trained on unlabeled data via SSL are more scalable.
pairs 22 65 30
However, most methods adopt the task-agnostic [CLS] token from pretrained ViTs as a global repre- sentation each tile [88] 20 and fed as WSI input [31 61 IO 33 40, 62 43 67.. This approach overlooks critical spatial information captured by other spatial tokens, which are particularly essential for modeling nuanced pathological variations in gigapixel WSIs.
Notably, some studies have attempted to address this issue by scaling up to larger models (UNI-
2 [10] using ViT-giant) or combining [average pooling] features with the [CLS] token (Virchow- 2 (90). Disappointingly, these approaches yield only marginal improvements. Consequently, we argue that the scalability and performance of FMs is fundamentally constrained by the trade- off between efficiency (using [CLS] only) and representational richness (using all patch tokens): Leveraging all spatial patch tokens
benefit WSI analysis but incurs nearly 200x higher storage and training costs as shown in Figure
Information Retention (Cosine Similarity).
Downstream Task Performance (Bubble Size), Memory Cost/Latency
0.55
[ (e.g.. 196 tokens in ViT221). To address this, introduce feature distillation via vector quantization (VQ) [7T] 53] on patch features, which efficiently com- presses spatial patch tokens using dis-
Topds
0.30
0.85
Coenne Simterky
0.80
Memory Cost
8.70
crete indices and decoder. Our method reduces token dimensional- ity from 1024 to 16, achieving a 64x compression rate while preserving re- construction fidelity. This compres- sion process retains original spatial and contextual information, ensuring that critical features are preserved for downstream tasks.
Top19
0.05
0:90
CLS) Decomelldiin
Quer keakoy
Vadio colntication
(CLS) Altanion Tuute
[CLA) Attanion Tup64
[CLi
0.90
Epoche
Figure 1: Evaluation on information loss via reconstruction training. Directly using the [CLS) token results in significant information loss, making it difficult to reconstruct all patch tokens, potentially discarding critical details for downstream tasks. In contrast, vector quantization retains more original information and show stronger result downstream task 'BRACS'. The TopK patch tokens via CLS tokens attention selection are included for comparison.
Furthermore, we employ a multi-scale VQ (MSVQ) strategy, which uni- fies patch-level and tile-level feature VQ. Intuitively, tile-level feature like [CLS) token can be seen adaptive combination all patch features. thus they share the same feature space and can be learned into single VQ model. The MSVQ not only enhances VQ reconstruction performance but also serves SSL supervision target for seamless slide-level pretraining objective (working By integrating slide SSL into our framework, we enable the model to learn rich, discriminative representations from unlabeled WSIs. addressing the challenge limited WSI samples in computational pathology downstream tasks. Built upon the quantized features of patches and supervision targets of tiles via MSVQ, we develop progressive convolutional module and slide-level SSL to extract representations with rich spatial information for downstream WSI tasks, leading to more accurate and interpretable predictions for
as a tokenizer thus can be pretrained like BERT 15 25 53

---

tasks like cancer diagnosis and prognosis. The contributions of our work can be summarized as follows:
1) Efficient Token Compression with VQ Distillation: We propose a novel VQ-based framework that compresses patch-level spatial tokens by 64x while retaining critical spatial and contextual information, enabling scalable and efficient WSI analysis.
2) SSL Supervision via offline tokenizer: Our improved MSVQ strategy not only enhances feature reconstruction but also serves as an SSL supervision target for slide-level mask prediction, providing new direction for pretraining WSI models.
3) Rigorous Validation: Extensive evaluations on multiple datasets demonstrate the effectiveness of our approach, achieving -tate-of-the-art performance in WSI analysis tasks, with practical implications for clinical applications.
By addressing the computational challenges of WSI analysis while preserving critical spatial infor- mation, our framework offers new perspective on the development of computational pathology foundation models, paving the way for more accurate and scalable cancer diagnostics.
Method
2.1 Preliminary
For WSI modeling, WSI X is first divided into N tiles: X X2,, ,,,,, which are then pro- cessed by the FM. The pretrained FM ViT converts tile image x into patches x [P1. P2...., Pal- where the most commonly used patch size is 16 x 16. The ViT outputs all patch representations within tile: [s: h1 h2 hn). where serves
summary [CLS] of the spatial tokens of all
patches (ST [h1 ha..... h.)). Most existing approaches [1]
84
10] rely on the [CLS] token R N NxD. These embeddings are
from each tile to form WSI input embeddings S = [1, $2, ................................ SN
subsequently aggregated for slide-level prediction: Y g(S where g(0) can be an attention 3 mechanism or a Transformer.
In contrast, this paper explores using all spatial tokens H ST,,ST,... ST x] E R XxXD for slide-level prediction. Here, N (the number of tiles) can easily exceed 5k, n 196 (the number of patches per tile), and the feature dimension D = 1024 (for UNI (10). So, directly leveraging these high-dimensional data (about million patch tokens) computationally prohibitive for WSI training.
2.2 Vector Quantization Learning
To mitigate the computational burden while incorporating all patches' ST representations, we intro- duce vector-quantization (VQ) learning the pretrained FM's patch ST, illustrated in Figure 33. This framework consists of an encoder, a quantizer, and a decoder. Additionally, we extend VQ to support both patch and tile representations via multi-scale VQ strategy.
2.2.1 VQ for Patches
The spatial tokens (ST) are mapped into discrete codes through vector quantization (VQ). Specifically, the tile-level representation ST [h,, h2................... h,] is first passed through an MLP encoder to reduce its dimensionality from D to d:
(e,,e,..... e,.... e, Enc [hi be...., h,))
(I)
where the resulting low-dimensional representations [e1, eg,..., e, are subsequently tokenized into discrete indices ST [z1, , ).). The codebook V = [V1, V2, ... vel E RCdd consists of C' learnable embeddings. Each patch-level representation e; is assigned its nearest neighbor in the codebook via:
2, arg min -2 (
(2)
where j f1,2...., C)2 and lg denotes L2 normalization used for distance computation, ensuring that each patch token matched to the most similar codebook vector.
After quantization, the selected embeddings Ez, Ezy...., E.,, are passed to multi-layer Trans- former decoder to reconstruct the original spatial token representation. During training, the decoder output o; is aligned with the target h, by maximizing their cosine similarity.
3

---

Region with 256 tiles.
7k (14, 14)
TK-1 (7.7)
WSI
ABMIL
match tokenized index distribution
Figure 2: Multi-scale Vector Quantization (MSVQ) visualization. Based on MSVQ, the tile- and patch-level can be quantified simultaneously for slide-level pretraining and feature compression, respectively. The region data can be used to pretrain ABMIL via token index frequency matching.
Since the quantization operation in Equation 2 is non-differentiable, we adopt the straight-through gradient estimator [71. which copies gradients from the decoder input to the MLP encoder output for backpropagation.
The overall training objective VQ is defined as:
cos(o, h,) - ([2((:)) - (2(v.))12
max
ZEM
where M denotes the dataset of image tiles. For simplicity, we omit the straight-through gradient path and stop-gradient notation [7T]. During optimization, the MLP encoder, codebook embeddings, and Transformer decoder are jointly trained to reconstruct the original spatial token representations.
2.2.2 Multi-Scale Vector Quantization
To simultaneously compress patch-level spatial token ST representations and generate an offline tokenizer for WSI self-supervised learning. we propose Multi-Scale Vector Quantization (MSVQ)
module. MSVQ encodes the FM tile ST features into K multi-scale discrete token maps R (ri, (r.,,k,..,,Kk)
MSVQ builds upon the VQ architecture described in Section 2.2.1 with the key addition of multi- scale quantization module. The encoding process is designed with residual paradigm [6 38 as detailed in Algorithm
Intuitively, when R (rz) as scale ratio, MSVQ reduces standard VQ applied to the average- pooled patch token representation- -kininnnn--iiinnn to the tile-level [CLS] token. On the other hand, when R (rx), MSVQ behaves identically the patch-level VQ introduced in Section 2.2.1 The general form R (r....., rx) enables vector quantization at multiple-scale semantic levels, including tile, patch, and intermediate resolutions. Please refer Figure for visual illustration.
shared codebook Z is employed across scales, ensuring that tokens from each rı are drawn from a unified vocabulary [Vo]. The decoding process mirrors the encoding pipeline in reverse order.

---

Algorithm 1: Multi-Scale VQ Encoding
Inputs: FM's spatial token feature ST [b,, hg,-..; hn] # #h, is the VTT outputs tokens despite of CL.S;
Hyperparameters: number of scales K, resolutions (He, Wx) for
f EST( encoded feature,
represents the residual list;
.... K
FA f Q(interpllate(f, Hg, W&)), get residual of resolution level k; R
prene_push(R.rg
Z₁ lookup(Z. (Z,
Z < interpolato(ze: Ha, W+):
I + f Zx;
Return: multi-scale tokens R and codebook indices Z [21,
zx]:
Pre-processing WSI
Region
Tiles
(14x14 Patches)
WSI-level modelling
b. Vector Quantization on Patch Spatial Tokens
CPathFM
Attention MIL
CLS
Spatial-inoo Enriched
Decoder
Tile Features (N, 1x1, 1024)
Reconstructed ST 114x14, 1024)
Alignment
Flattened Spatial Tokens (ST) (14x14, 1024)
Upsample
Conv
V0-Embedding (14:14, dim)
ST Indices (14, 14. 1)
Encoder
All P atch features (N. 14x14, dim]
Compressed (14x14, diml
Quantizer
tile
Embedding
of WS1
Layer
Codebook Lookup [C., dim)
Figure 3:
Overview of the proposed framework. (a) The pipeline for compressing spatial patch tokens using vector quantization (VQ) and multi-scale VQ (MSVQ). (b) Slide-level self-supervised learning (SSL) using MS MSVQ-generated tokenizers. (c) Downstream WSI task fine-tuning with compressed patch features.
2.3 Slide-Levvl Self-Supervised Learning
Leveraging the offline tokenizer generated by MSVQ for all WSI tiles, design self-supervised learning (SSL) pretraining framework tailored for WSI-MIL analysis. This framework compatible with both mainstream MIL architectures, including attention-base based MIL (ABMIL) and Transformer- based models.
2.3.1 ABMIL-Based Self-Supervised Learning
supervised ABMIL training (e.g., WSI classification), adaptive pooling or max-pooling typically employed to aggregate tile-level features for prediction the WSI level. Inspired by this paradigm, we formulate simple yet effective SSI. objective for ABMII., grounded in the level-1 quantized indices from MSVQ (as shown in Figure
5

---

Given a large-region crop from a WSI (e.g.. a region of size 14336 x 14336, corresponding 4096 tiles each sized 224 224), the SSL objective for each region x defined as:
( - Σ ge log po())
(4)
where the soft target distribution (x(x) is computed based on the frequency of MSVQ token indices within the region, normalized over all C' codebook categories. Specifically, go(x) represents the proportion of tiles in region x assigned to token class c. The predicted probability pe (x(x), for class
is obtained by passing the region through an ABMIL model followed by a classifier head with softmax activation:
po xx)) softmax (classifier [AtmPool(x)))
(5)
2.3.2 WSI Transformer-Base Self-Supervised Learning
We adopt masked image modeling (MIM) strategy inspired by MAE 25 and BEiT [53 but with key difference: instead of raw image patches, we operate on pre-extracted feature representations as input. Given an input region composed of k tiles, represented as x {t1, 1,,..., ). we randomly mask a subset of tiles indexed by M. The masked positions are replaced with a shared learnable embedding e[M]. and Rotary Positional Embedding (RoPE) (64] is applied to retain spatial coherence. The corrupted imput becomes:
Xcorrpt (t1,t/, 1/c. t.t......, tt)
66)
For each masked tile, softmax classifier is trained to predict the corresponding token index, which is obtained from the level-1 quantized output of the MSVQ tokenizer (see Section 2.2.2 This provides discrete and consistent supervision signal.
The training objective formulated as:
mmask-modeling (0) = - -E
PogPo(z; xxM),
XED FEM
where 2; denotes the MSVQ token index for the i-th masked tile, and D is the dataset of training regions. Compared to the online tokenizer used in iBOT and related frameworks, our MSVQ-basee offline tokenizer provides more stable and reliable supervisory signal for SSL pretraining.
2.4 WSI Downstream-Tass Task Fine- Tine-Tuning
As illustrated in Figure 3 the refined WSI input consists of patch feature embeddings with a compressed shape of (N, 14, 14, dim), where N represents the total number of tiles in WSI, and (14, 14) corresponds to the standard 2D patch arrangement in ViT for each tile. To enhance the feature representation for downstream tasks, first apply convolutional layers (Convs) with upsampling, increasing the output channel size while reducing to fewer tokens to better capture task-relevant information. The extracted features are then reshaped match the original CLS token representation of tile-based VIT.
It is notable that the encoding direction of ST is not controllable since the the encoded embeddings are the middle layers (after encoding, before decoding). To keep its feature space as original, we align the output of Convs to original level-1 tile feature during VQ pre-training This module will be further fine-tuned during slide-level task. Finally, the processed features are fed into downstream MIL models, including both ABMIL and WSI-Transformer (see Section 2.1).
2.5 Overall Framework and Implementation
We summarize the overall framework of our method below. The WSI pre-processing follows the approach used in previous work [49]:
Patch-Level VQ Learning (Figure 3 3D) This module aims to compress all patch token
features from FMs, making them trainable for downstream tasks. Multi-scale VQ learning
(Figure 2 further enables slide-level SSL supervision and subsumes patch-level VQ learning.

---

Leveraging the tile-level tokenizer learned via MSVQ, SSL can be effectively applied to both ABMIL and WSI-Transforner models.
Levee SSL (Figure 2)
WSI Downstream-Tass Fine-Tuning (Figure Fine-tuning serves two purposes: (a) transforming patch features into a more suitable representation for downstream tasks, and (b) fine-tuning the pretrained slide-level SSL model for improved performance.
Experiments
In this section, we evaluate the performance of the proposed method and compare it with various baselines. Additionally, we conduct ablation studies further analyze its effectiveness.
3.1 Pretraining Implementation Details
VQ Pretraining: We conduct VQ pretraining on IM randomly cropped 224 224 tiles extracted
from all TCGA [69] diagnostic pathology WSIs. During training, the FM backbone (e.g., UNI with ViT-Large) remains frozen. The codebook has a size of 8192 with an embedding dimension of 16. For MSVQ, we employ multi-scale resolution list: {1 x 1, 2, 4 x 4, 4,7 x 7, 14 x 14). The VQ encoder, decoder, and codebook are frozen after pretraining. The model is trained on RTX-3090 GPUs for 50 epochs using batch size of 128 tile images per GPU. The total training time approximately 22 hours.
We crop all TCGA diagnostic WSIs into regions of resolution 3584 x 3584, yielding dataset of approximately 250k regions. To facilitate SSL, a pretrained MSVQ model is used to extract the quantized indices of each tile within region, requiring only about 65MB for storage.
WSI-SSL Pretraining:
During pretraining, the indices of each region are first re-embedded via a frozen VQ module, resulting in feature representation of shape (256, 14 x 14, 16). The convolutional module consists of four conv layers with a stride of 2, progressively increasing the output channels from 128 - 256 > 512 1024. This process transforms the features into spatially enriched embeddings of shape (256, x 1, 1024). These embeddings are subsequently fed into either an ABMIL. model or a 6-layer WSI-Trranfformrr for pretraining.
3.2
Downstream Tasks
ROI Classification Performance
We primarily focus WSI classification and survival prediction. For dataset details, please refer Appendix A.2] For illustration purpose, we also run two experiments on ROI classifica- tion clarify [CLS] token not all we need.
CRC-100K C17. WLL
Agcurayy
The data processing and embedding procedure identical to the region-basse based approach but are applied at the WSI level. The (æ. y) coor- dinates of tiles are also stored to facilitate posi- tional encoding in the WSI-Traasfformrr.
During fine-tuning, both the convolutional mod- ule and the WSI model are trained with a batch size of 1 for 20 epochs. The learning rate is fixed at 10-4, with a weight decay of 1 x 10-4, us- ing the AdamW optimizer with default settings.
UNI
UNI last block
UNI PathliQ
UNI-2 probe
GigaPath probe
Figure 4: ROI classification. Obviously, further fine-tuning (FT) on the last block UNI ViT can further improve the downstream results. Path by compressing and reconstructing the patch spa- tial token, achieve comparable improvement. UNI-2, however, does not show consistency im- provement compared FT and Path VQ.
For ABMIL, both randomly initialized and pre- trained models are fine-tuned tuned using the same hyper-parameters and training protocol. For Tran-rransfrrmrr. LoRA 27 adaptation (ap- plied to all nn. Linear layers) used with rank 16 during fine-tuning of the pretrained model to mitigate overfitting. For Transformer initialized from scratch, full fine-tuning is employed.

---

3.2.1 Tile/ROI Classification
We evaluate tile/ROI classification performance using dataset CRC-100K
35 (9 categories) and
Camelyon-17 WILDS (36) tiles (binary). The result in shown in Figure 4 By only updating the last Transformer block of UNI, the result can be significantly improved. Our Path VQ method is also included and shown comparable improvement (freeze backbone and fed [CLS] token feature to classification head), UNI-2, also using linear-probe seems can scaling up with strong performance on every down-stream tasks.
FT. All these results are better than linear probe
3.2.2
WSI Tumor Classification
Table I:
Slide-Level Tumor Classification based FM. The results in the first-row are all trained on UNI, while the second-row we include some recent stronger FMs. The cyan rows are our methods including VQ and Slide-level Pre-Training (SPT). The orange rows demonstrate how much (A) of our PathVQ method and UNI-2 improved over UNI with ABMIL setting. The bold and underline denote the best and second-best result, respectively.
Tumor classification
Mutation Prediciton
BRACS
LGG-GBM
Method
F1
AUC
F1
AUC
CLAM-SB 49
0.640
0.844
0.672
0,842+0.03
DTFD-MIL [84
0.655+00.0
0.878=+0.22
0.697
0.857+0.02
TransMIL [61
0.592
0,859+0.02
0.678
0,847+0.03
ABMIL [31]
0.692++0.3
0.875
0.685
0.852++0.4
+PathVQ
0.730=0002
0.902+0.01
0.723+0004
0.871+0.04
A over UNI + ABMIL
3.8%
2.7% 1
4.8% f
1.9% t
+PathVQ SPT
0.747=0001
0.906+0.01
0.752+0.03
0,879+0.02
Roformer
0.678=00.05
0.882+0.01
0.675+0.03
0.861+0002
+PathVQ
0.711
0.892+0.01
0.739+0004
0,872+0.02
+PathVQ SPT
0.754+0002
0.910=+0.01
0.758
0.886++0.01
UNI-2 + ABMIL
0.698
0.88711002
0.699
0.859
A over UNI + ABMIL
0.6%
1.2%
1.4% f
0.7% f
GigaPath TITAN
0.677+0003
0.862++0.0
0.703
0.864
0.696
0.891
0.7 +0.03
0.868+0002
We first evaluate our method on the BRACS 4 dataset with three categories- negative. benign, and malignant cancer. We then evaluate on TCGA LGG-GBM [69] focus on R132 [2] gene mutation as binary classification. (We notice that popular-used WSI binary tumor classification tasks(c.g. Camelyon [47). TCGA-NSCLC 699 are nearly solved (AUC>97) given FMs progress. So here we mainly focus more difficult task, like more categories, and will explore and validate more difficult
datasets in near future.)
Compared Baselines: Since our method primarily focuses extracting improved tile-level features for WSI analysis, we compare it against various WSI analysis models with different architectural designs: ABMIL (31] DSMIL [39] (introduces max-pooling branch alongside the attention
mechanism), and DTFD-MIL 88) (employs sub-bags for hicrarchical learning). TransMIL [61]
(leveraging Nyström self-attention (78) for computational efficiency), Transformer with 2-d RoPE 64
54].
FMs like GigaPath (a 12-layers WSI- Transformer (efficiently implemented using Long ViT 74
pretrained on large-scale private data via MAE (25), with [CLS] token as tile feature). TITAN 77
17 (a 6-layers WSI-Transformes with 2D-ALiBi positional encoding 55 41, pretrained on large- scale private data using iBOT 88), with Conch-v1. 1.5 as the tile feature extractor ([CLS) token).), and UNI-2 are also included. Certain works that focus on orthogonal aspects, such overfitting mitigation, hard instance mining, etc. primary comparison.
67
80), are not included in our
8

---

For all the experiments, we report the macro-AUC and macro-Fl scores (over five-runs or five-fold cross validation) because of class imbalance.
WSI Classification Results Analysis:
The results are reported in Table | We can first observe that ABMIL and Roformer show significant improvement when combined with Pa compressor into UNI. The results difference of UNI+Path VQ+ABMIL (about 3% improvement with adding IM tile data) and UNI2+ABMIL (about 1% improvement with adding large-scale (>>1M) of tile data, and 2 ~ 6x model size) demonstrate that the scalability of previous FMs are bottlenecked by the [CLS] token information losses. In addition, the results of our slide-level pretraining (SPT) also show consistency improvement compared with random initialization.
3.2.3 WSI Survival Prediction
Table 2: Survival prediction Results of Path and baselines for measuring patient cisease-specific
survival. All methods in Prototype and MIL. use UNI features 100 Best performance in bold, second best underlined.
TCGA
BRCA
CRC
BLCA
UCEC
KIRC
H2T
0.672:10.7
0.639
0,5662005
0.715
0.703
Prototype
OT
50
0.755
0.622.00.9
0.603
0.747=0.08
0.695
(umsup. cox
PANTHER
63
0.758#0.06
0.66540010
0.612
0.757=0.10
0.71640.10
AttnMISL 81
0.627
0.639
0.485
0.581
0.649
ILRA 77
0.649
0.555
0.550
0.632=0002
0.637
TransMIL
0.612++0.07
0.684
0.595
0.695=0.08
0.6.10
ABMIL [31 f
0.644.10.05
0. 0.608
0.550:00.6
0.669=0007
0.684-10.06
ABMIL reproduce
0.633
0.61 0.601.100008
0.540
0.67 0.008
0.691 1:0.08
MIL
+ PathVQ
0.655
0.649+011
0.608
0.721 =0.10
0.760
(opervised. UND
A over ABMIL.
2.2% 1
3.7% f
6.8% 1
5.0% f
6.9% 1
+PathVQ + SPT
0.674
0.659
0.616:0005
0.748 -011
0.778=
Roformer
0.602400.09
0.617
0.572
0.721 =0.08
0.655
+PathVQ
0.644
0.5871009
0.597:000
0.741 =0009
0.748
+PathVQ + SPT
0.67310.07
0.6791008
0.603
0.734
0.765
UNI-2
ABMIL.
0.61440.02
0.618
0.539
0.672
0.659
CHIEF 76
0.73720004
0.680
0.599
0.758=0.10
0.736:1006
Slide-FMs
GigaPath
0.687:10.08
0.628
0.589
0.779
0.751 1:000
(SOTA, ckp--nlly)
TITAN
[cox loss)
0.713
0.710
0.657=+0.s
0.789 =0.09
0.774-t0.06
We evaluate survival prediction on five TCGA datasets: BRCA, BLCA, CRC, UCEC, and KIRC. The model is trained using the negative log-likelihood (NLL, notice that some compared models' are trained via Cox-loss generally gain better result, please check Appendix A.6 for details.) loss and evaluated using the c-index with f-lold cross validation (the result of last epoch is reported).
For fair comparison, follow the default training pipeline of PANTHER 63 including hyper- parameters and data splits, and integrate our proposed model modifications along with pretrained weights.
Compared Baselines: We categorize the baselines into three groups: Unsupervised Prototype-
Based Approaches: H2T [73 (clusters tile embeddings and pools them within each cluster), OT (50)
(aggregates patch features into set prototypes using Optimal Transport), PANTHER [63] (models
prototype tile embeddings via Gaussian Mixture Model). Supervised MIL Models: AttnMISL [8]
(combines prototype-baat based learning with MIL). ABMIL [1). TransMIL (61), ILRA (771, and Transformer with RoPE [64]. Slide-Level FMs: CHIEF [76]: A large-scale ABMIL prettripe
model using contrastive learning to predict organ source, with CTransPath as the tile feature extractor
(mean-pooled features), GigaPath [9] and TITAN 17 UNI-2 TT feature extractor with ABMIL
model.
Survival Prediction Results Analysis:
The results reported in Table We can observe that ABMIL and Roformer show significant improvement when combined with our PathVQ compressor into UNL. But for Roformer with large-scale of parameters, the performance get easily overfitting to labels thus showing interior performance, which demonstrate the necessity of pretraining. For UNI-2, the improvement is marginal, proving our previous claim the [CLS] bottleneck of scalability.

---

3.3 Ablations and Visualization
Complexiyy
Performance vs. Complexity across re- duction methods is shown in Figure 5 PCA: Reduces token dimensions by pro- jecting them onto a low-rank linear sub- space, preserving variance but potentially discarding discriminative details. The Fl- score reaches about 0.70.
VG-SPT
midt
698
Aggregates tokens by computing their mean, leading compact representations but often oversmoothing critical local information. We have con- ducted pooling original 196=14x14 token features into 4x4 and 7x7 tokens, resulting in about 10 and 2 times token reduction. However, the result only show similar per- formance compared to original CLS token.
Average Pooling:
Computebbral Complelty (haemyrum"
Figure 5: Performance-complextty trade-off across various token reduction strategies on BRACS. Our proposed VQ method achieves the best balance be- tween accuracy and efficiency, delivering the highest Fi-score (0.73) with significantly reduced token com- plexity. Other approaches, including PCA, average pool- ing, and CLSS -S-based methods, show varying trade-offs.
CLS and overall average-pooling formed in virchow, the CLS and average- pooling token can complement little to each other and resulting in around 1.0 point improvement.
per-
We also performed token selection via CLS token attention top-k ranking. The selected top64 can reach Fl Fl-score around 0.72 but is still too heavy in computation cost. The selected topl6 can reach a Fl-score of 0.70, can be seen as a trade-off, but lose too much on performance.
Our proposed VQ-based method achieves the best balance, attaining the highest Fl-score of 0.73 while significantly reducing token complexity. In contrast, other strategies demonstrate varying degrees of compromise between accuracy and computational cost.
Convs pretraining ablation: Please check Appendix Figure 6
For additional ablation studies and visualizations, please refer to the Appendix A.4
The main
findings: The VQ reconstruction performance remains relatively stable when varying the quantized embedding dimension (32) and codebook size (16384). The reconstruction results MSVQ show improvements.
4 Conclusion and Limitations
In this work, we introduced a novel vector quantization (VQ) distillation framework to address the inherent bottleneck of existing computational pathology foundation models in whole-slide image analysis. Furthermore, our multi-scale VQ strategy unifies patch- and tile-level features, not only improving feature reconstruction but also serving as an effective self-supervised learning supervision target for slide-level pretraining. The main limitation is that the VQ learning process need extra training data. And the VQ still lose some information though is acceptable. By efficiently compressing patch-level spatial tokens while preserving critical spatial and contextual information, our method significantly reduces storage and computational costs without compromising performance.
Acknowledgments
This study was partially supported by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (Grant 2025SDXHDX0003), the National Natural Science Foundation China (Grant No. No.62506306), and foundation of Muyuan Laboratory (Program ID: 14106022401,1 1,,002224,1110022240222 Furthermore. essential technical support was provided by the D-PathAl platform, including its hardware and software, which was developed by Hangzhou Dipath Technology Co., Ltd.
10

---

References
[1] Benjamin Bergner, Christoph Lippert, and Aravindh Mahendran. Iterative patch selection for high-resolution image recognition. In The Eleventh International Conference Learning Representations, 2023.
[2] Fonnet E Bleeker, Nadia A Atai, Simona Lamba, Ard Jonker, Denise Rijkeboer, Klazien Bosch, Wikky Tigchelaar, Dirk Troost, W Peter Vandertop, Alberto Bardelli, et al. The prognostic idhl r132 mutation associated with reduced nadpt-dependent idh activity in glioblastoma. Acta neuropathologica, 119:487-44 494, 2010.
[3] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint 1rXiv:2108007255 2021.
[4] Nadia Brancati, Anna Maria Anniciello, Pushpak Pati, Daniel Riccio, Giosuè Scognamiglio, Guillaume Jaume, Giuseppe De Pietro, Maurizio Di Bonito, Antonio Foncubierta, Gerardo Botti, Maria Gabrani, Florinda Feroce, and Maria Frucci. Bracs: A dataset for breast carcinoma subtyping in h&e histology images, 2021.
[5] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus Busam, Edi Brogi, Victor Reuter, David S Klimstra, and Thomas J Fuchs. Clinical Clinical-grade computational pathology using weakly supervised deep learning whole slide images. Nature medicine, 25(8):1301-1309 2019.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. CoRR, abs/21 104.110411 14294, 2021.
[7] Tsai Hor Chan, Fernando Julio Cendra, Lan Ma, Guosheng Yin, and Lequan Yu. Histopathology whole slide image analysis with heterogeneous graph representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15661-15670, 2023.
[8] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11315- 11325, 2022.
[9] Richard J. Chen and et al. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In CVPR, pages 16144-16155, June 2022.
[10] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards general- purpose foundation model for computational pathology. Nature Medicine, 30(3):850-862, 2024.
[11] Richard Chen, Ming Y. Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y. Chen, Drew F. K. Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. 2021.
[12] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. arXiv preprint arXiv:2002.05709, 2020.
[13] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint arXiv:2 02057. 2021.
[14] Yufei CUI, Ziquan Liu, Xiangyu Liu, Xue Liu, Cong Wang. Tei-Wei Kuo, Chun Jason Xue, and Antoni B. Chan. Bayes-MII.: A new probabilistic perspective on attention-based multiple instance learning for whole slide images. In The Eleventh International Conference on Learning Representations, 2023.
[15] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 181118100.00 10.048855 2018.
11

---

[16] Tong Ding. Sophia J Wagner. Andrew H Song, Richard J Chen, Ming Y Lu, Andrew Zhang. Anurag J Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, et al. Multimodal whole slide foundation model for pathology. arXiv preprint arXiv:2411 1. 19666,, 2024.
[17] Tong Ding, Sophia J. Wagner, Andrew H. Song, Richard J. Chen, Ming Y. Lu, Andrew Zhang, Anurag J. Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, Drew F. K. Williamson, Bowen Chen, Cristina Almagro-Peree Paul Doucet, Sharifa Sahai, Chengkuan Chen, Daisuke Komura, Akihiro Kawabe, Shumpei Ishikawa, Georg Gerber, Tingying Peng, Long Phi Le, and Faisal Mahmood. Multimodal whole slide foundation model for pathology, 2024.
[18] William H Equitz. A new vector quantization clustering algorithm. IEEE Iransuctions on acoustics, speech, and signal processing, (((0)))568-15777 1989.
[19] Christopher Fifty, Ronald Guenther Junkins, Dennis Duan, Aniketh Iyengar, Jerry Weihong Liu, Ehsan Amid, Sebastian Thrun, and Christopher Re. Restructuring vector quantization with the rotation trick. In The Thirteenth International Conference on Learning Representations, 2025.
[20] Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Alice Mac Kain, Charlie Saillard, and Jean-Baptiste Schiratti. Scaling self-supervised learning for histopathology with masked image modeling. medRxiv, 2023.
[21] Olga Fourkioti, Matt De Vries, and Chris Bakal. CAMIL: Context-aware multiple instance learning for cancer detection and subtyping in whole slide images. In The Twelfth International Conference on Learning Representations, 2024.
[22] Robert Gray, Vector quantization. IEEE Assp Magazine, 1(2):-29, 1984.
[23] Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang, Pei Dong, Jinxi Xiang. Wei Yang, Junzhou Huang, Yuyao Zhang, and Xiao Han. Node-aligned graph convolutional network for whole- slide image representation and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18813-18823, 2022.
[24] Metin N Gurcan, Laura E Boucheron, Ali Can, Anant Madabhushi, Nasir M Rajpoot, and Bulent Yener. Histopathological image analysis: A review. IEEE reviews in biomedical engineering, 2:147-171, 2009.
[25] Kaiming Hc, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CoRR, abs/2111 1.11777,77772 2021.
[26] Xinhai Hou, Cheng Jiang, Akhil Kondepudi, Yiwei Lyu, Asadur Chowdury, Honglak Lee, and Todd C Hollon. A self-supervised framework for learning whole slide representations. arXiv preprint 20xi:22402.06/88,, 2024.
[27] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li. Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint 20tiv:22/6.0900) 2021.
[28] Shigao Huang, Jie Yang, Simon Fong, and Qi Zhao. Artificial intelligence in cancer diagnosis and prognosis: Opportunities and challenges. Cancer letters, 471:61-71, 2020.
[29] Zhi Huang, Federico Bianchi, Ment Yuksekgonul, Thomas J Montine, and James Zou. visual-language foundation model for pathology image analysis using medical twitter. Nature medicine, 2939):337-2311 2023.
[30] Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mo- hammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Ime One million image-text pairs for histopathology. Advances in neural information processing systems, 36:37995-38017. 2023.
[31] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-basel deep multiple instance learning. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 2127-2136. PMLR, 10-15 Jul 2018.
12

---

[32] Guillaume Jaume, Paul Doucet, Andrew Song, Ming Yang Lu, Cristina Almagro Pérez, Sophia Wagner, Anurag Vaidya, Richard Chen, Drew Williamson, Ahrong Kim, al. Hest-lk: dataset for spatial transcriptomies and histology image analysis. Advances in Neural Information Processing Systems, 37:53798-5338 53833, 2025.
[33] Guillaume Jaume. Lukas Oldenburg, Anurag Vaidya, Richard J Chen, Drew FK Williamson, Thomas Peeters, Andrew H Song. and Faisal Mahmood. Transcriptomics-gudee slide represen- tation learning in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 9632-9644, 2024.
[34] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weines Limin Yu, and aa- ditya prakash. Additive MIL: Intrinsically interpretable multiple instance learning for pathology. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[35] Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue, April 2018
[36] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, al. Wilds: A benchmark of in-the-wild distribution shifts. In International conference on machine learning,
pages 5637-5664. PMLR, 2021.
[37] Tristan Lazard, Marvin Lerousseau, Etienne Decencière, and Thomas Walter. Giga-ssl: Self- supervised learning for gigapixel images. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4305-4314, 2023.
[38] Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Shin Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11523-11532 2022.
[39] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13318-14328, 2021.
[40] Hao Li, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. Generalizable whole slide image classification with fne-grained visual- semantic interaction. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11398 14077, 2024
[41] Honglin Li, Yunlong Zhang, Pingyi Chen, Zhongyi Shui, Chenglu Zhu, and Lin Yang. Rethink- ing transformer for long contextual histopathology whole slide image analysis. arXiv preprint arXiv:2410. 14195, 2024.
[42] Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bottleneck for weakly-supervised pathology whole slide image classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7454-7463, June 2023.
[43] Jiawen Li, Yuxuan Chen, Hongbo Chu, Qiehe Sun, Tian Guan, Anjia Han, and Yonghong He. Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11323-11332. 2024.
[44] Ruoyu Li, Jiawen Yao, Xinliang Zhu, Yeqing Li, and Junzhou Huang. Graph enn for survival analysis whole slide pathological images. In International Conference Medical Image Computing and Computer-Assinn
Intervention, pages 174-182. Springer, 2018
[45] Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang-Wen Chen. Interventional bag multi-instance learning on whole-slide pathological images. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pages 19830-19839, 2023. 2023,
13

---

[46] Lucas D Lingle. Transformer-vq: Linear-ting time transformers via vector quantization. arXiv preprint arXiv:2309.16354, 2023.
[47] Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob van de Loo, Rob Vogels, et al. 1399 h&e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset. GigaScience, 7(6))i 2018.
[48] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guil- laume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863 874, 2024.
[49] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6):555-570, 2021.
[50] Grégoire Mialon, Dexiong Chen, Alexandre d' Aspremoot, and Julien Mairal. A trainable optimal transport embedding for feature aggregation and its relationship to attention. arXiv preprint arXiv:2006.12008 2020.
[51] OpenAl. Gpt-4 technical report. arXiv, 2023.
[52] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint urXiv:2304.07 193, 2023.
[53] Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint 22X8 06366, 2022.
[54] Etienne Pochet, Rami Maroun, and Roger Trullo. Roformer for position aware multiple instance learning in whole slide image classification, 2023.
[55] Ofir Press, Noah A Smith, and Mike Lewis. Train short, test long: Attention with linear biases enables input length extrapolation. arXiv preprint arXiv:2 12409, 2021.
[56] Linhao Qu, xiaoyuan Luo, Manning Wang, and Zhijian Song. Bi-directional weakly supervised knowledge distillation for whole slide image classification. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103.0000 2021.
[58] Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelig images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
[59] Charlic Saillard, Rodolphe Jenatton, Felipe Siinres--óópes Zelda Mariet, David Cahané, Eric Durand, and Jean-Philippe Vert. H-optimus-0, 2024.
[60] Wei Shao, Tongxin Wang, Zhi Huang, Zhi Han, Jie Zhang, and Kun Huang. Weakly supervised deep ordinal cox model for survival prediction from whole-slide pathological images. IEEE Transactions Medical Imaging, (0(11))3739-374 2021.
[61] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang. Xiangyang Ji, and yongbing zhang. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 2136-2147, Curran Associates, Inc., 2021.
[62] Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu. Vila-mil: Dual-scale vision vision-language multiple instance learning for whole slide image classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 11 11248 1258, 2024.
14

---

[63] Andrew H. Song, Richard Chen, Tong Ding, Drew FK Williamson, Guillaume Jaume, and Faisal Mahmood. Morphological prototyping for unsupervised slide representation learning in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11566-11578, 2024.
[64] Jianlin Su, Yu Lu, Shengfeng Pan, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. 2021.
[65] Yuxuan Sun, Yunlong Zhang. Yixuan Si, Chenglu Zhu, Kai Zhang, Zhongyi Shui, Jingxiong Li, Xuan Gong, XINHENG LYU, Tao Lin, al. Pathgen- 1.6 m: 1.6 million pathology image-text pairs generation through multi-agent collaboration. In The Thirteenth International Conference on Learning Representations.
[66] Wenhao Tang, Sheng Huang, Xiaoxian Zhang, Fengtao Zhou, Yi Zhang, and Bo Liu. Multiple instance learning framework with masked hard instance mining for whole slide image classifica- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4078 4087, 2023.
[67] Wenhao Tang, Fengtao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang, and Bo Liu. Feature re-embedding: Towards foundation model-level performance in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11343 11352, 2024.
[68] Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng, and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. Advances neural information processing systems, 17:84839-84860 2025.
[69] Katarzyna Tomezak, Patrycja Czerwiáska, and Maciej Wiznerowiez. Review the cancer genome atlas (tega): an immeasurable source of knowledge. Contemporary Oncology/sspotczese Onkologia, 20 2015(1):68-77, 2015.
[70] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 139711 2023.
[71] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[72] Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, et al. Virchow: A million-slide digital pathology foundation model. arXiv preprint arX 2023.
[73] Quoc Dang Vu, Kashif Rajpoot, Shan Ahmed Raza, and Nasir Rajpoot. Handcrafted histo- logical transformer (h2t): Unsupervised representation of whole slide images. Medical image analysis, 85:10274,, 2023.
[74] Wenhui Wang, Shuming Ma, Hanwen Xu, Naoto Usuyama, Jiayu Ding, Hoifung Poon, and Furu Wei. When an image is worth 1,024 x 1,024 words: case study in computational pathology. arXiv preprint arXiv:2312.03355 2023.
[75] Xiyue Wang. Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Junzhou Huang, Wei Yang. and Xiao Han. Transpath: Transformer-based self-supervised learning for histopathological image classification. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pages 195. Springer, 2021.
[76] Xiyue Wang, Junhan Zhao, Eliana Marostica, Wei Yuan, Jietian Jin, Jiayu Zhang, Ruijiang Li, Hongping Tang, Kanran Wang, Yu Li, et al. A pathology foundation model for cancer diagnosis and prognosis prediction. Nature, 634(8035):970-975 2024.
[77] Jinxi Xiang and Jun Zhang. Exploring low-rank property in multiple instance learning for whole slide image classification. In The Eleventh International Conference on Learning Representations, 2023.
15

---

[78] Yunyang Xiong, Zhanpeng Zeng, Rudrasis Chakraborty. Mingxing Tan, Glenn Fung, Yin Li, and Vikas Singh. Nyströmformer. A nyström- based algorithm for approximating self-attention. Proceedings of the AAAI Conference on Artificial Intelligence, 2021.
[79] Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. A shole-slide foundation model for digital pathology from real-world data. Nature, pages 1 1-8, 2024.
[80] Shu Yang, Yihui Wang, and Hao Chen. Mambamil: Enhancing long sequence modeling with sequence reordering in computational pathology. In International Conference on Medical Image Computing and Compuler-Assisted Intervention, pages 296-306. Springer, 2024.
[81] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
[82] Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. arXiv preprint M..iv22110.04222 2021.
[83] Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng, Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion-tokenizer is key to visual generation. arXiv preprint arXiv:23 2023.
[84] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang. Sarah E. Coupland, and Yalin Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. ArXiv, abs/2203. 12081, 2022.
[85] Jingwei Zhang, Saarthak Kapse, Ke Ma, Prateek Prasanna, Joel Saltz, Maria Vakalopoulou, and Dimitris Samaras. Prompt-mil: Boosting multi-instance learning schemes via task speciiciii prompt tuning, 2023.
[86] Yunlong Zhang, Honglin Li, Yuxuan Sun, Sunyi Zheng, Chenglu Zhu, and Lin Yang. Attention- challenging multiple instance learning for whole slide image classification. arXiv preprint arXiv:2311.07125. 2023.
[87] Yunlong Zhang, Zhongyi Shui, Yunxuan Sun, Honglin Li, Jingxiong Li, Chenglu Zhu, Sunyi Zheng, and Lin Yang. Adr: Attention diversification regularization for mitigating overfit- ting in multiple instance learning based whole slide image classification. arXiv preprint arXiv:2406.1523 15303, 2024.
[88] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.078.. 2021.
[89] Wenhui Zhu, Xiwen Chen, Peijie Qiu, Aristeidis Sotiras, Abolfazl Razi, and Yalin Wang. Dgr- mil: Exploring diverse global representation in multiple instance learning for whole slide image classification. European Conference on Computer Vision, pages 333-351. Springer, 2024.
[90] Eric Zimmermann, Eugene Vorontsov, Julian Viret, Adam Casson, Michal Zelechowski, George Shaikovski, Neil Tenenholtz, James Hall, David Klimstra, Razik Yousfi, et al. Virchow2: Scaling self-supervised mixed magnification models in pathology. arXiv preprint 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction are accurately reflect the paper's 's contributions and scope, matching the empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer:[Yes]
Justification:
The main weakness is discussed in last section: need further pre-training, the training data mainly include TCGA.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer: [Yes]
Justification: We provide patch and tile visualizations to support the theoretical multi-scale Vector Quantization learning.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide comprehensive details of dataset selection and pre- pre-processing together with training details. We will release full code implementation.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
18

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All our data is public and use 'CLAM' for pre-processing. We will release full code implementation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
aublic//uuddes/Codeuubbsss wwt.ssiinnooooyylo for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wwtps/cccppplgguuuaaaeoeaec
Cttelee//ooSSSSSeSS..ee.a www.ssssiicogoooooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide model backbone selection and implementation details core paper and further provide data split and pre-processing in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: For each experiments we have reported the mean and std via multi-runs or cross-folds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our memory efficiency method is RTX3090 GPU (24g), detailed in core paper and appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't fidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripp: http://cc..pppplttttcccccceeeeee
Answer: [Yes]
Justification: We have read the NeurIPS Code of Ethics and understood the policies, and we believe that neither the manuscript nor the study violates any these.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: There is no negative societal impact our paper. Our work focus on computer aided diagnosis for potential medical use, which is discussed in conclusions of core paper.
Guidelines:
20

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work mainly focus on recognition for real-world medical pathology image, without any generative problem.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
All the data and tools are public accessible and well referenced.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
21

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, papprswipphaeeoooccaaappp thcode. com/ com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: There is no new assets right now.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/mooel as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: No human subjects were directly involved in this study. The research was con- ducted using publicly available datasets that have been previously collected with appropriate ethical approvals and anonymization. Therefore, no additional IRB approval was required.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
22

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientifie rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurd) (ttrgp//rearrrsscccCoofeeereeeee cct..coeeeeeeceee/222255 for what should or should not be described.
23

---

A Supplementary Materials
A.1 Related Work
A.1.1 Pathological Whole Slide Image Analysis
Whole Slide Images (WSIs) contain a wealth of visual information that plays a crucial role in pathological analysis [S] 49). However, obtaining detailed cell-level annotations is both labor-
intensive and time-consuming 5 49 9. posing a significant challenge for large-scale WSI analysis.
To address this, weakly-supervised learning has emerged as a promising direction in computational pathology.
Computational Pathology Foundation Models The performance of early WSI MIL approaches [1
39
49 (84 34 [56
[4
[
[1
44 23 7 [21] relied heavily on tile-level features extracted
from pre-trained models 42 20) To address this, FMs have been developed and shown significant
advancements both tile-level 775
48
10
20 72 59 and WSI-level analysis [79 These FMs
leverage visual Supe-super Learning (SSL) techniques 6 52 1 12
large-scale unlabeled 10 20 or organize pathology image-tex mextettexttyg pairs to learn multimodal representation [57] FMs have demonstrated superior performance in downstream tasks such cancer subtyping.
datasets [75]
17 48
survival prediction, and biomarker identification.
Recently, authors in Hest- 1k 132 observe that tile encoders like Conch 48 can be further fine-tuned to obtain better downstream tile task result. However, the key challenge that the computational cost high-resolution WSIs makes fine-tuning [2542] 42)
FMs with overwhelm parameters difficult. Most approaches [48] I0 20 resort to using pretrained [CLS] token representation of tile-level FM as slide-level inputs, which may lead to the loss of critical spatial information. Some models, such as UNI-2 [TO. attempt to scale up ViTs into larger-size size as tile encoders to extract better feature representations, but only achieve marginal improvements from ViT-Large to ViT-Giant We argue that this performance bottleneck stems from the spatial information loss inherent in [CLS] token representations. Other efforts, such as Virchow-2 [90], find that combining [CLS] tokens with [AVG] (average pooling of all spatial tokens) can yield some improvement (less than point). Motivated by these findings, we propose keep but compress all spatial patch tokens and further extract useful information for downstream WSI tasks analysis.
Slide-level FMs/ / SSL pretraining: Some recent slide-level FMs, e.g. GigaPath 79 and TITAN 17 are modeled via Transformer with 6 to 12 layers, then pretrained via MAE 25 and iBOT [8 respectively. Some other slide SSL models 226 37 also propose to employ slide-level augmentation with contrastive learning (CL) [12 52) for pretraining. But there are some training problems of these works: 1) The main augmentations in level to generate different views for CL is limited, like crop or random-drop. since the tile feature are pre-processed and stored. This hinders the performance of CL pretraining. 2) The self-supervised target. The iBOT, masked token to match online-toker tokenizer,,, is not so stable during training. The MAE in GigaPath [79] need to regress the feature of masked tiles, which may be too difficult fit and hinder downstream tasks. The CHIEF 776,, on the other way, pretrain ABMIL by constrastively predicting tumor organ source (extra information).
88 used in TITAN 117 predicting
Different to these work, this paper we train a offline tile tokenizer via vector quantization which offer superviuerv for both for pretraining and need no further information.
mask modeling and ABMIL. This is more stable
A.1.2 Vector Quantization
Vector quantization (VQ) (22) is a fundamental technique in signal processing and machine learning, widely used for data compression, clustering, and generative modeling [T8] 7 58). Recent devel- opments in deep learning have led to neural vector quantization methods, such as Vector Quantized Variational Autoencoders (VQ-VAEs, VAEs) [7]
58 and quantized transformers [46], which integrate VQ into end-to-end learning pipelines to enhance expressiveness and efficiency. To further improve VQ, techniques such as residual quantization
38 and rotation tricks [99 have been proposed. Recent studies {2 83 reveal that lower-dimensional quantized vectors (dimension size ranging from 8 to 32) can improve codebook usage and reconstruction performance, providing strong compression capabilities that benefit this study. Unlike recent VQ methods that focus primarily visual genera-
24

---

tion 71 68 58, our work focus on feature compression and distillation of pretrained pathology tile encoder features via a VQ quantizer.
A.2 Data Description
BReAst Carcinoma Subtyping (BRACS) collect H&E stained Histology Images, containing 547 WSIs for three lesion types, i.e., benign, malignant and atypical, which are further subtyped into seven categories. Here, since the WSIs number is limited, we only perform three class subtyping.
TCGA 69]: Breast Invasive Carcinoma (BRCA, n 1, 041, WSI 1, 111), Colon and Rectum Adenocarcinoma (CRC, 566, WSI 575), Bladder Urothelial Carcinoma (BLCA, = 373, WSI = 437), Uterine corpus endometrial carcinoma (UCEC, n = 504, WSI = 565). Kidney renal clear cell carcinoma (KIRC, 511. WSI = 517), Brain Lower Grade Glioma (LGG) and Glioblastoma Multiforme (GBM) constitute WSI 463. The train/val split is performed on the patient level.
A.3 Experimental settings
VQ Pretraining: We conduct VQ pretraining on 1M randomly cropped 224 224 tiles extracted from all TCGA [69] diagnostic pathology WSIs. During training, the FM backbone (e.g., UNI with ViT-Large) remains frozen. The input tile images are augmented using RandomCrop (minimum ratio: 0.4) and RandomHorizznnaalFF
(probability: 0.5). The codebook has size of C' 8192 with an embedding dimension of 16. The MLP encoder consists of two linear layers with a tanh activation in between, transforming the feature dimension from 1024 to 16. The decoder first upsamples the feature dimension from 16 to 768 using a linear layer, followed by three Transformer blocks. Another linear layer then maps the features from 768 1024, ensuring alignment with the original feature tokens.
For MSVQ, employ multi-scale resolution list: x1x 1,2 x 2, x 4, 7 x 14 14).
The model is trained on RTX-3090 GPUs for 50 epochs using batch size of 128 tile images. The total training time is approximately 22 hours. The learning rate is set to 2 x 10 with a 5-epoch warmup, followed by cosine decay minimum learning rate of 10-5 The weight decay is 1 x 10 and the AdamW optimizer is used with 3 parameters set to (0.9, 0.99).
10
WSI-SSL Pretraining: We crop all TCGA diagnostic WSIs into regions of resolution 3584 3584, yielding a dataset of approximately 250k regions. To facilitate SSL, a pretrained MSVQ model is used extract the quantized indices of each tile within region, requiring only about 65MB for storage.
All pretraining is conducted on 4 GPUs for 20 epochs with an initial learning rate of 5 x 10-1, The first 2 epochs serve as a warmup phase, followed by cosine decay to a minimum learning rate of 1 x 10-3, The AdamW optimizer is employed with 3 parameters set to (0.9, 0.98) to ensure fast convergence.
For ABMIL pretraining, a batch size of 64 is used due to the model's simplicity. For WSI- USS--Tranfoommer, the batch size to 32, with 96 masked tokens out of 256. The learning objective for both models is formulated as a cross-entrope loss over 8192 categories, with ABMIL additionally utilizing soft targets.
AUC
0.85
UNI
RD+FT
PT+FT
PT+FZ
Figure 6: The VQ feature with Convs need pretraining and aligning tile's level-0 feature to attain good feature space and compression. RD: random initialize Convs. FT: fine-tuning during WSI analysis. PT: pretrained during VQ learning (align to level-0 tile feature). FZ: freeze during WSI analysis.
25

---

14x14 Patch Tokens Feature Reconstruction Training
0.00
0.70
0.60
variila VG
Mul0i soal
10
20
50
Epochs
Figure Reconstruction using Multi-Scale (MSVQ) or not. MSVQ obviously improve the rec performance.
14x14 Patch Tokens Feature Reconstruction Training
0.90
0.80
0.00
0.80
VO 1-1K
VO C-
10
20
50
Epochs
Figure 8: Reconstruction ablation on quantization codebook size, 8k and 16k.
14x14 Patch Tokens Feature Reconstruction Training
www. www.eeewwwe0000000000000000000..
Simfonty
0.75
0.65
0.65
Figure 9: Reconstruction ablation on quantization codebook embedding, 16 and 32.
26

---

Region-level
VQ
Tile-level
VQ
Figure 10: The VQ performed on both tile-level and peteh- The quantized index can be as type of prototypes (n=8k) with strong interpretability. Be aware that since the codebook is too large, the heatmap different index may share similar color.
A.4 Ablations on VQ
Multi-Scale (MSV)) Please check Figure 7
Codebook Size Please check Figure g
Codebook Embedding Dimension Please check Figure
A.5
Illustrative Visualization
Please check Figure 10
A.6 Survival Prediction Loss Comparison
Most current MIL methods 31 61
use NLL loss since WSI batch size is limited (most settings are 1) when the MIL models need fine-tuned with large bag size (GPU memory limit). However, the NLL loss is not optimal for this setting [60] Cox loss [60], on the other hand, is better than NLL in performance but it need large batch size to calculate the hazard ranking matrix among different samples, which is currently inevitable for MIL models. However, this is easy to be implemented if the bag-size is small (e.g. using unsupervised learning) to prototyping instances' features like PANTHER (3] Or using a strong WII pretrained module like TITAN [17] pre-extract the slide-level representations. Though currently method can not surpass above methods, but it show strong
27

---

improvement compared to other baselines which also using NLL loss. And we will further explore this problem (combining Cox loss into fine-tuning WSIIlevel representation model) in the future.
28


Reviews:
[{"review_id": "b39X58Nap1", "reviewer": "Reviewer_LDx8", "summary": "Processing whole-slide pathology images poses a computational challenge due to their sheer size. When working on slide-level tasks, the slides are divided into thousands of usually 224x224 tiles, each having 196 16x16 patches. Most models summarize whole tiles into a single [CLS] token for slide-level prediction, which is computationally efficient but loses rich patch-level information. As using large tokens from each class is prohibitive, the authors introduce a Vector Quantization (VQ) scheme, where a fixed-size codebook is learned and used to summarize each patch. \n\nFurthermore, two related techniques for slide-level pretraining are introduced and feature alignment is performed when converting the patch-level codes into a final slide-level encoding. The effectiveness of the method is well illustrated on 3 different tasks (tumor classification, mutation prediction, survival prediction) and it performs favorably compared to many competing approaches.", "questions": "I've covered my concerns about the novelty of the paper in the \"Weaknesses\" section.  \n\nAs of now, I would strongly advocate for the acceptance of this paper in a medically focused conference, but have my doubts as to how interesting it is for the machine learning community at large. I am still veering on the side of acceptance due to the strong aggregation and evaluation which makes this contribution a strong model and baseline for other researchers and practicioners to base their work on. If, after the rebuttal, I am convinced that there is more novelty in the paper than I've noticed so far, I'd gladly raise my score from 4 to 5.", "limitations": "The coverage of limitations is token, authors state the main weakness of VQ is that \"it needs more training data\" and that \"there is information loss although it is acceptable\". These claims should be further substantiated in supplementary material and the grammar of the conclusion section reviewed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n* The problem being tackled and the proposed VQ-based solution is explained and illustrated clearly, and solves a tangible problem in WSI-based tasks. \n* Related work is covered sufficiently. \n* Evaluation on tasks and competing models is extensive. \n\nWeaknesses\n\n* There are no particularly novel technical contributions in the paper. It aggregates several techniques which have been previously studied: \n* VQ is a well known technique, as covered in the paper's related work. \n* The WSI-Transformer self-supervised learning objective which tries to predict the codebook assignment of tokens missing in a sequence is very similar to masked token prediction, which is fairly well known [1]. \n* The ABMIL self-supervised learning objective which tries to predict the global distribution of token assignments is similar to [2], where predicting the code distribution of cross-view images is a task. \n* The token-alignment during conv-upsampling performed while summarizing patches into tiles is straightforward. \n\n[1] Bao, Hangbo, et al. \"Beit: Bert pre-training of image transformers.\" arXiv preprint arXiv:2106.08254 (2021).\n[2] Gidaris, Spyros, et al. \"Moca: Self-supervised representation learning by predicting masked online codebook assignments.\" TMLR (2024).", "was_revised": true, "final_justification": "I still find the novelties contained in the approach to be too limited as I stated in my initial reviews, despite the interesting points raised by the authors during rebuttal. The most interesting part of the rebuttal: The rank based perspective, is not present in the paper and was not designed for, and needs a deeper look.", "rebuttal": "We thank the reviewer for the insightful comments and appreciate the reviewer's concerns regarding the novelty of our method. While our work builds upon established techniques such as VQ and SSL, our core contribution lies in **unifying two longstanding challenges in WSI foundation model pretraining** through a coherent and scalable **multi-scale VQ framework**:\n\n**(1) Addressing the CLS token bottleneck via patch-level (16x16) VQ:**\n\nCLS token–based representations often discard fine-grained spatial information, leading to suboptimal performance on downstream pathology tasks. To mitigate this, we apply patch-level (patch size = 16) VQ/autoencoding to preserve local semantics while significantly reducing spatial dimensionality. This allows us to compress the token space without resorting to a single global embedding.\n\n**(2) Improving the SPT (Slide-level Pretraining) objective via tile-level (224x224) VQ:**\n\nTraditional masked autoencoding (e.g., MAE) focuses on reconstructing pixel inputs, which is infeasible for WSI-scale images due to GPU memory limits. We instead mask and predict **intermediate tile-level feature representations**, but just like MAE learn too many low-frequency feature of raw-pixel image [1], directly using feature-level targets also not align well with task-specific semantics.\n\nTo resolve this, we adopt a **tile-level (tile size = 224) VQ tokenizer** that discretizes feature targets into informative codebook entries. This approach enables more stable and effective slide-level representation learning compared to raw pixel or continuous-feature reconstruction.\n\n**Unified framework via Multi-Scale VQ:**\n\nRather than treating patch-level compression and slide-level representation learning as disjoint processes, we introduce a **Multi-Scale VQ architecture** that links both levels: patch-level VQ captures local detail, while tile-level VQ supports robust and interpretable global objectives. This unified view of representation compression and learning makes our approach both practically scalable and theoretically grounded for WSI tasks.\n\nIn addition, for addressing issue (1)—the CLS token bottleneck—we tackle a fundamental limitation in transformer-based slide-level models: **insufficient input rank**, which constrains their capacity to represent complex WSI data. As noted in [2], the rank of the attention matrix in a standard transformer is theoretically bounded:\n\n$$\n\\text{rank}(QK^\\top) \\leq \\min(\\text{rank}(Q), \\text{rank}(K)) = \\min(n, d),\n$$\nwhere n is the sequence length and d is the feature dimension. When using a single CLS token (or averaged tile-level CLS features), the actual rank is severely limited. Though in [2] authors try to improve the rank via local attention and positional embedding, the input rank limitation still exists. For example, we observe that in **Conch**, the effective rank of WSI embeddings using tile-level CLS is approximately **50**, and in **UNI**, it is around **400**—both significantly lower than the typical WSI sequence length (often exceeding **5,000** tokens).\n\n| Model | Rank (Mean ± Std) | WSI Length (Mean ± Std) | AUC Transformer |\n| --- | --- | --- | --- |\n| **Conch** | 53.6 ± 6.2 | 3356.1 ± 1840.6 | 0.861 |\n| **UNI** | 367.3 ± 109.7 | 3356.1 ± 1840.6 | 0.882 |\n| **Ours** | 765.4 ± 88.3 | 3356.1 ± 1840.6 | 0.910 |\n\nIn contrast, our method reconstructs spatial tokens via VQ-based representations, enabling **a much higher-rank representation** of the WSI content. We can also **increase the feature dimensionality (e.g., to 2048)**, which is more effective than simply up-projecting CLS features from 1024 using a linear layer.\n\nFor context, in standard ViT-Large (e.g., ViT-L/16 with 224×224 inputs), a 1024-d embedding is used for just 196 tokens. Large language models (e.g., LLaMA, Qwen ranging from 3~7B) employ embeddings of 2048–4,096 dimensions to represent typical question lengths. By analogy, for long-sequence AI applications such as WSIs, **our approach provides a scalable and rank-preserving alternative perspective**, and may generalize to other domains facing similar sequence-length bottlenecks.\n\nRef:\n\n[1] Beit: Bert pre-training of image transformers\n\n[2] Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis"}, {"review_id": "CwnNXTcld7", "reviewer": "Reviewer_TKaA", "summary": "This paper proposes a novel method, PathVQ, which initially employs a feature distillation framework based on Vector Quantization (VQ) to compress spatial patch tokens into discrete indices. This achieves substantial compression (e.g., 64-fold from 1024 to 16 dimensions) while preserving high fidelity and rich spatial semantics. Subsequently, a Multi-Scale VQ (MSVQ) strategy is introduced to unify patch-level and tile-level feature quantization, thereby enhancing reconstruction quality and providing robust Self-Supervised Learning (SSL) supervision for slide-level pretraining. The MSVQ-based offline tokenizer serves as a stable and effective supervisory signal for SSL pretraining and is compatible with both ABMIL and WSI-Transformer architectures. Extensive experiments across multiple datasets validate that the proposed approach achieves state-of-the-art performance in whole slide image (WSI) analysis tasks.", "questions": "Could you please furnish a concise description of Figures 7, 8, 9, and 10?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n-  The paper introduces a VQ-based feature distillation framework to address the fundamental trade-off between efficiency and representational richness in WSI foundation models. It may offer a scalable and effective solution for high-performing pathology FMs\n- The methodology is well-detailed, explaining the VQ learning, MSVQ, and the slide-level SSL objectives.\n- The paper is generally well-structured.\n\n\nWeaknesses:\n- The novelty of the work appears to be relatively incremental, as both Vector Quantization (VQ) and autoencoders are well-established techniques for information compression.\n- The ablation studies section lacks a more thorough and detailed discussion.\n- Certain symbols—such as those presented in Algorithm 1—require further specification and clarification.", "was_revised": true, "final_justification": "After reviewing all the comments and responses, I tend to keep my initial score.", "rebuttal": "We thank the reviewer for the insightful comments and appreciate the reviewer's concerns. \n\n>W1:The novelty of the work appears to be relatively incremental, as both Vector Quantization (VQ) and autoencoders are well-established techniques for information compression.\n\nWhile our work builds upon established techniques such as VQ and SSL, our core contribution lies in **unifying two longstanding challenges in WSI foundation model pretraining** through a coherent and scalable **multi-scale VQ framework**:\n\n**(1) Addressing the CLS token bottleneck via patch-level (16x16) VQ:**\n\nCLS token–based representations often discard fine-grained spatial information, leading to suboptimal performance on downstream pathology tasks. To mitigate this, we apply patch-level (patch size = 16) VQ/autoencoding to preserve local semantics while significantly reducing spatial dimensionality. This allows us to compress the token space without resorting to a single global embedding.\n\n**(2) Improving the SPT (Slide-level Pretraining) objective via tile-level (224x224) VQ:**\n\nTraditional masked autoencoding (e.g., MAE) focuses on reconstructing pixel inputs, which is infeasible for WSI-scale images due to GPU memory limits. We instead mask and predict **intermediate tile-level feature representations**, but just like MAE learn too many low-frequency feature of raw-pixel image [1], directly using feature-level targets also not align well with task-specific semantics.\n\nTo resolve this, we adopt a **tile-level (tile size = 224) VQ tokenizer** that discretizes feature targets into informative codebook entries. This approach enables more stable and effective slide-level representation learning compared to raw pixel or continuous-feature reconstruction.\n\n**Unified framework via Multi-Scale VQ:**\n\nRather than treating patch-level compression and slide-level representation learning as disjoint processes, we introduce a **Multi-Scale VQ architecture** that links both levels: patch-level VQ captures local detail, while tile-level VQ supports robust and interpretable global objectives. This unified view of representation compression and learning makes our approach both practically scalable and theoretically grounded for WSI tasks.\n\nIn addition, for addressing issue (1)—the CLS token bottleneck—we tackle a fundamental limitation in transformer-based slide-level models: **insufficient input rank**, which constrains their capacity to represent complex WSI data. As noted in [2], the rank of the attention matrix in a standard transformer is theoretically bounded:\n\n$$\n\\text{rank}(QK^\\top) \\leq \\min(\\text{rank}(Q), \\text{rank}(K)) = \\min(n, d),\n$$\nwhere n is the sequence length and d is the feature dimension. When using a single CLS token (or averaged tile-level CLS features), the actual rank is severely limited. Though in [2] authors try to improve the rank via local attention and positional embedding, the input rank limitation still exists. For example, we observe that in **Conch**, the effective rank of WSI embeddings using tile-level CLS is approximately **50**, and in **UNI**, it is around **400**—both significantly lower than the typical WSI sequence length (often exceeding **5,000** tokens).\n\n| Model | Rank (Mean ± Std) | WSI Length (Mean ± Std) | AUC Transformer |\n| --- | --- | --- | --- |\n| **Conch** | 53.6 ± 6.2 | 3356.1 ± 1840.6 | 0.861 |\n| **UNI** | 367.3 ± 109.7 | 3356.1 ± 1840.6 | 0.882 |\n| **Ours** | 765.4 ± 88.3 | 3356.1 ± 1840.6 | 0.910 |\n\nIn contrast, our method reconstructs spatial tokens via VQ-based representations, enabling **a much higher-rank representation** of the WSI content. We can also **increase the feature dimensionality (e.g., to 2048)**, which is more effective than simply up-projecting CLS features from 1024 using a linear layer.\nFor context, in standard ViT-Large (e.g., ViT-L/16 with 224×224 inputs), a 1024-d embedding is used for just 196 tokens. Large language models (e.g., LLaMA, Qwen ranging from 3~7B) employ embeddings of 2048–4,096 dimensions to represent typical question lengths. By analogy, for long-sequence AI applications such as WSIs, **our approach provides a scalable and rank-preserving alternative perspective**, and may generalize to other domains facing similar sequence-length bottlenecks.\n\n>W2: The ablation studies section lacks a more thorough and detailed discussion.\n\nSorry for the non-thorough discussion on ablation section due to the page limitation. Here we provide an detailed analysis and will include this in next version:\n\n**Performance vs. Complexity across  reduction methods**: \nFigure 5 illustrates the trade-off between performance and complexity for different token reduction methods, evaluated on the BRACS dataset. \n\nPCA: Reduces token dimensions by projecting them onto a low-rank linear subspace, preserving variance but potentially discarding discriminative details.The F1-score reaches about 0.70.\n\nAverage Pooling: Aggregates tokens by computing their mean, leading to compact representations but often oversmoothing critical local information. We have conducted pooling original 196=14x14 token features into 4x4 and 7x7 tokens, resulting in about 10 and 2 times token reduction. However, the result only show similar performance compared to original CLS token.\n\nCLS and overall average-pooling: as performed in virchow, the CLS and average-pooling token can complement a little to each other and resulting in around 1.0 point improvement.\n\nWe also performed token selection via CLS token attention top-k ranking. The selected top64 can reach a F1-score around 0.72 but is still too heavy in computation cost. The selected top16 can reach a F1-score of 0.70, can be seen as a trade-off, but lose too much on performance.\n\nOur proposed VQ-based method achieves the best balance, attaining the highest F1-score of 0.73 while significantly reducing token complexity. In contrast, other strategies demonstrate varying degrees of compromise between accuracy and computational cost:\n\nOur ablation shows that VQ not only improves computational efficiency but also maintains or enhances discriminative capacity, compared to conventional reduction techniques.\n\n>W3: Certain symbols—such as those presented in Algorithm 1—require further specification and clarification.\n\nThank you for pointing this out. We will revise Algorithm 1 to clarify the notation and added detailed definitions for all variables used in next version.\n\n>Q1:  furnish a concise description of Figures 7, 8, 9, and 10\n\nThanks a lot for this advise and we will furnish it in next version.\n\n\nRef:\n\n[1] Beit: Bert pre-training of image transformers\n\n[2] Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis"}, {"review_id": "rVw8KXJ2oq", "reviewer": "Reviewer_udpH", "summary": "The authors propose PathVQ, a vector quantization-based framework for compressing spatial patch tokens in WSI analysis. To reduce computational burden while preserving rich contextual information, they introduce a multi-scale VQ strategy and apply it in a SSL setting to support scalable pretraining and downstream tasks such as classification and survival analysis. Results show improved performance over existing aggregation methods across several benchmarks.", "questions": "Please address the concerns above.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper addresses patch token aggregation and model throughput, important problems in CPATH and working with WSI foundation models\n- Experiments cover multiple downstream tasks, including survival, and include comparisons with recent MIL, prototype based, and FMs\n\nWeakness + questions:\n- Using vector quantization to resolve the trade off between using CLS and patch token is not well justified. While vector quantization compresses patch feature dimensions, the amount of memory needed still scales linearly to the number of patch tokens - instead of using CLS, which would be constant. Other simpler methods, such as using a linear projector, could achieve similar memory savings while maintaining similar model performances. \n- The motivation behind using SSL with VQ is unclear. How does the model perform with SPT without vector quantization?\n- The effect of multi-scale VQ is also unclear. How does the number of scales and their resolution affect the model performance?\n- It is concerning that PathVQ underperforms prototype/slide-FM based methods available, especially in survival prediction, which are the more challenging and complex down-stream tasks", "was_revised": true, "final_justification": "The authors have sufficiently addressed my concerns regarding model motivation and evaluation.", "rebuttal": "We thank the reviewer for the insightful comments and appreciate the reviewer's concerns. We hope the reviewer can re-evaluating our paper based on this response.\n\n>W1.1: Using vector quantization to resolve the trade off between using CLS and patch token is not well justified. While vector quantization compresses patch feature dimensions, the amount of memory needed still scales linearly to the number of patch tokens - instead of using CLS, which would be constant. \n\nWe have compressed the 196 tokens with d=1024 into d=16 / d=32, the cost comparison to CLS token is 196*16/ (1*1024)=3.0625, which is acceptable. In our experiments, we find that 2~16x input cost compared to CLS are acceptable in training GPU memory cost and latency.\n\nAs for token scales problem, there may be some mis-understanding. We do not input all tokens into ABMIL or Transformer, we utilize a upsampling layer (like conv with 4 stride size) to convert the feature into dim 1024 or 2048, this resulted feature is of high rank, make it more easily for down-stream task anaysis.\n\nwe tackle a fundamental limitation in transformer-based slide-level models: **insufficient input rank**, which constrains their capacity to represent complex WSI data. As noted in [2], the rank of the attention matrix in a standard transformer is theoretically bounded:\n\n$$\n\\text{rank}(QK^\\top) \\leq \\min(\\text{rank}(Q), \\text{rank}(K)) = \\min(n, d),\n$$\nwhere n is the sequence length and d is the feature dimension. When using a single CLS token (or averaged tile-level CLS features), the actual rank is severely limited. Though in [2] authors try to improve the rank via local attention and positional embedding, the input rank limitation still exists. For example, we observe that in **Conch**, the effective rank of WSI embeddings using tile-level CLS is approximately **50**, and in **UNI**, it is around **400**—both significantly lower than the typical WSI sequence length (often exceeding **5,000** tokens).\n\n| Model | Rank (Mean ± Std) | WSI Length (Mean ± Std) | AUC Transformer |\n| --- | --- | --- | --- |\n| **Conch** | 53.6 ± 6.2 | 3356.1 ± 1840.6 | 0.861 |\n| **UNI** | 367.3 ± 109.7 | 3356.1 ± 1840.6 | 0.882 |\n| **Ours** | 765.4 ± 88.3 | 3356.1 ± 1840.6 | 0.910 |\n\nIn contrast, our method reconstructs spatial tokens via VQ-based representations, enabling **a much higher-rank representation** of the WSI content. We can also **increase the feature dimensionality (e.g., to 2048)**, which is more effective than simply up-projecting CLS features from 1024 using a linear layer.\nFor context, in standard ViT-Large (e.g., ViT-L/16 with 224×224 inputs), a 1024-d embedding is used for just 196 tokens. Large language models (e.g., LLaMA, Qwen ranging from 3~7B) employ embeddings of 2048–4,096 dimensions to represent typical question lengths. By analogy, for long-sequence AI applications such as WSIs, **our approach provides a scalable and rank-preserving alternative perspective**, and may generalize to other domains facing similar sequence-length bottlenecks.\n\n\n\n\n>W1.2 Other simpler methods, such as using a linear projector, could achieve similar memory savings while maintaining similar model performances.\n\nUsing AE (without VQ), even PCA compression  can obtain similar result, as showing in Figure 5, but our VQ goal is not only for compression. While our work builds upon established techniques such as VQ and SSL, our core contribution lies in **unifying two longstanding challenges in WSI foundation model pretraining** through a coherent and scalable **multi-scale VQ framework**:\n\n**(1) Addressing the CLS token bottleneck via patch-level (16x16) VQ:**\n\nCLS token–based representations often discard fine-grained spatial information, leading to suboptimal performance on downstream pathology tasks. To mitigate this, we apply patch-level (patch size = 16) VQ/autoencoding to preserve local semantics while significantly reducing spatial dimensionality. This allows us to compress the token space without resorting to a single global embedding.\n\n**(2) Improving the SPT (Slide-level Pretraining) objective via tile-level (224x224) VQ:**\n\nTraditional masked autoencoding (e.g., MAE) focuses on reconstructing pixel inputs, which is infeasible for WSI-scale images due to GPU memory limits. We instead mask and predict **intermediate tile-level feature representations**, but just like MAE learn too many low-frequency feature of raw-pixel image [1], directly using feature-level targets also not align well with task-specific semantics.\n\nTo resolve this, we adopt a **tile-level (tile size = 224) VQ tokenizer** that discretizes feature targets into informative codebook entries. This approach enables more stable and effective slide-level representation learning compared to raw pixel or continuous-feature reconstruction.\n\n**Unified framework via Multi-Scale VQ:**\n\nRather than treating patch-level compression and slide-level representation learning as disjoint processes, we introduce a **Multi-Scale VQ architecture** that links both levels: patch-level VQ captures local detail, while tile-level VQ supports robust and interpretable global objectives. This unified view of representation compression and learning makes our approach both practically scalable and theoretically grounded for WSI tasks.\n\n\n>W2: The motivation behind using SSL with VQ is unclear. How does the model perform with SPT without vector quantization?\n\nThe motivation behind using SSL with VQ is elaborated above. We did make some experiment on SPT without VQ but find limited performance. One of the main reason is that direct predict the input feature space may not benifit downstream task (just like MAE predict raw-pixels may not fit classification best). Differently, the VQ can work as a kind of prototyping, it is inner categories of pathology images. What's more, the VQ is performed offlined, which is better compared to DINOv2 using online tokenizer for categories/index prediction.\n\n| Method                                   | AUC | F1-score |\n|------------------------------------------|----------------|----------|\n| UNI+Transformer                                 | 0.882          | 0.678    |\n| UNI+Transformer + feature alignment SPT  | 0.894          | 0.725    |\n| UNI+Transformer+ VQ SPT                        | 0.910          | 0.754   |\n\n>W3: The effect of multi-scale VQ is also unclear. How does the number of scales and their resolution affect the model performance?\n\nDespite the unified goal (feature compression and SSL objective) of MSVQ, the MSVQ can enhance the VQ recovering performance since the image are borned with multi-scale characteristic.\nFor ablation study, we have showed the VQ learning cos-similarity on patch-scale only and multi-scale in Figure 7 (page 18). \nFor more scales ablation, we here show the cos-similarity and downstream tasks performance comparison:\n| Method                                   | Cos-similarity | F1-score |\n|------------------------------------------|----------------|----------|\n| 14×14 VQ                                 | 0.851          | 0.720    |\n| 1×1 + 14×14 MSVQ                         | 0.859          | 0.726    |\n| 1×1 + 2×2 + 14×14 MSVQ                   | 0.872          | 0.725    |\n| 1×1 + 2×2 + 4×4 + 7×7 + 14×14 MSVQ       | 0.881          | 0.730    |\n| MSVQ + SPT                               | 0.881          | 0.747    |\n\n\n>W4: It is concerning that PathVQ underperforms prototype/slide-FM based methods available, especially in survival prediction, which are the more challenging and complex down-stream tasks.\n\nThe comparison to slide-FM is not fair since they have used over 10 times of  prviate WSI data than ours (only public avaiable), but we can still exceed them in WSI classification tasks, which demonstrate the potential of our method, especially the feature compression has not been explored in these previous methods.\n\nFor survival prediction tasks using prototyping (Panther) or slide-FM (TITAN), a key difference lies in their use of a larger training batch size and Cox loss (as noted in lines 300–301). In contrast, following most prior works such as ABMIL-based methods, we adopt a batch size of 1 and use NLL loss, which typically results in performance that is 3–5 points lower compared to settings with larger batch sizes and Cox loss. \n\nHere, for more fair comparison to Panther and TITAN, we show the comparable result with similar trainig setting (larger batchsize and cox loss):\n\n| Method  | BRCA | CRC | BLCA | UCEC | KIRC| avg|\n|- |- |-|-|-|-|-|\n| Panther (prototyping)   | 0.758          |  0.665    |  0.612|  0.757 |  0.716| 0.702 |\n| TITAN (slide-FM)     | 0.713         | 0.710    | 0.657 | 0.789|  0.774| 0.729 |\n| Ours, bs=1+ NLL loss | 0.673          | 0.679    |  0.603 |   0.734 | 0.765| 0.691 |\n| Ours, bs=16+ cox loss | 0.727          | 0.698    |  0.629 |   0.742 | 0.775| 0.714 |\n\nWe would like to emphasize that survival prediction inherently exhibits an upper-bound in performance (e.g., clinicians typically achieve C-indices below 0.75 [3]). Taking into account that our model is trained on merely 1/10 of the data compared to TITAN, the achieved performance is competitive and indicates the efficiency and effectiveness of our method.\n\n\nRef:\n\n[1] Beit: Bert pre-training of image transformers\n\n[2] Rethinking Transformer for Long Contextual Histopathology Whole Slide Image Analysis\n\n[3] Automated model versus treating physician for predicting survival time of patients with metastatic cancer."}, {"review_id": "K3dTXZ9Pwf", "reviewer": "Reviewer_nSUP", "summary": "This paper focuses on improving existing ViT-based pathology foundation models (FMs). Most current approaches represent each patch using a [CLS] token and aggregate multiple [CLS] tokens to obtain the representation of a WSI. In contrast, this work proposes a novel method that distills the tokens of each patch onto a codebook using vector quantization (VQ). This reduces the original tile representation from $n \\times D$ to $C \\times d$, achieving a trade-off between computational cost and information preservation.", "questions": "1, How does PathVQ perform on more foundation models?\n\n2, How does the training cost of PathVQ compare to fine-tuning?\n\n3, What is the effectiveness of MSVQ? Are there ablation studies?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n- The paper provides an accurate summary of existing ViT-based FMs in computational pathology.\n- The idea of post-processing the outputs of large-scale models is interesting and potentially useful for improving efficiency.\n\nWeaknesses\n\n1. Figure 1 is confusing. The meaning of \"attntop64/16\" is unclear. I assume the left vertical axis shows the cosine similarity to measure information reconstruction (possibly reflecting the preservation of spatial information), in which the proposed VQ method performs well. However, the memory consumption on the right appears suboptimal compared to other methods. Moreover, it's unclear how well the proposed method performs on downstream tasks and whether it generalizes across different foundation models.\n\n2. Lack of detail in the MSVQ description. The paper does not sufficiently explain how aggregation is done across different scales, nor does it clearly describe the final output structure. This section is hard to follow. \nAdditionally, the pseudocode in Algorithm 1 is difficult to interpret due to undefined variables and functions. A more precise definition and breakdown are needed. Ablation studies related to this part are also missing.\n\n3. Insufficient experimental validation for VQ. From Figure 4, it seems that VQ is only applied to UNI tokens, which is not a particularly resource-intensive operation. The study should have evaluated more commonly used foundation models and included more patch-level datasets to better validate the generalizability of the VQ approach.\n\n4. Unclear comparison in training efficiency. Based on Figure 4, the PathVQ method is compared against fine-tuning the last block of the UNI model. While fine-tuning performs slightly better on CRC and VQ is slightly better on C17, the paper does not report training or computational cost, making it hard to assess the efficiency gains of PathVQ over conventional fine-tuning. Similarly, in the slide-level classification tasks, it is unclear why the authors did not report PathVQ's performance on larger datasets like Gigapath and TITAN.", "was_revised": true, "final_justification": "My major concerns regarding the writing and experimental settings have been addressed. I appreciate the idea of enhancing foundation models and will raise my score accordingly.", "rebuttal": "We thank the reviewer for the insightful comments and appreciate the reviewer's concerns. We hope the reviewer can re-evaluating our paper based on this response.\n\n\n>W1: Figure 1 is confusing. The meaning of \"attntop64/16\" is unclear. I assume the left vertical axis shows the cosine similarity to measure information reconstruction (possibly reflecting the preservation of spatial information), in which the proposed VQ method performs well. However, the memory consumption on the right appears suboptimal compared to other methods. Moreover, it's unclear how well the proposed method performs on downstream tasks and whether it generalizes across different foundation models.\n\nFigure 1 is a intuitive illustration of our motivation on why we need use more spatial patch tokens to replace CLS. The attn-top64/16 indicates another aspect/baseline by using less tokens (token reduction, implemented by ranking the CLS token attention weight on other patch tokens, we select the top64 and 16 most attented tokens) compared to ours less dimension. \nThe left vertical axis shows the similarity of information preserving of the curves. The **right vertical axis** show the memory/computational cost of **the bubbles**. So **the memory consumption on the right appears suboptimal compared to other methods** is a mis-understanding. As shown by the position of bubbles, our method only need a few extra cost compared to CLS but much better than token reduction. In addition, the bubble size indicates the downstream tasks performance (BRACS AUC), as explained in Figure 1 caption. \n\nWe feel sorry about the confusing and will make it more clear in next version. For different foundation models, it may make the figure more confusing, so we will included it in appendix in next version, some of its experimental ablations are included in following content.\n\n>W2: Lack of detail in the MSVQ description. The paper does not sufficiently explain how aggregation is done across different scales, nor does it clearly describe the final output structure. This section is hard to follow. Additionally, the pseudocode in Algorithm 1 is difficult to interpret due to undefined variables and functions. A more precise definition and breakdown are needed. Ablation studies related to this part are also missing.\n\nThe description on MSVQ can be found in line 139-162 and the figure 2. Sorry for the unclear elaboration. Here is a brief understanding of MSVQ in two-scale (tile-level and patch-level): 1) the tile-level VQ first encodes the average-pooling of 14x14=196 tokens (denoted as AVG, one token can be seem as same to CLS token). 2) upsampling the AVG VQ feature back to 14x14,  calculate the residual (difference) between the the upsampled feature map and original patch-level feature map, then the quantization is performed on the residuals. This can be further extened into multi-scale (1x1, 2x2, ..., 14x14). We will make it more clear in next version including a more precise definition and breakdown on algorithm, figure and text.\n\nFor ablation study, we have showed the VQ learning cos-similarity on patch-scale only and multi-scale in Figure 7 (page 18). \nFor more scales ablation, we here show the cos-similarity and downstream tasks performance comparison:\n| Method                                   | Cos-similarity | F1-score |\n|------------------------------------------|----------------|----------|\n| 14×14 VQ                                 | 0.851          | 0.720    |\n| 1×1 + 14×14 MSVQ                         | 0.859          | 0.726    |\n| 1×1 + 2×2 + 14×14 MSVQ                   | 0.872          | 0.725    |\n| 1×1 + 2×2 + 4×4 + 7×7 + 14×14 MSVQ       | 0.881          | 0.730    |\n| MSVQ + SPT                               | 0.881          | 0.747    |\n\n\n>W3 & Q1: Insufficient experimental validation for VQ. From Figure 4, it seems that VQ is only applied to UNI tokens, which is not a particularly resource-intensive operation. The study should have evaluated more commonly used foundation models and included more patch-level datasets to better validate the generalizability of the VQ approach.\n\nWe here list the performance of implementing VQ  with Conch[1] considering it is trained via image-text constrastive learning aligment, thus may showing some difference to image-only SSL like UNI/Gigapath/Virchow, etc. Based on  the result, we find that we can get similar performance gains from PathVQ. In next version, we will inlcude more Path foundation models and its evaluation on more dataset when combined with our VQ methods.\n\n| Method                                   | AUC | F1-score |\n|------------------------------------------|----------------|----------|\n| UNI+ABMIL                                 | 0.875          | 0.692    |\n| UNI+PathVQ+ABMIL                        | 0.902          | 0.730    |\n| UNI+PathVQ+ABMIL+SPT                        | 0.906          | 0.747    |\n| Conch+ABMIL                                 | 0.872          | 0.683    |\n| Conch+PathVQ+ABMIL                        | 0.897          | 0.718    |\n| Conch+PathVQ+ABMIL+SPT                        | 0.910          | 0.751    |\n\n\n>W4.1 & Q2: Unclear comparison in training efficiency. Based on Figure 4, the PathVQ method is compared against fine-tuning the last block of the UNI model. While fine-tuning performs slightly better on CRC and VQ is slightly better on C17, the paper does not report training or computational cost, making it hard to assess the efficiency gains of PathVQ over conventional fine-tuning. \n\nDispite the VQ pretraining cost, the overall fine-tuning cost of PathVQ compared to fine-tuning is slight. For slide-encoder inference, the storage cost is about 4 times and the training cost is about 2~3 times. However, during the overall WSI inference after model deployment in clinical diagnosis, the tile-level encoder inference takes most time cost. Generally, the tile-level encoder takes about 2 minites but the slide encoder only takes less than 1 seconds (ABMIL, using VQ takes about 2 seconds) and 3 seconds for slide-transformer while using VQ takes about 5 seconds.\n\nHere we list the training cost of 1 epoch for training BRACS, the main training speed is bottlenecked by the data loading during 2-stage training, so it does not impact too much during inference.\n\n| Aspect |PathVQ|Baseline |\n| - | -|-|\n| Fine-tuning Cost of BRACS 1 epoch  | ~28 seconds | ~13 seconds       |\n|Storage Cost (Slide Encoder)     | \\~4×   | 1   |\n| Training Cost (Slide Encoder)   | \\~2–3×   |1    |\n| Inference Time (Tile Encoder)      | \\~1-2 minutes   | Dominates WSI inference time     |\n|Inference Time (Slide Encoder)     | ~2s (ABMIL + VQ)      | <1 second (ABMIL),  Slide-level inference is fast|\n| Inference Time (Slide Transformer) | ~3s (with VQ)  | 1s (w/o VQ)   |\n\nFor tile/RoI level experiments, the main purpose is to show that our PathVQ can attain most information of original tokens, thus may showing similar performance to end-to-end fine-tuning considering that the foundation models' ability. Since end-to-end can not be performed in slide-level, the PathVQ or preserving all generated patch tokens may strongly improve the WSI performance compared to end-to-end learning.\n\n>W4.2 Similarly, in the slide-level classification tasks, it is unclear why the authors did not report PathVQ's performance on larger datasets like Gigapath and TITAN. \n\nFor mroe dataset evaluation like TITAN, we have conducted about 7 WSI level and 2 RoI level in this paper, I believe it is sufficient to demonstrate the effectiveness of our method. We here include 2 extra slide tumor classification task (for rebuttal length) and will validate more in next version.\n\n| Method | BCNB-ER | TCGA-LUAD TP53 |\n|-|-|-|\n|-|AUC|AUC|\n| UNI+ABMIL | 0.867          | 0.758    |\n| UNI+PathVQ+ABMIL                        | 0.891          | 0.775    |\n\n>Q3: What is the effectiveness of MSVQ? Are there ablation studies?\n\nFor MSVQ ablation, please see response in W2.\n\nFor MSVQ novelty perspective, our work builds upon established techniques such as VQ and SSL. Our core contribution lies in **unifying two longstanding challenges in WSI foundation model pretraining** through a coherent and scalable **multi-scale VQ framework**:\n\n**(1) Addressing the CLS token bottleneck via patch-level (16x16) VQ:**\n\nCLS token–based representations often discard fine-grained spatial information, leading to suboptimal performance on downstream pathology tasks. To mitigate this, we apply patch-level (patch size = 16) VQ/autoencoding to preserve local semantics while significantly reducing spatial dimensionality. This allows us to compress the token space without resorting to a single global embedding.\n\n**(2) Improving the SPT (Slide-level Pretraining) objective via tile-level (224x224) VQ:**\n\n..Just like MAE learn too many low-frequency feature of raw-pixel image, directly using feature-level targets also not align well with task-specific semantics.\n\nTo resolve this, we adopt a **tile-level (tile size = 224) VQ tokenizer** that discretizes feature targets into informative codebook entries. This approach enables more stable and effective slide-level representation learning compared to raw pixel or continuous-feature reconstruction.\n\n**Unified framework via MSVQ:**\n\nRather than treating patch-level compression and slide-level representation learning as disjoint processes, we introduce a **MSVQ** that links both levels: patch-level VQ captures local detail, while tile-level VQ supports robust and interpretable global objectives. This unified view of representation compression and learning makes our approach both practically scalable and theoretically grounded for WSI tasks."}]

Author rebuttal:

Meta-review: This submission tackles WSI data processing with an aim to address the trade-off between computational efficiency and information richness of representations. The developed foundation model has been applied and validated on downstream tasks of tumor classification, mutation prediction, and survival prediction. All the reviewers acknowledged the importance of the paper's research topic, and appreciate the workload of experiments. Initial weaknesses pointed out by the reviewers included inadequacy of technical contribution (e.g., VQ and autoencoders are existing), unclear motivation/justification of the method design (e,g., reason of using SSL and VQ), and details in writing (e.g., reporting training cost). The authors' rebuttal has successfully addressed the raised comments and sufficiently alleviated the major concerns. As a result, all reviewers positively rated Borderline Accept. The authors may consider to add the clarifications provided in the rebuttal into the final version.
Decision: Accept (poster)