MGPATH: A Vision-Language Model with Multi-Granular Prompt Learning for Few-Shot Whole Slide Pathology Classification
Authors: Anh-Tien Nguyen
Duy Minh Ho Nguyen
Nghiem Tuong Diep
Trung Quoc Nguyen
Nhat Ho
Jacqueline Michelle Metsch
Miriam Cindy Maurer
Daniel Sonntag
Hanibal Bohnenberger
Anne-Christin Hauschild
Venue: Accepted by TMLR ()

Abstract: Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels, hindering model generalization. This paper introduces a prompt learning method to adapt large vision-language models for few-shot pathology classification. We first extend the Prov-GigaPath vision foundation model, pre-trained on 1.3 billion pathology image tiles, into a vision-language model by adding adaptors and aligning it with medical text encoders via contrastive learning on 923K image-text pairs. The model is then used to extract visual features and text embeddings from few-shot annotations and fine-tunes with learnable prompt embeddings. Unlike prior methods that combine prompts with frozen features using prefix embeddings or self-attention, we propose multi-granular attention that compares interactions between learnable prompts with individual image patches and groups of them. This approach improves the model’s ability to capture both fine-grained details and broader context, enhancing its recognition of complex patterns across sub-regions. To further improve accuracy, we leverage (unbalanced) optimal transport-based visual-text distance to secure model robustness by mitigating perturbations that might occur during the data augmentation process. Empirical experiments on lung, kidney, and breast pathology modalities validate the effectiveness of our approach; thereby, we surpass several of the latest competitors and consistently improve performance across diverse architectures, including CLIP, PLIP, and Prov-GigaPath integrated PLIP. We release our implementations and pre-trained models at this https://github.com/HauschildLab/MGPATH.

Published in Transactions on Machine Learning Research (09/2025)
MGPATH: A Vision-Languagee Model with Multi-Granular Prompt Learning for Few-Shot Whole Slide Pathology Clas- sification
Anh-Ties Tien Nguyen
wwb.L.ggugen@uniigiiee
Institute for Predictive Deep Learning for Medicine and Healthcore, Justus Liebig University Giessen, Germany Department of Medical Informatics, University Medical Center Gottingen, Germany
Duy Minh Ho Nguyen"
Mar Planck Research School for Intelligent Systems (IMPRS-IS) Germany University of Stuttgart, Germany German Research Center for Artificial Intelligence (DFKT), Germony
Nghiem Tuong Diep" & Trung Quoc Nguyen & Daniel Sonntag German Research Center for Artificial Intelligence (DFKI), Germany
Nhat Ho
The University of Texas at Austin, USA
Jacqueline Michelle Metsch & Miriam Cindy Maurer & Hanibal Bohnenberger University Medical Center Gottingen, Germany
enme-chrrsstnn. http:ainnuauccchi@eeii-eeeee de Institute for Predictive Deep Learning for Medicine and Healtheore, Justus Lichig Unéversity Giessen, Germany
Anne-Christee Christin Hauschild
Reviewed on OpenReview: https: // openreview. net/ forum? id= dd=000811
Abstract
Whole slide pathology image classification presents challenges due to gigapixel image sizes and limited annotation labels, hindering model generalization. This paper introduces a prompt learning method to adapt large lision-language models for few-shot pathology classification. We first extend the Prov-GigaPath vision foundation model, pre-trained on 1.3 billion pathology image tiles, into language model by adding adaptors and aligning it with medical text encoders via contrastive learning on 923K image-text pairs. The model is then used to extract visual features and text embeddings from few-shot annotations and fine-tunes with learnable prompt embeddings. Unlike prior methods that combine prompts with frozen features using prefix embeddings or self-attention, we propose multi-granular attention that compares interactions between Icarnable prompts with individual image patches and groups of them. This approach improves the model's ability to capture both fine-grained details and broader context, enhancing its recognition of complex patterns across sub-regions. To further improve accuracy, we leverage (unbalanced) optimal transport-based visual-text distance to secure model robustness by mil perturbations that might occur during the data augmentation process. Empirical experiments on lung, kidney, and breast, pathology modalities validate the effectiveness of our approach; thereby, we surpass several of the latest competitors and consistently improve performance across diverse architectures, including CLIP, PLIP, and Prov-Giga Path integrated PI.IP. We release our implementations and pre-trained models at this https:/ httsss//iitttb.com Htt../o///HuuuillaaaaMMAAAAAAA
Equal sccond contribution. t Corresponding author

---

Published in Transactions on Machine Learning Research (09/2025)
1 Introduction
Whole slide imaging (WSI) NNaai et al. 2019) has become essential in modern pathology for capturing high-resolution digital representations of entire tissuc samples, enabling casier digital storage, sharing, and remole analysis (Pantanowitz et al. 2011). Unlike conventional methods that depend on examining slides under a microscope, WSI provides faster, detailed structural and cellular insights essential for disease diagnosis across multiple tissue layers, which is particularly valuable in cancer screening (Barker al., [2016] Chong et al. 2021p. Nevertheless, WSIs are massive images, often containing billions of pixels (Farahani al, 2015 Song al. 2023), making detailed annotations and analysis difficult and expensive. To tackle these challenges, machine learning techniques incorporating few-shot and weakly supervised learning have been developed (Madabhushi & Lee 2016 Li et al. 2023 Lin et al. 2023 Ryu et al. 2023 Shi et al. 2024). Among these, multiple instance learning (MIL) and vision vision-language language models (VLMs) have gained particular attention for their ability to effectively manage limited annotations and interpret complex whole-slide pathology imagos.
Pe Prompts
Prior Work
Classifier
Vision Encoder Er(-)
Patch Jevel Attention
Prompt- guided
Prediction
Prefix Tuning
WSI
Patches
slide feature
MGPATH
Pe Prompts
Classifier
Vision Encoder Ei(-)
t+1 +-2
Multi- Granular Attention
Prompt- guided
Prediction
WSI
Patches
slide feature
Figure 1: Unlike previous methods that add prompts at prefix positions or patch-level attention disrupting structural correlations our MGPATH framework integrates prompts both regional and individual patch levels (multi-granular attention). The tokens + 1, 2, and t + represent the spatially adjacent patches within larger tissue region (highlighted by the larger orange box), which together contribute to the model'ss 's diagnostic decision. In contrast, + 9 represents more distant or isolated patch froma different region.
In MIL (Ise et al. 2018 Xu et al. 2019a Li et al. 2023 Lin et al. 2023: Tang et al. 2023 Shi et al. 2023
each WSI is first divided into smaller patches or instances. These instances are extracted feature embeddings using pre-trained vision encoders before being grouped into 'bag', i.e., a whole slide-level representation for the entire WSI. The MIL model mainly focuses on learning ensemble functions to identify patterns in specific patches, contributing to the overall label prediction for each bag (c.g., cancerous non-cancerous), hence reducing the need for detailed annotations. Nonetheless, these methods often struggle to select relevant patches due to complex correlations and tissue variability (Gadermayr & Tschuchnig. 2024 Qu et al. 2024by.
To overcome those obstacles, VLMs (Lu et al., 2023 Huang et al. [0023] Ikezogwo et al., 2024 Shi et al.,
2024) have emerged as a promising solution, combining slide-level visual features with textual descriptions to enrich contextual understanding and support predictions in sparse data scenarios with approaches such as zero-shot lcarning (Xu ct al. 2024 Ahmed et al. 2024).. Specifically, VLMs incorporate multi-scale images 2024 Han al.] 2024). permitting the extraction of global and local WSI features al. different.
Shi et. al.]
resolutions. To adapt the pre-trained vision-language model efficiently, prompt learning (Zhou et al. 2022b 2024 is employed where learnable prompts are treated as part of the input text to guide the
Gao ct al.
Yao et al.] 2024) are integrated into feature embeddings Despite their strong classification performance across diverse tasks, these approaches still encounter certain limitations.
model, and contextual prompts {Li & Liang] 2021
using a self-attention mechanism [Vaswani 2017
2

---

Published in Transactions on Machine Learning Research (09/2025)
First, (i) adapting prompt learning with frozen visual features often neglects the hierarchical relationships among learnable prompts and the visual features they interact with specifically, the mulli-gromular atlention between prompts to individual patches and groups of patches. This limitation lessens the model's ability to capture interdependence across distinct scales - from fine-grained local features to broader contextual information, leading to less accurale comprehension of complex palterns in pathology images. Second, (ii) many VLMs rely on the CLIP architecture (Radford et al., 2021), ; which was not explicitly pre-trained on pathology images, thereby limiting its adaptability in few-shot settings, especially when the architecture is primarily frozen and prompt learning is applied. While there exist recent works that have incorporated PLIP (Huang et al. 2023), model pre-trained on 200k pathology image-text pairs curated from Twitter and showed siguificant improvements, an open question remains whether scaling pre-training to millions or billions of pathology-specific samples could further boost performance. Lastly, (iii) most VLM models for whole-slide pathology rely on cosine similarity to align visual and textual features. This metric, however, can struggle with multiple text descriptions for sub-regions (Chen et al., 2023) and with augmented data perturbations (Nguyen et al. 2024b), as it lacks the precision to capture fine-grained aliguments between varied image-text pairs.
In this work, we present MCPATH, a novel vision-language model (VLM) designed to address the challenges of whole-slide pathology classification. Unlike existing approaches, our method integrates large-scale pathology pre-training with multi-granular prompt learning to capture both fine- and coarse-grained tissue characterist ics. To enhance aligument between visual and textual representations, we adopt a parameter-effiiiet strategy that scales to millions of image-text text pairs without requiring full model retraining. Next, we incorporate optimal transport (OT) method to measure the distance between prompt-fused visual embedding and multiple text. prompts, providing flexibility in aligning heterogencous data distributions.
Extensive evaluations on three benchmark datasets and across diverse backbones demonstrate that MGPATH consistently outperforms both multiple instance learning (MIL) and recent VLM approaches, achieving significant gains in F1, AUC, and accuracy over stato-of-tho-art basclines. Our contributions are summarizod as follows:
We propose MGPATH, a parameter-efficient vision language model for whole-slide pathology that leverages large-scale pathology pretraining with lightweight adaptor-based alignment.
We introduce multi-granular prompt learning framework with hierarchical attention, enabling effective integration of fine- and coarse caase-grained contextual information from whole-slide images.
We incorporate optimal transport to align prompt-fused fuspt--uused visual embeddings and multiple textual prompts, providing robustness to align heterogeneous data distributions in few-shot scenarios.
We evaluate MGPATH across three pathology benchmarks and multiple backbones, consistently out- performing satte-of-the-art MIL and VLM baselines, with notable improvements in F1, AUC, and accuracy.
2
Related Work
2.1
Large-scale Pre-trained Models for Pathology
Recent advancements in large-scale pre-trained models for pathology can be broadly classified into two categories. Vision models, such Virchow (Iezzogoo et al. 2024), Hibou (Nechaev et al. 2024)) UNI (Chen) ct al. 2024), and Prov-GigaPath (Xu et al. 2024) leverage massive pathology image datasets to learn robust visual representations. Among these, Crrov-GigaPata stands out as the largest. model, trained on 1.3 billion pathology image patches, and excels in resolving complex tissue patterns at high resolution. On the other hand, vision-language models (VLMs) like PLIP (Huang et al., 2023) (trained 200K image-text pairs), CONCH [u et al.] 2024) (1.17M), or QUILTNET (Ikezogwo al. 2024b (IM), integrate visual and textual information to enhance contextual understanding and improve pathology slide interpretation. In contrast, our MGPATH combines the strengths of both approaches by using purumeter-efficient adaptor to link Prov-GigaPath (the largest. pre-trained vision encoder) with a text encoder from VLMs like PLIP or CONCH, leveraging both rich visual features and semantic textual embeddings. Although we use the PLIP text encoder in our experiments duc to its common use in basclines, the mothod can be extended to other larger pre-trained text models.
3

---

Published in Transactions on Machine Learning Research (09/2025)
2.2
Few-shot learning in WSI
MIL treats WSI bag of patches and aggregates these instances into a bag of features, with early methods using non-parametric techniques like mean max pooling. However, since disease-related patches are rare, these methods can overwhelm useful information with irrelevant data. To address this, attention-basee methods, graph noural Networks (GNNs), and Transformer-basee methods have been introduced (Lu et al.)
2021 Chen et al. 2021 Tsse el. al. 2018 Li et al. 2021 Shao et al.) 202T Zheng et al.) 2022 In contrast, VLMs have gained popularity through contrastive learning, aligning image-text pairs to enhance performance on a variety of tasks. While collecting large-scale pathology image-text pairs remains challenging, models like MI-Zero, PLIP, and CONCH have been trained on hundreds of thousands to over million pathology image-text pairs QLuu et al. 2023 Huang et al. 2023 Lu et al. 2024). Some approaches also integrate multi-maguification images and multi-scale text to mimic pathologists diagnostic processes, especially for detecting subtle abnormalities (Shi et al. 2024 Han et al. 2024). Our MGPATH extends on the VLMs strategy by further amplifying the benefits of using a large pre-trained pathology VLM model and introducing a new parometer-efficien multi-gronalar prompt learning to adapt these models few-shot settings.
2.3 Prompt Learning for Vision-Language Adaptations
Prompt tuning is proposed to transfer large pre-trained model task-specific downstream tasks and has shown strong results in multimodal models like CLIP. Rather than design a heuristic template, several methods like
CoDp [Zhou et al.] 2022bb, CoCoOp (Zhou et al. 2022a), or MaPLe (Khattak et al. 2023) among others (Rao et al.) 2022 Shhu et al.] 2022) have allowed models to determine optimal prompts from multiple perspectives,
such domain generalization (Ge et al. 2023 Yao et al. 2024). knowlodge prototype (Zhang et al. 2022b
2022). However, these approaches locus on natural images and do not address the unique challenges of whole-slide pathology images, which require multi-scale and structural contextual information. While a few current methods typically integrate prompts with frozen visual features via self-attention (Shi et al. 2024 Qu et al. 2024a/, these approaches might struggle with the complex relationships in WSTs. Our solution introduces multi-granular prompt learning, bridging aldention on both individual image patches and spatial groups to better align with the hierarchical structure of WSI data.
Li et al.) 2024 or diversity (Lu et al., 2022: Shu et al.
3 Methods
This section presents MGPATH, a parameter-efficiet vision language framework for whole-slide pathology classification. We bridge the Prov-GigaPath visual encoder (Xu ct al., 2024) - -pretrained on 1.3B pathology patches and the PLIP text encoder (Huang et al., 2023) -rrained on 200K image text pairs using lightweight adaptor modules optimized with contrastive objective. To strengthen vision-tess -exon-texttoe alignment. without updating either backbone, we collect an additional 923K pathology image text pairs from ARCH (Gamper & Rajpoot 2021), PatchGastricADC22 (Tsuneki & Kanavati 2022). and Quilt- 1M (Ikezogwo et al. 2024) and train only the adaptors via cross-alignment (Gao et al. 2024 Cao et. al.] 2024 This yields a large-scale aligned representation while remaining highly sarameter-efficenn To our knowledge, MGPATH is the first parfmeter-efficient VLM for pathology trained at this data scale (923K pairs) on top of a 1.3B-patch visual pretraining.
Next, we leverage these pro-trained models for fow-shot WSI tasks by introducing learning. First, descriptive lext prompls are generated for image patches at dual magnilication (owhhigh) using a frozen large language model (LLM) and augment them with multiple learnable prompt tokens per magnification, capturing diverse subregional patterns (Han et al. 2024 Shi et al. 2024 Qu ct al. 2024a). Then, we integrate the learnable prompts with frozen visual features at two granularities: (i) patch-level attention over all patches and (ii) region-level attention over spatial groups built from patch coordinates via message passing. As illustrated in Figure [] unlike previous methods, which typically score tokens independently, based on their individual relevance scores to the prompts (t + 3 and t 9). Our method explicitly incorporates both regional attention adjacent, semantically linked tokens (1 + 1, L + 2, ++ 3), while preserving patch-level attention for isolated yet discriminative tokens (t+ 9). This results in richer prompt-guided slide features that better capture the underlying pathology. Concretely, we represent image patches from each WSI as a spatial graph, using bounding- box coordinates to enable region-level aggregation through message passing along local connections. This spatial structure encoded as tokens within the Key- Value matrices, which interact with Query matrices derived from prompt embeddings. By directing
prompt
4

---

Published in Transactions on Machine Learning Research (09/2025)
attention from Query to Key- Bey-Value matrices across both patch and region levels, approach effectively captures hierarchical information, enriching feature representation and selectively emphasizing features across diverse tissue areas. Finally, instead of cosine similarity, we align prompt-fuse
visual summaries to class
prompts using an optimal transport (OT) distance (Nguyen et al.) 2021 Pham et al. 2020 Séjourné et al.,
2023 Chen el al. 2023 Dong al., 2023 Nguyen el al. 2024bb Zhan el al. 2021); ) which is robust to noisy
augmentations and partial text-region overlap both common in WSIs.
Figure 2 provides overview of the key steps in method. We next detail the adaptor-based vision-text alignment (Sec. 3.1 the construction of multi- magnification descriptive prompts (Sec. 3.2) gramularit--wwar
visual prompting (Sec. 3.3 and the OT-based alignment objective (Sec. 3.4
Mulli-Granular Attention
Attention
Dol-Producl
Vision Encoder
Ey(-)
Pb
H
Message Passing
17
Multi-Granular Attention
H
Pe
10x
T
r
A whole olids irage of Lung addnncareiine cinnna low and gien resolution zeveala arzegular tunor zeçiona distarring an jenent seractures, smooth
descriptive featuzes characterize both low and high sesclutionn xithin the shole slide linege? Please sumnarise ioe into a single paxograph. Question
(cless
3 1LM
Encoder
Optimal Transport
margine indiowting frvant venere,,,,
Br(())
destures prominent hucleoli, rich varcularity, and cellolar beterogenelty.
cleas
Contextual Prompt
Visual Descriptive Text Prompt
Figure 2: The pipeline of the proposed MGPATH. Low- and high-resolution image patches are processed with large language models to generate visual contextual descriptions (Section 3.2 Visual prompts are integrated with frozen features through multi-granular attention at both patch and group-of-patch levels 3.3 The final output is obtained by aligning visual and text embeddings using optimal transport (Section 3.4
3.1 Bridging Pathology Visual and Text Encoders
To leverage Prov-GigaPath's extensive pre-trained visual features for pathology, we implement lightweight adaptors that map image patch-level features to an embedding space aligned with the PLIP text encoder. These adaptors allow us train joint image-text representations with parameter efficiency by updating only the adaptor weights.
Given a set of collected pathology image-text pairs {(I, T 1,2.,, N (Sec. 4 we denote by Ey(.) be a pre-trained vision encoder from Prov-GigaPath, extracting patch-level feature, and Er(.) the pre-trained text. encoder from PLIP model. Given a batch size of B samples, the image and text embeddings are computed as x; = (I,) e gd. t, = (TT,) Rdd,, We then design two trainable adaptors Ar(.) and Ar(.). that maps (x,, t.) into the same hidden dimension Rd and minimizes the noise contrastive loss (Oord et al. 2018
(cos( Ar(t:))/T) Loo Br log Σ, exp (cos (x) (t(t))//7)
(1)
where cos(.) is the cosine similarity, and 7 denotes for temperature of the softmax function. For parameter efficiency, train only the adaptors Ay), Ar(.) while keeping the Prov-GigaPath visual encoder and PLIP text encoder frozen. After optimizing equation [ we use the outputs of the adaptors visual and text. embeddings for downstream tasks. Unless otherwise specified, we refer to this model as MGPATH
5

---

Published in Transactions on Machine Learning Research (09/2025)
3.2
Multi-Magnification Descriptive Text Prompts
To improve vision-language models (VLMs) for whole-slide image (WSI) analysis, designing effective text prompts is essential. Pathologists typically examine WSIs by first assessing tissue structures at low magnifi- cation before zooming in to analyze finer details such as nuclear size and shape. Inspired by this diagnostic workflow and the inherently multi-scale nature of WSIs, recent studies (Shi et al. 2024 Han et al., 2024) have roducee dual-scale visual descriptive lext prompls to guide VLMs, leading to signilicant improvements in classification performance. Building on this observation, we further extend and refine this strategy enhance model elfectiveness.
First, to ensure that generated prompts remain robust across varying WSI magnifications, we design shared prompts that combine both high- and low-scale descriptive elements, treating them as contextual embeddings. Specifically, we leverage the API of a frozen language model (GPT-4) and query it with the prompt as Figure
LLM Prompt
What visually descriptive features characterize {class name} both low and high resolutions within the whole-slide image? Please summarize into single paragraph.
Figure LLM template prompt.
In the above query, we replace {class name} by specific categories, for e.g., they are invasive ductal carcinoma (IDC) and invasive lobular carcinoma. (ILCC in the TCGA-BRCA dataset.
Second, at each low/high scale, rather than inserting single learnable text prompt of length K alongside a frozen contextual prompt from LLMs (Shi et al.) 2024 Han et al., 2024), we propose using M learnable prompls. This approach aims to caplure different sub-regions or structural features within each patch that might be overlooked with only a single prompt. Specifically, we define visual descriptive text prompts for both low and high resolutions as follows:
T = { (wwf)) (4) 10 -..(2u [LLM context) M
(2)
( = (luf Wo 110 (.!!" I |LLMMM context]
where [ufli,s E E M1 are KM trainable textual prompts for each resolution / E (1,h).
3.3 Granularity-aware Visual Prompt Learning
We propose to adapt visual prompts frozen features extracted by pre-trained vision encoder in the VLM model by taking into account the image patch level and spatial groupings of patches. Specifically, for each WST W, we denote by { (W()) w(h) } are representations of W at low and high maguification. We define bag of multiple instances of W as = (1(), ()))),
I(h) N x.NN.NxX..X with Nr, Na indicate the number of low and high-resolution image patches and No is the patch size. Following prior works
/(m) where [0]_ E N
(Shi et al. 2024 Tsse et al. 2018 Lu et al. 2021 Han et al. 2024), we employ non-overlapping sliding window technique to extract patches / from the WSI.
3.3.1
Patches-based Prompting
The frozen image encoder Ef(.) (or (,)) in case of NGPATH feature vector H (H() Nixd H(h) Na where d denotes the feature dimension. To effectively consolidate the extensive set of patch features into final slide-level representation, we introduce set of learnable visual prompts P. E RdN,xd, which facilitate the progressive merging of patch features in H (similarly for (²)) (Figure 2). In particular, we formulate p. as Query and take all features in H())
used map patches I into a
the Keys k.) and Values vi(() in in self-attention [Vaswani 2017). We then associate p. with patch features
P.
p P.P Normalize
SoftMax
V( p
+ Pu
(3)
Vd

---

Published in Transactions on Machine Learning Research (09/2025)
where Normalize(.) and Softmax(.) indicate the layer normalization operator and activation function respectively. Intuitively, equation 3 computes the correlations between the visual prompt Ps and all individual feature patches in H', subsequently grouping patches with high similarity to form fused prompt embeddings. However, since cancerous tissucs in WSIs often appear as large, contiguous regions of adjacent image patches, this motivales the introduction of spatial patch group-based attention.
3.3.2 Spatial Patch Group-based Prompting
We build spatial correlations for multiple instances in I by using their image patch coordinates inside each WST W. In particular, taken 7() = 7 1 .... (()) with their corresponding extracted features in HD - 1 (1
H No ".. ). we construct graph G = (V"), to caplure regional tissue structures where the set of vertices V(O 149 and E E 10,1) is the set of edges. Edges in E can be defined by linking inner patches to their K-nearest neighbors based on the coordinates. We define the node-feature embedding X H N that associates each vertex us with its feature node x; H
(1)
We next design a trainable messaag-passing network ge) based on the graph attention layer (GAT) Veličkovid ct al. 2017) to capture the feature representation of cach node and its local neighbors. The message passing of the GAT layer is formulated as:
exp (o t, + 00
aij ENENCHU(E) explo O, x!"" + +o θ: x0))
(4)
1 = 0,,
an 0 Σ JEN(()
where x; is aggregated features of x; with its local region after GAT layer, a() is the LeakyReLU activation function, (i) denote the neighboring nodes of the i-th node, azj are the attention coefficients and as 0e, .,, O are weight parameters of (.).).
After doing message passing by ge(-), the graph of patch-image features G()) is updated to GOY, where each node now represents super-node that encapsulates its corresponding feature region. We then squeeze all feature nodes in G(V) as a vector H and treat. them as another Keys K and Values vi for region-level features. Similar to equation 3 we associate prompt Pv with those group-level features:
Ko ())
Pegr = Normalize
SoftMax
(())
+ Pp
(5)
Va
The final output of our multi-gramular is computed as:
(1) = (1 - a) p + a P p
(6)
which interpolates between image patches and spatial patch groups
3.4 Optimal Transport for Visual-Text Alignment
Given descriptive text prompts (²) and (M) (equation 2 and visual prompt-guided slide features p and (h) (equation 6) for low and high resolutions, our goal is to maximize the similarity between slide and text embeddings for each class c. Rather than relying on cosine distance, as in prior works (Zhou et al. 2022b
Zhao et al. 2024 Qu et al. 2024a Singh & Jaggi 2020), we propose using optimal transport (OT)) based distance to capture a more nuanced cross-alignment between visual and text domains. Although OT has
been explored for prompt learning in natural images and multi-modal learning (Kim et al. 2023 Chen et al,
2023 Nguyen et al. 20224a Séjourné et al. 2023). we are the first to adapt for whole-slide imaging (WST), effectively handling the alignment of multi mulli-magniiiiation patches to capture rich structural details across scales.
Recap OT: Given two sets of points (features), we can represent the corresponding discrete distributions as follows:
N
11
Σ prOfes
g5 j=l
Σ
(7)
7

---

Published in Transactions on Machine Learning Research (09/2025)
where 6, and 6g represent Dirac delta functions centered at f and 9, respectively, and M and N indicate the dimensions of the empirical distribution. The weight vectors p {pt and q {qi j=1 lie within the M and N-dimensional simplex, respectively, meaning they satisfy Ei= = and Σj= Uj = =1. The discrete optimal transport problem can then be expressed as:
M N ΣΣ C.
T arg min
TERMAN
s.t. T1 N μ T
(8)
where T is denoted as the optimal transport plan, which is optimized to minimize the total distance between the two probability vectors, C is the cost matrix which measures the distance between fa and 9j- We then define the OT distance between i and V as:
d) CT"
(9)
Objective functions: Given the visual prompt-guided slide features p E Nyxd in equation and the descriptive text prompts ( in equation we compute the textual embedding for T()) as P (T((()) e
(1)
R Mxd.
We next denote T as the imput text prompts, (D.²) as the extracted textual embedding, and (p. (p() as the visual prompt-guidee slide features associated with class e. We then aim to minimize the distance between T!!)) and Pr indicated as doT (T!", pF (!) in the paper, by computing optimal transport distance
(p!") (1) ( Specifically, we treat (D!)) +F= (f and (p.*) + + = ( - and compute the cost matrix C as (1= F G) E MMM, which used to compute T in equation for estimate optimal transport distance defined in equation Following the same procedure, we can also
between Pi
(1) and
compute dor T(h)) (p!*)) (h)) at high-resolution image patches. Then, the prediction probability is written
exp(2 - Exe(unn dor (p.t) (K) )
Pe
(10)
Σ exp(2 - EnELL,y dor (T()) (p! (k)
where Ds controls contribution of each-resolution. Finally, we can train the model with the cross-entropy as: Lelass Cross(P, GT),
(11)
with Cross(.) be the cross-entropy and GT denotes slide-level ground-truth.
The details for solvers of equation and relaxed version with unbalanced optimal transport are presented in Scctions D and (D.1) in Appendix. Intuitively, using OT, in this case, offers several key advantages over cosine similarity. Pathology images exhibit complex, heterogeneous patterns that can be described from multiple perspectives. OT models these relationships as distribution, enabling more holistic alignment that handles variability and incomplete details while reducing noise from irrelevant prompts. This enhances the model's ability to generalize to unseen or complex disease cases.
4 Experiments
4.1 Settings
Datasets for contrastive learning. PatchGastricADC22 Tsuneki & Kanavati 2022 consists of approxi- mately 262K patches derived from WST of H&F-sttined gastric adenocarcinoma specimens, each paired with associated diagnostic captions collected from the University of Health and Welfare, Mita Hospital, Japan. QUILT-1M [Ikezogwo al.) 2024] includes approximately 653K images and one million pathology image- pairs, gathered from 1,087 hours of educational histopathology videos presented by pathologists on YouTube.
ARCH Gamper & Rajpoot
2021 is pathology multiple-instance captioning dataset containing pathology

---

Published in Transactions on Machine Learning Research (09/2025)
images at the bag and tile level. However, work focuses on tile-level images from all datasets for our contrastive training strategy. In total, we collected approximately 923K images from these datasets.
Downstrcam tasks. For the classification task, the proposed method was evaluated in three datascts from the Cancer Genome Atlas Data Portal (The Cancer Genome Atlas (TCGA]]: TCGA-NSCIC, TCGA- RRGA and TCGA-BRCA. We followed the ViLa-MIL et al. (2024) experimental settings for TCGA-NSCIC and TCGA-RCC, randomly selecting proportions for training, validation, and testing. For TCGA-BRCA, we adapted the training and testing slide ID from MSCPT (Han et al. 2024 The detailed description is included in the appendix section.
Evaluation Metrics. For all experiments, we report the Area Under the Curve (AUC), F1 score (F1), and accuracy (ACC) as evaluation metrics. Each experiment is repeated five times, and we present the mean and standard deviation across these runs to ensure robustness and reproducibility.
Implementation Details. We followed the ViLa-MIL preprocessing pipeline for tissue region selection and patch cropping. To integrate our attention module with CLIP50 and PLIP, extracted tile-level embeddings from their frozen vision encoders (1024-dimensional for CLIP50 and 512 for PLIP). We used the visual encoder of Prov-GigaPath to produce 1536-dimensional embeddings. To align it with PLIP's frozen text encoder, we developed two MLP-based adaptors that project both encoders into a shared feature space during a contrastive learning process, using datasets outlined in Section 4
To implement spatial attention, we use a Graph Attention Network (GAT) to model spatial relationships between WSI patches. Each tile-level embedding serves as node, connected to its left, right, top, and bottom neighbors, ensuring local spa- tial dependencies are captured. We then integrate spatial patch group- atten- tion Pe.gr into patch- attention Pv.p using equation G The hyperparameter a (0 to 1) controls the balance between spatial context and prototype-bar based guid-
Table 1: Comparison of methods on TCGA-BRCA with 16-shot settings.
Methods
# Param.
TCGA-BRCA
AUC
F1
ACC
Mase-pooling
197K
60:42:1135
56. 10::0.33..8 58
06.55:16.54
Mean-pooling
197K
66.6424.21
60.70 78
71.7333.59
ImageNot Pretrained
ABMIL Ilise
2018
461K
09.24=3.00
61.72+3.36
72.77+33.15
CLAM-SD
La
2121
600K
67.80
00.5115.01
72 72.46:14.36
CLAM-MB
660K
60.81::4.87
55.18114.9
61..114.19
TTamMM]) Stan el
2 2.54M
65.62 20
60.75:24.04
67 07.52:14.11 16
DSMIL
462K
66. 18-23.33000 08
67.52+1
RECT-MIL
lang ct al:
2024
2.63M
08.31 4.30
61.14
71.2148..44
CoOp Zhon
20122
337K
68.86.14.35
61.6112.44
71.08:13222
ance.
CoCoOp Zhog m A1
201
370K
69. 69:13:1::2
61.18:22.00 62
72.11:1..7
Metaprony (2100
21124
360K
60. 12:-4.44444
63 33-3014-4.38
74.66 7.5--
TOP Qu 22124
2.11M
09.74:3.14
63.30--4466 62
T4.41+5.27
In this paper, we leverage whole slide im-
ViLa--mm (Si at al.
2024
2.77M
72:26=66166 16
62.04::
16.11
ages at two resolution levels low magni- fication (5x) and high magnification (10x)
MSCPT
(Than
1.35M
7..56=44544
6.85
75.82422.88
MGPATH (VTT)
592K
74.9611.98
64.60:55.39
77.10=2.39
CONCH
a 2024
110M
15.11
65.633110.44
73.24188.6
to capture both tissue-level context and fine cellular detail. TThe whole slide im- ages are processed by Otsu's 's algorithm to remove all the non-tissuc regions. The cropped patch size is 256 x 256. All exper- iments were executed on a single NVIDIA A100 GPU node.
QUILT
Leesugwo at.
2024
68M
13.18:10.57
63.78228.72
73.26:10.18
Max Mazs-pooling
197K
68.50
61.50
11.5744.82
Pathology Pretrained
ARMIL [Tipp al
Mesut-pollig
2018
461K
197K
72.41
71.62=240 41
66. 06..8422.00 96
03.044-3.62
74. 19
74.00-14.38
CLAM-SB
La al
2121
600K
6..34=6.17
65.51133.29
16.16:4436
CLAM-MB
al
600K
33.41
1..1
17.88:12.00
TTammMIL (Saa el
2027
2 2.54M
71.95 01
67. 07.5012:6.00 00
17.0116.14
DSMIL
mt 2021
462K
71.44=2.77 72
14.48:11644
75526-22222
RRT-MML
Tang of Ai
2.63M
71.21=6.40
14.1
75.9245.10
CoOp Zhoi et al 21
337K
71.50=2..44 45
64.84::2.46 2.40
74.22245.02
CoCoOp Zayg et all puzzee
72.65224.65
66.63113355
66. 13.335
4.2
Comparison to State-of-the-Arr
Metappaame
(40on mi 2014
360K
74.862-4 25
65.08:11.81
88::332222
TOP Quu
2.11M
76.13=6.01
06.65
78-58445.00
We compare our MGPath with state-of-the- art multi-instance learning methods, in- cluding Maxpooling, Mean-
ViLa-b011 [581 dt ml
2224
2.77M
74.06: 4.62
86.03::1.81
76. 78.12444.88
MSCPT [Ha 2012
135M
75.55
67. 2.336622443
79.1112.63
MGPATH (PLIP)
592K
79. 79.02:26.0 43
68 25:t:4442
79.65=1.72
MGPATH (PLIP-
5.35M
$..36:11.85
33.13:3.49
19.56-14.77
pooling, ABMIL (Isee et al.
2018
CLAM
[Lu et al.) 2021), TransMIL
(Shao
el al. 2021), DSMIL (Li el al. 2022). GTMIL Zheng et al. 2022), DTMIL (Zhang el al. 2022a), RRT-MIL Tang et al. 2024) and IBMIL (Lin et al. 2023), and vision-lanme language methods, including CoUp (Zhou et al. 2022b).
CoCoOp (Zhou et al. 2022a). Metaprompt (Zhao et al.
2024). TOP (Qu et al.] 2024a). ViLa-MIL (Shi et al.
2024 CONCH (Lu et al.) 2024). Among these, QUILT 2024), MSCPT (Han et al.] 2024). QUILT (Ikezogwo et al.
and CONCH are foundation vision-language models (FVM) M).
9

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Comparison methods on TCGA-NSCLC, and TCGA-RCC datasets with 16-shots settings.
TCGA-NSCLC
TCGA-RCC
Methods
# Param.
AUC
F1
ACC
AUC
F1
ACC
Max-pooling
197K
53.0#6.0
45.8+8.9
53.3=3.4
67.4+99
46.7+11.6
54.1+4.8
Mean-pooling
197K
67.477.2
61.1+5.5
61.9=5.5
83.346.0
60.948.5
62.377.4
ABMIL Tsse et al.
2018
461K
15.9
56.8111.8
61.216.1
83.613.1
64.41:.22
65.7:4.77
CLAM-SB
660K
16.7
13.9
64.0=7.7
90.1+2.2
75.3+7.4
77.6+7.0
CLAM-MB Lu ct al.
202
660K
68.8
60.3111.1
63,019..
90.91-4.1
76.214444
78.614499
TransMIL
Shao ct al
2021
2.54M
64.2.18.5
57.5.16.4
59.715.44
89.4.5.6
73.0.17.8
75.3
DSMIL
al.
2021
462K
67.948.0
61.0=7.0
61.3=7.0
87.6+
7L.5+6.6
72.8+6.4
GTMIL
Zheng et al.
2022
N/A
66.0+15.3
01.1=12.3
63.849.9
81.1=13.3
71.1+15.7
76.1+12.9
DTMIL
Zhang et al.
2022a
986.7K
67.5
57.3
66.617.5
90.014.66
74.455.3
76.815.2
IBMIL
al. 2023)
N/A
69.247.4
57.4+8.3
66.926.5
90.5+t.1
75.1
77.2+4.2
ViLa-MIL (Shi et al. 2024
8.8M/47M
74.7#3.5
67.0+4.9
67.7+4.4
92.6+3.0
78.3+6.9
80.3+6.2
CONCH Lu et al. 2024
110M
89.46.110.2
78.519.31
78.78.19.1
88.08.14.99
78.2114.2
71.67.199 19.4
QUILT Tkezogwo et
2024
63M
19.66113.19
13.30
12.42:11..24
96.92:11.6
86.3411.56 78.46:5.55
MGPath (CLIP)
1.6M/39M
77.241.
70.9+2.0
71.0=2.1
92.1 + 2.8
76.5 + 6.2
81.7 = 2.9
MGPath (PLIP)
592K
83.6 + 4.5
4.5
76.41 + 4.8
76.5 + 4.8
94.7 1.6
78.6 4.9
83.6 + 3.5
MGPath (PLIP-G)
5.35M
93.02+2.99
84.64+4.75
84.77+4.67
98.2+0.31
88.33+3.41
91.72+1.74
4.3 MGPATH versions
We release four MGPATH variants. The ResNet-50 backbone CLIP-based models (MGPATH(CLIP)) for TCGA-NSCLC and TCGA-RCC, and a ViT-B/16 backbone (MGPATH(ViT) for -CCAA pure PLIP ver- sion (MGPATH (PLIP)) is also provided. Finally, MCPATH(PLIP-G) configurationccombiing the PLIP text, encoder with the vision encoder from Prov-GigaPath pretrained on large-scale pathology dataset-is employed for all datasets.
4.4 Results on Few-shot and Zero-shot Settings.
MGPATH with CLIP and PLIP backbones outperform several competitive MIL and VLM methods. As shown in Tables and our MGPATH(CLIP), (GPATH(ViT), and MGPATH(PLIP) outperforms several baseline models and achieves significant. improvements over other VLMs with similar architectures, such as ViLa and MSCPT, under 16-shots setting. The performance gain is particularly notable with the PLIP backbone. For example, on TCGA-BRCA, MGPATH (ViT) achieves an accuracy of 77. 77.10% compared to 75.82% for MSCPT and 75.01% for VILA-MIL. Additionally, with the PLIP backbone, MGP (GPATH(PLIP) surpasses MSCPT and ViLa-MIL by margins of approximately 3.5% to 5%.
MGPATH(PLIP-G) is a strong pre-trained VLM. By incorporating pathology-specific features from Prov-GigaPath (Xu et al. 2024), which pre-trained on 1.3 billion pathology images, and PLIP text encoder, MGPATH (PLIP-G) model demonstrated strong performance across multiple metrics on the TCGA-NSCLC, TCGA-RCC, and TCGA-BRCA datasets. When compared to other foundation VLMs such as CONCH and QUILT, our model consistently outperforms them. For example, achieve 3% improvement in AUC over CONCH on both the TCGA-BRCA and TCGA-NSCLC datascts.
MGPATH(PLIP-G) achieves competitive performance in zero-shot tasks. We evaluate the zero- shot capabilities of our model on three datasets and compare its performance against foundation VLMs such as CONCH, QUILT, and PLIP. The results, summarized in Table 3 show that the proposed VLM model achieves the best average performance across datasets, followed by CONCH and PLIP. This consistent top-tier performance across multiple benchmarks underscores the robustness and generalizability of our model.
Evaluating MGPATH(PLIP-) with fewer Samples. To assess the impact of number of training samples (number of shots K) on NGPATH(PLIP-G),
conducted experiments under 2-, 4-, 8-, 16-shots on TCGA-BRCA. Figure presents box plots each summaries five independent runs that track Fl scores. Overall, MGPATH MCPATH(PLIP-C) consistently surpassed the basclincs across 2, 4, 8, and 16 shots.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 3: Zero classification performance on TCGA-NSCLC, TCGA-RCC, and TCGA-BRCA datasets. Metrics reported include balanced accuracy (B-Acc) and weighted Fi-score (W-F1).
TCGA-NSCLC TCGA-RCC
TCGA-BRCA
Average
Zero-shot
B-Ace
W-F1
B-Ace
W-F1
B-Ace
W-F1
B-Ace
W-F1
QuiltNet
61.3
59.1
51.8
51.3
57.23
49.33
CONCH
80.0
79.8
72.9
69.1
64.0
61.2
72.3
70.03
PLIP
70.0
68.5
50.7
46.0
64.7
63.8
61.8
59.43
PLIP-G (Our)
72.7
72.6
81.3
81.4
70.0
69.9
74.67
74.63
80
Model
MGPATH(PLIP-G)
MSCPT
Metaprompt
70
TOP
ViLa-MIL
60
50
40
2
4
8
16
Number Shots
Figure 4: Fl scores of (GPATH(PLIP-G) under 2- 4-, 8-, and 16-shots training on TCGA-BRCA.
4.5 Ablation Studies
Multi-Granule Prompt Learning. In Table 4 we show the performance of MGPATH(CLIP) and (PPIIH))L with and without multi-granular two datasets TCGA-NSCLC and TCGA-RCC. It shows that using M-Gran improves the final performance of MGPATH and MGPATH (PLIP-G) on both datasets. We further investigate the impact of ratio when combining prototype-guided attention with graph spatial attention on TCGA-NSCLC. Table 5 shows that with a ratio of 0.2/0.8 (0.2 for spatial attention obtained from graph structure and 0.8 for prototype-guiedd attention), MGPATH (CLIP) achieves the highest performance.
PLIP enhanced Prov-GigaPath. We validate the use of Prov-GigaPath and PLIP under the following settings: (i) using full vision-languaa PLIP model; (ii) combining Prov-GigaPath with PLIP through the MLP layer which was pre-trained the large-scale dataset; (iii) integrating Prov-GigaPath with PLIP through adaptor layers which were randomly initialized; (iv) utilizing Prov-GigaPath and adaptor layer to map to the class output and only train MLP and last FFN layer of slide encoder; (v) only using Prov-GigaPath and an MLP layer to map to the class output and train MLP, the query matrix of last layer and the last FFN layer of slide encoder. Table shows that using Prov-GigaPath combined with PLIP (MGPATH (PLIP-G) boosts the final performance compared to only using PLIP (MGPAT((PLIP)) or Prov-GigaPath.
OT Alignment between Contextual Prompts. Table validates the use of OT in our (GPATH(CLIPP) TCGA-NSCLC and TCGA-RCC. We that using OT helps boost the performance of MGPATH(CLIP)
11

---

Published in Transactions on Machine Learning Research (09/2025)
compared to the use of cosine. It also shows that the number of prompt vectors depends on each dataset. In the appendix, we also run with another version using unbalanced optimal transport (UoT). We observe that both UoT and OT provide good alignment quality, with UoT slightly outperforming OT. However, this advantage comes at the cost of increased running time.
Table 4: Ablation studies on multi-gran Table 6: Ablation studies on adaptor learning for Prov-GiGa- ular (M-Gran) on the proposed models' Path and PLIP. PLIP-G denotes for mixed version between performance.
Table
Prov- Gigr-Git and PLIP.
TCGA-NSCLC
Confgurrations
NCGA-NSCLC
AUC
F1
ACC
Methods
Param.
AUC
Fl
ACC
MGPATH (CLIP)
77:24133
70392.0
71 11.022.
MGPATH (PLIP)
83.6+445
76,41446
76.5544.8
w/o M-Grau (CLIP)
74.6422
67.842
8425
I (PLIP-G)
6.85
98.022299
84.824477
647744.67
MGPATH (PLIP-G)
91.743.6
8L22:1 8144445
MGPNTH(PLIP4. Raodoen Adaplors
6.35M
91,4+422
82.866.7
830+56
w/o M-Gran (PIPP-G)
90.644.55
82.455.7 82.545.7
Prov GigaPath Tuning (MLP + last FFN)
4.7M
62.713.5
64.8815.3
528134
Prot-GigaPath Tuning (MLP Q-ViT)
5.8M
85.12669
74.17.5
75.816.1
TCGA-RCC
MGPATH (CLIP)
92.1+28 76.5+5.2 81 7..9
Table 7: Contribution of OT and multiple descriptive text prompts.
w/o M-Gran (CLIP)
91.613.5
72.316.4 80.21-14
MGPATH (PLIP-G)
98.1
85.7 89.9:2.0
w/o M-Gran (PLP-GG
98.1 85.01-4.0 89.3:3.0
Table 5: Ablation studies investigate the Methods
TCGA-NSCLC AUC
effect of the ratio combines spatial atten- tion (a) and prototype-guided attention
ACC
MGPATH (OT, text prompts)
77.211. 70.942.0 71.012.1
in (CPIIH(CLIP).
MGPATH (OT, 2 text prompts)
77.211. 70.942.0 71.012.1
MGPATH (Cosine, text prompts) 75.813.7
68.314.5 68.414.5
TCGA-NSCLC
TCGA-RCC
Configurations
AUC
F1
ACC
MGPATH (OT, 4 text prompts)
92.1+2288 76.5 81.742.9
a 0.2
77.2+1.3 70.9+2.0 71002.1
MGPATH (OT, text prompts)
92.1+:266 75.643.9 80.4+2.4
0.5
73.743.1 67. 422.6 67.8277
MGPATH (Cosine, text prompts) 91.812.8 75.914.3 80.512.6
a 0.8
72.255.2 66 415.5 66.8-5.2
Effect of Large Language Models. To further validate the impact of different LLMs on NGGPAH(PPIP--0 's performance, we conducted experiments with text prompts generated by various LLMs, including GPT-4o (OpenAT 2024). Grok-3 [xAT 2025), GPT-o3 (OpenAT, 2025), Gemini-2.. --rorr (Team et al. 2023 Deepseek-R1 (DeepSeek-A] 2025 and Mistral Medium 3 (Mistral Al 2024). The results in 16-shot set- lings on TCGA-BRCA are represented in Table g The experimental results demonstrated that (PPA-H consistently exceeded the baseline across all text prompt sources. The model achieved the best performance with Mistral Medium 3 (Mistral AT 2024), which produced the highost AUC (93.95 + 2.71), F1 score (88.53 + 4.29), and accuracy (88.63 + 4.26). This experiment demonstrated the robustness of the performance of the proposed method in various LLMs and highlighted the critical role precise visual descriptions on model performance enhancement.
Table 8: Ablation experiments of input text prompts from diffcrent LLM
TCGA-BRCA
Methods
AUC
F1
ACC
GPT-4oo O pen Al
2024
93.02 - 2.99
84.64 + 4.75
84.77 + 4.67
Grok-3 XAI 2025
93.07 = 3.56
85.50 + 4.53
85.58 + 4.51
GPT-o3 OpenAl 2025
93.87 3.31
86.19 + 5.28
86.29 5.26
Gemini- 2.5- pro Team et
2.5-pro
2023
93.17 = 3.12
86.72 + 3.50
86.80 3.48
Deepseek-R1 DeepSerk Al
2025
93.40 2.80
86.87 + 4.68
87.01 4.63
Mistral Medium Mistral AI
2024
93.95 + 2.71
88.53 + 4.29
88.63 + 4.26
12

---

Published in Transactions on Machine Learning Research (09/2025)
4.6 Discussion
While we demonstrate significant improvements in few-shot and zero-shot WSI classification across several settings, this paper does not explore other important challenges. For example, how can we scale the current. attention mechanism to handle even larger image patches (e.g., using Flash Attention (Dao et al.) 2022)), ), or extend the model from classification to tumor segmentation tasks (Khened et al. 2021). Additionally, the potential for extending GiGaPath to integrate with other large-scale VLM models, such as CONCH (Lu el. al. 2024), remains unexplored.
5 Conclusion
High-resolution WSI is crucial for cancer diagnosis and treatment but presents challenges in data analysis. Recent VLM approaches, which utilize few-shot and weakly supervised learning, have shown promise handling complex whole-slide pathology images with limited annotations. However, many overlook the hierarchical relationships between visual and textual embeddings, ignoring the connections between global and local pathological details or relying non-pathology-speiis pre-trained models like CLIP, Additionally, previous metrics lack precision in capturing fine-grained aligmments between image-text pairs. To address these gaps, (i) we propose MGPATH (PLIP-G), which integrates Prov-GigaPath with PLIP, cross-aligning them with 923K domain-specifii image-text pairs. (ii) Our multi-gramular prompt learning approach captures hierarchical tissue details effectively, (iii) while OT-based visual-text distance ensures robustness against data augmentation perturbations. Extensive experiments on three cancer subtyping datasets demonstrate that MGPATH and its variants achieve state-of-the-art results in WSI classification. We expect that this work will pave the way for combining large-scale domain-specific models with multi-gramular prompt learning and optimal transport to enhance few-shot learning in pathology.
6 Broader Impact Statement
MGPATH is developed to investigate the feasibility of applying vision-language models (VLMs) to pathology under few-shot learning settings, simulating scenarios with extremely limited training samples, such as rare discases. However, the model is not intended for clinical use as a diagnostic tool or decision support system. It is not designed to replace professional medical judgment or diagnosis.
7 Acknowledgement
This work is supported in part by funds from the German Ministry of Education and Research (BMBF) under grant agreements No. 01D2208A and No. 01KD2414A (project FAIPPaCT). The authors gratefully acknowledge the computing time granted by the KISSKI project. The calculations for this research were conducted with computing resources under the project kisski-ung-faing fairpact-2 The authors also acknowledge the computing time granted by the Resource Allocation Board and provided on the supercomputer Emmy/Gai /reet at NHR-Norl&@Göttineee as part of the NHR infrastructure. The calculations for this research were conducted with computing resources under the project nim00014.
The project "Development of an intelligent collaboration service for Al-based collaboration between rescue services and central emergency rooms" (acronym: CONNECT ED) is funded by the German Federal Ministry of Education and Research under grant number 16SV8977 and by the joint project KISSKI under grant number 1IS22093E.
The authors thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting Duy M. H. Nguyen. Duy M. H. Nguyen and Daniel Sonntag are also supported by the XAINES project (BMBF, 01 017W20005), No-IDLE project (BMBF, 01 01/W23002), and the Endowed Chair of Applied Artificial Intelligence, Oldenburg University. Anh-Tien Nguyen was member of the Ph.D. program 'Genome Science" International Max Planck Research School.
13

---

Published in Transactions on Machine Learning Research (09/2025)
References
Faruk Ahmed, Andrew Sellergren, Lin Yang, Shawn Xu, Boris Babenko, Abbi Ward, Niels Olson, Arash Mohtashamian, Yossi Matias, Greg S Corrado, et al. Pathalign: A vision-language model for whole slide images in histopathology. arXiv preprint arXiv:2406. 19578, 2024.
Dosovitskiy Alexey. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv: 2010.11929, 2020.
Jocelyn Barker, Assaf Hoogi, Adrien Depeursinge, and Daniel L Rubin. Automated classification of brain tumor type in whole-slide digital pathology images using local representative tiles. Medical image analysis, 90: 2016.
Qinglong Cao, Zhengqin Xu, Yuntian Chen, Chao Ma, and Xiaokang Yang. Domain-controlled prompt learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 936 944, 2024.
Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision-language models. International Conference on Learning Repre- sentations, 2023.
Richard Chen, Ming Y Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction us- ing patch-based graph convolutional networks. In Medical Image Compuling and Computer Assisted Intereention-MICCAA 2021: 24th International Conference, Strasbourg, France, September 97-October 1, 2021, Proceedings, Part VIII 24, pp. 339-349. Springer, 2021.
Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine, 30(3):850 862, 2024.
Shenghua Cheng, Sibo Liu, Jingya Yu, Gong Rao, Yuwei Xiao, Wei Han, Wenjie Zhu, Xiaohua Lv, Ning Li, Jing Cai, et al. Robust whole slide image analysis for cervical cancer screening using deep learning. Nature commanications, 12(1):5639, 2021.
Lenaic Chizat, Gabriel Peyré, Bernhard Schmitzer, and François-Xava Xavier Vialard. Scaling algorithms for unbalanced optimal transport problems. Mathematics of Compulation, 87(314):2563 2609, 2018.
Marco Cuturi. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural informalion processing systems, 26, 2013.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficien exact allention with io-awareness. Advances in Neural Information Processing Systems, 2022.
Ae. Deepseek-rl: Incentivizing reasoning capability in lmms via reinforcement learning, 2025. URL
https: //arxi http:////axivv..g/aaa//a/// attg//ab//501..1.. 12948
Shunjie
Dong, Zixuan Pan, Yu Fu, Dongwei Xu, Kuangyu Shi, Qianqian Yang, Yiyu Shi, and Cheng Zhuo Partial unbalanced feature transport for cross cross- modality cardiac image segmentation. IEEE Transactions on Medical Imaging, 1777) 2023.
Navid Farahani, Anil Parwani, and Liron Pantanowitz. Whole slide imaging in pathology: advantages, limitations, and emerging perspectives. Pathology and Laboratory Medicine International, pp. 23-33, 2015.
Michael Gadermayr and Maximilian Tschuchnig. Multiple instance learning for digital pathology: A review of the stato-of-the-ate limitations & future potential. Computerized Medical Imaging and Graphics, pp. 102337, 2024.
Jevgenij Gamper and Nasir Rajpoot. Multiple instance captioning: Learning representations from histopathol- ogy textbooks and articles. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 16549-16555 2021.
14

---

Published in Transactions on Machine Learning Research (09/2025)
Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581 595, 2024.
Chunjiang Ge, Rui Huang, Mixue Xie, Zihang Lai, Shiji Song, Shuang Li, and Gao Huang. Domain adaptation via prompt learning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
Minghao Han, Linhao Qu, Dingkang Yang, Xukun Zhang, Xiaoying Wang, and Lihua Zhang. Mscpt: Few- shot whole slide image classification with multi-scale and context-focused prompt tuning, arXiv preprint or WrXiv:2408.11503 2024.
Zhi Huang, Federico Bianchi, Mert Yuksckgomul, Thomas J Montine, and Jamos Zou. A visual-language foundation model for pathology image analysis using medical twitter. Nature medicine, 2023.
Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathology. Advances in neural information processing systems, 36, 2024.
Maximilian Tsse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pp. 2127 2136. PMLR, 2018
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition, pp. 19113-19122 2023.
Mahendra Khened, Avinash Kori, Haran Rajkumar, Ganapathy Krishnamurthi, and Balaji Srinivasan. A generalized deep learning framework for whole-slide image segmentation and analysis. Scientific. reports, 11 (1):11579, 2021.
Kwanyoung Kim, Yujin Oh, and Jong Chul Ye. Zegot: Zero-shot segmentation through optimal transport of text prompts. arXiv preprint arXi::2301.1117/ 2023.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. Interna- tional Conference on Learning Representations, 2017.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318-14328 2021.
Honglin Li, Chenglu Zhu, Yunlong Zhang, Yuxuan Sun, Zhongyi Shui, Wenwei Kuang, Sunyi Zheng, and Lin Yang. Task-specific fine-tuning via variational information bot blenekckck for weakly-supervised pathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7454-7463, 2023.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers, 2021.
Zhuowei Li, Long Zhao, Zizhao Zhang, Han Zhang, Di Liu, Ting Liu, and Dimitris N Metaxas. Steering prototypes with cooppt-tuning for rehearsal-free continual learning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2523 2533, 2024.
Matthias Liero, Alexander Mielke, and Giuseppe Savaré. Optimal entropy-transport problems and a new hellinger kantorovich distance between positive measures. Inventiones mathematicae, 211(3):969 1117, 2018.
Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang- Wen Chen. Interventional bag multi-instance learning on whole-slide pathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 18830-19839) 2023.
Seang-Wen
15

---

Published in Transactions on Machine Learning Research (09/2025)
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555 570, 2021.
Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard J Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zcro-shot transfer for histopathology images. In Proceedings of the IEEE/CVF conference on compuler vision and pollern recognition, pp. 19764-19775, 2023.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863 874, 2024.
Yuning Lu, Jianzhuang Liu, Yonggang Zhang, Yajing Liu, and Xinmei Tian. Prompt distribution learning. In Proceedings of the BEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 5215,, 2022.
Anant Madabhushi and George Lec. Image analysis and machine learning in digital pathology: and opportumities. Medical image analysis, 33:170-175, 2016.
Challenges
Mistral AI. Mistral medium 3, 2024. URL https: wwt/new/missraalaneeeea///
Mistral Al.
Dmitry Nechaev, Alexey Pchelnikov, and Ekaterina Ivanova. Hibou: A family of foundational vision transformers for pathology. arXiv preprint (1i::2406.00074, 2024.
Duy MH Nguyen, Nghiem T Diep, Trung Q Nguyen, Hoang- Le, Tai Nguyen, Tien Nguyen, TrungTin Nguyen, Nhat Ho, Pengtao Xie, Roger Wattenhofer, et al. Logra-med: Long context multi-graph alignment for medical rision-languag model. arXiv preprint 0rXiv:2410.02611 2024a.
Duy MH Nguyen, An T Le, Trung Q Nguyen, Nghiem T Diep, Tai Nguyen, Duy Duong-Tran, Jan Peters, Li Shen, Mathias Niepert, and Daniel Sonntag. Dude: Dual distribution-awarc context prompt learning for large vision-language model. Asian Conference on Machine Learning (ACMT.)): 2024b.
Huy Nguyen, Khang Le, Quang Nguyen, Tung Pham, Hung Bui, and Nhat Ho. On robust optimal transport: Computational complexity and barycenter computation. In Advances in NewrIPS, 2021.
Muhammad Khalid Khan Niazi, Anil V Parwani, and Metin N Gurean. Digital pathology and artificial intelligence. The lancet oncology, 20(5))e253-c226) 2019.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arxiv preprint aixiv: 1xXxx:8807.00 1807.03788 2018.
OpenAL. GPT-o3, 2024. URL http:://ppeaaa..coonnnneeelllaep OpenAl GPT-4o.
OpenAL GPT-03, 2025. URL http:://ggpaaa...oo/na/rrtttatac and on-dio4- Large lan-
guage model, April 16 version.
Liron Pantanowitz, Paul N Valenstein, Andrew Evans, Keith J Kaplan, John D Pfeifer, David Wilbur, Laura C Collins, and Terence J Colgan. Review of the current state of whole slide imaging in pathology. Journal of pathology informaties, 2(1):36, 2011.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trendso in Machine Learning, 11(5-6):355 607, 2019.
Khiem Pham, Khang Le, Nhat Ho, Tung Pham, and Hung Bui. On unbalanced optimal transport: An analysis of Sinkhorn algorithm. In International Conference on Machine Learning, pp. 7673-7682. PMLR, 2020.
Linhao Qu, Kexue Fu, Manning Wang, Zhijian Song, et al. The rise of ai language pathologists: Exploring two-level prompt learning for few-shot reakly-supervised whole slide image classification. Advonces in Neural Information Processing Systems, 36, 2024a.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Linhao Qu, Dingkang Yang, Dan Huang, Qinhao Guo, Rongkui Luo, Shaoting Zhang, and Xiaosong Wang. Pathology-knowlekde enhanced multi-instance prompt learning for few-shot whole slide image classification. arXiv preprint orXic:2407.1081, 2024b.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu. Denseclip: Language-guided dense prediction with context-aware prompling. In Proceedings of the IEEE/CVE conference on computer vision and pattern recogmition, pp. 18082-18091, 2022.
Jeongun Ryu, Aaron Valero Puche, Jae Shin, Seonwook Park, Biagio Brattoli, Jinhee Lee, Wonkyung Jung, Soo Tek Cho, Kyunghyun Paeng, Chan-Young Ock, et al. Ocelot: overlapped cell on tissue dataset for histopathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1-902-239122 2023.
Thibault Séjourné, Gabricl Peyré, and Trangois-Xavier Vialard. Unbalanced optimal transport, from theory to numerics. Handbook of Numerical Analysis, 24:407-477 471, 2023.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer- based correlated multiple instances learning for whole slide image classification. Advances in newral information processing systems, 34:2136 2147, 2021.
Jiangbo Shi, Lufei Tang, Yang Li, Xianli Zhang, Zeyu Gao, Yefeng Zheng, Chunbao Wang, Tieliang Gong, and Chen Li. A structure-aware hicrarchical graph-based multiple instance learning framework for pt. staging in histopathological image. IEEE Transactions on Medical Imaging, 42(10):3000 3011, 2023.
Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu. Vila-mil: Dual-scale vision-language multiple instance learning for whole slide image classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11248-11258, 2024.
Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, and Chaowei Xiao. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Newral Information Processing Systems, 35:14274 14289, 2022.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 04:20045-22055 2020.
Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood, Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering, 1(12):930-9)) 2023.
Wenhao Tang, Sheng Huang, Xiaoxian Zhang, Fengtao Zhou, Yi Zhang, and Bo Liu. Multiple instance learning framework with masked hard instance mining for whole slide image classification. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp.
2023.
Wenhao Tang, Fenglao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang, and Bo Liu. Feature re-embedding: Towards foundation model-level performance in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11343-11352, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. Xiv preprint ar Xix:22312. 1 805, 2023.
The Cancer Genome Atlas (TCGA). Genomic Data Commons Data Portal (GDC). https://portal.gtt cancer gwt/pprojects/TTGA.. Bwweeeettt///GGBBBRCCACCA Accessed 07 Jul. 2023.
Masayuki Tsuncki and Fahdi Kanavati. Inference of captions from histopathological patches. In International Conference on Medical Imaging with Deep Learning, pp. 1235- 1250. PMT.R, 2022.
17

---

Published in Transactions on Machine Learning Research (09/2025)
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv preprint arXiv:1710.1000 2017.
Cédric Villani et al. Optimal transport: old and new, volume 338. Springer, 2009.
XAI. Grok-3, 2025. URL https://x.ai/news///
xAI Grok-3.
Gang Xu, Zhigang Song, Zhuo Sun, Calvin Ku, Zhe Yang, Cancheng Liu, Shuhao Wang, Jianpeng Ma, and
Wei Xu. Camel: A weakly supervised learning framework for histopathology image segmentation. Proceedings of the IEEE/CVF International Conference on computer vision, pp. 10682-10691, 2019a.
Hanwen Xu, Naoto Usuyama, Jaspreet Bagga, Sheng Zhang, Rajesh Rao, Tristan Naumann, Cliff Wong, Zelalem Gero, Javier González, Yu Gu, et al. whole-slide foundation model for digital pathology from real-world data. Nature, pp. 8, 2024.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? International Conference on Learning Representations, 2019b.
Hantao Yao, Rui Zhang, and Changsheng Xu. Tep: Textual-based class-aware prompt tuning for visual- language model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23438 23448, 2024.
Fangneng Zhan, Yingchen Yu, Kaiwen Cui, Gongjie Zhang, Shijian Lu, Jianxiong Pan, Changgong Zhang, Feiying Ma, Xuansong Xie, and Chunyan Miao. Unbalanced feature transport for exemplar-base image translation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
15028 11038,, 2021.
Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp- 18802-18812, 2022a.
Yue Zhang, Hougliang Fei, Dingcheng Li, Tan Yu, and Ping Li. Prompting through prototype: A prototype- based prompt learning on pretrained languaeee models. arXiv preprint arXim:2210.108: 2022b.
Cairong Zhao, Yubin Wang, Xinyang Jiang, Yifei Shen, Kaitao Song, Dongsheng Li, and Duoqian Miao. Learning domain invariant prompt for vision-language models. IEEE Transactions on Image Processing, 2024.
Yi Zheng, Rushin H Gindra, Emily J Green, Eric J Burks, Margrit Betke, Jennifer E, Beane, and Vijaya B Kolachalama. A graph-tran for whole slide image classification. IEEE transactions medical imaging, 41 41(11):3003 3015, 2022.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision- language models. In Proceedings of the IEEE/CVP conference on compuler vision and pullern recognilion, pp. 19816-16820 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for lision-language models. International Journal of Computer Vision, 130(9))2337-234) 2022b.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Appendix
A Description of Dataset Splitting
TCGA-BRCA. This dataset contains 1056 whole slide images breast invasive carcinoma. To conduct fair experiments, we adaptod training and testing slides provided by the GitHub repository of MSCPT (Han et al. 2024). In the MSCPT setup, 20% of the dataset was allocated for training, while the remaining 80% (833 slides) served as the test set. A fixed set of 16-shot WSIs was randomly sampled from the training set. Additionally, MSCPT specified the exact training and testing slides used in its experiments. However, there are 35 slides in which we got errors in the pre-processing steps; thus, we replaced those slides with the other ones (same
number of WSI per class) downloaded from Cancer Genome Atlas (TCGA) Data Portal (GDC) (The Cancer Genome Atlas (TCGA)).
TCGA-RCC TCGA-NSCLC. We adopt the same data splitting as in ViLa-MIL (Shi et al. 2024 using 16-shot samples for training in each dataset. For testing, 192 samples were used for TCGA-RCC and 197 samples were used for TCGA-NSCLC.
A.1 Other hyper- paramerr
For all experiments, we trained MGPATH with the Adam optimizer with a learning rate of 9 x 10-6 and a
weight docay of 1 x 10-5 fine-tune all versions of MGPATH presented in Tables and 2 The training process was conducted for a maximum of 200 epochs, with batch size set to 1. The best checkpoints are picked based validation performance with Fl score.
A.2 Baseline Setups
TCGA-BRCA: The baselines in Table are sourced from the MSCPT (Han et al. 2024 paper, where various methods are evaluated using two backbones: Vision Transformer (ViT) (Alexey 2020) from the CLIP model (top section of Table [ and PLIP (Huang et al. 2023 (bottom section of Table 1. In this context, we
introduce three variations of (V1T,H(V (PLIP, and MGPATH (PLIP-G), where all versions utilize frozen vision and text encoders.
TCGA-RCC & TCGA-NSCLC: The baselines in Table 2 are adapted from ViLa-MIL (Shi et al., 2024 where methods employ ResNet-50 from the CLIP model as the primary backbone. We present the results using three architectures: (CPATH(CLIP), (MPPIH)(PPI), and KGPATH(PLIP-G).. With ResNet-50, we follow the ViLa-MIL approach by training the text encoder and reporting performance for this setup. To assess efficiency, provide the total parameter counts for both ViLa-MIL and MGPATH, considering scenarios with frozen backbones and trainable text encoders. For (CPATH(PIPP) and (GPATH(PLIP-G), all visual and text encoders are kept frozen.
CONCH & QUILT: We download the pre-trained weights of these foundation models and adapt them for zero-shot evaluation on TCGA datasets following the authors' guidelines from (Lu et al. 2024), which randomly sample 75 samples for each class. For few-shot settings, since official implementations are not provided, we ialize the models with their pre-trained weights and allow fully fine-tune the text encoder and evaluate on the same subsets that use for other baselines. While CONCH provides prompts for the datasots in its publication, QUILT docs not. Therefore, we fine-tune the model using CONCH's prompts and our own generated prompts for QUILT.
B Impact of PLIP enhanced Prov-GigaPath
Figure presents the AUC curves for three randomly selected folds, illustrating the impact of Prov-GigaPath model performance. The results show that integrating Prov-GigaPath leads to consistently higher AUC values across all folds, demonstrating its effoctiveness in enhancing the proposed model. Notably, the improvements are most pronounced during the carly training epochs, where the model converges faster and achieves more stable performance compared to the baseline. This suggests that Prov-GigaPath facilitates better feature extraction and generalization, ultimately leading to more robust model.
19

---

Published in Transactions on Machine Learning Research (09/2025)
Figure 5: AUC performance comparison over epochs for PLIP (blue) MGPATH(PLIP-G) (red). MGPATH achieved achieving more stable and higher values, particularly in the carly epochs.
Table 9: Comparison of message passing algorithms MGPATH, including GAT-CONV, Graph Isomorphist sm Network (GIN), and Graph Convolutional Network (GCN). Performance is evaluated on the TCGA-NSCLC dataset using 5-fold cross-validation.
0.94
0.92
TCGA-NSCLC
0.88
Configurations
AUC
F1
ACC
0.86
MGPATH (GAT CONV)
77.2-13
70.9-2.0 71.012.1
0.84
MGPATH (GIN)
77.1=2.9
69.8+3.9 69.9+4.0
0.82
MGPATH (GCN)
75.1 67. 2..612.5 67.1
0.80
0.78
PLP
(PLIP GigaPath)
15 Epoch
C Ablation Study on Message Passing Networks
In Table g we evaluate the performance of (CLIP using the Graph Attention Network (GAT-CONV) against alternatives like the Graph Isomorphism Network (GIN) (Xu et al.) 2019b) and the Graph Convolutional Network (GCN) (Kipf & Welling 2017). The results show that MGPATH (GIN) achieves comparable performance to MGPATH (CL) (CLIIH(CLIP) (GAT-CONV) significantly outperforms the GCN-based version, likely due to GAT's ability to dynamically assign attention weights to neighboring image patches, enabling it to prioritize the most relevant neighbors for each node.
however, with higher variance. In contrast,
D Additional Details on Optimal Transport Distance
The following paragraphs will provide detailed information on the implementation of (un-balanced) optimal transport (OT) (Villani et al., 2009 Peyré et al. 2019 and specifically the alignment prompt-guided visual-text distances MGPATH.
D.1 OT Formulation and Efficient Solver
Given two set of fcature embeddings F = { [M e Rxd and = {g E Nxd, we can represent them as two discrete distributions l and by:
M 11
N Σ #=1
Σ Prog.
9,5g.
(12)
where of, and 0g, represent Dirac delta functions centered at F and G, respectively and the weights are elements of the marginal p = Ms and = {4: 1 and can be selected as the uniform weight with MM DE = E
Then we can compute the distance between F and G through ie and (Eq.(9)) as dor(H, (T", C).
(13)
where
M
T arg min TERMXN
ΣΣ Cij
s.t. NIN = M
(14)
with C E MxN is the cost matrix which measures the distance between f: E la and E v.
Because directly solving Eq equation 14 is high-compptttiional costs (O(n² log n) with n proportional to M
and N), Sinkhorn algorithm (Cuturi 2013) proposed approximate solution by solving regularized problem:
20

---

Published in Transactions on Machine Learning Research (09/2025)
M
T arg min TERMXN
ΣΣ ==
T Cij AH(T)
s.t. TI==M I
(15)
where H(T)= = Eij Tú TogTo be an entropy function and À > 0 is the regularization parameter. The optimization problem in Eq. equation 15 is strictly convex, allowing us to achieve a solution efficiently with lewer ilerations as outlined below:
T diag(a' ) exp(-C/A)diag(t
(16)
where t is the iteration and a' // exp C/X)b' and b' / exp(-C/A)a', with the initialization 6 1. In our experiments, we used t 100 and À 0.1 based on validation performance.
D.2 Relaxed Marginal Constraints with Unbalanced Optimal Transport
Due strict marginal constraints in Equqution 14 optimal transport may be unrcalistic in real-world scenarios where data distributions
noisy, incomplete, unbalanced. The Unbalanced Optimal Transport. (UoT) (Chizat et al. 2018 Liero et al. 2018) addresses this challenge by relaxing the marginal constraints, allowing for partial matching through penalties on mass creation destruction. In particular, UoT solves
M N
T = arg min
ΣΣT C. - (T(T)
(17)
TERMXN ==1==1
+ (L( 11)++ + 1
here, P1 and P2 represent the marginal regularization parameters, and KL (PI()) denotes the Kullback-Leibler divergence between two positive vectors. Similar to the classical OT formulation, there are solvers based the Sinkhorn algorithm that can address Eq. equation 18 (Pham et al. 2020). However, these solvers typically require more iteration steps to converge to optimal solutions due to the added complexity introduced by the relaxed marginal constraints.
E
Unbalance Optimal Transport (UoT)
Table 10: (GPATH((LIP) performance and rumning time (in second) comparison between OT and UoT.
TCGA-NSCLC
Methods
AUC
F11
ACC 1
Time (s) $
(GPATH(CLIP) (OT, 4 text prompts)
76.2+2.2
69.0
69.3+2.8
1482
(GPATH(CLIP)) (UoT, 4 text prompts) 77.0
70.243.4
70.4
3260
TCGA-RCC
MGPATH(CLIP) (OT, 4 text prompts) (CPIIH(CLIP) (UoT, text prompts) 92.8+2.4 76.8 82.4 2.4
92.1+2.8 76.5+5.2 81.7+2.9
1451
3049
To conduct comparative evaluation of the performance of MGPATH using optimal transport versus unbalanced optimal transport (Section D.2 given the more flexible constraints in UoT, we conducted additional experiment. To be specific, we test on the TCGA-NSCLC and TCGA-RCO datascts with NGPATH using a 4-text-prompl setting. Table 10 presents our findings where the running time is computed as seconds of average across five-folds. The results show that UoT outperforms OT with an approximate 1% improvement across all metrics. However, UoT is approximately times slower than OT. This increase attributed to the added flexibility and complexity introduced by relaxing the marginal constraints in the UoT formulation. Given this trade-off, we choose OT the main distance in MGPATH and leave the UoT version for further evaluation. It is also important to know that OT formulation leverages approximate solutions through the regularized formulation (Eq.equation 15 and produces smoothed optimal mappings T", which can implicitly help the model adapt to perturbations like UoT.
21

---

Published in Transactions on Machine Learning Research (09/2025)
F Ablation Studies on Data Augmentation
To evaluate the effect of data augmentation model performance, conducted experiments comparing MGPATH(PLIP-G) with and without augmentation. In addition, we trained a variant, MGPATH (PLIP-G) (COS), where the optimal transport module in the MGPATH architecture was replaced by cosine similarity. All experiments were carried out on the TCGA-NSCLC dataset under the 16-shot setting.
As illustrated in Table the cosine similarity configuration exhibits unstable behavior, with a sharp decline in F1 and ACC when data augmentation applied, indicating difffcully in adapting to helerogeneous data in the training data. In contrast, the optimal transport configuration not only achieves the best overall results but also remains consistently strong across augmented and non-augmented settings.
Table 11: Impact of data augmentation the performance of MGPATH using cosine similarity and optimal transport configurations the TCGA-NSCLC dataset.
TCGA-NSCL
Method
AUC
F1
ACC
(CPATH(PLIP-C)CCOS)
91.13 1 3.20
82.88 1 3.55
83.05 + 3.53
w/o augmentation
92.92 1 3.88
74.35 1 23.40
78.17 1 15.93
MGPATH(PLIP-G)
93.95 2.71
88.53 - 4.29
88.63 - 4.26
w/o augmentation
91.94 + 5.33
84.49 + 6.02
84.67 + 5.98
G Training and Inference Efficiency
To assess the computational efficiency of (GPATH(PLIP-G) based on OT and cosine similarity, MGPATH(PLIP-G) (OT) and MGPATH (PLIP-G) (COS), respectively. We conducted an additional experiments. To concreat, we evaluate training and inference using the metries Throughput (samples/se sec) on the dataset. TCGA-NSCLC with text prompts generated by Mistral Medium 3 (Mistral Al 2024 in 16-shots setting.
Table 12 shows that although the optimal transport configuration requires more computation and yields lower training and inference throughput compared to cosine similarity, it consistently achieves higher accuracy. This highlights the strength of optimal transport improving model generalization, as it is able to leverage complex data, distributions more effectively despite the additional computational overhead.
Comparison average Training and Inference Throughput (samplcs/scc) on a single NVIDIA A100 GPU with only batch size 1 for MGPATH (PLIP-G) (OT) and MGPATH (PLIP-G) (COS).
Table 12:
Training
Inference
Configuration
Throughput (samples/sec))
Throughput (samples/sec)
ACC1
(CPAAHHPPPP-G)CCOS))
16.00 0.29
31.59 0.32
83.05 3.53
(GPPT
10.67 2.52
19.05 5.59
88.63 4.26
22


Reviews:
[{"review_id": "pc5ePtTs98", "reviewer": "Reviewer_xKaA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The lightweight adaptor training strategy is efficient and elegant.\n- The idea of multi-granular prompt learning is insightful and novel.\n- The experiments are comprehensive and carefully designed.\n\nWeakness\n- Although the presentation is generally clear, some of the abbreviations and notations need clarification. I understand that MGPath has different versions depending on the CLIP backbone choices. Yet, can the authors clearly explain what MGPath, MGPath (ViT), MGPath (PILP), and MGPath (PLIP-G) are, preferably list all the different versions together, and explain their differences?\n- It's not very clear to me how the few-shot classification experiments are conducted on the comparative methods, especially for the vision-language models CONCH and QULTT. Are they fine-tuned with training data, or is there a classifier built on the vision encoder? \n- The authors use GPT-4 to generate a visual descriptive text prompt. Are there any quality controls on the GPT's output (i.e., evaluated by experts/consistency of responses)?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work developed MGPATH, a vision-language model for whole slide images. It aligns the encoder of a foundational vision model (Prov-GigaPath) with the text encoder of a foundational vision-language model (PLIP) using lightweight adaptors trained on 923K image-text pairs. This vision-language model is then used to conduct few-shot WSI classification with multi-granular prompt learning. Extensive experiments show that it outperforms state-of-the-art methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See the weakness.", "broader_impact_concerns": "There are no concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sHY341mbDo", "reviewer": "Reviewer_MdBd", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength:**\nThe paper presents a novel and well-motivated approach for few-shot whole slide image classification, introducing multi-granular prompt learning and optimal transport alignment. It is further backed by many experiments across multiple datasets and model backbones.\n\n**Weaknesses:**\n\n1. From an editorial standpoint, the paper requires substantial revision. For example, the Introduction section includes excessive technical detail that is more appropriate for the Methods section (specifically, the last three paragraphs). Additionally, inconsistent and duplicative terminology is used throughout the paper and tables—for instance, “GigaPath-PLIP” and “PLIP-G” appear to refer to the same model variant and should be unified for clarity. Important details are also missing from the main text: the number of shots used in few-shot experiments is mentioned only in the appendix, and a table reference is missing in Section 4.4.\n\n2. The few-shot experiments lack evaluations across a range of shot numbers, which is important to assess the model’s robustness under varying levels of supervision. Given the paper’s emphasis on both zero-shot and few-shot learning, this omission is important, especially since, in real-world clinical settings, few-shot scenarios often involve rare cancers with only a handful of annotated cases. I recommend evaluating the model using a broader range of shots (e.g., 1 to 32). For reference, consider the methodology in this paper: https://arxiv.org/pdf/2411.19666\n\n3. Figure 1 lacks clarity and is not sufficiently intuitive. Even after reading the caption, the meaning of “t+1”, “t+2”, etc., and their relation to patches and prompts remains unclear to me, particularly when comparing the two rows in the figure and the presence of different red square boxes in each patch grid. A clearer and more informative visual illustration would greatly improve reader comprehension.\n\n4. It appears that the authors pretrained different variations of their model using the 923K image-text pairs, while the baseline models were not pretrained on this dataset. If this is correct, the comparison may not be entirely fair due to differing pretraining data sources. In this case, I recommend including experiments where MGPath, ViLa-MIL, and MSCPT are pretrained on a comparable dataset to ensure a fair and meaningful comparison. This will emphasize whether the multi-granular attention is the main source of performance improvement or the extra training data.\n\n5. The authors also claim that MGPath is parameter-efficient; however, the number of parameters reported for the full model (MGPath with PLIP-G at 5.35M; Table 1) is actually higher than many baseline models it is compared against. This raises questions about the basis of the parameter efficiency claim, and the authors should clarify what exactly they mean.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces MGPath, a novel vision-language model that tries to address problems in few-shot whole-slide pathology classification. Key contributions include (i) extending Prov-GigaPath into a vision-language model using PLIP text encoder; (ii) proposing a multi-granular attention mechanism; and (iii) employing optimal transport as a robust alignment metric between visual and text embeddings instead of cosine similarity. Based on the experiments provided, the method shows performance improvements across 3 public datasets compared to MIL methods and other vision-language approaches.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I'd certainly recommend revising the text for improved clarity, structure, and consistency. Above, I have outlined specific weaknesses observed in the current version of the paper and my suggestions for change or improvement.", "broader_impact_concerns": "No concern.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fmvILsu5iO", "reviewer": "Reviewer_gSYA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\nThe manuscript Leverages the foundation model Prov‐GigaPath  trained on 1.3 billion pathology patches and fine-tune it with an additional 923K image–text pairs. \n\nThe authors propose to combine the patch‐level prompting and spatial patch group prompting,\n\nThe authors propose to utilize optimal transport (OT)‐based distance to address the limitations of cosine similarity.\n\nMGPATH was tested on three TCGA datasets, demonstrating consistent improvements over state‐of‐the‐art multiple‐instance‐learning and other vision‐language baselines.\n\nWeaknesses:\n\nThe high- and low-resolution descriptive prompts are generated using the GPT-4 model. Since GPT-4 is a general-purpose model, I am not sure whether the prompts are reliable. It is also unclear how much of the performance of MGPATH depends on the quality of the language model.\n\nIn the ablation study, the authors do not discuss the case when alpha = 0. I am suspicious about whether the spatial patch group-based prompting is helpful.\n\nThe authors do not provide sufficient details in the experiments. I cannot find the patch size and the magnification levels that define the high- and low-resolution images. It would be helpful if the authors can discuss training and inference time of the proposed model.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript presents MGPATH, a vision-language model for whole‐slide image (WSI) classification. The authors adapt the Prov‐GigaPath foundation vision model into a vision‐language framework by aligning it with a pathology‐specific text encoder PLIP. They introduce a multi‐granular prompt learning strategy that (1) generates multiple learnable prompts at both low and high magnification levels, (2) fuses prompt embeddings with frozen visual features at the individual patch and patch‐group level, and (3) measures the distance between visual and text embeddings using an optimal transport metric. Experimental results on three TCGA datasets demonstrate consistent improvements over state‐of‐the‐art multiple‐instance‐learning and other vision‐language baselines.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In the ablation study, it would be beneficial if the authors can discuss the cases when alpha = 0 and alpha = 1. I am also not very sure what \"without multi-granular\" mean. Does it mean that only high-resolution patches and texts are used in training? I am curious about how the model would perform, if only high-resolution patches are used as the image input but align with both high and low-resolution textual embeddings. \n\nIt would be beneficial if the authors can include details such as patch size, magnification levels and computational costs.\n\nPlease replace the placeholder “??” on page 11 with the correct table number.", "broader_impact_concerns": "The authors do not provide a Broader Impact Statement. It might be helpful if the authors can briefly discuss the potential diagnostic misuse  and the generalizability issues.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)