Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling
Authors: Bryan Wong
Jongwoo Kim
Huazhu Fu
Mun Yong Yi
Venue: NeurIPS 2025 poster ()
Keywords: computational pathology
whole slide image
multiple instance learning
vision-language model
multimodal learning
data-efficient learning
few-shot learning
Primary area: machine_learning_for_sciences

Abstract: Vision-language models (VLMs) have recently been integrated into multiple instance learning (MIL) frameworks to address the challenge of few-shot, weakly supervised classification of whole slide images (WSIs). A key trend involves leveraging multi-scale information to better represent hierarchical tissue structures. However, existing methods often face two key limitations: (1) insufficient modeling of interactions within the same modalities across scales (e.g., 5x and 20x) and (2) inadequate alignment between visual and textual modalities on the same scale. To address these gaps, we propose HiVE-MIL, a hierarchical vision-language framework that constructs a unified graph consisting of (1) parent–child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes on the same scale. To further enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism that removes weakly correlated patch–text pairs, and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments on TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MIL consistently outperforms both traditional MIL and recent VLM-based MIL approaches, achieving gains of up to 4.1% in macro F1 under 16-shot settings. Our results demonstrate the value of jointly modeling hierarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data. The code is available at https://github.com/bryanwong17/HiVE-MIL.

Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and
Modeling
Bryan Wong
Jong Woo Kim
Huazhu Fu? Mun Yong Yi
KAIST IHPC, A*STAR
(bryan. wong, gsds4885, munyi)0kaist.ae ac. kr hzfu@ieee.org
Abstract
Vision-language models (VLMs) have recently been integrated into multiple in- stance learning (MIL) frameworks address the challenge of few-shot, weakly supervised classification of whole slide images (WSIs). A key trend involves leveraging multi-scale information to better represent hierarchical tissue structures. However, existing methods often face two key limitations: (1) insufficient modeling of interactions within the modalities across scales (e-g.. 5x and 20x) and (2) inadequate alignment between visual and textual modalities on the same scale. To address these gaps, we propose HiVE-MIL, hierarchical vingun-language framework that constructs a unified graph consisting of (1) parent-child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes on the same scale. To further enhance semantic consistency, HiVE-MIL. incorporates a two-stage, text- eext-guided dynamic filtering mechanism that removes weakly correlated patch-text pairs, and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments TCGA breast, lung, and kidney cancer datasets demonstrate that HiVE-MII. consistently outper- forms both traditional MIL and recent VLM-based MIL approaches, achieving gains of up to 4.1% % macro Fl under 16-shot settings. Our results demonstrate the value of jointly modeling hicrarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data. The code available at /https://githut: http:/..oobbbbbbaaaae coom/bbyaawong117. MIL
Introduction
Whole slide image (WSI) classification a central task in computational pathology (CPath), enabling cancer diagnosis, subtyping, and prognosis prediction (50 51 10]. With gigapixel resolution, WSIs contain detailed spatial information ranging from coarse tissue structures to fine-grained cellular
morphology (20). which is crucial for accurate interpretation in these diagnostic tasks. To handle their large size and the absence of fine-graine annotations, multiple instance learning (MIL) is widely adopted [8 28 In MIL, each WSI treated as a bag of instances (patches), and a slide-level label is predicted through a feature aggregator with supervision only at the slide level (Figure I ()). However, 34 face key challenges in real- solld clinical settings [23]: (1) they rely on large labeled datasets, which are difficult to obtain due to privacy concerns (4) and the rarity of certain diseases 32 making them ineffective in fhw-shot scenarios where labeled
traditional MIL models 27 33
"Equal contribution.
CCorrespondigg author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(A) Traditional MIL
Single-scale VLM MIL
(C) Multi-scale VLM MIL
(D) Hierarchical VLM MIL
X
Fiee
2
Aupregstoor
Aggregasar
Apsrgaster
Bierarchtcal Hettrrogeneeaa Ape
Claznifeer
scale-en
Mihh acale
Low scale
Hinel Logla
Pinsl Logis
Loght
Lack of textual understanding
Lack of contextual
Lack of instance hierarchy modality types
Captures hierarchical instance relations modality types
undorstandlng
Figure 1: Comparison of four paradigms for FSWC. (A) Traditional MIL uses only visual features and requires extensive WSI labels. (B) Single-scale VLM-MIL introduces LLM-based prompts but lacks contextual and scale-aware reasoning. (C) Multi-scale VLM-MIL adds ccale-specific prompts but lacks structured modeling of hierarchical dependencies and modality interactions ( [green indicates missing modalltty-ppeciff modeling). (D) HiVE-MIL. (Ours) explicitly captures
coarse-to-tine hierarchical relationships and aligns visual enabling efficient and scalable learning.
and
textual
features at each scale,
WSIs are scarce variability 36 and domain shifts [7 48 These challenges call for a data-efficient approach that leverages prior domain knowledge for robust learning under limited supervision.
(2) they use only visual features, making them highly sensitive to staining 44
Vision-language models (VLMs) such as CLIP 46). BLIP [35]. and Flamingo 2 have demonstrated strong zero- and few-shot transfer performance by learning joint image-text embeddings through contrastive learning 33 55 49 12.. However, their training in natural images and generic captions limits their effectiveness in CPath, where domain-specific semanties are critical. To address this
issue, domain- VLMs such as PLIP |25 QuiltNet [26], and CONCH [39] leverage large- scale pathology patch-text datasets and improve robustness and generalization
the patch level.
Motivated by these capabilities, recent efforts 44 45 I9] have integrated them into MIL. frameworks to address the few-shot suakly-uupeeisedd WSI classification (FSWC) problem by incorporating domain knowledge via text prompts (Figure BB). Multi-scale VLM- based MIL. methods [48] [2 41 further enhance single-scale models by using scale-specific prompts that capture WSI multi-scale information to better represent hierarchical tissue structures.
Although recent multiscale VLM-based MIL methods effectively transferring VLM knowledge to MIL remains a serious challenge due to limited modeling of the complex hicrarchical structure of WSIs and insufficient integration of multiple modalities. Specifically, as illustrated in Figure [ existing methods face two key limitations, (1) Insufticient modeling of hierarchical interactions within the same modalities. Existing models process visual and textual features independently at each scale and combine them through simple summation or averaging at the final prediction stage. This naïve fusion fails to capture hierarchical relationships across scales within each modality. In the visual domain, it overlooks the semantic progression from coarse tissue-level patterns to fine-grained cellular morphology. Also, in the textual domain, it fails to represent the transition from general morphological descriptions to specific structural details, limiting the model's ability to leverage hicrarchical semantics effectively. (2) Inadequate alignment between modalities on the same scale. Existing models do not n fully explore interactions between modalities when constructing task tsse-fpcc knowledge, often relying on simpler alternative mechanisms that lack the strong inductive bias 4 [58) needed for fine- -inenne mooss- alignment. This limits their ability to effectively integrate visual and textual features.
48 22
have made impressive progress,
To this end, we propose HiVE-MIL (Hierarchical Vision- LanguagE MIL), unified framework that explicitly models hierarchical relationships within modalities and intra-scale alignments across settings (Tigure[[i) (1) To capture the hierarchical modalities in multi-scale HiVE-MIL constructs hierarchical edges between visual nodes and interactions across scales,
between textual nodes across coarse (5x) and fine (20x) scales based parent- child relationships.

---

and jointly introduces a .MMdality SeclallityS-aleeaee Attention (MSA) mechanism that handles these connections, allowing the model represent semantic progression from global context localized detail while preserving hierarchical consistency. To ensure semantic coherence in textual space, HiVE-MII. incorporates a Hierarchical Text Contrastive Loss (HTCI.) that aligns class-level text embeddings across scales. Unlike prior methods that fuse multi-scale features only at the output, HiVE-MIL. facilitates explicit hierarchical interaction for more coherent representations. intra-scale interactions across modalities, HiVE-MIL utilizes a heterogeneous graph that captures semantic connections between visual and textual nodes on the same scale. This design improves the alignment quality between modalities and contributes to semantically coherent multimodal integration. Furthermore, to improve alignment accuracy, HiVE-MIL introduces a (TGDF) module that filters out semantically irrelevant weakly matched patch-text pairs, such as when a normal patch is mistakenly paired with IDC or II-C-related text, using text-wise soft thresholding. Together, these components enable HiVE-MIL to model hierarchical and semantic dependencies across scales and modalities, improving robustness and accuracy in FSWC.
(2) To effectively model
Dynamic Filtering
The main contributions of this work are follows:
We construct a hierarchical graph with hierarchical edges between coarse (5x) and fine (20x) visual/textual nodes via parent-child links, and introduce Modality-Scale Attention (MSA) and Hierarchical Text Contrastive Loss (HTCL) to enforce text semantic consistency across scales.
We build a heterogeneous graph to connect visual and textual nodes on the same scale and apply Text-Guided Dynamic Filtering (TGDF) module to remove weak or irrelevant patch-text pairs. improving intra-scale alignment.
Extensive experiments on three weal-world WSI datasets, including lung. breast, and kidney cancers, show that HiVE-MIL. consistently outperforms traditional MIL. and VL.M-based MIL. baselines across diverse few-shot settings and pathology foundation models.
2
Related Work
2.1 Multiple Instance Learning in CPath
WSI classification is typically formulated as a weakly supervised learning task in the MIL. setting, where each slide is treated as a bag of unlabeled patches with supervision provided only at the slide level [14. Embedding- based MIL approaches 22 33 38 47
54 are generally more effective than instance-based models 77. [9] 43), as they learn discriminative patch embeddings for aggregation 6 Early methods rely on non-parametric aggregators (e.g., mean, max), while attention-based
techniques such as ABMIL 27). DSMIL (3), and CLAM [38] introduce mechanisms to weigh patch relevance. TransMIL
captures spatial dependencies via self-attention, while DTFD-MIL [4 employs a double-tier distillation framework with pseudo-bag supervision. Graph-based models improve contextual modeling by constructing structured graphs that capture spatial and relational dependencies between instances. Existing methods remain ineffective for FSWC due to their reliance on large labeled datasets and visual-only features, which make them sensitive to staining variability [36. I and domain shifts (37 48), highlighting the need for data-efficient approach that leverages prior domain knowledge for robust learning under limited supervision.
57
17
24 34
2.2 Vision-Language Models in CPath
and Flamingo 2 learn joint image-text embeddings through contrastive learning and enable zero- and few-shot transfer via prompting (30,
Vision-language foundation models such as CLIP 44). BLIP 35
49 12]. In CPath, PLIPP 25), QuiltNet (26), and CONCH (39] adapt these models using large-scale pathology image-text datasets to enhance robustness in patch-level tasks. Motivated by their few-shot capabilities, recent works extend VLMs to MIL settings for WSI classification. However, adaptation methods developed for natural images, such as CoOp (59). CLIP-Adapter [15) (56), overlook the gigapixel scale, hierarchical structure, and fine-grained semantics of WSIs, limiting their effectiveness in weakly supervised scenarios. To address this issue, several studies integrate VLMs into MIL frameworks [4]
55
and HeGraph
22 19. TOP [4] and FOCUS adopt prompt-based supervision and multi-stage compression, respectively, but are limited to single-scale inputs and cannot capture multi-scale context. Multi-scale VLae-based MII. methods
45 48
19
48 22 41 improve upon these approaches by introducing scale-specific prompts that reflect the hierarchical nature of WSIs, enabling more context-aware and semantically aligned representations.
3

---

Fasture Exsration ciee
TecsGluded Dynamic Fibrig
23 13l---itttro Graps Stradams
Haviretiaa Heturage Oreah Leariing
Figure Overview of the proposed HiVE-MII. framework.
Nevertheless, they still face key challenges: they fail model semantic progression across scales within each modality and they lack alignment between visual and textual features on the same scale, weakening semantic grounding and multimodal integration. Unlike previous work, HIVE-MIL explicitly addresses both limitations by modeling hierarchical relationships across scales and aligning visual and textual features within each scale, enabling more robust performance in FSWC.
Method
3.1 Multi-scale Hierarchical Feature Extraction
Unlike prior work 48 that employs multi-scale text prompts and patch features without mod- eling explicit hierarchical relationships, our method organizes visual and textual representations into a parent-child hierarchy, where each fine-scale (child) node is explicitly linked to its corresponding coarse-scale (parent) node. This hierarchical design reflects the diagnostic workflow of pathologists and captures the intrinsic multi-scale structure of WSIs.
Hierarchical Visual Feature Extraction. Each WSI is divided into non-overlapping patches at two scales: low-scale (5x) and high-scale (20x). We extract N low-scale patches x:(9) and encode them using frozen VLM image encoder, resulting in visual features zn Simg (I E D To
capture finer-grained details within each low-scale patch, we subdivide it into M = (20/5)2 16
high-scale patches In.m. arranged in 4 spatial grid. Here, E (1,...,M, MM denotes the relative position of each high-seale patch within the grid. fewer than 16 high-scale patches are available inserted to preserve consistent feature dimensions.
(h)
(e.g., due whitespace), zero embeddings
Each valid high-scale patch is encoded as 2n,m = fung (mn.m) E R... To simplify notation, we flatten
(h)
the hierarchical indices as r (n, m). resulting in the final encoded representation 2 This yields hierarchical visual features: ( RNNxD the low scale and ( E RRRD the high scale. These representations maintain spatial alignment across scales and serve as the basis for hierarchical modeling. Please refer the Appendix(A.. A.2 for hicrarchical patch extraction details.
Hierarchical Textual Feature Extraction. Tumor heterogeneity causes WSIs within the same class to exhibit spatially diverse morphological features, including coarse tissue-level structures (c.g.. Glandular Acinar Patterns) and fine-grained cellular traits (e.g., Nuclear Hyperchromasia) forming hierarchical pattern. To capture this, on the following template:
LLM to generate hierarchical textual prompts based
LLM Prompt Template
The task is to summarize the morphological features of the (dataset_ _ame] dataset for the {class name_1).... {class sas__nam__cc classes. For each class, list O representative morphological features observed 5x magnificatiin, followed by K finer sub-features observed at 20x magnification for each. Each description should include the morphological term along with an explanation of its defining visual features.
This yields hierarchical textual descriptions: Low-scale Text, and High-scale Text, ky where each high-scale (child) text is associated with its corresponding low-scale (parent) text, enabling semanti-

---

cally consistent hierarchical alignment. Following CoOp [59], each text is prepended with L learnable tokens x*²) E ER L2L resulting in prompt embeddings:
(I) These are encoded via frozen VLM text encoder eo fie (1o and eo.k fien fien( ((.))) For notational simplicity, we flatten the indices to (o, k),, producing hierarchical textual embeddings EQ E ROxD and E(h) E RSDD For examples of hierarchical text generated by the LLM, please refer to the Appendix A.3
[0""][[ to = 11 1 ... [Low-scale Texto]
t0 |0 1
10% (1) HHighighhhsaa Text
3.2 Text-Guided Dynamic Filtering
We propose two-stage Text-Guided Dynamic Filtering (TGDF) module that improves intra-scale visual-texture alignment by removing semantically irrelevant image-text pairs in a top-down manner. In the first stage, TGDF filters out low-scale patches (e.g., normal tissue) and patch-text pairs with low similarity to the low-scale text. In the second stage, it refines high-scale patch selection based on retained low-scale patches and filters out weakly aligned high-scale patch-text pairs. This prevents irrelevant patches from being connected to disease-specific prompts, which could confuse the model. The remaining meaningful visual-textual pairs are then used to guide intra-scale edge construction in the heterogeneous graph. See Appendix D for the detailed algorithm and pseudocode.
Stage (Low-seale Filtering). We compute the cosine similarity between low-scale patch features z E I and low-scale text features E E BOxD forming similarity matrix ( E IR xo, To discard weak or irrelevant matches, we apply text-wise soft thresholding by computing the mean Mo and standard deviation a, across all patches in the WSI for each text where a controls the filtering sensitivity (Eq. 2 and 3
( liltree (n, 0) = (S (n,
Mo +a
(2)
(t)) The filtered similarity matrix fiitered E RN XO implicitly serves as soft relevance mask, where non-zero entries indicate semantically valid patch-text pairs. This matrix is then propagated to guide high-scale filtering.
Stage 2 (High-scale Refinement). For each retained low-scale patch and associated text o, use the corresponding high-scale image patches (h)) and submorphology texts es a similarity matrix ((h) E Rx5 To maintain consistency with the first stage filtering, we mask irrelevant pairs using the filtered similarity score: (( (r, s) = s(h) (r, s) S (n, o), where r (n, m) and (o, k). We then apply text-wise soft thresholding s(i) by computing the mean M4 and standard deviation a, for each submorphology text
(N) and compute
fiterred (r.s) (r,
(bb (makkod (r, s) Me + a a..
(3)
The final filtered similarity matrices at both low and high scales identify semantically aligned patch- text pairs at each scale. These aligned pairs are then used to construct intra-scale edges between modalities at both scales in the graph. As text tokens updated during training (Eq.II, [T]. the resulting text features and similarity matrices vary, enabling dynamic filtering and non-fixed intra-scale edges.
3.3 Hierarchical Heterogeneous Graph Structure
To enable integration across multiple scales and modalities, propose Hierarchical Heterogemeous Graph (HHG), GиHC = (V, P, T, T, R). This graph encodes intra-scale semantics and hierarchical structure in a modality modality. The edge set È = gintra U ghier supports structured message passing across both scales and modalities. Detailed definitions of the node set V and the edge set & are provided below.
manner, allowing each node to play different roles depending its
Node Set. We define node types T = (img(") img(*), (c) (ex((1)), where each type specifies both the modality (image or text) and the scale (low or high). The full node set is:
V {2(0 J U (((h)) 1 U uee!")) yo U J
(4)
5

---

Edge Set. We define relation types as R = {intra (, intra (h), (img) hier(ts))) Intra-scale edges connect valid patch and text nodes on the same scale using the TGDF-filtered similarity matrix (Section 3.2). Hicrarchical edges capture hierarchical alignment within each modality: (img)
(h) via absolute coordinate mapping; hier(te)) is constructed analogously using hierarchical text structure (Section (. All edges are bidirectional (a ++ b) to support relation-awan message passing across scales and modalities. The edge set is defined as:
links low-scale patch nodes 2n to high-scale ones 2,
Eintr (211) + e
$ fiterrd (n, > {2 + e
((l) fittered (T, +0,
intra
(h)
ghier (2/0 ++ 2(11) (e + !
hicr(iux)
hior(ee))
3.4 Hierarchical Heterogeneous Graph Learning
We introduce hierarchical heterogeneous graph neural network (HHGNN) designed to operate the constructed HHG. HHGNN performs relation-specific message passing to model local semantic interactions within each scale and propagate hierarchical signals across scales, enabling robust representation learning on multi-modal, multi-scale graphs. Details of the message passing are provided in Appendix E
Intra-scale Aggregator. To capture intra-scale relationships between patch and text nodes on the same scale, we apply relation-specific GraphSAGE [21] operator SAGE(*) (v) for each edge type ). The intra-scale representation is then computed by averaging the outputs over all such relations: hi MEAN( (SAGE(())) r Rimaa) }). Initial node features hy
E Riinta (zn (1) e0 2r <>es")
(h)
are modality-specific embeddings.
Hierarchical Aggregator. To capture hierarchical interactions across both modalities and scales, we introduce Modality- Scclle Attention (MSA), an attention mechanism applied to hierarchical edges (1)), Each edge encodes both the scale direction and the modality type. The node features are first enhanced with scale embeddings and projected into query, key,
E Rhier (zn (h) (1) ++
and value vectors using relation-specife weights: go w.(") (h. + 8x), kis = -W(") (h., + su), and
Va = ( (ha + s.). The attention weights are computed as Bon softmax 9u Ri and the final
vd
output is:
To
Σ (EN,(1)
Bun Vv
(6)
3.4.1 Feature Update and Classification
The final node representation computed as h, = h + Aiier combining intra-scale and hierarchical
information. Let y((e) and v(e) denote the sets of patch and text nodes at scale E {1, h). with feature
matrices X (h. E yln)) and ((s) {h, E vla)). Class-wise logits computed by:
logit
7 Σ Le
TopKAvgHO
(Ix))((1o)) (T!N))
(T(N)
SET.
where Le denote the class-specific text index set, where each i La corresponds text prompt associated with class the logit scaling factor provided by the VLM, and k((*) the number of top similarity scores considered at scale s. The operator TopKAvg, computes the average of (op-k((s) scores from the i-th text (- in the scale-specific image-text similarity matrix, thereby aggregating signals from the most relevant images for each text prompt before the final summation.

---

3.4.2 Training Objectives
Hierarchical Text Contrastive Loss (HTCL). To encourage semantic alignment of morphological text embeddings across scales, we compute cosine similarity between low- and high-scale textual embeddings: sim... = cos(TT) T(a), ), where o is a parent (low-scale) and is its child (high-scale). For each anchor 8, positive pairs are defined as Pa = (o cls(Parent(s)) cls(o)} and negatives as N, fo cls cls(Paren((s)) # cls(o)). The loss is computed as:
1 N
1 PP
LHTCL=
Σ 1=1
log (simo,s)
log o (-simm,,)))
(8)
[
JEP
JEN,
Let 2; E RC denote the final class class-wise wise logits for sample i, obtained by summing the low- and high-scale outputs, i.e., 2, = logit i,e logit The total loss then:
Loaal = LCF (zi, y1)) + À LHTCE.
where Les is the standard cross-entropy loss, Yi E fl...., C} is the ground-truth truth WSI class label, and À balances the two loss terms.
4 Experiments
4.1 Experimental Settings
Datasets. We utilize three publicly available WSI datasets: TCGA-NSCLC (lung), TCGA-BRCA (breast), and TCGA-RCC (kidney), obtained from The Cancer Genome Atlas (TCGA) B Following 48), each dataset is split into training, validation, and test sets using fixed 4:3:3 ratio. For the few-shot setting, we randomly sample 4, 8, and 16 WSIs class from the training set. For detailed dataset statistics and preprocessing steps, please refer to the Appendix A
VLM and Baselines. We evaluate using three pathology vision-language foundation models: PLIP 25
QuiltNet (26). and the recent CONCH [39] Baselines include: (1) Pooling-basse bbaeed (max,
mean): (2) Traditional MIL-based (ABMIL 27). DSMIL (33,, CLAM-SB/MB (38], TransMIL. [7]
DTFD-MIL (AFS) [54], WiKG (34): (3) Single-scale VLM-based MIL method (FOCUS 19]
(4) Multi-scale VLM-based MIL methods (ViLa-MIL [48), MSCPT (22)). All single-scale models
including the pooling-based based 20 patches, whereas multi-scale models 5x and x0x, Please refer to the Appendix for more details.
Implementation Details. HiVE-MIL operates 5x and 20x patches, using GPT-4o ( to generate coarse-level texts and K = 3 fine-level substructures per class. We use = 16 learnable context tokens (Eq. D and apply the TGDF threshold
0.5 (Eas. 2 3 The HHG consists of two layers and a 2-head in MSA. HTCL is used with X = 0.5 (Eq.) We train using Adam optimizer 31 (learning rate: 1e-4, weight decay: le-5), batch size 1, for up 50 epochs with early stopping (patience 10). All experiments are run using PyTorch [42 on workstation with two NVIDIA RTX A100 GPUs. Please refer to the Appendix for additional implementation details.
Evaluation Metrics. We report accuracy (ACC), area under the curve (AUC), and macro score. To mitigate dataset split variability in few-shot settings, all experiments are repeated five times, reporting the mean and standard deviation. To assess whether HiVE-MIL captures hicrarchical text semantic alignment (i.c., child hierarchy in text), we introduce Hit Ratio (Section 4.4
4.2 Main Results
We adopt the 16-shot evaluation setting as our main experimental setup. following [48 22 41), and report results across three TCGA datasets (NSCLC, BRCA, RCC) and three rision-langusal pathol- ogy foundation models (PLIP, QuiltNet, CONCH). As shown Table T HiVE--MIL consistently achieves the best performance across all settings, outperforming both traditional and VLMM-aseed MIL. methods, including their single-scale and multi-scale variants. Compared to the state-of-the-ar
https://portal cancer.cc.... gov/

---

Table 1: 16-shot results on three datasets using three pathology VLMs. The best and second-best results are highlighted in bold and underlined. HiVE-MIL outperforms all baselines in all settings.
Dataset
ICGA NSCLC
TCGA BRCA
TCGA RCC
Model
ACC
AUC
Macro Fl
ACC
ALIC
Macro F1
ACC
ALC
Macro F1
Max: Pooling
55,00 23.88
57.33
24.63
53.96
24.36
57.29
a3.23
62.33
=2.94
53.68
26.91
66.82
26.94
80.58
26.68
61.38 28.68
200K Pathilogggg Imioge Tint Pains
Mean Pooling
61.73
68.29
+7.55
61.15
65,25
70.83
64:04
79.62
92.09
76.67
+3.40
ABMIL
22
70.64:2..0
78.44 13.63
70.37 23:00
65.83 25.35
=7.58 72.87
65.29 15.78
80.00 13.71
93.01 =1.53
77.95 =3.43
DSMIL
15
72.63 +1.8
79.88 +4.60
72.48 +3.96
71.38 +3.00
77.55 +1.00
71.04 +5440
86 41 11
96.44 -0.63
84.63 -1.51
PLIP 25
CLAM-SB
75.96
#3.21
71.75 43.57
+2.59
53.00
+1:51
#1.54
CLAM--M
73.46
82. 13 +3.41
73.42 +5.15
72.
+2.92
7M.39
+3.95
72.20 +2887
86.97
+1 05
+0.78 96.53
84,92 +1.00
TransILL
47
73.21 53,02
81.44 22.75
72.98
72.08 43:32
79:17
=3.71
71.94
87.05
96.51
81,96 =1.32
DTFD-MML
72.95 23.40
79.79 #4.63
71.25 6268
78.91
a3.16
70.86 12.76
86.74 20.79
95,94
WIKG
31
67.89 23.66
75.54 =4.05
67.51 +3.62
67.71 12.19
74.92
#4.16
67.15
83.07 40.89
91.34 =0.76
80,32 =1.40
ViLa-MII.
48
=1.07
80.63 s2.37
73.90
71.04 36.92
78.42
a3.96
70.56
85.
95.53
82:51 =2.30
MSCPT
76.86
84.93
76.82
19
72.71 +2.90
79.78
72.58 +2.81
86.21 44.54
95.84 -0.45
84.20 -0.81
POCUS 19
71.73
78.21 +5.93
71.65 s5.51
71. 06 +5.60
78.19 #4.51
71.36 15.69
87.82 +1.60
96.73 =0.70
85.54 =1.82
HIVE-MIL
87.28 61.76
75.21 6341
74.99
from 2nd-best
(+3.27)
(+2.35)
(+3.26)
(+2.50)
(+3.19)
(+2.41)
(+1.07)
(+0.85)
(+1.64)
Max Pooling
53.59 23.66
57.24 a5.97
51,36
55.83 24.04
56. #4.36
53.75 34
68 28 26.77
81.33 27.72
61:31
IM Puthology Imsger Teen Pairs
Mean Pooling
60.77 +4.66
65.
+6.04
60.48
65 +2.93
72.41
+2.27
79 62.62.1
92.09
76.67 +3.49
ABMIL
28
67,31 59.64
75.
15.13
66.81
25
68,
#4
76.84 =4.27
68.42 +5.45
88. 09..8944 +1.71
96.86 20.50
87.11 =2.44
QuiltNet 226
DSMIL
72.76
XX
CLAM-SB
72.82 22,8
79:17 #2.93
72.58 #2.74
71.46 43.8
80.09 =0.00
71.24 25.00
58,
#2 17
97.58 =0.01
87.00 22.99
CLAM-MB
33
73.27 23.56
80.53 23.76
73.25 s3.15
72.29 22.43
78.42
22.75
72:24 22:47
88. 74 a1 62
97.34
86.83 =2.50
TransMIL
71.60
78.59 #4.86
71.67 +3.75
#2.92
71.56
=0.01
#2.65
DTFD-MIL
54
70.51 25:37
77.38 15.26
70.33 +5.89
72:71 +2.02
79.28
=1.81
72.66 11.99
88.66 11 65
=0.71 96.74
87.06 =1.99
WiKG MIL
68.20
75.08 $4.66
67.98
68.75 43.16
75.51
68.59
83.99
11.70
95.13
81.51 =3.14
ViLa-MII.
73.27 =5.54
80.82 +6.41
73.24 +5.52
+3.03
77.67
72.35 63.92
84.60
04
=0.70
MSCPT
76.15 63.33
84.06 +3.02
76.13 +5.82
72.
16 68.75
78.59
#4.21
71 +5.11
87 20
61 90
+0.87 96.89
85.33 -2.41
FOCUS 19
69.04
74.64
69.00
+3.56
75.66
22.66
68.47
89 11.23
97.13
87.43
HIVE-MIL
79.23 +2.70
87.34 +4.08
79.09 +2.75
77.08 63.90
#4.22 84.31
76,80 +4.18
89.97 40.85
98.32 20.48
88.18 a1.25
è from 2nd best
(+3.08)
(+3.28)
(+2.96)
(+4.37)
(+4,22)
(+4,14)
(+0,85)
(+0.74)
(+0.75)
Max Pooling
75.85 21.78
87.43 a1.60
71 s2.99
24.53
70.91
80.15
1.PM Pathology Iini lixt Pains
Mean Pooling
79.55 +2.33
87 +2.78
79.47 +2.74
76.67 +2.0
86.08
+4.43
76.47 +2.81
K7 60.60
96.76 -0.47
86.06 -0.46
ABMIL 27
84.30 22:22
90.97 $0.00
84.28
22.21
11.04 43.05
87.50
=5.38
80.93
03.04
88.43 95
96.17
86,95 22.33
39
DSMIL
85.83 +2.08
94.
+1.20
85.76
82. 0K 4399
89.91
46.4
91 13.89
91
98.20 -0.23
90.87 +7.00
CLAM-SB
3
85.83 g4.25
93.
93.19 22.39
85.80
24
82.29
47.42
90.70
=t.73
82.24 27.1
92 40.52
98.17 =0.33
90,76 =0.85
CONCH
CLAM-MB
86.92
94.01 12.16
86.91
81 88 84
90.41
=5.14
81.84
91.42 11.15
98.15
89,96 21.11
TransMLL
17
35,90 23.36
93.38 12.11
85.88 +3:36
82.50 +5.37
89.69
=4.54
82.38 15.36
89.27 22 34
97.75
87.66 =2.95
DTFD-MIL
64
$8.40 +3.54
95.36 51.52
88.37 s3.56
53. 54 +3
1.30 91
83.48 65.83
91.65
k1 14
=0.09 97.99
90.38 =1.52
WiKG 14
82.24
11.17
87.84
ViLa-MII.
48
83.08 x3.63
91.10 +2.43
83.04 13.64
77.08 16.69
87.03 =5.01
76.98 15.73
89
12.32
97.48
=0.79
87.91 =2.88
MSCPT
72
80.06 +5.20
88.05 26.24
79.95 15.24
79.
58:22
87.33
16.78
79.69 1621
92.00 41.52
98.03 =0.15
90.89 =1.94
FOCUS
22
85.32
93.43 11.45
$2.50 55
=4.50
82.20
91.57 11.14
90.21 =1.37
HIVE-MIL
90.39 #1.57
96.49 +8.56
90.37 $1.58
87.29 +2.3
93.86 20.89
+2.85
92.34 +1.35
98.53 e0.13
91.32 =1.68
A from 2nd best
(+1.99)
(+1.13)
(+2.00)
(+3.75)
(+2.64)
(+3.76)
(+0,23)
(+0.33)
(+0.43)
baselines, HiVE-MIL. achieves significant improvements across all datasets, with gains of up to +4.37% in ACC, +4.22% in AUC, and +4.14% in macro Fl on BRCA, and notable margins NSCLC (up to +3.27% ACC, +3.28% AUC, +3.26% FI) and RCC (up to + +1.07% ACC, +0.85% AUC, +1.64% F1). Even when paired with CONCH, the best pathology VLM to date, HiVE-MIL maintains consistent improvements across all datasets. Traditional MIL methods are based solely on visual features and large WSI labels, making them ineffective in the FSWC setting. Existing VLM-MIL methods, including multi-scale approaches, fail to capture hierarchical interactions across scales and do not effectively align modalities within the same scale. In contrast, HiVE-MIL explicitly models cross-scale hierarchies and intra-scale modality alignments, leading to superior performance.
4.3 Robustness in Few-Shot Scenarios
TOGA NSCLC
TOGA BRCA
TOGA ACC
16
PLP
AVE 159
Figure 3 shows the performance of HiVE-MIL in 4-, 8-,
70-
and 16-shot settings across datasets and pathology-ppecffic VLMs. Even with extremely limited supervision, HiVE-
16
16
16
Quilmet
AVV 183
30
MII. consistently outperforms existing baselines. For ex- ample, the highest observed performance gains are +6,81% 1%
70
1.6
16
16
and +8.57% on NSCLC at 4- and 8-shot settings with Quilt- Net, and +3.48% and +3.83% on BRCA with QuiltNet. The consistent improvements across diverse scenarios flect the method's s effectiveness in FSWC settings.
CONCH
AUD PM
16
16
16
DANIL
HIVE-
Figure 3: Few-shot robustness.
8

---

4.4 Hierarchical Text Semantic Alignment
Figure 4 evaluates the relationship between hierar- chical textual semantic consistency and classifica- tion performance, amalyzing how effectively HiVE- MIL aligns parent-chile structures across low- and high-scale textual descriptions. For each low-scale patch, retrieve its top-K (K=2) most similar low- scale texts (parents). We then identify the high-scale patches linked to the same low-scale patch and check whether their most similar high-scale text (child) cor- responds any child text associated with the re- trieved parent texts. A hit is recorded such a match is found (see the Appendix for more details). This process is repeated across all low-scale patches in the test WSIs and the overall score is reported as Hit Ratio @ 2 (x-axis). Among the evaluated variants (PLIP and QuiltNet, 16-shot), the bidirectional (Bi.) HiVE-MIL consistently achieves the highest Hit Ratio, outperforming both unidirectional (Uni.) and no-interaction (No.) variants. The strong correlation between Hit Ratio and Maero Fl underscores the effectiveness of bidirectional message passing preserving hierarchical semantics across scales.
TCGA
TOGA RCC
Macro P1c5)
80-
TCGA
TCGA BRCAO
Bi Uni.
DA BRCA
70
80 Ratiog2((%)
90
50
60
70
80
90
Hit Ratiop2(%)
Figure 4: Performance of Hit Ratio 2 and
Macro F1 (16-shot).
4.5 Interpretability Analysis
To assess interpretability, we visualize how HiVE-MIL aligns visual patches with class-level semantics at each scale using visual textual sim- ilarity scores. We sample a WSI from the IDC class in the BRCA dataset and identify, at each scale, the patch with the highest similarity to the class text, referred to as the Anchor (Figure 5 We then select Positive patches with text distri- butions most similar to the Anchor, and Negative patches with the most dissimilar distributions. The anchor and positive patches show similar morphological patterns, whereas the negative patches differ clearly in structure. We also show that IDC-related text classes have higher probabilities than ILC-related ones for each Anchor. This supports ViVE-MIL's IDC prediction for WSI and provides interpretable evidence based the description of the contributing text.
Figure 5: Low- and high-scale patches with highest (Anchor, Positive) and lowest (Negative) similarity to the WSI label (BRCA, CONCH, 16-shot).
4.6 Ablation Studies
Variants of HTCL. We evaluate three variants of the hierarchical text contrastive loss (HTCL), each defined by distinct strategy for selecting anchors, positives, and negatives. In the Share- Parent variant, the anchor is the mean embedding of high-scale texts that share the same low-scale parent (T (1)) and the query is low-scale embedding (T))). Positives and negatives are defined as P. (j Parent(i) = j} and N, = (j Parent(i) * j}. The Instance- -iste variant uses each high-scale embedding T.( as the anchor, with positive and negative sets identical to those in Share-Parens Our proposed Class-Was Wise variant (Section 3.4.2) constructs positive and negative sets based on shared class labels on all scales, allowing supervision at the class level. As shown in Table 2 Class-Wiss Wise consistently achieves the best performance across all settings. Moreover, all HTCL variants outperform the No-Contrastive bascline that uses only cross-entropy loss, highlighting the benefit of contrastive supervision for hierarchical text semantic alignment. Please refer to the Appendix J.1 for details on the HTCL variants.
Effects of Module Components. Table 3 presents an ablation study evaluating the contributions of TGDF, HHG, and HTCL, The full model (d), which integrates three components, achieves the highest accuracy and Macro F1 across all datasets, highlighting their complementary effects. Removing HTCL (c) leads to a performance drop on BRCA, with a 2.04% decrease in Macro Fl, emphasizing the importance of enforcing textual semantic consistency across scales. Further

---

Table 2: HTCL Variants (PLIP, 16-shot).
Table 3: Module ablation (QuiltNet, 16-shot).
TCGA NSCLC
NSCIC
TCGA BRCA
Row HBC HTCL.
TCGA NSCLC
TOGA BRCA
TCGA RCC
Macro
F1
Macro F1
Nu-Contraztivee
78. 14 233 78.11 aa 73.96 a44 73.81 248
(M)
x
74.73 pa 73.65
69.58 asu 69 asu 56.23 and 84 203
Share- Parre-
78.46 uam 78.37 1331 74.17
73.83
(b)
77.01 em 7680=
2 avw 72.75 an 67.36 er 85.21
Jnstnnee Wine
=*w 44m a2a km
(0)
13 78.27
143 74.76
142 58.88.82 96.85
Class- Wise (Ours) 80.13
4n 80.08 142 75.21
-18 74,99
(4)
79.23
11 79.09
42
77.08
v2m 76.80 sus 89.97
u $8.18
removing HHG (b) results in additional 1.64% drop in Macro Fl on RCC compared to (c), indicating the critical role of hicrarchical message passing in capturing multi-scale relationships. The configuration without any of the modules (a) consistently yields the lowest performance, confirming the necessity of each component. Overall, these results demonstrate that TGDF, HHG, and HTCL are jointly essential for effective representation learning in few-shot WSI classification. Please refer to the Appendix J.2 for descriptions of the module component ablations.
Effects of Hierarchical Aggregator. As shown in Figure 6 the Modality-Scall At- tention (MSA) module consistently outper-
TOGA NSCLC (PUP)
TCGA BRCA (PLIP)
TOGA NSCLCC (CONCH)
TCGA BRCA (CONCH)
70-
Mecco
forms all baselines across the evaluation metrics, including Modality-Aware Atten- tion (MAA), Scale-Aware Attention (SAA), generic Attention (Aftm.,, and GraphSAGE (SAGE). MSA explicitly models both the modality and scale at the edge level, enabling more effective interaction throughout the hierarchy. In contrast, the other variants omit modality, scale, or both, resulting in a performance drop of 1-3%. This highlights the importance of jointly modeling modality-specific and scale-aware information. SAGE performs the worst, as it applies uniform intra-scale message passing without scale adapta- tion, unlike other variants that incorporate scale-specific design. Details on hierarchical aggregator methods are provided in the Appendix [.3]
72
Figure 6: Comparison of hierarchical aggregator meth- ods (16-shot).
Further Ablations. We provide additional ablation studies in Appendix M and hyperparameter sensitivity analyses in Appendix N The findings indicate that HiVE-MIL is robust to a range of hyperparameter settings. Additionally, usage in Appendix [ demonstrating that although our method incurs moderate computational overhead compared to the baselines, it remains efficient and delivers competitive performance.
report FLOPs, inference time, and maximum GPU memory
5 Discussion
Conclusion. We propose HiVE-MIL, hierarchical vision-language MIL framework that models hierarchical dependencies and intra-scale multimodal alignments through a unified hierarchical heterogeneous graph. Hierarchical edges enhance contextual understanding by message passing across scales, while intra-scale links ensure semantic consistency across modalities. These designs, combined with text-guided filtering and hierarchical contrastive loss, enable robust learning under few-shot supervision. HiVE-MIL consistently outperforms MIL and VLM-MIL baselines across three TCGA datasets, offering an effective solution to aware, multimodal WSI classification.
Limitations and Future Work. TGDF currently relies on non-learnable, similarity -based thresholds defined per WSI, which may limit generalization across datasets or backbones. As future work, plan to explore adaptive, learnable filtering mechanisms to enhance robustness and transferability. Additionally, the Hit Ratio metric assumes the correctness of LLM-generated parent-child text structures. We plan to incorporate expert human evaluation to validate these hierarchies and to assess the alignment between the highest text description contribution with the corresponding patch.
Acknowledgements. This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korean government (MSIT) under grant numbers RS-2022-NR068758 and
We are also deeply grateful for the generous support provided by the Seegene Medical Foundation in South Korea.
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint 1oX.::::303088774 2023.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 2022.
[3] Rohan Anil, Scbastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katic Millican, et al. Gemini: A family of highly capable multimodal models. corr, abs/2312. 1885, 2023. doi: 10.48550. arXiv preprint ARXIV233 pages 24-28, 2025.
[4] Peter Battaglia, Jessica Hamrick, Victor Bapst, Alvaro Banchez-Gonzaler Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:8006. 080601261. 2018.
[5] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Giuneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama, 31 (12(2)2119922210 2017.
[6] Johannes Butke. Tobias Frick, Florian Roghmann, Safaa F. El-Mashtoly, Klaus Gerwert, and Axel Mosig. End-to-end multiple instance learning for whole-slide cytopathology of urothelial carcinoma. In MICCA/ Workshop on Computational Pathology, pages 57-68. PMLR, 2021.
[7] Gabriele Campanella, Michael G. Hanna, Lisa Geneslaw, Amy Miraflor, Victor Werneck Krauss Silva, Klaus J. Busam, and Thomas J. Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, 25(8): 1301)1)301-109 2019.
[8] Mare-André Carbonneau, Veronika Cheplygina, Eric Granger, and Ghyslain Gagnon. Multiple instance learning: A survey problem characteristics and applications. Pattern recognition. 77:329 353, 2018.
Philemon Chikontwe, Minsu Kim, Scong Jun Nam, Heounjeong Go, and Sang Hyun Park. Multiple instance learning with center embeddings for histopathology classification. Medical Image Computing and Computer-Assistee Intervention- 2020: 23rd International Conference, Proceedings, Part V. pages 519-528. Springer International Publishing, 2020.
[10] Miao Cui and David Y Zhang. Artificial intelligence and computational pathology. Laboratory Investiga- tion, 411-4)2412-422, 2021.
[11] Jan Deuschel, David Firmbach, Christoph I Geppert, Maximilian Eckstein, Arndt Hartmann, Valentin Bruns, and Matthias Benz. Multi-prototype few-shot learning in histopathology. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2021.
[12] Guneet S Dhillon, Pratik Chaudhari, Avinash Ravichandran, and Stefano Soatto. A baseline for few-shot image classification, arXiv preprint (eXivv1909.022 2019.
[13] Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint arXiv: 02428, 2019.
[14] Michael Gadermayr and Maximilian Tschuchnig. Multiple instance learning for digital pathology: review of the otate-of-the-arr, limitations & future potential. Computerized Medical Imaging and Graphics, 112:102337 2024.
[15] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better languaeee models with feature adapters. International Journal of Computer Vision, 132(2):581- 595, 2024.
[16] Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuchao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, et al. A survey on llm-as-a-judge. arXiv preprint arXiv:2411. 2024.
[17] Yonghang Guan, Jun Zhang, Kuan Tian, Sen Yang, Pei Dong. Jinxi Xiang, Wei Yang. Junzhou Huang. Yuyao Zhang, and Xiao Han. Node-all aligne graph convolutional network for whole-slide image represen- tation and classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18813 18823, 2022.
11

---

[18] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma. Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[19] Zhengrui Guo, Conghao Xiong. Jiabo Ma, Qichen Sun, I Lishuang Feng, Jinzhuo Wang, and Hao Chen. Focus: Knowledge-enhaneee adaptive visual compression for few-shot whole slide image classification. arXiv preprint arXiv:2411 147443, 2024.
[20] Metin N Gurean, Laura E Boucheron, Ali Can, Anant Madabhushi, Nasir M Rajpoot, and Bulent Yener. Histopathological image analysis: A review. IEEE reviews in biomedical engineering, 2:147-171, 2009.
[21] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning large graphs. Advances in neural information processing systems, 30, 2017.
[22] Mingaao Han, Linhao Qu, Dingkang Yang. Xukun Zhang, Xiaoying Wang, and Lihua Zhang. Msept: Few-shot whole slide image classification with multi-scale and context-focused prompt tuning. arXiv preprint arXiv:2408. 1505, 2024.
[23] Mahdi S Hosseini. Babak Ehteshami Bejnordi. Vincent Quoc-Huy Trinh, Lyndon Chan, Danial Hasan, Xingwen Li, Stephen Yang, Tachyo Kim, Haochen Zhang. Theodore Wu, et al. Computational pathology: survey review and the way forward. Journal of Pathology Informatics, 15: 100357, 2024.
[24] Wentai Hou, Lequan Yu, Chengxuan Lin, Helong Huang, Rongshan Yu, Jing Qin, and Liansheng Wang-
H' 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 2022.
[25] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307-23 2316, 2023.
[26] Wisdom Ikezogwo, Saygin Seyfinglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mohammed, Pa- van Kumar Anand. Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathology. Advances in neural information processing systems, 36:37995-38017, 2023.
[27] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International conference machine learning, pages 2127-2136. PMIR, 2018.
[28] Maximilian Ilse, Jakub M Tomezak, and Max Welling. Deep multiple instance learning for digital histopathology. In Handbook of Medical Image Computing and Computer Assisted Intervention, pages 521-546. Elsevier, 2020.
[29] Guillaume Jaume, Anurag Vaidya, Andrew Zhang, Andrew H. Song, Richard J. Chen, Sharifa Sahai, Dandan Mo, Emilio Madrigal, Long Phi Le, and Faisal Mahmood. Multistain pretraining for slide representation learning in pathology. In European Conference on Computer Vision, pages 19-37. Springer, 2024.
[30] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig, Scaling up visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMLR, 2021.
[31] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6900 2014.
[32] Jinyoung Lee, Chih-Wei Liu, Jihie Kim, Zhe Chen, Yizhen Sun, Jared R. Rogers, and Chunhua Weng- Deep learning for rare disease: A scoping review. Journal of Biomedical Informatics, 135:104227, 2022.
[33] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14318-14328, 2021.
[34] Jiawen Li, Yuxuan Chen, Hongbo Chu, Qiehe Sun, Tian Guan, Anjia Han, and Yonghong He. Dynamic graph representation with knowledge-aware attention for histopathology whole slide image analysis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11323- 11332, 2024.
[35] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language -maggeiiaeeeeee pre- te-iiraiing for unified pages 12888 18888-12900 12900. PMLR, 2022.
understanding and generation. In International conference on machine learning.
12

---

[36] Sylwia Libard, Dzenita Cerjan, and Irina Alafuzoff. Characteristics of the tissue section that influence the staining outcome in immunohistuchemiitry Histochemistry and Cell Biology, 151:91-96, 2019.
[37] Tiancheng Lin. Zhimiao Yu, Hongyu Hu, Xu, and Chang-Wen Ween Chen. Interventional bag multi-instance learning on whole-slide pathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19830- 19839, 2023.
[38] Ming Y Lu. Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.
[39] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Lc, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863-874, 2024.
[40] Bradley A. Malin, Khaled Emam, and Christine M. O'Keefe, Biomedical data privacy: problems, perspectives, and recent advances. Journal of the American Medical Informatics Association, 20:2-6, 2013.
[41] Anh Tuan Nguyen, Duc Minh Hieu Nguyen, Nguyen Tuan Diep. Thi Quynh Nguyen, Nam Ho, Julian M. Metsch, and Anne C. Hauschild. Few-shot whole slide pathology classification with multi -uatilgranular vision-language models. In ICLR 2025 Workshop on Foundation Models in the Wild, 2025.
[42] A Paszke. Pytorch: An imperative style, tigh-performance deep learning library. arXiv preprint arXiv: arXiv:1912.00 1912.0177.7 2019.
[43] Linhao Qu, Xiaoyuan Luo. Shuai Liu. Manning Wang. and Zhijjan Song. Dgmil: Distribution guided multiple instance learning for whole slide image classification. In International Conference on Medical Image Computing and Computer-Assister Intervention, pages 24-34. Springer Nature Switzerland, 2022.
[44] Linhao Qu, Kexue Fu, Manning Wang, Zhijian Song, et al. The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. Advances Neural Information Processing Systems, 36:67551-67564 2023.
[45] Linhao Qu, Dingkang Yang, Dan Huang, Qinhao Guo, Rongkui Luo, Shaoting Zhang, and Xiaosong Wang.
enhanced multi-instance prompt learning for few-shot whole slide image classification. In European Conference Computer Vision, pages 196-212. Springer, 2024.
[46] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763 PmLR, 2021.
[47] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processngg systems, 34:2136-2147, 2021.
[48] Jiangbo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu. Vila-mil: Dual-scale vision-language multiple instance learning for whole slide image classification. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11248-1122 11258, 2024.
[49] Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pages 266-282. Springer, 2020.
[50] Hamid Reza Tizhoosh and Liron Pantanowitz. Artificial intelligence and digital pathology: challenges and opportunities. Journal of pathology informatics, 9(1):38, 2018
[51] Jeroen Van der Laak, Geert Litjens. and Francesco Ciompi. Deep learning in histopathology: the path to the clinic. Nature medicine, 2755)77757784 2021.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[53] xAL Grok 3. https::/ ttt::://rookcco com 2025. Accessed: 2025-05-22.
13

---

[54] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18802-18812, 2022.
[55] Yuhao Zhang. Hang Jiang. Yasuhide Miura, Christopher D Manning, and Curtis Langlotz. Contrastive learning of medical visual representations from paired images and text. In Machine learning for healthcare conference, pages 2-25, PMLR, 2022.
[56] Yumiao Zhao, Bo Jiang, Xiao Wang, Qin Xu, and Jin Tang. Hegraphadapter: Tuning multi-modal modal vision- languae models with heterogeneous graph adapter. arXiv preprint arXiv:2410.0785 2024.
[57] Yi Zheng, Rushin H Gindra, Emily Green, Eric Burks, Margrit Betke, Jennifer E Beane, and Vijaya Kolachalama. A graph-transformn for whole slide image classification. IEEE transactions on medical imaging, (1 1):3003-3311 2022.
[58] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changeheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. Al open, 1:57-81, 2020.
[59] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Learning prompt for vision- languale models, International Journal of Computer Vision, 130(9)23337 22448, 2022.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our Abstract and Section Jacurately reflect our paper's contributions and
scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed descriptions of our experimental setup in Section 4.1 with additional details in Appendix H
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [Yes]
Justification:
We provide a GitHub link to our code in the Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https Cooeeeoodeesssseeeeee //nips.com htte:/uulllllliiiee/e//eeeuuaasl casssiinnccccyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We provide detailed descriptions of all the training details in section 4.1 We further provide the data splits and generated texts in the GitHub link.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We report the standard deviation in our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
We provide information on the computational resources in Section 4.1 and details on FLOPs, inference time, and maximum GPU memory usage are included in Appendix O
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: wwtp:uuriippp.ppppeeeeeeeee httppp/cccccttticccccii........
Answer: [Yes]
Justification: Since all datasets employed this work are publicly available, our study complies with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the broader impacts of work in both the Abstract and Conclu- sion, with further details in Appendix P
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
The assets used in our paper are properly credited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
The code we provided includes a detailed README document.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: All the data we used comes from public datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: All the data we used comes from public datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We describe the use of an LLM for text generation in Section 3.1 and specify the LLM version employed in Section 4.1
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
21

---

Appendix for Few-Shot Learning from Gigapixel Images via Hierarchical Vision-Language Alignment and Modeling
Dataset and Input Construction
A.1 Dataset Descriptions
We use three public WSI datasets from The Cancer Genome Atlas (TCGA): NSCLC, BRCA, and RCC slide undergoes Otsu's 's thresholding to remove background regions and is normalized for stain variation. Patches are extracted at two scales: 256 x 256 pixels at x (low scale) and 20 (high scale). Table summarizes the number of slides and patches extracted for each subtype.
Table 4: Summary of TCGA datasets used in this study.
Dataset
Subtype (Abbr.)
Slides
Patches (5x)
Patches (20x)
Lung Adenocarcinoma (LUAD)
531
452,664 424,008
6,917,186
TCGA NSCLC
Lung Squamous Coll Carcinoma (LUSC)
511
6,451,978
Carinomm (IDC)
844
9,276,899
TCGA BRCA
Invasive Lobular Carcinoma (ILCC
211
140,974
2,092,908
Clear Cell Renal Cell Carcinoma (CCRCC)
455
424,338
6,487,063
TCGA RCC
Papillary Renal Coll Carcinoma (PRCC)
296
258,288
3,909,258
Chromophobe
(CHRCC)
1.725,880
Following [48), we split each dataset into training, validation, and test sets using a 4:3:3 ratio. For few-shot evaluation, we repeat the random-splitting process five times and report averaged results. Due to the class imbalance in the TCGA BRCA (approximately 20:80), we balance the test set during sampling to avoid misleading final metric results. All details regarding the data splits used are provided in the splits directory of our GitHub repository (linked in the Abstract).
A.2
Hierarchical Patch Extraction
High Level
(x20)
Complex Branching
Apical Epithelial
Papillae
Vessels
Low Level
Glandular Acinar Pattern
Papiliary Configurations
(x5)
WSI Level
LUAD
LUSC
(x1)
TCGA NSCLC
Whole Slide Image
Pathological Annotations
Figure 7: Illustrative example of hierarchical visual patches and hierarchical texts within a WSI.
typically process patch features at different scales indepen- dently, without preserving any hierarchical structure between them. This design hinders the ability
Existing MIL methods 33
48
22
https://portal. www.er.ccccooooe gov/
22

---

to model structured cross-scale interactions, which are essential in tasks such as WSI classification, where coarse tissue-level patterns and fine-grained cellular features are often interdependent. To address this, we propose hierarchical extraction pipeline with an explicit 1-to- 16 mapping across scales: each low-scale (5x) patch corresponds to a spatial region at the high scale (20x), which is uniformly subdivided into fixed 4 x 4 grid, yielding 16 high-scale patches. This structured mapping ensures that each high-scale patch can be directly and uniquely traced back to its low-scale parent, enabling consistent alignment and hierarchical reasoning across scales.
To facilitate downstream processing, we flatten the two-level indices (n, m), where denotes the low-scale patch index and m E . 15} denotes the position of the grid within its corresponding high-scale region, into a single index 16n + m. This indexing convention enables the effective retrieval of parent-child relationships and is used throughout all scale-sensitive components. It forms the basis of our multi-scale vision-langee siioo-la alignment framework, supporting semantic propagation across scales while preserving spatial coherence within the WSI.
A.3 Hierarchical Textual Prompt Construction
We present the hierarchical textual descriptions used in our study to guide visual-language alignment in HiVE-MIL. The final version of the text is generated using GPT-4o [[] although we also explore variants generated by other LLMs and demonstrate that HiVE-MIL's performance is robust the choice of LLM (see Appendix M.2). These descriptions capture representative morphological features two levels: coarse tissue-level patterns at the 5 scale and fine-grained cellular structures the x scale. Each coarse-level feature (four in total) serves as a parent, linked to three fine-level child features, forming a structured hierarchy. These hierarchical descriptions are used as textual input in our HiVE-MIL framework.
23

---

TCGA NSCLC (LUAD) (5x 20x)
Glandular Acinar Patterns: Well-cireumseribed acinar structures composed of atypical cuboidal to columnar epithelial cells, often containing intraluminal mucin and exhibiting mild nuclear pleomorphism.
Irregular Gland Outlines:
Angulated, non-uniform glandular profiles with irregular lumina, nuclear crowding, and focal mucin, indicating infiltrative glandular transformation.
Nuclear Hyperchromasia:
Darkly stained, enlarged nuclei with coarse chromatin and prominent nucleoli, often accompanied by atypical mitotic figures.
Fibrotic Stromal Response: Dense fibrous stroma with activated fibroblasts and peritumoral lymphocytic infiltration, reflecting the desmoplastic reaction around the malignant glands.
Branching papillary structures supported by delicate fibrovas- cular cores, lined by stratified columnar tumor cells with nuclear sseudo-stratificale and occasional apical tufting.
Papillary Configurations:
Complex Branching Papillae: Elaborately branched papillary fronds lined by multilayered atypical cells, often with central necrosis and early signs of stromal infiltration.
Fibrovascular Core Vessels: Central capillaries within papillary stalks, sur- rounded scant inflammatory cells and sometimes showing endothelial prolif- eration or changes in the basement membrane.
Apical Epithelial Tufts: Protruding apical structures formed by columnar tumor cells with eosinophilic cytoplasm and low-grade nuclear stratification, occasion- ally secreting mucin.
Tumor cells spreading along intact alveolar septa without architectural distortion, preserving pulmonary parenchyma and lacking stromal invasion.
Lepidic Growth:
Flat Monolayer Proliferation: A single layer of atypical cells that line the alveolar walls, maintaining the underlying architecture with subtle nuclear atypia and minimal stromal response.
Preserved Alveolar Architecture: Retention of the alveolar septa and capil- lary framework despite tumor cell proliferation, often accompanied by mild inflammation.
Extracellular mucin accumulation within alveolar spaces, bordered by flattened tumor cells, suggesting early mucin mucin-producing differentia- tion.
Mucinous Secretions:
Solid Sheets with Mucin Production: Compact tumor cell sheets without glandular formation, exhibiting cytoplasmic mucin vacuoles, nuclear pleomorphism, and high muclear-to-cytoplammic ratios.
Coalescent Cell Clusters: Irregularly bordered nests or aggregates of poorly dif- ferentiated tumor cells with hyperchromatic nuclei and scant glandular features.
Intracellular Mucin Pools: Cytoplasmic mucin displacing the nucleus the periphery, creating signet-ring-like cells typical of mucinous adenocarcinoma variants.
Central Necrotic Foci: Necrotic zones within solid tumor areas, surrounded by viable malignant cells, showing karyorrhexis, inflammation, and early cavitation.
24

---

TCGA NSCLC (LUSC) (5x 20x)
Keratin Pearls: Concentric layers of cosinophilic keratinized material encircled by malignant squamous cells, hallmark of well-differentiated squamous carcinoma.
Concentric Lamellae: Onion-skin-lin layers of compact keratin, indicating advanced squamous maturation in well-differentiated tumors.
Central Keratin Accumulation: Dense eosinophilic keratin material in the center of tumor nests, displacing surrounding malignant cells outward.
Peripheral Reactive Stroma: Fibroblastic and inflammatory stromal response surrounding keratinizing nests, often with multinucleated giant cells.
Intervellular Bridges: Cytoplasmic connections between adjacent tumor cells, repre- senting retained desmosomal junctions typical of squamous differentiation.
Desmosomal Thickenings: Pronounced desmosomal plaques visible at tumor cell junctions, reinforcing epithelial cohesion in squamous cells.
Bridging Spines: Flongated cytoplasmic projections maintaining intercellular contacts, characteristic of squamous cell architecture.
Intercellular Gaps: Narrow spaces between adjacent tumor cells with intact
desmosomal attachments, allowing minimal extracellular fluid passage.
Dyskeratotic Cells: Isolated eosinophilic tumor cells undergoing premature keratiniza- tion, appearing dense, glassy bodies within cell clusters.
Premature Keratin Accumulation: Early keratin buildup within individual cells, causing nuclear condensation and cytoplasmic eosinophilia in dyskeratotic zones.
Eccentric Hyperchromatic Nuclei: Peripheral, dark-staining nuclei compressed by cytoplasmic keratin, showing irregular contours and coarse chromatin.
Focal Cell Lysis: Localized necrosis with keratin extrusion and surrounding inflammatory infiltrates, occurring within tumor nests.
Squamous Eddies: Swirling arrangements of keratinizing squamous cells, forming eddy-like patterns often seen in keratinizing regions.
Spiralized Cellular Streams: Corkscrew-like arrangements of tumor cells show- ing organized squamous maturation with partial keratinization.
Vortex-Lit Whorls: Dense, spiral patterns of malignant squamous cells with central keratinization and peripheral nuclear reorientation.
Peripheral Flattening: Peripheral squamous cells becoming flattened along the edges of nests, demarcating mature keratinizing foci.
25

---

TCGA BRCA (IDC) (5x 20x)
Infiltrative Tumor Regions: At 5x, irregularly shaped, ill-defined tumor regions invade surrounding tissue, displacing or distorting adjacent structures.
Pleomorphic Tumor Cells: At 20x, tumor cells display marked variation in size and shape, with irregular contours and disorganized architecture.
Irregular Nuclear Contours: Nuclei show jagged, wrinkled borders, with coarse chromatin and frequent mitotic figures.
High Nuclear-to- Co-oo-llaactt Ratio: Tumor cells have large hyperchromatic nuclei and scant cytoplasm, reflecting aggressive cellular proliferation.
Dense Tumor Cellularity: Low-power view reveals solid nests or trabecular cords of tumor cells occupying large portions of the parenchyma.
Prominent Nucleoli: Nuclei often feature conspicuous eosinophilic nucleoli, characteristic of high-grade IDC.
Increased Mitotic Activity:
Numerous mitoses, including abnormal forms, are visible, particularly at invasive fronts.
Tumor architecture includes malformed gland-like ducts or linear trabeculae amidst fibrotic stroma.
Glandular or Trabecular Formations:
Necrotic Zones: Focal areas of necrosis, sometimes centrally located within tumor nests, give rise to ghost cell zones and debris.
Ghost Cell Debris: Necrotic
contain pyknotic nuclei, faded cytoplasm, and fragmented cell remnants.
Peripheral Viable Tumor Rim: Viable tumor cells form a around necrosis, often with hyperchromatic nuclei and mitotic activity.
Mixed Inflammatory Infiltrate: Neutrophils and macrophages infiltrate around necrotic zones, indicating the tumor-host interaction.
Distorted Stromal Architecture: Stroma shows reactive changes and desmoplasia due to infiltrative and space-occupying tumor growth.
Desmoplastic Stromal Response: Fibroblast-rich, fibrotic stroma surrounds tumor nests and glandular elements.
Reactive Fibrosis: Dense collagen bundles with stromal retraction and scattered lymphocytes surround invasive regions.
Peritumoral Lymphocytic Aggregates: Clusters of immune cells the tumor- stroma borders suggest the host response to invasion.
26

---

TCGA BRCA (ILC) (5x 20x)
Single-File Growth Pattern: At 5x, tumor cells infiltrate in linear rows between stromal fibers, forming the classic Indian file architecture.
Small, Uniform Tumor Cells: At 20x, tumor cells monomorphic, with small round-to-oval nuclei and inconspicuous nucleoli.
Round or Oval Nuclei: Nuclear morphology bland, with smooth contours and fine chromatin.
Minimal Pleomorphism: Cellular features are uniform, with very limited vari- ability in size, shape, or staining.
Diffuse Infiltration: Tumor cells are widely dispersed across the stroma, lacking a mass-formin mass-forming architecture.
Intracytoplasmic Mucin: Clear cytoplasmic vacuoles displace nuclei, creating a targetoid signet-ring-like appearance.
Lack of Glandular Structures: No lumen formation epithelial polarity is observed, which differentiates ILC from IDC.
Diffuse Growth Pattern: Tumor spreads diffusely through the stroma, often preserving native tissue landmarks.
Scattered Tumor Cells: Neoplastic cells appear loosely distributed, often without clustering or clear nest formation.
Sparse Stromal Reaction: Fibrous stroma is loose and minimally reactive, in contrast to the desmoplasia IDC.
Isolated Single Cells: Tumor cells often appear as single entities, lacking inter- cellular adhesion junctions.
Perivascular or Periductal Infilration: Tumor cells wrap around existing ducts and vessels without destruction, maintaining normal tissue outlines.
Loss Cohesive Structures: Absence glandular trabecular patterns, with minimal distortion of pre-existing tissue architecture.
Nuclear Monotony: Cells exhibit uniform nuclear size, shape, and chromatin, reflecting low-grade morphology.
Clear Cytoplasm with Vacuoles: Cytoplasm appears pale with discrete vacuoles, often mistaken for benign tissue.
Loss of E-cadherin Expression: Absence cell adhesion molecules explains discohesive behavior, visible through architectural disarray.
27

---

TCGA RCC (CCRCC) (5x 20x)
Clear Cytoplasm Dominance: At 5x, tumor cells appear in sheets or nests with optically clear cytoplasm due lipid/glycogen, bounded by crisp membranes.
Intracytoplasmic Glycogen/liipee Accumulation: At 20x, cytoplasm is vacuo- lated and optically clear due to the lipid or glycogen content.
Peripheral Nuclear Displacement: The nuclei are eccentrically located and pushed to the periphery of the cell by abundant cytoplasm.
Fine Cell Membrane Borders: Well-demarcated cell borders are easily identifi- able at high magnification.
Alveolar-Nested Architecture: Low power reveals pseudo-alveolar structures formed by small tumor nests and intervening thin fibrovascular septa.
Small Tumor Nests: Compact, round nests of tumor cells mimic alveolar units, often surrounded by fine capillaries.
Intervening Thin Fibrovascular Septa: Nests are separated by delicate fibrovas- cular septa lined by flat endothelial cells.
Lack of Papillary Structures: No fibrovascular cores or true papillae are evident, which helps to distinguish the subtype.
Prominent Sinusoidal Vasculature: Numerous sinusoidal capillaries form arborizing patterns that wrap around tumor clusters.
Rich Capillary Networks: 20x shows dense, branching vascular beds extending between and around tumor clusters.
Endothelial Wrapping: Capillaries encase the nests with endothelial cells form- ing tight boundaries.
Erythrocyte-Fille Filled Lumina: Sinusoidal spaces frequently appear engorged with red blood cells.
Minimal Nuclear Atypia: Tumor nuclei show minimal pleomorphism, with low-grade features distributed evenly across nests.
Low Nuclear Grade: Nuclei are round with uniform chromatin, corresponding to Fuhrman grade I-II.
Inconspicuous Nucleoli: Nucleoli are small absent under 20x, suggesting low proliferative activity.
Uniform Chromatin: Evenly distributed chromatin supports low-grade tumor morphology.
28

---

TCGA RCC (CHRCC) (5x- 20x)
Pale Eosinophilic Cytoplasm: At 5x, tumor cells exhibit pale pink cytoplasm with central nuclei and minimal architectural variation.
Prominent Perinuclear Halo: Clear perinuclear zones caused by microvesicular cytoplasm dominate the cell morphology.
Reticulated Cytoplasm: Fine vesicle-like reticulations are visible in cytoplasm under higher magnification.
Dense Cell Borders: Polygonal tumor cells have well-defined borders and cytoplasmic outlines.
Tumor grows in broad, cohesive sheets with minimal stromal interruption or patterning.
Solid Sheet Growth Pattern:
Broad Cell Plates: Cells form expansive and cohesive units with minimal architectural disruption.
The growth pattern lacks central fibrovascular structures, reinforcing a solid architecture.
Lack of Fibrovascular Core:
Sparse Mitoses: Few mitotic figures are observed, suggesting low-grade prolif- crative activity.
Perinuclear Clearing: Halo-like clearing around nuclei imparts a distinct plant-cell morphology.
Clear Perinuclear Cytoplasmic Zones: Large halos perinuclear clearing disrupts cytoplasmic uniformity.
Irregular Nuclei: Nuclear contours are wrinkled or raisinoid in appearance.
Binucleation: Multiple nuclei within single cell are commonly observed.
Plant-like Mosaic Growth: Geographic arrangements of cells create a tiled mosaic- like appearance under low magnification.
Geographic Cell Grouping: Tumor cells form clustered patches resembling tiles or islands.
Peripheral Cytoplasmic Accentuations: Borders are thickened accentuated the cell periphery.
Eosinophilic Granularity: Cytoplasm contains fine, pink-staining granules.
29

---

TCGA RCC (PRCC) (5x 20x)
Papillary/Trabecular Architecture: At 5x, tumor growth includes true papillae and trabecular structures with alternating cords and sheets.
True Papillae: Well-formed fibrovascular cores lined by single pseudo- stratified tumor epithelium are present.
Pseudopapillary Areas: Incomplete or collapsed papillary structures lacking central cores are seen.
Trabecular Slits: Linear cords of tumor cells form slit-like spaces within loose fibrotic stroma.
Foamy Macrophage Aggregates:
Pale yellow zones containing lipid-laden macrophages are seen within tumor stroma and papillae.
Intraluminal Clusters:
Macrophages aggregate in luminal spaces or within
papillary cores.
Vacuolated Cytoplasm: Lipid content gives macrophages foamy, vacuolated appearance.
Hemosiderin Pigmentation: golden-brown pigment granules are deposited within macrophage cytoplasm.
Pseudostratified Tumor Epithelium: Papillae are lined by tumor cells with crowded, elongated nuclei mimicking stratification.
High Nuclear Crowding: Elongated nuclei densely crowd near the apical surface of epithelial layers.
Hyperchromatic Nuclei: Nuclei are darkly stained, often irregular in shape.
Mitotic Figures: Frequent mitotic activity is visible, especially in higher-grand cases.
Calcified concentric structures are visible in the cores or adjacent stroma under low magnification.
Psammoma Body Formation:
Concentric Calcification: The bodies of psammoma appear as round layered calcifications in the stroma.
Stromal Mineralization: Stromal areas show scattered calcific debris.
Associated Necrosis: Focal necrotic zones are seen near the papillae or within the stroma.
30

---

B Validation of LLM-Generated Descriptions
While the descriptions generated by the LLM have yet been verified by pathologists, we ad- dress this limitation through a series of alternative validation strategies designed to assess their class-iiscimminative relevance and semantic reliability capturing morphology- and concept-level information.
B.1 Dlas-miiscrmminative Relevance
We evaluate the faithfulness of the generated descriptions using the
[66] where
LLMs are prompted to infer class labels based solely on the generated descriptions. We employ two evaluators: GPT-4o and GPT o3, to assess descriptions generated by several LLMs: Deep Seek R1 18]. Grok 3 [53], Gemini 2.5 Pro [3), and GPT-4o [.. Although not perfect, the number of correct predictions is still consistently high, suggesting that the generated text is generally reliable while
capturing class- discriminative signals.
Table 5: Class-diseriminative relevance (LLM-as-a-Judge))
Total Descriptions (#5x #20x)
GPT-40
GPT-4o
GPT
Generator Evaluator
(TCGA NSCLC)
(ICGA BRCA)
(TCGA NSCLC)
(TOGA BRCA)
DeepSeck
(8/24)
(8/20)
28
(8/23)
28
Grok 33
32 (8/24)
28 (7/21)
28 (8/20)
32 (8/24)
21 (8/23)
Pro 6
(8/24)
(8/22)
(7/22)
(7/23)
(8/24)
GPT-4o
32 (8/24)
30 (8/22)
30 (7/23)
31 (8/23)
32 (8/24)
Total
(32/96)
(31/5))))
(29/5)
(31/3)))
(32/9)
B.2 Morphology- and Concept-Level Semantic Validation
reppots intra- and inter-category text similarity scores, where intra denotes similarities among descriptions of the same morphology or concept, and inter denotes similarities across different ones. Similarity is computed separately at each scale and jointly across both scales using the CONCH text encoder [9 and cosine similarity. These results demonstrate that the descriptions encode highly discriminative and consistent semantics across both morphological groups and scales, and across different LLMs.
Table 6: Intra- and inter-category text similarity scores. Intra denotes similarities within the same morphology concept across scales, and inter denotes similarities across different ones.
TCGA NSCLC
TCGA BRCA
LLM
5x (intra/inter)
20x (intra/inter)
x (intra/inter)
5x (intra/inter)
20x (intra/inter)
5x x0 (intrafinter)
DeepSeek
[K
(0.157)
(0. 164)
0.456 (0. (58)
16))
141)
(0.138)
Grok 3 53
1.000 (0.123)
0.511 (0.170)
0.434 (0.160)
1.000 (8.186)
0.490 (0.214)
0.415 (0.212)
Gemini 2.5 Pro 3
1.000 (0.111)
0.558 (0.165)
0.478 (0.150)
1.000 (0.206)
0.491 (0.127)
0.426 (0.142)
GPT-4o
1.000 (0.066)
0.516
0.454 (0.109)
1.000 16.1
0.477 (0.108)
0.413 (0.125)
We further validate the performance of HiVE-MIL. using descriptions generated by different LLMSs which are provided in Appendix[M]) M.2
31

---

C
Notations
Table 7: Summary of the notations. For simplicity, we describe the notation based on a head-branch.
Notation
Definition
General Notations
Number of low-scale (5x) patches per
M
Number high-scale (20x) patches per low-scale patch (default: 16)
Total number high-scale patches:
N.
Number of low-scale text prompts per class Number of high-scis text prompts per low-scale prompt
Total number high- scale text prompts:
Dimensionality of visual/textual embecklings
Y
Logit scaling factor from the pretrained VLM
Visual and Textual Features
2n
Visual feature of low-scale patch n, 2n R
20.0
(h)
Visual feature of high-scale patch m low-scale patch n
(6)
Flattened high-scale visual feature where mn.
2(
Low-scale patch features, 2() E RNDD High-scale patch features, 2th) E R
2(h)
d0)
Prompt embedding for low-scale text O
(1))
Prompt embedding for high-scale
& under low-scale o
(0]
Encoded feature of ( co R.D
6.6
(h)
Encoded feature of ",,k 6,,6
(h) (h) R
(h)
Flattened high-scale text feature where (o,
EX0
Low-scale text features, g(0) E ROxD
E(A)
High-scale text features, [ R5xD
(*)
Patch feature matrix at scale (1,h}
((s)
Text feature matrix at scale s 43
Similarity Matrices and Filtering
5()
scale similarity matrix, 500 R-x
(
High-scale similarity matrix, g(h) E RRS
s(1)
Binary mask for low-scale similarity: above μ4 a do
High-scale similarity masked hy low-scale filtering: 51h). S
maeked
(N)
Bllered
Final filtered high-scale similarity: ahove fl,
logits (5)
Logits computed by patch-text alignment at scale
Graph Definitions
GanG
Hierarchical Heterogeneous Graph Node set of GauG
V
E
Edge set of Grmo. & gintra gliie
T
Node types: (img"), img(") (ex (e)
R
Edge relation types: (intra () intra hier!ine) hier
Aggregation and Prediction
A
intra
Intra-scale aggregated node feature for node v Hierarchical aggregated node feature via MSA
Aiier
Final node feature: hy
jiittra haer
Loss Terms
Lee
Cross-entropy loss Hierarchical Text Contrastive Loss
LITTLL
Hy perparameeesss
Threshold sensitivity for TGDF filtering
Weighting factor for Lurret
To)-
Top-k similarity scores at
32

---

D Text-Guided Dynamic Filtering Pseudocode
Algorithm 1 Text-Guided Dynamic Filtering (TGDF)
Require:
Low-scale patch features 7(1) E RRNx low-scale text features E(U) E RRODD, high-scale
patch features ((h) RRxD high-scale text features E(h) RRSD,, sensitivity parameter
Ensure: Filtered similarity matrices s(t)
Liltered E TR=xxO,
(shh) (((h)
fillered E RRxs
1: Stage 1: Low-scale Filtering
2: 5()) -- cosine caieeessiillayy2222 (Z ,E!)
3: for
O do < mean( (e [ o])
4:
5: do f- std(s ol)
6: threshold, - 1 + a 0o
7: 5(()
of + (1, 2 threshold,)
8: end for
9: Stage High-scale Refinement
10: 5(h) cosine_similarde (2(),, E(h)
11: for to R do
12:
parent_patchh_ieeeee
(Parent patch index)
13: for to do
(Parent text index
15: 5(hh]
(r, (r, + 5th) (r, 8) Sitered (n, o)
end for
17: end for
for
S do
19: Ns + mean ((h)
masted s))
20: O. f- (S(h) maskes s])
21: threshold, f- Me (S(h) maskod threshold,)
((h) Fiteredlt. x-
23: end for
S(l)) (hh)
return "iltered liltere
33

---

E
Message Passing Details
HiVE-MIL introduces a newly designed hierarchical heterogeneous graph to jointly capture structural differences across scales and modalities. Previous approaches fail explicitly encode hierarchical, coarse-to-fine structures across scales. To address this limitation, we apply relation-specific message passing operators to both intra-scale and hierarchical relationships.
The message passing process as follows. During neighbor aggregation, information collected from neighboring nodes connected via specific relation type (i.e., edge type). These include intra-scale relations, such as valid patch-text connections at low or high scale, and hierarchical relations, such connections between low-scale and high-scale patches, between low-scale and high-scale text. The aggregated neighbor features are then concatenated with the node's s own features and passed through a relation-specte linear transformation (i.e., weight matrix). For intra- scale relations, we adopt the GraphS AGE [21] operator via the HeteroConv module, whereas for hierarchical relations, employ Modality-Scale Attention (MSA) mechanism that accounts for both scale directionality and modality type. Finally, non-linear activation function (ReLU) is applied enhance representational capacity. By decoupling both aggregation and transformation parameters by relation type, our model effectively encodes hetcrogeneous and hierarchical information.
F Hierarchical Text Semantic Alignment Evaluation (Hit Ratio)
Algorithm 2 Hierarchical Text Semantic Alignment Evaluation (Hit Ratio @2) 2)
Require: Low-scale patch features ( RNND high-scale patch features 2 E RNNMMD
Low-scale text features E(U) E OxD, high-scale text features ((() E RSD
Hierarchical map : {1,..., O} (Maps parent index to list of child indices)
Ensure: Hit Ratio@2 E [0, 1]
1: hit f 0
2: for 1 to N do
3: in + z
4: s(!) + cosine_ cainunssmillrttyee en _E)) ())) E R
5:
[or, 02] t- ToppIInicess(s))
6:
Cn x H((o)) U ((22) (Candidate child indices)
7:
for m = to M do
8:
2n.m + (h)
(h)
9:
(h)) + cosine_ baiieessmmllrityyaaaee E(h)) E R$
10:
S" + arg max, (h)
11:
s" E C, then
12:
hit + hit +
13:
break (One hit is enough) 14: end if
end for
16: end for
return Hit Ratio@2 hit/N
To assess whether the model preserves the intended hierarchical structure between coarse (low-scale) and fine-grained (high-scale) textual semanties, introduce quantitative evaluation metric termed Hit Ratio@2. While the main paper presents the high-level idea, we describe here the full procedure and rationale consistent with the implementation in Algorithm 2 The core objective is evaluate whether the high-scale patch features selected by the model semantically align with the child-level text descriptions that are hierarchically linked to the most relevant low-scale (parent) prompts. Each low-scale (5x) patch is associated with a parent-level textual description (c.g., Diffuse Infiltration) and each parent prompt is linked to set of fine- -ree--rriddd child prompts (e.g., Intracytoplasmic Mucin, Lack of Glandular Structures) via predefined hicrarchical mapping H. (1.....).
Step-by-step protocol. For each low-scale patch embedding 21 E ( we first compute cosine similarities with all low-scale text features E(I), yielding similarity vector g() 12 We then
34

---

select the indices of the top-2 most similar parent prompts, denoted 01 and 02. The corresponding candidate child indices are obtained as C, H((o)) U H((2).
Next, for each high-scale patch 2n,m within the spatial region of the n-th low-scale patch, we compute cosine similarities to all high-scale text features E resulting in a score vector ((h) E R We identify the top-1 most similar child text index 8 = arg max; s(h), A hit is recorded if 8* E C, and we immediately terminate the further comparisons for the remaining high-scale patches in that region (i.e., only one hit is counted per low-scale patch). The final metric is computed as the ratio of low-scale patches with at least one correct child-level alignment among the top-2 parent candidates:
Hit Ratio # of low-scale patches with valid child match
N
This procedure provides a controlled and interpretable evaluation of hierarchical alignment across visual-textual levels and validates whether the model's fine- ane-grained decisions respect the structural guidance implied by the coarse-scale semantics.
G Spatial vs. Semantic Connectivity
Although HiVE-MIL does not explicitly encode spatial adjacency, it does capture structural context indirectly through hierarchical and semantic design. At the 20x scale, each high-resolution patch is linked to its corresponding 5x parent patch via predefined spatial alignment to be reflected hierarchically. At both 5x and 20 scales, semantically related patch-teas text pairs are selected using TGDF, allowing patches to form connections through shared textual descriptions. These semantic-based connections serve intermediate mechanism for modeling inter-patch relationships beyond physical proximity.
coordinate mappings, enabling
This design reflects the characteristics of pathological images, where spatially adjacent patches often differ in tissue structure, while semantically meaningful relationships can occur across distant regions. Therefore, we construct the intra-scale graph using semantic filtering, which allows connections between distant patches that share similar meanings based on textual descriptions. We argue that simple distance-basee connections may overlook semantially important patterns.
H Implementation Details
We set the logit scaling factor 7 to 4.5871 (PLIP (25)). 4.6052 (QuiltNet (26). and 4,0315 (CONCH (3).. using the values provided by each pre-trained VLM. We retain the top-2 patches 5x and the 10p-100 patches at 20x scales to compute the final logits. Pooling- and MIL-based methods use only the image encoder, while "LM-based MIL methods leverage both image and text encoders. Most methods operate on single-scale 20x patches, whereas DSMIL (33), ViLa-MIL (48), MSCPT [22). and HiVE-MIL (Ours) utilize multi-scale inputs from both 5x and 20x patches.
Our implementation is based on the official ViLa-MIL codebase [48] and all baselines as well as HiVE-MIL are implemented within this unified framework to ensure fair comparison. Our graph- based modules are implemented using PyTorch Geometric [13]. All experiments are conducted on Ubuntu 20.04.6 using workstation equipped with NVIDIA A100 GPUs (40 GB each); however, only one GPU is used for training each model. Complete package versions and dependencies are listed in the requirements. txt file available in our GitHub repository (linked in the Abstract).
35

---

I Baselines
We compare HiVE-MIL and other baseline models using image and text encoders from the following recent vision-language pathology foundation models.
PLIP [25]: A vision-language pathology model pretrained on OpenPath, a large dataset of 208,41 414 image-text pairs curated from public platforms like medical Twitter, enabling strong zero-shot classification and case retrieval image or language queries.
QuiltNet [ A vision-language model pretrained on Quilt, a IM-pair dataset curated from YouTube, medical Twitter, and academic sources, enabling strong zero-shot classification and cross-modal retrieval across diverse histopathology datasets.
CONCH 139 A recent state-of-the-art vision-language pathology foundation model, pretrained on 1.17 million image-caption pairs from diverse biomedical sources, enabling strong zero-shot and transferable performance across 14 benchmarks spanning classification, segmentation, captioning, and retrieval tasks.
We provide details for the baselines used for the few-shot WSI classification tasks.
ABMIL [27]:
An attention-based pooling framework that assigns adaptive weights to instances for more informative bag-level aggregation.
DSMIL 33 A dual-stream MIL model where one stream identifies a critical instance via
max pooling, while the other aggregates instances by distance-weighted similarity.
weakly-supervised MII. framework that combines attention pooling with instance-level clustering enhance interpretability and slide-level prediction.
CLAM 38
TransMIL 47
transformer-bast MIL model that captures inter-instance correlations by modeling both spatial and morphological relationships among patches.
DTFD-MIL A double-tier MIL framework that introduces pseudo-bage improve training diversity and derives instance probabilities under an attention- based setting.
WiKG (34] A graph-based MIL approach that constructs WSIs as knowledge graphs with directed edges and updates features using knowledge-aware attention.
ViLa-MIL [88] dual-scale VLM-MIL model generates LLM-based prompts and uses a prototype-guided image decoder and context-guided text decoder.
MSCPT (22]: multi-scale VLM MIL model that integrates multi-scale visual inputs with LLM-generated prompts through graph-based reasoning and cross-scale aggregation.
FOCUS [19] A single-scale VLM-MIL model that integrates pathology foundation models with language priors for focused analysis of diagnostic regions.
Baselines for Main Ablation Studies
This section introduces all baselines used in the ablation studies presented in the main paper.
J.1 HTCL Variant Baselines
No-Contrastive. In the No-contrastive variant, only the standard cross-entropy loss is used; no additional loss, such as the hicrarchical text contrastive loss (HTCL), is applicd.
Share-Parent. the Share-Parent variant, the anchor defined as the mean embedding of all high-scale children sharing the same parent, i.e., T:(h) = c()) Ee (ht) where C(t) denotes the set of high-scale children for parent node i. The similarity between the anchor and each low- scale (parent) embedding is computed sim,,; = cos (T() T()). Positive pairs are defined as P: = [j Parent(i) = =}), and negative pairs as M,, = {i Parent(i) サ.}. This approach encourages the semantic alignment of all the children nodes under the same parent.
36

---

Instance- In the Instance- Wise variant, each high-scale embedding serves as the anchor. The similarity is calculated as sim:,j cOs TT))) ). with positives and negatives defined by the parent relationship. i.e., P. {j Parent(i)= = j} and Mr {i Parent(i) * j}. This strategy makes the embedding of each instance to be more similar to its own parent node.
J.2
Module Components
Disables the model's ability suppress irrelevant or weakly aligned patch-text pairs during graph construction. As a result, all patch and text nodes within each scale are densely connected in the intra-scale graph, irrespective of semantic relevance. This lack of filtering introduces spurious and noisy edges, which can degrade intra-scale alignment quality and propagate noise during message passing.
No TGDE.
No HTCL. Removes explicit supervision that aligns parent and child text embeddings across scales. The model is then trained solely with standard cross-entropy loss, without constraints that enforce hierarchical consistency between coarse- and fine-seale textual semantics. Consequently. the embeddings of low- and high-scale text nodes may become semantically misaligned, thereby weakening hierarchical text semantic aligmment.
No HHG. Eliminates all hierarchical edges connecting low- and high-scale nodes in both the visual and textual branches. The model operates with only intra-scale message passing, without leveraging the hierarchical structure. This prevents the flow of contextual information across scales and inhibits the modeling of coarse-to-fine semantic relationships between 5x and 20 features.
J.3 Hierarchical Aggregator Baselines
Scale-aware Attention (SAA). To disentangle the contribution of the modality, we define scale- aware attention by removing the modality-specific transformation (i.c., using shared weights W WR. W. for all relations). The vectors computed qu Wy(h, + sv), k. W4 (h sa).
Va = (hu + su), and the output is
1u Ru
hacale To
Σ (EN+(v)
Bou where Bou softmax
(10)
Vd
awdae Attention (MAA). To isolate the effect of scale, we define modality-aware attention by removing the scale embedding (i.e., not adding &u su). The vectors computed as Zo W Ny. hig w((r hus Vu = W hu, and the output is
T ku Le Vd
mad 9o
+
Bralay where Bor softmax
(11)
aEA.(())
Attn. For this Attn. variant, we apply standard attention ing both scale and modality types.
52 without any modifications. disregard-
GraphSAGE. For hierarchical -cale message passing, we adopt the standard GraphSAGE aggregation without any additional attention mechanism or modifications.
37

---

K Generalization to Camelyon16
To evaluate generalizability beyond TCGA datasets (BRCA, NSCLC, RCC), we further test HiVE- MIL and baselines the Camelyon16 dataset [5], using the same experimental settings as described in the main paper. HiVE-MIL achieves the highest performance, outperforming the second-best model, ViLa-MII, by 1.25% in Macro FI.
Table 8: Performance comparison on the Camelyon16 dataset (CONCH, 16-shot).
Camelyon16 B
Model
ACC
AUC
Macro Fl
ABMIL
88.33
44444
92.97
87.50 1:4.6
DSMIL 33
90.33
1:2.7
94.95
1
89.92 1:2.85
CLAM-SB
33
85.17
14.16
85.07
4-2002
80.46
+224.8
DTFD MIL (AFS)
38
91.17 4.225
93.35
13336
91,92 +:4.43
VIT a- MIL
92.33 55004
96.37 22.18
91.97
1:.22
MSCPT
88.50
88.43
1:344
88.29
FOCUS 19
90.83
14,44
91.62 1:6.2
90.23 1:4.4
HIVE-MIL
93.33 4.86
96.72
13.57
93.22 1:006
L Comparison with WSI Foundation Model
We conduct an additional experiment to compare against MADELEINE [29], a representative WSI foundation model (not MIL-based), as it is pre-trained on CONCH patch features to generate slide- level embeddings. This setup ensures a fair comparison with our method (CONCH + HiVE-MIL).
Table 9: HiVE-MIL vs. WSI Foundation Model (CONCH, 16-shot).
NSCLC
Model
ACC
AUC
Macro Fl
ACC
AUC
Macro Fl
MADELEINE 29
(Linear Probing)
83.00
90.30
83.00 24.0
80.40 1:
88.40 1:640
80.30 16.70
HIVE-MIL
90.39 1.5
96,49 10.56
90.37 11.5
87.29 1:2883
93.86 1.0.9
87.24 1285
M Further Ablations
M.1
Text Format
As detailed in Appendix A.3 the input text generated by the LLM follows a structured format consisting of term and an explanation, e.g., Dyskeratotic Cells (term) and Isolated eosinophilic tumor cells undergoing premature keratinization, appearing as dense, glassy bodies within cell clusters (explanation). To evaluate the robustness of HiVE-MIL different textual input formats, conduct ablation studies using three variants: (i) term only, (ii) explanation only (denoted as Exp.), and (iii) + explanation, which serves the default format in our main experiments. As shown in Table 10 the term + explanation format achieves the highest performance on both the TCGA NSCLC and BRCA datasets. The term-only and explanation-oni variants also perform competitively, with only marginal degradation. These results confirm that HiVE-MIL is robust to variations in textual input structure and suggest that combining concise terms with descriptive context yields more effective guidance.
Table 10: Text format ablation (CONCH, 16-shot).
TCGA NSCLC
TCGA BRCA
Text Format
ACC
AUC
Macro Fl
ACC
AUC
Macro Fl
Term
89.55 +:4.77
96.92 +0.70
89,50 4:434
86.64 12000
94.22 11.12
86.57 +:000
Exp.
90.00 12.,3
96.47 20.6
89.99 4285
86.77 12.80
93.35 11.01
86.66 22.95
Exp.
40.56
90.37 +18
+2883
38

---

M.2 Robustness to LLM Variants
To evaluate the robustness of HiVE-MIL to variations in textual input, we generate descriptions
using four different LLMs: DeepSeek RI [18], Grok 3 [53]. Gemini 2.5 Pro [3], and GPT-4o [.. The results in Table T1 demonstrate that HiVE-MII. performs robustly across all LLMs. Although GPT-40 yields the best overall results, the performance differences among the L.MSs relatively small, indicating that HiVE-MIL does not heavily depend on a specific L.M.. Importantly, regardless of the LLM used generate textual descriptions, HiVE-MIL consistently outperforms all baseline methods. We expect performance to improve further as pathology-specie LLMs become available in the future, since the generated texts will be more accurate and less biased.
Table 11: LLM variants ablation (CONCH, 16-shot).
TCGA NSCLC
LLM
ACC
AUC
Macro F1
ACC
AUC
Macro F1
SeepSeck Ri 78
89.36 188
95.77
11.68
89.33 1.99
85.60 +4422
93.08 12.43
85.49
24436
Grok 3 53
90.19 2233
96.33
11.00
90.18
12,33
85
1:3.27
93.62 11.80
85.71 1:1.32
Gemini 2.5 Pro 3
90.00 -252
96.47 1:0.99
89.98 2:255
86.65 4-2100
93.08 1.79
86.61 4:.10
GP
90.39
-1.5
96.49 1.056
90.37 1:.58
87.29 2
93.86 +0.89
87.24 #2.85
M.3 TGDF Component
To evaluate the contribution of individual components in the Text-Guided Dynamic Filtering (TGDF) module, we conduct ablation studies by selectively disabling its submodules. The first variant (w/o Mask Prop. + Low Fil.) removes both cross-scale mask propagation and threshold-based filtering at the low scale, retaining only normalization for the low-scale similarity matrix while preserving high-scale thresholding. The second variant (w/o Mask Prop.) disables only the cross- scale mask propagation, keeping threshold filtering active at both scales. The full TGDF configuration incorporates all components: cross-scale mask propagation, low-scale filtering, and high-scale filtering. As shown in Table [2] this complete TGDF module achieves the best performance across the TCGA NSCLC and BRCA datasets. These results confirm that both low-scale filtering and high-scale guidance via propagated masks are essential for effective multi-scale visual-textn alignment.
Table 12: TGDF component ablation (QuiltNet, 16-shot).
TCGA NSCLC
TCGA BRCA
ACC
AUC
ACC
AUC
TGDF (wlo Musl: Prop. Low Fil.)
76.55 14.43
84.67 +3.92
76.06 +4.40
75.79 +4.27
83.95 14.06
75.46 2:424
TGDF (wlo Mask Pmp.)
77.63
85.51 1604
77.57 77004
76.63 13.42
84,11 12.30
76.45 1:447
TGDF (Ours)
79.23 12200
87.34 14008
79.09 12.75
77.08 1:300
84.31 1:422
76,80 1:415
M.4 GNN vs. Alternative Interaction Models
HiVE-MIL is designed to model WSIs that inherently involve heterogeneous modalities (e.g., visual and textual features) and multiple scales (e.g., 5x and 20x). To reason over both intra-scale interactions and coarse-to-fine hierarchical relationships, the model must operate on complex and structured representations. GNNs offer a strong inductive bias for this purpose, as they are well suited to capturing such structured dependencies 4 58). Through localized message passing conditioned on edge types, GNNs enable HiVE-MIL to semantically align visual and textual nodes while preserving structural granularity across scales. In contrast, attention- or MLP-based approaches are limited in their capacity to effectively capture dependencies between nodes, and they struggle to incorporate prior structural knowledge, such as semantic links across modalities or hierarchical relationships across scales.
To verify the necessity of GNNs, we replace them with MLP or attention modules in both the intra-scale and hierarchical components and report the results in T3] The comparison confirms that GNNs play crucial role in integrating intra-scale and hierarchical information, while simpler alternatives fall short in capturing these relationships effectively.
39

---

Table 13: GNN vs. alternative interaction models (PLIP, 16-shot).
Module
TCGA NSCLC
TCGA BRCA
Intra-scale
Hierarchical
ACC
AUC
Macro F1
ACC
AUC
Mucro F1
MI.P
MI.P
69.04 10.55
76.15 13.56
65.69 16.73
63.91 12.50
58.88 1351
58.96
1:552
Attention
Attention
71.62 19.3
78.54
7.96
68.24 15.96
67.76
1:304
69.24 1:400
64.1 +3383
Graph
MLP
74.21
81.04
71.13
71.03
Graph
Atiention
76.99 +:223
84,50 +3.46
76.89
+:324
73.11 +3383
81.41 +:224
72.96 +:355
Graph
14773
2276
14472
13367
M.5
Single-Scale vs. Multi-Scale Interaction
To assess the contribution of scale-level interaction, we evaluate three configurations: (a) using only low-scale features, (b) using only high-scale features, and (c) combining both through multiscale integration. Table [14] highlights the consistent superiority of the multi-scale configuration over both single-scale variants on the TCGA NSCLC and BRCA datasets. Low-scale features primarily encode coarse structural context, while high-scale features capture fine-grained morphological detail. Limiting the model single scale restricts its ability fully exploit the semantic richness of WSI data. The superior performance achieved by the multi-scale setting validates our design choice to model hierarchical relationships and highlights the importance of integrating complementary information across scales in visual-textual alignment.
Table 14: Single multi-scale interaction (CONCH, 16-shot).
Instance
Logit
TCGA
TCGA BRCA
Low High
Low High
ACC
AUC
Macro F1
ACC
AUC
Macro
(6)
/
X
X-x x
84,99 12.0
89.23 1:153
84.92 12.17
79.17 13.49
85.45 11:13
78.97 1335
(b)
86.17 12222
91.33 1:105
86.15 11.22
83.17 1:2.0
87.36 1:100
83.05 1:997
90.39 1:577
96.49 4-0.56
90.37 +1.58
87.29 4-283
93.86 +-0.89
87.24 +2885
N Hyperparameter Sensitivity
N.1 TGDF Hyperparameter
The TGDF module uses hyperparameter a to compute the threshold that determines the sensitivity of patch-text alignment filtering at both low and high scales. Specifically, a influences the number of retained entries in the similarity matrices s(!) and 5(h) litered thereby affecting which visual-textual pairs preserved for downstream processing. Table 15 showhw how overall performance varies with different TGDF filtering thresholds. To clarify, = O does not imply "no TGDF"; rather, it sets the filtering threshold as μ + a, meaning the threshold equals μ when proportion of filtered patch-text pairs, we calculate the filtered-pair ratio per WSI, average it across WSIs, and report the mean over five runs.
To compute the
When TGDF is not applied at all, performance is lowest due to aligning all image-text pairs, which results in incorreet image-ter -ext pairings. As increases, more patch- text pairs are filtered out at both scales, leading to improved performance, with a = 0.5 yielding the best result, indicating that TGDF effectively removes semantically irrelevant image-text pairs. However, when 1, filtering becomes too aggressive, discarding many meaningful pairs and reducing performance.
Table 15: TGDF threshold a hyperparameter sensitivity (QuiltNet, 16-shot).
TCGA NSCLC
TCGA BRCA
Filtered Pairs (5 x20x)
Macro F1
Filtered (5x/20x)
Macro
No TGDF
0% 70%
75.13 + 4,21
0% 0%%
74.24 + 5.27
17.21%
12.46%
4.10
0.5
28. 25% / 30.022%
78.52
1 3.67
30.11% /31.0555
76.17
4.68
46.34% /1
79.06
48.53%
3.47
0.5
67.02% 70.65%
79.09 = 2.75
67.92% 77.80%
76.30
4.15
85.21% 86.03%
78.55 = 4.52
84.16% 85.43%
76.09 3.41
40

---

N.2 HTCL Hyperparameter
The total loss defined as the sum of the standard cross-entropy loss (CE) and the hierarchical text contrastive loss (HTCL), i.e., Lioaal = LCe + ACHICL where À controls the influence of HTCL. To assess the sensitivity of the model performance this hyperparameter, we perform an ablation study by varying À (0, 0.1, 0.5, 1). Incorporating HTCL (A > 0) consistently improves performance over the baseline (λ 0), confirming the benefit of hierarchical text supervision (TTble T6p. All non-zero values yield comparable results, with À 0.5 achieving the best overall performance. These results demonstrate that HTCL robust hyperparameter selection and effective guiding multi-scale visual-textual alignment.
Table 16: HTCL hyperparameter sensitivity (PLIP, 16-shot).
TCGA NSCLC
TCGA BRCA
ACC
AUC
Macro F1
ACC
AUC
Macro F1
0
78.14
13.55
86.02
13889
78.11
33.
73.96 1:4.42
80.50
16.02
73.81
::4.6
0.1
79.30 +:4.00
86.19 4-3.0
79.26 3398
74.50 4:3.6
82.08
46.51
74.22
#:391
0.5
80.13
4:473
87.28 1.776
80.08 4.73
75.21 #351
83.19 14772
74.99 1367
79.46
8:3.78
86.62
83.13
78.37
33788
75.16
4:3.61
82.94 4:3.44
74.94
+3.68
N.3 Top-k Logit Hyperparameter
To further assess the robustness of HiVE-MIL with respect to the top-k hyperparameters used for logit computation, we conduct sensitivity analysis by varying the number of selected patches at both scales. Specifically, we vary K E (2,5. 10} at the low scale and N E (50, 100, 200} at the high scale. The results are shown in Figure g using a bar plot to depict performance in combinations of (K, N). The variation in performance across the nine configurations is minimal, indicating that HiVE--III is robust and largely insensitive the specific choice of K and N within the tested ranges.
Plot (K
AUC
10g
(High)
(Low)
50
Figure 8: AUC sensitivity top-k logit selection low (K) and high (N) scales (TCGA BRCA.
PLIP, 16-shot). HiVE-MIL. shows stable performance across all (K, N) combinations.
41

---

O
Computational Efficiency & Scalability
We evaluate HiVE-MIL along with comparison models (2024 onwards) in terms of FLOPs, inference time, and maximum GPU memory usage. The computation time is measured using an NVIDIA A100 GPU. To evaluate runtime overhead in realistic setting, we first use WSI from TCGA BRCA containing 1,880 patches at 5 and 28,249 patches at 20 (Left). For scalability, we also report results on the largest WSI in the dataset, which contains 2,633 patches and 40,777 patches at 20x (Right).
Table 17: Computational efficiency and scalability.
(5x: 1880, 20x: 28,249 patches)
WSI 2 (5x: 2.633, 20x: 40.777 patches)
FLOPs (G)
Inference
Max GPU
FLOPs
Inference
Max GPU
Memory Usage (MB)
(G)
Time
Memory Usage (MB)
WKGG 24
891.82
0.0775
4630.56
1810.44
0.1539
8617.82
ViLa-MLL H8
87.65
0.0242
528.
116.00
0.0199
679.41
MSCPT 12
860.21
0.1189
2074.8 8774.89
894.17
0.1368
2695 .2
FOCUS (19
HiVE-MIL
624.74
0.2037
4738.36
738.26
0,2583
6456.60
Optimizations planned. Working with WSIs is undeniably challenging due to their gigapixel size. To make our method more practical for large-scale deployment, we will further develop parallel version of HiVE-MIL that can better handle the computational load. This would help reduce processing time and make the model more efficient for deployment clinical settings.
P Potential Societal Impact
Positive Impacts. HiVE-MIL enables data-efficient, few-shot classification of WSIs by modeling hierarchical dependencies and intra-scale multimodal alignments through a unified hierarchical heterogeneous graph. This approach has the potential to improve diagnostic support in resource- limited settings, where annotated datasets and expert pathologists are often unavailable. By aligning visual patches with descriptive prompts on multiple scales, the model improves interpretability and can help clinicians make more informed decisions. Beyond pathology, hierarchical design should also benefit other domains that involve limited data and structured semantic inputs.
Negative Impacts. The model relies on --M-generated prompts to construct hierarchical text de- scriptions, which may introduce factual errors or biases, particularly in the absence of expert oversight. In clinical settings, excessive reliance on automatically generated outputs without independent review and validation can lead to biased misleading results. This underscores the importance of expert participation in ensuring both the accuracy and reliability of the model's explanations.
42


Reviews:
[{"review_id": "Fap2nV5DBS", "reviewer": "Reviewer_6Jif", "summary": "This paper proposes HiVE-MIL, a hierarchical vision-language framework that constructs a unified graph with (1) parent-child links between coarse (5x) and fine (20x) visual/textual nodes to capture hierarchical relationships, and (2) heterogeneous intra-scale edges linking visual and textual nodes at the same scale. To enhance semantic consistency, HiVE-MIL incorporates a two-stage, text-guided dynamic filtering mechanism to remove weakly correlated patch-text pairs and introduces a hierarchical contrastive loss to align textual semantics across scales. Extensive experiments on TCGA breast, lung, and kidney cancer datasets demonstrate its effectiveness.", "questions": "Besides the concerns mentioned in the weaknesses section, there is another issue:\nThe description of coarse and fine morphological features is crucial in the proposed framework, and these features are generated by the LLM. Is there any validation or guarantee regarding the accuracy of the content generated by the LLM?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1) the authors did take the hierarchy of features (5X, 20X) into consideration, which is one of the most important things in analyzing pathological images.\n2) there is a carefully designed graph-based module that could connect visual and textual nodes on the same scale\n3) the models they compared with are selected from the top-tier journal and trained with large amount of data. The experiments could clearly demonstrate the superior performance of the proposed method.\n\nMajor Weaknesses:\n1) the author only mentioned 5X and 20X. In real-world scenario, there are 10X and 40X. Is there any way of handling these?\n2) there is some methods that could handle giga pixel without MIL-related method, like  Giga-Path. I would like to see some comparison experiment on those methods.\n\nMinor Problems:\n1) font size of Figure 2 is too small.", "was_revised": true, "final_justification": "Given the original quality of the paper and the response experiments, this paper could be accepted.", "rebuttal": "We thank you for acknowledging the importance of our hierarchical design, the careful graph-based visual-textual modeling, and the strong experimental comparisons. We provide the following responses to address your concerns.\n\n| **W1. Other scales (10x and 40x)**|\n|---|\nWe chose the (5x, 20x) combination because it is the **most commonly used setting** in multi-scale MIL research. This setup has been widely adopted in prior work, such as MSCPT \\[1\\]. Thus, we followed it for consistency and fair comparison. Moreover, our framework is flexible and can be easily adapted to other scale pairs, such as (10x, 40x), using the same hierarchical design.\n\nFor the combination of using all of the four scales (5x, 10x, 20x, 40x), the proposed method can support this configuration, although adding more scales naturally increases model complexity, inference time, and memory usage, as is also the case for other multi-scale MIL models. However, HiVE-MIL mitigates this overhead through TGDF, which filters out semantically meaningless image–text pairs at each scale and retains only the meaningful ones for constructing heterogeneous intra-scale edges. TGDF also operates in a *top-down* manner, meaning that when a low-scale patch such as 5x is removed, all its corresponding higher-scale patches (10x, 20x, 40x) are automatically discarded, thereby avoiding unnecessary computation. The parent–child mapping between scales is determined by the square of the scale ratio. For example, a single 5x patch corresponds to 4 patches at 10x $( (10/5)^2 = 4 )$ and 16 patches at 20x $( (20/5)^2 = 16 )$. Additional text prompts can also be generated for each scale, although this requires creating separate scale-specific descriptions.\n\n\\[1\\] Han et al. MSCPT: Few-shot Whole Slide Image Classification with Multi-scale and Context-focused Prompt Tuning, IEEE TMI 2025\\.\n\n| **W2. Comparison with non MIL-related method**|\n|---|\nFollowing the comment, we have conducted an additional experiment to make a comparison with MADELEINE \\[2\\] as the representative of the non-MIL related method instead of GigaPath for the following reasons:\n \n(1) GigaPath’s slide encoder requires 1536-dim inputs from its own patch encoder, which is incompatible with the 512-dim features (PLIP, QuiltNet, CONCH) used in our work.\n\n(2) MADELEINE is pre-trained on CONCH patch features to create a slide embedding, making the comparison with (CONCH + HiVE-MIL) fair.\n\n(3) MADELEINE has been shown to outperform GigaPath in its original paper, making it a stronger baseline. \n\nThe experimental results are as follows (CONCH, 16-shot)**:**\n\n| | **TCGA NSCLC** |  |  | **TCGA BRCA** |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Model**  | **ACC** | **AUC** | **Macro F1** | **ACC** | **AUC** | **Macro F1** |\n| MADELEINE (Linear Probing) | 83.00 ± 4.10  | 90.30 ± 3.50 | 83.00 ± 4.10 | 80.40 ± 6.70 | 88.40 ± 6.40 | 80.30 ± 6.70 |\n| **HiVE-MIL** | **90.39 ± 1.57** | **96.49 ± 0.56** | **90.37 ± 1.58** | **87.29 ± 2.83** | **93.86 ± 0.89** | **87.24 ± 2.85** |\n\nOverall, our method significantly outperforms MADELEINE (Linear Probing) on both datasets. We will add these additional experimental results to our final version of the paper.\n\n\\[2\\] Jaume et al. Multistain Pretraining for Slide Representation Learning in Pathology, ECCV 2024\\.\n\n| **Minor. Font size of Figure 2**|\n|-|\nThank you for pointing this out. We will update the font size in Figure 2 to improve readability in the final version.\n\n| **Q1. Validation of LLM-generated descriptions**|\n|---|\nThank you for asking this important question. While the descriptions generated by the LLM have not yet been verified by pathologists, we address this limitation through a series of alternative validation strategies designed to assess their class-discriminative relevance (1) and semantic reliability (2).\n\n(1) We evaluate the *faithfulness* of the generated descriptions using an \"LLM-as-a-Judge\" \\[3\\], where LLMs are prompted to infer class labels based solely on the generated descriptions. We employ two evaluators: GPT-4o and GPT o3, to assess descriptions generated by several LLMs: Deep Seek R1, Grok 3, Gemini 2.5 Pro, and GPT-4o. Although not perfect, the number of correct predictions is still consistently high, suggesting that the generated text is generally reliable while capturing class-discriminative signals.\n\n\n\n| |  | **Evaluator: GPT-4o** |  | **Evaluator: GPT o3** |  |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| **Generator**  |**\\#Total Descriptions (\\#5x/\\#20x)**  | **TCGA NSCLC** | **TCGA BRCA** | **TCGA NSCLC** | **TCGA BRCA** |\n| DeepSeek R1 | 32 (8/24) | 28 (8/20) | 28 (8/20) | 31 (8/23) | 28 (8/20) |\n| Grok 3 | 32 (8/24) | 28 (7/21) | 28 (8/20) | 32 (8/24) | 21 (8/23) |\n| Gemini 2.5 Pro | 32 (8/24) | 30 (8/22) | 29 (7/22) | 30 (7/23) | 32 (8/24) |\n| GPT-4o | 32 (8/24) | 30 (8/22) | 30 (7/23) | 31 (8/23) | 32 (8/24) |\n| Total | 128 (32/96) | 116 (31/85) | 114 (29/85) | 124 (31/93) | 113 (32/89) |\n\n\\[3\\] Gu et al. A Survey on LLM-as-a-Judge, arXiv 2025\\.\n\n(2) The table below reports intra- and inter-category text similarity scores, where *intra* denotes similarities among descriptions of the same morphology or concept, and *inter* denotes similarities across different ones. Similarity is computed separately at each scale and jointly across both scales using the CONCH text encoder and cosine similarity. These results highlight that the descriptions encode highly discriminative and consistent semantics for each morphological or conceptual group, with robustness across scales and LLMs. \n\n\n\n| | **TCGA NSCLC** |  |  | **TCGA BRCA** |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **LLM**  | **5x (intra/inter)** | **20x (intra/inter)** | **5x,20x (intra/inter)** | **5x (intra/inter)** | **20x (intra/inter)** | **5x,20x (intra/inter)** |\n| DeepSeek R1 | 1.000 (0.157) | 0.515 (0.164) | 0.456 (0.158) | 1.000 (0.165) | 0.438 (0.141) | 0.412 (0.138) |\n| Grok 3 | 1.000 (0.123) | 0.511 (0.170) | 0.434 (0.160) | 1.000 (0.186) | 0.490 (0.214) | 0.415 (0.212) |\n| Gemini 2.5 Pro | 1.000 (0.111) | 0.558 (0.165) | 0.478 (0.150) | 1.000 (0.206) | 0.491 (0.127) | 0.426 (0.142) |\n| GPT-4o | 1.000 (0.066) | 0.516 (0.114) | 0.454 (0.109) | 1.000 (0.163) | 0.477 (0.108) | 0.413 (0.125) |\n\nFurthermore, we validated the performance of HiVE-MIL using descriptions generated by different LLMs, as detailed in Supplementary Material H.5.\n\nAgain, we thank you for your thoughtful and constructive comments."}, {"review_id": "RM3dvgVPfJ", "reviewer": "Reviewer_S9yU", "summary": "The paper proposes HiVE-MIL, a hierarchical vision-language framework for few-shot weakly supervised classification of whole slide images (WSIs) in computational pathology. Existing methods often fail to model the hierarchical relationships within visual and textual modalities across different magnifications and lack effective alignment between modalities at the same scale. To overcome these limitations, HiVE-MIL constructs a hierarchical heterogeneous graph that incorporates parent–child links between low- and high-magnification nodes and connects visual and textual nodes within the same scale. The results demonstrate the value of jointly modeling hierarchical structure and multimodal alignment for efficient and scalable learning from limited pathology data.", "questions": "- What is the impact of the vision-language backbone choice? The paper evaluates HiVE-MIL with multiple pretrained VLMs, but the architectural changes are tightly coupled with their embedding quality. How sensitive is the proposed framework to the backbone choice? \n- Could the authors elaborate on computational efficiency and scalability? The proposed hierarchical graph structure and attention mechanisms introduce additional complexity compared to standard MIL. Can the authors provide more insight into the runtime and memory overhead, especially for larger WSIs? Are there any bottlenecks or optimizations planned for deployment in clinical settings?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The experimental evaluation is comprehensive, covering three TCGA datasets and multiple vision-language models. Ablation studies and other analysis further support the robustness and effectiveness of the proposed method.\n- The proposed HiVE-MIL framework integrates well-established components with novel adaptations.\n- The motivation is clear, and the methodology is logically developed. \n\n**Weaknesses:**\n- While the empirical results are strong, the reliance on frozen vision-language backbones may limit its adaptability. \n\nIn summary, the paper has high technical quality, clear contributions, and practical relevance, though future work may need to address limitations in adaptivity and complexity.", "was_revised": true, "final_justification": "The paper is solid and my concerns are solved by the authors' rebuttal. My final rating is 5: Accept.", "rebuttal": "We thank you for the positive feedback and for recognizing the high technical quality, clear contributions, and practical relevance of our work. We provide the following responses to address your concerns.\n\n| **W1. Reliance on frozen vision-language backbones**|\n|---|\nWe appreciate the comment and agree that freezing the VLM backbones may limit their adaptability. Also, please note that, in few-shot settings, where only a small amount of training data is available and due to the risk of overfitting, it is common to use train-time adaptation methods such as CoOp \\[1\\], which adapt pre-trained (frozen) VLMs to downstream tasks by updating only a small number of parameters. Because methods like CoOp are effective only for single images (i.e., patch-level), we transfer this knowledge to the WSI level and leverage pathology-specific prior knowledge from the pre-trained VLMs by integrating CoOp with our proposed HiVE-MIL framework, which effectively models the hierarchical and contextual information present in WSIs.\n\n\\[1\\] Zhou et al. Learning to Prompt for Vision-Language Models, IJCV 2022\\.\n\n| **Q1. On sensitivity to vision-language backbone choice**|\n|---|\nHiVE-MIL is designed to be modular and independent of any specific vision-language backbone. Its core components, including the hierarchical heterogeneous graph, TGDF, and HTCL, operate separately from the embedding source and consistently enhance representation quality. To assess sensitivity, we evaluated HiVE-MIL using three pathology-specific VLMs trained on progressively larger pre-training datasets: PLIP (208K pairs), QuiltNet (1M pairs), and CONCH (1.17M pairs). Across all of the backbones and datasets, HiVE-MIL consistently outperforms baseline methods. Even when using PLIP, which is pre-trained on the smallest dataset, HiVE-MIL achieves a 3.26% improvement in macro F1 over the second-best method on the TCGA NSCLC dataset (Table 1, main paper). These results demonstrate that HiVE-MIL is robust to variations in VLM quality. We also expect its performance to further improve as stronger pathology-specific VLMs become available.\n\n| **Q2. Computational efficiency and scalability**|\n|---|\nWe have evaluated HiVE-MIL along with comparison models (2024 onwards) in terms of FLOPs, inference time, and maximum GPU memory usage. The computation time was measured using an NVIDIA A100 GPU. To evaluate runtime overhead in a realistic setting, we first used a WSI from TCGA BRCA containing 1,880 patches at 5x and 28,249 patches at 20x (Left). For scalability, we also report results on the largest WSI in the dataset, which contains 2,633 patches at 5x and 40,777 patches at 20x (Right).\n\n\n\n| | **# 5x: 1,880, #20x: 28,249** |  |  | **5x: 2,633, #20x: 40,777** |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Model** | **FLOPs (G)** | **Inference Time (s)** | **Maximum GPU Memory Usage (MB)** | **FLOPs (G)** | **Inference Time (s)** | **Maximum GPU Memory Usage (MB)** |\n| WiKG | 891.82 | 0.0775 | 4630.56 | 1810.44 | 0.1539 | 8617.82 |\n| ViLa-MIL | 87.65 | 0.0242 | 528.20 | 116 | 0.0199 | 679.41 |\n| MSCPT | 860.21 | 0.1189 | 2074.89 | 894.17 | 0.1368 | 2695.92 |\n| FOCUS | 39.52 | 1.4500 | 353.21 | 46.20 | 1.9484 | 429.51 |\n| HiVE-MIL | 624.74 | 0.2037 | 4738.36 | 738.26 | 0.2583 | 6456.6 |\n\n**Optimizations planned.** Working with WSIs is undeniably challenging due to their gigapixel size. To make our method more practical for large-scale deployment, we will further develop a parallel version of HiVE-MIL that can better handle the computational load. This would help reduce processing time and make the model more efficient for deployment in clinical settings.\n\nAgain, we thank you for your thoughtful and constructive comments."}, {"review_id": "COURfQW6Jy", "reviewer": "Reviewer_FSfv", "summary": "This work introduces HiVE‑MIL, a novel framework for few‑shot, weakly supervised whole‑slide image classification that builds a hierarchical vision‑language graph to jointly model coarse‑to‑fine tissue structures and align visual/textual modalities; it incorporates a text‑guided dynamic filtering module and a hierarchical contrastive loss to refine patch–text pairings and enforce cross‑scale semantic consistency, and on TCGA breast, lung, and kidney cancer datasets under 4‑, 8‑, and 16‑shot settings, HiVE‑MIL outperforms both traditional and recent VLM‑based MIL methods by up to 4.1 % in macro F1 (16‑shot) .", "questions": "1.The paper introduces a relation‑specific GraphSAGE operator but does not detail its internal steps. Could you provide a concise illustration or pseudocode showing how neighbor aggregation, relation‑specific transformations, and activation functions are composed within this operator?\n2.Your Text‑Guided Dynamic Filtering (TGDF) and Hierarchical Text‑Contrastive Loss (HTCL) rely on textual features generated by a vision‑language model. Could you elaborate on the exact VLM used, the prompt‐engineering process, and how sensitive your results are to different prompt templates or VLM backbones?\n3.GNN necessity: The model relies on a hierarchical graph neural network to model inter‑ and intra‑scale interactions—could you clarify why a GNN is the most appropriate mechanism here, and whether alternative interaction schemes (e.g., attention‑based fusion or message passing via simpler MLPs) were considered?\n4.You set the TGDF filtering threshold α=0.5 by default—can you share how this value was chosen, what proportion of patch–text pairs it typically filters out, and how sensitive overall performance is to variations in α?", "limitations": "While proposing a GNN‑based hierarchical graph, the paper lacks discussion on why graph neural networks are necessary versus simpler fusion mechanisms. The authors should acknowledge this choice’s assumptions and consider evaluating alternative interaction models.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1.Thoughtful integration of modalities: The paper is the first to integrate a hierarchical heterogeneous graph structure with vision‑language models (VLMs), enabling coarse‑to‑fine modeling of tissue architecture in WSIs.\n2.Sound design: The combination of Text‑Guided Dynamic Filtering (TGDF) and Hierarchical Text‑Contrastive Loss (HTCL) effectively improves patch–text alignment and enforces cross‑scale semantic consistency.\n3.Comprehensive evaluation: Experiments on TCGA breast, lung, and kidney cancer datasets under 4‑, 8‑, and 16‑shot settings show consistent performance gains, with up to a 4.1 % macro F1 improvement in the 16‑shot scenario.\nWeaknesses\n1.Lack of theoretical justification for using GNNs: The paper does not sufficiently argue why a graph neural network is the necessary choice to model the proposed relationships, nor whether alternative interaction mechanisms might be more efficient.\n2.Insufficient hyperparameter analysis: Key settings such as the TGDF threshold (α = 0.5) are not justified; it is unclear what proportion of patch–text pairs this value filters and how sensitive performance is to this choice.\n3.Dependence on text quality: The effectiveness of TGDF and HTCL hinges on the quality of the generated textual prompts; any biases or inaccuracies in VLM‑generated text could degrade performance.", "was_revised": true, "final_justification": "My concerns have been resolved. The paper is technically acceptable.", "rebuttal": "Thank you for your positive feedback and for recognizing the thoughtful integration of modalities, sound design, and comprehensive evaluation in our work. We provide the following responses to address your concerns.\n\n| **W1/Q3/L1. Justification for using GNNs**|\n|--- |\nWe appreciate the reviewer’s thoughtful question. HiVE-MIL is designed to model WSIs that inherently involve heterogeneous modalities (e.g., visual and textual features) and multiple scales (e.g., 5x and 20x). To reason over both intra-scale interactions and coarse-to-fine hierarchical relationships, the model must operate on complex and structured representations. GNNs offer a strong inductive bias for this purpose, as they are well suited to capturing such structured dependencies \\[1,2\\]. Through localized message passing conditioned on edge types, GNNs enable HiVE-MIL to semantically align visual and textual nodes while preserving structural granularity across scales.\n\nIn contrast, attention- or MLP-based approaches are limited in their capacity to effectively capture dependencies between nodes, and they struggle to incorporate prior structural knowledge, such as semantic links across modalities or hierarchical relationships across scales.\n\n\\[1\\] Battaglia et al. Relational inductive biases, deep learning, and graph networks, arXiv 2018\\.  \n\\[2\\] Zhou et al. Graph Neural Networks: A Review of Methods and Applications, AI Open 2021\\.\n\n**GNN vs. alternative interaction models.** To verify the necessity of GNNs, we replace them with MLP or attention modules in both the intra-scale and hierarchical components and report the results below (PLIP, 16-shot). The comparison confirms that GNNs play a crucial role in integrating intra-scale and hierarchical information, while simpler alternatives fall short in capturing these relationships effectively. \n\n|  |  |  |  |  |  |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Module** |  | **TCGA NSCLC** |  |  | **TCGA BRCA** |  |  |\n| **Intra-scale** | **Hierarchical** | **ACC** | **AUC** | **Macro F1** | **ACC** | **AUC** | **Macro F1** |\n| MLP | MLP | 69.04 ± 10.55  | 76.15 ± 8.56 | 65.69 ± 16.73 | 63.91 ± 2.50 | 58.88 ± 3.51 | 58.96 ± 3.52 |\n| Attention | Attention | 71.62 ± 9.83 | 78.54 ± 7.96 | 68.24 ± 15.96 | 67.76 ± 3.04 | 69.24 ± 4.03 | 64.11 ± 3.82 |\n| Graph | MLP | 74.21 ± 9.11 | 81.04 ± 7.36 | 71.13 ± 15.18 | 71.67 ± 3.57 | 80.56 ± 4.55 | 71.03 ± 4.11 |\n| Graph | Attention | 76.99 ± 3.23 | 84.50 ± 3.46 | 76.89 ± 3.24 | 73.11 ± 3.83 | 81.41 ± 2.24 | 72.96 ± 3.55  |\n| **Graph** | **HGNN** | **80.13 ± 4.73**  | **87.28 ± 2.76** | **80.08 ± 4.73** | **75.21 ± 3.51** | **83.19 ± 4.72** | **74.99 ± 3.67** |\n\nWe will add these rationales and experimental results to the final version of the paper.\n\n| **Q1. Details on the message passing**|\n|--- |\nWe apologize for not clearly detailing the steps. This study introduces a newly designed heterogeneous hierarchical graph to jointly capture structural differences across modalities and scales. While prior approaches may distinguish between modalities, they often fail to explicitly encode hierarchical, coarse-to-fine structures across scales. To address this limitation, we apply relation-specific message passing operators to both intra-scale and hierarchical relationships.\n\nThe message passing process is as follows. During neighbor aggregation, information is collected from neighboring nodes connected via a specific relation type (i.e., edge type). These include *intra-scale relations*, such as valid patch-text connections at low or high scale, and *hierarchical relations*, such as connections between low-scale and high-scale patches, or between low-scale and high-scale text. The aggregated neighbor features are then concatenated with the node’s own features and passed through a relation-specific linear transformation (i.e., weight matrix). For intra-scale relations, we adopt the GraphSAGE operator via the HeteroConv module, whereas for hierarchical relations, we employ a Modality-Scale Attention (MSA) mechanism that accounts for both scale directionality and modality type. Finally, a non-linear activation function (ReLU) is applied to enhance representational capacity. By decoupling both aggregation and transformation parameters by relation type, our model effectively encodes heterogeneous and hierarchical information. The corresponding pseudocode will be included in the final version.\n\n| **W2/Q4. TGDF filtering threshold**|\n|--- |\nThe results below (QuiltNet, 16-shot) shows how overall performance varies with different TGDF filtering thresholds. To clarify, α \\= 0 does not imply \"no TGDF\"; rather, it sets the filtering threshold as μₒ \\+ α·σₒ (Section 3.2, Main Paper), meaning the threshold \\= μₒ when α \\= 0\\. To compute the proportion of filtered patch-text pairs, we first calculate the ratio of filtered pairs for each WSI and then take the mean across WSIs. The reported value is the average of these means over five independent runs.\n\nWhen TGDF is not applied at all, performance is lowest due to aligning all image-text pairs, which would result in incorrect image-text pairings. As α increases, more patch-text pairs are filtered out at both scales, leading to improved performance, with α \\= 0.5 yielding the best result, indicating that TGDF effectively removes semantically irrelevant image–text pairs. However, when α \\= 1, the filtering becomes too aggressive, discarding many meaningful pairs and reducing performance.\n\n\n\n|| **TCGA NSCLC** |    | **TCGA BRCA** |  |\n| :---: | :---: | :---: | :---: | :---: |\n |  **α**|   **Filtered Pairs (5x/20x)** |   **Macro F1** | **Filtered Pairs (5x/20x)** |  **Macro F1** |\n| No TGDF  | 0% / 0% | 75.13 ± 4.21 | 0% / 0% | 74.24 ± 5.27 |\n | \\-1 | 11.58% / 17.21% | 77.02 ± 3.25 | 12.46%/16.85% | 75.12 ± 4.10 |\n|  \\-0.5 | 28.25% / 30.02% | 78.52 ± 3.67 | 30.11% / 31.05% | 76.17 ± 4.68 |\n|   0 | 46.34% / 51.23% | 79.06 ± 4.40 | 48.53% / 52.45% | 76.00 ± 3.47 |\n|   0.5 | 67.02% / 70.65% | 79.09 ± 2.75 | 67.92% / 71.80% | 76.80 ± 4.15 |\n|  1 | 85.21% / 86.03% | 78.55 ± 4.52 | 84.16% / 85.43% | 76.09 ± 3.41 |\n\n| **Q2. Details on text generation**|\n|--- |\nThank you for the insightful question. We would like to first clarify that the textual features used in TGDF and HTCL are generated by the LLM, not VLM. Specifically, we use GPT-4o as the primary model (note: we also experimented with several other LLMs, as shown below) to generate text descriptions using the prompt detailed in Section 3.1 of the main paper. For your convenience, we also include the prompt here:\n\n> The task is to summarize the morphological features of the `{dataset_name}` dataset for the `{class_name_1}`, `{class_name_2}`, … classes. For each class, list **O** representative morphological features observed at **5×** magnification, followed by **K** finer sub-features observed at **20×** magnification for each. Each description should include the morphological term along with an explanation of its defining visual features.\n\nThe examples of the descriptions generated by the LLM are provided in Supplementary Material A.3.\n\n| **W3/Q2. Sensitivity to prompt template and LLM backbone**|\n|---|\nTo assess sensitivity to textual inputs, we conducted two sets of robustness analyses: \n\n(1) **Prompt Template Sensitivity** (Supplementary Material H.4): We evaluated three formats: Term-only, Explanation-only (Exp.), and Term \\+ Explanation (Term \\+ Exp.). The Term \\+ Explanation format consistently achieves the best performance, suggesting that combining concise terms with descriptive context yields more effective guidance (CONCH, 16-shot).\n\n\n\n|  | **TCGA NSCLC** |  |  | **TCGA BRCA** |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **Prompt Template** | **ACC** | **AUC** | **Macro F1** | **ACC** | **AUC** | **Macro F1** |\n| Term | 89.55 ±4.27 | **96.92 ±0.70** | 89.50 ±4.34 | 86.64 ±2.01 | **94.22 ±1.12** | 86.57 ±2.00 |\n| Exp. | 90.00 ±2.83 | 96.47 ±0.86 | 89.99 ±2.85 | 86.77 ±2.80 | 93.35 ±1.01 | 86.66 ±2.93 |\n| **Term \\+ Exp.** | **90.39 ±1.57** | 96.49 ±0.56 | **90.37 ±1.58** | **87.29 ±2.83** | 93.86 ±0.89 | **87.24 ±2.85** |\n\n(2) **LLM Backbone Sensitivity** (Supplementary Material H.5) : We generated descriptions using four different LLMs: DeepSeek R1, Grok 3, Gemini 2.5 Pro, and GPT-4o. GPT-4o achieves the best overall performance. The results are consistent across different LLMs, demonstrating that HiVE-MIL is robust to the choice of LLM backbone (CONCH, 16-shot).\n\n\n\n| | **TCGA NSCLC** |  |  | **TCGA BRCA** |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n|  **LLM** | **ACC** | **AUC** | **Macro F1** | **ACC** | **AUC** | **Macro F1** |\n| DeepSeek R1 | 89.36 ±3.88 | 95.77 ±1.68  | 89.33 ±3.92 | 85.60 ±4.22 | 93.08 ±2.43 | 85.49 ±4.36 |\n| Grok 3 | 90.19 ±2.33 | 96.33 ±1.01 | 90.18 ±2.33 | 85.80 ±3.27 | 93.62 ±1.80 | 85.71 ±3.32 |\n| Gemini 2.5 Pro | 90.00 ±2.52 | 96.47 ±0.99 | 89.98 ±2.55 | 86.65 ±2.10 | 93.08 ±1.79 | 86.61 ±2.10 |\n| **GPT-4o** | **90.39 ±1.57** | **96.49 ±0.56** | **90.37 ±1.58** | **87.29 ±2.83** | **93.86 ±0.89** | **87.24 ±2.85** |\n\nAgain, we thank you for your thoughtful and constructive comments."}, {"review_id": "Hf7LcCYcPy", "reviewer": "Reviewer_BfPp", "summary": "HiVE-MIL proposes a few-shot learning framework for whole-slide image classification that integrates multi-scale image patches with LLM-generated textual descriptions. By constructing a hierarchical heterogeneous graph linking visual and textual features across scales, and applying a lightweight graph neural network with modality-scale attention, the method achieves strong performance on TCGA benchmarks while maintaining parameter efficiency and offering interpretable outputs.", "questions": "1. Are the disease-specific textual descriptions generated by the LLM verified by pathologists for accuracy?\n2. The experiments are limited to TCGA datasets. Could the authors validate the generalizability of their method on external datasets such as CAMELYON or PANDA?", "limitations": "Please see weaknesses and questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. This paper is well written and easy to follow.\n2. The design of TGDF combined with hierarchical graphs enhances the model's interpretability. Additionally, the use of a lightweight graph head and frozen VLM encoders makes the model parameter-efficient and stable under few-shot settings.\n3. The experimental comparison is comprehensive, and the ablation studies are also very thorough.\n\n### Weaknesses\n\n1. In line 57, the authors claim that *\"current models do not explicitly model the interactions between visual and textual features on the same scale.\"* However, this claim is not entirely accurate. Existing works such as ViLa-MIL and MSCPT already incorporate cross-modal feature interaction at the same scale. For example, ViLa-MIL (Figure 2) includes a context-guided text decoder and slide-text similarity computation; MSCPT (Figure 2) introduces image-text similarity-based graph prompt tuning. This weakens the paper’s stated motivation and contribution.\n2. In Equation 7, the index notation is confusing, mixing the letters *i* and *c*. Moreover, the explanation of text features in the methods section only accounts for low-scale feature count *O* and high-scale feature count *K* or *S*, but neglects the class number *C* (e.g. Equation 4). This sudden appearance of *C* in Equation 7 may confuse readers.\n3. The modeling of image features only considers hierarchical relationships between low and high scales, without incorporating spatial adjacency within the same scale. This may hinder the model from capturing visually recognizable patterns such as gland–stroma–inflammatory cell arrangements.\n4. The fixed threshold α in TGDF is heuristic and may not generalize across datasets or backbones. An inappropriate α could result in the omission of rare but critical tumor regions, reducing diagnostic reliability.", "was_revised": true, "final_justification": "Thanks for the valuable feedback, it well address my concerns, so I will raise my score to 4.", "rebuttal": "Thank you for your positive feedback and for recognizing that our paper is well written and easy to follow. We also appreciate your acknowledgment of the model’s interpretability and the comprehensiveness of our experiments. We provide the following responses to address your concerns.\n\n| **W1. Clarification regarding motivation and contribution**|\n|--- |\nWe apologize for the confusion. What we intended to convey is that while current models such as ViLa-MIL and MSCPT do perform cross-modal feature interaction at the same scale, the alignment between visual and textual features remains *inadequate*. This point is reflected in line 7 (Abstract) and line 57, which state: *“Inadequate alignment between modalities on the same scale.”* Existing models do not *fully* explore the interactions between different modalities in constructing task-specific knowledge. In contrast, our method addresses this limitation through a heterogeneous graph that explicitly models semantic connections between visual and textual nodes at the same scale (line 74).\n\nWe acknowledge that the sentence referenced by the reviewer may have unintentionally implied that existing models lack any form of cross-modal modeling, which was not our intention. We will revise the wording in the final version to clearly reflect our intended message.\n\n| **W2. Clarification regarding equation**|\n|--- |\nWe thank the reviewer for pointing out the confusion in Equation 7 regarding the index notation and the insufficient explanation of class-wise text prompt grouping.\n\nWe agree that the use of indices in Equation 7 was potentially confusing, especially due to the mixing of the image index $i$ and class index $c$. In the revised version, we will clarify the notation by introducing a class-specific text index set $\\mathcal{I}_c$, where each $i \\in \\mathcal{I}_c$ corresponds to a text prompt associated with class $c$. This allows us to express the class logit computation more clearly. In the equation below, the $ \\mathrm{TopKAvg}_k$ function computes the average of the top $k$ similarity scores from the $i$-th column (.,i) of the image-text similarity matrix. This step aggregates the signal from the most relevant images for each text prompt before the final summation.\n\n\n`logit_c^{(s)} = (γ / |I_c|) * Σ_{i∈I_c} TopKAvg_k([X^{(s)} (T^{(s)})ᵗ]_{·,i})`\n\n\nThis avoids index overloading and makes the operation easier to follow.\n\n\n\n| **W3. Not incorporating spatial adjacency**|\n|---  |\nThank you for your valuable feedback. Although HiVE-MIL does not explicitly encode spatial adjacency, it does capture structural context *indirectly* through a hierarchical and semantic design. At the 20x scale, each high-resolution patch is linked to its corresponding 5x parent patch via predefined absolute-coordinate mappings, enabling spatial alignment to be reflected hierarchically. At both 5x and 20x scales, semantically related patch–text pairs are selected using TGDF, allowing patches to form connections through shared textual descriptions. These semantic-based connections serve as an *intermediate* mechanism for modeling inter-patch relationships beyond physical proximity. \n\nThis design reflects the characteristics of pathological images, where spatially adjacent patches often differ in tissue structure, while semantically meaningful relationships can occur across distant regions. Therefore, we construct the intra-scale graph using semantic filtering, which allows connections between distant patches that share similar meanings based on textual descriptions. We argue that simple distance-based connections may overlook semantially important patterns. Nevertheless, we acknowledge the reviewer’s suggestion to explicitly incorporate intra-scale spatial adjacency as a valuable extension and we plan to explore ways to integrate spatial and semantic relationships in parallel in future work.\n\n| **W4. Heuristic α may omit rare tumors**|\n|---  |\nWe appreciate the reviewer’s concern that the fixed threshold α in the TGDF module could potentially miss rare but critical tumor regions, thereby reducing diagnostic reliability. However, we would like to clarify that the purpose of α is not to filter or suppress tumor regions, but rather to prevent irrelevant image patches from being connected to unrelated textual prompts, such as background patches paired with disease-specific text, which could confuse the model during learning. In other words, TGDF filters out semantically meaningless image–text pairs and retains only the meaningful ones, which are then used to form edges in the heterogeneous graph, thereby encouraging the model to focus on learning coherent and semantically rich visual–language associations.\n\nFurthermore, as shown in the sensitivity analysis presented in Supplementary Material H.2, we evaluated a range of α values (0, 0.1, 0.5, 1.0) and observed that the model’s performance remains relatively stable across settings. The default value α \\= 0.5 yields the most consistent results overall. That said, we fully acknowledge the reviewer’s point that a fixed threshold may not be optimal across all scenarios. As such, we plan to explore adaptive, learnable filtering mechanisms in future work to further improve generalization across datasets or backbones.\n\n| **Q1. Pathologists verification**|\n|---  |\nThank you for asking this important question. While the descriptions generated by the LLM have not yet been verified by pathologists, we address this limitation through a series of alternative validation strategies designed to assess their class-discriminative relevance and semantic reliability.\n\n(1) We evaluate the *faithfulness* of the generated descriptions using an \"LLM-as-a-Judge\" \\[1\\], where LLMs are prompted to infer class labels based solely on the generated descriptions. We employ two  evaluators: GPT-4o and GPT o3, to assess descriptions generated by several LLMs: Deep Seek R1, Grok 3, Gemini 2.5 Pro, and GPT-4o. Although not perfect, the number of correct predictions is still consistently high, suggesting that the generated text is generally reliable while capturing class-discriminative signals.\n\n| |  | **Evaluator: GPT-4o** |  | **Evaluator: GPT o3** |  |\n| :---: | :---: | :---: | :---: | :---: | :---: |\n| **Generator**  |**\\#Total Descriptions (\\#5x/\\#20x)**  | **TCGA NSCLC** | **TCGA BRCA** | **TCGA NSCLC** | **TCGA BRCA** |\n| DeepSeek R1 | 32 (8/24) | 28 (8/20) | 28 (8/20) | 31 (8/23) | 28 (8/20) |\n| Grok 3 | 32 (8/24) | 28 (7/21) | 28 (8/20) | 32 (8/24) | 21 (8/23) |\n| Gemini 2.5 Pro | 32 (8/24) | 30 (8/22) | 29 (7/22) | 30 (7/23) | 32 (8/24) |\n| GPT-4o | 32 (8/24) | 30 (8/22) | 30 (7/23) | 31 (8/23) | 32 (8/24) |\n| Total | 128 (32/96) | 116 (31/85) | 114 (29/85) | 124 (31/93) | 113 (32/89) |\n\n\\[1\\] Gu et al. A Survey on LLM-as-a-Judge, arXiv 2025\\.\n\n(2) The table below reports intra- and inter-category text similarity scores, where *intra* denotes similarities among descriptions of the same morphology or concept, and *inter* denotes similarities across different ones. Similarity is computed separately at each scale and jointly across both scales using the CONCH text encoder and cosine similarity. These results highlight that the descriptions encode highly discriminative and consistent semantics for each morphological or conceptual group, with robustness across scales and LLMs.\n\n\n\n|| **TCGA NSCLC** |  |  | **TCGA BRCA** |  |  |\n| :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n| **LLM**  | **5x (intra/inter)** | **20x (intra/inter)** | **5x,20x (intra/inter)** | **5x (intra/inter)** | **20x (intra/inter)** | **5x, 20x (intra/inter)** |\n| DeepSeek R1 | 1.000 (0.157) | 0.515 (0.164) | 0.456 (0.158) | 1.000 (0.165) | 0.438 (0.141) | 0.412 (0.138) |\n| Grok 3 | 1.000 (0.123) | 0.511 (0.170) | 0.434 (0.160) | 1.000 (0.186) | 0.490 (0.214) | 0.415 (0.212) |\n| Gemini 2.5 Pro | 1.000 (0.111) | 0.558 (0.165) | 0.478 (0.150) | 1.000 (0.206) | 0.491 (0.127) | 0.426 (0.142) |\n| GPT-4o | 1.000 (0.066) | 0.516 (0.114) | 0.454 (0.109) | 1.000 (0.163) | 0.477 (0.108) | 0.413 (0.125) |\n\nFurthermore, we validated the performance of HiVE-MIL using descriptions generated by different LLMs, as detailed in Supplementary Material H.5.\n\n| **Q2. Generalization beyond TCGA datasets**|\n|---  |\nTo evaluate generalizability beyond TCGA datasets (BRCA, NSCLC, RCC), we further tested HiVE-MIL and baselines on the **Camelyon16 dataset**, using the same experimental settings as described in the main paper. HiVE-MIL achieves the highest performance, outperforming the second-best model, ViLa-MIL, by 1.25% in Macro F1 (CONCH, 16-shot).\n\n\n\n| **Model**           | **ACC**           | **AUC**           | **Macro F1**      |\n|---------------------|-------------------|-------------------|-------------------|\n| ABMIL               | 88.33 ± 4.44      | 92.97 ± 4.64      | 87.50 ± 4.67      |\n| DSMIL               | 90.33 ± 2.77      | 94.95 ± 1.82      | 89.92 ± 2.85      |\n| CLAM-SB             | 85.17 ± 14.16     | 85.07 ± 20.02     | 80.46 ± 22.48     |\n| DTFD-MIL (AFS)      | 91.17 ± 4.25      | 93.35 ± 3.36      | 91.92 ± 4.43      |\n| ViLa-MIL            | 92.33 ± 5.04      | 96.37 ± 2.28      | 91.97 ± 5.22      |\n| MSCPT               | 88.50 ± 8.79      | 88.43 ± 13.44     | 88.29 ± 13.03     |\n| FOCUS               | 90.83 ± 4.44      | 91.62 ± 6.52      | 90.23 ± 4.84      |\n| **HiVE-MIL**        | **93.33 ± 4.86**  | **96.72 ± 3.57**  | **93.22 ± 5.06**  |\n\n\nAgain, thank you for your thoughtful and constructive comments."}]

Author rebuttal:

Meta-review: The paper addresses a current problem in Digital histopathology in modeling interactions across different scales when using VLMs. After rebuttal reviewers raised their scores and are all positive towards acceptance. The authors should try to incorporate their rebuttal clarifications to the final paper
Decision: Accept (poster)