Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Authors: Zhengrui Guo
Qichen Sun
Jiabo MA
Lishuang Feng
Jinzhuo Wang
Hao Chen
Venue: ICML 2025 poster (2025)
Keywords: Computational Pathology
Whole Slide Image
Cancer Diagnosis and Prognosis
Primary area: applications->health_medicine

Abstract: Whole slide image (WSI) analysis presents significant computational challenges due to the massive number of patches in gigapixel images. While transformer architectures excel at modeling long-range correlations through self-attention, their quadratic computational complexity makes them impractical for computational pathology applications. Existing solutions like local-global or linear self-attention reduce computational costs but compromise the strong modeling capabilities of full self-attention. In this work, we propose **Querent**, *i.e.*, the **quer**y-awar**e** long co**nt**extual dynamic modeling framework, which achieves a theoretically bounded approximation of full self-attention while delivering practical efficiency. Our method adaptively predicts which surrounding regions are most relevant for each patch, enabling focused yet unrestricted attention computation only with potentially important contexts. By using efficient region-wise metadata computation and importance estimation, our approach dramatically reduces computational overhead while preserving global perception to model fine-grained patch correlations. Through comprehensive experiments on biomarker prediction, gene mutation prediction, cancer subtyping, and survival analysis across over 10 WSI datasets, our method demonstrates superior performance compared to the state-of-the-art approaches. Codes are available at https://github.com/dddavid4real/Querent.

Context Matters:
Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Zhengrui Guo
12 Qichen Sun
Jiabo Ma
Lishuang Feng? Jinzhuo Wang
Hao Chen
Abstract
Benign patches
Cancerous patches in the WSI
Whole slide image (WSI) analysis presents signif- icant computational challenges due to the massive number of patches in gigapixel images. While transformer architectures excel at modeling long- range correlations through self-attention, their quadratic computational complexity makes them impractical for computational pathology appli- cations. Existing solutions like local-global linear self-attention reduce computational costs but compromise the strong modeling capabilities of full self-attention.
cancerous B query. results in high attention acom with cancerous patch
When benign patch A works as leads attention soore cancerous patch
Figure I. Illustration of context-dependen patch relationships in whole slide images. When a benign patch (A) interacts with can- cerous patch C, shows low correlation, while cancerous patch (B) shows high correlation with patch C. This demonstrates how the same patch (C) have fundamentally different relationships with other patches depending on the biological context.
this work, we propose Querent, i.e., the query-aware long contextual dynamic modeling framework, which achieves a theoretically bounded approximation of full self- attention while delivering practical efficiency. Our method adaptively predicts which surrounding re- gions are most relevant for each patch, enabling focused yet unrestricted attention computation only with potentially important contexts. By us- ing efficient region-wise metadata computation and importance estimation, our approach dramati- cally reduces computational overhead while pre- serving global perception model fine-graine grained patch correlations. Through comprehensive ex- periments on biomarker prediction, gene mutation prediction, cancer subtyping, and survival anal- ysis across over WSI datasets, our method demonstrates superior performance compared the state-of-the-art approaches. Codes are here.
ligence and deep learning to analyze the growing collec- tions of whole slide images (WSIs) from medical facilities (Van der Laak et al., 2021; Cui & Zhang, 2021; Zheng et al., 2025). By digitalizing traditional pathology workflows, this emerging field enhances clinical maiiiinnnmainng through more standardized diagnoses, enables the identification of new biomarkers, and helps predict treatment outcomes for patients (Niazi et al., 2019; Song et al., 2023). These WSIs, also known as gigapixel images, typically contain between 10, 0002 100, 0002 pixels. The challenge lies in iden- tifying critical diagnostic features that may be dispersed across various tissue regions within these highly informa- tive images, i.e., analogous to finding needle in a haystack (Jin et al., 2023). These unique challenges have driven the development of supervised learning paradigm for WSI analysis (Campanella et al., 2019; Lu et al., 2021; Shao et al., 2021; Xu & Chen, 2023; Zhou & Chen, 2023; Yang et al., 2024).
learning (MIL), weakly-
1. Introduction
Computational pathology (CPath) represents a transforma- tive shift in clinical diagnostics, leveraging artificial intel-
'Hong Kong University of Science and Technology, IKK, China 'Beijing Institute of Collaborative Innovation, Beijing, China e "Peking University, Beijing, China. Correspondence to: Zhengrui Guo <zguobe wontoee@@ooonnnttthhc.iee Hao Chen <jhe@cse.ust.ni
MIL performs slide-level analysis through three key steps: (1) segmenting and cropping tissues in WSI into smaller image patches; (2) extracting representation features from these patches using a pretrained encoder (typically a CPath foundation model (Chen et al., 2024; Lu et al., 2024; Ma et al., 2024; Xu et al., 2024; Du et al., 2025; Xiong et al., 2025a)); and (3) aggregating the patch features through
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
specific principles obtain the slide-level representation. Among various feature aggregation approaches, transformer architectures have emerged as a particularly promising so- lution due to their powerful ability to model long-range dependencies through self-attention mechanisms (Vaswani, 2017: Dosovitskiy, 2020). This capability is especially valu- able in WSI analysis, where diagnostically relevant features often manifest through complex spatial relationships be- tween distant tissue regions. However, the quadratic com- putational complexity (nn) of the standard transformer's self-attention mechanism presents a significant challenge when applied to WSIs, as typical slide can contain thou- sands to tens of thousands of patches (Wang et al., 2023; Song et al., 2024b; Guo et al., 2025; Xiong et al., 2025b).
importance of patch relationships through efficient region metadata computation and selectively applies full attention to the most relevant interactions. This query-aware strat- egy allows each patch to have its unique attention pattern, better capturing the heterogeneous nature of histological features while remaining computationally tractable for gi- gapixel WSIs. Theoretically, we prove that Querent's query- aware attention mechanism maintains expressiveness within small constant bound of full demonstrate Querent's 's effectiveness across multiple CPath tasks, including biomarker prediction, gene mutation pre- diction, cancer subtyping, and survival prediction, where it consistently outperforms state-of-th of--th---aaan-- art models on over 10 WSI datasets.
Empirically, we
To address this computational barrier, several adaptations of transformer architectures have been proposed for WSI analysis. These include linear attention methods that reduce complexity to (n) (Shao et al., 2021), and local-global attention approaches that restrict attention computation to predetermined spatial patterns (Chen et al., 2022: Li et al., 2024a). While these modifications successfully reduce com- putational overhead, they inevitably compromise the trans- former's 's inherent modeling capabilities. As demonstrated by Dao et al. (2022) and Han et al. (2023), linear approxima- tion of self-attention leads to only sub-optimal modeling per- formance. Meanwhile, local-global attention makes strong assumptions about which spatial relationships are important, failing to adapt to the highly variable and context-dependent nature of pathological features in WSIs.
The main contributions of this work are as follows: (I) We propose novel query-aware attention mechanism that dynamically adapts to each patch's 's unique context in gi- gapixel WSIs, maintaining full attention's 's expressiveness while achieving computational efficiency; (2) We develop ef- ficient region- metadata summarization and importance estimation modules that enables dynamic sparsification of attention patterns while preserving modeling capabilities; (3) We establish Querent's effectiveness through theoreti- cal bounds on its expressiveness and extensive empirical validation across diverse CPath tasks.
2. Related Work
Given the massive image size of WSIs and GPU memory re- strictions (Araujo et al., 2019), researchers have widely adopted MIL. for WSI analysis. Recent MIL-based ap- proaches have achieved promising results in diagnosing diseases and predicting patient outcomes under the formula- tion of weakky--uuerrvieed learning (Campanella et al., 2019; Chikontwe et al., 2020; Li et al., 2021b; Xiang et al., 2022: Hou et al., 2022; Zheng et al., 2022; Wang et al., 2022; Yu et al., 2023; Xiong et al., 2023; Lin et al., 2023; Xiong et al., 2024a; Song et al., 2024a; Xiong et al., 2024b).
These limitations motivate the need for a more adaptive ap- proach to modeling inter-patch relationships in WSI analysis. A key observation is that the relevance of surrounding tissue regions varies significantly depending on which specific re- gion is being examined (Heindl et al., 2015; Yuan, 2016). As shown in Fig. I, when analyzing a tumor boundary region, nearby regions showing the tumor-stroma interface might be highly relevant, while distant regions of normal tissue might be less informative. Conversely, when examining area of inflammation, regions with similar inflammatory pat- terns across the slide might be more relevant than adjacent but histologically different regions. This context-dependee nature of patch relationships suggests that an ideal attention mechanism should dynamically prioritize relevant interac- tions for each query patch while maintaining the capability to model range dependencies when necessary.
MII. primarily addresses the challenge of aggregating patch- level features into a comprehensive slide-level representa- tion for diagnostic purposes. Early MIL approaches em- ployed straightforward, non parameeriiic aggregation tech- niques, such as max and mean pooling operations, to com- bine these features (Campanella al., 2019). Recent ad- vances in MIL have emphasized the development of more sophisticated aggregation strategies to effectively capture diagnostic patterns scattered across numerous patches. AB- MIL (Ilse et al., 2018) introduces an attention-based frame- work that computes importance weights for each patch using side network during the aggregation process. CLAM (Lu et al., 2021) enhanced this attention mechanism by incor- porating cuustering-constrainet learning and flexible atten- tion branches to enhance WSI classification performance.
Based on this insight, we propose Querent, novel frame- work for dynamic long-range contextual modeling of gi- gapixel images through adaptive determination of patch relationships. Our approach maintains the modeling power of full attention while achieving computational efficiency through dynamic sparsification. Rather than using fixed pat- terns or uniform approximations, it estimates the potential
2

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
DSMIL (Li al., 2021a) a) develops a dual-stream architec- ture that capitalizes on the hierarchical structure of WSIs by integrating features across multiple magnification levels. Further, DTFD-MIL (Zhang et al., 2022) introduces novel double-tier framework that leverages pseudo-bags to maxi- mize feature utilization and minimize the bag-instance im- balance problem in WSI analysis. Taking a graph-based per- spective, WiKG (Li et al., 2024b) reformulates WSI analysis by representing patches nodes in knowledge graph and utilizing head-to-tail embeddings to generate dynamic graph representations. Combined with the State Space Models (Gu & Dao, 2023), MambaMIL (Yang et al., 2024) features a Se- quence Reordering Mamba module that processes instance sequences through both original and reordered pathways to enhance long-range dependency modeling while maintain- ing linear complexity. The field has further evolved with the integration of multimodal information, as researchers have begun incorporating pathology image captions (Luu et al., 2023), diagnostic reports (Guo et al., 2024), as well as genomic profiles (Sun et al., 2025) to enhance both inter- pretability and diagnostic accuracy.
benefits from focusing on local contextual patterns.
This observation motivates our query-aware dynamic long sequence modeling approach, which adaptively determines relevant regions for each patch based on its pathological characteristics rather than using predetermined fixed atten- tion patterns. Our method efficiently computes region-level metadata through min-max networks to estimate the impor- tance of different tissue regions for each (query) patch. By focusing only on the most relevant regions for each patch, we maintain the expressiveness of full attention while sig- nificantly reducing computational cost.
3. Methods
3.1. Problem Formulation and Solution Overview
Problem Formulation: Given a WSI W, we first par- tition it into set of non-overlapping smaller patches {p1, Pa...., (p,.P2,.... } at a fixed magnification level. Each patch is then processed through pre-trained encoder <6-) to ob- tain feature representations. Specifically, a CPath founda- tion model named PLIP (Huang et al., 2023) is used for this purpose. Formally, for each patch ps, we obtain its feature representation x; = p(p:). where 24 E Rdd is d- dimensional feature vector. This results in a bag of instances X = {x1, T2, ..... IN representing the entire WSI. The goal of WSI analysis is to learn mapping function f : X + y that predicts the slide-level label y based on all patches within the WSI through a weakly-supervised manner. The key challenge lies in effectively aggregating information from thousands of patches while capturing both their spe- cific contexts and inter-patch semantic relationships.
Building upon these advancements, another significant line of research has focused on leveraging Transformer archi- tectures to model long range dependencies among patches in WSIs. Despite the effectiveness of self-attention in mod- eling long-range correlation, the quadratic computational complexity of standard Transformer architectures poses sig- nificant challenges when processing tens of thousands of patches in a WSI. To address this limitation, Transformer- based MIL methods could be categorized into two classes, i.e. linear approximation (Shao et al., 2021) and local- global attention (Chen et al., 2022; Guo et al., 2024; Li et al., 2024a). TransMIL (Shao et al., 2021), pioneering work in linear approximation, adopts the Nyströmformer (Xiong et al., 2021) to achieve linear complexity in WSI modeling. However, this approximation strategy often leads to sub-optimal performance due its inherent limitations in capturing pairwise token interactions, creating an infor- mation bottleneck that compromises the model's 's ability to learn complex dependencies (Dao et al., 2022). In con- trast, local-global attention methods like HIPT (Chen et al., 2022) introduce a hierarchical approach, using a three-level Transformer architecture to progressively aggregate infor- mation from cellular features to tissue phenotypes through non-overlapping region slicing. Similarly, HistGen (Guo et al., 2024), RRT-MIL. (Tang et al., 2024), and LongMII. (Li et al., 2024a) employ a two-stage strategy, first process- ing patches within local attention windows before pooling them to enable global attention across the reduced sequence. However, these methods rely on fixed window sizes, which fail to capture the inherently adaptive nature of pathological analysis: malignant regions often require attention dis- tant but visually similar areas, while normal tissue typically
Solution Overview: Here, we describe the overview of our method comprising 4 major steps as depicted in Fig. 2.
For each WSI, we partition the WSI into local regions, where each region contains multiple patches. For each region, we compute region-level metadata through summarization mechanism. This metadata encapsulates the key characteristics of each region through statistical measures (e.g., min/max features) and serves as a compact representation for importance estimation.
Step 1:
Step 2: Given a query patch, we leverage the pre-computed region-level metadata to identify the top-K most relevant regions through importance scoring efficiently. This en- ables prioritized processing of regions most likely to contain meaningful contextual information for the current query.
Step 3: Then, we employ query-aware selective attention to process the query with the most relevant regions. This step involves computing dense self-attention between the query and patches within selected regions, allowing for detailed analysis of patch-to-pate relationships while maintaining computational efficiency. The selection is guided by the
3

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Relevant Region Identification
1
Query Feature
Top-K Most Relevant Regions
Feature Representations of Patches
Summary Vectors of Each Region
Encoder
(1)
Feature
Extraction
Patch Feature in Selected Regions
Value
Region-levet Meta-data
Patching
Summarization
Query
Dense Self-Attontion
Whole Slide Images
(4)
Query-Aware Selective Self-Attention
TrejonnleeelImportanen Estimation
A local region split from WSI
Attentive Pooling
Feature Aggregation
A patch cropped from WSI
Features after Query-aware Attn
Prediction
Figure 2. Illustration of the proposed Querent framework, which models WSI via four key steps: (1) region- level metadata summarization from partitioned WSL, detailed in Fig. (2) identification relevant regions for query patches through efficient importance scoring, (3) query-au aware selective self- attention computation between query patch and patches in selected regions, and (4) feature aggregation with attentive pooling for final prediction. The framework enables dynamic modeling of long-range contextual relationships in gigapixel WSIs through efficient region relevance identification and query-aware selective attention computation.
relevance scores computed using the region-level metadata.
learnable projections:
Step 4: Finally, we aggregate the refined features through an attentive pooling mechanism generate slide-level pre- dictions, which combines the contextually enhanced patch representations while emphasizing the most diagnostically relevant features.
m Smin (m mim ); 1.
f fmux max
where Smin and mmax are neural networks that project the sum- mary vectors into a shared embedding space. This projection allows the metadata to be directly comparable with query features in the subsequent importance estimation step. The resulting region-level metadata provides a compact yet in- formative representation of each region's 's content, enabling efficient relevance assessment without requiring exhaustive patch-level computations.
3.2. Querent for dynamic long sequence WSI modeling
3.2. 1. REGION-LEVEL METADATA SUMMARIZATION
Given the extracted patch features X = {x1, EN we first organize them into regions R = (R1, R2 RM where each region R; contains a fixed number of K patches.
For each region R,, we compute metadata vectors that cap- ture the statistical characteristics of all patches within that compute two
3.2.2. REGION IMPORTANCE ESTIMATION FOR QUERY
Given a query patch feature E IRd our goal is to efficiently identify the most relevant regions for this query using the pre-computed region-level metadata. For each region R, we estimate its importance score based on the potential interaction between the query and its metadata vectors.
region, illustrated in Fig. 3. Specifically, types of summary vectors: minimum and maximum feature values across all patches in the region. Formally, for region R; containing patches {I.I: Fi2.... R.K) ), we compute:
mi
Tij min min Tijs m. JE(I...K) jE(I..K)
(1)
Specifically, we first project the query feature into the same embedding space as the region metadata q (g). where f, is a learnable projection network. The importance score for region R, is then computed by evaluating the maximum possible interaction between the query and the region's 's meta-
where m. m E Rd represent the wise min- imum and maximum values across all patches in region R. These summary vectors are then transformed through

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Region Meta-data
attention head. The output of the attention layer and multi- head attention are computed as:
Summary Vector
Concat
O AV
(6)
Projector
Projector
O = MultiHead Q, K, V) -Concat OH W
Computo Min-Value
Compute Max-Value
(7)
where Wo E Rdd is a learnable projection matrix. Through this selective attention mechanism, each query patch attends only to patches within the most relevant regions, achieving a balance between computational effi- ciency and comprehensive contextual modeling. The com- putational complexity is reduced from O(N²) to O K), where N is the total number of patches and K the number of patches in selected regions.
The selective attention mechanism described above pro- vides an efficient approximation of full self-attention while maintaining its key properties. To formally characterize the relationship between our query-aware selective attention and standard self-attention, we establish the following the- oretical guarantee. This theorem demonstrates that under reasonable conditions regarding the input distribution and model parameters, our selective attention mechanism can approximate full self-attention with bounded error. Specifi- cally, show:
Figure 3. Illustration of the region-level metadata summarization process. Each region from the WSI is represented by summary vectors computed from its constituent patches. These summary vectors capture statistical characteristics (minimum and maxi- mum values) across all patches within each region, providing an efficient representation for subsequent importance estimation.
data bounds:
8; max( 1,
((. m
(3)
Theorem 3.1 (Query-Aware Attention Approximation). Let A be the query-aware attention matrix (Def. B.2), and B be the full self-attenton matrix (Def. B.1). Assume at- tention scores decay exponentially with spatial distance: exp( exp(-addy (oddd(i,jj) bounds the attention score decay for dis- tance d(i,j). For any input sequence X. with L patches and dimensions, there exist random projection matrices
where ( denotes the dot product operation. The impor- tance score 8; provides an upper bound on the potential relevance of any patch within region R; to the current query. Based on these scores, we select the top-K regions with the highest importance scores:
R2 = TopK( 1
(4)
Wo. WK E Rdddd such that || - -BIIA S (2+ v2) with probability least 1 8, provided:
This efficient scoring mechanism allows us to identify the most promising regions for detailed attention computation without examining every patch. significantly reducing the computational complexity while maintaining the ability to capture long-ran aangrangg dependencies.
The hidden dimension satisfies, and the number of se- lected regions per query satisfies:
log(L/8) 2
k C2
d C1
log
3.2.3. QUERY-AWARE SELECTIVE ATTENTION
where C₁ 8B (from JL inner-product preservation (Kaban, 2015)), and C2 = 2.
After identifying the relevant regions R4 for query patch q. we compute dense self-attention between the query and patches within these selected regions. For a selected region R, E Rq. we first compute query (Q), key (K), and value (V) representations through linear projections, followed by attention score computation:
2. For each region R2 the diameter satisfies:
(L.va'd)
1
diam(R)) S min
3. Regions are spatially separated such that:
softmax QKT J
C3s
Vi ti j. d(Ri, R2)
Q2 K. V] = W (a),(a),
(5)
a
where C3 = 1 ensures - e Cam < E.
where Wak E Rdx3d is a learnable parameter matrix, x represents the concatenation of the query patch and patches from the selected region, and d, is the dimension of each
Proof. The detailed derivation of proof can be found in Appendix B.
5

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Table 1. Results of biomarker prediction, gene mutation prediction, and cancer subtyping tasks, with accuracy, AUC, Fl score reported. The best results are in bold, and the second-best results are underlined. Rows in gray color represent Self-Altm-based methods.
BCNB-ER (n 1038)
TCGA-LUAD TP53 (n 469)
UBC-OCEAN (n 527)
Methods
ACC
AUC
Fl Score
ACC
AUC
Fl Score
ACC
AUC
Fl Score
Mcan Pooling
0.806 047
0.82010.084
0.700
0.63910.061
0.67210.081
0.630 063
1.78810.017
2.0011
0.75910.038
Max Pooling
0.81010.016
1..191
0.72310.029
0.63910.049
0.66010.048
0.65010.016
0.80010.031
1.94610.015
0.78210.033
ABMIL
0.825 10.008
0..251
0.726 10.088
0.639
0.68810.077
0.630
0.82310.035
0.94210.019
0.79610.009
DS-MIL
0.817
0.814
0.704
0.62910.04F
0.68410.072
0.61410.046
0.781
0.933810.018
0.75310.024
CLAM-SB
0.821 10.062
0.821 10.050
0.73220.070
0.615
0.647
0.60710.077
g0.021
0.9422
0.79610.025
DTFD
0.786 029
0.835-10.010
0.715
0.64420.057
0.680
0.636 00.02
0.793
0.941120.044
0.77920.029
WiKG
0.794 027
0.815
0.717
0.654
0.642 0.66110.076
048
0.81920.065
0.944420.02
0.78220.060
MambaMII.
0.825
0.82010.0er
0.71910.011
0.63910.06
0.68510.075
0.627 10
0.80810.021
0.941
0.78510.031
TransMII.
0.802 0.780+0.086
0.66420.063
0.61540.028
0.65840.094
0.596
0.928
0.698
HIPT
0.796
0.758
0.681
0.600 +0.029
0.672+0.062
0.588+0.031
0.781 +0.040
0.913
#0.068
HistGen
0.79 0.791 10.027
10.901
0.695_0.049
0.66810.061
0.658 0.66510.036
0.78440.025
0.93441.009
0.764
RRT-MIL
0.8 221220.mm
0.814
0.688
0.600
0.65740.029
0.58 10.077
0.804
0.93910.016
0.783
LongMIL
0.781 20.018
20.05
0. 0.653
0. 2..63
0. 40.00
0.63740.077
0. 0.760
00.06
2.74220.08
Querent (Ours) 0.836g0.043 0.848
0.739
0.678
0. 0.706
0.672+0.070
0.835
0.956 0.806
3.2.4. ATTENTIVE FEATURE AGGREGATION
(CE) loss Lee. For survival prediction tasks, we utilize the negative log-likelihood (NLL) survival loss Laurv (Cox, 1972; Katzman et al., 201 8; Zadeh & Schmid, 2020). The detailed formulations of these loss functions and their opti- mization procedures are provided in Appendix C.
After obtaining contextually enhanced representations for all patches through query-awar selective attention, we employ an attentive pooling mechanism to aggregate these features for slide-level prediction. Given the refined patch features {z', x22, <'y J, we compute attention weights through a learnable attention network:
4. Experiments
exp(w:) Ejn exp(w;)
4.1. Tasks and Datasets
=fa((z)))
(8)
We evaluate Querent across four types of tasks over 11 publicly available datasets (see Appendix D for details):
where fa is multi-layer perceptron that produces a scalar score for each patch, and a is the sigmoid activation function. The attention weights aj are normalized through softmax to ensure they sum to one. The final slide-level representation computed as the weighted sum of all patch features, which is then passed through final classification layer:
(1) Biomarker prediction. BCNB (Xu et al., 2021) (n 1038), a dataset of early breast cancer core-needle biopsy WSI is used to predict tumor clinical characteristic estrogen receptor (ER).
(2) Gene mutation prediction. A subset of The Can- cer Genome Atlas (TCGA) (Tomezak et al., 2015) is used (TCGA-LUAD, = 469) to predict TP53 gene mutation from lung adenocarcinoma WSIs.
2
9 = fc)z)
(3) Cancer subtyping. UBC-OCEAN (n = 527), an ovar- ian cancer dataset, is used to predict 5 cancer subtypes.
where fo is a fully connected layer that outputs the predicted class probabilities. This attentive pooling mechanism al- lows the model to emphasize diagnostically relevant patches while suppressing the influence of irrelevant or background regions in the final prediction.
(4) Survival Analysis. 8 TCGA subsets are used for this
task, including TCGA-BRCA (n 1025), TCGA-UCEC (n = 497), TCGA-STAD (n = 365), TCGA-LUAD (n 457). TCGA-LUSC (n = 454), TCGA-SKCM (n = 417), TCGA-KIRC (n = =500), and TCGA-KIRP (n = -263..
Our model is trained end-to-ent using task- approprpoee loss functions. For classification tasks, we employ cross-entropy
6

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Table Results of survival prediction on 8 TCGA subsets with C-Index score reported. The best results are in bold, and the second-best
results underlined. Rows in gray color represent Self-Altn-based methods.
Methods
BRCA
UCEC
STAD
LUAD
LUSC
SKCM
KIRC
KIRP
Avg. (*)
Mean Pooling
40.033
0.685
0.5044
0..044
4..044
+0.045
0.61640.068
0.643
0.641
Max Pooling
0.67 40.037
0.649
0.53940.054
0.519
0.672
0.581 0.60640.064
0.581 +0.031
0.593
0.612
ABMIL
0.000
0.660
0.59840.075
0..83
#0.063
0.707
0.663+0.084
0.606
00.07
0.641
DS-MIL
0.078
0..05
0.547
0.5082
0.705
0.671 0.671 +0.045
0.58220.079
0.593
0.619
DTFD
0.095 0.67220.032
0.597
0.570
0.71640.046
0.64940.048
0.631
0.598 20.023
0.641
WiKG
0.06940.019
0.648
00.066
0.588
0.71 +0.030
0.65440.083
0.610
0.651 #0.030
0.647
MambaMIIL
0.69440.012
0.60820.022
0.590
0.609
0.69220.044
0.606+0.046
0.61440.067
0.641 #0.042
0.647
TransMIL
10.43
1..429
0.56810.082
0.4981
10.056
0.58610.042
10.043
0.567 1.0092
0.595
HIPT
1..0710.006
0.647
10.031
10.085
0.5361
0.603
0.61610.039
0.578 10.036
0.559 1:0. 103
0.606
HistGen 10.081
0.66410.041
10.044
0.56210.068
10.028
10.057
10.999
0.560 10.055
0.628
RRT-MIL
10.022
0.67210.029
10.004
10.069
0.66910.070
0.65210.063
0.59210.048
0.56110.041
0.561
10.041
0.623
LongMIL
0.65210.024
0..032
1599110.04
0.535
0.67410.074
0.65810.026
0.631
0.626110.05
0.625
Querent (Ours)
0.720
0.691
0.636
0.608 10.050
0.732
0.68220.045
0.62610.038
1..024
0.670
These four tasks represent key challenges across the compu- tational pathology pipeline: from molecular-level analysis (biomarker/mutation prediction) to clinical assessment (sub- typing) and patient outcomes (survival).
tion prediction (TCGA-LUAD TP53), and cancer subtyping (UBC-OCEAN), Across all three tasks, our method consis- tently achieves state-of-the-ar performance. For BCNB-ER, Querent achieves an accuracy of 0.836, AUC of 0.848, and Fl score of 0.739, outperforming all MIL baselines and monnsfomer-bbseed counterparts. On TCGA-LUAD TP53, our method demonstrates superior performance with an ac- curacy of 0.678 and AUC of 0.706, showing particular ad- vantages over other transformer-based methods. For the complex task of 5-class cancer subtyping on UBC-OCEAN, Querent achieves significant improvements with accu- racy of 0.835 and AUC of 0.956, surpassing the second-best method by 1.2% and 1.0%, respectively.
We use 5-fold cross-validation for model training and evalua- tion and report the results' mean and standard deviation. For classification tasks, accuracy (ACC), area under the curve (AUC), and Fl Score are reported for comparison. For sur- vival analysis, the concordance index (C-Index) (Harrell et al., 1982) used for evaluation.
4.2. Baselines
We employ SOTA models including Max/Mean Pooling, ABMIL (Ilse al., 2018), 8), DS-MIL (Li et al., 2021a), DTFD (Zhang et al., 2022), WiKG (Li al., 2024b), and Mam- baMIL (Yang al., 2024) for comparison. For Transformer- based MIL models, we involve TransMIL (Shao et al., 2021), HIPT (Chen et al., 2022), HistGen (Guo et al., 2024), RRT- MIL (Tang et al., 2024), and LongMIL (Li et al., 2024a). For all models in comparison, we use PLIP (Huang et al.,
Moreover, Tab. 2 demonstrates the survival prediction re- sults across eight TCGA cancer types. Querent achieves the highest average C-Index of 0.670, representing substan- tial improvement over the second-best methods (WiKG and MambaMIL, 0.647). The superior performance is consistent across different cancer types, with our method achieving the best results on BRCA (0.720). UCEC (0.691), STAD (0.636), LUSC (3.73 SKCM (0.682) and KIRP (0.667). This consistent improvement across diverse cancer types demonstrates the robustness and generalizability of our method capturing prognostic features from WSIs.
2023) as the patch feature encoder. Further information on all baselines and implementation details are available in Appendix E and Appendix F, respectively.
5. Results
5.2. Ablation Study
5.1. Comparison to SOTA Methods
Region-level Metadata Summarization Strategy. We eval- uate the effectiveness of different region-level feature sum- marization strategies by comparing pairwise distance rela-
Tab. 1 shows the evaluation results on classification tasks, including biomarker prediction (BCNB-ER), gene muta-

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
(a) Pearson Correlation
(b) Mean Squared Error
K=8
K=16
K=24
K=32
K=64
Correlation Coefficient
1.00
08-
0.15
Pc0.005
0.95
0.90
0.10
0.75
0.9
0.85
Score
0.05
0.80
0.7
0.-
0.00
Pc0.005
077
min
make
mean
meanstd
momax
min
max
meatt
meanstd
minmax
Accursay
AUIC
Acouracy
AUC
F1 Scorr
Figure 4. Ablation on Querent using min, max, mean, and mean tst strategies compared to our min-max method on TCGA-
(a) ICGA-LUAD TP53
(b) UBC-OCEAN
LUAD TP53 gene mutation dataset (details Appendix G.I).
Figure 5. Ablation on Querent using different region size K, with reported results on TCGA-LUAD for TP53 gene mutation predic- tion and UBC-OCEAN for ovarian cancer subtyping.
tionships between regions before and after summarization. As shown in Fig. our proposed min-max summarization strategy demonstrates superior performance as measured by both the Pearson correlation coefficient and mean squared error (MSE). Specifically, our min-max approach achieves the highest correlation (0.975) and lowest MSE (0.008), sig- nificantly outperforming (p < 0.0055) alternative methods including individual min (correlation: 0.937, MSE: 0.012) or max summarization (correlation: 0.959, MSE: 0.018), mean (correlation: 0.902, MSE: 0.058), and mean-std ap- proaches (correlation: 0.897, MSE: 0.062), indicating that combining both minimum and maximum values effectively preserves the structural relationships between regions.
(a) Memory Requirements
Compunatoonal Comp city (GPL0PM) GP.0Ps)
(b) Computational Complexity
Full Self Attentttennoon
Rull Self-Attantio
20
Query-Aware Attertion
Query Awarre Attention
Memoo
20
100
Langth
Langth (x10
Figure 6. Computational efficiency comparison between full self- attention and our query-aware aware approach. (a) Memory requirements in gigahytes and (b) computational complexity in GFLOPs across different sequence lengths. See detailed analysis in Appendix H.
Table Ablation on the importance estimation module of Quer- ent, with reported results on TCGA-LUAD TP53 mutation and UBC-OCEAN ovarian cancer datasets (details in Appendix G.2).
cer subtyping (UBC-OCEAN), where our approach reaches 0.835 accuracy and 0.956 AUC, representing absolute im- provements of 8.9% in accuracy and 4.2% in AUC over random selection, demonstrating the effectiveness of our region importance estimation strategy.
Importance Estimation
Accuracy
AUC
Fl Score
TCGA-LUUD TP53 Gene Mutation Prediction
Region size Querent. Fig. shows the impact of apply- ing different region size for our Querent model during the region metadata summarization process. For TP53 mutation prediction in TCGA-LUAD, moderate region sizes (K=24) yielded optimal results with an AUC of 0.706, while both smaller (K=8) and larger (K=64) regions showed decreased performance. In UBC-OCEAN ovarian cancer subtyping, K=16 emerged as the clear optimal choice, achieving the highest accuracy (0.835) and AUC (0.956), with perfor- mance gradually declining as region size increased. These results demonstrate that moderate-sine sized regions are most effective for both tasks. This aligns with pathological intu- ition, as larger regions may dilute the discriminative local tissue patterns by aggregating potentially heterogeneous ar- cas, while smaller regions might miss important contextual information.
Estimation Side Network
0.58020.078
0.660go.012
0.56840.073
Random Region Selection
0.649
0.686go.071
0.64340.061
Querent (Ours)
0.678
0.706 0.672+0.070
UBC-OCEAN Ovarian Cancer Subtyping
Estimation Side Network
0.731
0.903_a.018
0.690+0.028
Random Region Selection
0.74640.056
0.914g0.036
0.697
Querent (Ours)
0.835
0.956+0.019 0.806
Region Importance Estimation Strategy, Meanwhile, we conduct an ablation study comparing our proposed region importance estimation module against random region se- lection and an estimation side network, reported in Tab. 3. Our approach consistently outperforms both baselines across TCGA-LUAD and UBC-OCEAN datasets. For TP53 mutation prediction, our method achieves 0.678 accuracy and 0.606 AUC, showing moderate improvements over the baselines. The gains are more substantial in ovarian can-
5.3. Computational Efficiency
We analyze the computational efficiency of our query- attention mechanism compared to full self-attention (illus-
8

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
trated in Fig. 6). While full self-attention reaches 37GB memory and 10,000 GFLOPs at 100k sequence length with quadratic scaling, our approach achieves near-linear scaling, requiring only IGB memory and 500 GFLOPs for a 100k patch sequence. This significant reduction in computational overhead enables our model to efficiently process longer sequences, which crucial for gigapixel whole-slide image analysis.
V. E., Klimstra, D. S., and Fuchs, T. J. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8); 1301-1309, 2019.
Chen, R. J., Chen, C., Li, Y., Chen, T. Y., Trister, A. D., Kr- ishnan, R. G., and Mahmood, F. Scaling vision transform- ers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16144- 16155, 2022.
6. Conclusion
In this study, we present Querent, query-aware dynamic modeling framework for understanding long-range contex- tual correlations in gigapixel WSIs. Inspired by pathological observations, our method aims to dynamically adapt to the unique context of each patch while remaining computa- tionally efficient through region-level metadata summariza- tion and importance- based attention. Extensive experiments across multiple CPath tasks demonstrate the superior perfor- mance of our model compared to state-of-the-art approaches, establishing its effectiveness in whole slide image analysis.
Chen, R. J., Ding, T., Lu, M. Y., Williamson, D. F., Jaume, G., Song, A. H., Chen, B., Zhang, A., Shao, D., Shaban, M., et al. Towards a general-purpose foundation model for computational pathology. Nature Medicine. 30(3): 850-862, 2024.
Chikontwe, P., Kim, M., Nam, S. J., Go. H., and Park, S. H. Multiple instance learning with center embeddings for histopathology classification. In Medical Image Comput- ing and Computer Assisted InterrentionMIICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part V 23, pp. 519-528. Springer, 2020.
Acknowledgements
This work was supported by the National Natural Science Foundation of China (No. 62202403), Hong Kong Innova- tion and Technology Commission (Project No. MHP/002/22 and ITCPD/11-9). and Research Grants Council of the Hong Kong Special Administrative Region, China (Project No. R6003-22 and C4024-22GF).
Cox, D. R. Regression models and life-tables. Journal of the Royal Statistical Society: Series B (Methodological), 34(2):187-202, 1972.
Cui, M. and Zhang, D. Y. Artificial intelligence and com- putational pathology. Laboratory Investigation, 101(4): 412 422, 2021.
Impact Statement
Dao, T., Fu, D., Ermon, S., Rudra, A., and Ré, C. Flashat- tention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Process- ing Systems, 35: 35:16344-16359 2022.
This work advances computational pathology through a novel query-aware dynamic long contextual modeling framework for analyzing gigapixel whole slide images. Our research utilizes publicly available pathology datasets with appropriate institutional approvals. While our work demon- strates potential for improving the efficacy and efficiency of histopathological analysis, it is currently intended for research purposes only. Clinical implementation would re- quire extensive external validation and regulatory approval. Our methodology and findings are shared to advance scien- tific understanding in computational pathology rather than for immediate clinical application.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010.11922 2020.
Du, Y., Zhuang, J., Zheng, X., Cong, J., Guo, L., He, C., Luo, L., and Li, X. Beyond h&e: Unlocking pathological insights with polarization via self-supervised learning. arXiv preprint 2xXiv:5503.09335 2025.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752. 2023.
References
Araujo, A., Norris, W., and Sim, J. Computing receptive fields of convolutional neural networks. Distill, 4(11): c21, 2019.
Guo, Z., Ma, J., Xu, Y., Wang, Y., Wang. L., and Chen, H. Histgen: Histopathology report generation via local- global feature encoding and cross-modal context inter- action. In International Conference on Medical Image Computing and Computer-Assiste Intervention, pp. 189 199. Springer, 2024.
Campanella, G., Hanna, M. G., Geneslaw, L., Miraflor, A., Werneck Krauss Silva, V. Busam, K. J., Brogi, E., Reuter,

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Guo, Z., Xiong, C., Ma, J., Sun, Q., Feng, L.,, Wang, J., and Chen, H. Focus: showledge-enhancel adaptive visual compression for few-shot whole slide image classifica- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2025.
Li, H., Yang, F., Zhao, Y., Xing, X., Zhang, J., Gao, M., Huang, J., Wang, I.., and Yao, J. Dt-mil: deformable transformer for multi-instance learning on histopathologi- cal image. In Medical Image Computing and Computer Assisted Intervention-MICCA 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24, pp. 206-216. Springer, 2021b.
Han, D., Pan, X., Han, Y., Song, S., and Huang, G. Flatten transformer: Vision transformer using focused linear at- tention. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 019555969777 2023.
Li, H., Yang, L.,, Wang, Y., Wang, L.,, Chen, G., Zhang, L., and Wang, D. Integrative analysis of tp53 mutations in lung adenocarcinoma for immunotherapies and prognosis. BMC bioinformatics, 24(1):155, 2023.
Harrell, F. E., Califf, R. M., Pryor, D. B., Lee, K. L., and Rosati, R. A. Evaluating the yield of medical tests. Jama,
247(18))2543-2446 1982.
Li, H., Zhang, Y., Chen, P., Shui, Z., Zhu, C., and Yang, L. Rethinking transformer for long contextual histopathol- ogy whole slide image analysis. In Globerson, A., Mackey, I.., Belgrave, D., Fan, A., Paquet, U., Tomezak, J., and Zhang, C. (eds.), Advances in Neural Informa- tion Processing Systems, volume 37, pp. 101498-101522 Curran Associates, Inc., 2024a.
Heindl, A., Nawaz, S., and Yuan, Y. Mapping spatial het- erogeneity in the tumor microenvironment: a new era for digital pathology. Laboratory investigation, 95(4): 377-384, 2015.
Hou, W., Yu, I.., Lin, C., Huang, H., Yu, R., Qin, J., and Wang, L. 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In Proceedings of the AAAI confer- ence on artificial intelligence, volume 36, pp. 933-941, 2022.
Li, J., Chen, Y., Chu, H., Sun, Q., Guan, T., Han, A., and He, Y. Dynamic graph representation with knowledge- aware attention for histopathology whole slide image analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition, pp. 11323- 11332. 2024b.
Huang. Z.. Bianchi, F., Yuksekgonul, M., Montine, T. J., and Zou, J. A disual-languaae foundation model for pathology image analysis using medical twitter. Nature medicine, 2909):2377- 2023.
Lin, T., Yu, Z., Hu, H., Xu, Y., and Chen, C.-W. Interven- tional bag multi ialti-iinttacee learning on whole-slide patho- logical images. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 19830-19839, 2023.
Ilse, M., Tomezak, J., and Welling, M. Attention-based deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018.
Lu, M. Y., Williamson, D. F., Chen, T. Y., Chen, R. J., Bar- bieri, M., and Mahmood, F. Data-efficient and weakly su- pervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.
Jin, C., Guo, Z., Lin, Y., Luo, L., and Chen, H. Label-efficient deep learning in medical image analy- sis: Challenges and future directions. arXiv preprint arXiv:2303. 12484, 2023.
La, M. Y., Chen, B., Zhang, A., Williamson, D. F,, Chen, R. J., Ding, T., Le, L. P., Chuang, Y.-S., and Mahmood, F. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEE/VV conference on computer vision and pattern recognition, pp. 19764-19775, 2023.
Kaban, A. Improved bounds on the dot product under ran- dom projection and random sign projection. In Proceed- ings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 487-496, 2015.
Katzman, J. L., Shaham, U., Cloninger, A., Bates, J., Jiang, T., and Kluger, Y. Deepsurv: personalized treatment rec- ommender system using a cox proportional hazards deep neural network. BMC medical research methodology, 18: 1-12, 2018
Lu, M. Y., Chen, B., Williamson, D. F., Chen, R. J., Liang, I. Ding. T., Jaume, G., Odintsov, I., Le, L. P., Gerber, G., et al. A visual-language foundation model for com- putational pathology. Nature Medicine, 30(3):863-874 2024.
Li, B., Li, Y., and Eliceiri, K. W. Dual-stream multiple instance learning network for whole slide image clas- sification with supervuue contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318-14328, 2021a.
Ma, J., Guo, Z.,, Zhou, F., Wang, Y., Xu, Y., Cai, Y., Zhu, Z., Jin, C., Jiang, Y. L. X., Han, A., et al. Towards a generalizable pathology foundation model via unified knowledge distillation. arXiv preprint arXiv:2407.18449 2024.
10

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Niazi, M. K. K., Parwani, A. V., and Gurean, M. N. Digital pathology and artificial intelligence. The lancet oncology, 20(5):e253<2261, 2019.
Wang, X., Xiang, J., Zhang, J., Yang. S., Yang, Z., Wang, M.-H., Zhang, J., Yang. W., Huang, J., and Han, X. Scl-we: Cross-slide contrastive learning for weakly- supervised whole-slide image classification. Advances in neural information processing systems, 15:18009-18021, 2022.
Shao, Z., Bian, H., Chen, Y., Wang, Y. Zhang, J., Ji, X., et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147, 2021.
Xiang, T., Song, Y., Zhang, C., Liu, D., Chen, M.. Zhang, F., Huang, H., O' O'Donnell, L.,, and Cai, W. Dsnet: A dual-stream framework for weakly-supervised gigapixel pathology image analysis. IEEE Transactions on Medical
Song, A. H., Jaume,, G., Williamson, D. F., Lu, M. Y., Vaidya, A., Miller, T. R., and Mahmood, F. Artificial intelligence for digital and computational pathology. Na- fure Reviews Bioengineering, 1(12):930-92) 94999999999,00 2023.
Imaging. 41(8):2180 80-2090, 2022.
Xiong, C., Chen, H.,, Sung. J. J., and King. I. Diagnose like a pathologist: Transformer-enabled hierarchical attention- guided multiple instance learning for whole slide image classification. arXiv preprint arXiv:2301.088 000-223010022... 125, 2023.
Song, A. H., Chen, J., Ding, T., Williamson, D. Jaume, G., and Mahmood, F. Morphological prototyping for un- supervised slide representation learning in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11566- 11578, 2024a.
Xiong, C.,, Chen, H., Zheng, H., Wei, D., Zheng, Y., Sung, J. J., and King, I. Mome: Mixture of multimodal ex- perts for cancer survival prediction. In International Conference on Medical Image Computing and Computer- Assisted Intervention, pp. 118-328. Springer, 2024a.
Song, A. H., Chen, R. J., Jaume, G., Vaidya, J., Baras, A., and Mahmood, F. Multimodal prototyping for cancer sur- vival prediction. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A., Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Leurning, volume 235 of Proceedings of Machine Learning Research, pp. 46050-46073. PMLR, 21 21-27 Jul 2024b.
Xiong, C., Lin, Y., Chen, H., Zheng, H., Wei, D., Zheng, Y., Sung, J. J., and King, I. Takt: Target-aware knowledge transfer for whole slide image classification. In Inter- national Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 503-513. Springer, 2024b.
Sun. Q., Guo, Z., Peng, R., Chen, H., and Wang, J. Any-to- any learning in computational pathology via triplet mul- timodal pretraining. arXiv preprint arXiv:2505. 12711, 2025.
Xiong, C.,, Chen, H., and Sung, J. J. A survey of pathology foundation model: Progress and future directions. arXiv preprint arXiv:2504.04045, 2025a.
Xiong, C., Guo, Z., Xu, Z., Zhang, Y., Tong, R. K.-Y,, Yeo, S. Y., Chen, H., Sung, J. J., and King. I. Beyond linearity: Squeeze-ame -nd recclacali blocks for few-shot whole slide image classification. arXiv preprint arXiv:2505. 15504, 2025b.
Tang, W., Zhou, F., Huang. S., 7hu, X., Zhang, Y., and Liu, B. Feature re-embedding: Towards foundation model- level performance in computational pathology. In Pro- ceedings of the IEEF/CVF Conference on Computer Vi- sion and Pattern Recognition, pp. 11343-11352, 2024.
Xiong, Y., Zeng, 7., Chakraborty, R., Tan, M., Fung, G., Li, Y., and Singh, V. Nyströmformer: A nyström-baad based algorithm for approximating self-attention. In Pruceed- ings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14138 14148, 2021.
Tomczak, K., Czerwinska, P., and Wiznerowicz, M. Review the cancer genome atlas (tcga): an immeasurable source of knowledge. Contemporary OncologyWsspócceena Onkologia, 2015(1):68-77, 2015.
Xu, F., Zhu, C., Tang, W., Wang, Y., Zhang, Y., Li, J., Jiang, H., Shi, Z., Liu, J., and Jin, M. Predicting axillary lymph node metastasis in early breast cancer using deep learning on primary tumor biopsy slides. Frontiers in Oncology. pp. 4133, 2021.
Van der Laak, J., Litjens, G., and Ciompi, F. Deep learning in histopatholooy: the path to the clinic. Nature medicine, 27(5):775-774, 2021.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Xu, Y. and Chen, H. Multimodal optimal transport-based co-attention transformer with global structure consistency for survival prediction. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 21241 21251, 2023.
Wang, W. Ma. S., Xu, H., Usuyama, N., Ding, J., Poon, H., and Wei, F. When an image is worth 1,024 x 1,024 words: A case study in computational pathology. arXiv preprint 23Xiv:2312.03555 2023.

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Xu, Y., Wang, Y., Zhou, F., Ma, J., Yang, S., Lin, H., Wang,
X., Wang, J., Liang, L., Han, A., et al. A multimodal knowledge-enbanced whole-slide pathology foundation model. arXiv preprint arXiv:2407.15362 2024.
Yang, S., Wang. Y., and Chen, H. Mambamil: Enhanc- ing long sequence modeling with sequence reordering in computational pathology. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 296-306. Springer, 2024.
Yu, J.-G., Wu, Z., Ming. Y., Deng, S., Li, Y., Ou, C., He, C.,, Wang, B.. Zhang, P., and Wang, Y. Prototypical multiple instance learning for predicting lymph node metastasis of breast cancer from whole-slide pathological images.
Medical Image Analysis, 85:102748, 2023.
Yuan, Y. Spatial heterogeneity in the tumor microenviron- ment. Cold Spring Harbor perspectives in medicine, 6 (8):0025583, 2016.
Zadeh, G. and Schmid, M. Bias in cross-entropy-basee training of deep survival networks. IEEE transactions puttern analysis and machine intelligence, 43(9):3126-
3137, 2020.
Zhang, H., Meng, Y., Zhao, Y., Qiao, Y., Yang, X., Coup- land, S. E., and Zheng, Y. Dtfd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 18802-18812, 2022.
Zheng, X., Wen, J., Zhuang, J., Du, Y., Cong, J., Guo, L., He, C., Luo, L., and Chen, H. Diffusion-basee virtual staining from polarimetric mueller matrix imaging. arXiv preprint ar arXiv:2503.01352 2025.
Zheng, Y., Gindra, R. H., Green, J., Burks, E. J., Betke, M., Beane, J. E., and Kolachalama, V. B. A graph- transformer for whole slide image classification. IEEE transactions on medical imaging, 30011)30033-315, 2022.
Zhou, F. and Chen, H. Cross-modal translation and alignment for survival analysis. In Proceedings of the IFFF/CVF International Conference on Computer Vision pp. 21485-21494, 2023.
12

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
A. Algorithm Pseudo Code of Querent
Algorithm 1 Querent: Query-Aware Dynamic Long Sequence Modeling for Gigapixel WSI Analysis
. N region size K, number of regions M 2: Output: Slide-level prediction and attention weights
Input: Patch features X {a1, ...,
3: Parameter: Query projection fq. metadata projections f attention W glv, Wo, classifier fc
4: // Phase 1: Region-level Metadata Summarization
5: for i 1 to MM do
6: Extract region patches: R2 = ( Tik
7: Compute min/max metadata: m, minjee (1...K) Zij. m, maxje(1...K] Zij
Project metadata; m. fmin (m?iin), mi Smax (m
9: end for
10: // Phase 2: Region Importance Estimation
11: for each query patch y do
12: Project query: ④ = (9)
13: for = to M do
14: Compute importance: 8; = max( (i. fi 1. m,))))))
end for
Select top regions: R2
(;))
17: end for
18: // Phase 3: Query-Aware Selective Attention
19: for each selected region R; E R2 do
20:
Compute QKV: [Q, K, V] Wzkr(z)
Compute attention scores: A = softmax OK")
21:
vd,
22: Compute attention output: O = AV
23:
Multi-head attention: O = Concat O1,...,,0) Wo
24: end for
// Phase 4: Attentive Feature Aggregation
Compute attention weights: W/ (x)(
27:
Normalize weights: a. = ΣN exp
expl
28: Aggregate features: 2 = -m 0,01
Compute final prediction: y fz(2)
Return ÿ. &
B. Theoretical Analysis
B.1. Query-Aware Sparse Attention: Theoretically Bounded Approximation of Full Self-Attention
B.I.1. DEFINITIONS AND PRELIMINARIES
Definition B.1 (Full Self-Attention Matrix). Let X E Lxd be the feature matrix sequence of patches, where L. the sequence length and is the feature dimension. The full self-attention matrix B is defined as:
q K
B = exp
d
where qi is the query vector for patch and K, is the key vector for patch j.
Definition B.2 (Query-Aware Attention Matrix). Let X E Ixd be the feature matrix of a sequence of patches, where L is the sequence length and dl is the feature dimension. The query-aware attention matrix is defined as:
exp
9.
if E R4
A
otherwise
where R; the set of top-K relevant regions for query patch qu and Kj the key vector for patch j.
13

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Definition B.3 (Region-Level Metadata). For each region R2 the metadata consists of summary statistics computed from the patches within the region. The primary statistics are the minimum and maximum feature vectors:
min
min Xj. JER.
max
mAx Xj JER.
These vectors are projected via Lipschitz-continuuus neural networks smin and mmax into shared embedding space:
fmin mia). m = Fmax (mxx
min
where fuin and fuax L-Lipschitz continuous with constant L.
B.1.2. TECHNICAL LEMMAS
Lemma B.4 (Region Metadata Approximation). Let B be bound on the input norms [| S B. For any query patch and region R2 assuming L-Lipschitz continuous projection functions, the interaction scores satisfy:
max (q, x3) (q., (iuim S B diiam(R)) = E1
jeRt
and similarly for iy max where diam(R) R2) is the diameter of region R4 in feature space.
Proof. Fix any patch Xj E R. By construction:
Xx
where 3 denotes element-wise comparison. By Lipschitz continuity:
| - -min (x,) S L L|[miin - -xyll S L didmmii
Using Cauchy-Schwarz and the norm bound B:
lgq x,) - (q. m S ||q|- L diam(R,) < B L -diam(R)) = E1
Lemma B.5 (Ranking Stability). Let s; the top-K regions selected by 8; and true interactions differ by most sel-suboplimal regions.
minn (. m If maxjeR, liq. x,) (q, miinn S E1. then
Proof of Lemma B.5. Let SK denote the true lop-K regions ranked by (q, Xg), and SK any region R,
upprox denote those selected by For
SK SK its approximate score satisfies: approx
8;
min
SR
S
R
SK appos
By the approximation error bound E1. we have:
(q. xx) 2 28 8:-66 -1
and (q,xx) S te Vj E R..
Thus, any region R; SK Spppox must satisfy:
(q.
max SR
E1.
RESK
This implies R2 most ssh-sutptl compared to the true top- K regions.
14

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
B. 1.3. MAIN THEOREM
Theorem B.6 (Quer--Aware Attention Approximation), Let A be the query-aware attention matrix (Def. B.2), and B be the full self-attention matrix (Def. B.1). Assume attention scores decay exponentially with spatial distance: exp(-od(i.) j)) bounds the attention score decay for distance d(i.j). For any input sequence X with L patches and d dimensions, there exist random projection matrices W Q, WK E IRddd such that:
B vd
|A F 1
2+
with probability at least 5, provided:
1. The hidden dimension satisfies:
d 2 C1 log(L./5)
è²
where C1 8B* (from inner-product preservation (Kaban, 2015)).
2. The number of selected regions per query satisfies:
k: 2 C2 log =
a
where C2 2 (derived from Step 3).
3. For each region Rú, the diameter satisfies:
c 1 L. 21)
(Ram(R,) S min
4. Regions are spatially separated such that:
Vi d(R, R2) 2 C33
where C3 ensures Em e Csm S.
Proof of Theorem B.6. Step 1: Region Metadata Summarization.
By Lemma B.4, projections m min.x approximate interactions within R; with error e B L diam R4) S
Be vd
The
condition (R.) & 1 ensures region-level interactions respect the exponential decay rate a.
Step 2: Region Importance Estimation.
Using Lemma B.5, approximate scores 8; preserve the true top-K regions up error e1. Thus, R₂ includes all regions with 8; max, 8j 2ET-
Step 3: Bounding Non-Selected Regions.
Under spatial separation d( R(, R2) 2 Co the tail sum becomes:
Cam CC---11
Σ app(-ad(t,)) <
1 Cs Se
JER.
m 1+
Solving for k gives k: 2 I log ([---155)
Setting C3 = 1/2 simplifies this to k 2 2 log (t) (i.e., C2 = 2).
C.
Step 4:
Guarantec.
Using random matrices Wo. Wx (Kaban, 2015), for d 2 8B1 log T./8)/22, we have:
Pr (.. K2) (q. K s 2 1 - 8
15

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Step 5: Frobenius Norm Aggregation.
Normalize e/ absorb the I,2 scaling. The entrywise enror becomes:
Be
2
Be
et//L,
B Vd
- F $ $12
L2
2
Vdl
=
2E
+
Vd
e
B.1.4. DISCUSSION
The theorem shows query-aware attention effectively approximates full attention via region metadata, sufficient embedding dimension, and region selection. Error bounds depend on I,, d, B, ki a, and region diameters. The spatial separation condition ensures the exponential decay property meaningful.
C. Task-specific Losses
C.1. Cross-entropy Loss for Classification Tasks
For classification tasks (biomarker prediction, gene mutation prediction, and cancer subtyping), employ the standard cross-entropy (CE) loss. Given WSI X with slide-level label y, and model prediction ý f(X;0) where ý IRC represents the predicted probabilities over C' classes, the CE loss is defined as:
C Lee = Σ Ua log(ūe)
(10)
where ye is the one-hot encoded ground truth label for class c. The final classification loss is averaged over all WSIs in the training set:
Lels
(()
(11)
N
where N is the number of WSIs in the training set and (1) is the CE loss for the i-th WSI.
This loss function encourages the model to output probability distributions that assign high probabilities to the correct classes while minimizing probabilities for incorrect classes, effectively training the query-aware attention mechanism to focus on diagnostically relevant regions within each WSI.
C.2. Negative likeliood (NLL) Loss for Survival Analysis
The NLL survival loss (Zadch & Schmid, 2020; Song et al., 2024b) generalizes the standard negative log-likelihood loss to accommodate censored data. The goal is to predict patient survival based on their patient-level embedding. Fuatiem E Rd2d For each patient, we consider and indicates the last known follow-up, and (2) time-to-event 44 representing either time until death 0) time until last follow-up (c 1).
key pieces of information: (1) censorship status c, where e 0 indicates observed death
Rather than directly predicting t, we discretize time into non-overlapping intervals (fy-1.ty), where j E [1, mi], based on the quartiles of observed survival times, and denote each interval as Yý- This transforms survival prediction into a classification problem, where each patient's outcome defined by (apatients c).
For each interval, we compute a hazard function frazard (gy paatteett S(ūr). ). where S is the sigmoid activation. Intuitively, Imazand (yy ( represents the probability of death occurring within interval (4)-1,4z). We also define survival function faurv (Uy paatee)) = I[ (1 frazadd ( yk ppatien)) that represents the probability survival up to interval (ty-1,ty)-
The NLL survival loss for dataset of Np patients can then be formalized as:
16

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
L (11 patient ,/y, ( = Σ -c(() (og(fsrrry)" ponien))
- -11 (1- log( (w)=
pubsem )) - (1)c()
log log( (v)
| 15
patient ))
(12)
The first term enforces high survival probability for patients still alive at their last follow-up. The second term ensures high survival probability up to the time interval before death for uncensored patients. The third term promotes accurate prediction of the death interval for uncensored patients.
D. Datasets
BCNB/ (Xu et al., 2021) for biomarker prediction. Short for Early Breast Cancer Core-Needle Biopsy WSI, the BCNB dataset includes core-needle biopsy WSIs of early breast cancer patients and the corresponding clinical data. 1038 WSIs are annotated with their corresponding estrogen receptor (ER) expression, which is a favorable prognostic parameter in breast cancer, and predictor for response endocrine therapy.
TCGA-LUAD* (Tomezak et al., 2015) for TP53 gene mutation prediction. TCGA-LUAD TP53 dataset includes 469 lung adenocarcinoma WSIs with their corresponding annotation of mutation of the TP53 tumor suppressor gene, which one of the most mutated genes in lung adenocarcinoma and represents a vital role in regulating the occurrence and progression of cancer (Li al., 2023).
UBC-OCEAN for ovarian cancer subtyping. UBC-OCEAN is large-scale ovarian cancer dataset collected from over 20 medical centers across four continents. The dataset collects 527 ovarian cancer WSIs, representing five major ovarian carcinoma subtypes: high-grade serous carcinoma, clear-cell carcinoma, endometrioid, low-grade serous, and mucinous carcinoma, along with several rare subtypes. Each subtype exhibits distinct morphological patterns, molecular profiles, and clinical characteristics, making accurate subtype classification crucial for treatment planning. The challenge of accurate subtype identification is particularly relevant given the growing emphasis on subtype-specific treatment approaches and the limited availability of specialist gynecologic pathologists in many regions.
TCGA Subsets* (Tomezak et al., 2015) for survival analysis. In this work, we include 8 TCGG sub-datasets for unimodal survival prediction:
1. TCGA-BRCA: Breast Invasive Carcinoma dataset from TCGA, containing 1025 whole slide images with associated survival outcome data for breast cancer patients. This comprehensive collection represents one of the largest breast cancer datasets, featuring diverse histological patterns and molecular subtypes including ductal and lobular carcinomas. The survival data encompasses overall survival time and vital status, enabling robust prognostic model development.
2. TCGA-UCEC: Uterine Corpus Endometrial Carcinoma dataset from TCGA, encompassing 497 WSIs and survival data from endometrial cancer cases. This dataset includes various histological grades and stages of endometrial carcinoma, providing valuable insights into the progression and prognosis of gynecologic cancers. The images showcase diverse morphological patterns characteristic of endometrial malignancies.
3. TCGA-STAD: Stomach Adenocarcinoma dataset from TCGA, which includes 365 WSIs and patient survival infor- mation for gastric cancer cases. The dataset represents different anatomical locations within the stomach and various histological subtypes of gastric adenocarcinoma. The survival data is particularly valuable for understanding the relationship between morphological features and patient outcomes in gastric cancer.
4. TCGA-LUAD: Lung Adenocarcinoma dataset from TCGA, containing 457 WSIs and survival outcome data from lung adenocareinoma patients. This collection captures the heterogeneous nature of lung adenocarcinomas, including various growth patterns and degrees of differentiation. The survival information is crucial for developing predictive models for one of the most common types of lung cancer.
https://portall.dd
htw.ppooorrrrgg...ccee...ooooooo cancer.gov/
hhtps:/wwwwkkgge/ www.ww.wggeeeo.oooo
'httss//portall www.orttalgg....m..o... www..cance.goo...oo
17

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
5. TCGA-LUSC: Lung Squamous Cell Carcinoma dataset from TCGA, providing 454 WSIs and survival data for lung squamous cell carcinoma cases. These images showcase the distinct morphological features of squamous cell carcinomas, including keratinization and intercellular bridges. The dataset enables comparative studies between different types of lung cancers and their prognostic factors.
6. TCGA-SKCM: Skin Cutaneous Melanoma dataset from TCGA, consisting of 417 WSIs and survival information from melanoma patients. This collection includes primary and metastatic melanoma cases, capturing the diverse histological patterns and progression stages of this aggressive skin cancer. The survival data is particularly relevant for understanding metastatic potential and treatment response.
7. TCGGA-IIRC Kidney Renal Clear Cell Carcinoma dataset from TCGA, containing 500 WSIs with survival outcome data for kidney cancer patients. The dataset showcases the characteristic clear cell morphology and various grades of renal cell carcinoma. The survival information helps in understanding the prognostic implications of morphological variations kidney cancer.
8. TCGA-KIRP: Kidney Renal Papillary Cell Carcinoma dataset from TCGA, providing 263 WSIs and survival data for papillary renal cell carcinoma cases. This collection represents a distinct histological subtype of kidney cancer, featuring papillary architecture and different cellular patterns. The survival data enables comparative analysis with other renal cancer subtypes.
All these datasets are part of The Cancer Genome Atlas (TCGA) program and contain high-resolution histopathology whole slide images along with corresponding patient survival data. Each dataset has been digitized following standardized protocols and includes detailed clinical annotations. The images are typically scanned at 40x magnification, providing high-detail visualization of cellular and architectural features. These comprehensive resources enable researchers to develop and validate computational methods for survival prediction based on histopathological features, facilitating advances in precision oncology and personalized medicine approaches.
E. Baselines
Max/Mean Pooling. Mean Pooling represents one of the most straightforward aggregation strategies in multi-instance learning. This approach processes all instances within a bag equally, combining their features through average pooling to create a single, unified representation. While simple in nature, this method effectively captures the collective characteristics of all instances, making it particularly suitable for cases where every instance contributes meaningful information to the overall bag classification. Max Pooling, on the other hand, adopts a more selective approach by identifying and utilizing the most prominent features across all instances. This strategy operates under the assumption that the most distinctive or highest-activater features are the most relevant for classification. By focusing on these peak features, max pooling can effectively highlight the most discriminative patterns within the bag, though it may potentially overlook more subtle, collective patterns that could be captured by other methods.
ABMIL (Isse et al., 2018). Attention-Base Multiple Instance Learning (ABMIL) enhances bag-level feature aggregation by incorporating a learnable attention mechanism that adaptively weights the importance of different instances within a bag. Unlike fixed pooling strategies, ABMIL computes attention scores for each instance based on their learned representations, enabling the model to emphasize the most informative elements while downweighting less relevant ones. This adaptive weighting scheme, combined with its inherent interpretability through attention weights, makes ABMIL particularly effective for scenarios where instances have varying levels of relevance to the classification task.
DS-MIL (Li et al., 2021a). Dual-Stream Multiple Instance Learning (DS-MIL) advances bag-level feature aggregation through a novel two-stream architecture that combines the benefits of instance-level and bag-level learning. The model's first stream employs max pooling to identify the most discriminative instance (critical instance), while the second stream computes attention weights for all instances based on their learned distance to the critical instance. By fusing these complementary streams and incorporating trainable distance measurements, DS-MIL creates more nuanced decision boundary that better captures the relationships between instances, making it particularly effective for scenarios with complex instance distributions within positive bags.
DTFD (Zhang et al., 2022). DTFD introduces an innovative dual-tier framework for whole slide image classification that addresses the inherent challenges of limited training samples with high instance counts. The model's key innovation lies in its pseudo-bag strategy, which virtually increases the number of training bags by randomly splitting instances from each
18

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
original slide into smaller subsets. A two-tier architecture processes these pseudo-bags: the first tier applies attention-based MIL to the pseudo-bags, while the second tier distills features from the first tier's outputs generate final slide-level predictions. This hierarchical approach, combined with multiple feature distillation strategies, enables more effective leamnng from limited training data while maintaining robustness against noise in positive instance distributions.
WiKG (Li et al., 2024b). WiKG introduces a dynamic graph representation framework for whole slide image analysis that conceptualizes the relationships between image patches through a knowledge graph structure. The model employs a dual-stream approach: head embeddings that explore correlations between patches and tail embeddings that capture each patch's 's contribution others. These embeddings are then used to dynamically construct directed edges between patches based on their learned relationships. A knowledge-awaal attention mechanism aggregates information across patches by computing attention scores through the joint modeling of head, tail, and edge embeddings. This architecture allows WiKG to both capture long-range dependencies between distant patches and model directional information flow, overcoming key limitations of conventional graph-based and instance- based approaches for histopathology image analysis.
MalbaMII. (Yang et al., 2024). MambaMIL. introduces a novel approach to whole slide image analysis by incorporating Selective Scan Space State Sequential Model, i.e., Mamba (Gu & Dao, 2023), into multiple instance learning with linear complexity. The model's core innovation lies in its Sequence Reordering Mamba (SR-Mamba) architecture, which processes instance sequences in dual streams- one preserving the original sequence order and another utilizing reordered sequences. This dual-stream approach enables more comprehensive modeling of relationships between instances while maintaining computational efficiency. The SR-Mamba module dynamically reorders instance sequences within non-overlapping segments, allowing the model to capture both local and global dependencies between patches. Through this sequence reordering mechanism and linear-time sequence modeling inherited from Mamba, the model effectively addresses common challenges in whole slide image analysis such as overfitting and high computational overhead.
TransMIL (Shao et al., 2021). TransMIL introduces a novel correlated multiple instance learning framework that fundamentally reimagines how relationships between instances are modeled in whole slide image analysis. Unlike traditional MIL approaches that assume instances are independent and identically distributed, TransMIL leverages Transformer architecture to capture both morphological and spatial correlations between patches. The model employs a TPT (Transformer Pyramid Translation) module that combines multi-heat atte-antention for modeling instance relationships with position-aware encoding to preserve spatial context.
HIPT (Chen et al., 2022). HIPT (Hierarchical Image Pyramid Transformer) is a novel architecture designed specifically for analyzing gigapixel whole-slide images in computational pathology. It processes images in hierarchical manner across multiple scales- from cell-level features (16x16 pixels) to larger tissue regions (4096x4096 pixels)- mirroring how pathologists examine slides by zooming in and out. This hierarchical approach allows HIPT to capture both fine- ine-grained cellular details and broader tissue patterns simultaneously. The model employs a series of Vision Transformers arranged in a pyramid structure to aggregate information across these different scales, enabling it to learn representations that incorporate both local and global tissue contexts.
HistGen (Guo et al., 2024). HistGen is a hierarchical architecture designed for efficient processing and representation learning of gigapixel whole slide images (WSIs). At its core, HistGen employs novel local-global hierarchical encoder that processes WSIs in region-to-slide manner, capturing both fine-grained tissue details and broader contextual patterns. The framework first segments WSIs into regions, processes them using a pre-trained vision transformer backbone, and then hierarchically aggregates information across different scales through a series of attention-based modules. This hierarchical design allows the model to effectively manage the extreme size of WSIs while maintaining meaningful spatial relationships. Particularly noteworthy is HistGen's ability to learn robust and transferable representations through its region-aware aware approach, which helps bridge the gap between local tissue patterns and global slide-level characteristics.
RRT-MIL (Tang et al., 2024). RRT-MIL introduces novel re-embedding paradigm for multiple instance learning in computational pathology that addresses a key limitation of existing approaches - the inability to fine-tune pre-extracted image features for specific downstream tasks. At its core is the Re-embedded Regional Transformer (R2T), which processes pathology images through a hierarchical structure that respects the natural organization of tissue at different scales. The framework first divides whole slide images into regions and processes them using a pre-trained feature extractor. Then, it employs two key components: a Regional Multi-head Self-attention module that captures fine mme-grained local patterns within each region, and a Cross-region Multi-head Self-attention module that models relationships between different regions. This regional approach allows RRT-MIL to efficiently handle the extremely large size of pathology images while maintaining the ability to capture both local and global tissue patterns.
19

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
LongMIL (Li et al., 2024a). LongMIL proposes novel hybrid Transformer architecture specifically designed for whole slide image analysis. The method addresses key challenges in processing gigapixel pathology images by introducing local-global attention mechanism that balances computational efficiency with modeling capability. By analyzing the low-rank nature of attention matrices in long sequences, LongMIL incorporates local attention patterns in lower layers while maintaining global context through selective attention in higher layers. This design not only reduces computational complexity but also improves the model's ability to handle varying image sizes and spatial relationships in histopathology slides.
F. Implementation Details
We implement Querent using PyTorch. For the feature extraction backhone, we utilize CPath pre-trained vision-language foundation model PLIP (Huang et al., 2023) to obtain 512-dimensional patch features. Each region contains 16/24/28 patches (depending on the datasets used), which provides good balance between computational efficiency and contextual coverage. The model consists of attention heads, with the hidden dimension set to 512.
For the region-level metadata networks (fmin and fmax), we use single-layer perceptrons with GELU activation. The query projection network f shares the same architecture. During the region importance estimation, we select the top-16 most relevant regions for each query patch, which empirically provides sufficient contextual information while maintaining computational efficiency.
The attention pooling network fa consists of a two-layer MLP with a hidden dimension of 512 and GELU activation. Dropout with a rate of 0.1 is applicd throughout the network to prevent overfitting. We train the model using the Adam W optimizer with a learning rate of 1e-4 for classification tasks and 2e- 4 for survival analysis, with a weight decay of 1e-5, The model trained for 50 epochs with batch size of WSI per GPU. We employ gradient clipping with maximum norm of 1.0 ensure stable training.
G. Additional Experiment Details
G.1. Ablation on Region-level Metadata Summarization Strategy
To thoroughly evaluate the effectiveness of our proposed min-max region-level metadata summarization approach, we conducted a comprehensive ablation study comparing it against several alternative summarization strategies:
Min-only: Using only the element- wise minimum values across patches within each region
Max-only: Using only the element-win maximum values across patches within each region
Mean: Using the average feature values across patches within each region
Mcan-std: Using both mean and standard deviation of features across patches
The evaluation framework compares the pairwise distance relationships between regions before and after summarization. For each region containing N patches with D-dimensional features (N D matrix), we compute two distance matrices: one using the original high-dimensional features, and another using the summarized representations (e.g., D-dimensional for min/max/mean, 2D- -imeennsion for min main-max/mean-tthe For each strategy, we computed pairwise distance matrices between regions using both the original patch features and the summarized metadata, then measured the correlation between these matrices using two metrics: Pearson correlation coefficient and Mean Squared Error (MSE), indicating how well each summarization method preserves the original structural relationships between regions.
The experiment was conducted on the TCGA-LUAD TP53 dataset, with statistical significance assessed using paired t-tests (p 0.005). Our proposed min-max approach demonstrated superior performance in both metrics, achieving significantly higher correlation and lower MSE compared to all alternative strategies. This suggests that capturing both the lower and upper bounds of feature distributions within each region provides more comprehensive and accurate representation of the region's 's characteristics.
Particularly noteworthy is the substantial improvement over the mean-based approaches, indicating that extreme values (minimums and maximums) carry important discriminative information that might be lost when only considering average
20

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
statistics. This aligns with pathological intuition, where both the presence of certain distinctive features (captured by maximums) and their absence (captured by minimums) can be diagnostically relevant.
G.2. Ablation Region Importance Estimation Strategy
To evaluate the effectiveness of our proposed region importance estimation module, we conducted detailed ablation study comparing it against two baseline approaches: random region selection and trainable estimation side network. This study aims to validate the benefits of our query-aware dynamic region importance estimation strategy.
The random region selection baseline serves as a lower bound, employing uniform random sampling to select regions for attention computation. This approach uses a fixed random seed for reprodueibility and maintains the same number of selected regions as our proposed method. While simple, helps quantify the importance of adaptive region selection.
The estimation side network baseline represents network that directly predicts region importance scores. This network consists of two fully-connected layers with GELU activation and processes each region independently. Unlike our proposed method, it doesn't 't consider query-region relationships when estimating importance.
sophisticated approach, implementing trainable neural
We evaluated these approaches on both TCGA-LUAD TP53 mutation prediction and UBC-OCEAN cancer subtyping tasks. The results demonstrate that while both baseline approaches achieve reasonable performance, our query-aware estimation strategy consistently outperforms them. The performance gap is particularly pronounced in the more complex UBC-OCEAN dataset, where accurate region selection becomes crucial due to the heterogeneous nature of ovarian cancer subtypes.
For training stability, we implemented an adaptive update mechanism for the estimation module with a moving average of attention accuracy and periodic updates every 200 forward passes. This strategy helps prevent overfitting and ensures stable convergence of the importance estimation. Additionally, we employed hybrid loss function combining binary classification and ranking components to guide the learning process effectively.
H. Complexity Analysis
We analyze the computational and memory complexity of the Query-Aware Transformer MIL method, comparing it to standard transformer self-attention. Let N denote the total number of patches in a whole slide image (WSI), represent the hidden dimension, be the number of regions (pages), and A indicate the number of selected regions per query. The page size p is the number of patches per region.
H.1. Standard Transformer Self-Attention
Standard transformer self-attention has quadratic computational complexity:
(NV2d)
(13)
This arises from computing attention scores and weighted aggregation for all patch pairs. Memory requirements scale similarly:
O + Nd)
(14)
to store the full attention matrix and key/value representations. This quadratic scaling makes standard self-attention computationally prohibitive for gigapixel WSIs.
H.2. Querent: Query-Aware Dynamic Long Sequence Modeling Framework
Our method operates three phases, with each phase optimized through efficient chunking strategies:
1.2.1. REGION METADATA COMPUTATION
This phase computes min/max metadata for each region:
((Na)
(15)
21

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
Memory requirements
((Rd)
(16)
where R [Npp] (number of regions). For constant p. R scales linearly with N. This phase is highly efficient as it requires only a single pass over the data with constant memory overhead per region.
H.2.2. REGION IMPORTANCE ESTIMATION
This phase estimates relevance scores between queries and region metadata:
(d)
(17)
While the theoretical memory requirements are:
O(NR)
(18)
in practice, we employ chunking strategy with fixed chunk size C' (typically 8192):
(min((N,C) C).
(19)
This chunked processing effectively bounds the memory usage while maintaining computational efficiency through vectorized operations.
H.2.3. SELECTIVE ATTENTION COMPUTATION
This phase performs attention only selected regions:
(Nkpd)
(20)
The theoretical memory requirements are:
O(Nkp + Nd)
(21)
However, with chunked processing (chunk size C):
O(min(N, C)kp + Nd)
(22)
Where and p are constants (typically & 16, p 16), ensuring linear scaling with in both computation and memory.
H.3. Total Complexity
Combining all phases, the total computational complexity is:
O(Nd + N + Nkpd)
(23)
For constant & and p. this simplifies to:
O(N++NRd
(24)
The theoretical memory complexity is:
O NN+N Nkp+Na
(25)
which simplifies to:
O(NR + Nd)
(26)
However, with chunked processing, the practical memory complexity becomes:
(Rd + min(N, C)R min(N, C)kp C)kpp
Nd)
(27)
where C' the chunk size.
22

---

Context Matters: Query-aware Dynamic Long Sequence Modeling of Gigapixel Images
H.4. Practical Efficiency
In practice, our method achieves near-linear scaling (ONd due
Constant & and p reducing the impact of Nkpd
Chunked processing with size 8192 providing bounded memory usage
Efficient vectorized operations in Py Torch minimizing constant factors
Memory-efficiet implementations of matrix operations
Empirical results (Fig. demonstrate that our method requires 1% of the memory and 5% of the computational cost of standard self-attention for large WSIs (100k+ patches). This significant reduction in resource requirements enables the practical processing of gigapixel-scale images while maintaining the modeling power of attention mechanisms.
23


Reviews:
[{"review_id": "ioZose1ho7", "reviewer": "Reviewer_uN59", "summary": "To alleviate the self-attention o(n^2) complexity when modeling WSI, this paper introduces query-based lager-region pruning method to replace linear-attention and local-global attention mechanisms. By ignoring the irrelevant regions to current patches, all the computational cost between current patch and all patches in these regions can be pruned. The evaluations in experiments demonstrate the computational efficiency and performance effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable comments. We respond with detailed interpretations as follows:\n\n`Claims and Evidences`\n> The claim in line 023-025 (abstract) 'the query-aware ...\n\nWe should clarify that Querent provides a theoretically bounded approximation of full self-attention rather than claiming it maintains the exact same expressive power. As demonstrated in Theorem 3.1, our approach maintains the expressiveness within a small constant bound of full self-attention while significantly reducing computational complexity. We will revise this wording in the final version to be more precise: \"Querent achieves a theoretically bounded approximation of full self-attention and meanwhile delivers practical computational efficiency for modeling gigapixel images.\"\n\n`Relation with Full Self-Attn (FlashAttn)`\n\nSince the reviewer's major concern is the necessity of developing a new dynamic attention pattern instead of directly using Flash Attention for full self-attention implementation, here we address these concerns step by step.\n\n**1. Comparison with Full Self-Attention via FlashAttention**\\\nFollowing the reviewer's suggestion, we have implemented a full self-attention-based MIL model (FlashMIL) using FlashAttention to enable the processing of long sequences without memory limitations. Specifically, our implementation applied 4 flash-attn layers to model the WSI patch sequence, followed by a mean operation to obtain the slide-level representation for prediction. The comparison results on our three classification datasets are shown in Table 1 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md). While FlashMIL achieves comparable performance on the BCNB-ER dataset, Querent significantly outperforms it on the more complex TCGA-LUAD TP53 and UBC-OCEAN datasets. This demonstrates that our method provides benefits beyond just addressing memory efficiency.\n\n**2. Relationship Between Querent and Flash Attention**\\\nWhile both Querent and FlashAttention address the computational challenges of self-attention, they do so through fundamentally different approaches. FlashAttention optimizes the implementation of full self-attention through IO-aware algorithms and hardware optimizations, but still computes attention between all pairs of patches. In contrast, Querent introduces a context-dependent attention mechanism that dynamically identifies and focuses only on the most relevant regions for each query patch.\n\nThe superior performance of Querent over FlashMIL can be explained by this contextual selectivity, which serves as an implicit regularization mechanism. By focusing only on relevant regions, Querent filters out noise and irrelevant information that could potentially confuse the model, especially in highly heterogeneous WSIs. This is particularly important in computational pathology, where diagnostically relevant features may be sparsely distributed across the gigapixel image.\n\n**3. Why Querent Improves Results Beyond Memory Efficiency**\\\nThe improved performance of Querent over Flash Attnetion-based implementation can be attributed to several factors. First, the context-aware attention dynamically determines which surrounding regions are most relevant for each patch, allowing Querent to adapt to the heterogeneous nature of WSIs, where different tissue types require different contextual considerations. Second, selective attention acts as a form of implicit regularization by reducing the influence of irrelevant or noisy patches, which is particularly beneficial in weakly-supervised settings with limited training data. Third, while reducing computational overhead, our min-max region metadata approach ensures that potentially important long-range dependencies are still captured, unlike fixed local-global attention patterns that make strong assumptions about which spatial relationships matter. These advantages explain why Querent outperforms Flash-Attention-optimized self-attention.\n\n**4. Compatibility with FlashAttention**\\\nRegarding the potential combination of Querent with FlashAttention: Yes, our method is compatible with and complementary to FlashAttention. While FlashAttention optimizes how attention is computed through IO-aware algorithms, Querent determines which attention computations are most valuable to perform. A combined approach could leverage FlashAttention's efficiency for computing the selected region attention in our Step 3 (Query-Aware Selective Attention), potentially providing even greater computational benefits. This represents an interesting direction for future work.\n\nOur current implementation already demonstrates significant efficiency gains over standard attention (as shown in Figure 6), requiring only ~1% of the memory and ~5% of the computational cost for 100k patches. Even compared to FlashAttention, Querent offers advantages in computational complexity (near-linear vs. quadratic time complexity) while achieving superior performance on CPath tasks.", "claims_and_evidence": "The claim in line 023-025 (abstract) 'the query-aware long contextual dynamic modeling framework, which maintains the expressive power of full self-attention while achieving practical efficiency' is not evident.'\nIf the full self-attention is maintained, how to speed up. I think this expression should be refined", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense.", "theoretical_claims": "I have reviewed all the proofs and found no issues.", "experimental_designs_or_analyses": "I have checked the experimental design. An issue of experiment in this paper is that the performance of full self-attention is not included. To the best of my knowledge, the full self-attention can be implemented via FlashAttention to avoid out-of-memory problem in WSI tasks.", "relation_to_broader_scientific_literature": "The method of this paper may also be applied in other tasks with long-sequence modeling using Transformer, e.g. document-level language understanding and AI4Science tasks with long-sequence DNA, RNA.", "essential_references_not_discussed": "I find that all the essential references are discussed.", "other_strengths_and_weaknesses": "Strengths: The proposed method is novel and can highly speed up Transformer WSI modeling.\n\nWeakness: \n1) The claim on the relationship between 'Querent' and 'full self-attention' should be further discussed.\n2) The Flash-attention (with full self-attention) but linear-memory and quadratic time cost should be compared.\n3) A very important issue: There seems no explanation or motivation on why the method can improve the results. If the motivation is just like weakness 1), the weakness 2) should be validate in rebuttal. If full self-attention (implemented via Flash-attention) cannot reach the results just like Querent, how to explain it?", "other_comments_or_suggestions": "The author should provide more discussion on Flash-attention. \nThe Flash-attention is of linear-memory cost and quadratic-time cost, but the time cost or speed was accelerated by their hardware operations optimization.\nDoes your method can be combined with it? How much of your method surpass it?", "questions_for_authors": "No further question.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aoyOwHrTfx", "reviewer": "Reviewer_Zr67", "summary": "This paper introduces Querent, a query-aware dynamic modeling framework for analyzing whole-slide images in computational pathology. To address the computational inefficiency of standard transformer architectures, which struggle with the quadratic complexity of self-attention in large-scale WSI analysis, the authors propose a novel approach that dynamically adapts attention computation to the most relevant regions for each query patch. The framework includes: 1) Region-Level Metadata Summarization, 2) Query-Aware Attention Mechanism and 3）Efficient Importance Estimation. Experiments on biomarker prediction, gene mutation prediction, cancer subtyping, and survival analysis demonstrate that Querent achieves state-of-the-art performance while significantly reducing computational overhead, enabling efficient processing of gigapixel WSIs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments. We respond to the comments one by one as follows:\n\n`Weakness`\n> The performance of the model depends on the quality of the region-level metadata ...\n\nWe acknowledge that any approach to summarizing region-level metadata, whether it be min/max, mean, or mean-std, will inherently lose some local information. This trade-off is necessary to achieve computational efficiency when handling gigapixel whole slide images. In our work, the min-max strategy was specifically chosen because it captures the extreme values of feature distributions, which are critical for preserving discriminative patterns — especially in heterogeneous tissues. Our additional analysis (see Figure 1 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md)), using the \"Adjusted Average Distance to Summary\" metric, shows that the min-max approach outperforms other summarization methods, particularly in high-heterogeneity regions, where it achieves significantly lower error compared to mean or mean-std methods.\n\nFurthermore, our query-aware attention mechanism complements the summarization by dynamically selecting and weighting the most relevant patches, which helps mitigate the loss of local information and filters out noise. Although some information loss is unavoidable with any summarization method, our experimental results demonstrate that the min-max approach, in combination with our selective attention, provides an effective and robust representation that leads to superior performance across all tasks, even in challenging heterogeneous and noisy scenarios.\n\n`Other Comments or Suggestions`\n> The experiments did not leverage features from latest foundation models ...\n\nWe appreciate this suggestion. As demonstrated in Table 2 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md), we conducted additional experiments using state-of-the-art foundation models (CHIEF and Virchow). Results show that Querent consistently outperforms other methods with these advanced feature extractors, confirming that our method's superiority stems from its long contextual modeling capability rather than the choice of feature extractor. This indicates Querent's contribution is complementary to advances in foundation models and will continue to provide advantages as foundation models evolve.\n\n> The region size in Querent significantly impacts performance, potentially ... \n\nWe appreciate the reviewer's comment on region size impact. While region size is a hyperparameter, our ablation studies (Figure 5) show that moderate-sized regions (16-24) consistently deliver strong performance across diverse datasets. This pattern aligns with pathological intuition, *i.e.*, region size should capture meaningful local context without diluting distinctive tissue patterns. This provides a reliable starting point that significantly narrows the hyperparameter search space, enhancing Querent's practical applicability without extensive tuning.\n\n`Questions`\n> The method can visualize the original WSI corresponding to the K regions selected ...\n\nWe thank the reviewer for this great suggestion and will include visualizations in the revised version.\n\n> It is not clear how the method can avoid the situation where the top K ...\n\nWe address this important concern through two mechanisms: (1) our region importance estimation algorithm provides theoretical guarantees (Theorem 3.1) that selected regions are at most 2$\\epsilon$1-suboptimal compared to the true top-K regions, ensuring minimal information loss; and (2) our min-max summarization strategy (superior in Fig. 4, p<0.005) effectively captures extreme feature distributions, making the metadata highly discriminative for identifying diagnostically relevant areas. Our ablation studies confirm that our approach significantly outperforms random region selection (Table 3, 8.9% accuracy improvement on UBC-OCEAN), demonstrating that our method reliably identifies regions containing key diagnostic information.\n\n> It is uncertain what the advantages of the proposed method are compared with the latest methods ... through Mamba ...\n\nWe directly compare with both MambaMIL and MamMIL (see Table 3 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md)) and Querent consistently outperforms these methods across all metrics and datasets.\n\nThe fundamental difference between Querent and these methods lies in our query-aware dynamic modeling approach. While MambaMIL uses sequence reordering and MamMIL employs graph-based representations with MST, both still process all patches with predetermined patterns. In contrast, Querent adaptively determines which surrounding regions are most relevant for each patch based on content, focusing computational resources only where needed.\n\n**We welcome any further questions or clarifications regarding our rebuttal and are happy to provide additional details if needed.**", "claims_and_evidence": "Yes, the claims made in the manuscript are largely supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "Yes, the proposed method(s) and/or evaluation criteria (e.g., benchmark datasets) are appropriately justified for the current problem or application.", "theoretical_claims": "Yes, I reviewed the principles of the method proposed in the paper, primarily focusing on Query-Aware Attention Approximation. I examined the detailed derivation provided in the paper, and after referencing the cited literature (Kaban et al., 2015), I found the methodological derivation to be reasonable.", "experimental_designs_or_analyses": "Yes, I have checked the validity of experimental designs. The experimental design and analysis conducted for the proposed method in the article are methodologically sound and empirically valid.", "relation_to_broader_scientific_literature": "The paper introduces query-aware sparse attention, which dynamically selects relevant regions for each query patch, maintaining the expressive power of full self-attention while achieving near-linear computational complexity. This aligns with findings from prior studies (e.g., HIPT, Chen et al., 2022) that proposed various attention mechanisms and region selection strategies to improve computational efficiency and model performance in WSI analysis.", "essential_references_not_discussed": "To the best of my knowledge, I think the authors have already provided sufficient explanation and discussion.", "other_strengths_and_weaknesses": "Strengths:\n1.The paper introduces a novel query-aware attention mechanism that dynamically adapts to the context of each patch, addressing the computational bottleneck of standard transformers in large-scale WSI analysis.\n2.The paper provides theoretical guarantees for the query-aware attention mechanism, proving its error bounds in approximating full self-attention.\n\nWeaknesses:\nThe performance of the model depends on the quality of the region-level metadata. In the computation of region-level metadata, using min/max/mean/mean-std feature to summarize the patch features within a region may lead to the loss of important local information, especially in regions with high tissue heterogeneity or significant noise.", "other_comments_or_suggestions": "Some minor issues:\n1.The experiments did not leverage features from latest foundation models (e.g., UNI and CONCH). Incorporating these advanced features could potentially reduce the performance disparity between existing MIL methods with Querent.\n2.The region size in Querent significantly impacts performance, potentially limiting its generalizability across diverse WSI datasets. When applied to new datasets, it may require careful tuning, posing challenges for real-world applications.", "questions_for_authors": "1.The method can visualize the original WSI corresponding to the K regions selected by the Querent and their min/max feature to prove the accuracy of the method. The author could provide some visualization results to demonstrate the effectiveness of the proposed method. For example, visualizing the metadata feature score for each region could help illustrate that the model indeed selects influential patches.\n2.It is not clear how the method can avoid the situation where the top K regions miss the regions containing key information.\n3.It is uncertain what the advantages of the proposed method are compared with the latest methods that can also achieve efficient and fast classification through Mamba (such as MambaMIL, MamMIL) in existing research. MamMIL can also perceive the topological structures among the instances and incorporate short-range feature interactions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dLkybDfFlQ", "reviewer": "Reviewer_E5SM", "summary": "This paper introduces Querent, a framework for dynamic long-range contextual modeling of gigapixel WSIs through the adaptive determination of patch relationships. The key idea is to maintain the modeling power of full self-attention while achieving computational efficiency through dynamic sparsification. The method adaptively predicts which surrounding regions are most relevant for each patch, enabling focused yet unrestricted attention computation only with potentially important contexts. By using efficient region-wise metadata computation and importance estimation, their approach dramatically reduces computational overhead while preserving global perception to model fine-grained patch correlations. The effectiveness of the proposed method is validated on benchmark datasets, showing improvements over existing approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for these constructive comments. We answer the reviewer's questions in the following one by one:\n\n`Experimental Designs or Analyses`\n> However, the comparative methods delineated in the paper appear to diverge from the tasks ...\n\nWe appreciate the reviewer's concern regarding task alignment between our evaluation and original studies. We want to clarify that our evaluation framework was intentionally designed to be comprehensive, spanning multiple computational pathology tasks (biomarker prediction, gene mutation prediction, cancer subtyping, and survival prediction) to demonstrate the robustness and generalizability of our method across diverse clinical applications. While some baseline methods like RRT-MIL were originally evaluated on specific tasks, we adapted all methods for multiple tasks using standardized feature extraction and training protocols to ensure fair comparison. This approach provides stronger evidence of our method's versatility than limiting evaluation to a single task type would, as demonstrated by Querent's consistent performance advantages across all tasks in Tables 1 and 2. We believe our comprehensive evaluation strategy enhances rather than diminishes the persuasiveness of our experimental findings by showing our approach's effectiveness across the spectrum of computational pathology applications.\n\n`Weakness`\n> however, the analysis appears somewhat deficient. For instance, the absence of results derived from global attention ...\n\nWe appreciate the reviewer's concern about determining the source of our method's superior performance. It's worth noting that directly applying global self-attention to tens of thousands of WSI patches leads to out-of-memory problems, which explains why existing methods use alternatives like local-global or linear attention approximations. To respond to the reviewer's comment directly, we implemented a full global attention approach using FlashAttention (which achieves linear memory complexity) and compared it with our method with the same PLIP feature extractor and same training protocol (see Table 1 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md)). The experimental results show that Querent consistently outperforms the global attention approach (FlashMIL). This confirms that our performance improvements stem specifically from the proposed query-aware dynamic attention mechanism rather than other components, as these elements were identical across both compared methods.\n\n`Other Comments or suggestions`\n> Some sections, particularly the theoretical derivations, could be elaborated ...\n\nIn response to this comment, we acknowledge that some elements of Theorem 3.1 would benefit from additional clarification. Specifically, the parameter B in Theorem 3.1 represents the bound on input norms ($||q_i||$, $||K_j||$ $\\leq$ $B$), which is critical for establishing the error bounds of our query-aware attention approximation. This parameter is properly defined in Lemma B.4 of the appendix but should have been explicitly introduced in the main text for clarity. We will ensure this and other theoretical elements are more thoroughly explained in the revised version to enhance readability and comprehension of our technical contributions.\n\n> Exploring the integration of the proposed method with other advanced models ...\n\nWe agree with the reviewer that integrating our query-aware dynamic attention mechanism with advanced vision transformer architectures represents a promising direction for future work. Our current implementation focuses on efficient modeling of long-range dependencies in gigapixel images, but the core principles of our approach — dynamic region-level metadata summarization and importance-based selective attention — could be readily adapted to enhance various vision transformer frameworks. We appreciate this valuable suggestion and plan to explore such integrations in our future research.\n\n`Questions`\n> Should a comprehensive global attention mechanism be employed ...\n\nWe have responded to this comment with detailed interpretation in the `Weakness` section.\n\n> Would the adoption of alternative patch feature extractors—such ...\n\nWe appreciate this insight. As demonstrated in Table 2 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md), we conducted additional experiments using state-of-the-art foundation models (Virchow and CHIEF). Results show that Querent consistently outperforms other methods with these advanced feature extractors, confirming that our method's superiority stems from its long contextual modeling capability rather than the choice of feature extractor. This indicates Querent's contribution is complementary to advances in foundation models and will continue to provide advantages as foundation models evolve.\n\n**We welcome any further questions or clarifications regarding our rebuttal and are happy to provide additional details if needed.**", "claims_and_evidence": "The authors claim that their method outperforms existing techniques in both efficiency and accuracy when analyzing WSIs. The experimental results presented partially support these claims, showing improvements in key metrics.", "methods_and_evaluation_criteria": "The proposed query-aware attention mechanism dynamically adapts to the unique context of each patch in gigapixel WSIs, preserving global attention while substantially reducing computational complexity. This results in enhanced computational efficiency, making it suitable for the intended application.", "theoretical_claims": "The paper includes theoretical justifications for the proposed approach, particularly in the modeling techniques used. The proofs and derivations appear sound.", "experimental_designs_or_analyses": "The experimental design and analyses are sound, offering lucid delineations of datasets, metrics, and methodologies. However, the comparative methods delineated in the paper appear to diverge from the tasks reported in the original studies. For instance, the performance of the RRT-MIL method, as reported on the TCGA-BRCA dataset, pertains to a sub-typing task, whereas this study utilizes the BRCA dataset for survival prediction. A comparative evaluation of the sub-typing task’s performance could enhance the persuasiveness of the experimental findings.", "relation_to_broader_scientific_literature": "The paper builds upon existing work in sequence modeling, introducing novel adaptations for WSIs. It contributes to the literature by addressing specific challenges associated with WSI and proposing a method that integrates context-aware mechanisms.", "essential_references_not_discussed": "The paper covers relevant literature.", "other_strengths_and_weaknesses": "Strengths:\n- The integration of query-aware mechanisms with dynamic sequence modeling in WSI analysis.\n- The paper is clearly written and well-structured.\n- The proposed methodology has yielded commendable performance across a diverse array of tasks and datasets.\n\nWeaknesses:\n- The primary contribution of the paper lies in its ability to reduce computational complexity while preserving global attention. As evidenced in Tables 1 and 2, the proposed methods outperform existing approaches; however, the analysis appears somewhat deficient. For instance, the absence of results derived from global attention computations raises questions about whether the superior performance of the proposed method stems predominantly from the MIL paradigm, the extraction of patch features via PLIP, or the novel Dynamic Attention mechanism introduced in the text.", "other_comments_or_suggestions": "- Some sections, particularly the theoretical derivations, could be elaborated for better clarity. For instance, the meaning of B in Theorem 3.1 should be promptly elucidated.\n- Exploring the integration of the proposed method with other advanced models, such as vision transformers, could be a valuable direction.", "questions_for_authors": "- Should a comprehensive global attention mechanism be employed instead of this approximate variant, what impact might it have on performance? \n- Would the adoption of alternative patch feature extractors—such as CHIEF, UNI/UNI2, Virchow/Virchow2, PRISM, or GigaPath—yield analogous conclusions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "8uUevJNqRt", "reviewer": "Reviewer_qU9c", "summary": "The paper introduces Querent, a query-aware long contextual modeling framework for whole slide image (WSI) analysis, addressing the challenge of computational efficiency in gigapixel images. Unlike standard transformer architectures with quadratic complexity, Querent dynamically selects relevant regions for each patch using region-wise metadata summarization and importance estimation. This enables efficient self-attention while preserving long-range dependencies. The method outperforms existing approaches in biomarker prediction, gene mutation prediction, cancer subtyping, and survival analysis across multiple WSI datasets. Empirical results show that Querent achieves state-of-the-art accuracy while significantly reducing computational costs.\n\n## update after rebuttal\nI think the authors well address my concerns, so I will raise my score.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable suggestions. We answer the reviewer's questions in the following one by one:\n\n`Weakness`\n> The assumptions in the theoretical proofs may be challenging to satisfy ...\n\nWe appreciate the reviewer's concerns. Regarding the Lipschitz continuity of $f_{min}$ and $f_{max}$: while strict $L$-Lipschitz continuity is a common theoretical assumption used to bound approximation errors, in practice we implement these functions as single-layer perceptrons with ReLU activations. Moreover, we apply regularization techniques such as normalization and weight clipping during training to encourage Lipschitz-like behavior, which empirically ensures that the projections behave in a sufficiently smooth manner for our theoretical guarantees to hold approximately.\n\nAs for the four conditions in Theorem B.6, they are idealized guidelines to understand the behavior of our query-aware selective attention mechanism. In our implementation, we approximate these conditions by: (1) using a large hidden dimension (e.g., $d=512$) to meet JL lemma requirements; (2) selecting an appropriate number of regions based on spatial decay analysis; and (3) designing region sizes  (e.g., $K=16,24$) that balance the need for a small diameter with computational efficiency. Although perfect adherence is challenging, our parameter choices, guided by validated performance, effectively control the approximation error and preserve the key theoretical properties, as supported by our empirical results.\n\n> Could you provide a comparison of training and inference times ...\n\nWe appreciate this question on computational efficiency. In our experiments (see Figure 2 in the [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md)), Querent trains in 72.33s per batch — faster than more complex models like HIPT (473.49s) yet slightly slower than simpler baselines. It converges in a similar number of epochs as other transformer-based methods while achieving state-of-the-art accuracy. Notably, Querent has the lowest memory footprint (2286 MB) among the compared methods. For inference, Querent processes a slide in 0.1328s. \n\nAs the reviewer correctly points out, patch feature extraction (typically 2~3mins per slide) dominates the WSI processing pipeline, while feature aggregation accounts for less than 1% of the total time. We believe that the slight increase in computation time compared to simpler methods is a worthwhile trade-off given Querent's state-of-the-art performance across all tasks.\n\n\n`Other Comments or Suggestions`\n> It appears that there is an error in Equation 12 ...\n\nWe thank the reviewer for the careful review of Supplementary Materials. We will correct this equation in the revised manuscript to ensure mathematical accuracy. \n\n`Questions`\n> In the second set of ablation experiments, why does the Estimation Side Network ...\n\nThis counter-intuitive result stems from fundamental limitations in the Estimation Side Network approach. By attempting to predict region importance independently without considering query context, this network struggles with optimization challenges and fails to capture the relational information critical for accurate importance assessment. It also tends to overfit to region patterns seen during training. Random Region Selection, while simple, provides diverse contextual sampling that occasionally includes relevant regions by chance, avoiding biased selection. Our query-aware approach resolves these issues by dynamically assessing region importance relative to each specific query, leading to significantly better performance than both alternatives. We will clarify this explanation in the revised manuscript.\n\n> In the final step of Lemma B.4, ...\n\nWe thank the reviewer for the insightful comment regarding the feature space transition in Lemma B.4. To clarify, although $q$ and $\\hat{q}$ reside in different feature spaces, the projection functions are assumed to be Lipschitz continuous, which allows us to control the distortion when moving from the original space to the projected space, with detailed deviation shown in Figure 3 in this [anonymous link](https://anonymous.4open.science/r/ICML_PaperID10-A4EE/README.md).\n\n> In the first set of ablation experiments and in Appendix G.1 ...\n\nIn our distance matrix calculations, pre-summarization distances between regions are computed by first flattening all patches in each region into a single vector, then calculating Euclidean distances between these region vectors. Post-summarization distances are simply the Euclidean distances between the metadata vectors. \n\n> Is the $\\alpha$ in Theorem 3.1 the same as ...\n\nNo, these are different quantities. In Theorem 3.1, $\\alpha$ is the exponential decay rate parameter for attention scores with spatial distance. In Equation 8, $\\alpha$ represents normalized attention weights for feature aggregation. We'll use distinct notation in our revision to prevent confusion.", "claims_and_evidence": "The authors have demonstrated, through theoretical analysis and experimental validation, that their proposed query-aware attention mechanism possesses expressiveness comparable to that of full self-attention, while achieving greater computational efficiency. Moreover, the effectiveness of the region-level metadata summarization and importance estimation modules introduced by the authors has also been empirically substantiated.", "methods_and_evaluation_criteria": "I believe that the methods and evaluation criteria proposed by the authors are well-aligned with the problem at hand.", "theoretical_claims": "Upon reviewing the authors' theoretical proofs, I have some reservations and cannot guarantee their complete accuracy.", "experimental_designs_or_analyses": "The experimental design and analysis conducted by the authors are reasonably sound.", "relation_to_broader_scientific_literature": "The paper builds on MIL and transformer-based WSI analysis, addressing efficiency challenges seen in TransMIL and HIPT. While prior work uses linear approximations (Shao et al., 2021) or local-global attention (Chen et al., 2022), Querent introduces query-aware attention, dynamically selecting relevant regions, inspired by context-dependent tissue relationships (Heindl et al., 2015). This improves efficiency while maintaining long-range modeling, advancing adaptive sparse attention in pathology AI.", "essential_references_not_discussed": "No, there aren’t.", "other_strengths_and_weaknesses": "### Strengths:\n\n1. The paper introduces query-aware attention, a novel approach to dynamically selecting relevant regions in WSIs, improving upon rigid local-global attention and linear approximations.\n2. By significantly reducing computational costs while maintaining long-range dependencies, Querent advances scalable WSI analysis, impacting biomarker prediction, cancer subtyping, and survival analysis.\n3. Strong performance across 11 datasets and multiple CPath tasks demonstrates robustness, outperforming state-of-the-art MIL and transformer-based models.\n\n### Weaknesses：\n\n1. The assumptions in the theoretical proofs may be challenging to satisfy in practical code implementation, which undermines their persuasiveness. For instance, the authors did not specify how to ensure that the neural networks $f_{min}$ and $f_{max}$ satisfy the L-Lipschitz continuity. Additionally, the fulfillment of the four conditions in Theorem B.6 during actual network training was not addressed.\n2. Could you provide a comparison of training and inference times between this method and other networks? How does the training convergence speed of this method fare? In practical inference for WSI, the most time-consuming part is likely the extraction of patch features, with aggregation taking up a relatively small portion of the time. How much does this method improve the overall inference time compared to other methods during inference?", "other_comments_or_suggestions": "It appears that there is an error in Equation 12 (NLL survival loss) in the appendix. The second and third terms on the right side of the equation should be preceded by a minus sign rather than a plus sign. Additionally, the second term should be $y^{(i)}_{j-1}$ instead of $y^{(i)}_j-1$.", "questions_for_authors": "1. In the second set of ablation experiments, why does the Estimation Side Network perform worse than the Random Region Selection, which serves as the lower bound?\n2. In the final step of Lemma B.4, since $q$ and $\\hat{q}$ are not in the same feature space, how can they be combined to yield the final result? Could you provide a detailed derivation?\n3. In the first set of ablation experiments and in Appendix G.1, it is mentioned that distance matrices are calculated. After summarization, each region has a feature vector, but before summarization, since a region contains multiple patches, it has multiple feature vectors. How, then, are the distance matrices calculated in this case? Could you elaborate?\n4. Is the $\\alpha$ in Theorem 3.1 the same as the $\\alpha$ in Equation 8？", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes to improve whole slide prediction though a dynamical attention mechanism. The strength of the method is tested on various downstream tasks. All reviewers are positive about the paper and are satisfied with the rebuttal.
Decision: Accept (poster)