Exploiting Low-Dimensional Manifold of Features for Few-Shot Whole Slide Image Classification
Authors: Conghao Xiong
Zhengrui Guo
Zhe Xu
Yifei Zhang
Raymond Kai-yu Tong
Si Yong Yeo
Hao Chen
Joseph JY Sung
Irwin King
Venue: ICLR 2026 Poster (2026)
Keywords: Computational Pathology
Whole Slide Image Classification
Few-shot Learning
Manifold Hypothesis
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Few-shot Whole Slide Image (WSI) classification is severely hampered by overfitting. We argue that this is not merely a data-scarcity issue but a fundamentally geometric problem. Grounded in the manifold hypothesis, our analysis shows that features from pathology foundation models exhibit a low-dimensional manifold geometry that is easily perturbed by downstream models. This insight reveals a key potential issue in downstream multiple instance learning models: linear layers are geometry-agnostic and, as we show empirically, can distort the manifold geometry of the features. To address this, we propose the Manifold Residual (MR) block, a plug-and-play module that is explicitly geometry-aware. The MR block reframes the linear layer as residual learning and decouples it into two pathways: (1) a fixed, random matrix serving as a geometric anchor that approximately preserves topology while also acting as a spectral shaper to sharpen the feature spectrum; and (2) a trainable, low-rank residual pathway that acts as a residual learner for task-specific adaptation, with its structural bottleneck explicitly mirroring the low effective rank of the features. This decoupling imposes a structured inductive bias and reduces learning to a simpler residual fitting task. Through extensive experiments, we demonstrate that our approach achieves state-of-the-art results with significantly fewer parameters, offering a new paradigm for few-shot WSI classification. Code is available in https://github.com/BearCleverProud/MR-Block.

Published as a conference paper at ICLR 2026
EXPLOITING LOw-DIMENSIONAL MANIFOLD OF FEA- TURES FOR FEW-SHOT WHOLE SLIDE IMAGE CLASSI-
FICATION
Conghao Xiong!.2 1.2 Zhengrui Guo Zhe Xu Yifei Zhang'
Raymond Kai-yu Tong*
Si Yong 2e02.5,6 Hao Chen Joseph J. Y. Sung2. Irwin King'
'Te Chinese University of Hong Kong Centre of AI in Medicine, Singapore
The Hong Kong University of Science and Technology "Nanyang Technological University Lee Kong Chian School of Medicine, Nanyang Technological University "ieAiisAl Lab, Singapore
Email: (chxiong21, king) csse. cuhk. edu, hk
ABSTRACT
Few-shot Whole Slide Image (WSI) classification is severely hampered by over- fitting. We argue that this is not merely data-scarcity issue but fundamentally geometric problem. Grounded in the manifold hypothesis, our analysis shows that features from pathology foundation models exhibit low-dimensional manifold geometry that is easily perturbed by downstream models. This insight reveals a key potential issue in downstream multiple instance learning models: linear layers are crometry-agnostie and, as we show empirically, can distort the manifold geometry of the features. To address this, we propose the Manifold Residual (MR) block. a plug-and module that is explicitly geometry-awane the linear layer as residual learning and decouples it into two pathways: (1) fixed, random matrix serving as geometric anchor that approximately preserves topol- ogy while also acting as spectral shaper to sharpen the feature spectrum; and (2) a trainable, low-rank residual pathway that acts residual learner for task-specific adaptation, with its structural bottleneck explicitly mirroring the low effective rank of the features. This decoupling imposes structured inductive bias and reduces learning to a simpler residual fitting task. Through extensive experiments, we demonstrate that our approach achieves state-of-the-art results with significantly fewer parameters, offering a new paradigm for few-shot WSI classification. Code Block
The MR block reframes
is available in https: /tl:://iite /githbit./uu/// www//earrleeverrrruuueee
INTRODUCTION
Histopathology is the gold standard for disease diagnosis (Chen al. 2024 Guo et al. 2024 Xiong ct al. 2024a b). and its computational analysis via Whole Slide Images (WSIs) operates under a uniquely challenging paradigm defined by two constraints. First, the gigapixel nature of WSIs makes Multiple Instance Learning (MIL) the de facto setting, where each slide is represented as a bag of patch features (Ilse al., 2018 Li al., [2021] Zhang al., 2022). Second, expert-intensive annotation limits supervision, yielding datasets that are both small (few-shot) and weakly labeled, typically with only slide-level labels (Lu al. 202T). This few-shot, weakly supervised setting forces models to identify discriminative patterns from thousands of unlabeled instances using only handful of slide-level labels (Xiong al. 2023 Guo et al.) 20256), often leading to severe overfitting.
To understand the root cause of overfitting, we look beyond the learning algorithm and into the intrinsic structure of the features themselves. Grounded in the manifold hypothesis (Tenenbaum et al. 2000), we investigate the intrinsic geometry of Camelyon16 (Litjens et al., (018) under different feature extractors, including CONCH (Lu et al. 2024), UNI (Chen et al. 2022)) , and ResNet- 50 (He et al. 2016) (UNI and ResNet-50 results are reported in Sec. B. and present multi-faceted evidence that these representations lie on a low-dimensional. nonlinear manifold. First, spectral analysis (Skean et al. 2025) reveals low effective rank of 29.7 (vs. the 112-dimensional CONCH features), confirming low dimensionality (Fig. [a). Second, t-SNE visualizations reveal a distinct cluster topology (Fig. [b)). Finally, and most critically, tangent space analysis shows non-flat,

---

Published as a conference paper at ICLR 2026
Eipenvolue Spectrum: CONCH
Original Features: CONCH
Random Uniform Transfurme: CONCH
Tamgent Space Analysis: CD NCH
Oriiiall 35, 8.011 FRoIT
Tongon algrmon)) 5
Oowaeoden regler
Reider Undde Coparl
MOKE
C
8
Normal
Detre: 43%
4.5g
Eienoluee Index
1:SKE
1:
Graph tog.ditance
Figure 1: Feature geometric analysis reveals a low-dimensional manifold, which linear layers distort and our method preserves. (a) Spectral analysis confirms now-dimensionality. (b, c) t-SNE shows cluster topology, and random transformation preserves it. (d) Tangent space analysis visualizes this preservation: our MR block maintains the manifold curvature, while linear layer causes distortion.
distance-dependent growth of geometric drift (Fig. dd), Original), providing quantitative evidence of intrinsic manifold curvature, thus excluding the pure linear subspace hypothesis.
Therefore, we posit that one important contributing factor overfitting is geometric: while features from pathology foundation models exhibit low-dimensional manifold geometry, existing MIL models often fail to preserve this fragile structure. This is not incidental but a limitation rooted in their
ubiquitous building block: the linear layer. Across MIL models, e.g. ABMIL (Ilse et al., 2018))
Lu et al. 2021; Li et al. (02T), Transformers (Shao et al., 2021 Xiong et al., 2023] Tang et al. 2024), graph-based models (Chen et al., 2021] Li et al. (2024b), information-bottleeeee
(Zhang et al.
Huang et al. 2024) and density-basee (Zhu et al. (2024) approaches, the linear layer serves as the indispensable component. However, the linear layer is inherently geometry-agnostie lacking an architectural bias to respect the manifold structure. Our tangent space analysis provides direct, empirical evidence of this destructive tendency: as shown in Fig. [dd), trained linear layer (Vanilla Linear in the figure) severely distorts the intrinsic geometry of the manifold compared to the original (blue line). Consequently, especially under the few-shot setting, these linear layers are prone to learning an overly complex mapping that does not respect the low-rank nature of features and distorts their structure, thereby discarding the geometric priors that the pathology foundation models learned during large-scale pretraining. This points to a fundamental research gap: lack of mechanisms designed to actively preserve and leverage this crucial, yet fragile, manifold structure.
To bridge this research gap, we propose a plug-and-play Manifold Residual (MR) block. The MR block aims to better preserve the geometric structure of the pretrained features from the Pathology Foundation Models (PFMs). It reframes the feature mapping residual learning and decouples it into two parallel paths: a frozen base path for geometric preservation and a trainable Low-rank Residual Path (LRP) for task-specific adaptation. The base path employs fixed random matrix with Kaiming uniform initialization that serves as both a geometric anchor and a spectral sharpener. Theoretically, random projection theory shows that such a transformation provides a preenetty-preervingg embed- ding that, with high probability, approximately preserves pairwise Fuclidean distances (Johnson et al. 1984 Dasgupta& & Gupta (003) and local geodesic structure (Baraniuk & Wakin 2009), and acts as a near-isometry on tangent spaces under standard smoothness assumptions (Papaspiliopoulos 2020). Empirically, Fig. [ shows that the geometric anchor preserves the cluster topology (c), retains the original distance-drin profile in tangent space (d), and sharpens the eigenvalue spectrum (a). The LRP employs a bottleneck structure to model the inherently low-rank task task- -peeific residuals, explicitly mirroring the low-rank features. Although LRP alone substantially distorts the geometry (d). LRP), this behavior is expected because geometric anchor counterbalances this effect (Fig. dd). Random Uniform). Together, the full MR block (Fig. dd), MR Block) preserves the manifold geometry while capturing task-specific signals. The final representation is the sum of the two paths, balancing structural stability with adaptivity.
designed learn task-specific residuals. The
The contribution of our paper can be summarized as follows:
1. We provide quantitative evidence that foundation- model features lie on a low-dimensional manifold and identify a common failure mode: linear layers can disrupt this fragile structure.
2. We propose the theoretically grounded bug--and-play Manifold Residual block that mitigates manifold degradation by combining a geometric anchor with low-rank residual.
2

---

Published as a conference paper at ICLR 2026
3. We conduct extensive experiments, including comparative experiments, ablation studies, and sensitivity analyses, to validate both the efficacy and efficiency our method.
RELATED WORK
2.1 MULTIPLE INSTANCE LEARNING
In MII, WSIs are tessellated into patches, which are then input into a pretrained neural network to obtain features. In terms of how slide-level predictions are computed from patch features, MIL can be categorized into two streams: instance-level and embedding-leve approaches, In the instance- level paradigm, logits are first computed for each patch and then aggregated to yield slide-level prediction (Campanella et al., 2019; Chikontwe et al. [2020] Hou et al., 2016 Kanavati et al. 2020 approaches first aggregate patch features via pooling
Pan et al.] 2025). Conversely, embedding
or aggregator networks to form a slide-level embedding, which is subsequently used to generate the
final prediction (Guo et a 2025a Huang et al. 2024; llse et al. 2018: Lin et al.) [023 Li et al.
2024b
Lu et al.
2021 Shao et al.) 2021: Xiong et al. [023]] //hang et al. 2022 Yu et al. 2025
2.2 PATHOLOGY FOUNDATION MODELS AND FEW-SHOT WSI CLASSIFICATION
ResNet (He et al. 2016). However, the domain discrepancy between natural images and WSIs often limits their efficacy in the pathology domain. To address this, both vision and vision-language pathology foundation models have been developed (Xiong et al.] (025). For example, Virchow (Vorontsov al.) 2024). UNI (Chen al.) 2024). and GPFM (Ma et al. 2024) are pretrained on pathology image datasets, and VLMs, including PLIP (Huang al. 2023), CONCH (Lu al. 2024), mSTAR (Xu al. (024) and MUSK (Xiang et al. 2025), leverage image-text pairs to enhance multimodal tasks. These pathology foundation models enable few-shot weakly-supervised learning (FSWSL) in WSI classification, which is espe-
Previously, feature extraction relied on ImageNet-pretree
cially appealing due to the substantial costs and labor incurred by WSI annotations. FSWSL was
introduced in TOP (Qu et al. 2023), and subsequent studies (Fu et al. 2024 Li al. 2024a Qu
et al. 2024
Shi et al., 2024 Guo et al. 2025b) have further expanded on this paradigm.
Beyond -ppecif FSWSL, generic few-shot and metric-based paradigms (Song et al. 2023 can also operate on bag-level embeddings. Prototypical Networks (Snell et al.) 2017), for example, replace the final linear classifier with a distance-based prototype head. In this work, we keep the standard linear readout used in prior WSI MIL studies and do not apply MR to this small classifier head. Instead, MR is inserted into internal linear layers of the MIL backbone, where our tangent space analysis reveals substantial distortion of the fow-dimensional manifold of pretrained pathology features. Prototype-base based heads endow the embedding space with a simple Euclidean metric but do not explicitly control this internal tangent drift. MR is in principle orthogonal to either linear prototype-based readouts and can be used together with these two techniques.
2.3 GEOMETRIC DEEP LEARNING AND REPRESENTATION ANALYSIS
The manifold hypothesis, central geometric deep learning, posits that high-dimensional data often lie on a now-dimensional manifold (Tenenbaum et al. 2000; Fefferman et al. 2016 Brahma et al. 2016). The geometric properties this learned manifold are shaped by the pretraining objectives, for instance, contrastive methods like CLIP (Radford et al.) 2021) encourage topology of well-separate semantic clusters (Tian et al. 202T); self-distillation approaches like DINO (Oquab et al. 2024) preserve fine-grained local structures; while supervised learning often prioritizes linear separability. These manifolds can be quantitatively characterized using spectral analysis of the Gram matrix (Skean et al.) 2025), which yields metrics like the effective rank (Roy & Vetterli 2007 Vershynin 2009). To further probe for nonlinearity, tangent space analysis directly measures the curvature of the manifold, with significant non-zero curvature being the signature of nonlinear structure (Papaspiliopoulos/ 2020). Our work leverages these geometric tools to conduct a rigorous geometric analysis of the pathology features, revealing the properties our MR block is designed preserve.
3

---

Published as a conference paper at ICLR 2026
Output
MIL Backbone
IJnpaxx
MR
W; c. E.s2
Lineat
A R Rd
B Rd
Activation
gulerr
Linear
Cancer Normal
Linear
Geometric Anchor
LRP
+
Linear Classifier
Input
WSI
Feature
MIL Model
Prediction
Figure 2: Illustration of the linear transformation (a), our proposed MR block (b), and the overall pipeline of MIL for WSI classification task and where our MR block takes effect (c). Specifically, the MR block is applied only to some of the internal linear layers within the MIL backbone, while the final linear classification head (the classifier) is left unchanged.
3 METHODOLOGY
We first formalize the bag-level MIL setting and introduce the geometric tools used to probe the feature manifold of the pathology foundation models, and then detail the Manifold Residual block, mitigating manifold degradation while enabling parameter-efficien task-specific adaptation. Finally, we present empirical evidence and theoretical support for our MR block.
3.1 PRELIMINARIES
3.1.1 BAG-LEVEL MULTIPLE INSTANCE LEARNING FORMULATION
In this work, we focus on bag-level MIL. In MIL, we first partition the WSI into N non-overlapping patches. A pretrained feature extractor then encodes each patch into feature vector f. 17 where dp is the output dimension. The set of these patch features, known as the bag F = {f:) aggregated by an MIL model to produce a single bag feature f = MIL(F) E R where dy is the dimension of the bag feature. Finally, this bag feature is fed into classifier to obtain the slide-level logits CLS(F) E RCC with C being the number of classes. Notably, most modern MIL. models employ attention pooling to compute f. which simultaneously yields both the bag feature and interpretable importance scores for cach patch. This framework is illustrated in Fig. 2 (right).
3.1.2 SPECTRAL ANALYSIS OF FEATURE REPRESENTATIONS
Spectral analysis is a classic technique used to estimate the intrinsic dimensionality of the feature representations. The process begins by computing the Gram matrix K F, from the nor- malized feature matrix Fy- An eigendecomposition is then performed on the Gram matrix K to obtain the eigenvalues (XK K whose plotted distribution forms the spectral curve for visual analysis. For quantitative analysis, we normalize these eigenvalues into probability distribution p: (K)) Σ, (K) and use it to compute the Von Neumann entropy S E, Ps log Pt, and finally, the effective rank Rot = exp(S) (Skean et al.
2025) (N >> dp).
3.1.3 TANGENT SPACE ANALYSIS
To probe for nonlinearity beyond the low-dimensionality, we measure the local curvature of manifold via tangent space analysis (Zhang & Zha (004). This technique provides evidence of nonlinear structure (Lim et al. 2024) by quantifying how the local geometry changes across the feature space. We approximate the geodesic paths by constructing d-nnarest-neighhor graph (Tenenbaum et al. 2000) (k 12) over the features F, with cosine similarity and compute the local tangent space T, at each point f. using local PCA (Kambhatla & Leen (997). Given the orthogonal bases of the d.-dimensional tangent spaces V, and Vj. the Frobenius norm [[, the drift between the tangent
spaces of points, T and Ty. can then be expressed as,
D(T. Ty) =1
V d
(1)
4

---

Published as a conference paper at ICLR 2026
We use these metrics qualitative diagnostics rather than direct predictors of performance gains. In practice, a single geometric statistic cannot reliably forecast the exact improvement on each dataset and backbone, because the final performance also depends various other factors, such as architecture design, which layers are replaced, label difficulty, and the number of shots. Therefore, only use these metrics to describe how vanilla linear layers and our MR block transform the pretrained feature manifold, rather than as dataset-level performance predictors.
3.2 MANIFOLD RESIDUAL BLOCK
The design of the MR block an architectural response to the geometric deficiencies of vanilla linear layers. We first detail its architecture, and then its empirical motivation and theoretical guarantees.
3.2.1 ARCHITECTURE
To mitigate the geometric degradation introduced by a vanilla linear layer, we introduce the MR block, a parameter-effiiieet in Fig. [c., the MR block reframes linear map as the sum of two pathways: a fixed geometric anchor for manifold preservation and a trainable LRP for task-specific adaptation.
plug-and substitute that is explicitly geometry-aware. As illustrated
Given the input X E R. N.Nxd xdo with the input feature dimension being do our MR block is given by,
[MR(X) = GELU(X W + BBB
(2)
with output dimension d1. Here GELU(-) is the element-wise GELU activation function (Hendrycks) & Gimpel 2016); the LRP uses W₂ E RRdoxr and Wi E Rrxdd with rank parameter < min(do. dg. di ): the anchor uses a fixed random matrix B Rdoxd, initialized with Kaiming uni- form distribution (He et al. 2015). For initialization. we set W1 0 and draw W2 from Kaiming uniform distribution, so the residual path initially contributes zero and fXR (x) During training. the LRP is activated only if it improves the training objective. Therefore, even though Fig. dd) shows that the tangent space drift of the LRP alone can be significant, the anchor balances this effect and the overall MR mapping better preserves manifold geometry.
XB at step zero.
Beyond preserving geometry, the MR block also reduces the number of trainable parameters. For any < (dod))/(e (/do + d1), the trainable parameter count of LRP, r(do + di), is strictly fewer than that of a vanilla linear layer (dod1), ) theoretically reducing the risk of overfitting (see Sec. [or details).
3.2.2 EMPIRICAL EVIDENCE
First, we show that features are inherently low-dimensional by spectral analysis (Sec. 3.1.2p. The eigenvalue spectrum decays rapidly, yielding an effective rank of 29.7 (& 512). This provides quantitative evidence that variance is concentrated within a compact, low-dimensional structure.
Second, we provide evidence that the representation forms curved manifold rather than a linear subspace. To distinguish a curved manifold from a linear subspace, we probe nonlinearity via tangent space analysis (Sec. 3.1.3) As shown in Fig. dd) geometric drift increases with graph-hop distance. This nonzero curvature indicates a curved, nonlinear manifold geometry that a trained linear layer fails to preserve. This geometric fragility is the central problem the MR block is designed to address.
3.2.3 COMPONENT JUSTIFICATION AND THEORETICAL GUARANTRES
The geometric anchor preserves the manifold structure that the vanilla linear layers distort. The fixed random projection serves as geometric anchor. Random projection theory shows that this transformation approximately preserves most of the essential geometric properties (see Sec. D
F empirically corroborates that the anchor approximately preserves cluster and neighborhood structure (Fig. [c)), maintains the intrinsic curvature pattern in tangent space drift (Fig. d)), and
(c)) sharpens the eigenvalue spectrum, improving spectral concentration and separation (Fig. [a).
The LRP enables task-specific adaptation. Although our LRP is similar to LoRA (Hu al. 2022),
they serve fundamentally different purposes. The LRP structure is motivated by the low effective rank of features extracted by pathology foundation models. Its low-rank bottleneck imposes a structured inductive bias, ensuring that task-specific adaptation remains consistent with the low-rank nature of the manifold, evidenced in Fig. aa). This reduces learning simpler residual-fitiing task.
5

---

Published as a conference paper at ICLR 2026
Table Experimental results comparing our method with baselines are presented under various shot settings. Performance metrics in which our method outperforms the baseline are highlighted in italic blue and the best metrics are in bold red. "#P" is the number of parameters (million).
Camelyon16 Flt
NSCLC F11
RCC FI1
k Methods
NP
AUCt
Acc.1
AUC1
Acc.t
AUC
Acc.1
ViLaMII,
40.7
53.612.4
48 4
62.32.3
82.9s
73.9a.
74.32.6
96.0h.4
81.7s.
85.52.9
FOCUS
86,9
63.316.s
56.418.2
68.7u.a
87.92.
79.42.2
79.52.:
97.72.1
87.22.
89.52.7
ABMIL
0.26
4
49.27.2
56.1s.2
86.5s.1
80.1s.9
80.21.
89.3n.4
74.211.9
76.711.3
MR- MR-ABMIL
0.10
46.718.7
43.810.0
56.6s.2
86.82.6
79.32.s
79.42.1
96.5a.
85.5a.0
87.85.6
TransMIL
2.41
51.910.3
46.77.6
51.0s.2
79,92.9
71.0g.8
71.62.5
96.51.a
83.5g.a
86.61.7
MR MrlnsMil..
2.01
55.211.0
50.9..a
58.6s.6
83.02..
75.42.8
75.52.5
97.01.2
84.0a.a
87.42.5
CATE
1.93
59.712.4
50.0p.s
58.36.1
81,4a.4
72.34.1
72.8a.6
96,91.8
83.92.
86.52.5
MR CR-CATE
0.84
62./16.
48.
62.3.s.5
87.0g.9
79.4a.2
79.49.2
98,21.2
88.72.
90.51.
DGRMIL
4.08
68.02..6
$9.42..
66.01s.4
71.5
60.3g.5
62.4,.7
96.4n.9
85.32.1
87.41.7
MR DGRMIL
3.29
65.4
55.718.4
63.615.2
80.Ja.4
72.5a.9
73.12.
95.82.0
83.2s.8
86.32.7
RRTMIL
2.44
$7.84.4
54.8s.2
57.16.3
65.74.8
60.82.9
61.5g.a
96.01.9
81.62.7
84.72.a
MR-RRTMIL
2.04
46.5
42.75.6
62.96.6
69.9%
63.96.
64.0s.o
96.71 76.
82.5
86. /2.2
ViLaMII.
40.7
74.0
65.922.1
71.81..5
88.46 2
79.2s.
79.2s.7
96.80.9
86.22.1
88.01.7
FOCUS
86.9
87.2v.0
79.29.4
80.20.4
93.7a.8
85.0s.s
85.
18.
86.72.a
88.92.0
ABMIL
0.26
49.2s.8
48.9s.a
59.74.s
87.76.9
81,0%
81.1s.4
91.8a.a
76.6s.4
79.17.9
MR- AR-ABBIL
0.10
56.516.5
53.813.9
61.96.7
94.0a.a
85.9g.3
86.0g.a
08.1o.a
87.02.s
89.62.1
TransMIL
2.41
57.52.5
52.06.a
53.67.4
87.96
80.3s.
80,46.
97.96.6
86.5s.9
88 As.1
MR MransMil.
2.01
65.38.6
54.29.a
63.63.8
91.64.1
82.44.1
82.64.
97.9p.4
86.72.5
8..
CATE
1.93
4..8
69.12..0
76.012.6
87.76.7
78.01.9
78.24.9
97.80.8
86.45.0
88.32.1
MR-CATE
0.84
88.014.8
81.71e.0
82.915.a
93.56.s
84.6..r
84.76.8
98.6n.3
89.1
90.71.0
DGRMIL
4.08
84.710.o
72.51n.8
77.412.6
85.56.a
73.4a.6
74.2
96.71.1
86.82.9
88.72.a
MR-DGGMI
3.29
93.40.6
86.0s.5
86.7a.8
91.67.a
K2.89.9
83.06.4
97.61.0
85.1
87.91.7
RRTMIL
2.44
68.817.8
62.615.4
63.715.2
82.5s.9
73.96.7
74.26.5
97.01.1
85 85.2a.3
86.9s.2
MR-RRTMII.
85.316.3
79.216.6
81.713.3
87.37.8
79.67.a
79.87.8
97.8o.4
86.31.5
87.81.1
ViLaMIL
40.7 81.42a. 1
77.522.0
82.01s.o
92.82.
84.92.r
85.02.7
97.41.1
85.5s.0
87.92.2
FOCUS
86.9
93.22.6
89.0s.0
89.8s.0
96.51.
89.32.7
89.32.7
98.2n.6
89.03.1
90.61.6
ABMIL
0.26
73.2
71.215.4
75.710.2
93.62.4
87.8a.4
87.8a.4
94.76.2
81.0r.s
82.8s.6
MR-ABMILL
0.10
79.02a.1
74.522.4
78.017.1
96.31.
89./1.0
89./1.0
98.30.7
89./2.8
91.12.0
TransMII.
69.715.7
62.51a.2
69.315.3
45.61.7
88.23.6
88.2
98.40.3
89.31.6
90.96.7
MR- TransMIl.
2.01
19.3
75.1
77.1
93.24.4
84.0a.1
84.1a.o
98.6n.3
89.31.
91.21.0
CATE
1.93
92.2g.1
87.5 87.5s.2
88.52
94.22.1
85.12.0
85.22.
97.9p.6
87.42.7
89.32.0
MR-CATE
0.84
92.43.2
86.7s.9
87.8a.4
96.11.4
88.6g.3
88.7a.3
98.6n.s
89.8s.1
91.62.a
DGRMIL
4.08
88.5s.0
79.410.2
80.91o.a
92.4g
85.84.0
85.81.
94,81.4
86.31.0
88.71.2
MR-IIIMMIL
3.29
92.3p.
87.6p.8
NK.75.2
93.91.7
86.6g.
86.72.1
97.70.s
88.02.
RETMIL
2.44
84.016.9
74.812.8
75.512.7
90.7a.2
82.64.2
82.74.1
98.30.6
86.92.
88 88.8
MR. RRTMIL
93.71.8
89.02.4
89.82.3
91.0%.4
82.57.8
82.8;.1
97.8p.s
86.5..1
88. 88.92.9
Approximation behavior of the MR block. Theoretically, our MR block can approximate any linear layer to arbitrary precision, thereby providing worst-case performance guarantee: if the MR block cannot improve over the linear layer then it can match behavior of the target optimal linear layer arbitrarily closely, ensuring loss in worst-case performance. In addition, rather than directly proving our formulation, we establish more fundamental and challenging result without activation functions. Subsequently, with the universal approximation theorem (Barron) 1993 Cybenko 1989 Funahashi 1989 Hornik et al. 1989: Lu & Lu 2020). our architecture with nonlinear activation function can achieve even superior approximation precision. The proof is given Sec.
EXPERIMENTS AND RESULTS
4.1 DATASET DESCRIPTIONS AND IMPLEMENTATION DETAILS
Datasets. We conduct extensive experiments Camelyon16 Litjens et al. 2018), NCGA- and TCGA-RCC datasets. The details of the datasets used in this work are available in Sec. G
Evaluation Metrics. We employ four complementary metries for comprehensive assessment of these data-imbalanced datasets: Area Under the Receiver Operating Characteristic Curve (AUC), Area Under the Precision-Rece Recall Curve (AUPRC), macro F1-score, and Accuracy.
Implementation Details. To ensure a fair and robust comparison, we use a consistent set of hyperparameters and training and evaluation protocols for all methods, as detailed in Sec. G.2 The

---

Published as a conference paper at ICLR 2026
Table Ablation studies of our proposed method across varying shot settings on the Camelyon TCGA-NSCLC, and TCGA-RCC datasets. Metrics highlighted in bold red indicate the top result.
Camelyon16 FI1
NSCLC FI
RCC FIf
&
Methods
#P
AUCt
Acc.1
AUC1
Ace.
AUC
Acc.t
Ours
2.01
71.414.9
65.512.1
69,97,3
95.13.5
86.14.7
86.24.6
96.81.3
84.3,.4
87.04.4
+ B FT
2.54
55.31.1
55.82.7
60.62.3
88.72.1
81.7a.3
81.8g.a
96.41.9
84.33.6
86,43.2
B
2.01
55.30.6
6
58.41.9
88.2s.a
81.0p.9
81.2g.
97.61.1
86.4..u
88.61.4
Bax
2,01
56.32.s
55.82.9
59.12.7
92.72.7
84.7s.4
84.9g.4
98.1o.4
87.4s.7
89.3s.0
-LRP
1.89
51.017.7
49.414:2
58.9s.s
89.46.a
81.6o.s
81.7g.8
98.1o.s
86.92.
89.32.1
Ours
2,01
86.411.7
80.912.0
81.911.8
96.01.7
88.71.4
88.71.4
98.3o.a
87.72.a
90.02.0
B FT
2.54
51.39.4
47.27.a
61.14.5
93.12.4
87.21.a
87.21
97.41.1
85.42.8
87.92.4
B
2,01
61.1s..
52.88.s
63.37.4
91.92.4
84.72.4
84.82.
97.71.3
86.0
88.33.6
Ba 2.01
72.219.8
67.619.9
73.513.0
95.50.7
87.41.0
87.41.0
98.50.4
88.4g.1
90.61.6
LRP
1.89
68.320.2
65.117.8
69.812.8
93.72.0
86.41.1
86.41.
98.3o.6
87.01.
89.61.0
Table 3: sapacity-matchent comparison between ABMIL and MR-ABMIL under various shot settings. MR-ABMIL uses an MR block with the same number of trainable parameters as the vanilla ABMIL.
Camelyon16 F11
NSCLC FI1
RCC F11
Methods
WP
AUC1
Acc.t
AUC1
Acc.t
AUC
Acc.7
ABMIL.
0.26
45.57.6
49.27.2
56.1a.a
86.5s.1
80.15.0
80.24.9
89.36.4
74.2
76.711.a
MR-ABMIL
0.26
56.616.3
54.0
58.413.
78.74.0
70.8s.7
71.13.5
97.71.3
86.92.5
89.32.6
ABMIL
0.26
49.2s.s
48.93.0
59.74.
87.7s.9
81.0..4
81.15.4
91.8g.s
76.6x.4
79.17.9
MR-ABMIL
0.26
73.216.0
70.716.9
72.615.8
88.9s.6
80.0s.1
80.1s.2
98.60.6
88.73.a
90.92.9
ABMIL
0.26
73.217.1
71.215.4
75.710.2
93.62.4
86.92.s
87 87.0g.s
94.7p.a
81.07.s
82.8s.5
AR-IB
0.26
84.311.7
79.311.2
80.311.1
93.61.9
87.83.4
87.8g.4
98.8g.3
89.72.0
91.71.0
Table 4: Performance of MR-ABMIL with replaced linear layers under varying shot settings. "U"
and "V" denote two matrices in MR-ABMII..
Camelyon16 F11
NSCLC FIf
RCC F11
Methods
AUC1
Acc.t
AUC1
Acc.
AUC1
Acc.
V
U
0.10
46. 13.77
43.8
56.63,2
86.82.5
79.32.3
79.42.1
96.5
85.56.0
87.8,.s
4
V
0.18
45.60.a
43.90.2
59.81.4
86.
78.2s.2
78.4a.1
94.04.8
79,07.a
81.77.a
U
0.18
52.810.a
47.96.6
59.42.
85.56.s
77.2p.5
77.6g.
90.4g.4
73.411.
77.410.3
0.10
56.51s.s
53.81a.9
61.9s.7
94.0s.3
85.9s.a
86.0g.a
98.
87.62.9
89.62.1
8
V
0.18
71.414.9
65.5
69.97.3
95.13.5
86.
86.2 8
96.81.3
84.35.4
87.0a.4
U
0.18
72.819.1
67.820.0
71.516.8
94.64.7
85.8
85.96.7
95.1
79.811.7
83.67.9
specific locations where linear layers are replaced by MR blocks are described in Sec. G.4 The FLOP and the wall-clock inference time statistics are available in Sec. G.5
4.2 COMPARISON WITH OFTTE-OF-THE METHODS
Tables 1 and to present few-shot results (typically with k {2, 16), where available)
on multiple datasets, including tumor detection (Camelyon16) tumor subtyping (TCGA-NSCLC, TCGA-RCC), and treatment response (Boehmk, Trastuzumab; ABMIL variants only in Table 9) Three significant findings emerge from this analysis. First, across both artificially constructed few-shot datasets on large cohorts and inherently few-shot treatment response datasets, models augmented with the MR block consistently outperform their corresponding basclines over datasets and shot counts. On Camelyon 16, TCGA-NSCICC. and TCGA-RCC, they match or surpass the state-of-the-an ViLaMII. (Shi et al., 2024) and FOCUS (Guo et al.] 2025b) while using far fewer trainable parameters. Second, our MR block improves parameter efficiency: replacing the vanilla linear layer with our MR block yields smaller models that deliver better performance across multiple MIL backbones, indicating that MR provides a beneficial low-rank inductive bias rather than simply adding capacity. Third, performance increases steadily as & grows for all methods, with MR variants showing the largest gains at moderate shots and remaining competitive at higher shots, where RCC results approach ceiling. We also observe reduced variability across runs for several MR settings,
7

---

Published as a conference paper at ICLR 2026
Table Experimental results on different initialization methods in MR-CATE. "K." and "X." stands for Kaiming and Xavier initialization. "U." and "N." stands for uniform and normal distribution. The number following is the input parameter for initialization. The gray rows stand for our main setting.
Camelyon16 FIf
NSCLC FI
RCC
k Methods
AUC1
Acc.
AUC
Acc.t
AUC
F11
Acc.f
K. N.O O
63.0
19.5
61.7
82.24.2
75.0..7
75.2
98.01.4
87.8,
90.1 1.8
K. U.O
62.417.0
48.916.9
63.7
86.54.4
79.4a.2
79.4s.1
98.11.4
88.23.0
90.32.5
4
K. U. v5
16.1
58.415.8
62.313.5
87.03.9
79,4a.2
79,43.2
98.21.2
88.72.0
90.51.a
X. O
66.214.11
48.820.1
64.712.9
84.0s.a
76.34.2
76.44.2
98.11.4
87.9a.4
90.12.9
X.
61.117.1
51.217.8
63.612.8
86.44.6
79.
12.
98.1
88.22.9
90.22.6
N.
93.81.7
88.94.
89.64.5
95.42.2
86.92.8
87.02.s
98 98.60.4
88 88.22.2
89.62.0
K. U.O
92.22.2
87.51.
88.42.0
94.62.3
86.32.2
86 86.42.1
98 98.6g.4
88 $8.82.3
90.92.0
16
K. U. V5
92.43,2
86.73.9
87.8g.4
96.1
88.63.3
88. 88.73.3
98.6u.
89.83.1
91.62.3
X. O.O
94.51.5
90.11.5
90.71.6
95.62.0
88.51.4
88.51.4
98.7a.5
88.92.
90.91.4
X.O.O
92.73.7
88.53.5
89.13.4
95.81.3
88.73.1
88.8s.1
98.7g.4
89.5g.1
91.22.5
Table 6: Experimental results on different pretrained feature extractors under various shot settings. "R50" stands for ResNet50, and "MR-" stands for MII. model with our MR block.
ABMIL
CATE
Methods
NSCLC
RCC FI?
NSCLC
RCC
Acc.f
AUCT
Acc.f
AUC1
Acc:t
AUCT
Acc.1
R:0
59.0u.
54.7a.1
57.0az
74.6c
36.6..
58.51o.7
64.9h
57.4106
60.0g.
82.61.
64.210.
68 68.9r.
MR-R50
64.3422
59.7a.
60.1ga
80.8
62.36.
63.92
63.5v.4
60.7s.7
61.4a.
89.5gs
75.0v
76. 76.6s.u
UNI
72.86.2
73.16.2
98.60.4
88.72.1
90.31.
82.77.
72.4s.:
73.0s.1
98.1o.s
87.8s.4
89.3g.a
MR-UNI
91.35.
81.7g.4
81.8g.
98.0...
91.21.
92.31
8886.6
8006
80.8..
98.5p.1
90.81
91.8:
R50
62.3,.u
62.5x.
67.7a.
66.3mm.
56.212
61.17.4
90.0s.o
74.7,a
78.0..
MR-R50
67.26.6
63.75.0
63.94a
85.92
70.82.4
73:41.
66.3124
39.215.1
63.0g.2
9V.74.2
81.46.1
82.45.0
16
UNI
91.127
83.6g.
836aa
98.96
89.21.0
91.0
87.27.5
75.8a
76.96
98.0g.
88.0a.
90.3..
MR-UNI
87.0g.
87.0ga
91.12.4
91.74.
82.6gu
82.9.
98.5u.
92.3,.
suggesting improved training stability. Overall, MR is a plug-in replacement for vanilla linear layers in MIL models that improves accuracy, data efficiency, and robustness without increasing model size.
4.3 ABLATION STUDIES AND SENSITIVITY ANALYSIS
Ablation Study of MR Block Components. Our ablation studies in Table validate our decoupled design. Removing the LRP term (f-LRP = B) consistently degrades performance, confirming its necessity for task adaptation. Removing the geometric anchor, either while retaining the residual connection (f-B GELU(X (f-Bx = GELU 2) W1). J. is detrimental in both cases, underscoring its dual role as a geometric anchor and spectral shaper. More importantly, making the anchor trainable leads performance collapse. These observations provide direct empirical support for our core hypothesis: an unconstrained linear layer tends shatter the feature manifold, whereas our design preserves it.
+ X GELU(X W2 + X) while removing it
catastrophic
Capacity-Matee Matched Analysis of MR Block Effects. To disentangle parameter reduction from the geometric inductive bias introduced by our MR block, we construct a capacity- MR-ABMIL in which the MR block has exactly the same number of trainable parameters as the original gated attention layer in ABMIL; the results are reported in Table 3 For gated attention layer with input dimension do 512 and output dimension d1 256, we set the rank to r (dod1)
d1) = 170, so that MR and the vanilla linear layer are exactly matched in parameter count. Across Camelyon16 and RCC, this capacity-matee matched MR-A MRIIBMII. consistently achieves substantial improvements in all metrics over the ABMIL baseline for all shot settings. On NSCLC, performance remains comparable, with slight drop 4-shot and similar better performance at 8- and 16-shot. Since these gains are obtained under exactly matched trainable parameter counts, they provide direct evidence that the geometry aware design of MR as structural inductive bias, rather than parameter reduction alone, plays a central role in improving few-shot performance.
Layer-wise Performance Analysis. We conducted layer- laser-wise ablation study to evaluate how our method affects performance when implemented at different layers within MIL. We selected ABMIL (Ilse al. 2018) due to its structural simplicity, with results reported in Tables 4 and 10
8

---

Published as a conference paper at ICLR 2026
MR-ABMIL Rank Sensitivity: NSCLC
MR-ABMIL Rank Sensitivity: RCC
1.00
0.95
0.98
0.90
0.96
0.85
0.94
0.92
0.75
6-hhot Daseliie: 5.60
d-shot Dhaseline: 0.803)
B shot (asselin:
shott (aaseline:
16-ehat (pa0e
0.90
16-ehot (haneline:
0.70
16
32
48
128
32
48
64
128
Rank Value
Rank Value
MR-CATE Rank Sensitivity; NSCLC
MR-CATE Rank Sensitivity: RCC
1.00
1.00
0.95
0.99
0.98
0.85
0.97
0.96
0.80
Gahat (asseline 0.144)
4shot Baselline (.600)
Daseline:
0-shot Basseline 6370)
0.95
16-shot Braseline
16
32 Rank Value
48
128
48
64
128
Rank Value
Figure 3: Sensitivity analysis of parameter r across multiple datasets using MR-ABMIL and MR- CATE. Dashed lines indicate the performance of the baseline methods (i.e., ABMIL and CATE).
This study investigates two components of ABMIL [lsse et al. 2018), U and V (detailed in Sec. G.4). analyzing performances when selectively replacing these components with the MR block. The results demonstrate that optimal performance is achieved when both components are replaced with our MR block, and our method is robust to positions in MIL.
Choice of Initialization. We investigated the sensitivity of the MR block to the initialization scheme of B, by testing four schemes: Kaiming/Xavier Normal/Unifone The results in Tables 5 and IT demonstrate that our method is broadly insensitive to the specific choice of initialization. Across most settings, the performance differences between these methods marginal, confirming that the benefits the geometric anchor are not sensitive to initialization. Our chosen default, Kaiming Uniform with v5, consistently delivers superior performance, validating it as an effective choice without the need for additional hyper- paremmeerr tuning.
Foundation Model Selection. We investigated whether our approach demonstrates compatibility across different foundation model feature extractors. Using CONCH, UNI (Chen et al. 2024), and ResNet50) (He et al., 2016) with ABMIL and CATE, we present results in Tables 6 and 12 Our method demonstrated consistent improvements over baselines across all model combinations, substan- tiating its robustness across diverse MIL and foundation models. CONCH embeddings consistently outperformed UNI and ResNet50, aligning with expectations as ResNet50 lacks pathology pretraining. The dimensional difference between UNI (1,024) and CONCH (512) likely contributes to lower performance of UNI due inflated nigher-dimensionn spaces under few-shot settings.
Sensitivity Analysis of the Rank. The residual rank is key design hyperparameter that directly controls how much information can flow through the low-rank path. We perform a sensitivity analysis over and the results are shown in Fig- Across these datasets and shot settings, model performance saturates around r 32. On the datasets and backbones evaluated, this empirically observed saturation point aligns closely with the theoretically predicted effective rank of the features, indicating that a low-rank path is sufficient to capture the essential task-specific information residing in the manifold structure. The simpler MR-ABMIL model peaks sharply = 32, while the more expressive MR-CATE exhibits only marginal gains at higher ranks, which we attribute to its additional capacity for modeling finer-grained feature interactions beyond the primary manifold structure.

---

Published as a conference paper at ICLR 2026
Analysis Beyond the Extreme Few-Shot Regime. To assess whether MR continues to provide benefits when data volume increases, we conduct experiments on TCGA-NSCLC and TCGA-RCC with larger shots, & (32, 64), and report the results in Table [3 Across both datasets and both shot settings, MR-ABMIL and MR-CATE consistently outperform their corresponding vanilla baselines on all metrics. These observations align with our interpretation of MR geometry- the relative gains are typically stronger in the extreme few-shot regime, tend to decrease as supervision increases, yet remain positive and do not degrade performance even when more data is available.
regularizer:
4.4 MODEL INTERPRETARIL.TT ANALYSIS UNDER EXTREME LIMITED RESOURCE.
Figure 6 illustrates the heatmaps generated by our MR-CATE, which demonstrates remarkable robustness in the extreme 2-shot setting where the standard model (CATE) often fails to produce meaningful heatmaps. In the original images (upper left), blue curves delineate the approximate tumor boundaries, and the corresponding heatmap is shown the lower left panel. Notably, our method captures additional fine-grained boundaries not presented in the original annotations, as shown in the right panel. Although all regions on the right fall within the blue-delineated tumor boundaries, model precisely identifies the boundaries between distinct morphological patterns, demonstrating its sensitivity in capturing tumor and normal tissue heterogeneity.
CONCLUSION
In this work, we introduced a new geometric perspective on the problem of overfitting in few-shot WSI classification. We provided both quantitative and visual evidence that features from pathology foundation models exhibit a fragile, low-dimensional manifold geometry, and identified common model failure mode of the current MIL models: the systematic degradation of this manifold structure by the geometry-agnoste We proposed the Manifold Residual block, plug-and-play module that preserves this geometry via fixed random geometric anchor while enabling parameter-efficien task-specific adaptation through a low-rank residual path. Our extensive experimental results not only achieve state-of-te state-of-the-art results but also empirically support our geometric diagnosis, thereby establishing a new, geometry aware paradigm for developing robust models with implications beyond computational pathology.
linear layer, which is the indispensable component of modern MIL models.
REPRODUCIBILITY STATEMENT
Code is available in https:: http::://githbb htwp://ww...coeaaaaaeeeeerrooooo Boc.o
In addition, we point to all components needed replicate our results. The MIL setup and geometry tools are in Secs. 3.1 3.1.2 and 3.1.3 The MR architecture is in Sec. 1.2.1 full experimental protocols (datasets, few-shot k, backbones, metrics) and implementation details are in Sec. Ablations and sensitivity studies hompppnew/plccmmmenntttaaalttoao
appear in Tables 2 and 5 and Fig.
and their appendix counterparts. Assumptions and complete proofs are Sees. Dand]
ACKNOWLEDGEMENT
The research presented in this paper partially supported by the Research Grants Council of the Hong Kong Special Administrative Region, China (CUHK 2410072, RGC R1015-23) and (CUHK 2300246, RGC C1043-24G).
REFERENCES
Richard G Baraniuk and Michael B Wakin. Random projections of smooth manifolds. Foundations of computational mathematics, 91):511-77 2009.
Andrew R Barron. Universal approximation bounds for superpositions of sigmoidal function. IEEE Transactions on Information theory, 39(3):930-99)) 945, 1993.
Kevin M Bochm, Emily A Aherne, Lora Ellenson, Ines Nikolovski, Mohammed Alghamdi, Ignacio V Garefa, Dmitriy Zamarin, Kara Long Roche, Ying Liu, Druv Patel, al. Multimodal
10

---

Published as a conference paper at ICLR 2026
data integration using machine learning improves risk stratification of gigh-grade serous ovarian cancer. Nature cancer, (16):723-733, 2022.
Pratik Prabhanjan Brahma, Dapeng Wu, and Yiyuan She. Why deep learning works: A manifold disentanglement perspective. IEEE Transactions Neural Networks and Learming Systems, 27 (10):1997-2008, 2016. doi: 10. 10.1109//T1 2015.2.9.211..49994....
Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus Busam, Edi Brogi, Victor Reuter, David Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature medicine, 25(8):1301-1399 2019.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650 9660, 2021.
Richard Chen, Ming Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware survival prediction using patch-based graph convolutional networks. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 339-349. Springer, 2021.
Richard Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Bowen Chen, Andrew Zhang, Daniel Shao, Andrew H Song, Muhammad Shaban, et al. Towards a general- purpose foundation model for computational pathology. Nature Medicine, 2024.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Philip Chikontwe, Meejeong Kim, Soo Jeong Nam, Heounjeong Go. and Sang Hyun Park. Multiple instance learning with center embeddings for histopathology classification. In Medical Image Computing and Computer Assisted MMMCCAA 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part V 23, pp. 519-528. Springer, 2020.
George Cybenko. Approximation by superpositions of sigmoidal function. Mathematics of control, signals and systems, 204):303-3,,, 1989.
Sanjoy Dasgupta and Anupam Gupta. An elementary proof of a theorem of johnson and lindenstrauss. Random Structures & Algorithms, 22(1):60-655 2003.
Tong Ding, Sophia Wagner, Andrew H Song. Richard J Chen, Ming Y Lu, Andrew Zhang, Anurag J Vaidya, Guillaume Jaume, Muhammad Shaban, Ahrong Kim, et al. A multimodal whole-slide foundation model for pathology. Nature Medicine, pp. 1-13, 2025.
S Farahmand, AI Fernandez, FS Ahmed, DL Rimm, JH Chuang, E Reisenbichler, and K Zarringhalam. Her2 and trastuzumab treatment response h&e slides with tumor roi annotations. The Cancer Imaging Archive), 2022.
Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan. Testing the manifold hypothesis. Journal of the American Mathematical Society, 2944):93-1009, 2016.
Kexue Fu, Linhao Qu, Shuo Wang, Ying Xiong, Ilias Maglogiannis, Longxiang Gao, Manning Wang. et al. Fast: A dual-tier few-shot learning paradigm for whole slide image classification. Advances in Neural Information Processing Systems, 37: 17:100090-1051 13, 2024.
Ken-Ichi Funahashi. On the approximate realization of continuous mappings by neural networks. Neural networks, 193) 1989.
Robert Geirhos, Homm-Herrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 11:66)66:777 07255663,2200 2020.
Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah. Multimodal neurons in artificial neural networks. Distill, 6(3)xe30, 2021.

---

Published as a conference paper at ICLR 2026
Zhengrui Guo, Jiabo Ma, Yingxue Xu, Yihui Wang, Liansheng Wang, and Hao Chen. Histgen: Histopathology report generation via local-global feature encoding and cross-modal context interaction. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 189-199. Springer, 2024.
Zhengrui Guo, Qichen Sun, Jiabo Ma, Lishuang Feng, Jinzhuo Wang, and Hao Chen. Context matters: Query-aware dynamic long sequence modeling of gigapixel images. arXiv preprint arXiv:2501.18984, 2025a.
Zhengrui Guo, Conghao Xiong, Jiabo Ma, Qichen Sun, Lishuang Feng, Jinzhuo Wang, and Hao Chen. Focus: Knowledge-enhand adaptive visual compression for few-shot whole slide image classification. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp.
15590-15600, 2025b..
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. 1oXv::1606.08415.. 2016.
Gaussian error linear units (gelus). arXiv preprint
Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural networks, 2(5):359-366) -3659 1989.
Le Hou, Dimitris Samaras, Tahsin M Kurc, Yi Gao, James E Davis, and Joel H Saltz. Patch-based convolutional neural network for whole slide tissue image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2016.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 1(2):3, 2022.
Yanyan Huang, Weiqin Zhao, Yihang Chen, Yu Fu, and Lequan Yu. Free lunch in pathology foundation model: Tasc-specific model adaptation with concept-guidee feature enhancement, In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual- language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307-2316 2023.
Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pp. 2127-2136. PMLR, 2018
William B Johnson, Joram Lindenstrauss, et al. Extensions of lipschitz mappings into hilbert space. Contemporary mathematics, 26(189-206):1, 1984.
Nandakishore Kambhatla and Todd K. Leen. Dimension reduction by local principal component analysis. Neural Computation, 1433-19911 1997. doi: 10. 10.1162/neco. http:/ecoo.1999..30 1997.9.7.1493.
Fahdi Kanavati, Gouji Toyokawa, Seiya Momosaki, Michael Rambeau, Yuka Kozuma, Fumihiro Shoji, Koji Yamazaki, Sadanori Takeo, Osamu lizuka, and Masayuki Tsuneki. Weakly-supervised learning for lung carcinoma classification using deep learning. Scientific reports, 10(1):9297, 2020.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 14318-14328, 2021.
Hao Li, Ying Chen, Yifei Chen, Rongshan Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. Generalizable whole slide image classification with fine- fine-graine grained visual-semantie inter- action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11 1139--1 11407, 2024a.
12

---

Published as a conference paper at ICLR 2026
Jiawen Li, Yuxuan Chen, Hongbo Chu, Qiehe Sun, Tian Guan, Anjia Han, and Yonghong He. Dynamic graph representation with knowledge-awain aware attention for histopathology whole slide image analysis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17223-11322 2024b.
Victor Weixin Liang, Yuhui Zhang, Yongchan Kwon, Serena Yeung, and James Y Zou. Mind the gap: Understanding the modality gap multi-modal contrastive representation learning. Advances in Neural Information Processing Systems, 35:17612-17625, 2022.
Uzu Lim, Harald Oberhauser, and Vidit Nanda. Tangent space and dimension estimation with the wasserstein distance. SIAM Journal on Applied Algebra and Geometry, 8(3):650-688 2024.
Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang- Thang-Wenn Chen. Interventional bag multi- instance learning whole-slide pathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19830-19839, 2023.
Geert Litjens, Peter Bandi, Babak Ehteshami Bejnordi, Oscar Geessink, Maschenka Balkenhol, Peter Bult, Altuna Halilovic, Meyke Hermsen, Rob van de Loo, Rob Vogels, Quirine F Man- son, Nikolas Stathonikos, Alexi Baidoshvili, Paul van Diest, Carla Wauters, Marcory van Dijk, and Jeroen van der Laak. 1399 H&E-stained sentinel lymph node sections of breast cancer patients: the CAMELYON dataset. GigaScience, 7(6), June 2018. ISSN 2047-217X. doi: 10. 10.0091/jigaacienc//ivg... URL URL https://///csteemii oup cotpg/gaaiigciciccce////
article/doi//in att11///...7....//////aaaaa////2
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, In International Confer- net forum?idd
ence on Learning Representations, 2019. URL
Bky
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570) 2021.
Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard Chen, Ivy Liang, Tong Ding, Guillaume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30:863-874, 2024.
Yulong Lu and Jianfeng Lu. A universal approximation theorem of deep neural networks for expressing probability distributions. Advances in neural information processing systems, 33: 3094-3105, 2020.
Jiabo Ma, Zhengrui Guo, Fengtao Zhou, Yihui Wang, Yingxue Xu, Cai, Zhengjie Zhu, Cheng Jin, Yi Lin, Xinrui Jiang, et al. Towards a generalizable pathology foundation model via unified knowledge distillation. arXiv preprint arXiv:24007. 18449, 2024.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po- Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features with- out supervision. Transactions on Machine Learning Research, 2024. ISSN 2835-8856. URL Featured Certification.
net forum (at.:ummid-aa888000 6zFt
Wentao Pan, Donghuan Lu, Jiangpeng Yan, Zhe Xu, Conghao Xiong, Dong Wei, Xian Wu, and Yixuan Yuan. Improving instance- based whole slide image classification with logit-based log-sum-exy aggregator and contextual awareness. In 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). pp. 3979-3984. IEEE, 2025.
Omiros Papaspiliopoulos. High- digeh-immnsional probability: An introduction with applications in data science, 2020.
Linhao Qu, Kexue Fu, Manning Wang. Zhijian Song, al. The rise of ai language pathologists: Ex- ploring teo-level prompt learning for few-shot weakly-supervised whole slide image classification. Advances in Neural Information Processing Systems, 36:67551-611 -67564, 2023.
13

---

Published as a conference paper at ICLR 2026
Linhao Qu, Dingkang Yang, Dan Huang. Qinhao Guo. Rongkui Luo, Shaoting Zhang, and Xiaosong Wang. Hathology-knowleee enhanced multi-instance prompt learning for few-shot whole slide image classification. In European Conference on Computer Vision, pp. 196-212. Springer. 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 1748-8763. PmLR, 2021.
Maithra Raghu, Chiyuan Zhang, Jon Kleinberg, and Samy Bengio. Transfusion: Understanding transfer learning for medical imaging. Advances in neural information processing systems, 32, 2019.
Olivier Roy and Martin Vetterli. The effective rank: A measure of effective dimensionality. In 2007 15th European signal processing conference, pp. 606-610. IEEE, 2007.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34:2136-2147, 2021.
Jiangoo Shi, Chen Li, Tieliang Gong, Yefeng Zheng, and Huazhu Fu. Vila-mil: Dual-scale vision- language multiple instance learning for whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11248-11258, 2024.
Oscar Skean, Md Rifat Arefin, Dan Zhao, Niket Nikul Patel, Jalal Naghiyev, Yann LeCun, and Ravid Shwartz-Ziv. Layer by layer: Uncovering hidden representations in language models. In Forty- second International Conference on Machine Learning, 2025, URL /tttps::/ 7UdvTX
fet//orummid
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
Yisheng Song, Ting Wang, Puyu Cai, Subrota K Mondal, and Jyoti Prakash Sahoo. A comprehen- sive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys, 55(13s):1 1-00: 2023.
Wenhao Tang, Fengtao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang, and Bo Liu. Feature re-embedding: Towards foundation model-level performance in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11343-11352, 2024.
Joshua B Tenenbaum, Vin de Silva, and John C LLngford. A global geometrie framework for nonlinear dimensionality reduction. science, 290(5500):2319-2333, 2000.
Yuandong Tian, Xinlei Chen, and Surya Ganguli. Understanding self-supervised learning dynamics without contrastive pairs. In International Conference on Machine Learning, pp. 10268-10278
PMLR, 2021.
Roman Vershynin. High-dimensional probability, 2009.
Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Kristen Severson, Eric Zimmermann, James Hall, Neil Tenenholtz, Nicolo Fusi, et al. A foundation model for clinical-grade computational pathology and rare cancers detection. Nature medicine, 30(10): 2924 2935, 2024.
Jinxi Xiang. Xiyue Wang, Xiaoming Zhang, Yinghua Xi, Feyisope Eweje, Yijiang Chen, Yuchen Li, Colin Bergstrom, Matthew Gopaulchan, Ted Kim, et al. A vision-language foundation model for precision oncology. Nature, pp. 1-10, 2025.
Conghao Xiong, Hao Chen, Joseph J.Y. Sung, and Irwin King. Diagnose like pathologist: Transformer-enablled hierarchical attention -eenttdn-guided multiple instance learning for whole slide image classification. In Proceedings of the Second International Joint Conference Artificial Intelligence, IJCAI-23, pp. 1587- 1595, 8 2023. doi: 10.24963/ijcai.22223777 URL 2023/176
fttts:://do..ogg// www.orrr/...........
Main Track.
14

---

Published as a conference paper at ICLR 2026
Conghao Xiong. Hao Chen, Hao Zheng, Dong Wei, Yefeng Zheng, Joseph JY Sung, and Irwin King. Mome: Mixture of multimodal experts for cancer survival prediction. In International Conference Medical Image Computing and Computer-Assisted Intervention, pp. 318-328 Springer, 2024a.
Conghao Xiong, Yi Lin, Hao Chen, Hao Zheng, Dong Wei, Yefeng Zheng, Joseph JY Sung, and Irwin King. Takt: Target-aware knowledge transfer for whole slide image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 503-513. Springer, 2024b.
Conghao Xiong, Hao Chen, and Joseph J. Y. Sung. A survey of pathology foundation model: Progress and future directions. James Kwok (ed.), Proceedings of the Thirty-Fourth International Joint Conference Artificial Intelligence, IJCAI-25, pp. 10751-10760. International Joint Conferences doi: 10.24963//jcai. 2025/1119... URL https
on Artificial Intelligence Organization, 8 2025.
7/doi www.orgg/go.... 10. 0. 24 24 jcai 2005
Survey Track.
Yingxue Xu, Yihui Wang, Fengtao Zhou, Jiabo Ma, Shu Yang, Huangjing Lin, Xin Wang, Jiguang Wang, Li Liang, Anjia Han, et al. A multimodal knowledge-eniee eenannee whole-slide pathology foundation model. arXiv preprint arXiv:2407.15362 2024.
Dianzhi Yu, Conghao Xiong, Yankai Chen, Wenqian Cui, Xinni Zhang, Yifei Zhang, Hao Chen, Joseph JY Sung, and Irwin King. Consurv: Multimodal continual learning for survival analysis. arXiv preprint arXiv:2511.09853, 2025.
Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah Coupland, and Yalin Zheng. Difd-mil: Double-tier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the TEEE/CVF conference on computer vision and pattern recognition, pp. 18802-18812 2022.
Yilan Zhang, Yingxue Xu, Jianqi Chen, Fengying Xie, and Hao Chen. Prototypical information bottle- necking and disentangling for multimodal cancer survival prediction. In The Twelfth International Conference on Learning Representations, 2024.
Zhenyue Zhang and Hongyuan Zha. Principal manifolds and nonlinear dimensionality reduction via tangent space alignment. SIAM journal scientific computing, 26(1):313-33 338, 2004.
Wenhui Zhu, Xiwen Chen, Peijic Qiu, Aristeidis Sotiras, Abolfazl Razi, and Yalin Wang. Dgr- mil: Exploring diverse global representation in multiple instance learning for whole slide image classification. In European conference on computer vision, pp. 333-351. Springer, 2024.
Barret Zoph, Golnaz Ghiasi, Tsung-Y Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoe Le. Rethinking pre-training and self-training Advances in neural information processing systems, 33: 3833-3845, 2020.
15

---

Appendix
LLM Usgge
17
Spectral Analysis Results
17
Complexity Analysis
18
D
Proof of Random Recalibration Matrix Preserving Input Characteristics
19
Proof of Approximation Capabilities of Manifold Residual Block
24
Additional Proofs
26
Experiment Setups
28
Additional Experimental Results
29
Limitations and Future Work
29
16

---

Published as a conference paper at ICLR 2026
Eigenvalue Spectrum: UNI
Features UNI
Features:
Random Uniform Transform: UNI
Normalzzed NomalizedEggemall Eigenvllue (log)
100
ARt 0.951
Unifore 5-523,
1211
leyien
124
I-SNE 2
L-SNE 2
12"1
-25
Delta R:
-10
Delta TE09995-502-344
75
-100
-120
Eigenvalue
DSSNE1
MSNE
Figure 4: Random transformation preserves the intrinsic low-dimensional manifold of UNI features. (a) Spectral analysis confirms the low-rank structure. (b) t-SNE reveals the cluster topology of the manifold. (c) High ARI and consistent coloring visually and quantitatively prove this topology is robustly preserved.
Eigenvalue Spectrum: ResNet50
Original Features: ResNet50
Random Uniform Transform: ResNet50
Normalzzed Eigenvalue (log)
Original 5=3.97 & PaWk
100
ARt:
Randors Uniferm (50.57 8-21.6, Pixd.3
Comprasiion regios
72
SD
I-SNE
2
12-
Delta 5:-22:00 6%
-30
-10
60-1
Delta TBG-95-312-440
134
Delta PR: 41.8%
130
-100
100
203
300
592
-100
-50
50
100
-53
0
Eigenvalue Index
t:
t:SNE 1
Figure 5: Random transformation preserves the intrinsic low-dimensional manifold of ResNet50 features. (a) Spectral analysis confirms the low-rank structure. (b) t-SNE reveals the cluster topology of the manifold. (c) High ARI and consistent coloring visually and quantitatively prove this topology robustly preserved.
A LLM USAGE
The authors utilized LLMs writing assistant during the preparation of this manuscript. Its role was primarily focused on tasks related to language refinement and improving the clarity of the narrative. Its application included: (1) enhancing the conciseness and academic tone of sentences and paragraphs; (2) restructuring complex sentences for better readability; and (3) brainstorming alternative phrasings for key scientific arguments.
is important to emphasize that all core scientific ideas, experimental design, results, and conclusions were exclusively developed by the human authors. The LLM served sophisticated editing and brainstorming tool. All text generated or modified by the LLM was critically reviewed, edited, and ultimately approved by the authors ensure accurately reflected our original intent and scientific findings.
B SPECTRAL ANALYSIS RESULTS
We apply the same procedures to UNI (Chen et al. 2024) and ResNet50 (He et al. 2016) features of Camelyon 16 (Litjens et al., 2018) dataset, and present the results in Figs. 4 and [5] From these figures, we can that features from these two feature extractors share similar characteristics as those from CONCH (Lu et al.) 2024]: rapid decaying spectrum and low effective rank. In addition, after applying a random uniform transformation on the features, the high ARIS and the cluster visualizations demonstrate that the random transformation can preserve the manifold structure well, both visually and quantitatively.
Interestingly, our analysis reveals a significant variance in the degree of this compression across models, which reflects their distinct, training-induced information encoding strategies. At one extreme,
17

---

Published as a conference paper at ICLR 2026
SrageeNet--retrainee ResNet50 exhibits dramatic rank collapse (from 52.8 21.6). We argue this is not a sign of superior and compact structure discovery, but a domain-inducee information collapse. This aligns with foundational studies transfer learning, which show that ImageNet priors are often suboptimal for specialized domains like pathology (Raghu et al. 2019; Zoph et al. 2020). leading reliance superficial "shorteut" features rather than genuine histological patterns (Geirhos) itt al. (020). At the other extreme, CONCH, trained with text-alignment, is incentivized to learn highly abstract, semantic-driven representation that captures key diagnostic concepts (Goh et al.) 2021). ). behavior consistent with findings in the vision- language models (Liang et al. 2022). This results in highly compressed, specialized feature manifold. In contrast, UNI trained via self- supervision, is driven to capture a broader spectrum of morphological diversity. Its objective of instance discrimination forces the model to learn features sensitive to fine-grained visual details (Chen) et al. 2020, Caron et al., (02T), resulting in richer, higher-dimensions (yet still low-rank) feature manifold. Crucially, despite these differences, the features from both expert PFMs (CONCH and UNI) provide far richer and more reliable structural prior than the collapsed representation of the generalist model.
C COMPLEXITY ANALYSIS
C.I SPACE COMPLEXITY ANALYSIS
In the linear layer, the matrix A, contains dgdı trainable parameters; however, in our approach, B is frozen, the only trainable part is LRP, with (do + d1) )r trainable parameters. Defining the reduction ratio r, (do + di) d (ddod)), and enforeing r, < 1, yields the constraint (ddd1)/(cn) + d1). If 2 1, the trainable parameter count in the MR block would match or exceed dod1, the trainable parameter count A., which makes our MR block meaningless.
C.2 TIME COMPLEXITY ANALYSIS
We begin by analyzing a general case time complexity and then use the result to analyze our method.
In this section, we derive the computational cost of both the forward and backward passes through single linear layer. We show that each pass scales linearly with the total number of weights dgd1.
For convenience of the following derivations, define the notations follows,
M [M,y] E Rdxdd,
(3)
x [z,]] Rd.
(4)
2 = Mr = [2]] E R4
(5)
8 = V I. = [6:] E Rd.
C.3 FORWARD PASS
To compute the pre-activation vector each component performs a dot product between the ith row of M and the input æ, the 24 is given by,
do Mis Tz-
Since there are d such outputs and each requires do multiplications (and roughly the same number of additions), the total cost of the forward pass is (ddd1).
C.4 BACKWARD PASS
The backward pass consists of three main steps: computing the gradient with respect to the weights, propagating the gradient to the inputs, and updating the weights.
18

---

Published as a conference paper at ICLR 2026
Weight Gradients chain rule, is given by,
The derivative of the loss L with respect to each weight My follows by the
OL aL d2
85 zz.
(8)
OM,j 0z, iM
Collecting these into the full gradient matrix yields the outer product,
MML = =
(9)
which requires computing one scalar product for each the djd1 entries. Hence, the cost of this step is (dod1).
Input Gradients
To propagate the enror back into the input space, we compute,
V&L= M d.
(10)
This is matrix- product of dimensions dy d1, again costing O(dgd1) )
Weight Update
A typical gradient descent update modifies each entry of M by
dL
Mij Mej " OM,
(11)
Updating all dod1 weights elementwise thus costs
(d d1).
(12)
C.5 TOTAL TIME COMPLEXITY
Summing the costs of the forward pass and all backward-pass components, we obtain O(ddd;) complexity. Therefore, single backpropagation step for one example in this layer has time complexity linear in the number of parameters dud1.
D PROOF OF RANDOM RECALIBRATION MATRIX PRESERVING INPUT CHARACTERISTICS
We categorize the geometric characteristics of the input features into variance and covariance: inner products and norms; cosine similarity; pairwise distances; condition numbers; the restricted isometry property; subspace embeddings; manifold geometry; cluster structure; nearest-neighhor relationships; and simplex volumes. We then demonstrate that each of these properties satisfies precise invariance conditions, ensuring that the recalibration matrix preserves the underlying geometric structure.
In this proof, we take the Kaiming uniform initialization with a /55 as an example, which is the default initialization method for linear layers.
D. VARIANCE AND COVARIANCE PRESERVATION
Intuitively, this result says that when you multiply your data covariance by "random" matrix M, all of the original variance gets spread out evenly across the new dj dimensions, up to the constant factor d1/(3du), In other words, particular direction in the original space is favored suppressed on average.
To derive the scaling of total variance under projection, start from
de
tr(M ΣM)
Σ (MT EMMEM) pp
Ma. Eg.r Mr
(13)
Taking expectation and using linearity gives
di do
Eitr(M x.)
ΣΣ
Eg E [[Mg. Mrr.)
(14)
19

---

Published as a conference paper at ICLR 2026
Because the entries My. independent with E[M O, only terms with g survive:
E Mg Mr
Var(M
3
q= r,
(15)
0,
q Tr.
Substituting back, we have
E = d1 do
1
d1
de
d1
tr(E).
(16)
4.9
3
3
9.9
300
p=1g=1
q=1
D.2 INNER PRODUCTS AND NORMS
A random projection with independent, zero-mean entries treats every coordinate of your vectors an unbiased, "democratic" way. On average, the dot product between any two vectors u and simply gets scaled by the factor d1 d3/(3 do), but its sign and relative magnitude remain the same. Geometrically, this means angles and lengths are preserved in expectation, no particular direction in the original space is preferred, and the overall similarity structure of the data survives the embedding up to known scale.
To how pairwise inner products scale, write
d1 do
(M u, M MD
Σ (M u)p (Mv)
Σ
Mp Us Mp Vgg-
(17)
Taking expectation:
u, M u)] = Σ
u E Mp. Mp.]
(18)
p.j.
By independence and zero mean, only i terms contribute:
= j,
E [[Mps Mp =
3 :
(19)
0.
+ j.
80
d
1
di
[[(M u, M v)} Similarly, the norm squared follows as the special case u = U,
Up Us
(u, v).
(20)
3
3 do
2m
d1
E[IM u/ =E(MM u, M u]]
|u||
(21)
do
D.3 COSINE SIMILARITY
Cosine similarity measures only the angle between two vectors, not their lengths. Since the random projection scales all lengths and dot products by the same constant in expectation, it does not distort angles on average. In effect, no particular direction is stretched more than any other, so the typical cosine similarity between any two points remains exactly as it was before projection.
Since both inner products and norms incur the same factor similarity is exactly preserved:
di in expectation, the mean cosine 3
Elcos 0*]
EKM u, M
v M
3
d (u, v)
cos 0.
(22)
E
2
E M 2||
di 3d
|
1111 3d
d1
D.4 PAIRWISE DISTANCES
This property guarantees that random projection will, with overwhelming likelihood, maintain the original distances between every pair of data points up to small, controllable emror. In effect, the
20

---

Published as a conference paper at ICLR 2026
projection spreads each original distance across many independent components, and by concentration of measure those components collectively reproduce the true separation very closely. As a result, one can embed high-dimensional data into much viwer-dimensional space without significantly distorting the geometric relationships, ensuring that algorithms relying on interpoint distances, such as clustering or nearest-neighhor retrieval, continue perform reliably.
Let U = I2 zz.- Then
do
? p.qq Uig
M e112
(23)
9
Define for each p.
de
E|y,1 0, Var(Y,) ||0|13
y Σ Mp pi Dgg
(24)
d
By concentration of Ep y2 p. with probability 2 0, 6,
(1- 8) ||u||2 V M e112 - (1 +++))) ||13,
(25)
provided
di 2 C Ix(88)
(26)
D.5 CONDITION NUMBER
This property ensures that random projection will not substantially worsen the numerical stability of the data. In other words, the projection maintains the balance between the directions in which the data stretches the most and the least. As a result, any algorithms that depend on solving linear systems or performing matrix decompositions will continue to behave reliably after projection, because the projected matrix remains nearly as well-conditioned as the original.
For data matrix X and any unit u,
|X M ull = TTTMXX
X X Mu. u.
(27)
Applying matrix concentration on XM shows that
d 2 C do.
(28)
then with high probability
(1 (1- X X u T M X x M u S x Xxu,
(29)
so
1 1+
<(XM) < 1-E
n: (X) ~ (1x))
(30)
D.6 RESTRICTED ISOMETRY PROPERTY
This property ensures that a random projection acts almost like perfect length-preserver on all sparse vectors at once. In practice, means that if the data has an underlying sparse structure, we can compress it drastically without losing the ability to tell sparse signals apart or to reconstruct them reliably. By making the projection dimension grow in line with how many nonzeros we expect, we guard against any sparse pattern being overly distorted.
For any K K-sparse x,
di
Mpsrig) )
M x 2
Σ
(31)
p=1
(Esupp(ι)
A union bound over all supports yields that if
di d 2 K In do
(32)
K"
then for all K K-sparse
(1- || & M x|| - (1 6) 1|
(33)
21

---

Published as a conference paper at ICLR 2026
Table 7: Experimental results comparing our method with baselines are presented under various shot settings. Performance metrics in which our method outperforms the baseline are highlighted in italic blue and the best metrics are in bold red. "#P" is the number of parameters (million).
Camelyon16 Flf
NSCLC F11
RCC
k Methods
AP
AUC
Acc.t
AUC1
Acc.t
AUC
F1
Ace.1
ViLaMII.
40.7|
53.812.7
4
61.1z
77.6s.0
69.4 69.42s
69.79.s
90,31,3
74.5
79.7s.7
ABMIL
0.26
41.2;.6
43.1
46.5a.
77.416.1
68.4p.s
69.1a.4
84.47.s
63.2a.1
64.7u.1
MR-ABBIL
0.10
4
43.2s.6
57.16.
85.5o.5
75.97
76.37.9
92.66.3
78.
81.87.2
TransMIL
2,41
47.5g.4
45.04.9
62.31.7
74.5s.1
65.76.a
66.64.
92.54.2
75.
79.69.4
MR- MIL-TTanMILL
2.01
51.210.0
47.5a.9
64.0a.8
79,47a
71.45.6
71.76.4
93.5a.9
77.15.4
81.7a.8
CATE
1.93
45.76.
52.9r.
78.9s.8
70.83.2
71.0a.
94.7a.1
81.7g.
84.02.5
MR- CATE
0.84
54.918.
51.916
54.316.6
83.27.
75.1
75.37
96.0a.a
82.4a.1
85.31.
DGRMIL
4.08
42.66.4
42.7g.7
62.32.2
68.19.5
60.4g.7
61.87.6
93.72.6
77.8s.7
81.61.2
MR-DO DGRMIL
3.29
59.72.5
52.0s.4
58.8s.7
74.1s.a
62.97.s
64.8s.s
92.7a.s
75.84.9
80.7s.s
RRTMIL
2.44
56.0a.7
56.66.0
62.96.4
61.26.7
55.56.9
56.06.5
91.6a.6
74.1
78.06.5
MR RRTMI..
2.04
47.6g.1
49.06.s
61.74.2
55.418.1
53.37.5
55.26.5
91.64.4
74.56.0
78.75.0
ViLaMIL
40.7
53.6.2.4
48.49.1
62.32a
82.92.1
73.92.9
74.32.
96.01.4
81.7a.a
85 85.52.9
ABMIL
0.26
45.5;.s
49.2,.2
56.1s,2
86.5..1
80.1s.0
80.21.9
89.3
74.211.9
76.7
MR-ABMIL
0.10
46.718.7
43.810.0
56.662
86.82.5
79.32.9
79,42.1
96.5a.
85.5a.0
87.85.5
TransMIL
2,41
$1.91o.a
46.77.s
51.0p.2
79.9a.9
71.02.8
71.62.5
96,51.3
83.5g.a
86.61.7
Transll.M
2.01
50.96.3
58.66.5
83.05.5
75.42.8
75.5p.5
97.01.2
84.00a
87.42.5
CATE
59.712.4
50.0
58.36.
81.4.4
72.3
72.8g
96.91.
83.92.9
86.52.5
CATE
0.84
62.716.9
58.41..8
62.313.5
87.0s.a
79.4s.2
79.4a.2
98.21.2
88.72.0
90.51.
DGRMIL
4.08
68.024.6
59.422.0
66.018.4
71.51a.6
60.3o.5
62.47.7
96.40.9
85.32.4
87.41.7
MIR-DGRMIL
3.29
65.4.0.1
55.718.4
63.615.2
80.1s.4
72.5s.0
73.12.8
95.8 82.0
83.2a.9
86.32.7
RRTMIL MII.
2.44
57.84.4
$4.86.2
57.1
65.74.8
60.82.9
61.5a.3
96.01.9
81.62.7
84.72.a
MR MR-RRTMIL
2,04
46.5a.s
42.7a.6
62.96.
69.96.
63.95.
64.05.a
96.71.4
82.52.7
86, 1..
ViLaMIL
40.7
74.0,7.2
65.9
71.816.
88.4s.2
79.2s.7
79.2,.7
96.8n.9
86.2
88.0,.7
ABMIL
0.26
49.26.s
48.93.0
59.74.8
87.7s.9
81.05.4
81.16.4
91.86.3
76.6s.4
7,91
MR-ABMIL
0.10
56.515.5
53.81a.9
61.99.7
94.0s.a
85,93.3
86.03.3
98.To.8
87.62.9
89.62.1
TransMII.
2.41
57.52.5
$2.0k.
53.67.4
87.96.1
80.3s
801.4.
47.90.
86.5a.
88.4s.1
MR- TransMIL.
2.01
65.3g.5
54.2n.3
63.0g.8
91.6g.1
82.44.1
82.6,.
97.96.4
86,72.5
89.1
CATE
1.93
79.8.4.7
69.120.0
76.01..6
87.7s.7
78.0.
78.24.9
97.8p.s
86.4g.
88.32.1
MR-CATE
0.84
88.014.8
81.716.0
82.916.3
93.56.a
84.6s.7
84.76.9
98.60.3
89.11.3
90.71.0
DGRMIL
4.08
84.7,o.0
72.5
77.412.
85.5e.a
73.4a.
74.2;.7
96.71.1
86.82.9
88.72.a
MR- DGRMIL.
3.29
93.40.5
86.0g.s
86.7
91.67.a
82.8g.a
83.06.
97.61.
85.
87.91.7
68.8
62.615.4
63.715.2
82.5p.a
73.96.
74.26
97.0h.1
85.2g.3
86.95.a
MR RRTMIL
2.04
85.3
79.210.6
81.7.3.3
87.3v.a
79.67.9
79.8;
97.80.4
86.31.a
87.81.4
ViLaMIL
40.7
81.42s.1
77.522.0
82.016.0
92.82.9
84.92.7
85.02.7
97.41.1
85.5a.o
87.92.2
ABMIL
0.26
7
71.215.4
75.710.2
93.62.4
87.83.4
87.8
94.75.a
81.07.s
82.8
MR ABMII.
0.10
79.0aa. 1
74.522.
78.017.1
96.5,.2
89.1..0
89.11.0
98.30.7
/2.8
91.12.
TransMIL
2.41
69.7
62.516.2
69.313.3
95.61.7
88.2s.6
88.23.6
98.4p.3
89.3
90.9o.7
MR TransMIL.
2.01
79.3
75.
77./13.2
93.24.4
84.0c.1
84.1e.o
98.60.3
89.31.
91.21.0
16 CATE
1.93
92.2g.1
87.5s.2
88.52.
94.22.1
85.12.0
85.22.0
97.9a.6
87.42.7
89.32.0
MR-CATE
92.4a.2
86.7a.9
87.8s.4
96.1
88.65.a
88.73.3
98.6p.
89.8..1
91.62.3
DGRMIL
4.08
88.5g.0
79.410.3
80.91..3
92.45.4
85.84.
85.8a.9
94.81.4
86.31.
88.71.2
MR- DGRMIL
3.29
92.3g.s
87.6s.s
88,76.2
93.91.7
N6.62.7
N6.72.1
97.70.s
88020
W0.31.5
RRTMIL
2.44
84.0.6.9
74.812.9
75.512.7
90.7s.2
82.6
82.74.1
98.3n.s
86.92e
88.82.4
MR- RRRMIRIIL
2.04
93.71.8
89.02.1
89.8
91.0s.1
82.57.8
82.8,
97.8p.s
86.5..1
88.92.
D.7 SUBSPACE EMBEDDING
This property guarantees that random projection acts near-perfect isometric embedding for any fixed low-dimensional subspace. It means that all geometric relationships, lengths and angles, within that subspace are preserved almost exactly after compression. As result, any downstream algorithm that relies on the subspace structure (for example, solving linear least-squares problems or performing spectral decompositions) will perform almost identically in the reduced space, yet with greatly reduced computational and storage costs.
For d-dimensional subspace with basis A E dddoxd one shows with a net argument that if
di = 0(=
(34)
then with high probability
(1
(35)
22

---

Published as a conference paper at ICLR 2026
Table 8: Experimental results comparing our method with baselines are presented under various shot settings using data-imbalance metrics (AUPRC and F1-score) Performance metrics in which our method outperforms the baseline are highlighted in italic blue and the best metrics are in bold red.
Camelyon16
NSCLC
RCC
h Methods
#P
AUPRCT
F1f
AUPRCT
FI
AUPRCT
FIf
ABMIL
0.26
35.86.6
43.16.5
74.012.0
68.49.8
74.711.5
63.2
MR-ABMII.
0.10
37.912.1
43.2s.8
81.29.3
75.97.2
86.411.3
78. /7.5
CATE
1.93
40.79.9
45.76.1
74.66.5
70.83.2
89.65.0
81.73.2
MR-CATE
0.84
48.221.2
51.916.1
80./
75.17.3
91.06.0
82.44.1
ABMIL
0.26
40.76.9
49.27.2
84.7s.8
80.1s.u
82.7g.4
74.211.9
MR-ABMIL
0.10
39.0 39.0sa.5
43.810.0
81.96.1
79.32.3
93.66.1
85.56.u
CATE
1.93
52.612.1
50.0p.a
76.94.9
72.34.1
94.32.5
83.92.9
CARCCATE
0.84
54.319.6
58.415.6
83.97.9
79.43.2
96.12.4
88.72.0
ABMIL
0.26
45.44.1
48.93.0
81.77.2
81.05.4
87.7s.2
76.68.4
ABBIL
0.10
53.019.1
53.813.9
91.75.2
85.93.a
96.41.2
87.62.9
8
CATE
1.93
76.819.0
69.
84,9a.6
78.04.9
96.01.1
86.43.0
MR CAREEATE
0.84
84.920.9
81.716.0
92.66.0
84.6s.7
97.20.5
89.11.3
ABMIL
0.26
72.5
71.215.4
91.04.4
87.83.4
90.8s.
81.07.8
16 MR-ABMII
0.10
77. 77.926.5
74.522.4
96.31.6
89.11.0
96.41.8
89. /2.8
CATE 1.93
92.52.7
87.53.2
92.83.7
85.12.0
95.91.3
87.42.7
MR-CATE
0.84
92.2a.3
86.73.9
96.01.8
88.63.3
97.21.1
89.83.1
Table 9: Experimental results on two treatment response datasets.
Boehmk F1
Trastuzumab F11
k
Methods
AUCT
Acc.t
AUC1
Acc.f
ABMIL
0.26
56.2g.7
45.96.a
49.7
47.75.6
40.33.9
42.42.
4
MR-ABMII.
0.18
65.411.3
56.96.5
65.13.7
56.92.3
40.96.7
58.80.0
ABMIL
0.26
53.45.1
46.14.6
50.9s.2
49.4s.7
48.810.8
54.17.7
8
MR-ABMIL
0.18
63.4g.5
57.67.1
62.3g.8
55.14.4
40.34.5
54.16.4
ABMIL
0.26
52.65.5
46.97.3
50.3s.0
46.310.0
36.11.3
48.26.4
16
MR-ABMIL
0.18
63.54.1
55.13.4
63.46.5
57.70.s
43.68.3
56.5s.2
D.8 MANIFOLD GEOMETRY
This property ensures that a random projection will faithfully reproduce the intrinsic geometry of any low-dimensional manifold embedded in high-dimensional space. Even though the data may lie on curved, nonlinear surface, we can compress it down to far fewer dimensions and still retain nearly all of the true "geodesic" distances along that surface. The key idea is that the manifold can
Table 10: Performance of MR-ABMIL with replaced linear layers under varying shot settings on Camelyon16, TCGA-NSCLC, and TCGA-RCC datasets. "U" and "V" denote two matrices in MR-ABMIL.
Camelyon16 F11
NSCLC FI1
RCC FI1
& Methods
#P
AUC1
Acc,t
AUCt
Acc.t
AUC1
Acc.f
+
0.10
46.1
43.8
56.65.2
86.82.
79.3
79.4g
96.5g.6
85.56.0
87.8..5
0.18
45.60.3
43.96.2
59.81.4
86.13.5
78.24.2
78.4a.1
94.04.8
79.07.5
81.77.a
0.18
52.8.o.s
47.9%.
59.42.s
85.5g.
77.2a.5
77.6a
90.4s.a
73.41..5
77.410.3
V +
0.10
56.51s.s
$3.813.9
61.9 61.9s.7
94.0u.3
85.9u.3
86.0g.a
98.10.8
87.62.9
89.62.1
0.18
71.41..9
65.5
69.97.3
95.1s.s
86.
86.2..a
96.81.
84.3s.4
87.04.4
0.18
72.81o.1
67.820.0
71.5
94.61.7
85.8a.s
85.9g.7
95.1s.a
79.811.7
83.67.9
+
0.10
79.0ss.1
74.522.4
78.0 78.017.1
96.5
89.11.0
89 89.1
98.30.7
89.12.8
91.12.0
0.18
86.411.7
80.9:2.0
81.911.
96.01.7
88.71.4
88.71.4
98.30.8
87.72.3
90.02.0
82.51s.4 0.18
77.918.3
81.9
96.91.1
89.81.5
89.81.5
97.71.1
86.3s.9
88.4g.5
23

---

fundamental result Published as a conference paper at ICLR 2026
be approximated by finite set of local patches, and the projection preserves each patch's 's geometry so accurately that the entire shape remains intact. As a result, analyses that rely on the manifold's structure, such nonlinear dimensionality reduction manifold-bases learning, remain valid and effective after compression.
Cover d-dimensional manifold of volume V by N (V/r)² balls. If
di 2 Ce In
(36)
then geodesic distances are preserved within e.
D.9 CLUSTER LABELS
This criterion ensures that the worst-case contraction of the gap between clusters still exceeds the worst-case expansion of each cluster's size. In practice it means that random projection will not cause any overlap between clusters that originally well separated. As result, any clustering classification based on distance remains unchanged, and all original labels are preserved.
Clusters of diameter D separated by A satisfy label preservation if
(1- (1+
(37)
D.10 NEAREST NEIGHBORS
The nearest-neighbor property refers to the guarantee that, after random projection, each point's set of & closest points (its k-nearest-neighhbr graph) remains exactly the same as in the original high-dimensional space.
Fast JL transforms M PHD satisfy the same distortion bounds and run in time Odg log do + +d1); the N--N graph is unchanged if
7
(38)
2D
D.11 SIMPLEX VOLUME
A simplex in this context is the most elementary convex polytope determined by one "base" point together with a set of other points that do not all lie in a lower-dimensional subspace. In two dimensions it is triangle, in three it is a tetrahedron, and in higher dimensions the generalization of those. Its volume quantifies the amount space enclosed by those corner points. Because the projection acts almost the total enclosed volume can only change by the small, precisely bounded factor implied by the near-isometry.
exact length-preserve on each independent direction in the simplex,
For + 1 affinely independent points,
1 Vol(A) à det(zi 2o, .. xol
(39)
Since M is an approximate isometry on this d-dimensional span, its singular values lie in [VI E, /1 11++ E), giving
(1 (1-s)d/2Voll) Vol(A) S Vol(A') < (1 + +2)//2 Vol(A),
(40)
hence squared volume changes by most (1 + =)
E PROOF OF APPROXIMATION CAPABILITIES OF MANIFOLD RESIDUAL B1.OCK
In this section, we prove that our MR block can approximate any linear layer to arbitrary precision, thereby providing worst-case performance guarantee: if the MR block cannot improve over the linear layer then it can match its behavior arbitrarily closely, ensuring no loss in worst-case performance. In addition, rather than directly proving our formulation, establish
24

---

Published as a conference paper at ICLR 2026
Table 11: Experimental results on different initialization methods in MR-CATE. "K." and "X." stands for Kaiming and Xavier initialization. "U." and "N." stands for uniform and normal distribution. The number following is the input parameter for initialization. The gray rows stand for our main setting.
Camelyon16 FIf
NSCLC FI
RCC
k Methods
AUC1
Acc.
AUC
Acc,t
AUC
F11
Acc.f
K. N.O
63.0
49.5
61.7
82.24.2
75.0..7
75.2
98.01.4
87.8,
90.1 1.8
K. U.O
62.417.0
48.916.9
63.710.5
86.54.4
79.4a.2
79.4s.1
98.11.4
88.23.0
90.32.5
4
K. U. v5
12.1
58.415.8
62.313.5
87.03.9
79,4a.2
79,43.2
98.21.2
88.72.0
90.51.a
X. O
66.214.1
48.820.1
64.712.9
84.0s.a
76.34.2
76.44.2
98.11.4
87.9a.4
90.12.9
O.
61.117.1
51.217.8
63.612.8
86.44.6
79.
79.
98.1
88.22.9
90.22.6
N.
89.513.8
78.1
83.312.2
90.4s.9
I2.
82.2s.9
98.5p.5
89. 89.1
91.11.9
K. U.O
87.413.8
77.720.9
82.811.3
91.94.8
82.94.4
83.04.s
98.5p.5
89. 89.12.0
90.71.6
8
K. U. V5
88.0
81.7
82.915.3
93.55.3
84.6s.7
84.7s.6
98.60.1
89.1
90.71.0
N.
19.1
75.921.1
81.211.0
92.17.1
86.
84.26.7
98.6a.4
90,31.1
91.51.0
X.
85.614.7
76.219.5
81.611.1
93.54.7
85.0s.3
85.0s.2
98.7g.4
90.0,
91.5
N.
93.81.7
88.9..6
89.61.a
95.42.2
86.92.
87 $7.02.8
98. 60.
88. 88.22.2
89.62.0
K. U.O
92.22/2
87.5
88.42.0
94.62.3
86.32.2
86.42.1
98.6g.a
88.8
90.92.0
16
K. U. /5
92.4a.2
86.73.9
87.83.4
96.11.4
88.63.a
88 88.7a.3
98 60.6
89 89.8g.1
91.62.3
X. O.
94.51.5
90.11.5
90.71.6
95.62.0
88.51.4
88 38.51.4
98.70.5
88.92.2
90.91.4
U.
92.73.7
88.5a.5
89.13.4
95.81.3
88.7a.1
88.8s.1
98.7a.4
89.5g.1
91.22.5
Table 12: Experimental results on different pretrained feature extractors under various shot settings. "R50" stands for ResNet50, and "MR-" stands for MII. model with our MR block.
ABMIL
CATE
Methodis
NSCLC F11
RCC F11
NSCLC F11
RCC F11
AUCT
Acc.t
AUC
Acc.
AUC1
Acc:t
AUC1
Acc.1
R50
$3.5as
57.1aa
74.75
$7.3r.6
61.1as
$3.2a
40.310.5
$1.0aa
77.8..
47.0s.a
61.7s.
MR-R50
62.36x2
35.311.7
39.0a.
80.2s
63.77.
67.0ae
$1,4b.
41.311
51.442
78.62z
48
59.6
UNI
74.5
68.57.1
69.07.0
96.62
83.8
$6.12.
64.6s.s
41.612
53.37.6
90.5u.
68.8s.a
71.4g.
MR-UNI
86.52
77.56,7
78.2a.1
97.9:
87.5g.a
89.6a
59.01
49.718.7
56.15.7
92.1as
73.5p.
76.9.
R50
59.0s.7
54.7a.1
57.0a.c
74.6c.
56.6..0
58.5
64.97.
57.416.
60.0c.
82.611.
64.210a
68.9r.
MR-R50
64.34
39.7a.
60.1aa
80.8
62.36.
63 63%
63.5v.
60.76.
67.45.
89.3a
75.0v
76 76.0.
UNI
72.86.2
73.1e.2
98.60
88.72.1
90.31.a
82.77
724s..
73.0s.1
98.
89.3g.
MR-UNI
91.35.0
81.74.4
$1.8g
98.60.
91.21.6
92.31.4
88.8.cc
80.6g
80.86.7
98.56.4
90.81
91.8.1
R50
62.3,.0
67.7as
78
MR-R50
67.2p.6
63.7
63.94a
85.92
70.82.
73:f1.
66.3124
39.215.1
63.0g.2
98.74.2
81.4..1
82.4..
16
UNI
91.127
83.6g.0
83.6a.
98.96
89.21.0
91.01.s
87.27.5
75.8p
76.95.
98.0g.a
58.02.
90.3..6
MR-UNI
Table 13: Performance of ABMIL and CATE, with and without our MR block (MR-ABMIL and MR-CATE), under 32- and 64-shot settings TCGA-NSCLC and TCGA-RCC datasets.
NSCLC F1
RCC
Methods
AUC1
Acc.t
AUC1
F11
Acc.t
ABMIL
0.26
96.21.a
89.7
89.73.6
97.61.8
91.12.0
93.12.0
32 MIR-BBMIL
0.18
98.1
93.11.7
93.11.7
99.50.3
93.40.s
94.90.8
CATE
1.93
97.0p.4
91.11.8
91.11.8
99.30.3
91.02.0
93.11.8
MR-CATE
0.84
97.7p.5
92./1.6
92.
99.60.1
92.92.2
94.51.9
ABMIL
0.26
97.5p.9
91.51.a
91.5
99.0p.s
91.62.9
93.22.5
MR-ABMIL
0.18
98.2g.a
92.81.2
92.81.2
99.60.2
94.31.1
95.61.4
64
CATE
1.93
97.7
91.11.4
91.11.4
99.4v.2
93.61.1
95.0p.s
MR CATE
0.84
97.9p.1
92.41.1
92.41.1
99.8p.1
95.31.0
96.6u.s
without activation functions. Subsequently, with the universal approximation theorem (Barron 1993 Cybenkol 1989 Funahashi 1989 Hornik et al. 1989 Lu & Lu 2020), our architecture with nonlinear activation function can achieve even superior approximation precision.
Theorem E.1 (Universal Approximation by Manifold Residual Block). Let A, E xdd have full rank A min min(do, di Draw frozen matrix B dd with uub-Gaussian entries. Then almost
25

---

Published as a conference paper at ICLR 2026
surely: for every > 0 there exist integers TA and matrices W2 and W1 such that,
||A, [ + W2 Wi) S, where W2 E Rdoxr, and W1 E Bxxd
(41)
Proof. Full-Rank Guarantees and Singular- Value Bounds. We prove that B is full rank almost surely in Sec.
F.1 Let 6, E (0, 1), according to
Lemma, we have,
11-8811 (1-5)|1a112
B 2 Σ (1+6)||=12 (a d (1 5), do (1+ (42)
with probability at least
if the input dimension satisfies do 2 C(di + In(1/n)/88.
Truncated SVD of the Residual. Let A. B. Its singular value decomposition is given as E = -UEV.. By the Eckart- Eckart-Young Mirsky theorem, we choose r S and we have,
rank(E)
E - Erllr
Σ
of(E) Se, e, where E,
Σ o. u, D,_
(43)
Low-rank Correction Construction. From the last step, set W and W2 as,
W₂ = [Ui,..., U. E// W E//² [2), (i,,..., Ur J, where Σ, diag or); (44)
which satisfies W = E, and therefore, for sufficiently large we have
| (B W2WWl ErIle <
(45)
F ADDITIONAL PROOFS
F.1 RANDOMLY INITIALIZED MATRIX HAS FULL RANK
Theorem F.1. Let W Rddadd have entries drawn i..d. from a continuous distribution (e.g. Kaiming uniifor- with gain V5). Then with probability one,
rank( W) min((do d1}.
(46)
Proof. Set mo. d.). Define the singular set
= (W rank(W)) < <r).
(47)
matrix in S must make every minor vanish. Label those minors by index sets 7, and write
Z, [W det W) = 0}.
(48)
Each dety is nonzero polynomial in the entries of W, so by basic measure theory its zero set Z has Lebesgue measure zero. Since there are finitely many
s s
(49)
a finite union of null sets, hence also null. Finally, the distribution on entries is absolutely continuous with respect to Lebesgue measure, so
Pr(W& E S) = 0,
(50)
and therefore W has full rank almost surely.
n
26

---

Published as a conference paper at ICLR 2026
3mm
3rzzmn
3mmm
500mm
500ym
500
Figure MR-CATE --Tergeee heatmaps for tumor058 (Camelyon16) The left panel shows heatmaps for three biopsy regions, while the right panel displays fine grained patches with their heatmaps. Red indicates high attention (tumor), whereas blue indicates low attention (normal tissue).
F.2 MAXIMUM RANK OF THE LRP
Theorem F22 (Rank of Low-Rank Product). Let W2 E mmxr and W1 R with < min,m,n n). Then
rank(W2W) S r.
(51)
Proof. Observe that
rank 5 r, rank W1) S r,
(52)
since W2 has only columns and W, has only r rows. A standard rank-inequality for matrix products states
rank(AB) & min(rank(A),, rank(B))
(53)
for any conformable A, B. Applying this with A W2 and B W1 gives rank(W,We s min(rank(W.)) (W2), rank (W1) S r.
(54)
27

---

Published as a conference paper at ICLR 2026
G EXPERIMENT SETUPS
G.1 DATASET DESCRIPTIONS
1amelyon16 (Litjens et al. 2018) comprises 397 WSSs of lymph node sections from breast cancer patients, annotated for metastatic presence. The dataset is officially partitioned into 268 training specimens (157 normal, 111 sumor-containing) and 129 testing specimens.
TCGA-RCCI
encompasses 940 WSIs from three distinct renal cell carcinoma subtypes: Kidney Chromophobe (TCGA-KICH, 121 WSIs from 109 cases), Kidney Renal Clear Cell Carcinoma (TCGA-KIRL KIRC, 519 WSIs from 513 cases), and Kidney Renal Papillary Cell Carcinoma (TCGA- KIRP, 300 WSIs from 276 cases).
TCGA-NSCLC consists of 1.053 WSIs representing two major lung cancer histological subtypes: Lung Squamous Cell Carcinoma (TCGA-LUSC, 512 WSIs from 478 cases) and Lung Adenocarci- noma (TCGA-LUAD, 541 WSIs from 478 cases).
Boehmk (Boehm et al. 2022) is a treatment- treatment-response cohort of high-grade serous ovarian cancer patients, with H&E-stained WSIs and associated progression-free free survival (PFS) metadata. We frame this as binary response prediction task using 12-month landmark analysis: patients with PFS 2 12 months are labeled as responders, those with documented progression before 12 months as non-responders, and patients censored before 12 months are excluded.
Trastuzumab (Farahmand et al. 2022) is a HER2-positive breast cancer cohort comprising pre- treatment H&E tumor WSIs and clinical metadata describing response to trastuzumab-basee therapy. We treat this as an intrinsically few-shot, slide-level treatment-respone task by assigning each patient binary label (responder vs. non-responder) based on the curated clinical "Responder" status.
G.2 EXPERIMENTAL PROTOCOL AND
SETTING
Experimental Protocol. We benchmark our approach on established multiple instance learning frameworks: AB-MIL (Ilse et al., 2018) (attention-based method), TransMIL. (Shao et al.) 2021 TTaasformer-basee method), CATE (Huang et al.) 2024) information-theoo ter--tthhoy-beeeddd -ased method), DGR- MIL (Zhu al. (224) (density modeling method), and RRTMIL (Tang al., 2024) (re-embedding method). Regularization techniques such as dropout, weight decay are used exactly as in the official implementations. In addition, we also compare our methods with ViLaMIL (Shi et al. 2024), the recent visual language models specifically proposed for few-shot WSI classification. All implementa- tions utilize their official codes. We employ early stopping with a patience of 20 epochs on validation loss, while enforcing a minimum training duration of 50 epochs. For our experiments, we adhere to the official training and test partitions for the Camelyon 16 dataset. We further split the official training set into new training and validation sets using 70/30 ratio. For the rest of the datasets, perform our own random split, partitioning the data into training, validation, and test sets with a 60/20/20 ratio. From these constructed training sets, we then create our few-shot scenarios by randomly sampling WSIs per class to create five different subsets for each fold of our cross-validation. The average performance and standard deviation across these five runs reported.
prrpe-pprramters Details. To rigorously assess the universal, plug-and-play value of our block, we adopt a strict paired-comparison protocol under unified hyperparameter setting. For each baseline architecture, we evaluate its original and MR-enhancee versions while keeping all other conditions identical, ensuring the observed performance gain is solely attributable our block. We use the Adam W [Loshchilov & Hutter 2019) optimizer with learning rate and weight decay being 5 x 10-1 and 10 1 linear scheduler is utilized for our methods with starting factor and end factor being 0.01 and 0.1, respectively. Dropout is set to 0.25. We use CONCH (Lu et al. 2024). UNI (Chen ctt al. 2024) and ResNet-50 (He et al. 2016) to extract features from the non-overlapping 224 x 224 patches, obtained from 20x magnification of the WSIs, and the resulting feature dimensions are 512, 1,024 and 1,024, respectively. The CONCH features are utilized throughout our experiments as they demonstrate superior performance the other two features. The UNI and ResNet-50 features are only utilized for investigating the robustness of our method against features from different foundation
'The TCGA data used in our work is available in http://pottalgdd.comcrroo
28

---

Published as a conference paper at ICLR 2026
models. The rank is set to 64. GELU (Hendrycks & Gimpel 2016) is chosen as it yields the best performance among the alternatives.
G.3 COMPUTER RESOURCES
We conduct our experiments on an NVIDIA A100 GPU and Intel(R) (e) Silver 4410Y CPU, with Ubuntu system version 22.04.5 LTS (GNU/Linux 115.0-136-geneed x86 x86_64). More detailed Python environment would be released with the code.
G.4 REPLACEMENT POSITION OF THE MODELS
Preliminaries of ABMIL. ABMIL can be formulated follows.
04 exp w (tanh(Vh]] @ (ig ))}
aghs:
(55)
2j=1 exp T (lanh( VhJ) ) @ sigm( sigm(Uh)]) (Uh}))}
where z is the slide-level feature.
ABMIL: replace both the and matrix with our MR block.
TransMiL only replace the out matrix in Nyström attention implementation with our MR block.
CATE replace all linear layers in x_lin__a interv_l lev_iinnerrrrrr feature output linear layer and the second linear layer in encoder IB with MR block.
DGRMIL replace linear layers in self f. fel of opt pootimiee_tte ipptizeer_tippl triple and self. crossffn
DGRMIL.
RRTMIL replace linear layer in sel f proj of InnerAttention.
More details will be available along with the code.
G.5 COMPUTATION CONSUMPTION MR BLOCK
In this section, present FLOPs and wall-clock inference time for the MR-enhanced backbones under different bag sizes and feature dimensions in Table
H ADDITIONAL EXPERIMENTAL RESULTS
The additional experimental results are presented Tables 10 to 12 The discussions these tables
are already available in Sec.
I LIMITATIONS AND FUTURE WORK
This work introduces MR block to mitigate the manifold degradation of the vanilla linear layer while enabling parameter-efficient task-specific adaptation. However, our empirical evaluation has several limitations. First, fix the rank across all layers, choice that is unlikely be optimal in real-world scenarios. A more comprehensive exploration of different rank values within a single model would better characterize the trade-off between model size and accuracy. Second, our study of insertion positions is limited to the ABMIL architecture; to establish broader applicability, the MR block should also be tested in more sophisticated MIL frameworks such TransMIL and CATE. Third, our geometric anchor now is frozen and does not contain any dataset-level information. interesting extension is replace the random anchor with data-driven PCA projection, at the cost of dataset-specific precomputation for more comprehensive utilization of dataset-level information. Fourth, we do not evaluate MR on top of slide-level pathology foundation models
29

---

Published as a conference paper at ICLR 2026
512-d features (CONCH)
1024-d features (UNI/ResNe(50)
Model
Bag Size
FLOPs (G)
Time (ms)
FLOPs (G)
Time (ms)
100
0.053
0.90
0.420
1,44
ABMIL
1000
0.525
2.66
4.197
2.73
10000
5.251
2.78
41,974
3.37
100
0,333
2.69
1,541
2,43
CATE
1000
3.328
3.12
15,410
3.12
10000
33.283
10.36
154.102
10.05
100
1.153
12.33
4.822
12.27
TransMIL
1000
6,076
16.25
26.403
15.66
10000
49.832
46.23
220,299
44.54
100
1.332
12.19
5.560
12.18
DGRMIL
1000
4.862
12.75
21.566
12.77
10000
43.431
31.36
194.718
31.59
100
1,008
3.01
4.243
3.13
RRTMIL
1000
3.227
3.52
15,006
5.92
10000
30.286
13.04
142.116
46.47
Table 14: FLOPs and inference time for different MIL models under varying bag sizes and feature dimensions.
such as TITAN (Ding et al. 2025). ) because this would require non-trivial architectural redesign (MR cannot simply replace the tiny slide-level classifier, and TITAN is tightly coupled to CONCH v1.5 while our experiments span CONCH, UNI, and ResNer50). In addition, our geometric analysis relies on approximately 30,000 patch-level features per configuration, whereas TITAN only provides slide-level embeddings per WSL, directly reusing the same analysis would yield much less stable statistics. treat these limitations as future directions of our work.
We anticipate that the proposed MR block will broadly advance few-shot learning within computa- tional pathology and beyond. minimal deployment requirements render it readily applicable in wide range of settings. In computational pathology, it may enhance tasks such as few-shot report gen- eration, survival outcome prediction. mutation inference, and treatment- others. More generally, the MR block could improve performance in computer vision applications, including few-shot segmentation and classification, and may even be adapted to natural-language- processing tasks. Further exploration across these domains is therefore warranted.
modeling, among
30


Reviews:
[{"review_id": "q3X0DfXSeP", "reviewer": "Reviewer_jYL1", "summary": "The authors introduce a novel layer to preserve low-dimensional manifold geometry within modern Multiple Instance Learning (MIL) frameworks for few-shot classification of whole slide pathology images. They first show that embedding spaces from well-known pathology foundation models manifest low-dimensional manifold geometries and that these are not well-preserved in popular attention-based MIL framework. Authors show that this tends to be due to linear layers such as those present within gated-attention mechanism used in ABMIL and they propose a novel layer, called the Manifold Residual (MR) block, to better preserve geometry. The latter is decomposed in 2 parts operating on a feature matrix $X$: (1) a fixed random matrix transforming linearly $X$ useful to preserve topology; (2) a trainable low-rank residual pathway (LRP). Authors study the theoretical properties of their method, demonstrate its relevance to improve many MIL models for few-shot classification tasks and perform a range of ablation studies.", "questions": "I invite authors to discuss the weaknesses mentioned above, knowing that I am really inclined to increase my initial grade. A last question:\n\nQ1. Could authors clarify whether there are correlations between geometric properties of the different datasets with the results observed in the ablation studies reported in Table 2, Table 3 and Figure 3?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The authors propose a relevant analysis to emphasise low-dimensional manifold properties of a range of foundation models for pathology.\n- Propose a novel layer for few-shot MIL, the MR block with a custom training strategy.\n- They provide theoretical results on a range of geometric/statistical properties preserved by perturbations by random matrices.\n- Demonstrate a universality approximation theorem for the MR block. \n- Show on 3 datasets that the MR blocks, instead of linear layers, within 5 MIL frameworks improve few-shot WSI classification while leveraging 3 different types of pretrained models (CONCH, UNI, ResNet50)\n- Perform ablations on the 2 parts included within the MR blocks, which tend to show that coupling these 2 parts brings the best performance.\n- Conduct several sensitivity analyses, to question where to replace linear layers with MT blocks within ABMIL, how to initialize the MR blocks and whether the MR blocks are robust to their rank hyperparameter.", "weaknesses": "- **W1 : clarity** There are several points in the paper that would benefit from clarification and/or further detail:\n   - a) L63: \"linear layer\". For people knowing the MIL literature it is not clear at this stage, about which linear layers you are referring to, e.g those included in the gated-attention layer of ABMIL or actually the linear classifier at the end of the architecture, which can also have an effect. This should be clarified.\n   - b) The dataset used for the geometric studies reported in Fig 1 and 5 is never mentioned.\n   - c) Figure 2: The box \"MIL model\"  explaining how are supposed to intervene the MR blocks is really not clear.  \n   - d) Section 2.2: it is not clear to me why more generic few-shot learning paradigms/literature (see e.g [A]), applicable to any bag representations in MIL is omitted in the related work. For instance well-known prototypical neural networks [B] could be applied as a readout within ABMIL instead of a linear layer and their inherent dependence to distances could be a good proxy to preserve geometric properties. This observation underlines that it is not clear in the paper why different readout strategies are not discussed. I invite authors to do so during the rebuttal.\n   - e) Section 3.1: While both instance-level MIL and embedding/bag-level MIL are mentioned in Section 2.1, Section 3.1 only formalizes bag-based approaches. It can be sufficient to mention that in Section 3.1 with a disclaimer that only bag-based methods are benchmarked in the paper.\n   - f) To improve readibility of most tables, I suggest authors to express everything in % instead of 0.x.\n \n\n[A] Song, Y., Wang, T., Cai, P., Mondal, S. K., & Sahoo, J. P. (2023). A comprehensive survey of few-shot learning: Evolution, applications, challenges, and opportunities. ACM Computing Surveys, 55(13s), 1-40.\n\n[B] Snell, J., Swersky, K., & Zemel, R. (2017). Prototypical networks for few-shot learning. Advances in neural information processing systems, 30. \n\n- **W2: benchmarks and ablations**:\n   - a) Most tested datasets are significantly imbalanced hence I don't think that the choice of metrics such as the accuracy and AUC are the most appropriate. I believe that simply presenting macro F1 scores in the main paper could be sufficient to share the main messages, and potentially include AUPRC [C] for completeness in the main paper or supplementary. \n   - b) In most ablation studies and sensivity analyses, many strong claims are made by authors when the results hold for at most 2 out of 3 datasets. Therefore I strongly encourage authors to include at least 2 other WSI datasets in their experiments to better support these claims.\n   - c) Authors argue that a central issue of MIL methods for few-shot classification is overfitting. Nonetheless, there is a pletora of implicit or explicit regularizations (e.g dropout, attention dropout, norm constraints etc) that could be envisioned. Hence the scope of the baselines chosen by authors is not clear and should be further justified by authors or completed by including different regularization techniques. \n   - d) Hyperparameters of benchmarked MIL models are not present in the paper and should be added.\n\n- **W3: asymptotic analysis**: While authors mention that their method brings less improvements in 16-shots WSI classification than with less supervision, it could be interesting to stress test their methods with higher ranges of shots on the larger datasets like TCGA-NSCLC.\n\n[C] McDermott, M., Zhang, H., Hansen, L., Angelotti, G., & Gallifant, J. (2024). A closer look at auroc and auprc under class imbalance. Advances in Neural Information Processing Systems, 37, 44102-44163.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kQeEIW9cVL", "reviewer": "Reviewer_tHAq", "summary": "This paper tackles few-shot whole-slide image classification by arguing that the root cause of overfitting lies in a geometric mismatch between pretrained pathology features and downstream linear classifiers. The authors propose a Manifold Residual block, which introduces a random geometric anchor and a trainable low-rank residual path to preserve manifold structure while reducing model capacity. Experiments across several MIL backbones show accuracy improvements and parameter reductions. The paper positions itself as introducing a geometry-aware inductive bias for few-shot computational pathology.", "questions": "1. Could the authors disentangle the improvement due to parameter reduction from the claimed geometric preservation? A control experiment using an equally sized models would clarify this.\n\n2. Does the MR block help when training data is not extremely limited? This would clarify whether the method primarily acts as a regularizer rather than a geometry-preserving transformation.\n\n3. Please refer to the other weaknesses for additional concerns.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper identifies a real and practically significant issue in computational pathology. The connection between feature geometry and data efficiency is conceptually interesting and relevant to current efforts in adapting large pretrained models for medical imaging.\n\n2. The proposed MR block is lightweight, easy to implement, and compatible with a wide range of MIL backbones. It can be viewed as a structured parameter-efficient adapter.\n\n3. The paper reports consistent accuracy gains across multiple models with substantial parameter reductions.", "weaknesses": "Major: \n\n1. The paper attributes few-shot overfitting to the “destruction” of pretrained feature manifolds by downstream linear layers. This interpretation is not entirely convincing. Linear mappings are expected to reshape representations to achieve class separability, which is the very purpose of a classifier. The observed overfitting could instead result from limited data or excessive model capacity rather than geometric distortion. The causal link between ‘destruction’ and overfitting is not yet well established and could be further clarified with additional controlled experiments.\n\n2. The proposed MR block introduces a fixed random matrix \\(B\\) as a geometric anchor. The t-SNE panel shows a non-trivial disagreement (~14%) in neighborhood structure. From an intuitive perspective, once the input features are multiplied by \\(B\\), much of the pretrained manifold structure and discriminative geometry are likely disrupted. Classifying on \\(XB\\) rather than on the original \\(X\\) would likely reduce performance. Even with sufficient data, a full-rank MIL training setup might not learn to counteract this direct perturbation of pretrained features, let alone a low-rank adaptation like LoRA. In contrast, linear layers transform pretrained features into a task-relevant space in a data-driven manner. Injecting random noise in this way fits more closely with the definition of “destruction” than “preservation.”\n\n3. It is not entirely clear why extreme few-shot WSI classification is a key constraint here. The computational bottleneck typically lies in patch-level feature extraction and pretraining, not in the downstream classifier. Moreover, pretrained slide-level feature extractors such as TITAN already exist, which weakens the motivation for emphasizing few-shot adaptation at the classifier level.\n\nMinor:\n\n1. The finding that pretrained pathology features exhibit low-dimensional manifold structure is broadly consistent with prior work in vision and contrastive representation learning. Classification layers are expected to transform pretrained features into spaces that better align with downstream tasks, which naturally alters the geometry.\n\n2. The reported performance gains may primarily arise from substantial parameter reductions. The current experiments do not separate this effect from the claimed geometric contribution, making it difficult to assess which factor drives the improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V776POuOMD", "reviewer": "Reviewer_k14d", "summary": "The study proposes **Manifold Residual block** to address the issue of overfitting in few-shot whole slide images (WSI) classification. It argues that overfitting not just from data scarcity but a fundamentally geometric problem. Features from pathology foundation models lie on a low-dimensional, nonlinear manifold that linear layers in MIL models distort.", "questions": "1) how does the MR block perform if B is a non-random, fixed matrix, such as an identify matrix?\n\n2) can you confirm if the same geometric distortion problem exists for pathology slide-level foundation models such as TITAN [1]?\n\n3) Given this strong evidence for **r=32**, why were all main experiments (Table 1) and ablation studies (Table 2) run with **r=64**?\n\n4) Could you elaborate on the methodology used to apply **MR** block to TransMIL and CATE?\n\n**Reference**:\n\n1. Ding, T., et al. \"Multimodal whole slide foundation model for pathology (2024).\" URL https://arxiv. org/abs/2411.19666 2411.", "limitations": "", "rating": 10, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The study provides quantitative proof that CONCH features exhibit a low-dimensional manifold with nonlinear curvature, which linear layers disrupt.\n\n2) The study proposes **MR Block Innovation** with a fixed random geometric anchor and a trainable low-rank residual pathway, reducing overfitting and parameter count.\n\n3) The study provides a extensive validation to demonstrates the generalization of the proposed method.\n\n4) **MR Block Innovation** demonstrates SOTA performances on three datasets across 4, 8, and 16 shots settings.", "weaknesses": "1) The study does not provide comparison with SOTA methods for whole slide images classification in few-shot settings such as MGPATH [3], MSCPT [2], FOCUS [3].\n\n2) The study does not report inference time and FLOPs for the proposed method.\n\n3) The study does not fully explain the effective of rank on the model's performance. For example, the sensitivity analysis (Fig. 3) shows that performance saturates around a rank of **r=32**. The authors note this **aligns remarkably** with the features effective rank of 29.7. However, all main experiments in Table 1 and the ablation studies in Table 2 were run with **r=64**. In Fig. 3, NSCLC 8-shots, **r=64** performs worse than **r=32** or **r=48**, suggesting **r=64** may be a suboptimal.\n\n4) The study lacks a clear description of how to apply **MR** block to complex methods such as TransMIL or CATE.\n\n**Reference**:\n\n1. Nguyen, A.-T., Nguyen, D. M. H., Diep, N. T., Nguyen, T. Q., Ho, N., Metsch, J. M., Maurer, M. C., Sonntag, D., Bohnenberger, H., & Hauschild, A.-C. (2025). MGPATH: A vision-language model with multi-granular prompt learning for few-shot whole-slide pathology classification. Transactions on Machine Learning Research (2025).\n\n2. Han, Minghao, et al. \"Mscpt: Few-shot whole slide image classification with multi-scale and context-focused prompt tuning.\" IEEE Transactions on Medical Imaging (2025).\n\n3. Guo, Zhengrui, et al. \"Focus: Knowledge-enhanced adaptive visual compression for few-shot whole slide image classification.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MUIyt1VHVA", "reviewer": "Reviewer_7Ghg", "summary": "This paper identifies a novel cause for overfitting in few-shot WSI classification: the distortion of low-dimensional feature manifolds by standard linear layers . The authors propose a plug-and-play \"Manifold Residual (MR) block\" that replaces these layers, using a fixed random matrix as a \"geometric anchor\" to preserve manifold structure and a separate low-rank pathway for task adaptation", "questions": "- Why was $r=64$ used for the main comparison tables, while it is shown that performance saturates at $r=32$ (Fig. 3)? Does this choice, which doubles the parameters of the LRP, potentially understate the true parameter efficiency and performance of the MR block at its optimal rank? It may be useful to report the $k=16$ results for MR-CATE with $r=32$ in the text to show that the SOTA performance holds at this more theoretically-motivated rank.\n- Is a random matrix the optimal choice for preserving the specific, learned structure of a foundation model's feature manifold? For instance, would a fixed projection based on the principal components (PCA) of the training set features serve as a more \"informed\" (but still fixed) geometric anchor?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is built on a strong, clear, and insightful hypothesis. The diagnosis of overfitting as a geometric problem (i.e., manifold distortion by geometry-agnostic layers) rather than purely a data-scarcity problem is a novel and compelling contribution to the field.\n\n- The core hypothesis is well-supported by quantitative analysis before the method is introduced. The use of spectral analysis to show low effective rank (Fig. 1) and tangent space analysis to demonstrate both the manifold's curvature and its distortion by standard linear layers (Fig. 1) provides a solid and convincing foundation for the proposed solution.", "weaknesses": "- Limited evaluation datasets. While many MIL methods are tested, the number of different types of tasks (classification only) and number of organs (limited to 3) is quite low for demonstrating a robust method improvement. \n- The tasks are also artificial few shot tasks. These tasks (e.g., NSCLC subtyping) have 1000s of data points, but the few shot splits are artificially sampled. I recommend trying some real few shot tasks, such as treatment response prediction. This type of task will always be few shot in nature and helping improve performance in this domain will carry tremendous benefit for the field, which is not true for a rather solved task of RCC and NSCLC subtyping. 10+ treatment response prediction tasks can be found at: https://huggingface.co/datasets/MahmoodLab/Patho-Bench", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)