Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement
Authors: Yanyan Huang
Weiqin Zhao
Yihang Chen
Yu Fu
Lequan Yu
Venue: NeurIPS 2024 poster (2024)
Keywords: Computational Pathology
Whole Slide Image
Vision Language Model
Mutiple Instance Learning
Primary area: machine_learning_for_healthcare

Abstract: Whole slide image (WSI) analysis is gaining prominence within the medical imaging field. Recent advances in pathology foundation models have shown the potential to extract powerful feature representations from WSIs for downstream tasks. However, these foundation models are usually designed for general-purpose pathology image analysis and may not be optimal for specific downstream tasks or cancer types. In this work, we present Concept Anchor-guided Task-specific Feature Enhancement (CATE), an adaptable paradigm that can boost the expressivity and discriminativeness of pathology foundation models for specific downstream tasks. Based on a set of task-specific concepts derived from the pathology vision-language model with expert-designed prompts, we introduce two interconnected modules to dynamically calibrate the generic image features extracted by foundation models for certain tasks or cancer types. Specifically, we design a Concept-guided Information Bottleneck module to enhance task-relevant characteristics by maximizing the mutual information between image features and concept anchors while suppressing superfluous information. Moreover, a Concept-Feature Interference module is proposed to utilize the similarity between calibrated features and concept anchors to further generate discriminative task-specific features. The extensive experiments on public WSI datasets demonstrate that CATE significantly enhances the performance and generalizability of MIL models. Additionally, heatmap and umap visualization results also reveal the effectiveness and interpretability of CATE.

Free Lunch in Pathology Foundation Model: Task-specific Model Adaptation with Concept-Guided Feature Enhancement
Yanyan Huang
Weiqin Zhao The University of Hong Kong
Yihang Chen
The University of Hong Kong yanyanh@connect. hku hk
The University of Hong Kong yihangc0connect hku. hk
wgphaa988connecc. hku. hk
Yu Fu
Lequan Yu"
Lanzhou University fuyu@lzu. edu.
The University of Hong Kong lqyu@hku. hk
Abstract
Whole slide image (WSI) analysis is gaining prominence within the medical imag- ing field. Recent advances in pathology foundation models have shown the potential to extract powerful feature representations from WSIs for downstream tasks. How- ever, these foundation models are usually designed for general- purposs pathology image analysis and may not be optimal for specific downstream tasks or cancer types. In this work, we present Concept Anchor-guided Task-specific Feature Enhancement (CATE), an adaptable paradigm that can boost the expressivity and discriminativeness of pathology foundation models for specific downstream tasks. Based on set of task-specific concepts derived from the pathology rision-languse model with expert-designed prompts, we introduce two interconnected modules to dynamically calibrate the generic image features extracted by foundation models for certain tasks or cancer types. Specifically, we design a Concept-guided Informa- tion Bottleneck module to enhance task-relevant characteristics by maximizing the mutual information between image features and concept anchors while suppressing superfluous information. Moreover, Concept-Feature Interference module is proposed to utilize the similarity between calibrated features and concept anchors further generate discriminative task-specific features. The extensive experiments public WSI datasets demonstrate that CATE significantly enhances the per- formance and generalizability of MIL models. Additionally, heatmap and umap visualization results also reveal the effectiveness and interpretability of CATE, The source code is available hntps:://gthhb..om/HHHff wttpr//tw...oooe/KK-eeee////CCCA
Introduction
Multiple Instance Learning (MIL) 26 34 23
is widely adopted for weakly supervised analysis in of patch features generated
computational pathology, where the input of MIL is typically pre-trained feature extractor (i.e., image encoder). Although promising progress has been achieved, the effectiveness of MIL models heavily relies on the quality of the extracted features. A robust feature extractor can discern more distinctive pathological features, thereby improving the predictive capabilities of MIL models. Recently, several studies have explored using pretrained foundation models on large-scale pathology datasets with self-supervised learning as the feature extractors for WSI analysis 37 7 36). Additionally, drawing inspiration from the success of Contrastive
Cooresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
(b)
reslvvelevant Information
Feature Extractor
Necrosis
Distribution:
Fibrosis
Site
CATE
Tesk-refevant Information
relernnn Information
Site
Task-specific Concepts
Figure (a) Illustration of the key idea of concept-guided information bottleneck to enhance the task-relevant information and discard the iisk-irrelevant information. (b) Task-specific model adaptation with CATE to enhance the generalization across different data sources.
Language- Eagguagg--mageeaaee Pretraining (CLIP) 30 20 in bridging visual and linguistic modalities, works have aimed to develop a pathology vision- lasion-annguae foundation model (VLM) to simultaneously learn representations of pathology images and their corresponding captions [15 25 The intrinsic consistency between the image feature space and caption embedding space in the pathology VLM enables the image encoder to extract more meaningful and discriminative features for downstream WSI analysis applications 25
Although the development of these pathology foundation models has significantly advanced com- putational pathology, these models are designed for general-purpose pathology image analysis and may not be optimal for specific downstream tasks or cancer types, as the features extracted by the image encoder may contain generic yet task-irrelevant information that will harm the performance of specific downstream tasks. For example, as illustrated in Figure [a), the features extracted by the image encoder of a pathology VLM can include both task-relevan information (e.g., arrangement morphology of tumor cells) and task iireeereelevntt clements(such as background information, stain styles, etc.). The latter information may act as "noise", distracting the learning process of MIL models tailored specific tasks, and potentially impairing the generalization performance of these models across different data sources. Consequently, it is crucial to undertake task-specific adaptation to enhance feature extraction of generic foundation models and enable MIL models concentrate on lask-relevant information and thus improve analysis performance and generalization [3] 38).
In this paper, we propose novel paradigm, named Concept Anchor-guided Task-specific Feature Enhancement (CATE), to enhance the generic features extracted by the pathology VLM for specific downstream tasks (e.g., cancer subtyping). Without requiring additional supervision significant computational resources, CATE offers an approximately "free lunch" " in the context of pathology VIM. Specifically, we first derive set of task-specific concept anchors from the pathology VLMM with task-specific prompts, and these prompts rely on human expert design or are generated through querying large language models (LLMs), necessitating certain level of pathological background knowledge. Based on these concept anchors, we design two concept-driven modules, i.e., the Concept- guided Information Bottleneck (CIB) module and the Concept- Feature Interference (CFI) module, to calibrate and generate task-specific features for downstream analysis. Particularly, with the task- specific concepts as the guidance, the CIB module enhances task-relevant features by maximizing the mutual information between the image features and the concept anchors and also eliminates iusk-irrelevant information by minimizing the superfluous information, shown in Figure [T). Moreover, the CFI module further generates discriminative task-specifi features by utilizing the similarities between the calibrated image features and concept anchors (i.e., concept scores). By incorporating the CATE into existing MIL. frameworks, we not only obtain more discriminative features but also improve generalization regarding domain shift by eliminating task-irrelevant features and concentrating on pertinent information, shown in Figure bb)
In summary, the main contributions of this work are threefold:
We introduce novel method, named CATE, for model adaptation in computational pathology. To the best of our knowledge, this the first initiative to conduct task-specific feature enhancement based on the pathology foundation model for MIL tasks.
We design new CIB module to enhance the task-relevant information and discard irrelevant information with the guidance of task-specific concepts, and a new CFI module to generate task-specific features by exploiting the similarities between image features and concept anchors.

---

Extensive experiments on Whole Slide Image (WSI) analysis tasks demonstrate that CATE significantly enhances the performance and generalization capabilities of MIL models.
2 Related Work
Multiple Instance Learning (MIL) for WSI Analysis. MIL is the predominant paradigm for WSI analysis, treating each WSI as a bag of patch instances and classifying the entire WSI based on aggregated patch-level features. Attention-basee their ability to determine the significance of each instance within the bag. For instance, llse el al. [[6]
methods [T6 26 17 39 are highly regarded for
cestering-constraingg attention to refine this mechanism further. To model the relationships among instances, graph-based and Transformer-based methods have been developed [TO] 13 For example, Chen et al.
introduced an attention-based MIL model, while Lu al. [26] proposed
introduced Transformer-bast hierarchical network to capitalize on the inherent hierarchical structure of WSIs.
Pathology Foundation Model. With the advancement of foundation models in computer vision, several pathology foundation models have been developed to serve as robust image encoders for WSI analysis. Riasatian et al. [31] proposed fine-tuning the DenseNet [12] on the TCGA dataset, while Filiot et al. 7 utilized iBOT 42 to pretrain vision Transformer using the Masked Image Modeling framework. Recently, Chen el al. [3] pre-trained a general-purpose foundation model on large-scale pathology datasets using DINOv2 (88]. which has demonstrated strong and readily usable representations for WSI analysís. Inspired by CLIP 30), Ikezogwo et al. [15), Huang et al. [14], and Lu et al. (25) developed vision-language foundation models by training on large-scale pathology datasets with image-caption pairs. These foundation models have demonstrated superior performance in downstream tasks due to their ability to extract more discriminative features for WSI analysis.
Feature Enhancement in Computational Pathology. Several methods have been developed to obtain more discriminative features for WSI analysis by adapting pathology foundation models [4] 24 or designing new plug-and-play modules (35]. For instance, Zhang al. 44] suggested aligning the image features with text features extracted from pre-trained natural language model to enhance the feature representation of WSI patch images, while it operates solely at the patch level, without considering the informational relationship between image and text features. Recently. Tang et al. [35] introduced Re-embedded Regional Transformer for feature re-embedding, aimed enhancing WSI analysis when integrated with existing MII. methods. However, while this method considers the spatial information of WSIs and adds flexibility to MII. models, it falls short in extracting task-specific discriminative information for WSI analysis.
Method
3.1 Overview
The proposed CATE can be seamlessly integrated with any MIL framework to adapt the existing pathology foundation model (Pathology VLM) for verformance-impred specific enhancement. Specifically, consider a training set D = {(x. y)} of WSI-label pairs, where x [m1, 22 ..., xvv } is set of patch features with dimension of C' (i.e., 2; R extracted by the image encoder of pathology VLM, N denotes the number of patches, and y is the corresponding label. The objective of CATE is obtain the corresponding enhanced task-specific feature set from the original feature x with the guidance pre-extracted concepts anchors e (see description below) for downstream usage:
WSI analysis via task-
z CATE (x, ,), = MIL (2).
As illustrated in Figure 2 we design two different modules to enhance the extracted features from foundation models: (1) concept-guided Information Bottleneck (CIB) module calibrates original image features with the guidance of concept anchors with information bottleneck principle; and (2) Concept- Interference (CFI) module generates discriminative task-specific features by leveraging the similarities between the calibrated image features and concept anchors. Specifically, the enhanced patch features can be represented as z = {21, 22, NN), where 2, the concatenation of the calibrated feature or, and the interference feature B, generated by CIB and CFI module:
2, Concat [ax, [,] Concat [CIB (æt, c). CFI (x,, c)]
(2)
3

---

(a) Overview
CATE
Enhanced Features
Image of
CIB
Image Encoder
MIL Model
Text x Encoder
CFI
(z,)1
Original
1V
(c) CIB
(d) CFI
H(a
Feature Feature Entropy
vBB image of
MLPP
M.P d
H(a)
Encoded Feature Entropy
B
Hic)
Concept.Archoc Entropy
A SNN
H(y)
Label Information
vmage Kf
(a(a)a))
líck
:
I(áe c)
Predictive Information
[(:: d/c)
Colibrated
Concept
Concept
Interference
02
I( Superfluous Information
Feature
Anchors
Vector
Figure 2: (a) Overview of CATE: the outputs of the CIB and CFI modules are concatenated to form the enhanced feature for downstream MIL models. (b) Task-relevant concept generation. (c) Concept- guided Information Bottleneck (CIB) module. (c) Concept- Feature Interference (CFI) module.
Concept Extraction. We extract two kinds of task-specific concept anchors, c fce c( com- prising speciis concepts c [c]t N (e.g., subtyping classes) and class-agnostic concepts c = = [c 1== (e.g., adipose, connective. and normal tissues), with m and representing the num- bers of class-specific and class-agnostic concepts, respectively. These concepts are generated by the text encoder of pathology VI.M with prompt p. Each prompt consists of a class name (e.g.. "invasive ductal carcinoma") and template (e.g., "An image of <CIASSNAME>>)) To obtain more robust concepts, use multiple prompts for each class and the final concept anchor is the average of the embeddings generated by different prompts. Details of class names and templates for various tasks provided in Appendix G Note that due to the inherent consistency between the image and text embedding space in VLM, these extracted concepts can also be regarded as image concept vectors.
3.2
Concept-guided Information Bottleneck
The objective of this module is to find distribution p(a{(x) that maps the original image feature into a representation a, which contains enhanced fask-discriminative characteristics and suppressed task-irrelevant information. WSIs typically contain various cell types or tissues (e.g., tumor cells, normal cells, adipose tissue, connective tissue), while only a subset of patches (e.g., with tumor cells) is crucial for certain tasks such tumor subtyping. We thus define x { S as the representative subset of the original feature set (e.g., tumor tissue patches), where & denotes the number of representative patches. Note that this selection of image features with class-specifee concepts (see discussion below). To this end, the corresponding enhanced feature set is á {á, R C and we want to find the conditional distribution pl ( [ to map the selected patch feature à; E RC into a more discriminative enhanced feature á which is discriminative enough to identify the label y.
be conducted with simple comparison
Sufficiency and Consistency Requirements. To quantify the informativeness requirement of the calibrated fcature à we consider the sufficiency of & for As defined in Appendix [E.] the encoded feature & derived from the original feature & is sufficient for determining the label if and only if the amount of task-specific information remains unchanged after calibration, i.e., 1 ( y) = I [(ā;)) However, the label y pertains the slide level and specific labels cannot be assigned each instance due to the absence of patch-level annotations.
To address this challenge, we propose using the task-specific concept anchor as the guidance for each single & Specifically, we posit that the concept anchor e distinguishable for the task and contains task-relevant information for label y. Given the consistency between image and text features in pathology VI.M, any representation â containing information accessible from both image feature E and concept e will also encapsulate the discriminative information required for the

---

label. This consistency requirement is detailed in Appendix E.1 Thus, if & is sufficient for c (i.e., [ c à 0), then is as predictive for label y as the joint of original feature and concept anchor c. Applying the chain rule of mutual information, we derive:
[ &c)
[(ācc)
Predietive Information
Superituous Information
According the consistency between the concept anchor and original feature, the mutual information term / ((x; c) represents the predictive information for the task, while the conditional information term I (6;; ò (c)c) denotes task-irrelevant information (i.e., superfluous information) in original patch feature à, which can be minimized to enhance the robustness and generalization ability of downstream MIL models. As result, the main objective of the feature calibration in this module can be formalized as maximize predictive information I ( c) while minimize the superfluous information [ â(c)).
Predictive Information Maximization (PIM). The predictive information in Equ (3) equals the mutual information between the calibrated feature and concept anchors. To maximize this, we choose the InfoNCE [27] to estimate the lower bound of the mutual information, which can be obtained by comparing positive pairs sampled from the joint distribution å, cpus p(ā, ct)) to pairs d, cjj p (c and class-agnostic concepts
and x, egf built using a set of negative class concepts cf 65 (c
k Σ
f f(c as)) Σラ=1 I (cf,, σ.)
INCE (ár;c)=
E Óe cpes
log
post
+ -z= I (C
We set f (c,, āc) exp (atec/) with 7 > O in practice following 27 By maximizing this mutual information lower bound, / (c, (z;) will be proportional to the density ratio p (c, (x)/p(c) (ás) as proved in (27). Hence, f (c, ox)) preserves the mutual information between the calibrated feature and concept anchor. The detailed derivation can be found in Appendix E.2 can be denoted as:
The loss function for PIM
) (5)
Es J
à oj
LPIM = E
pos
E Ce
Σ log
exp
exp
Superftuous Information Minimization (SIM). To compress task- information, we aim to minimize the superfluous information term as defined in Equ (3. This objective can be achieved by minimizing the mutual information [(æ; o). In practice, conduct SIM for all patches in the subset x, as each patch may contain insk-iirelevant information. Following [. it can be represented as:
T(x: a.) = f p (zx, ox) log (a(z)) dæ:do= - f (ax) log (ax) da.
After that, we let the distribution of a: r ((x) (e.g., Gaussian distribution in this work), be variational approximation the marginal distribution p (ax,, and we can obtain the upper bound for / (æ:; a):
T(x; a:) S / (ax) p (a(l)) log
p p(x(a) (x)
dado..
r (a))
Furthermore, use a variational distribution go (a((z) with parameter 0 to approximate (a()z) and we implement the parameterization of the variational distribution with MLP by predicting the mean and variance of the Gaussian distribution and sample the calibrated feature from this distribution:
ar ~ N (MIPH (æ),MLPP MLP (a:)
(8)
In practice, we implement this by utilizing the reparameterization trick (18) to obtain an unbiased estimate of the gradient and further optimize the variational distribution. The detailed derivation can be found in Appendix E.3 The minimization of the upper bound of I(æ; a equals to the minimization of the Kullback-Leible divergence between go (a((z) and r (ax). Therefore, the loss function can be represented as:
k
DeL (go (arl2:) (a:la)))) (I(())))
CSTM =
5

---

Discussion. We further provide explanation of CIB module with the information plane 8 6 in Appendix It should be noted that the PIM supervises only the representative subset x containing task-relevant information (selected by the similarity between image features and corresponding class-specific concepts). Meanwhile, the SIM applied to all patches in x, as any patch may earry information irrelevant to the task (e.g., background information and stain styles). Besides, SIM cannot be directly optimized without the guidance of concept anchors (i.e., PIM) due to the absence of patch-level labels. As demonstrated in the ablation study in Section 4.44 the absence of concept anchor guidance leads to the collapse of discriminative information the calibrated feature, adversely affecting downstream task performance. By maximizing predictive information and minimizing superfluous details, the CIB module effectively enhances the discriminative capacity of the original features and aligns them with the task task-specific concept anchors for improved prediction.
3.3 Concept-Feature Interference
We also propose the Concept- Interference (CFI) module utilize the similarity characteristic between calibrated features and concept anchors to further obtain robust and discriminative infor- mation for the downstream tasks. Our primary focus on the class-specific concept anchors c Specifically, for each CIB encoded feature ag we calculate the cosine similarity between a and each class-specific concept cft It important to note that the number of class-specific concepts m is larger than the number of classes, as we use multiple <CLASSNAME> and templates to generate the concept anchor for each class, as shown in the Appendix G Thus, we can obtain the similarity vector by concatenating the similarity scores between a and each class-specific concept ct*. To integrate the interference information (similarity relationship) into the enhanced feature, we align the similarity vector with the calibrated feature a, using This allows us to obtain the final interference vector Bg of CFI:
Network (SNN) layer [99
B. = SNN (Concat ({Sim (oa; +)))
(10)
The interference vector contains superficial information that indicates the similarity between the calibrated feature and concept anchor directly. This is completely different from the calibrated feature of the CIB module, which contains discriminative latent information for the downstream tasks. Therefore, integrating the interference feature can further provide robust and discriminative information for the downstream tasks.
Discussion. The CFI module is designed to utilize the similarity characteristic between calibrated feature and concept anchor discriminative feature, which be further integrated into the calibrated feature for downstream tasks. This is different from other studies that directly compare the similarity between visual features and textual concept features of different classes perform zero-shot classification 25
3.4 Training Objective
The overall training objective of the CATE framework can be represented as the combination of the cross entropy loss Lce for the downstream tasks, the predictive information maximization loss LPIM, and the superftuous information minimization loss LSTMi
= Lae + XpLPrM + ASLSTM,
(11)
where Ap and Äs are hyperparameters and influence of them is discussed in Appendix
Experiments
4.1 Experimental Settings
Tasks and Datasets. We conducted cancer subtyping tasks on three public WSI datasets from The Cancer Genome Atlas (TCGA) project: Invasive Breast Carcinoma (BRCA), Non-Small Cell Lung Cancer (NSCLC), and Renal Cell Carcinoma (RCC). Detailed dataset information is available in
Appendix D

---

Table 1: Cancer Subtyping Results on BRCA of MIL Models Incorporated with CATE.
BRCA (Nin=1))
Method
CATE
OOD-AUC
Gain
OOD-ACC
Gain
IND-AUC
Gain
IND-ACC
Gain
ABMIL
X
0.914
NIA
U.852++00014
N/A
0.963=0.044
N/A
0
NIA
CLAM
0.907
0.802400053
N/A
0,965-0:049
0.888=0.088
DSMIL
X
0.925
NIA
0.836:200048
N/A
0.969=0:040
N/A
0.900=00080
N/A
DIFD-MIL
X
0.912:10012
NIA
0.858:100000
NIA
0.044
N/A
0.894=0.070
N/A
TransMLL
X
0,918
NIA
0.832::0.066
N/A
0.969
N/A
0,918 0.067
N/A
R R'T-MI.''
X
200027
N/A
0.816220.051
N/A
0.965=0.035
N/A
0.894=0:022
N/A
ABMIL
0.901
14.05%
0.897
15.28%
0.998
13663%
0.965
18.67%
CLAM
0.951
14.85%
0.906
112.97%
0.998_0.006
13.42%
0.968_0.037
18.67%
0.936+:00007
11.19%
0.8664:00036
13.59%
0.990-0:022
12.17%
0.959-0:044
16.56%
DIFD-MII.
0.947
13.84%
0.906100000
15.59%
0.985-0.028
14.34%
0.953=0.042
76.60%
TransMIL
0.938
12.18%
0. 0.800
15.77%
0.998 0.006
+2.99%
15.1
BRCA (NiND=
Method
CATE
OOD-AUC
Gain
OOD-ACC
Gain
IND-AUC
Gain
IND-ACC
Gain
ABMIL
X
0.899:00035
NIA
0.8922:0019
NIA
0.967=0.019
N/A
0.941=00024
N/A
CLAM
X
0.893
NIA
0.8022100019
N/A
0.960-0.042
N/A
0.935-0.027
N/A
DSMIL
0.881 0:0002
NIA
0.852:10.005
N/A
0.946=0.057
0,940-0.08
NIA
DTFD MIL
X
0.909+
NIA
N/A
0.923
N/A
0.945-0041
N/A
TransMIL
X
0.904-10.023
NIA
0.852.10.000
NIA
0.966-0.031
N/A
0.936-00052
N/A
R'T-MLL
0.902++00028
0.803-4-0027
N/A
0.906-00060
0.929
ABMIL
0.003
14.89%
0.007110.018
+1.68%
0.081=00018
+1.45%
0.948
10.74%
CLAM
0.945:00006
15.82%
0.896200000
13.94%
0.976-0.023
11677%
0.938-0.048
10.32%
DSMIL
0.919
14.31%
0.000
12.00%
0.958
11.27%
0.949_0.024
10.96%
DIFD-MIL
0.946 0:000
14.07%
0.887
+1.03%
0,977
10.41%
0,946
10.11%
TranMIL
0.920+00011
11.77%
4-0066
11.76%
0.968 0.045
10.211%
0.940-0:026
10.43%
The best results are highlighted bold, and the second-best results undertined.
R.T.MI.. is designed for feature re-emhedding that utilize ABMII as hase MII model.
IND and OOD Settings. The datasets in the TCGA contains samples from different source sites (i.e., different hospitals or laboratories), which are indicated in the sample barcodes" And different source sites have different staining protocols and imaging characteristics, causing feature domain shifts between different sites [ Therefore, MIL models trained on several sites may not generalize well others. To better evaluate the true performance of the models, we selected several sites IND) data (in-domain, the testing and training data are from the same sites), and used data from other sites as OOD data (oum-of-domain, the testing and training data are from different sites), and reported the testing performance on both IND and OOD data. Specifically, we designated NIND sites as IND and the remaining as OOD. Each experiment involved splitting the IND data into training, validation, and testing sets, training the models on IND data, and evaluating them on both IND and OOD testing data. For the BRCA dataset, we randomly selected one or two sites as IND data and used the remaining sites as OOD data. However, for NSCLC (2 categories) and RCC (3 categories) datasets, each site contains samples from only one subtype. Therefore. we cannot select only one site as IND data, as it will include one category/subtype in the training data. Instead, we randomly selected one or two corresponding sites for each category as IND data for NSCLC and RCC, and used the other sites as OOD data. Finally, we obtained or 2 IND sites for BRCA, 2 or 4 for NSCLC, and 3 or 6 for RCC.
Evaluation. We report the area under the receiver operating characteristic curve (AUC) and accuracy for the OOD and IND test sets, respectively, with means and standard deviations over 10 runs of Monte-Carlo Cross Validation. Notably, the OOD performance emphasized for NSCLC and RCC, where each site contains samples from only one cancer subtype. Traditional MIL models tend to recognize site-specific patterns (e.g., staining) as shortcuts and excel in-domain evaluations, rather than identifying useful class-specific features, making performance less reflective of the models' actual capability. Therefore, OOD performance more accurately reflects the models' discriminative and generalization capabilities.
Comparisons. Given that CATE is an adaptable method, we evaluated the performance variations across various MIL models both with and without the integration of CATE. We specifically focused on the following -tate-of-the-ar MIL models: the original ABMIL [66, CLAM (26), DSMIL (21).
http:///loogggg..eeeeeeeeeoooeor Barcode/
7

---

Table 2: Cancer Subtyping Results on NSCLC and RCC.
NSCLC (NNn==2)
NSCLC (Nisn=1)
Method
OOD-AUC
OOD-ACC
IND-AUCC
IND-ACC
OOD-AUC
OOD-ACC
IND-AUC
IND-ACC
ABMIL
0.874=0.021
0.803-20001
0.997::00004
0.954:20.028
0.951
0.883=0.029
0.974+0018
0.910=0036
CLAM
0.87520.000
0.801::0021
0.997:30007
0.963200042
0.931
0.870=0.0%
0.977:0023
0.926120048
DSMIL
0.839=00046
0.764+00043
0,993-4:0004
0.963-40.028
0.934
0.864=0.026
0.974+00113
0.913++00042
DTFD MII.
0.903-00:025
0.8
0.990=:0.009
0.958:20.049
0.949110000
0.893-0.012
0.981=0012
0.918
TransMIL
0.790=0.028
0.712:100004
0,997::0.004
0.954:000033
0.917:100022
0.832=0.031
0.977=10014
0.923
R² T-MIL
0.739
0.690:10.075
0,999110.002
0.971:10.036
0.892:10.041
0.800
0.977:1:0018
0.9161:0045
CATE-MIL
0.945
0,985 0.840
0.938+00007
0.069
0.906
0.967440.19
0,905
RCC (Nisu=3)
RCC (NND=6)
Method
OOD-AUC
OOD-ACC
IND-AUC
IND-ACC
OOD-AUC
OOD-ACC
IND-AUC
IND-ACC
ABMIL
0.973=00005
0.891:00017
0,9971:0004
0.901
0:000
0.885=0.010
0.973:0010
0,897
CLAM
0.972:a0.004
0.893:100012
0.000
0.961:10.032
0.909:10.009
0.975 0.015
0.896:10.011
DSMIL
0.977=00.002
0.893++0.010
0,996+50.006
0.965
0.969-40000
0.883-00016
0,980++0.112
0,901 +00022
DTFD MII.
0.975=00.005
0.897:20.012
0.996:200004
0.943-00.46
0:971
0.893=0.0117
0.974
0.878
TransMIL
0.901
0.8641:0:022
0.994::0.04
0.930:20.030
0:947110017
0.828=0.007
0.975::0013
0.894100007
R²T T-L
0.956=0.018
0.847:40.022
0,991
0.936#00000
0.9
0.803=0.048
0.974#00012
0.897
CATE-MIL
0.983
0.911
0.989+00009
0.944+400031
0.979
0.905
0.963440001
0.882
The best results are highlighted in bold, and the second-best results are underlined.
R designed for feature embedding that utilize ABMIL hase model.
The in-domain performance of NSCLC and RCC does not represent the true ability of the models, each site contains only samples from one cancer subtype. We primurily focus on the OOD performance for these two datasets.
TransMIL B3 DTFD-MIL [40]. and R (35. The R 355 is a feature re-embedding method that utilizes ABMIL as the base MIL model.
Implementation Details. This study begins the image feature extraction process by segmenting the foreground tissue and then splitting the WSI into 512x512 pixels patches at 20x magnification. Subsequently, these patches processed through pre-trained image encoder from CONCH (25] to extract image features. For concept anchors, we utilize CONCH's text encoder to derive task-relevant concepts from predefined text prompts, with detailed prompt information available in Appendix Model parameters are optimized using the Adam optimizer with a learning rate of 10-5. The batch size is set to 1, and all the experiments are conducted on single NVIDIA RTX 3090 GPU.
4.2 Experimental Results
Quantitative Results on BRCA Dataset. To fully evaluate the effectiveness of CATE, we assessed its impact on several ofate-of-the-art MIL models using the BRCA dataset. The results are shown in Table [ where the MIL models integrated with CATE outperform their original counterparts in both in-domain (IND) and out-of-domain (OOD) testing, which demonstrates the effectiveness and generalization capabilities of CATE. Comparing with R'T-MIL, which is a feature re-embedding method that utilizes ABMIL as the base MIL model, CATE incorporated with ABMIL consistently achieves better performance in terms of both OOD and IND testing. To further investigate the effectiveness of CATE, conducted experiments by altering the in-domain sites and applying traditional settings. Detailed results are available in Appendix B
Qualitative Analysis. To qualitatively investigate the effectiveness of CATE, we visualized attention heatmaps, UMAP, and the similarities between original features and corresponding class concept features, as well calibrated features and class concept features, as shown in Figure 3 Additional visualization results are provided in Appendix H As shown in Figure 3 (a&b), attention heatmap comparisons reveal that CATE-MIL focuses more intensely on cancerous regions, with a clearer delineation between high and low attention areas. By comparing the similarities of original and calibrated features class concept features in Figure33 in cancerous regions is significantly higher than in original features. Moreover, the disparity between cancerous and non-cancerous regions' similarities is also expanded, which further verifies the ability of CATE to enhance task-relevant information and suppress irrelevant information. We further performed a UMAP visualization of class concept features, original features, and calibrated features. As depicted in Figure (f), calibrated features are notably closer to the corresponding class (IDC)
it is evident that the enhanced similarity
8

---

concept features compared to the original features, which demonstrates CATE's ability to effectively align features with fask-relevant concepts and enhance lask-relevant information.
(a) CATE-MIL Heatmap
(c) Similarity after CATE
9.24
(e) Original WSI
TCGA-3C-ALL- AAL--10-11-000 1010-00M
-2.93
(b) ABMIL Heatmap
(d) Original Similarity
0.47
(f)
UMAP Visualization
1DC Concupt Enhanced Featrne
Festure
0.10
Figure 3: (a) Attention heatmap of CATE-MIL. (b) Attention heatmap of the original ABMIL. (c) similarity between the calibrated features and the corresponding class concept feature. (d) similarity between the original features and the corresponding class concept feature. (e) Original WSL. (f)
UMAP visualization of class concept features, original features, and enhanced features.
4.3 Results on Additional Datasets
For clarity and to highlight the superiority ABMIL when enhanced with CATE, we developed CATE-MIL by incorporating CATE into ABMIL and compared it against other leading MIL models on NSCLC and RCC datasets. The comparative results caniie that CATE-MIL. consistently outperforms other models in both OOD and IND) performance. However, it is noted that CATE-MIL performs poorly on the in-domain testing data for NSCLC and RCC. This underperformance may be attributed to the elimination of task-irrelevant information, including site-specific patterns, by CATE, potentially degrading performance on in-domain data for these datasets. Consequently, OOD performance more accurately reflects the discriminative and generalization capabilities of the models.
4.4 Ablation Analysis
We conduct ablation studies to assess the effectiveness of each component within CATE, and the results are shown in Table 3. Initially, incorporating Predictive Information Maximization (PIM) enables ABMIL to achieve improved performance in most experiments, which demonstrates PIM's efficacy in extracting task-relevant information. However, using Superfluous Information Minimization (SIM) alone results in performance degradation across most experiments, which suggests that SIM may discard some task-relevant information without guidance from task-relevant concept anchor. Incorporating both PIM and SIM consistently enhances ABMIL's performance in all experiments, which further verifies that their combination effectively boosts the generalization capabilities of MIL models. We also conduct experiments by only using the interference features in CFI the input of ABMIL, and the results show that the interference features are also informative for WSI classification tasks. More ablation analysis about the weights of PIM and SIM in CIB module the number of representative patches are in Appendix C
5 Conclusion and Discussion
In this paper, we introduce CATE, new approach that offers "free lunch" for task-specific adapta- tion of pathology VLM by leveraging the inherent consistency between image and text modalities.

---

Table 3: Ablation study of CATE.
BRCA (Nisp=1)
BRCA (NisD=2)
NSCLC (NisD=2)
NSCLC
RCC
Method
PIM
SIM
CFI
(Ninn=4)
(NisD=3)
(NinD=6)
ARMII.
0.914200015
0.87420.021 0.899=60035
0,951200025
0.973=0:005
0.970
0.932:10011
0.939=0.012
0.906
0,901
0.976=0.005
0,973:20.00
0.895-60.022
0.885=0.190
0.656:60.041
0.952=0.014
0,954++0.015
V
0.936=20.010
0,942=0.010
0.910=0.030
0.960=0.011
0.979-0:014
0,977=+0.005
0.913:0.024
0.884=0.032
0.850=0000%
0.918=0:029
0,975=0:008
0.941
CATE-MIL
0.951=00005
0.943=00000
0.945=0.016
0.970
0.983
0.979
CATE shows the potential to enhance the generic features extracted by pathology VLM for specific downstream tasks, using task-specific concept anchors as guidance. The proposed CIB module calibrates the image features by enhancing task-relevant information while suppressing task-irrelevant information, while the CFI module obtains the interference vector for each patch to generate discrimi- native task-specific features. Extensive experiments on WSI datasets demonstrate the effectiveness of CATE in improving the performance and generalizability of state-of-the-ur MIL methods.
Limitations and Social Impact. The proposed CATE offers a promising solution to customize the pathology VLM for specific tasks, significantly improving the performance and applicability of MIL methods in WSI analysis. However, the performance of CATE heavily depends on the quality of the concept anchors, which, in turn, relies on domain knowledge and the quality of the pre-trained pathology VLM. Additionally, while CATE is optimized for classification tasks such as cancer subtyping, it may not be readily applicable to other analytical tasks, such survival prediction. However, there might be a potential solution to address this challenge. For instance, we could leverage LLMs or retrieval-based LLMs to generate descriptive prompts about the general morphological appearance of WSIs for specific cancer types. By asking targeted questions, we summarize reliable and general morphological descriptions associated with different survival outcomes or biomarker expressions and further verify these prompts with pathologists. Moreover, since medical data may contain sensitive information, ensuring the privacy and security of such data is crucial.
6 Acknowledgements
This work was supported in part by the Research Grants Council Hong Kong (27206123 and T45-401/22-N), in part by the Hong Kong Innovation and Technology Fund (ITS/274/22), in part by the National Natural Science Foundation of China (No. 62201483), and in part by Guangdong Natural Science Fund (No. 2024A1515011875..
References
[1] Alexander Alemi, Ian Fischer, Joshua V Dillon, and Kevin Murphy. Deep variational information bottleneck. arXiv preprint arXiv:1612.0044 2016.
[2] Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 161 14-11155,,, 2022.
[3] Richard J Chen, Tong Ding, Ming Y Lu, Drew FK Williamson, Guillaume Jaume, Andrew H Song, Bowen Chen, Andrew Zhang, Daniel Shao, Muhammad Shaban, et al. Towards a general- purpose foundation model for computational pathology. Nature Medicine, 30(3):850-88) 2024.
[4] Shenghua Cheng, Sibo Liu, Jingya Yu, Gong Rao, Yuwei Xiao, Wei Han, Wenjie Zhu, Xiaohua Lv, Ning Li, Jing Cai, et al. Robust whole slide image analysis for cervical cancer screening using deep learning. Nature communications, 12(1):5639, 2021.
[5] Kevin Haan, Yijie Zhang, Jonathan E Zuckerman, Tairan Liu, Anthony Sisk, Miguel FP Diaz, Kuang- Yu Jen, Alexander Nobori, Sofia Liou, Sarah Zhang, et al. Deep learning-base based
10

---

transformation of h&ee stained tissues into special stains. Nature communications, 12(1):1-13, 2021.
[6] Marco Federici, Anjan Dutta, Patrick Forré, Nate Kushman, and Zeynep Akata. Learning robust representations via multi view information bottleneck. arXiv preprint arXhv:2002.07017 2020.
[7] Alexandre Filiot, Ridouane Ghermi, Antoine Olivier, Paul Jacob, Lucas Fidon, Alice Mac Kain, Charlie Saillard, and Jean- Baptiste Schiratti. Scaling self- self-super learning for histopathology with masked image modeling. medRxiv, pages 2023-07, 2023.
[8] Bernhard C Geiger. On information plane analyses of neural network classifiers a review. IEEE Transactions on Neural Networks and Learning Systems, 33(12):703977051 2021.
[9] R. Devon Hjelm, Alex Fedorov, Samuel Marchilaa,,,,,,,,, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net, 2019.
[10] Wentai Hou, Lequan Yu, Chengxuan Lin, Helong Huang, Rongshan Yu, Jing Qin, and Liansheng Wang. H' 2-mil: exploring hierarchical representation with heterogeneous multiple instance learning for whole slide image analysis. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 933-941, 2022.
[11] Frederick M Howard, James Dolezal, Sara Kochanny. Jefiee Schulte, Heather Chen, Lara Heij, Dezheng Huo, Rita Nanda, Olufunmilayo Olopade, Jakob Kather, et al. The impact of site-specific digital histology signatures deep learning model accuracy and bias. Nature communications, 12(1))4423, 2021.
[12] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700-4708, 2017.
[13] Yanyan Huang, Weiqin Zhao, Shujun Wang, Yu Fu, Yuming Jiang, and Lequan Yu. Conslide: Asynchronous hierarchical interaction transformer with breakup-reorganizz rehearsal for contin- ual whole slide image analysis. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 21349-22 -13600 2023.
[14] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual visual-language foundation model for pathology image analysis using medical twitter. Nature medicine, (9(9):2307-2316 2023.
[15] Wisdom Ikezogwo, Saygin Seyfioglu, Fatemeh Ghezloo, Dylan Geva, Fatwir Sheikh Mo- hammed, Pavan Kumar Anand, Ranjay Krishna, and Linda Shapiro. Quilt-Im: One million image-text pairs for histopathology. Advances in Neural Information Processing Systems, 36, 2024.
[16] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention based deep multiple instance learning. In International conference on machine learning, pages 2127-21 2136. PMLR, 2018.
[17] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Weiner, Limin Yu, and Aa- ditya Prakash. Additive mil: Intrinsically interpretable multiple instance learning for pathology. Advances in Neural Information Processing Systems, 55.20689-207022 2022.
[18) Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 13/2.61 14, 2013.
[19] Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. Advances in neural information processing systems, 30, 2017.
[20] Zhengfeng Lai, Zhuoheng Li, Luca Cerny Oliveira, Joohi Chauhan, Brittany N Dugger, and Chen-Nee Chuah. Clipath: Fine-tune clip with visual feature fusion for pathology image analysis towards minimizing data collection efforts. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 2374-2380 2023.

---

[21] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEF/CVF conference computer vision and pattern recognition, pages 14318-14328, 2021.
[22] Hao Li, Ying Chen, Yifei Chen, Rongshhn Yu, Wenxian Yang, Liansheng Wang, Bowen Ding, and Yuchen Han. Generalizable whole slide image classification with fine-grained visual- semantic interaction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1 11398-11 14077, 2024.
[23] Jana Lipkova, Richard J Chen, Bowen Chen, Ming Y Lu, Matteo Barbieri, Daniel Shao, Anurag Vaidya, Chengkuan Chen, Luoting Zhuang, Drew FK Williamson, et al. Artificial intelligence for multimodal data integration in oncology. Cancer cell, 40(10):1095-1110, 2022.
[24] Jiaxuan Lu, Fang Yan, Xiaofan Zhang, Yue Gao, and Shaoting Zhang. Pathotune: Adapting visual foundation model to pathological specialists. arXiv preprini arXiv:2403. 16497, 2024.
[25] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guil- laume Jaume, Igor Odintsov, Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, pages 1-12, 2024.
[26] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature biomedical engineering, 5(6):555-570, 2021.
[27] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.033448 2018
[28] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint Xix::3304.07193 2023.
[29] Linhao Qu, Kexue Fu, Manning Wang. Zhijian Song, et al. The rise of ai language patholo- gists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. Advances in Neural Information Processing Systems, 36, 2024.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748 8763. PMLR, 2021.
[31] Abtin Riasatian, Morteza Babaic, Danial Maleki, Shivam Kalra, Mojtaba Valipour, Sobhan Hemati, Manit Zaveri, Amir Safarpoor, Sobhan Shafiei, Mehdi Afshari, et al. Fine-tuning and training of densenet for histopathology image representation using tega diagnostic slides. Medical image analysis, 70:102032, 2021.
[32] Alexander Robey, George J Pappas, and Hamed Hassani. Model-based domain generalization. Advances in Neural Information Processing Systems, 14:202100200221 2021.
[33] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems, 34:2136-2147, 2021.
[34] Artem Shmatko, Narmin Ghaffari Laleh, Moritz Gerstung, and Jakob Nikolas Kather. Artificial intelligence in histopathology: enhancing cancer research and clinical oncology. Nature Cancer, 1026 2022.
[35] Wenhao Tang, Fengtao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang, and Bo Liu. Feature re-embedding: Towards foundation model-level performance in computational pathology. arXiv preprint 272.2 2024.
[36] Eugene Vorontsov, Alican Bozkurt, Adam Casson, George Shaikovski, Michal Zelechowski, Siqi Liu, Philippe Mathieu, Alexander van Eck, Donghun Lee, Julian Viret, et al. Virchow: million-slide digital pathology foundation model. arXiv preprint arXiv:2309.07778 2023.
12

---

[37] Xiyue Wang, Sen Yang, Jun Zhang, Minghui Wang, Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Transformer-basee unsupervised contrastive learning for histopathological image classification. Medical image analysis, 81: 81:102539, 2022.
[38] Guile Wu and Shaogang Gong. Collaborative optimization and aggregation for decentralized do- main generalization and adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 6484 6493, 2021.
[39] Cui Yufei, Ziquan Liu, Xiangyu Liu, Xue Liu, Cong Wang, Tei-Wei Kuo, Chun Jason Xue, and Antoni B Chan. Bayes-mil: A new probabilistic perspective on attention-based multiple instance learning for whole slide images. In The Eleventh International Conference Learning Representations, 2022.
[40] Hongrun Zhang, Yanda Meng, Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Difd-mil: Double-tiee -ier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18802-18812, 2022.
[41] Yunkun Zhang, Jin Gao, Mu Zhou, Xiaosong Wang, Yu Qiao, Shaoting Zhang, and Dequan Wang. Text-guided foundation model adaptation for pathological image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 272-282. Springer, 2023.
[42] Jinghao Zhou, Chen Wei, Huiyu Wang. Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.08832, 2021.
13

---

A Overview
The structure of this supplementary material shown below,
Appendix B presents additional experimental results, including changes in in-domain and out-of-domain site splitting, further comparisons on the BRCA dataset under traditional settings, additional results the PRAD dataset, and results under site-preserved cross- validation.
Appendix discusses additional ablation study results concerning hyperparameters, includ- ing the impact of loss weights for PIM and SIM, the effect of the number of representative patches k, and further ablation studies on the CIB module.
Appendix D presents detailed descriptions of datasets and experimental settings.
Appendix provides the detailed definition and formula derivation.
Appendix elaborates on the Concept Information Bottleneck (CIB) module using the information plane.
Appendix details the prompts used in our experiments.
Appendix presents additional visualization results.
B Additional Results
B.1 Additional Cancer Subtyping Results with Different IND and OOD Sites
To further evaluate the generalization ability of the proposed CATE method, we conduct additional experiments on TCGA-BRCA dataset with different settings of in-domain (IND) and out-of-domain (OOD) sites. The detailed experimental settings are shown below:
BRCA (NiND=1): One site as the in in-domain data, and the other sites as the out-of-domain data. The details of the in-domain site are shown below:
AR: 44 slides of IDC and 15 slides of ILC
BRCA (NinD=2): Two sites as the in-domain data, and the other sites as the out-of-domain data. The details the in-domain sites are shown below:
AR: 44 slides of IDC and 15 slides of ILC.
B6: 39 slides of IDC and slides ILC.
The additional results presented in Table It is evident that the proposed CATE-MIL consistently delivers superior performance in both in-domain and out-of-domain settings, demonstrating its effectiveness in enhancing task-specific information and minimizing the impact of irrelevant data,
B.2 Additional Cancer Subtyping Results with Traditional Experimental Settings
In the main paper, we divided the dataset into in-domain and out-of-domain sites to evaluate the generalization ability of the proposed CATE method. To further evaluate its effectiveness, conducted additional experiments under traditional settings, randomly splitting the dataset into training, validation, and testing sets. We report the performance metries, including means and standard deviations over 10 Monte-Carlo Cross-Validation runs, in Table Additionally, we provide comparisons of training times and parameter sizes for various methods.
is evident that the proposed CATE-MIL consistently outperforms others in both AUC and ACC metrics, underscoring its superiority. Furthermore, CATE-MIL benefits from shorter training times and smaller parameter sizes compared to the R"T-MIL method.
B.3 Additional Gleason Grading Results on PRAD
CATE is a general framework that can be applied to more complex tasks beyond cancer subtyping, such as Gleason grading in prostate cancer. We have conducted conducted experiments on the TCGA-PRAD dataset to evaluate the performance of CATE-MIL in Gleason grading. Specifically,
14

---

Table 4: Supplementary Cancer Subtyping Results BRCA.
BRCA (NNDD=1)
Method
OOD-AUC
OOD-ACC
IND-AUC
IND-ACC
ABMIL
0.916 = 0.017
0.882 + 0.022
0.977 + 0.029
0.953 + 0.036
CLAM
0.895 + 0.031
0.842 + 0.049
0.973 + 0.035
0.938 + 0.043
DSMIL
0.892 + 0.017
0.863 + 0.016
0.967 + 0.022
0.924 + 0.048
TransMIL
0.917 + 0.008
0.878 + 0.016
0.980 + 0.031
0.955 + 0.038
DTFD-MIL
0.920 = 0.023
0 0.024
0.982 + 0.030
0.957 + 0.038 0.038
R
0.902 + 0.022
0.835 + 0.063
0.963 + 0.037
0.944 + 0.039
CATE-MIIL
0.938 + 0.014
0.895 + 0.021
0.984 + 0.021
0.959 + 0.038
BRCA (NIND=2)
Method
OOD-AUC
OOD-ACC
IND-AUC
IND-ACC
ABMIL
0,914 + 0.012
+ 0.017
0,954 + 0.035
0,910 + 0.043
CLAM
0.914 = 0.021
0.890 0.013
0.958 + 0.034
0.926 + 0.050
DSMIL
0.931 = 0.007
0.971
TransMIL
0.929 = 0.009
0.897 0.006
0.962 0.030
0.903 + 0.039
DTFD-MIL
0.898 = 0.009
0.868 + 0.017
0.950 + 0.034
0.890 + 0.037
TT
0.919 = 0.017
0.893 = 0.017
0.962 + 0.025
0.906 + 0.026
CAIE--II
0.947 = 0.005
0.920 = 0.004
0.980 + 0.015
0.939 + 0.022
The best results are highlighted in bold, and the second-best results are underlined. denotes the methods for feature re-embedding that utilize ABMIL as base MIL model.
Table 5: Results on BRCA under Traditional Settings.
Method
AUC
ACC
Training Time
Params Size
ABMIL
0.922
0.902+0099
67.16 S
1.26 MB
CLAM
0.928
0.912
67.99 S
2.01 MB
DSMIL
0.934=10.39
0.910
72.91 S
1.26 MB
TransMIL
0.927:0.044
0.906=00m2
72.39 S
5.39 MB
DTFD-MIL
0.940=0.030
2.90|
70.99 S
8.03 MB
R'T-MIL
0.936
0.903
68.87 S
9.28 MB (1.26+8.02)
CATE-MIL
0.945
0.917
68.53 S
4.26 MB (1.26+3.00)
The best results are highlighted bold, and the second-best results are underlined. denotes the methods for feature re-embedding that utilize ABMIL base MIL model.
Table 6: Supplementary Gleason Grading Results PRAD.
PRAD
Method
OOD-AUC
OOD-ACC
IND-AUC
IND-ACC
ABMIL
0.704 + 0.034
0.510 + 0.075
0.742 + 0.060
0.575 + 0.051
CATE-MIL
0.755 + 0.050
0.567 + 0.067
0.797 0.044
0.643 0.075
Table 7: Supplementary Results under Site-Preee Cross- Valis-Valiitiio...
Dataset
Method
OOD-AUC
IND-AUC
ABMIL
0,912 + 0.012
0.905 + 0.043 0.942 = 0.038
BRCA
CATE-MIL
0.935 0.014
ABMIL
0.942 0.016
0.941 - 0.013
NSCLC
CATE-MIL
0.951 = 0.015
0.943 0.008
ABMIL
0.980 0.001
0.986 + 0.010
RCC
CATE-MIL
0.983 + 0.001
0.989 = 0.007
15

---

To provide more comprehensive evaluation of the proposed CATE-MIL, we conduct additional experiments using siteppreervee tross-validation [I]], where the samples from the same site are preserved in the same fold. For each fold, we split the data into training and testing sets, and these the samples in PRAD dataset are divided into Gleason pattern 3, 4, and 5, and the task is to classify the samples into these three categories. The results are shown in T It evident that CATE-MIL consistently outperforms the base model ABMIL in both in-domain and out-of-domain settings, demonstrating its effectiveness in enhancing task-specific information and minimizing the impact of irrelevant data. In the future, more studies reveal the connection between morphological features and molecular biomarkers and more powerful pathology VLMs are developed, our framework has the potential benefit more complex tasks.
B.4 Additional Experiments using Site-Preserved Cross- Validation
testing sets are regarded in-domain testing data. And the other sites are used as out-of-domain testing data. The results are shown Table 7 It evident that CATE-MIL consistently outperforms ABMIL in both in-domain and out-of-domain settings.
C
Additional Ablation Study
C.1 Ablation Study the Weight of PIM and SIM Losses
The overall objective of the proposed CATE-MIL is weighted sum of the PIM and SIM losses, along with the classification loss:
C= APPPIM AgLSIM.
(12)
We note that the hyperparameters for the weights of the PIM and SIM losses were not specifically tuned the main paper. To investigate the effect of the weight of PIM and SIM losses on the model performance, we conduct an ablation study on BRCA (NNN==2) with CATE-MIL (ABMIL integrated with CATE), varying the weights of PIM and SIM losses. The results are shown in Figure
AUC
ABMIL Baseline
DOD AUC
0.50
5.22
0.91
8:93
15
20
25
50
90
2.02
0.00
IND
534
0.04
X
Às
Figure 4: Ablation study of the weight of PIM and SIM losses on the model performance.
From Figure evident that CATE enhances the performance of the base model ABMIL in most scenarios, particularly in out-of-domain settings. When the weight of PIM loss Ap is too low, model performance suffers, underscoring the significant role PIM loss in enhancing task-specific information. Conversely, excessively high Ap also diminishes performance, the model overly
16

---

prioritizes maximizing mutual information between the original and calibrated features at the expense of optimizing classification loss. Regarding the weight SIM loss As, optimal performance is achieved when Äş is approximately 30. If the Às is too low, the model fails to effectively eliminate irrelevant information, thereby impairing performance. Conversely, if As is too high, the model risks discarding task-relevant information, leading to performance degradation.
0OD AUC
IND AUC
ABMIL Basaline
0.6
0.50
IND AUC
0.92
0.97
0.58
0.50
0.58
102
k
k
Figure Ablation study of
C.2 Ablation Study of the Number of Representative Patches k:
In this section, we conduct an ablation study to investigate the impact of the number of representative patches, k, model performance, discussed in Section 3.2 In practice, the representative patches are selected based on the similarities between the original image feature and the corresponding class-specific concept anchor.
As shown in Figure 5 it is evident that CATE generally enhances the performance of the base model, ABMIL. When the number of representative patches is too small, model performance degrades due to insufficient capture of task-specific information. Conversely, an excessively large k also leads to performance degradation in oft-of-domain scenarios, it introduces noise from irrelevant information. Consequently, we have set the number of representative patches k to 10 in the main paper.
C.3 Additional Ablation Study of CIB Module
We further conducted experiments CATE-MIL without concept alignment (discarding PIM loss and SIM loss the CIB module) and replaced the CIB module with an MLP to investigate effect of concept alignment and the increased number of parameters. The results shown in Table g The performance CATE-MIL significantly decreases in both cases, demonstrating the importance of concept alignment in the CIB module and that the improvements of CATE are not due to the increased number parameters.
Table 8: Supplementary Ablation Study CIB Module.
Mathod
BRCA (N
BRCA
NSCLC (NixD=2)
NSCLC (Niu
RCC
RCC (Npo=*
(Njs==2)
(Nisp=3)
CATE-MIL who CF1 (Baseline)
0.936=00.00
0.942-00010
0.910=0.030
0.960=20011
0.979=20.014
0.977=20:05
wro concept alignment
0.900
0.854-40083
0.742
0.807
0.961 +00011
0.932++00016
Replace CIB with MLP
0.888:20:02
0.902-00077
0.816:20:440
0.931
0.966-20.006
0.000
D Datasets Description and Detailed Experimental Settings
In this section, we provide detailed descriptions of the datasets used in the experiments and the detailed experimental settings that we used in the experiments in Section 4.2
17

---

D.1 Datasets Description
TCGA-BRCA:
The dataset contains nine disease subtypes, and this study focuses the classification of Invasíve ductal carcinoma (IDC, 726 slides from 694 cases) and invasive lobular carcinoma (ILC,, 149 slides from 143 cases). The dataset is collected from 36 sites with 20 of them having both IDC and ILC slides, and the other 16 sites only have IDC slides ILC slides.
TCGA-NSCLC: The dataset contains two discase subtypes, including lung adenocarcinoma (LUAD, 492 slides from 430 cases) and lung squamous cell carcinoma (LUSC, 466 slides from 432 cases). The dataset is collected from 66 sites. Different from TCGA-BRCA each site only contains one disease subtype.
TCGA-RCC: The dataset contains three disease subtypes, including clear cell renal cell carcinoma (CCRCC, 498 slides from 492 cases), papillary renal cell carcinoma (PRCC, 289 slides from 267 cases), and chromophobe renal cell carcinoma (CHRCC, 118 slides from 107 cases). The dataset is collected from 55 sites. Similar to TCGA-NSCLC, each site only contains one disease subtype.
D.2 Detailed Experimental Settings
To validate that the proposed CATE effectively improves MIL model performance by enhancing the task-specific information and eliminating the disturbance of task-irrelevant information, we conduct extensive experiments across three datasets under various in-domain and out-of of-domain settings.
Specifically, for TCGA-BRCA dataset, conduct the following experiments:
BRCA (NinD=1): One site as the in-domain data, and the other sites as the out-of-domain data. The details of the in-domain site are shown below:
D8: 59 slides of IDC and 8 slides of ILC.
BRCA (NiND=2): Two sites as the in-domain data, and the other sites as the out-of-domain data. The details of the in-domain sites are shown below:
A8: 48 slides of IDC and slides of ILC.
D8: 59 slides of IDC and slides of IL...
For TCGA-NSCLC dataset, we conduct the following experiments:
NSCLC (NiND==): Two sites as the in-domain data, and the other sites as the out-of-domain data. The details of the in-domain sites are shown below:
44: 43 slides of LUAD and 0 slides of LUSC.
22: 0 slides of LUAD and 36 slides of LUSC.
NSCLC (NiNdD=)): Four sites as the in-domain data, and the other sites as the -ft-oin
data. The details of the in-domain sites are shown below:
44: 43 slides of LUAD and 0 slides of LUSC.
50: 20 slides of LUAD and 0 slides of LUSC.
22: 0 slides of LUAD and 36 slides of LUSC.
56: slides of LUAD and 35 slides of LUSC.
For TCGA-RCC dataset, we conduct the following experiments:
RCC (NiND==): Three sites the in-domain data, and the other sites as the out-of-domain
data. The details of the in-domain sites are shown below:
-3:: 48 slides of CCRCC, 0 slides of CHRCC, and 0 slides of PRCC.
-KL: O slides of CCRCC, 24 slides of CHRCC, and O slides of PRCC.
27: 0 slides of CCRCC, 0 slides of CHRCC, and 23 slides of PRCC.
RCC (NiND=6): Six sites as the in-domain data, and the other sites as the out-of-domain data. The details of the in-domain sites are shown below:
A3: 48 slides of CCRCC, 0 slides of CHRCC, and 0 slides of PRCC.
18

---

AK: 15 slides of CCRCC, slides of CHRCC, and slides of PRCC.
KL: 0 slides of CCRCC, 24 slides of CHRCC, and 0 slides of PRCC.
KM: 0 slides of CCRCC, 20 slides of CHRCC, and 0 slides of PRCC.
27: slides of CCRCC, 0 slides of CHRCC, and 23 slides of PRCC.
- 5P: slides CCRCC, 0 slides of CHRCC, and 15 slides PRCC.
E Detailed Definition and Formula Derivation
E.1 Definition of Sufficiency and Consistency
In this part, we will define the sufficcincy and consistency in the context of the concept information bottleneck (CIB) module.
First, define the mutual information between two random variables X and Y as:
T(X;Y)= =
p(2,,)) E1(,,
p(x, v) p(x)p(y)
(13)
Sufficiency: To quantify the requirement that the calibrated features should maximally informative about the label information y. define the sufficiency the mutual information between the label information and the calibrated features:
Sufficiency. & is sufficient for y > 1 = = T( = T(á:y).
The sufficiency definition requires that the calibrated features â encapsulate all information about the label information that is accessible from the original features à. In essence, the calibrated feature ā of original feature is sufficient for determining label y if and only if the amount of information regarding the specific task is unchanged after the transformation.
Since the image feature è and the concept anchor are consistent in the pathology VLM, we posit that any representation containing all information accessible from both the original feature and the concept anchor also encompasses the necessary discriminative label information. Thus, we define the consistency between the concept anchor and the original feature as:
Consistency:
Consistency. is consistent with & for
I(y; c c) 0.
The consistency definition requires that the concept anchor and the original feature á should be consistent with the label information y.
E.2 Maximize Predictive Information with InfoNCE
In this part, we will prove that the predictive information for the concept anchor can be maximized by maximizing the InfoNCE mutual information lower bound, which is defined Equation 4 As in (27). the optimal value for f (c, â.) should be proportional to the density ratio:
f f(, x (c,
(14)
((
19

---

And we can get:
p Spm cx.)
k
P Tin (c(()))
pott (à: c)
log
c
1=1
j=1 p(c) p((.)
p( ..,
+ +
p(c))p((.)
p(ct.ā.))
p ( (oss) p (át))
p p
+ P (c ac)
E
log
1 +
p (CGoos Oci)
P ((c)) p(())(
p(
j
1 p p(((e)) p ((xi)
p p(c ac) & p p p(a:)
p p(c ac)
E
log
1m- R
E
6 ci
p ( (C ais)
& p (e)))
p (ā;))
1 p ( (pos) (āx;)
log
(m + n 1)
& pps
p ( αii)
p (C p ((xs)
]
E
log
(m + n))
d ct
c Oia)
1 (cpési āc:) - k log (m + n))
i=1
7 (cpos: (t) - - -koog (m + n),
(15)
where Cneg denotes the negative concepts, including both negative class concepts and other type concepts. We can see that the InfoNCE estimation is a lower bound of the mutual information between the concept anchor and the calibrated features:
I (cpées ④x) 2 NCE (áx;) c) & log (m + + 2 INCE (á:;))
(16)
Thus, the predictive information of calibrated features for the concept anchor can be maximized by maximizing the InfoNCE mutual information lower bound.
E.3 Superftuous Information Minimization
As shown in Equation 3 the mutual information between the original features and the calibrated features ( àx can be decomposed into the mutual information 1 ((x;; c) between the calibrated features and the concept anchor and the mutual information 1 ( á/c) between the concept anchor and the calibrated features:
ax;;
[(xcc) c)
1(;; à
(17)
Predictive information for
Superfluous information
As introduced in section 3.2 the first item is the predictive information for the concept anchor, which can be maximized by maximizing the InfoNCE mutual information lower bound. Thus, the second superfiuous information item can be minimized by minimizing [(ē; &). In practice, the superfluous information minimization conducted on all patches the subset x, since the task- imreevant information is distributed across all patches. Thus, the superfluous information can be minimized by minimizing [(x; az). in 11, this mutual information can be represented as:
1 = f (2t a)) log
p (a x))
dxda
p(a))
(18)
= f p (xx ox) log (a(a) dxdx / (ox) log (a)) da.
However, the computing of the marginal distribution p (a) I (a(z) p (œ) dx is intractable. Thus, we can let the Gaussian distribution r (a) approximate the marginal distribution p (a). Since the Kullback-Leible divergence between two distributions is non neg-tega,iv, we can get:
f p (a) logg da 2 p (xx log r (a) da.
(19)
20

---

Thus, the mutual information ((:; a) have the upper bound:
/ p a a)) (a)a)) dxda - / p(a))lorr (a) da
1 a)=
(20)
/
(()()) dæde.
p (z) p (())) log r r())
In practice, to compute this upper bound, approximate distribution p (2z) with the empirical distribution:
p (ax)
Szz (m)
(21)
N i=1
Thus, the upper bound of the mutual information I (xx; a) can be computed as:
/ p (()))()()) log log
p(a(2)) dado ((x)
qo (ax,)
qo (ax; [x)) log
(22)
r r(as))
where qo (a (z) is a variational distribution with parameter 0 to approximate p (al(z) and we imple- ment this variational distribution with MLP:
4o (a x)) = N (a|MMPP (L) (x))
(23)
where MLPH and MLP are implemented with MLPs which output the mean / and covariance matrix E of the Gaussian distribution. practice, we utilize the reparameterization trick [18] to sample from the Gaussian distribution to get unbiased estimate of the gradient to optimize the opjective. Thus, the upper bound can be optimized by minimizing the KL divergence between the variational distribution qo (ax (x) and the Gaussian distribution (ar):
min Eg [D (ge (a(() ||r (a))].
(24)
Explanation of CIB with Information Plane
To better understand the concept information bottleneck (CIB) method, provide an explanation of CIB with the information plane g
Ily: 2))
Ideal Situation
CIB (Ours)
InfoMAX
yd::
Sufficiency
Consistency
Feasible Region
[db;; 2)
0
I(;;y) [(;; c) H()ly)
H ((:)
Figure 6: Explanation of CIB module with Information Plane.
As shown in Figure 6 the information plane is two-dimensional space where the x-axis represents the mutual information between the original features and the calibrated features of CIB module [(x: @), and the y-axis represents the mutual information between the label information and the calibrated features @.) The line colored orange in the information plane defines sufficiency, indicating where the calibrated features are maximally informative about the label information The consistency definition falls to the left of the sufficiency line, denoting a state where the calibrated features, containing all information from both the original feature and the concept anchor, also include the label information. The InfoMAX principle (9] states that the optimal calibrated feature should be maximally informative about the original feature.
21

---

Ideally, a good calibrated feature & should be maximally informative about the label information y (the sufficiency line in the information plane) while minimally informative about the original features di (i.e., the ideal situation in the information plane). This situation can be achieved by directly supervised learning. However, the weakly supervised nature of MII. methods complicates this goal. As demonstrated in the ablation analysis in Section 44 if we directly perform superfluous information minimization without the guidance of concept anchor (i.e., predictive information maximization), the calibrated features will be less informative about the label information and the model performance will be degraded significantly. To address this issue, the proposed CIB module introduces the concept anchor to guide the learning process of information bottleneck for each instance. As shown in Figure 6 the CIB module could effectively reduce the mutual information between the original features and the calibrated features [ &x) while preserving the mutual information between the label information and the calibrated features (l;())
G Prompts
In this section, we provide the detailed prompt templates and concepts for the datasets, which are used to generate the concept anchor for the CIB module. The prompt templates are shown in Table 9 and the concepts for TCGA-BRCA, TCGA-NSCLC, and TCGA-RCC are shown in Table TO Table
and Table 12 respectively. The templates and the class-agnostic prompts are referred from the
original paper of CONCH [25]. and the class-specific prompts are generated by querying the LLM
with the question such as 'In addition to tumor tissues, what types of tissue cells present in whole slide images of breast cancer?' The quality of LLM generated prompts has been demonstrated in several recent studies [29 (2). In principle, we can LLM (e-g., GPT-4) generate reliable expert-designed prompts and further verified by pathologists. This strategy can ensure the scalability and reliability of the prompts.
Table 9: Prompt Templates.
Templates
<CLASSNAME>.
photomicrograph showing <CLASSNAME>.
photomicrograph of <CIASSNAME>. an image of <CIASSNAME>.
an image showing <CLASSNAME>. an example of <CLASSNAME>. <CLASSNAME> is shown.
this is <CLASSNAME>. there is <CLASSNAME>>
histopathological image showing <CLASSNAME>,
histopathological image of <CLASSNAME>.
histopathological photograph of <CIASSNAMES.
histopathological photograph showing <CLASSNAMES. shows <CLASSNAME>
presence of <CLASSNAME>
<CLASSNAME> is present.
an H&E stained image of <CLASSNAMES.
H&E stained image showing <CLASSNAME>.
an H&E image showing <CLASSNAME>.
H&E image of <CLASSNAME>. <CLASSNAME>, H&E stain.
<CIASSNAME>, H&E
22

---

Table 10: Concepts for TCGA-BRCA.
Concept Type
Classes
Concept Prompts
invasive ductal carcinoma breast invasive ductal carcinoma
IDC
invasive ductal carcinoma of the breast
invasive carcinoma of the breast, ductal pattern idc
Class-specific Concept
invasive lobular carcinoma breast invasive lobular carcinoma invasive lobular carcinoma of the breast invasive carcinoma of the breast, lobular pattern ilc
ILC
adipocytes adipose tissuc fat cells tissue fat
Adipocytes
connective tissue stroma fibrous tissue collagen
Class Class-agnostic Concept
Connective tissue
necrotic tissuc necrosis
Necrotic Tissue
normal breast tissuc normal breast cells
Normal Breast Tissue Cells
normal breast
Table 11: Concepts for TCGA-NSCLC.
Concept Type
Classes
Concept Prompts
adenocarcinoma
lung adenocarcinoma
LUAD
adenocarcinoma of the lung luad
Class-specific Concept
squamous cell carcinoma lung squamous cell carcinoma
LUSC
squamous cell carcinoma the lung luse
connective
stroma fibrous tissue
Connective tissue
collagen
Class-agnostic Concept
necrotic tissue
Necrotic Tissue
necrosis
normal lung tissue normal lung cells
Normal Lung Tissue Cells
normal lung
23

---

Table 12: Concepts for TCGA-RCC.
Concept Type
Classes
Concept Prompts
clear cell renal cell carcinoma
renal cell carcinoma, clear cell type renal cell carcinoma of the clear cell type clear cell rcc
CCRCC
papillary renal cell carcinoma
renal cell carcinoma, papillary type renal carcinoma the papillary type papillary roc
Class-specific Concept
PRCC
chromophobe renal cell carcinoma renal cell carcinoma, chromophohe type renal cell carcinoma of the chromophobe type
CHRCC
chromophobe rec
adipocytes adipose tissue
Adipocytes
cells fat tissue fat
connective tissue stroma fibrous tissue collagen
Class-agnostic Concept
Connective tissuc
necrotic tissue necrosis
Necrotic Tissue
normal kidney tissue normal kidney cells
Normal Kidney Tissue Cells
normal kidney
H More Visualization
To further illustrate the effectiveness of the proposed CATE-MIL, we provide more visualization results in this section. The visualization results for IDC and ILC in TCGA-BRCA shown in Figure 7 and Figure respectively.
24

---

Invasive Ductal Carcinoma (IDC)
(a) CATE-MIL Heatmap
Similarity after CATE
Original
TGGA3C-44E-020000
-2.92
(b) ABMIL Hoatmap
(d) Original Simiiorty
(f)
DCCospt Eahased Fastor
(a) Heatmap
Similarity
9.21
(8) Original WSI
-2.82
(b) ABMIL Heatmap
(d) Original Similarty
(A UMAP Visualization
IDComeep #Diercalfetoo
(6) CATE-MIL Heatmap
(c) Similarity after CATE
(c) Original WSI
TCG4 00.001
Original Similrrty
0.45
UMAP Visualization
Conoe Fattre
2.11
Figure 7: Visualization results for samples of IDC in TCGA-BRCA. (a) Attention heatmap of CATE-MIL. (b) Attention heatmap of original ABMIL. (c) similarity between the calibrated features and the corresponding class concept feature. (d) similarity between the original features and the corresponding class concept feature. (c) Original WSL. (f) UMAP visualization.
25

---

Invasive Lobular Carcinoma (ILC)
(8) CATE-MIL Heatmap
(c) Similarity after CATE
6.39
Original
TEGA S00-3LAATT1122 002.00022..00000
4.11
[b) ABMIL Heatmap
(d) Original Simiariyy
0.54
UMAP Visualization
COREET Echured fuatue Oc @hignal fosture
(a) CATE-MIL Heatmap
(c) Similarity
7.31
[0] Original WSI
-3.67
[b) ABMIL Heatmap
(d) Original Similrity
(1) UMAP Visualization
DCCompp folunaa fature 1Consbii Tuker
(a) CATE-MIL Heatmap
(c) Similarity after CATE
[c] Oriinnal WSI
TOSA AD 4221-12.00000000
4.76 0.56
(d) Original Similarty
UMAP Visualiztioo
DC Coneet Extaradd Fuaur eic Greai Orgga
408
Figure 8: Visualization results for samples of ILC in TCGA-BRCA. (a) Attention heatmap of CATE-MIL. (b) Attention heatmap of original ABMIL. (c) similarity between the calibrated features and the corresponding class concept feature. (d) similarity between the original features and the corresponding class concept feature. (c) Original WSL. (f) UMAP visualization.
26

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discuss the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification: For each theoretical result, the paper provide the full set of assumptions and complete (and correct) proof.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: The paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions the paper.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
The paper provide open access the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cc.
aublic/guudees/Cdeettuii.. ott.:eewwwwiisiiiooooooooooo.g for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ wttps/cccppplcccuuiuaeaeeecccuea
Stteee//CooSSSemsssS.Si.a www.ssssiicggcooo for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
The paper report error bars suitably and correctly defined.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
29

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The paper provide sufficient information the computer resources needed reproduce the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neuripss.
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discuss both potential positive societal impacts and negative socictal impacts of the work performed.
Guidelines:
30

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
31

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: New assets introduced the paper well documented and is the documentation provided alongside the assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
32

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
33


Reviews:
[{"review_id": "JsNxUN2lfS", "reviewer": "Reviewer_ZzEi", "summary": "The authors propose an approach to enhance the performance of Multiple Instance Learning (MIL) models by incorporating concept prompts within the context of pathological Vision-Language Models (VLMs). Two modules, i.e., the Concept-guided Information Bottleneck (CIB) module and the Concept-Feature Interference (CFI) module, are designed to calibrate and inject similarity characteristics between features and concepts. Experiments are conducted on three Whole Slide Image (WSI) datasets, yielding interesting results. The paper includes qualitative and ablation analyses.", "questions": "The paper is technically sound, but I am not convinced by the provided experimental results.\n\n1) Could the authors suggest more principled ways to obtain expert-designed prompts? Relying on hand-crafted prompts may not be scalable or generalizable.\n\n2) Please refer to the weakness and clarify the experimental settings. More detailed qualitative and quantitative analyses are needed.\n\n3) Lack of comparison with ref [1], which also uses the Information Bottleneck theory to improve the feature quality of MIL.\n\n\n[1] Task-specific Fine-tuning via Variational Information Bottleneck for Weakly-supervised Pathology Whole Slide Image Classification. CVPR 2023.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Technically Sound Method:\nThe proposed approach demonstrates a well-founded methodology, leveraging concept prompts within pathological Vision-Language Models for enhancing Multiple Instance Learning models.\n\nWell-Written Paper:\nThe paper is clearly and coherently written, providing a logical flow and structure that makes the complex ideas accessible and easy to follow.\n\nWell-Illustrated Figures:\nThe figures are well-designed and effectively illustrate key modules, significantly aiding in the understanding of the methodology.", "weaknesses": "Dependence on Text Prompts: The performance heavily relies on the quality of text prompts/concepts. A more principled approach to generating expert-designed prompts would be beneficial. Only the hard-crafted ones make the  \"free lunch\" not that free. \n\nExperimental Settings: The choice of the number of IND and OOD sites for different datasets is unclear. How to choose the number of IND and OOD sites? It would be helpful to see performance under more traditional settings on NSCLC and RCC. What's more, it is noticed that the gain of CATE decreases with an increasing number of sites, raising questions about the practicality of the main experimental settings in real-world WSI analysis.\n\n\nAttention Maps: The attention maps are not that convincing to me. Since CATE-MIL provides almost identical attention maps to the baseline ABMIL except for intensity. If ABMIL predicts incorrect attention, CATE-MIL could potentially worsen the situation.\n\n\nUnderperformance on NSCLC and RCC: The authors claim that the underperformance of CATE-MIL on NSCLC and RCC due to the elimination of site-specific patterns is concerning. Clarification on why site-specific information for the IID test is not considered task-relevant is needed, especially as this issue does not appear with BRCA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your insightful comments and constructive suggestions. We would like to address your concerns below:\n\n> **1. Dependence on text prompts and principled ways to obtain prompts**\n\nIn our experiments, in addition to the expert-designed 'class-specific prompts' from the original CONCH article, we also employed GPT-4 to generate 'class-agnostic prompts', asking questions such as 'In addition to tumor tissues, what types of tissue or cells are present in whole slide images of breast cancer?' \nThe quality of GPT-4 generated prompts has been demonstrated in several recent studies [2,3]. \nIn principle, we can use LLM (e.g., GPT-4) to generate reliable expert-designed prompts and further verified by pathologists. This strategy can ensure the scalability and reliability of the prompts.\n\n> **2. Experimental Settings**\n\n- As we discussed in the general response, each dataset in the TCGA project contains samples from different source sites. The different source sites have different staining and imaging characteristics, causing feature domain shifts among different sites. Therefore, MIL models trained on one site may not generalize well to others. \nTo better evaluate the model performance, we report the testing performance on IND data (in-domain, the testing and training data are from the same sites) and OOD data (out-of-domain, the testing and training data are from different sites).\n\n- For the BRCA dataset, we randomly selected one or two sites as IND data and used the remaining sites as OOD data. \n**For NSCLC (2 categories) and RCC (3 categories) datasets, each site contains samples from only one subtype**. Therefore, we cannot select only one site as IND data, as it will include one category/subtype in the training data. Instead, we randomly selected one or two corresponding sites for **each category** as IND data and used the other sites as OOD data, resulting in 1 or 2 IND sites for BRCA, 2 or 4 for NSCLC, and 3 or 6 for RCC.\n\n\n> **3. Performance under a more traditional setting and gain of CATE**\n\n- As mentioned above, for NSCLC and RCC, different categories correspond to different sites. In a traditional setting (i.e., training and testing data are from the same sites), MIL models tend to **use site-specific features (e.g., staining) as shortcuts** for high performance, rather than identifying useful class-specific features, making performance less reflective of the models’ actual capability.\nTherefore, we did not report the results of the traditional setting on NSCLC and RCC. If necessary, we can provide these results in the comments.\n\n- The gain of CATE on OOD performance is closely related to the performance of the baseline MIL model. As the performance of MIL models has been very high (over 0.95 OOD-AUC for NSCLC) with more training data, further enhancement becomes harder, reducing the gain from CATE. However, **CATE’s benefit remains positive**. As shown in Table 5 in the Appendix, when all sites are used as IND data for the BRCA dataset (under traditional settings), CATE-MIL still outperforms vanilla ABMIL.\n\n> **4. Attention Maps**\n\nWe argue that although the attention maps of CATE-MIL and ABMIL are similar, they are **not identical**. CATE enhances pre-extracted patch features by preserving task-relevant information and eliminating task-agnostic information, which can strengthen the attention scores of cancerous patches and reduce the scores of non-cancerous patches.\nAs shown in the more detailed attention maps in the **accompanying PDF**, **some non-cancerous patches mistakenly assigned high attention by ABMIL are corrected by CATE-MIL** (e.g., the upper left area of the second sample), demonstrating CATE-MIL's ability to correct ABMIL's incorrect predictions.\n\n> **5. Underperformance in NSCLC and RCC**\n\n- Site-specific information such as staining style and imaging characteristics is not relevant to the WSI analysis tasks, while it can be used to distinguish different sites. \nFor the NSCLC and RCC datasets, as each site contains samples from only one category/subtype, the model can use these site-specific features as shortcuts for subtyping. In other words, the model learns **how to distinguish different sites, not different subtypes**.\nOur CATE-MIL eliminates these site-specific features, so the performance is lower than other MIL models using these spurious site-specific shortcuts. However, our method is more generalizable, as evident by the OOD performance.\n\n- This issue is not very severe in the BRCA dataset, as each site contains samples from two different categories simultaneously. The MIL models need to learn to classify based on task-relevant features rather than site-specific ones. Therefore, CATE's elimination of site-specific patterns does not decrease BRCA's IND performance but improves it by enhancing task-relevant features.\n\n> **6. Comparison with ref [1]**\n\nThe comparison results between CATE with ref [1] are shown below. It can be observed that ref [1] underperforms vanilla ABMIL on OOD data for NSCLC and RCC. This shows that ref [1] overfits IND data and focuses on shortcuts rather than task-relevant features. Furthermore, it involves three separate stages, making it time-consuming and resource-intensive.\n\n||Method|OOD-AUC|\n|---|---|---|\n|BRCA($N_{\\text{IND}}$=1)|ABMIL|0.914±0.015|\n||ABMIL+ref [1]|0.934±0.003|\n||ABMIL+CATE|0.951±0.003|\n|NSCLC($N_{\\text{IND}}$=2)|ABMIL|0.874±0.021|\n||ABMIL+ref [1]|0.713±0.009|\n||ABMIL+CATE|0.945±0.016|\n|RCC($N_{\\text{IND}}$=3)|ABMIL|0.973±0.005|\n||ABMIL+ref [1]|0.953±0.004|\n||ABMIL+CATE|0.983±0.002|\n\n[1] Task-specific fine-tuning via variational information bottleneck for weakly-supervised pathology whole slide image classification. CVPR 2023.\n\n[2] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeruIPS 2023.\n\n[3] Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction. CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YB6Xf9n28p", "reviewer": "Reviewer_9fs8", "summary": "The authors introduce a new tool called Concept Anchor-guided Task-specific Feature Enhancement (CATE) for analysis of whole slide images.  The authors are taking advantage of the open source weight now available for the CONCH pathology vision language model trained on paired captions of images from journals and other publicly available image-caption pairs.  The high level conceptulization is that language prompts related to a task at the whole slide image level can be enrich for the features most relevant to the downstream task.  The title cleverly asserts this is a free lunch when using a vision language encoder for feature extraction the expertly curated concepts can \"bring foward\" the most relevant features when aggregating at the whole slide level.  \n\nInterestingly, they split up the CATE task into Predictive Information Maximization to enhance features associated with related concepts and Superfluous Information Minimization is used to supress the concepts likely to be associated with unimportant features. \n\nThe outputs of PIM and SIM are used during aggregation task to determine weighting that MIL should use for a given feature vector derived from the tile. \n\nFinally a module called concept-feature interference is used calibrate the CIB scores. \n\nThese module provide enhanced features to the aggregation function that provides relative weighting based on the relationship between the semantic concepts relationship to the downstream task. \n\nCATE and weights are not backpropogated through the feature embedding tasks and they are static in the MIL function (to the best of my understanding). \n\nThe authors chose RCC, NSCLC, and BRCA tasks from the TCGA to demonstrate performance", "questions": "Out of distribution (OOD) is a term that means many different things to different people.  Will you please spare a paragraph to better explain what you mean by OOD. If you find that OOD is not the best term for what you mean, I think it would be better to use a less loaded term. \n\nHave you considered a hybrid approach where the vision feature vectors can be extracted with a different foundation model but the VLM is use for the concept guidance?\n\nHow the normal distribution of SIM in equation (8) align with Bernoulli distribution of ABMIL? Are there alternative distributions that could be used here? (may be Poisson distribution to define random occurance of a non-informative patch?)\n\nMinor comments for improvement in readability: \n1) In figure 2a, the image encoder cartoon is larger on the side of image which implies the feature vectors are larger than the original patches. \n2) Equation (6) LHS should be I(x; alpha | c)", "limitations": "Limitations section is reasonable.  I have provided further feedback in the weaknesses section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model is build such that one can introduce to any MIL-like framework that uses a VLM encoder. \n\nIt is a clever idea to guide the aggregation function with the concepts one can get from VLM encoder. \n\nThe paper is clearly written and claims are not over stated.  \n\nOn the chosen tasks, it is reasonably clear that a performance boost is acheived consistently and in some cases by a dramatic margin.  Given that is is a bolt-on method these are very good apples to apples comparisons. \n\nThe ablations study suggests that the 3 novel modules (PIM, SIM, and CFI and  parts are additive to peformance, at least on these tasks.", "weaknesses": "I agree with the weaknesses the authors have put forth.  \nThe model scope is ~mostly~ limited to downstream task for the the concepts are part of the VLM training.  The tasks that are chosen for benchmarking are actually tasks that are likely well represented in VLM training sets with robust descriptions of SCC vs. LUAD, etc.  For tasks that source material does not contain common descriptions, say more challenging tasks like predicting mutations, it is unclear to what extent, this work would provide advantage over other approaches.  \nMy presumption is that VLM encoders are less robust for feature extraction than the now available large foundation models trained with 10^6 plus whole slide images. If one is not able to use these models and must use an inferior model, then one might argue it is not a \"free lunch\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your positive feedback and constructive suggestions. We would like to address your concerns as follows:\n\n> **1. More diversified downstream tasks**\n\nThank you very much for your valuable comments. As we discussed in the paper,  we agree that the performance of CATE relies on the well-presented concepts generated by pathology VLM. For other tasks such as survival prediction, the concepts may be more complex and hard to write, which may limit the applicability of CATE. \nHowever, CATE has the potential to benefit more complex tasks beyond subtyping like Gleason grading. We have conducted experiments and shown that CATE can enhance Gleason grading performance. We will add the complete results in the revised manuscript. \nIn the future, as more studies reveal the connection between morphological features and molecular biomarkers, our framework has the potential to benefit more complex tasks.\n\n|Method|OOD-AUC|OOD-ACC|IND-AUC|IND-ACC|\n|---|---|---|---|---|\n|ABMIL|0.704±0.034|0.510±0.075|0.742±0.060|0.575±0.051|\n|CATE-MIL|0.755±0.050|0.567±0.067|0.797±0.044|0.643±0.075|\n\n> **2. Dependence on VLM encoders**\n\nThank you very much for your kind comments, we believe that with the development of pathology research and pathology VLMs, pathology VLMs will have broader prospects, and more powerful VLMs will be developed, reducing the gap between the pure pathology foundation models.\n\n\n> **3. Explanation of OOD**\n\nThank you for your valuable suggestion. In this work, OOD represents **out-of-domain**. We use the datasets from the TCGA project, which contain samples from different **source sites**. The diverse source sites may have different imaging technologies and staining protocols, and each site can be regarded as a domain, causing feature **domain shifts** between different sites. Therefore, we select several sites for training, referred to as in-domain (IND) data. Consequently, data from other sites with different feature distributions are called out-of-domain (OOD) data. We will add this explanation to the revised manuscript.\n\n> **4. Hybrid approach of different foundation models**\n\nYes, in our preliminary experiments, we have considered a similar hybrid approach that uses a more powerful vision foundation model to extract image features and a more powerful text encoder to extract text features, followed by a bridge layer to project these images and text features into a shared embedding space. \nThis approach could bring more flexibility to the model design and improve applicability to more challenging tasks.\nHowever, the end-to-end training of such a hybrid model is more complex and requires much more computational resources, which is beyond the scope of this work. \nIn the future, we will explore more parameter-efficient fine-tuning strategies and continue to explore this direction.\n\n\n> **5. Distribution alignment in equitation (8)**\n\nWe apologize for the misunderstanding. In fact, the primary goal of SIM is to minimize superfluous information, which can be seen as a form of regularization and is task-agnostic. It does not need to align directly with the Bernoulli distribution of ABMIL. Additionally, we greatly appreciate your suggestion; using the Poisson distribution to define the random occurrence of a non-informative patch is indeed a very promising idea. We will explore this in our future work.\n\n> **6. Minor comments**\n\nThank you very much for your valuable suggestions. We will adjust the figure and correct the equation in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7OpkHuUT6S", "reviewer": "Reviewer_R2V2", "summary": "The authors introduce CATE, a novel approach designed to enhance the generalizability of histopathology models by leveraging task-specific concepts derived from the text endoder of pathology vision language models(VLM). CATE includes two modules: CIB and CIF, which work synergistically to improve model robustness. The CIB module identifies and retains features that contribute positively to predicting the task outcome, while it eliminates features that contain irrelevant or superfluous information.  Additionally, CIF specializes in creating task-specific features by leveraging similarities between the enhanced image features the and predefined concept anchors. The authors demonstrate the effectiveness of CATE across an extensive series of histopathology datasets, showcasing superior results in model performance and generalization.", "questions": "- Why did you use accuracy as an evaluation metric? Accuracy is straightforward to interpret, but it can be misleading in imbalanced datasets. Metrics like F1-score, weighted accuracy, precision, and recall offer more nuanced insights into model performance, particularly in contexts where class distribution is skewed such as the BRCA dataset.\n\n- What strategy was used to split the dataset into in-domain and out-of-domain test sets? Understanding this process ensures transparency in evaluating the model's generalizability across different clinical settings or data sources. \n\n- How do you define the class-agnostic concepts?\n\n- What is the intuition behind maziming the distance between the enhanced featrures and the class-agnostic concepts? When features are overly tailored to the training samples by maximizing their separation from class-agnostic concepts, the model may become too specialized and prone to overfitting.", "limitations": "- Exploring the potential of concept anchors for applications beyond subtyping, such as mutation predictio could be an intersting avenue to explore.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The concept introduced by CATE is highly relevant as it addresses a major challenge in histopathology: ensuring models generalize effectively across diverse datasets and clinical scenarios. In computational pathology, the lack of external validation cohorts often leads to uncertainty about whether models have learned genuine signals or have simply overfitted to confounding factors or batch effects. To validate their approach, the authors split existing datasets into indoor and outdoor test sets, demonstrating CATE's robustness across different conditions.\n- The paper is well-presented and clearly written.\n- The strength of the paper lies in its innovative approach of repurposing general-purpose vision language models to extract rich, meaningful representations for downsream tasks.", "weaknesses": "- CATE's performance hinges largely on the quality of its concept anchors, which in turn depends on domain expertise and the quality of the pre-trained pathology VLM, a weakness that is also highlighted by the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate your positive comments and valuable suggestions. We would like to address your concerns below one by one:\n\n> **1. The quality of concept anchors**\n\nThank you very much for your understanding. As we discuss in the paper, we agree that CATE's performance highly relies on the quality of concept anchors. Fortunately, some operations can be done to improve the quality of the prompts and concepts.\n\nFor example, to ensure the quality of the concepts, we can ensemble multiple prompts with various templates and use their average embedding for each class to generate robust and stable class concepts, reducing the risk that the concepts cannot fully capture the characteristics of the class. Additionally, the quality of the concepts will be further improved with the development of pathology research and pathology VLMs.\n\n> **2. Evaluation metrics**\n\nThank you very much for your valuable comments. In the original paper, we follow the experiment settings of previous papers [1,2] and chose AUC as the primary metric and accuracy as the secondary metric. \nWe agree that the accuracy metric can be misleading in imbalanced datasets. Therefore, we mainly focused on the AUC metric in the original paper. To further improve the reliability of the results, we will supplement the F1-score metric in the revised manuscript.\n\n\n> **3. Dataset split strategy**\n\nThank you for your suggestion. We would like to provide more relevant details. We define the in-domain (IND) and out-of-domain (OOD) data based on the source sites of the TCGA dataset.\nSpecifically, each dataset in the TCGA project contains samples from different source sites. The different source sites have different staining and imaging characteristics, causing feature domain shifts among different sites. Therefore, MIL models trained on one site may not generalize well to others. To better evaluate the model performance, we report the testing performance on IND data (in-domain, the testing and training data are from the same sites) and OOD data (out-of-domain, the testing and training data are from different sites).\n\nFor the BRCA dataset, we randomly selected one or two sites with samples from two categories as IND data and used the remaining sites as OOD data. \nFor NSCLC (2 categories) and RCC (3 categories), **each site contains samples from only one category**. We randomly selected one or two corresponding sites for **each category** as IND data and used the other sites as OOD data, resulting in 1 or 2 IND sites for BRCA, 2 or 4 for NSCLC, and 3 or 6 for RCC.\nFor the IND data, we also randomly split it into training, validation, and testing sets; training the models on the IND training set, and evaluating them on both IND testing sets (IND performance) and OOD data (OOD performance). \n\n\n> **4. Definition of class-agnostic concepts**\n\nIn our paper, class-agnostic concepts refer to the common tissues that are agnostic to the classification task. For example, in the cancer subtyping task, different subtypes have subtype-specific attributes directly related to the task, known as class-specific concepts. \nMeanwhile, there are many pieces of information in the WSI that are agnostic to different subtypes, such as adipose tissue, connective tissue, and normal tissues. We define these common tissues as class-agnostic concepts.\n\n> **5. The intuition behind maximizing the distance**\n\nThanks for your comments. We understand your concern. However, the model will not become too specialized and prone to overfitting the training data. The class-specific and class-agnostic concepts are **general across different domains and are not specific to the training data**.\nBy bringing the enhanced features closer to the class-specific concepts and further away from the class-agnostic concepts, the model can avoid distractions unrelated to the classification task and focus on the general task-relevant concepts.\nThis prevents overfitting on the training dataset, and the maximization of the distance between enhanced features and class-agnostic concepts helps the model generalize better to out-of-domain data.\n\n> **6. Potential applications beyond subtyping**\n\nThank you for your valuable suggestions. We have conducted experiments to show that CATE can also benefit to other more complex tasks beyond subtyping like Gleason grading, as shown in the following table.\n\nOther tasks like mutation prediction are indeed more challenging compared to subtyping. \nIn the future, as more studies reveal the connection between morphological features and molecular biomarkers or patient prognosis, we will continue to explore whether our CATE can be applied to a wider range of pathology analysis tasks.\n\n|Method|OOD-AUC|OOD-ACC|IND-AUC|IND-ACC|\n|---|---|---|---|---|\n|ABMIL|0.704±0.034|0.510±0.075|0.742±0.060|0.575±0.051|\n|CATE-MIL|0.755±0.050|0.567±0.067|0.797±0.044|0.643±0.075|\n\n\n[1] Transmil: Transformer based correlated multiple instance learning for whole slide image classification. NeurIPS 2021.\n\n[2] Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. CVPR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1SrzzUHzuB", "reviewer": "Reviewer_VcAb", "summary": "The paper proposes a way to enhance image features from VLM pathology foundation models for downstream tasks by aligning them with task-specific text prompts. The authors use two modules -- one which extracts an image representation aligned with task concepts and other which measures similarity of representation with tasks, to enrich image representations for downstream tasks.\nThe authors show results for the cancer subtyping task across 3 datasets indicating the usefulness of the proposed approach.", "questions": "Its unclear how much improvements are coming from the patch filtering step and the increased number of parameters and if those alone are sufficient enough.\n\nI think its worth trying the alignment on the slide-level representations instead of patch-level representations (similar to SC-MIL) which makes the approach more generic and limits the drawbacks with using slide-level labels for patches.", "limitations": "Highlighted the limitations in weaknesses.\n\nProvided some suggestions and improvements in the questions. \n\nGiven the weaknesses and the lack of evidence and limited applicability of the approach to more complex MIL problems like scoring (gleason grading, HER2 grading) which need aggregation of information across patches or other analytical tasks like survival prediction, molecular signature prediction where there might not be possible to write text descriptions, I am basing my decision", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**Clarity and Quality**:\n\nThe paper is well written and the key ideas are explained clearly. \nThe experiments are well setup, the results and ablations provide good evidence of the proposed method for the specific problems and the limitations are discussed.\nThe code is shared and the supplementary section provides more details around ablations and the theoretical assumptions.\n\n**Originality**\nThere has been prior work in using task anchors to improve MIL using learnt or clustered task prototypes (ProtoMIL, TPMIL, etc) \nThe idea of using textual prompts to create task anchors and use these to filter patches and enrich patch representations is interesting. \nThe predictive information maximization (PIM) objective is interesting and is similar to applying CLIP on-the-fly with fixed text embeddings, to extract more task-specific image features.\nThe SIM objective seems useful to suppress task-irrelevant information.\n\n**Significance**\nThe proposed approach is a simple addition to MIL and can be applied in conjunction with existing MIL methods.", "weaknesses": "The main weaknesses are around some of the assumptions made in the paper listed below which limit the applicability of the approach.\n\n\n**Aligning patches with tasks**\n\nIn WSIs the labels are only available at slide-level and not patch-level and for many problems only a fraction of the patches in the slide have task-relevant info and for many others the information may need to be aggregated from multiple patches.\n\nEven though there is a filtering step to filter patches, the step of aligning patches to tasks is noisy when many patches dont have task info and it may not even be appropriate for problems which need aggregation of information across patches (as the approach homogenizes representations of all patches belonging to the same class).\n\nThe authors show results on only one type of task -- subtyping where most patches on the slide have the sub-type information and is also typically a simpler MIL task indicated by the strong results on the benchmarks for most methods. \n\nInstead of aligning patch representations, it would be better to align slide-level representations with task-concepts, which addresses both the drawbacks listed above. There is also evidence from prior works around usefulness of this like SC-MIL (https://arxiv.org/abs/2303.13405 which align representations of slides belonging to same-class) and they also show aligning patch-level representations (patch SC-MIL) does poorly compared to aligning at the slide-level\n\n\n**Unclear Improvements**\n\nIt is unclear how useful the task-alignment objective with the information bottleneck layer is. \nThe key part seems like the filtering step which extracts relevant patches for the task and I suspect if thats contributing to majority of the improvements.\nOnce filtering is done, is the alignment step really needed as it seems like a dimensionality reduction step to extract task specific features. \nA sufficiently strong model should be able to able to extract task relevant features from the embedding dimensions with cross-entropy loss.\nThe alignment step might also be unsuitable as it homogenizes patch representations which may not be good for many MIL tasks and directly optimizing with cross entropy on task-labels might be more flexible and general.\n\nIts also unclear if the improvements are coming from a bigger model due to the CIB and CFI modules. Scaling ABMIL to similar number of params would provide more context on this.\n\n\n**Task Concept Alignment**\nThe other key assumption is using text prompt embeddings as task anchors which is very dependent on factors like the quality of the pre-trained VLM, the textual prompts and descriptions used and the idea that it is possible to write morphological descriptors for tasks which is typically not possible for many MIL tasks like survival or molecular signature prediction.\n\nThe sufficiency and consistency constraints imposed could be limiting specially when any of the factors above break. The SIM objective specifically could be problematic when text doesnt fully capture the required features for the task and other relevant image features could be suppressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your detailed and constructive comments. We will address your concerns as follows:\n\n> **1. Aligning patches with tasks**\n\nThanks for your comments. We address your concerns from the following aspects.\n\n- The motivation behind patch-level alignment in this work is to enhance the patch features extracted by the pathology VLM to adapt them to specific downstream tasks and make the CATE more general and able to be incorporated with any MIL method.\nThe alignment is based on the information bottleneck principle and does not explicitly rely on the patch labels. For patches that don't contain task-relevant information, the CIB module will filter out the task-agnostic information, making the alignment more accurate and less noisy. \n\n- We agree that slide-level alignment is more straightforward. However, aligning only at the slide level may be difficult, as the slide-level representation (aggregated from patch features) is noisy without patch-level feature enhancement, hindering the model’s training, as shown in the following table.\nIncorporating the slide-level alignment with our patch-level alignment can further improve the model’s performance, which demonstrates the importance and rationale of patch-level alignment in the CATE.\n\n|Method|Patch-level Alignment|Slide-level Alignment|BRCA($N_{IND}$=1)|BRCA($N_{IND}$=2)|NSCLC($N_{IND}$=2)|NSCLC($N_{IND}$=4)|RCC($N_{IND}$=3)|RCC($N_{IND}$=6)|\n|---|---|---|---|---|---|---|---|---|\n|ABMIL|||0.914±0.015|0.899±0.035|0.874±0.021|0.951±0.023|0.973±0.005|0.971±0.007|\n|Aligning Patch (CATE-MIL w/o CFI)|&#10004;||0.936±0.010|0.942±0.010|0.910±0.030|0.960±0.011|0.979±0.004|0.977±0.005|\n|Aligning Slide||&#10004;|0.899±0.019|0.883±0.025|0.853±0.018|0.945±0.017|0.970±0.004|0.961±0.011|\n|Aligning Patch+Slide|&#10004;|&#10004;|0.941±0.007|0.944±0.011|0.923±0.019|0.969±0.004|0.982±0.002|0.978±0.003|\n\n\n> **2. Unclear improvements**\n\nThanks for your comments. \nThe filtering step alone, without concept alignment, does not provide any benefit, and the increased number of parameters in the CIB and CFI modules does not directly lead to performance improvement.\n\n- The filtering step is used to obtain a subset of the most task-relevant patch features to align with concepts and supervise the training of the information bottleneck. During the training of MIL and the prediction process, all patch features are input to the CIB module to filter task-agnostic information. Without concept alignment, the CIB module degenerates into an encoder of VAE and does not provide any benefit.\n\n- To show this, we conducted experiments on CATE-MIL without concept alignment (discarding PIM loss and SIM loss of the CIB module) and replaced the CIB module with an MLP to investigate the effect of **concept alignment** and the **increased number of parameters**. The results are shown in the following table. The performance of CATE-MIL significantly decreases in both cases, demonstrating the importance of concept alignment in the CIB module and that the improvements of CATE are not due to the increased number of parameters. We will add this results in the revised manuscript.\n\n|Method|BRCA($N_\\text{IND}$=1)|BRCA($N_\\text{IND}$=2)|NSCLC($N_\\text{IND}$=2)|NSCLC($N_\\text{IND}$=4)|RCC($N_\\text{IND}$=3)|RCC($N_\\text{IND}$=6)|\n|---|---|---|---|---|---|---|\n|ABMIL|0.914±0.015|0.899±0.035|0.874±0.021|0.951±0.023|0.973±0.005|0.971±0.007|\n|CATE-MIL|0.936±0.010|0.942±0.010|0.910±0.030|0.960±0.011|0.979±0.004|0.977±0.005|\n|CATE-MIL w/o concept alignment|0.900±0.017|0.884±0.033|0.742±0.059|0.897±0.022|0.961±0.011|0.932±0.016|\n|Replace CIB with MLP|0.888±0.027|0.902±0.037|0.816±0.040|0.931±0.023|0.966±0.006|0.951±0.021|\n\n\n> **3. Task concept alignment and applicability to more complex MIL problems**\n\n- As we pointed out in the original paper, we agree that the performance of CATE relies on the quality of the pathology VLM and it may not be difficult to be applied to other analytical tasks that are difficult to describe with prompts. \nHowever, we argue that the main contribution of this work is that it offers a novel and promising idea to adapt the pathology VLM to downstream tasks by leveraging the inherent consistency between image and text in pathology VLM. \n\n- The proposed CATE can benefit more complex tasks beyond subtyping, such as Gleason grading. Experiments on the TCGA-PRAD dataset show that CATE enhances Gleason grading performance, demonstrating that CATE does not affect the aggregation of patch features in the MIL model.\n\n\n|Method|OOD-AUC|OOD-ACC|IND-AUC|IND-ACC|\n|---|---|---|---|---|\n|ABMIL|0.704±0.034|0.510±0.075|0.742±0.060|0.575±0.051|\n|CATE-MIL|0.755±0.050|0.567±0.067|0.797±0.044|0.643±0.075|\n\n\n- In the future, as more studies reveal the connection between morphological features and molecular biomarkers and more powerful pathology VLMs are developed, our framework has the potential to benefit more complex tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are sincerely grateful to the reviewers for your insightful comments, constructive suggestions, and acknowledging the clarity and quality of our paper, as well as our contributions in terms of novelty and effectiveness in feature enhancement for better WSI analysis.
We have carefully read and considered all the comments and suggestions from the reviewers. We would like to first respond to the most common concerns raised by the reviewers as follows:

> **1. Experimental Settings on IND and OOD Data**:

To help readers better understand our experimental settings, we would like to provide more background and motivation behind the split of in-domain (IND) and out-of-domain (OOD) data for each dataset and the underperformance of CATE on NSCLC and RCC datasets.

- The dataset (e.g., BRCA, NSCLC, and RCC) in the TCGA contains samples from different source sites (i.e., different hospitals or laboratories). **Different source sites have different staining protocols and imaging characteristics**, causing feature domain shifts between different sites [1,2]. Therefore, MIL models trained on one or more sites may not generalize well to others. To better evaluate the true performance of the models, we selected several sites as IND data (in-domain, the testing and training data are from the same sites), and used data from other sites as OOD data (out-of-domain, the testing and training data are from different sites), and reported the testing performance on both IND and OOD data.

- For the BRCA dataset, we randomly selected one or two sites as IND data and used the remaining sites as OOD data. **For NSCLC (2 categories) and RCC (3 categories) datasets, each site contains samples from only one subtype**. Therefore, we cannot select only one site as IND data, as it will include one category/subtype in the training data. Instead, we randomly selected one or two corresponding sites for **each category** as IND data and used the other sites as OOD data, resulting in 1 or 2 IND sites for BRCA, 2 or 4 for NSCLC, and 3 or 6 for RCC.

- Site-specific information such as staining style and imaging characteristics is not relevant to the WSI analysis tasks, while it can be used to distinguish different sites. For the NSCLC and RCC datasets, as each site contains samples from only one category/subtype, the model can use these site-specific features as shortcuts for subtyping. In other words, the model learns **how to distinguish different sites, not different subtypes**. Our CATE-MIL eliminates these site-specific features, so the IND performance is lower than other MIL models using these spurious site-specific shortcuts. However, our method is more generalizable, as evident by the OOD performance.
In contrast, for the BRCA dataset, each IND site contains samples from two different categories simultaneously, requiring MIL models to classify based on task-relevant features rather than site-specific ones. Therefore, CATE's elimination of site-specific patterns does not decrease BRCA's IND performance but improves it by enhancing task-relevant features.

> **2. Application beyond Subtyping**:

As we pointed out in the original paper, CATE is optimized for classification tasks such as cancer subtyping. Moreover, CATE has the potential to benefit more complex tasks beyond subtyping, such as Gleason grading. We have conducted experiments and shown that CATE can enhance Gleason grading performance. We will add the complete results in the revised manuscript. In the future, as more studies reveal the connection between morphological features and molecular biomarkers and more powerful pathology VLMs are developed, our framework has the potential to benefit more complex tasks.
|Method|OOD-AUC|OOD-ACC|IND-AUC|IND-ACC|
|---|---|---|---|---|
|ABMIL|0.704±0.034|0.510±0.075|0.742±0.060|0.575±0.051|
|CATE-MIL|0.755±0.050|0.567±0.067|0.797±0.044|0.643±0.075|

[1] Robust whole slide image analysis for cervical cancer screening using deep learning. Nature communications.

[2] Deep learning-based transformation of H&E stained tissues into special stains. Nature communications.

[3] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeruIPS 2023.

[4] Generalizable Whole Slide Image Classification with Fine-Grained Visual-Semantic Interaction. CVPR 2024.

Meta-review: The paper introduces Concept Anchor-guided Task-specific feature Enhancement (CATE), a novel approach that enhances the performance of pathology foundation models by tailoring them to specific downstream tasks. By leveraging task-specific concepts derived from vision-language models and incorporating a Concept-guided Information Bottleneck module, the method effectively maximizes the relevance of extracted features while minimizing irrelevant information. Adding a Concept-Feature Interference module further refined the discriminativeness of these features. The reviewers acknowledged the novelty and potential impact of this approach, especially its ability to incorporate domain-specific knowledge into whole slide image (WSI) classification. While some concerns were raised regarding the dependency on vision-language model quality and potential overfitting, the additional experiments and clarifications provided by the authors in their rebuttal were sufficient to address most of these issues. The consensus is to accept this paper.
Decision: Accept (poster)