The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification
Authors: Linhao Qu
xiaoyuan Luo
Kexue Fu
Manning Wang
Zhijian Song
Venue: NeurIPS 2023 poster (2023)
Keywords: multiple instance learning
whole slide image classification
prompt learning
vision-language model
few-shot learning

Abstract: This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. A solution is proposed based on prompt learning and the utilization of a large language model, GPT-4. Since a WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached as a Multiple Instance Learning (MIL) problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC is to classify both bags and instances with only a limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose a two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based on the bag features. Our approach incorporates the utilization of GPT-4 in a question-and-answer mode to obtain language prior knowledge at both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, a learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification. All codes will be made publicly accessible.

The Rise of AI Language Pathologists: Exploring Two-level Prompt Learning for Few-shot Weakly-supervised Whole Slide Image Classification
Linhao Qu Xiaoyuan Luo Kexue Fu 22, Manning Wang Zhijian Song Digital Medical Research Center, School Basic Medical Science, Fudan University. Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention.
Abstract
This paper introduces the novel concept of few-shot weakly supervised learning for pathology Whole Slide Image (WSI) classification, denoted as FSWC. solution is proposed based on prompt learning and the utilization of large language model, GPT-4. Since WSI is too large and needs to be divided into patches for processing, WSI classification is commonly approached problem. In this context, each WSI is considered a bag, and the obtained patches are treated as instances. The objective of FSWC to classify both bags and instances with only limited number of labeled bags. Unlike conventional few-shot learning problems, FSWC poses additional challenges due to its weak bag labels within the MIL. framework. Drawing inspiration from the recent achievements of vision-language models (V-L models) in downstream few-shot classification tasks, we propose two-level prompt learning MIL framework tailored for pathology, incorporating language prior knowledge. Specifically, we leverage CLIP to extract instance features for each patch, and introduce a prompt-guided pooling strategy to aggregate these instance features into a bag feature. Subsequently, we employ a small number of labeled bags to facilitate few-shot prompt learning based the bag features. Our approach incorporates the utilization of GPT-4 in a question- and-answer mode to obtain language prior knowledge both the instance and bag levels, which are then integrated into the instance and bag level language prompts. Additionally, learnable component of the language prompts is trained using the available few-shot labeled data. We conduct extensive experiments on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, demonstrating the notable performance of the proposed method in bag and instance classification. Codes will be available at httttps: h/gittu/b/bbtttt..// thub. com/miccaiif/TOP
Multiple Instance Learning (MIL)
1 Introduction
The automated analysis of pathology Whole Slide Images (WSIs) plays crucial role in contemporary cancer diagnosis and the prediction of treatment response natural images, WSIs typically possess a gigapixel resolution, rendering them unsuitable as direct inputs for deep learning models. To address this problem, common approach involves dividing WSIs into non-overlapping small patches for subsequent processing. However, due to the vast number of patches within single WSI, it impractical to assign fine- ene-grained labels to these small patches, rendering instance-level supervised methods unfeasible [39 9 7 27 29). Consequently, Multiple Instance Learning (MIL), a popular weakly supervised learning paradigm, has emerged as an effective solution to overcome these challenges. In the MIL framework, each WSI is considered "bag". and
28
31
26
15
43 3). Unlike
"Corresooningg Authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the extracted patches are regarded as instances within this bag. In positive bag, there exists at least one positive instance, while in negative bag, all instances are negative. During training, only the bag labels are known, whereas the instance labels remain unknown 35 49
Deep learning- -aased WSI classification typically involves two tasks: bag-level classification, accurately predicting the category of the target bag, and instance-level classification, accurately identifying positive instances within positively labeled bags.
MIL methods for WSI classification can be broadly categorized into instance-based methods 4
35 23
46]. Instance-based approaches involve training an instance classifier using artificially g pseudo labels to estimate the probability of each instance being positive. These individual predictions are then aggregated to obtain the bag-level prediction. On the other hand, bag-basce based methods have emerged as the predominant approach for WSI classification. These methods initially extract features for each instance and subsequently employ an aggregation function to combine the features of all instances within a bag into a single bag feature. Finally, a bag classifier trained using the known labels of the bags. Recently, attention- -basd aggregation methods [1, [6 61 54
and bag-based methods [8 22 56
29
41 22 56 29] have demonstrated promising performance, and they could leverage attention scores assigned to each instance for instance-level classification.
Most existing MIL methods for WSI classification assume the availability of a substantial amount of labeled data at the bag level. However, in clinical practice, limitations such as patient privacy concerns, challenges in obtaining pathological samples, or the diagnosis of rare or emerging diseases
often result a scarcity of pathological data [34 39 9 21. Consequently, existing methods are ill-equipped to handle such few-shot learning scenarios. In this paper, we present novel WSI classification problem termed Few-shot Weakly Supervised WSI Classification (FSWC). Traditional few-shot learning strives to achieve good classification performance with very few labeled support samples per class (usually only 1, 2, 4, 8, or 16 samples). Similarly, in FSWC, only a few bags are labeled for training (only 1, 2, 4, 8, or 16 per class). Notably, FSWC diverges from traditional few-shot learning on natural images due to the absence of instance-level labels, with only bag-level labels provided. The MIL setting and the absence of instance labels make FSWC considerably more challenging than traditional sho-shot learning problems. The primary objective of FSWC is to achieve precise bag- bag-level and instance-level classification with very few training bags. Figure A and B provide intuitive illustrations of the existing WSI classification and FSWC tasks.
Recently, significant advancements have been made in visual representation and transfer learning with the emergence of vision- laiiuag models (V-L models) such as CLIP [38, . ALIGN (20), and FLIP [55]. These models have demonstrated remarkable success, indicating their ability to learn universal visual representations and perform effectively in zero-shot or few-shot settings for downstream tasks. Motivated by these achievements, we apply V-L models to address the FSWC problem. Unlike traditional visual frameworks, V-L models employ a two-tower architecture consisting of an Image Encoder and a Text Encoder for pre-training on extensive image-text pairs. The objective is to align images and their corresponding description texts within a shared feature space. For zero-shot classification tasks, a carefully designed prompt template, such as "a photo of a [CLS]." is utilized to classify the target image through similarity matching in the corresponding feature space. To enhance transfer performance and eliminate the need for handcrafted prompt templates, methods like CoOp 660) replace manual prompts with learned prompt representations. These approaches adapt V-L models for few-shot image recognition tasks using small number of labeled images from the target dataset, with only prompt parameters being trained while the V-L model parameters remain fixed. Figure D provides an intuitive depiction of the fine-tuning paradigm based on pre-trained V-L models and prompt learning in the context of natural images. A straightforward application of V-L models in FSWC involves using the Image Encoder to extract instance features within each bag and aggregating these instance features into bevelevel representations using established aggregation functions. Subsequently, prompt learning algorithms like CoOp [60] can be employed the bag level. However, our experimental results demonstrate unsatisfactory performance with this approach. The main challenges lie in the absence of efficient instance aggregation methods and the complexity of designing appropriate bag-level prompts.
In order effectively utilize V-L models for addressing the FSWC problem, we present a T-O-level Prompt Learning MIL framework guided by pathology language prior knowledge, referred to as TOP. The main concept of TOP is illustrated in Figure D. Initially, we employ the Image Encoder of V-L models to extract instance features within each bag. Subsequently, we introduce prompt guided pooling as a means to aggregate these instance features into a bag-level feature. To facilitate this

---

A
Extensive ueizing bags
B
big houndary mntuneebuuningg
Very fewtraining bogs
Pocitive slide
Nogative shde
bounday bounlaty
Positie bag
Negalive beg
Langaage Modele
Posilive inslance
Negalive inslance
Similatty
Images
ICLASS]
Tt
Pale
[G [German shenhert]] [Golden retnever)
retriovel
Text Encoder
3
Image
Encoder
Rag leve prompt group
Similarity
destriplive prompt ensembles
Bag featume
Rag-level CLASS
Mix
Leamable prompt
Text Encoder
Bag
Bag bext tokon
Prompt guided
pooling
Mutiple instance- prompt groups
Instances
Protolype descriplive prompt ensembles
Learaable prompt
Text
Image Encoder
Instance Prototype_
Encoder
Instance protolypes
Instence Featiuees
Figure 1: A. Existing WSI classification tasks. B. Our proposed FSWC task based on language models. C. Existing prompt learning paradigm using pre-trained V-L models, where the parameters of the V-L models are always frozen. D. Our proposed Two-level Prompt Learning paradigm.
aggregation process, we leverage the capabilities of the large language model GPT-4 33 to generate multiple instance-level prompt groups These prompt groups serve instance-level pathology language prior knowledge, effectively guiding the aggregation process. Furthermore, GPT-4 333 is utilized construct bag-level prompt group, which matched with the bag feature facilitate few-shot prompt learning at the bag level. Throughout this process, the bag-level prompt group acts as pathology language prior knowledge, providing guidance for the few-shot prompt learning process. In TOP, both the instance-lever and bag-level prompts comprise three components. The first component encompasses the task label at the instance bag level, such as "an image patch [Lymphocytes]" and "a WSI of [Lung adenocarcinoma]. The second component consists of combination of various visual description texts associated with the given task label. important to note that these visual descriptions are not manually designed, but are obtained through a question -anseerinn approach employing GPT-4. For example, we generate prompts such as "What are the visual pathological forms of Lung adenocarcinoma!". Our experimental findings consistently demonstrate the criticality of incorporating task-specific visual descriptions tailored to the pathological WSI classification task. Inspired by CoOp (60), we design the third component as learnable continuous prompt representation, enabling automatic adaptation and further enhancing transfer performance.
The main contributions of this paper are as follows:
We proposed and effectively solved the novel Few-shot Weakly Supervised WSI Classification (FSWC) problem.
We propose a Two-level Prompt Learning MIL framework, referred to as TOP. At the instance level, we leverage pathology language prior knowledge derived from GPT-4 to guide the aggregation of instance features into bag features. In addition, at the bag level, we create comprehensive level prompt group by incorporating bag-level pathology categories and visual pathology descriptions as prior knowledge to facilitate few-shot learning under the supervision of bag labels.

---

We conducted comprehensive evaluations on three real-world WSI datasets, including breast cancer, lung cancer, and cervical cancer. TOP demonstrates strong bag classification and instance classification performance under limited bag-level labeled data, achieving state-of-the of-the--rttttttt results.
Related Work
2.1 Multiple Instance Learning for WSI Classification
follow two-step process: they extract features for each instance and then aggregate these instance features to obtain a represennattive bag feature. Subsequently. bag classifier is trained using the bag features and corresponding labels. Notably, attention-buse aggregation methods [18] [6
Most MIL methods for WSI classification [8
56
46
19
37
61 [54] 41 22 56 29
have demonstrated promising results, where attention scores assigned to each instance contribute to instance-level classification. However, these approaches heavily rely on substantial number of labeled bags, which not the case our proposed FSWC task, where only limited number
of labeled bags are available for training. While some recent studies [22] 56
40
36
[
6 (30]
employed pre-trained networks to extract instance features, these models were solely pre-trained on self-supervised learning or ImageNet data. In contrast, explore the utilization of pre-trained
V-L models (as detailed Section 2.3) for extracting instance features. Additionally, we present a
Two-level Prompt Learning MIL framework guided by pathology language prior knowledge.
2.2 Few Shot Classification
The primary objective of Few-shot Classification (FSC) is to accurately classify test samples by utilizing limited labeled support examples. Typically, only 1, 2, 4, 8, or 16 examples per category are available. This classification process leverages learned knowledge and prior information [2] 42
45
47
Recently, uisionllagguag models (referred to as V-L. models, detailed in Section 2.3 such as
ALIGN [20], and FLIP (55) have demonstrated significant success in FSC. This success suggests that these large models have acquired universal visual representations and exhibit improved performance in downstream tasks, particularly in zero-shot or few-shot scenarios. Nevertheless, the task of few-shot WSI classification under bag-level supervision has not yet been investigated. In this paper, we introduce this paradigm as few-shot reakll--uuperiiee WSI classification (FSWC) and propose an effective solution by employing ELIP-based prompt learning.
CLIP
38
2.3 Vision-language Models and Prompt Learning
ALIGN 20). and FLIP (55), which have been trained on extensive image-text pairs, exhibit remarkable potential in visual representation These V-L models employ dual-tower architecture that comprises visual and text encoders. They utilize contrastive learning to align text- to-image and image-to- saage-to-texx te-text in the feature space. The pre-trained V-L models, including CLIP [8] demonstrate remarkable transferability in image recognition. By carefully designing text descriptions, referred to as "prompts," " to align with the corresponding image features in the feature space, these models enable zero-shot or few-shot classification. Building on the accomplishments of CLIP, CoOp 60 replaces manually created prompts with a learned prompt representation and adapts V-L. models to downstream FSC tasks. Motivated by the triumph of V-L. models in FSC within the domain of natural images, we propose several techniques to effectively adapt pre-trained V-L models for addressing the FSWC problem.
Pre-trained Vision-Language (V-L) models, such as CLIP
38
and transfer learning (57 60 59
50 51
52 53
58).
Preliminaries
3.1 Problem Formulation
XN) Given a dataset = (X,,XX,...,, X2) comprising N WSIs, and each WSI X; partitioned into non-overlapping small patches {.j.j. = 1, 1.2....n.. nz). where represents the number of patches obtained from X, patches within X, collectively form bag, and each patch serves as an instance of that bag. The bag is assigned label Y; 10, where i {1, 2, .J. The labels of each

---

instance fui..J= = 2,2, not are associated with the bag label in the following manner:
{ O, Y; = 1,
Bi =0
Vi.j
else
This implies that all instances within negative bags are assigned negative labels, whereas positive bags contain at least one positive-labeled instance. In the context of weakly-supervised MII., only the bag label is provided for the training set, while the labels of individual instances remain unknown. The Few-shot Superi-s WSI Classification (FSWC) task poses allows only a limited number of labeled bags for training. In the FSWC task, "shot" refers to the number of labeled slides. In N-shot experiments, the training set uses only N pairs of positive and negative slides for training, and the model is evaluated on the complete testing set. Typically, only a small number of bags per class, such as 1, 2, 4, 8, or 16, are available for training. The objective of FSWC is to accurately classify both the bags and individual instances, despite the scarcity of labeled training bags.
greater challenge as
3.2 Vision-Language Pre-training and Few-shot Prompt Learning
We first provide concise overview of the pre- pre-training of V-L models and few-shot prompt learning. In this paper, we use the V-L model of CLIP 38 but our method is also applicable to other CLIPP-like V-I. models.
Model and Pre-training. The CLIP framework comprises an image encoder and a text encoder. The image encoder employs ResNet-50 or ViT to extract image features, while the text encoder uses Transformer to generate text features. CLIP's primary training objective to establish an embedding space that align image features with their corresponding text features by means of a contrastive loss. During training, a batch of image-text pairs is used, and CLIP maximizes the cosine similarity of matching pairs while minimizing the cosine similarity of all other non-matching pairs. To facilitate the acquisition of diverse visual concepts that can be readily applied to downstream tasks, CLIP is trained on a large-scale dataset consisting of 400 million image-text pairs, which includes medical data.
Zero-shot Inference. CLIP has the inherent ability to perform zero-shot classification because is pre-trained to predict whether an image matches a given text description. Specifically, the approach involves using the image encoder to extract features from the image to be classified, using the text encoder to extract features of the text descriptions as "prompts") " and then calculating the degree of match between the image features and all the text features to determine the classification category. Formally, let z be the image feature extracted by the image encoder for image x, and let {w;} 1 be set of weight vectors generated by the text encoder. Here, K represents the number of candidate categories, and each w; comes from the prompt "a photo of [CLASS]", where the CLASS token is replaced with a specific class name, such as "cat", "dog", or "car". The predicted probability of each category is calculated using Equation
all candidate categories (referred to
plu i = exp (cos (w; f) /T)
Σラ= exp (cos (w.. f) /T)
Here, 7 is the temperature coefficient learned by CLIP, and cos(.,) )) represents cosine similarity.
Few-shot Prompt Learning. Research has shown that the construction of prompts plays a crucial role in the downstream task classification performance, CoOp [60 learns an end-to-end continuous vector as a supplementary prompt using limited labeled data from the downstream task. Formally, the overall prompt is designed as:
t VVMMMM....VMLCCAASSS..
where each [V]m (m {I,..., M)) is a vector of the same dimension the word embedding (i.e.. 512 for CLIP), and M is a hyperparameter that specifies the number of context tokens. The text encoder of CLIP is used to encode the prompt to obtain classification weight vector {w;), K that represents the visual concept, and then the prediction probability of each category calculated according to Equation 2 A small amount of labeled data in the downstream task is used to train
5

---

levd prompt group
Task descciptive pcumpt Eccembles
level [CLASS]
mng upuanesss
Leamable prompt
Text
Long Adesscetirenee
Those sttepes pas can
way
Encodle
Luzg Squaranus Cell
[Lug abencaadioona digtas legatr.
Bay token
Carconarus
accopppilany andsold
frss
Instance prompt guided pooling
Image Encadar
Stmmliatty Manit
frus
Rig featur F
Beg X
Insturess
Insiazze fediuee
Multple instance- level prompr. groupe
Instunce [Protutype]
Instaree pcototype desctiptive premupts
Testancs protetypes P
Kilanduker spitheltare) Call. arroyed glass-tios
sptthetum
stractats, pespoesile for secrinng vabous
camablle prompt
Text Ensoder
[C olamnur spilasium) thaped
Adgase theer
Elaoza oussacivv tistee See ebt moril
[Squamous epttheluma Thn, dit cells resembiing pllle, will suclen locatd Be cantet
Frest
Figure 2: Framework of proposed TOP.
and optimize [V]m in the prompt using cross-entropy loss. Other parameters, including the Image Encoder and Text Encoder of the V-L. model, are frozen.
Loss = CEE(pp))
(4)
where y represents the ground truth label and p represents the predicted probability of each class based on the prompt.
Method
Figure presents our proposed TOP framework. First, we use the Image Encoder Simg of CLIP to extract instance features Zi for all instances within bag X,. Then, we propose instance prompt guided pooling to aggregate these instance features into bag feature F; During this process, use large language model GPT-4 to generate multiple instance-level prompt groups and input them into the Text Encoder frezt of CLIP to generate instance prototypes P. which serve instance-level language priors to guide the aggregation process. Next, we use GPT-4 to construct bag-level prompt group and input it into the Text Encoder ftezt to generate Bag text token B, We then match B, with the aggregated bag feature F; to complete bag-level few-shot prompt learning. During this process, the bag-level prompt group serves as bag-level language prior guide the few-shot prompt learning process. The loss function for few-shot prompt learning is shown in equation small amount of bag-level labeled data to optimize the learnable prompt vectors [V]. in the bag-level and instance-level prompt groups (the two [V]m. vectors are different). In addition, during training, we also constrain the minimum correlation between each Instance prototype in P to prevent all prototypes from being too similar and causing degradation. The construction of instance and bag-level prompts will be introduced in Section 4.1 and the instance prompt guided pooling method will be introduced in Section 4.2
in Section 3.2 and the overall training objective is to
During inference, for bag classification, calculate the matching degree between the image features and all target class bag prompt features to determine the classification category, as shown in equation 2 in Section [3.2] For instance classification, obtain the classification score of each instance by averaging the similarity weights established between each instance feature and multiple text- based instance prototypes.
4.1 Construction of Instance and Bag-level Prompt
We utilize GPT-4 to construct instance and bag-level prompts as efficient language priors guide the instance-level feature aggregation and the bag-level few-shot prompt learning.

---

Instance-level prompt groups are designed to generate visual descriptions of various instance phe- notypes as prior knowledge to effectively guide instance-level feature aggregation. Each prompt group corresponds to potential tissue phenotype that may appear in WSI and consists of three parts. The first part text description of various instance phenotypes in the form of "an image patch of [lyympoccytes]" Typically, instances in a WSI contain information about different phenotypes, such as different forms of tumor cells, lymphocytes, epithelial cells, etc. For the second part, we used question-answeriing mode with GPT-4 to obtain common visual descriptions of different instance phenotypes in WSIs. Note that these visual descriptions do not require manual design but obtained from GPT-4, as shown in the Supplementary Materials. For each instance phenotype, focus on guiding GPT-4 to describe the visual characteristics it has from a visual perspective, thereby establishing visual priors for these phenotypes. Inspired by the learnable prompt in CoOp [60], we design the third part as a learnable prompt representation.
The bag-level prompt group is designed to guide the few-shot prompt learning process at the bag level. It also consists of three parts. The first part is the description of the task label at the bag level in the form of "a WSI of [Lung adenocareinomal"" The second part is combination of various visual descriptions for this task label, which are also obtained from GPT-4, as shown in the Supplementary Materials. For each classification task, we guide the GPT-4 model to describe the complex medical concept from a visual perspective, so as to establish a visual prior for the complex medical concept from the textual description. The third part is also designed as a learnable prompt representation.
Detailed bag-level and instance-level task descriptive prompt ensembles are presented in the Supple- mentary Materials and will be fully open-source.
4.2 Instance Prompt Guided Pooling
We propose a prompt guided pooling strategy to aggregate instance features into bag features. The main idea is to first calculate similarity weights between the image features of each instance and the prototypes of multiple text descriptions, and then use the weighted average of all instance features in a bag as the bag feature.
Mathematically, assuming that X, represents the current bag containing n, instances, use the Image Encoder sang to extract instance features Z; E Rm where m represents the dimension of the features.
Z = fimg (X;)
(5)
Then, we input multiple instance-level prompt groups T into the Text Encoder frent obtain multiple instance prototypes, P E "R","m, where np represents the number of prototypes, which corresponds to the number of instance-level prompt groups. It varies depending on the specific classification task.
P = fiet (T)
(6)
Next, we calculate the dot product of instance features Z; and set of prototypes P, and then perform softmax normalization by column (each column corresponds to a prototype), obtaining aggregation weights W; E Ru;X. for each prototype with respect to the current bag. Next, we use the aggregation weights to obtain set of weighted features W Zi E R"pxm, and finally average the weights of all prototypes to obtain the bag feature F, E R IIix
W; Softmax
Z2 P
F mean W Z,)
In addition, during training, we also constrain the minimum correlation between each instance prototype in P to avoid degeneration, which means that all prototypes are too similar to each other, with the following loss:
Loss min ( x W)
This auxiliary loss aims to separate instance prototypes learned by each instance prompt, ensuring distinct phenotypes representing WSIs. Crucial instance prototypes for slide classification stand out during aggregation.
7

---

Table 1: Performance of bag-level classification the Camelyon 16 Dataset.
Method
shot
S-shot
4-shot
2-shot
I-shot
Linear. Frobe (Mean-ponling)
Lineur Probe (Max pooling)
0.5947
0.5814
0.5417
0,5401
0,5268
Libear- Probb (Allention pooling)
0.7866
CoOp (Attenionn pooling)
0.8012
0.6746
0.6639
0.6525
0,6462
Instance Prompt Learning (Ours)
0.8301
0.7287
0.7151
Table 2: Performance of instance-leve classification on the Camelyon 16 Dataset.
Method
shot
shhho
shot
2hoho
shot
Linear-Prokf (Atienhoon pooooonnnogng))
0.7508
0.6630
0.6366
0.6177
0.5934
CoOp (Attention pooling)
0.8567
0.6648
0.6333
0.6221
0.5923
Instance-Bag Bag Prompt Learning (Ours)
0.8896
0.7200
0.7142
0.7019
0.6938
5 Experiment
5.1 Datasets, Evaluation Metrics and Comparison Methods
We comprchensively evaluated the instance classification and bag classification performance of TOP FSWC tasks using three real-world datasets of different cancer types from different centers: the Camelyon 16 Dataset [2] for breast cancer, the TCGA-Lung Cancer Datase for lung cancer, and an in-house Cervical Cancer Dataset for cervical cancer. See Supplementary Material for detailed introductions to the datasets.
For both instance and bag classification, we use Area Under Curve (AUC) as the evaluation metric. However, it should be noted that only the Camelyon 16 Dataset has the true labels for each instance, while the other two datasets only have bag-level labels. Therefore, we evaluate the instance and bag classification performance of each method on the Camelyon 16 Dataset, and only evaluate the bag classification performance on the latter two datasets.
Because existing few-shot learning methods cannot be used in FSWC task, we constructed four baselines based on the current state-of-the-ar few-shot learning methods CoOp (60) and Linear Probe [3] (1) Linear- Probe (Mean pooling), (2) Linear-Probs (Max pooling), (3) Linear-Prov
(Attention
pooling), and (4) CoOp (Attention pooling). Specifically, we first used CLIP as the image feature extractor to extract all instance features within each bag. Then, we aggregated all instance features within a bag using simple Mean, Max pooling or learnable Attention Pooling [8 to obtain the bag feature. Linear- Probbe indicates that we used linear layer to perform bag-level classification on the aggregated bag feature. CoOp indicates that we used the bag-level label and learnable prompt to perform bag-level classification through prompt learning. We conducted few-shot classification experiments with 1, 2, 4, 8, and 16 labeled bags for each class.
5.2 Implementation Details
We used the image encoder and text encoder of CLIP as the feature extractors for both images and text. The number of learnable parameters empirically set to 10 tokens for both instance and bag prompt and other quantities can be used in practice. During training, we fixed all weights of CLIP and only trained the learnable parameters of bag prompt and instance prompt. For experiments with different shots, randomly trained the network five times with fixed labeled bags and reported the average performance each method. All comparative methods utilize the same labeled bags.
5.3 Results on the Camelyon 16 Dataset
The bag classification and instance classification performance the Camelyon 16 dataset shown in Tables [ and [] respectively. can be seen that TOP achieved the best bag and instance classification performance in all few-shot settings, and significantly outperformed all comparison methods by a large margin. It can be observed that Linear-Prod with Mcan/Max pooling can hardly work. Although using trainable attention pooling helps learn the importance of each instance and improves the performance of Linear-Prohe, still has limitations in performance. Prompt learning with fully trainable prompts in CoOp outperforms Linear-Probe, In contrast, our method used two-level prompt learning paradigm, which achieved the best performance on both bag and instance
http://wwww.canerrgovvggg
8

---

Table 3: Performance of bag-level classification the TCGA-Lung Cancer Dataset.
Method
Ib
Shshon
4-shot
2.4hol
T-shol
MMan-ppolingg
0.6022
0.5418
0.9934
0,49U8
0.4646
Linear Prohe (Max pooling)
0.6227
0.5547
0.5155
0.4985
0.4876
Proce (Attention pttolntion-oollnn) poolingg
0.7178
0.6539
0.6248
0.5832
0.5713
CoOp (Ateentin pooling)
0.7840
0.6824
0.6811
0.6772
0.6801
Instance+Bag Prompt-I.eare leamning (Ours)
0.8235
0.8059
0.7531
0.7245
0.7123
Table 4: Performance of bag-level classification the Cervical Cancer Dataset.
Method
T6-shor
8-shor
4-shot
2-4hot
I-shor
Lincar-Probe (Mean-pooling)
0.6756
0.6684
0.6593
0.6246
0.6011
Lineur- Probe (Max (Max-pooling)
0.6322
0.6249
0.6038
0.5884
0.5869
Lincar-Probe pAtcotioon-oooling)
0.7345
0.7282
0.7155
0.6873
0.6137
CoOp (Atrention-poollng)
0.7565
0.7349
0.7271
0.6927
0.6484
Instance+Bag Prompt-Learning (Ours)
0.8007
0.7869
0.7618
0.7052
classification, with average improvement of 4.2% and 7.0%, respectively, over the second-best method.
5.4 Results on the TCGA-Lung Cancer Dataset and the Cervical Cancer Dataset
The results on the TCGA-Lung Cancer Dataset are shown in Table 3 It can be seen that TOP still achieves the best bag classification performance in all few-shot settings, and significantly outperforms all competitors by a large margin, with an average improvement of 6.3% over the second-best method.
The results on the Cervical Cancer Dataset are shown in Table 4 It should be noted that this task is extremely challenging, and even pathologists cannot make direct judgments. TOP displays strongest performance, with average improvement of 6.3% over the second-best method.
Table 5: Ablation results of bag-level classification on the Camelyon Dataset.
Method
16-shot
K-shot
4-shot
2-shot
T-shot
Bag Prompt+ Comppt+AAttettiion Amentoo-onnn
08168
0.6980
0.6706
0.6673
0.6483
Atrenton pooling
0,8012
0.6746
0.6639
0.6525
0.6462
CoOp+ Promptt guided pooling
0,8216
0.7079
0.6833
0.6732
0.6699
Bag Prompt-Prome guided pooling (Ours)
0.8301
0.7287
0.7151
0.6958
0.6783
Table 6: Ablation results of instance-level classification on the Camelyon 16 Dataset.
Method
shot
8-shot
4 shol
2-shot
shol
Bag Prompts Attention-pool
0.8699
0.6753
0.6425
0.6399
0.5961
Attemtion pooling
0,8567
0.6648
0.6333
0.6221
0.5923
CoOp-Prompt guided pooking
0,8754
0.6912
0.6707
0.6515
0.6045
Bag Prompt-Promppt guided pooling (Ours)
0.8896
0.7200
0.7142
0.7019
0.6938
6 Ablation Study
We conducted ablation experiments the two key components of TOP, the instance prompt guided pooling and bag-level prompt group. Experiments are conducted on the Camelyon 16 Dataset, and the bag classification and instance classification results are shown in Table Table 6 respectively. In the two tables, "Prompt guided pooling" represents the use of our proposed instance prompt guided aggregation method while "Attention-pooling" represents the use of attention-basse to aggregate instance features into bag features; "Bag Prompt" represents the use of our proposed Bag-level prompt group while "CoOp" represents using only the learnable part and bag labels as prompt at the bag-level prompt learning.
method
Effectiveness of instance prompt guided pooling, In both Table SableSand and Table matter 'Bag Prompt' or 'CoOp' is used for the bag-level prompt learning, 'Prompt guided pooling' consistently results in significantly better performance than 'Attention pooling', which indicates the effectiveness of our proposed Prompt guided pooling strategy for instance feature aggregation.
Effectiveness of Bag-level prompi group. In Table 5 and Table 6 matter what instance feature aggregation method is used, 'Bag-prompt" always outperforms "CoOp', which indicates that our proposed Task descriptive prompt is better than current SOTA methods of fully learnable prompt.

---

Table 7: Ablation and sensitivity tests of the auxiliary loss the Camelyon 16 dataset.
Shot
Weight
25 (ours)
50
Bag AUC
0.6691 0.6837
0.6759 0.6908
0.6783
0.6771
1-shot
Instance AUC
0.6938
0.6913
Bag AUC
0.6867
0.6924
0.6958
0.6945
2-shot
Instance AUC
0.6924
0.7008
0.7019
0.7011
Bag AUC
0.7087
0.7123
0.7151
0.7154
4-shot
Instance AUC
0.7033
0.7095
0.7142
0.7107
Bag AUC
0.7137
0.7218
0.7287
0.7254
8-shot
Instance AUC
0.7005
0.7188
0.7200
0.7243
Bag AUC
0.8245
0.8281
0.8301
0.8299
16-shot
Instance AUC
0.8834
0.8879
0.8896
0.8894
Effectiveness of Auxiliary Loss. Ablation and sensitivity tests on the Camelyon 6 dataset (Table demonstrate that our method is not highly sensitive to the loss weight, but its addition significantly improves performance compared to not using it.
Discussion
In this paper, we introduce the novel problem of Few-shot Weakly-supervised WSI Classification (FSWC) for the first time. We proposed Two-level Prompt Learning MIL framework named TOP to solve the FSWC problem effectively. TOP utilizes GPT-4 to generate both instance-level and bag-level visual descriptions to facilitate instance feature aggregation and bag-level prompt learning. Experiments on three WSI classification tasks shows the high performance of TOP in FSWC tasks.
We carefully consider the validity, rationale, motivation and potential model updates' impact for using GPT-4's knowledge. For validity and rationale: We rigorously reviewed pathology knowledge descriptions from GPT-4 with three senior pathologists and found them accurate and detailed. Nori et al. 32 supports GPT-4's reliability in producing medical domain knowledge due its vast medical expertise in training data. For motivation and importance: Leveraging GPT-4's knowledge as templates enhances efficiency versus manual design. This approach aligns with the few-shot learning goal, easing pathologist annotation. Manual templates might not cover all aspects; specialized doctors' templates could be needed for varied cancer types/tasks. Additionally, different doctors' descriptions vary, lacking a standardized manual description. By leveraging GPT-4's versatility, our aim is to attain knowledge descriptions for multiple cancer types and tasks while avoiding manual domain biases. For impact of model updates: GPT-44's 4s language descriptions contributed to training pathology models in our research. We will publicly share all used descriptions, codes, and models. This disclosure ensures reproducibility in reported tasks without the need of invoking GPT-4 for inference GPT-4's upgrades won't influence current outcomes. We'll explore if GPT upgrades generate new descriptions and their effect results.
training.
As widely acknowledged, training a pathological foundation model demands substantial corpus of correlated pathological images and textual data, in addition to significant computational resources. This poses considerable challenges in the field of computational pathology, characterized by high data privacy, scarce annotations, and extensive storage requirements. More recently, prominent research endeavors 25 17 have entered the spotlight, aiming to develop comprehensive vision-lang vision-language model within the domain of digital pathology. Nevertheless, the focus of this study is to some extent orthogonal to these studies. The objective of our method is to develop new prompt-learning strategies based an existing large Dision-language model for few-shot learning. More importantly, method can be combined with any large Dision-langure models (cither in general natural image domain or pathology-spen -specccficcccc domain) for few-shot WSI classification, not limited to the CLIP model used in the paper.
However, this paper has certain limitations, mainly due to the effectiveness of the proposed prompt depending on the capabilities of the large model. If the large model fails to provide effective descriptions, it can affect the model's performance. This study aims inspire further research that combines foundational models with large-scale language models for the classification pathology Whole Slide Images. Such research endeavors will herald new in AI pathology.
10

---

References
[1] Peyman Bateni, Raghav Goyal, Vaden Masrani, Frank Wood, and Leonid Sigal. Improved few-shot visual classification. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1499--1452, 2020.
[2] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA, 318(22):2199 22100, 2017.
[3] M Alvaro Berbís, David S McClintock, Andrey Bychkov, Jeroen Van der Laak, Liron Pan- tanowitz, Jochen K Lennerz, Jerome Y Cheng, Brett Delahunt, Lars Egevad, Catarina Eloy, et al. Computational pathology in 2030: a delphi study forecasting the role of ai in pathology within the next decade. EBioMedicine, 88, 2023.
[4] Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraflor, Vitor Werneck Krauss Silva, Klaus J Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas J Fuchs. Clinical-grade computational pathology using weakly supervised deep learning on whole slide images. Nature Medicine, 25(8) 2019.
[5] Richard J Chen, Chengkuan Chen, Yicong Li, Tiffany Y Chen, Andrew D Trister, Rahul G Krishnan, and Faisal Mahmood. Scaling vision transformers to gigapixel images via hierarchical self-supervised learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16144-16155, 2022.
[6] Richard J Chen and Rahul G Krishnan. Self-supervised vision transformers learn visual concepts in histopathology. arXiv preprint arXiv:2203.00588, 2022.
[7] Richard J Chen, Ming Y Lu, Jingwen Wang, Drew FK Williamson, Scott J Rodig, Neal I Linde- man, and Faisal Mahmood. Pathomic fusion: integrated framework for fusing histopathology and genomic features for cancer diagnosis and prognosis. IEEE Transactions on Medical Imaging, 41(4):757-770, 2020.
[8] Richard J Chen, Ming Y Lu, Wei-Hung Weng, Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 4015-4025, 2021.
[9] Veronika Cheplygina, Marleen de Bruijne, and Josien PW Pluim. Not-so-superrised: survey of semi-supervised, multi-instance, and transfer learning in medical image analysis. Medical Image Analysis, 54:280-296. 2019.
[10] Philip Chikontwe, Meejeong Kim, Soo Jeong Nam, Heounjeong Go, and Sang Hyun Park. Multiple instance learning with center embeddings for histopathology classification. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 519-528. Springer, 2020.
[11] Mohammad Mahdi Derakhshani, Enrique Sanchez, Adrian Bulat, Victor Guilherme Turrisi da Costa, Cees GM Snoek, Georgios Tzimiropoulos, and Brais Martinez. Variational prompt tuning improves generalization of vision-language models. arXiv preprint arXi::2210.02399. 2022.
[12) Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International Conference on Machine Learning (ICML), pages
1 1126-1135. PMLR, 2017.
[13] Chelsea Finn, Kelvin Xu, and Sergey Levine. Probabilistic model-agnostic meta-learning. Advances in Neural Information Processing Systems (NeurIPS), 31, 2018
[14] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang. Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with feature adapters. International Journal of Computer Vision, pages 1-15, 2023.

---

[15] Qinhao Guo, Linhao Qu, Jun Zhu, Haiming Li, Yong Wu, Simin Wang, Min Yu, Jiangchun Wu, Hao Wen, Xingzhu Ju, et al. Predicting lymph node metastasis from primary cervical squamous cell carcinoma based on deep learning in histopathological images. Modern Pathology, page 100316, 2023.
[16] Noriaki Hashimoto, Daisuke Fukushima, Ryoichi Koga, Yusuke Takagi. Kaho Ko, Kei Kohno, Masato Nakaguro, Shigeo Nakamura, Hidekata Hontani, and Ichiro Takeuchi. Multi-scale domain-adverssee multiple-instance cnn for cancer subtype classification with unannotated histopathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2852-3861, 2020.
[17] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual-anguagee foundation model for pathology image analysis using medical twitter. Nature Medicine, pages 1-10, 2023.
[18] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International Conference Machine Learning (ICML), pages 2127-2136. PMLR, 2018.
[19] Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Weiner, Limin Yu, and Aa- ditya Prakash. Additive mil: intrinsically interpretable multiple instance learning for pathology. Advances in Neural Information Processing Systems (NeurIPS), 15-20689-207022 2022.
[20] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning (ICML), pages 4904 4916. PMLR, 2021.
[21] Fahdi Kanavati, Gouji Toyokawa, Seiya Momosaki, Michael Rambeau, Yuka Kozuma, Fumi- hiro Shoji, Koji Yamazaki. Sadanori Takeo, Osamu lizuka, and Masayuki Tsuneki. Weakly- supervised learning for lung carcinoma classification using deep learning. Scientific Reports, 10(1):9297, 2020.
[22] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 14318-14328, 2021.
[23] Tiancheng Lin, Hongteng Xu, Canqian Yang, and Yi Xu. Interventional multi-instance learning with deconfounded instance-level prediction. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAT), volume 36, pages 1601-1609, 2022.
[24] Tiancheng Lin, Zhimiao Yu, Hongyu Hu, Yi Xu, and Chang Wen Chen. Interventional bag multi-instance learning whole-slide pathological images. arXiv preprint arXiv:2303.06873, 2023.
[25] Ming Y Lu, Bowen Chen, Andrew Zhang, Drew FK Williamson, Richard J Chen, Tong Ding, Long Phi Le, Yung-Sung Chuang, and Faisal Mahmood. Visual language pretrained multiple instance zero-shot transfer for histopathology images. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19764-19775, 2023.
[26] Ming Y Lu, Richard Chen, Dehan Kong, Jana Lipkova, Rajendra Singh, Drew FK Williamson, Tiffany Y Chen, and Faisal Mahmood. Federated learning for computational pathology on gigapixel whole slide images. Medical Image Analysis, 76:102293 2022.
[27] Ming Y Lu, Richard Chen, Jingwen Wang. Debora Dillon, and Faisal Mahmood. Semi- supervised histology classification using deep multiple instance learning and contrastive predic- tive coding. arXiv preprint arXiv:1910) 1910.1025,, 2019.
[28] Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, Melissa Zhao, Maha Shady, Jana Lipkova, and Faisal Mahmood. Ai-based pathology predicts origins for cancers of unknown primary. Nature, 594(7861): 2021.
12

---

[29] Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Matteo Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology on whole-slide images. Nature Biomedical Engineering, 5(6):555 570, 2021.
[30] Xiaoyuan Luo, Linhao Qu, Qinhao Guo, Zhijian Song, and Manning Wang. Negative instance guided self-distillation framework for whole slide image analysis. IEEE Journal of Biomedical and Health Informatics, 2023.
[31] Faisal Mahmood, Daniel Borders, Richard J Chen, Gregory N McKay, Kevan Salimian, Alexander Baras, and Nicholas J Durr. Deep adversarial training for multi-organ nuclei segmen- tation in histopathology images. IEEE Transactions on Medical Imaging, 39(1 1):3257-3261
2019.
[32] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz. Capa- bilities of gpt-4 on medical challenge problems. arXiv preprint arXiv:2303.13375, 2023.
[33] OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[34] Linhao Qu, Siyu Liu, Xiaoyu Liu, Manning Wang, and Zhijian Song. Towards label-efficient automatic diagnosis and analysis: comprehensive survey of advanced deep Harming-based weakly-supervised, semi-supervised and sulf-superviee techniques in histopathological image analysis. Physics in Medicine & Biology, 2022.
[35] Linhao Qu, Xiaoyuan Luo, Shaolei Liu, Manning Wang, and Zhijian Song. Dgmil: Distribution guided multiple instance learning for whole slide image classification. In Medical Image Computing and Computer Assisted Intervention (MICCAI), pages 24-34. Springer, 2022.
[36] Linhao Qu, Xiaoyuan Luo, Manning Wang, and Zhijian Song. Bi-directional weakly supervised knowledge distillation for whole slide image classification. Advances in Neural Information Processing Systems (NeurIPS), 011155568838888 2022.
[37] Linhao Qu, Yingfan Ma, Xiaoyuan Luo, Manning Wang, and Zhijian Song. Rethinking multiple instance learning for whole slide image classification: A good instance classifier is all you need. arXiv preprint 2tt.::::00000229999 2023.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748-8763 PMLR, 2021.
[39] Jérôme Rony, Soufiane Belharbi, Jose Dolz, Ismail Ben Ayed, Luke McCaffrey, and Eric Granger. Deep weakly-supervised learning methods for classification and localization in histology images: a survey. arXiv preprint arXiv: 190..09999..3.. 2019.
[40] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in Neural Information Processing Systems (NeurIPS). 04-2136-2147 2021.
[41] Xiaoshuang Shi, Fuyong Xing, Yuanpu Xie, Zizhao Zhang, Lei Cui, and Lin Yang. Loss-based attention for deep multiple instance learning. In Proceedings of the AAAI Conference Artificial Intelligence (AAAI), volume 34, pages 5742-5749, 2020.
[42] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in Neural Information Processing Systems (NeurIPS), 30, 2017.
[43] Andrew H Song, Guillaume Jaume, Drew FK Williamson, Ming Y Lu, Anurag Vaidya, Tiffany R Miller, and Faisal Mahmood. Artificial intelligence for digital and computational pathology. Nature Reviews Bioengineering, pages 1-20, 2023.
[44] Chetan L Srinidhi, Ozan Ciga, and Anne L Martel. Deep neural network models for computa- tional histopathology: A survey. Medical Image Analysis, 67:101813 2021.
[45] Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning to compare: Relation network for few-shot learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1199-1208, 2018
13

---

[46] Chao Tu, Yu Zhang, and Zhenyuan Ning. Dual-curricllm contrastive iustii-nsstance learning for cancer prognosis analysis with whole slide images. Advances in Neural Information Processing Systems (NeurIPS), 35:29484- 29497, 2022.
[47] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in Neural Information Processing Systems (NeurIPS), 29, 2016.
[48] Xiyue Wang, Jinxi Xiang, Jun Zhang. Sen Yang. Zhongyi Yang. Ming-Hui Wang. Jing Zhang, Wei Yang, Junzhou Huang, and Xiao Han. Scl-we: Cross-slide contrastive learning for weakly- supervised whole-slide image classification. Advances in Neural Information Processing Systems (NeurIPS), 35:1 18009-1808,,,, 2022.
[49] Yongluan Yan, Xinggang Wang, Xiaojie Guo, Jiemin Fang, Wenyu Liu, and Junzhou Huang. Deep multi-instance learning with dynamic pooling. In Asian Conference on Machine Learning (ACML), pages 662-677. PMLR, 2018.
[50] Dingkang Yang. Zhaoyu Chen, Yuzheng Wang, Shunli Wang, Mingcheng Li, Siao Liu, Xiao Zhao, Shuai Huang, Zhiyan Dong, Peng Zhai, and Lihua Zhang. Context de-confounded emotion recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19005-19015, June 2023.
[51] Dingkang Yang. Shuai Huang, Haopeng Kuang, Yangtao Du, and Lihua Zhang. Disentangled representation learning for multimodal emotion recognition. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM), pages 1642-1651, 2022.
[52] Dingkang Yang. Shuai Huang. Shunli Wang, Yang Liu, Peng Zhai, Liuzhen Su, Mingcheng Li, and Lihua Zhang. Emotion recognition for multiple context awareness, In Proceedings of the European Conference on Computer Vision (ECCV), volume 13697, pages 144-162, 2022.
[53] Dingkang Yang, Haopeng Kuang, Shuai Huang, and Lihua Zhang. Learning poddllty-sppccine and -agnostic representations for asynchronous multimodal language sequences. In Proceedings of the 30th ACM International Conference on Multimedia (ACM MM), pages 1708-1717, 2022.
[54] Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learning networks. Medical Image Analysis, 65:101789, 2020.
[55] Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training, arXiv preprint arXiv:2111.07783 2021.
[56] Hongrun Zhang, Yanda Meng. Yitian Zhao, Yihong Qiao, Xiaoyun Yang, Sarah E Coupland, and Yalin Zheng. Difd-mil: Double-tie -ier feature distillation multiple instance learning for histopathology whole slide image classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18802-18812, 2022.
[57] Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey. arXiv preprint arXiv:2304.00683 2023.
[58] Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-fres clip-adapter for better vision- languanlanguage modeling. arXiv preprint urXiv:2111.0390 2021.
[59] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu, Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 16816-16825, 2022.
[60] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009)22337-2348,
2022. 2022.
[61] Xinliang Zhu, Jiawen Yao. Feiyun Zhu, and Junzhou Huang. from whole slide histopathological images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7234-7242, 2017.
Wsisa: Making survival prediction
14


Reviews:
[{"review_id": "43rMLjtb4D", "reviewer": "Reviewer_VPd6", "summary": "This paper introduces an innovative problem setting known as Few-shot Weakly Supervised WSI Classification (FSWC), wherein the availability of labeled WSIs is severely limited. In order to address this challenge, the authors propose a novel Two-level Prompt Learning MIL framework, named TOP, which leverages the power of the VL model and GPT-4 to enhance the learning process of the model.", "questions": "It would be greatly appreciated if the author could consider making the code, trained models, and specific train-validation-test data splits publicly available for the associated public datasets used in the different methods. This step is crucial for ensuring the reproducibility of the results. Access to these resources would significantly facilitate the verification and replication of the finding. Other questions please refer to the weekness part.", "limitations": "Please refer to the weekness part.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThis paper demonstrates exceptional writing quality, distinguished by its clear and well-organized structure. The authors adeptly utilize intuitive figures and well-crafted paragraphs to effectively summarize their methods and contributions. Furthermore, the inclusion of extensive experiments and detailed ablation studies serves as a robust validation for their proposed method.\n2.\tThe proposed instance prompt guided pooling is a captivating and fitting approach for the vision language model task. It exhibits an intriguing method that holds potential for improving the performance of the model in this context.\n", "weaknesses": "1.\tThe problem addressed in this paper is not entirely novel within the field of WSIs, as previous studies like MI-Zero[1] have also recognized the challenge of limited labeled WSIs.\n2.\tThe utilization of pathology language prior knowledge derived from GPT-4 is indeed an interesting aspect of this paper. However, it is essential to address the concerns and questions raised regarding the validity and importance of this knowledge in the task at hand. To assess the correctness and relevance of the knowledge obtained from GPT-4, the paper should ideally outline an evaluation method or standards based on the output of GPT-4. This would help establish the reliability and usefulness of the information derived from the model. Additionally, the authors should clarify their rationale for choosing GPT-4's knowledge over manually designed and professionally validated visual descriptions. It would be beneficial to discuss how they compared the performance and efficacy of the two approaches and why GPT-4-based knowledge was deemed more appropriate. Regarding the generation of questions for GPT-4, it is crucial to provide insights into the process of selecting and designing these questions. The paper should clarify whether there were any specific criteria or standards used to ensure the validity and reasonableness of the questions. Addressing potential biases introduced by the question selection process is also important. Considering that GPT-4 may undergo updates over time, it is essential to acknowledge the potential impact on the answers generated by the model. The authors should discuss the implications of model updates on the overall performance and reproducibility of their proposed framework. In summary, addressing these questions and providing more detailed explanations would enhance the clarity and credibility of the paper, particularly concerning the validity and reliability of the pathology language prior knowledge derived from GPT-4.\n3.\tThe technical novelty of the paper is acknowledged to be limited. Although the paper introduces the instance prompt guided pooling method, which is derived from prototype learning and presents some novelty, it may not be considered sufficient to significantly differentiate it from existing approaches. Furthermore, it appears that the use of a Vision Language (VL) model to extract features and the utilization of learnable prompts from CoOp are commonly employed techniques within the field[1]. These aspects may not contribute significantly to the novelty of the proposed framework. Considering these limitations, it would be valuable for the authors to address the relative novelty of their contributions more explicitly. It could be helpful to discuss how their approach builds upon or improves existing methods, and to identify the specific aspects that differentiate their work from prior research in the field.\n4.\tI have a concern regarding the comparison and reporting of results in the paper. Randomly training the network multiple times with different labeled bags and reporting the highest performance of each method may introduce some ambiguity and make the results appear misleading or tricky. To ensure a more robust and fair comparison, it would be preferable to adopt a standardized evaluation methodology. This could involve training and testing the network multiple times using consistent sets of labeled bags for each method, and then reporting the average performance across these trials. This approach would provide a more reliable representation of the methods' benefits and allow for a more accurate comparison between them. By employing a standardized evaluation methodology, the authors would enhance the credibility and trustworthiness of their reported results, addressing concerns about potential biases or inconsistencies introduced by the current approach.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1 & Q3**. Novelty compared with MI-Zero.\n\n**Response**. \n\nOur approach differs significantly from MI-Zero [1] in the following ways: \n\n*a)* MI-Zero focuses on training a large model like CLIP for instance-level or slide-level zero-shot transfer and classification, while we leverage CLIP as a backbone for instance and slide-level few-shot learning from a prompt-learning perspective. Our concepts and experimental approaches are distinct. \n\n*b)* As highlighted by reviewer 5SeZ in \"Strengths\", our work explores both instance and bag-level prompts, while MI-Zero only explores \"instance-level\" prompts and uses simple top-k pooling solely for bag-level classification. \n\nOur work presents notable innovations: \n\n*a) Conceptually*, we address the novel Few-Shot Weakly Supervised WSI Classification (FSWC) problem and pioneer bag and instance-level prompt learning with large vision-language model (CLIP) and large language model (GPT-4) in WSI classification. \n\n*b) Technically*, we propose a Two-level Prompt Learning MIL framework. At the instance level, we use pathology language prior knowledge from GPT-4 to guide feature aggregation into bag features. At the bag-level, we introduce bag-level pathology categories and visual pathology descriptions as prior knowledge for few-shot learning supervised by bag labels. These ideas are fundamentally different from previous studies.\n\n**Q2**. Validity, rationale, motivation and potential model updates' impact for using GPT-4’s knowledge.\n\n**Response**. \n\n*a) Validity and rationale*: We rigorously reviewed pathology knowledge descriptions from GPT-4 with three senior pathologists and found them accurate and detailed. Literature [2] supports GPT-4's reliability in producing medical domain knowledge due to its vast medical expertise in training data.\n\n*b) Motivation and importance*: Leveraging GPT-4's knowledge as templates enhances efficiency versus manual design. This approach aligns with the few-shot learning goal, easing pathologist annotation. Manual templates might not cover all aspects; specialized doctors' templates could be needed for varied cancer types/tasks. Additionally, different doctors' descriptions vary, lacking a standardized manual description. By leveraging GPT-4's versatility, our aim is to attain knowledge descriptions for multiple cancer types and tasks while avoiding manual domain biases.\n\n*c) Impact of model updates*: GPT-4's language descriptions contributed to training pathology models in our research. We will publicly share all used descriptions, codes, and models. This disclosure ensures reproducibility in reported tasks without the need of invoking GPT-4 for inference or new training. GPT-4’s upgrades won't influence current outcomes. We'll explore if GPT upgrades generate new descriptions and their effect on results.\n\n**Q4**. Consistent labeled bag sets for each method.\n\n**Response**. \n\nFirstly, \"training five times with non-fixed labeled bags\" in the original text means random selection of different labeled bags in each run based on shot count to compose the training set. The test set is pre-divided and remains unseen during training. All comparative methods use the same training and test sets in each run, ensuring fair comparisons.\n\nIn the few-shot learning setup, limited labeled bags for training can cause performance disparities due to varying bag representativeness (like selecting slides with larger or smaller tumor areas). This leads to high standard deviation (STD) across runs with different labeled bags for training. This work is one of the first few-shot WSI learning studies, and there is not a relevant benchmark. Therefore, we conducted five randomized trainings with non-fixed labeled bags, studying the algorithm's holistic performance across diverse slide representativeness. As suggested by multiple reviewers, we supplemented five runs on Camelyon16 dataset using non-fixed bags, reporting Mean AUC (left panel) and STD (right panel) in Table 1 (bag AUC) and Table 2 (instance AUC). As the shot count increases, standard deviations of all methods tend to stabilize. Our proposed method still achieves the best performance. Mean AUC and STD on the other datasets will be provided in a camera-ready version. \n\nFurthermore, we supplemented five rounds of training using fixed labeled bags, employing the same bag in each run for training. Mean AUC (left panel) and STD (right panel) are presented in Table 3 (bag AUC) and Table 4 (instance AUC). The consistent labeled bags notably decrease variability and our method still achieves the best performance.\n\n[1] Lu et al. Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images, CVPR. (2023).\n\n[2] Nori et al. Capabilities of gpt-4 on medical challenge problems. arXiv:2303.13375 (2023).\n\n**Q5**. Code, data and model availability.\n\n**Response**. \n\nWe will publicly share all used descriptions, codes, and models.\n\n\n\n\n"}, {"review_id": "bsrTEttSCi", "reviewer": "Reviewer_5SeZ", "summary": "The authors introduce a two-level prompt learning framework for label-efficient classification of WSIs using pretrained visual language encoders. The main novelty of the proposed method seems to be that at the instance-level, domain-knowledge + GPT-4 guided prompt prototype groups are used to guide pooling of instance embeddings into the slide-level embedding and while the use of learnable prefix tokens + pretrained text embeddings for few-shot classification follows closely of the established V+L few-shot literature. Experimentation was performed on several WSI datasets (CAMELYON16, TCGA, and an in-house cervical cancer dataset).\n", "questions": "Summarizing the concerns above:\n1. What is the performance when using the median versus the max?\n2. Effect of reducing the # of prefix tokens so that the total # of parameters is the same?\n3. Ablation study concerning the auxiliary loss?", "limitations": "- Limitations discussed in this work include the effectiveness of the prompt depending on the quality of visual representations.", "rating": 4, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 3, "strengths": "- This work explores several interesting ideas regarding instance- and bag-level prompts. Previous and concurrent works (e.g. - MI-Zero, PLIP [1,2]) have only explored the application of \"instance-level\" prompts (text prompts correlated with pathology region-of-interests) for computational pathology. The application of using bag-level prompts in combination of with GPT-4 provides new mechanisms for training and evaluating models for slide-level tasks in pathology.\n\n1. Lu et al. 2023, Visual Language Pretrained Multiple Instance Zero-Shot Transfer for Histopathology Images. CVPR 2023.\n2. Huang et al. 2023, Leveraging medical Twitter to build a visual–language foundation model for pathology AI. bioRxiv 2023.", "weaknesses": "While the study presents interesting ideas which the reviewer believes to be valuable to the machine learning and computational pathology community. the reviewer has several concerns regarding the evaluation framework and ablation studies.\n\n1. In the methodological details of this work, because the results in the few-shot settings can be highly variable, therefore for each shot, 5 randomly sampled sets of training bags are used, and the highest performance for each model is reported. The reviewer find this practice somewhat questionable given that stability/consistency of the algorithm should be an important factor in deciding which algorithms works well in the few-shot setting. By only reporting the highest performing run on the test set, we can have the scenario where Algorithm 1 obtains consistently reasonable, above chance performance (e.g. [0.7, 0.7, 0.7, 0.7, 0.7]) while Algorithm 2 obtains performance below chance but a single lucky run (e.g. [0.4, 0.4, 0.4, 0.4, 0.75]) and the conclusion will be that Algorithm 2 outperforms Algorithm 1 by a wide margin of 5% - which is not a reasonable conclusion. In the few-shot setting reported in the paper, if the SD of the 5 runs can be as high as 10% - how can the reviewer be certain the proposed model in fact does outperform the other methods consistently when only the highest performing run is reported? I would encourage this work to instead use the median for comparing the different models instead of the max, which is more robust to outliers compared to the mean. Additionally, the exact numbers of all 5-runs should be reported as a box plot in the supplement, and other metrics besides AUC (e.g. balanced accuracy) should be reported as well.\n\n2. This work does not make comprehensive comparison to other methods/in the ablation experiments. An important concern being that the proposed method uses both learnable prefix tokens at the instance level and at the slide-level compared to CoOP, therefore effectively doubling the number of learnable parameters. (a) What happens when the proposed method uses the same number of learnable parameters as CoOP? (b) What happens if we cut the number of prefix tokens in half at both the slide-level and the instance-level relative to CoOP such that the total parameter count is the same?\n\n3. Some design choices in the proposed method do not seem to be properly ablated, for instance. During training, an auxiliary loss (equation 9) is used to encourage de-correlation of different prompt prototypes, but the effectiveness of this choice does not seem to be ablated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. Using the median versus the max.\n\n**Response**. \n\nFirstly, \"training five times with non-fixed labeled bags\" in the original text means random selection of different labeled bags in each run based on shot count to compose the training set. The test set is pre-divided and remains unseen during training. All comparative methods use the same training and test sets in each run, ensuring fair comparisons.\n\nIn the few-shot learning setup, limited labeled bags for training can cause performance disparities due to varying bag representativeness (like selecting slides with larger or smaller tumor areas). This leads to high standard deviation (STD) across runs with different labeled bags for training. This work is one of the first few-shot WSI learning studies, and there is not a relevant benchmark. Therefore, we conducted five randomized trainings with non-fixed labeled bags, studying the algorithm's holistic performance across diverse slide representativeness. As suggested by multiple reviewers, we supplemented five runs on Camelyon16 dataset using non-fixed bags, reporting Mean AUC (left panel) and STD (right panel) in Table 1 (bag AUC) and Table 2 (instance AUC). As the shot count increases, standard deviations of all methods tend to stabilize. Our proposed method still achieves the best performance. Mean AUC and STD on the other datasets will be provided in a camera-ready version. \n\nFurthermore, we supplemented five rounds of training using fixed labeled bags, employing the same bag in each run for training. Mean AUC (left panel) and STD (right panel) are presented in Table 3 (bag AUC) and Table 4 (instance AUC). The consistent labeled bags notably decrease variability and our method still achieves the best performance.\n\n**Q2**. Comparison with CoOp using equal learnable parameters\n\n**Response**. \n\nWe conducted the experiment on the Camelyon 16 dataset, and the results are presented in Table 6 (mean bag AUC) and Table 7 (mean instance AUC). Experiments marked with the same \"*\" or \"#\" have the same learnable parameters.\n\n*a)* We maintained the baseline's parameter size (1. CoOp+Attention pooling) and reduced the tokens in (4. Our approach) and (3. Ablation method) to match the baseline's size. Despite this reduction, our method still outperformed others marked with \"*\" in Table 6 and Table 7.\n\n*b)* Comparing (5. CoOp+Prompt guided pooling) and (6. Our approach) with equal parameter size (#), our method again achieved superior performance in Table 6 and Table 7.\n\n*c)* Although reducing the parameter size led to some performance degradation, our approach consistently achieved the best performance.\n\n**Q3**. Ablation study concerning the auxiliary loss.\n\n**Response**. \n\nThe auxiliary loss aims to separate instance prototypes learned by each instance prompt, ensuring distinct phenotypes representing WSIs. Crucial instance prototypes for slide classification stand out during aggregation. New ablation and sensitivity tests on the Camelyon16 dataset (Table 5) demonstrate that our method is not highly sensitive to the loss weight, but its addition significantly improves performance compared to not using it.\n\n\n\n\n\n\n\n\n\n\n\n"}, {"review_id": "jDhJQgciD6", "reviewer": "Reviewer_7zsM", "summary": "This work proposes TOP - a framework using Vision-Language models for few shot Multiple Instance Learning on pathology datasets. TOP uses features generated by passing natural language prompts through a LLM to guide the pooling process for the bag level aggregation in MIL. Results show improvements over other few shot MIL approaches in 3 pathology datasets.", "questions": "- For comparison on instance-level predictions, [1] uses the Additive MIL framework to get exact instance-level classwise predictions. Can this method be added to the comparison?\n- Any specific reason for restricting the learnable params to 10 tokens for instance and bag prompts? ", "limitations": "- Can the authors give a ballpark estimate of the cost associated with using LLMs for the description generation as a function of the number of prototypes?", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Few shot learning for MIL in histopathology is important since the effective number of data points is greatly reduced when predicting at WSI-level instead of patch-level. This paper thus addresses this highly relevant problem by providing bag level predictions with only a few training bags.\n- The prompt-guided pooling idea, where similarity of natural language embeddings of text descriptions of different kinds of pathology to image embeddings is used to ground and calculate the instance-level weights, makes a lot of intuitive sense.", "weaknesses": "- My main concern pertains to section 3 in supplementary material, where it is mentioned that \"..we randomly trained the network five times with different labeled bags and reported the highest performance of each method..\". This implies that the performance comparison is biased towards TOP.\n- There can be additional ablations around using pathology specific LLMs [1] instead of generic ones to check if this improves the large variance in performance for TOP across lower number of shots\n- The motivation behind adding the correlation loss between the instance prototypes can be clarified further. Also, details around the sensitivity of the model performance to the relative weight of this loss should be shared.\n\n[1] - Santos et al, PathologyBERT - Pre-trained Vs. A New Transformer Language Model for Pathology Domain\n\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. performance comparison is biased.\n\n**Response**. \n\nFirstly, \"training five times with non-fixed labeled bags\" in the original text means random selection of different labeled bags in each run based on shot count to compose the training set. The test set is pre-divided and remains unseen during training. All comparative methods use the same training and test sets in each run, ensuring fair comparisons.\n\nIn the few-shot learning setup, limited labeled bags for training can cause performance disparities due to varying bag representativeness (like selecting slides with larger or smaller tumor areas). This leads to high standard deviation (STD) across runs with different labeled bags for training. This work is one of the first few-shot WSI learning studies, and there is not a relevant benchmark. Therefore, we conducted five randomized trainings with non-fixed labeled bags, studying the algorithm's holistic performance across diverse slide representativeness. As suggested by multiple reviewers, we supplemented five runs on Camelyon16 dataset using non-fixed bags, reporting Mean AUC (left panel) and STD (right panel) in Table 1 (bag AUC) and Table 2 (instance AUC). As the shot count increases, standard deviations of all methods tend to stabilize. Our proposed method still achieves the best performance. Mean AUC and STD on the other datasets will be provided in a camera-ready version.\n\nFurthermore, we supplemented five rounds of training using fixed labeled bags, employing the same bag in each run for training. Mean AUC (left panel) and STD (right panel) are presented in Table 3 (bag AUC) and Table 4 (instance AUC). The consistent labeled bags notably decrease variability and our method still achieves the best performance.\n\n**Q2**. Ablations using pathology specific LLMs [1] instead of GPT-4’s ones.\n\n**Response**. \n\nWe thoroughly examined the model in [1], but it appears restricted to masked language modeling and lacks question-answering abilities like GPT-4. Nevertheless, the concept of using pathology-specific LLMs to generate domain-tailored knowledge shows promise. We intend to investigate this approach further in our future research.\n\n**Q3**. The motivation, ablation and sensitivity tests of auxiliary loss.\n\n**Response**. \n\nThe auxiliary loss aims to separate instance prototypes learned by each instance prompt, ensuring distinct phenotypes representing WSIs. Crucial instance prototypes for slide classification stand out during aggregation. New ablation and sensitivity tests on the Camelyon16 dataset (Table 5) demonstrate that our method is not highly sensitive to the loss weight, but its addition significantly improves performance compared to not using it.\n\n**Q4**. Add Additive MIL [1] to the comparison.\n\n**Response**. \n\nOur prompt learning framework can be combined with any bag-level and instance-level MIL methods, including the Additive MIL [1] mentioned. We will include this experiment in the camera-ready version of our work.\n\n**Q5**. Reason for restricting the learnable params to 10 tokens.\n\n**Response**. \n\nToken selection is based on empirical choices, and other quantities can be used in practice. For a comparison of parameter size experiments, please see **Response to Q5 of reviewer 5SeZ**.\n\n**Q6**. Ballpark estimate of the cost using LLMs for generation.\n\n**Response**. \n\nWe employed GPT-4's question-answering mode to obtain slide-level and instance-level knowledge descriptions. The generation of prompts for a single task takes around 2 minutes, and experienced pathologists spend approximately 10-20 minutes organizing and reviewing these prompts. These descriptions do not need to be regenerated during inference.\n\n\n\n\n"}, {"review_id": "dHuSYPPOck", "reviewer": "Reviewer_r2fX", "summary": "The paper introduces a new approach called Few-Shot Weakly Supervised Learning for Pathology Whole Slide Image classification (FSWC), which aims to classify bags and instances within a WSI with only a limited number of labeled bags. The proposed solution utilizes a large language model, GPT-4, and prompt learning. The approach leverages CLIP to extract instance features for each patch and uses a prompt-guided pooling strategy to aggregate these instance features into a bag feature. The language prior knowledge is obtained using GPT-4 in a question-and-answer mode at both the instance and bag levels. The method is evaluated on three real WSI datasets encompassing breast cancer, lung cancer, and cervical cancer, and demonstrates notable performance in bag and instance classification. ", "questions": "1.Unlike general few-shot scenarios，what does shot represent in medical images ? And how is the dataset divided ?\n2.In one part of the instance-level prompt, various instance phenotypes are generated. For different datasets, whether the generated functional descriptions are the same or not, without professional knowledge, how to determine these functional descriptions ?\n3.What is learnable in prompt learning ?\n4.The description of methodology is not clear enough. For example, how does the guided pooling work ?\n5.More comparative experiments based on MIL should be added.\n6.The reviewer would suggest the author to proofread the manuscript and the quality of the writing and the presentation should be significantly improved.\n", "limitations": "n/a", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "\nThe idea is interesting, and is one of the early works trying to apply large language and cross-modal models to WSI (Whole Slide Imaging) image analysis problems. Due to the knowledge learned wit the prompt, requirement on large number of traing samples can thus be alleviated.", "weaknesses": "While the idea is interesting, there are significant problems in presentation and languages of the paper. Lots of technical details are not clearly explained. The experiments are not intensive as well, more sota methods need to be compared.\n\nPls see the section of questions for details.\n\nAfter reading other reviewer's comments and the rebuttal, I think the paper reauires significant works to improve it, and currently it's not ready to be included in NeurIPS", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1**. Meaning of Shot and Division of the dataset.\n\n**Response**. In Few-shot WSI classification, \"shot\" refers to the number of labeled slides. For publicly available datasets, we follow official guidelines to split the data into training (including validation) and testing sets. In-house datasets are randomly divided into training (including validation) and testing sets. In N-shot experiments, the training set uses only N pairs of positive and negative slides for training, and the model is evaluated on the complete testing set. In **Response to Q1 of the General Response**, we have included the results of five trials using randomly selected N slides for training and five trials using fixed N slides for training.\n\n**Q2**. Are generated instance prompts varying from different datasets? how to determine?\n\n**Response**. Instance prototype descriptive prompts vary based on the specific task. For tumor detection and lung cancer subtyping, prompts consider different tissue phenotypes. In the task of determining lymph node metastasis from primary lesions, the focus is on phenotypic information related to tumor cells. Representative prompts for each task are in Supplementary materials Figure 1, 2, and 3.\n\nWe rigorously reviewed pathology knowledge descriptions from GPT-4 with three senior pathologists and found them accurate and detailed. Literature [1] supports GPT-4's reliability in producing medical domain knowledge due to its vast medical expertise in training data.\n\n[1] Nori et al. Capabilities of gpt-4 on medical challenge problems. arXiv:2303.13375 (2023).\n\n**Q3**. What is learnable in prompt learning?\n\n**Response**. In our approach, the last 10 tokens in the bag-level and instance-level prompt groups (referred to as learnable prompts) are trainable and treated as learnable parameters. These tokens are encoded into 10 features of 512 dimensions. On the other hand, other descriptive prompts are fixed and directly encoded by the language encoder in CLIP. During the training process, the tokens in the learnable prompts are dynamically learned. For detailed formulas and training methods, please refer to the main text, Section 3.2.\n\n**Q4**. How does the guided pooling work?\n\n**Response**. Prompt guided pooling (PGP) is designed for few-shot WSI classification (FSWC). The key challenge in MIL is aggregating instance features critical for slide classification. In FSWC, limited supervised information hinders the effectiveness of max/mean pooling methods in aggregating crucial instances. Using a separate attention module may lead to overfitting. PGP utilizes GPT-4 to generate visual descriptions of instance phenotypes, guiding feature aggregation. It calculates similarity weights between instance features and text prototypes, then uses the weighted average of all instance features as the bag feature.\n\n**Q5**. More comparative experiments based on MIL should be added.\n\n**Response**. Limited research exists on prompt learning+MIL for Few-Shot Weakly Supervised WSI Classification (FSWC), and methods available for direct comparison are quite limited. We constructed four baselines based on the SOTA few-shot learning methods CoOp and Linear Probe: Linear-Probe+Mean/Max/Attention pooling, and CoOp+Attention pooling. More comprehensive experiments are supplemented in the rebuttal and please see **Response to Q1 and Q2 of General Response** and **Response to Q2 of Reviewer 5SeZ**. In future work, we aim to compare more MIL methods under the FSWC setting to advance research in this area.\n\n**Q6**. Improve the quality and presentation.\n\n**Response**. We will further improve the paper in the camera-ready version.\n\n\n\n\n\n\n\n"}]

Author rebuttal:
We sincerely thank all reviewers for your valuable comments. We first reply to questions raised by multiple reviewers and then other questions from every reviewer. We have uploaded all seven tables into a PDF file, and we sincerely appreciate your downloading and reviewing them.

**Q1**. Average AUC for 5 runs instead of max AUC. (Reviewer VPd6, 5SeZ and 7zsM)

**Response**. 

Firstly, "training five times with non-fixed labeled bags" in the original text means random selection of different labeled bags in each run based on shot count to compose the training set. The test set is pre-divided and remains unseen during training. All comparative methods use the same training and test sets in each run, ensuring fair comparisons.

In the few-shot learning setup, limited labeled bags for training can cause performance disparities due to varying bag representativeness (like selecting slides with larger or smaller tumor areas). This leads to high standard deviation (STD) across runs with different labeled bags for training. This work is one of the first few-shot WSI learning studies, and there is not a relevant benchmark. Therefore, we conducted five randomized trainings with non-fixed labeled bags, studying the algorithm's holistic performance across diverse slide representativeness. As suggested by multiple reviewers, we supplemented five runs on Camelyon16 dataset using non-fixed bags, reporting Mean AUC (left panel) and STD (right panel) in Table 1 (bag AUC) and Table 2 (instance AUC). As the shot count increases, standard deviations of all methods tend to stabilize. Our proposed method still achieves the best performance. Mean AUC and STD on the other datasets will be provided in a camera-ready version. 

Furthermore, we supplemented five rounds of training using fixed labeled bags, employing the same bag in each run for training. Mean AUC (left panel) and STD (right panel) are presented in Table 3 (bag AUC) and Table 4 (instance AUC). The consistent labeled bags notably decrease variability and our method still achieves the best performance.

**Q2**. The motivation, ablation and sensitivity tests of auxiliary loss. (Reviewer 5SeZ and 7zsM)

**Response**. 

The auxiliary loss aims to separate instance prototypes learned by each instance prompt, ensuring distinct phenotypes representing WSIs. Crucial instance prototypes for slide classification stand out during aggregation. New ablation and sensitivity tests on the Camelyon16 dataset (Table 5) demonstrate that our method is not highly sensitive to the loss weight, but its addition significantly improves performance compared to not using it.

**Q3**. Validity, rationale, motivation and potential model updates' impact for using GPT-4’s knowledge. (Reviewer VPd6 and r2fX)

**Response**. 

*a) Validity and rationale*: We rigorously reviewed pathology knowledge descriptions from GPT-4 with three senior pathologists and found them accurate and detailed. Literature [1] supports GPT-4's reliability in producing medical domain knowledge due to its vast medical expertise in training data.

*b) Motivation and importance*: Leveraging GPT-4's knowledge as templates enhances efficiency versus manual design. This approach aligns with the few-shot learning goal, easing pathologist annotation. Manual templates might not cover all aspects; specialized doctors' templates could be needed for varied cancer types/tasks. Additionally, different doctors' descriptions vary, lacking a standardized manual description. By leveraging GPT-4's versatility, our aim is to attain knowledge descriptions for multiple cancer types and tasks while avoiding manual domain biases.

*c) Impact of model updates*: GPT-4's language descriptions contributed to training pathology models in our research. We'll publicly share all used descriptions, codes, and models. This disclosure ensures reproducibility in reported tasks without the need of invoking GPT-4 for inference or new training. GPT-4’s upgrades won't influence current outcomes. We'll explore if GPT upgrades generate new descriptions and their effect on results.

[1] Nori et al. Capabilities of gpt-4 on medical challenge problems. arXiv:2303.13375 (2023).



Meta-review: While some concerns about the evaluation of the method remain, the ideas on how to generate guidance in the few shot bag-level scenario are novel and will potentially have an impact on the field. The authors responses and comparisons appear satisfactory to the AC.
Decision: Accept (poster)