FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification
Authors: Kexue Fu
xiaoyuan Luo
Linhao Qu
Shuo Wang
Ying Xiong
Ilias Maglogiannis
Longxiang Gao
Manning Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Whole Slide Image Classification
Few-shot Learning
Vision-Language Model Adaption
Multimodal Large Model
Primary area: machine_vision
TL;DR: We propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification.  This paradigm can effectively reduce the fine-grained annotation cost of WSI while fully utilizing limited WSI data.

Abstract: The expensive fine-grained annotation and data scarcity 
  have become the primary obstacles for the widespread adoption of deep learning-based Whole Slide Images (WSI) classification algorithms in clinical practice. Unlike few-shot learning methods in natural images that can leverage the labels of each image, existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training in order to avoid expensive fine-grained annotation. They lack sufficient mining of available WSIs, severely limiting WSI classification performance. To address the above issues, we propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification, named FAST. FAST consists of a dual-level annotation strategy and a dual-branch classification framework. Firstly, to avoid expensive fine-grained annotation, we collect a very small number of WSIs at the slide level, and annotate an extremely small number of patches. Then, to fully mining the available WSIs, we use all the patches and available patch labels to build a cache branch, which utilizes the labeled patches to learn the labels of unlabeled patches and through knowledge retrieval for patch classification. In addition to the cache branch, we also construct a prior branch that includes learnable prompt vectors, using the text encoder of visual-language models for patch classification. Finally, we integrate the results from both branches to achieve WSI classification. Extensive experiments on binary and multi-class datasets demonstrate that our proposed method significantly surpasses existing few-shot classification methods and approaches the accuracy of fully supervised methods with only 0.22% annotation costs. All codes and models will be publicly available on https://github.com/fukexue/FAST.

FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification
Kexue Fu 16+ Xiaoyuan 23+ Linhao Qu 23- Shuo Wang Ying Xiong",
Iaas Maglogiannis', Longxiang Gao 161, Manning Wang 231
Key Laboratory of Computing Power Network and Information Security, Ministry of Education, Shandong Computer Science Center (National Supercomputer Center in Jinan), Qilu University of Technology (Shandong Academy of Sciences), Jinan, China Diigitll Medical Research Center, School of Basic Medical Sciences, Fudan University 3 SShanghai Key Lab of Medical Image Computing and Computer Assisted Intervention Fudan University Shandong Provincial Key Laboratory of Computing Power Internet and Service Computing, Shandong Fundamental Research Center for Computer Science, Jinan, China (fukx, gaolx}6sdaa (ooox)x)ddasoorg (19111010030, mnwang) @f udan. edu. cn
University of Piraeus
Abstract
The expensive fine-grained annotation and data scareity have become the primary obstacles for the widespread adoption of deep learning-based Whole Slide Images (WSI) classification algorithms in clinical practice. Unlike few-shot learning methods in natural images that leverage the labels of each image, existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training in order avoid expensive fine-grained annotation. They lack sufficient mining available WSIs, severely limiting WSI classification performance. To address the above issues, we propose a novel and efficient dual-tier few-shot learning paradigm for WSII classification, named FAST. FAST consists of dual-level annotation strategy and a dual-branch classification framework. Firstly, to avoid expensive fine-grained annotation, we collect very small number of WSIs the slide level, and annotate extremely small number of patches. Then, to fully mining the available WSIs, we use all the patches and available patch labels to build cache branch, which utilizes the labeled patches to learn the labels of unlabeled patches and through knowledge retrieval for patch classification. In addition the cache branch, we also construct a prior branch that includes learnable prompt vectors, using the text encoder of visual-language models for patch classification. Finally, we integrate the results from both branches to achieve WSI classification. Extensive experiments on binary and multi-class datasets demonstrate that our proposed method significantly surpasses existing few-shot classification methods and approaches the accuracy of fully supervised methods with only 0. 0.22% annotation costs. All codes and models will be publicly available on https:://// / //gitti giithu.......... com/ foomffukee htt//uukeeeeeeeee FFSS
Introduction
With the advent of Whole Slide Images (WSI) scanners, automated diagnosis based on WSIs has
become a critical problem in the field of computational pathology [53 48 26]. Due to the huge size of WSI [51], deep barning-basee methods typically divide it into series of patches and apply classification models to each patch individually. Fully supervised methods annotate each patch
*Equal Contribution #Corresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

All WSL
All WSS
All WSS
Few-shot data
Pateh-level shott
Slide-level
Slide-level shot
Patch labeling
Slide labeling
Patch labeling
Fev-shot aiatis
Few-shoo datt
Patch-level fow shot
Instance few shot
(b) Bag few shot
(c) Instance and Bag few shot
Figure 1: Different few-shot learning paradigms for WSI classification. (a) The instance few-shot method divides all WSIs into a series patches, then selects a few samples at the patch level and annotates them at the patch level. The red box represents positive samples, and the blue box represents negative samples. (b) The bag few-shot method directly selects few WSIs at the slide level and annotates them weakly the slide level. (c) Our method first selects few WSIs at the slide level, then annotates a few patches for each selected WSI. Compared to (a) and (b), our method significantly reduces annotation costs while providing patch-level supervision information.
and then train the classification model in an end-to-end manner annotation of WSIs requires expert knowledge and is extremely expensive. To overcome these issue, weakly supervised methods formulate the WSI classification task as a multi-instance learning (MIL) problem [8
However, fine-grained
18 27]. In MIL, each WSI (or slide) is bag containing thousands of instances (patches) cropped from the slide. They only use slide-level labels train the classification model. These studies assume the availability of abundant WSIs in clinical settings. However, due to wide staining variations 30 S multiple cancer types 333, and rare diseases [25]. many clinical scenarios can only access a limited number of WSIs. The dual obstacles of fine-grained annotation difficulties and data scarcity severely limit the available supervised information for model training. Therefore, how to avoid expensive fine-graing grained annotations while fully utilizing limited WSIs has become a key issue in the field WSI classification.
55 59 in natural images have demonstrated the effectiveness of few-shot learning under limited data. Inspired by these studies, some methods [8]
Recent works 37 61 10,
42
20
6
49 45 gathered
all patches obtained from dividing WSIs and randomly selected some patches for annotation, as shown in Figure a)) termed instance few shot, and then used existing few-shot learning methods from natural images for classification. These methods provide strong supervision signals for network training, discard large number of unlabeled patches. Our comparative experiments indicate that the methods from natural images perform poorly in few-shot WSI classification. Another methods combine weakly supervised learning with few-shot learning, such as TOP [38], which annotates only few slide labels, as depicted in Figure lbb), termed bag few shot. Compared to instance few shot, bag few shot methods can utilize all patches. However, slide labels belong to weak supervision signals, and the few-shot scenario leads to greater scarcity of supervision information for MIL-based weakly supervised learning methods. Therefore, the accuracy of TOP exhibits significant gap compared the fully supervised learning methods [35]. Overall, the key reason for the poor performance of existing few-shot WSI classification methods is that these methods cannot simultaneously utilize strong supervision from patch labels and the remaining unlabeled patches, lacking sufficient mining of available WSIs.
In this paper, propose novel and efficient dual-tier Few-shot learning pAradigm for WSI classificaTionn, named FAST. consists and sarametee-effficientt dual-branch WSI classification framework, which fully utilizes existing laniunel foundation models and can be rapidly applied to various WSI classification tasks.
unnotation-efficien dual-level WSI annotation strategy
Specifically, we first design dual-level WSI annotation strategy, shown in Fcc. Under this strategy, small number of WSIs are selected the slide-level, followed by labeling small number of patches within each selected WSI. Experts only need to annotate very small number of patches without needing to perform fine-grained annotation on the entire WSI, significantly reducing the cost of fine-grate fine-grained annotation while increasing the speed of annotation. Based the proposed

---

annotation strategy, we formulate the few-shot WSI classification task as dual-tier few-shot learning problem. Unlike conventional few-shot learning [37], which uses an "N-way K-shot" setting, FAST's "shots" consists of two levels: bag and instance. Subsequently, under the setting of dual-tier few-shot learning, we propose a dual-branch few-shot WSI classification framework that combines vision- language (V-L.) models [4. To fully utilize the prior knowledge of V-L models and the limited WSIs, the classification framework includes a cache branch and a prior branch. For the cache branch, we use the image encoder of the V-L model CLIP 41 to extract features of all patches, then construct a cache model using the labeled instances, and finally classify each instance through knowledge retrieval. However, due to the very limited number of annotated instances, the cached model has only limited knowledge and lacks generalization capability. To improve the cache model's s performance, incorporate a large number of unlabeled instances from the WSIs into the cache model, and treat their labels as learnable parameters, which effectively increases the knowledge capacity of the cache model. During training, use annotated instances as supervision to optimize these parameters. For the prior branch, we first use GPT4-V [ to obtain task-related prompts, and then utilize CLIP's text-image matching prior and prompt-learning techniques to design a leamnble visual-language classifier. Finally, we integrate the outputs of the cache and prior branches to obtain the final classification results. This framework based on the foundational model, requires minimal optimization of parameters (cache model and prompt vectors), and can achieve parameter-efficient fine-tuning with small amount of WSIs and labels. Additionally, by leveraging the prior knowledge of the foundational model, FAST can maintain good accuracy and generalization even with extremely limited annotated data. These characteristics make FAST suitable for rapid adaptation various WSI classification tasks. In summary, our main contributions are as follows:
We propose novel few-shot learning paradigm for WSI classification, which achieves high-accuracy WSI classification and rapid adaptation to various WSI classification tasks under low-cost annotation.
We propose an efficient dual-Ievel WSI annotation strategy, which can provide patch-level supervisory information at a cost close to that of slide-level annotation.
We propose a learnable cache model based on the foundation model, which fully utilizes both annotated and unannotated patches. Furthermore, we utilize the prior knowledge of vision-language foundation models to construct linualelanguage classifier, combining both to further enhance the performance of the classification framework.
Extensive experiments demonstrate that our method achieves state-of- ti-thhe-rtt art performance on the CAMELYONIC dataset and the TCGA-RENAL dataset. In addition, compared to fully supervised methods, the annotation cost is only 0.22% of that.
2
Related Work
WSI Classification According to supervised information, WSI classification methods can be divided into two categories: fully supervised learning methods based on patch-label and weakly supervised learning methods based on slide-label. Fully supervised learning methods directly draw inspiration from supervised learning methods in natural images 223
[5| 58 (2]. Relevant
studies on diseases such as breast cancer 44 43), lung cancer 7 3
and prostate cancer indicate that such methods have approached or even surpassed expert diagnostic accuracy However, expensive fine-graines annotation prevents their widespread adoption in clinical
31
22
29
36
33 formulate the WSI classification task as multi-instance learning problem, avoiding expensive fine-grained annotation. Although great progress has been made, these studies rely on large amounts of training data and and cannot address the common issue of data scarcity encountered in practical clinical settings. In this paper, we propose a novel WSI classification paradigm composed of an efficient annotation strategy and a prior knowledge classification framework. The proposed method not only alleviates the poor performance issues of existing methods caused by data scarcity and difficulties in fine- grained annotation, but also enables rapid adaptation to various disease WSI classification tasks.
practice. Weakly supervised learning methods [8 27 39
19
48
28
Few-shot Learning for WSI Classification Inspired by the success of few-shot learning natural images, similar research has emerged in pathology images. Some studies used meta-learning methods, such as MAML. 10 42, prototypical networks [50,9] and matching networks [54) for tasks like whole-genome doubling prediction (6] and cancer classification 44 45). Limited by the scale of
3

---

pre-training data, these methods only achieve limited generalizability. Another group of studies borrowed the idea of fine-tuning V-L models, with related research still in the early stages. For example, CITE 662 applied visual prompt fine-tuning techniques to few-shot learning in pathology images. CLIIPahh 24 fine-tuned learnable network layer on top of a frozen foundation model to transfer foundation model knowledge. However, these studies are only applicable to small-sized pathology image patches and cannot directly handle entire WSI. Qu et al. [38] leveraged the powerful generalization capabilities of CLIP, successfully achieving WSI classification tasks using only slide labels. Limited by the weakly supervised slide label, there still exists a significant gap in classification accuracy compared to fully supervised methods. PLIP [16] and CONCH [34 fine-tuned multimodal large models like CLIP [4I] and CoCa [56] to perform pathology classification tasks. However, they still rely on sarge-scale pathology image-text pair datasets for effective performance. Different from previous works, we propose a dual-level few-shot annotation strategy and a dual-tier few-shot learning formulation approach for WSI classification, which balances annotation cost and granularity. achieving excellent classification accuracy close to fully supervised methods.
Vision-Language Model Adaptation V-L foundation models such as CLIP 41. ALIGN 21 and Florence [57] have demonstrated remarkable generalization capabilities. How to fine-tune these V-L foundation models for adaptation to downstream tasks is crucial. The popular adaptation strategies can be divided into two groups: prompt tuning and feature adapter. CoOp [63] is a representative method of prompt tuning, which optimizes set of learnable prompt tokens to enhance the performance of V-L models in downstream tasks. A representative method of feature adapter is CLIP-Adapter 13. which fine-tunes the CLIP model by adding lightweight residual module after the encoder. Furthermore, Tip-Adapter [61] constructs key-value cache model to integrate the knowledge from few-shot training set directly into the CLIP model, effectively speeding up model convergence during fine-tuning. In the field of natural images, many subsequent works based on Tip-Adapter have also made significant contributions the development of foundation model adaptation. For example, CaFof60) effectively combines the different prior knowledge of various pre- pre-trained models by cascading multiple foundation models. [43 goes step further by considering both general and open-world scenarios, designing a text-guided fusion adapter to reduce the impact of noisy labels. Similarly, for open-world world few-shot learning. (ell-[47) proposes filtering out less probable categories through inverse probability prediction, significantly improving performance. APE[64] proposes an adaptive prior refinement method that significantly enhances computational efficiency while ensuring nigh-precision classification performance. Due to the huge size and the lack of pixel-level annotations, these methods cannot effectively solve the classification problem of WSIs. However, existing methods are only applicable to fully annotated data and cannot effectively utilize unannotated patches in pathology images. Our work is inspired by Tip-Adapter, but it differs from Tip-Adapter The key-value cache model built by Tip-Adapter only allows the key to be learnable. To fully utilize all patches in WSIs, we built a cache model where keys and values are learnable. This effectively facilitates the learning of correct label information for a large number of unlabeled patches, significantly enhancing the performance of CLIP for WSI classification.
3 Method
In this chapter, we first explain how to efficiently annotate WSIs, then describe how to formulate the few-shot WSI classification problem under the novel annotation strategy, and finally introduce the classification framework of FAST.
3.1 Annotation Strategy and Problem Formulation
For a better understanding, we first present fully labeled WSI dataset. Given dataset Xtrain = {X1, Xg...XX consisting of M WSIs, where each WSI X; (æv,1, T2,2..... ...
2,,U,} ,,,,...,,,,)) consisting of U patches. Each patch zi, is considered an instance, and all patches in X; form bag. For X,, we have a bag-level label Y.a. and YB Yrain ( Y YB For Tij. have an instance-level label Wi.j and Y Vrasn (V1.1.91.2-- V1 }. {12,1, U2.2. 12 ,,,.U ...., (yM,1 VM.2.... UM,UU }}. Similarly, for the testing set, the data is denoted as Xtest, and the labels consist of B teat and Y

---

Cache Branch
Fem-ssoo Wis
Few-shot Labels
Xirain
Visual Encoder
Férain
Learaohle
Viram
Imate
Prate
Few Shot Know Retrivval
Xaver
Visual Encoder
Few-shot Knowledge Retrieval
Tost Slide
Priches
goucher
Piror Branch
+ text prompt
angur 1J F
tranegus
yortor
x (3-0)
Pateh-Lare Chasfficatin
Stie-Lavel Clavifictioon
%
Encoder
CLIP's Knowlsdge
GPILY
Figure 2: The structure of the FAST classification framework.
As shown in Figure Tc). the steps of the dual-tier few-shot annotation strategy are as follows: firstly, collect small number of WSIs X Xtrain' smaller than M. Then, for each WSI Xtrain', only L patches are labeled, resulting in Yirii (1 , 21,2....: V1,L } 132,1 ......... ,/,,),... ..... {KK,1, 1,K,,,.......................... J. where L is much smaller than U. In this annotation strategy, the size of Xtrain much smaller than Xtrain and the number of instance labels Etrain' is much smaller than Ytrain-
{X1, X2.... xx), where K is much
In conventional few-shot learning, "N-way K-shot" training set is provided, where N-way represents categories, and K-shot represents K labeled samples. Under our dual-tier few-shot annotation strategy, the 'shot' includes bag-level few-shot and instance-level few-shot. Therefore, formulate the few-shot WSI classification task as "N-way K-bshot & L-ishot", where K-bshot represents K bags, and L--ihoot represents I. laheled instances. During training, only Xtrain and Y Ytrain' are used for training, but during testing, the complete test dataset Xtest and Ytest are used for testing.
3.2 Classification Framework
As shown in Figure 2 the classification framework consists of two branches: learnable image cache branch and a CLIP prior knowledge branch. During training, only a very small number of instances need to be annotated to train the cache model and the learnable prompt tokens. During testing, the prediction results from both the image cache branch and the CLIP prior knowledge branch integrated to obtain instance-level classification results. Finally, the instance-level classification results are pooled to yield a bag-level classification result.
1) Image Cache Branch The cache model consists of feature cache and a label cache, and its construction method is illustrated in Figure 2 First, all patches from these WSIs are input into image encoder for feature extraction, and the extracted features are stored in the feature cache.
Firain VisualEncoder (Xtrain')
The VisualEncodee represents the image encoder within CLIP. Simultaneously, the labels of the annotated instances are transformed into one-hot encoding and stored in the label cache. For the remaining instances without labels, we their labels learnable parameters Ptrain and store them in the label cache.
L train Cat (Yérain) ..Prrann)
Ptrain sents the pseudo-labels of all unannotated instances in Xtrain', where p is a Iearnable high- dimensional vector. As illustrated in the few-shot knowledge retrieval module in Figure through knowledge retrieval, we can obtain the prediction of the cache branch. Specifically, we employ an attention mechanism to implement the knowledge retrieval module, and treat the features and labels key-value pairs. The Features Firain serve keys, denoted by K. The labels Y/ serve as values, denoted by V. The feature of patch be retrieved serve query, denoted by Q The retrieval result o (T).. where (0-) soʃ sollma(()).
....I....,,PPPAA. PI,L+Ki) (P..L+......PLLKKKK). ...., (P..L....... Pr, ...L-+I.....,,LLK.. repre-
We train the cache model using Ftrain and Ytrain'. According to the idea of knowledge retrieval, given a query instance Ztroin Xtrain and its encoded feature ftrain E Firains and the prediction
5

---

result obtained from the few-shot knowledge retrieval is grache = ftrain F Xtrai" Subsequently, the cross-entropy between the predicted values Giyche from the cache model and the ground truth Vad calculated the loss function.
CacheLoss.,j = CE (Dighh, (1.))) - CE (Fig Virain" Vi.j)
The learnable labels in the cache model can be optimized by minimizing the loss function. Addition- ally, to further optimize the feature space of CLIP, we also set Ftrain to be learnable as follow.
P Pirain
min Σ 1.j
CacheLosss.. Firain Firain
min E 1.
CacheLosss,
Peraii
The parameters of the VisualEncoder are frozen. is important to note that in practical training, when there is an excess of unlabeled data, incorporating all the unlabeled instances into the cache model can exceed memory constraints. In such cases, we use K-means clustering algorithm to select representative instances to construct core set. Only the core set is incorporated into the cache model for training and inference.
2) CLIP Prior Knowledge Branch To further leverage the prior knowledge of the vision-language foundation model, we construct prompt-learnable vision-language instance classifier based on CLIP's text encoder. Since the feature spaces of CLIP's text encoder and image encoder are aligned, image classification can be achieved by calculating the similarity between text and images. However, unlike natural image classification, pathological image classification requires more specialized and targeted prompts. As shown in the prior branch Figure 2 we input a small number of annotated instance images into GPT-4V, which generates descriptions of the images combining relevant pathological knowledge, forming prompts for detecting each type of WSI. Then, these setegorr-speciffe prompts feature-extrace by CLIP's text encoder,
f
pleat Encoder (Te), Te = [W] [
where frrunt is the text feature for category c, ftrunt E Ftext. Te is the prompt encoding for category consisting of multiple word vectors [W]. Finally, given the CLIP-encoded test image feature ftests the classification result under N-class text descriptions is,
exp (cos (flext, Steat) /T) ΣS N exp (cos (ffent, Stest) /T)
gerior = p ftest)
where 7 is the temperature coefficient, learned by CLIP during the pre-training phase.
Furthermore, inspired by CoOp, we make the -ppecfcc prompts learnable. Specifically, we add D learnable tokens to the prompts generated for each category, combining them into the final learnable prompts T'. The new text feature as follows,
ptext' Text (Tz), TV = [W),,
Based on the new text feature and the image feature ftj, the classification result of prior branch is,
cxp cos (feer Fis) /T)
perior = - (yi. = fi.) Σn exp (cos (ffen ffa) /T)
Similar to the image cache branch, also use all annotated instances train the prior knowledge branch. The optimization function and loss function of the prior branch are as follows,
[V] = min
Σ Leat where T TeatLoss,, CE [V]
pprior
The parameters of the TextEncoder are also frozen. Finally, we combine the predictions and loss functions from the image cache branch and the prior knowledge branch obtain the overall model's instance-level classification result and loss function,
ÿ = a jeacke + +(1 (1-a): .grior
Loss Σ Lossi + Σ TestLossi

---

where a is the fusion weight of the two branches, which can be considered hyperparameter. We divide the fusion weight a into equal intervals with step size of 100, then sequentially calculate the classification accuracy for each fusion ratio, and finally select the fusion weight that yields the highest classification accuracy as the fusion weight a for this task. Since the entire network has few parameters and fast inference speed, this parameter can be quickly optimized through grid search to obtain the optimal value.
4 Experiments
4.1 Datasets and Few-Shot Scenario Simulation
We evaluated our method the public WSI datasets CAMELYON16 4 and TCGA-RENAL [17 CAMELYON 16 is a binary dataset used to detect whether it contains breast cancer metastasis. TCGA- RENAL is a multi-class dataset for classifying renal cancer subtypes, covering clear cell renal cell carcinoma (ccRCC), papillary renal cell carcinoma (pRCC), and chromophobe renal cell carcinoma (chRCC), For more details about the datasets, please refer the supplementary material A. For few-shot scenario simulation, we first simulate the collection of few-shot WSI data in clinical settings by sampling 1, 2. 4, 8, and 16 WSIs for each category. Then, we simulate the process of expert annotation of few-shot instances within sampled bags, initially selecting 10% instances as a core set by K-means clustering [2], then randomly sampling 16 instances per category from the core set as labeled instances, with others remaining unlabeled. Through this simulation, for CAMELYON and REGGARENALL... datasets, we obtained few-shot training datasets with 1, 2, 4, 8, and 16 bag shots and 16 instance shots. Considering the randomness in few-shot learning, we conducted five random samplings and model trainings for all scenarios and reported the mean and variance of classification results.
4.2 Comparing Methods and Evaluation Metrics
First, we compared FAST with zero-shot learning method CLIP 336 and fully supervised method using all instance-level labels [35]. which can respectively be seen as the lower and upper bounds of deep learning method performance in few-shot WSI classification. Subsequently, due to lack of effective few-shot WSI learning methods, we compared the latest few-shot learning methods Tip-Adapter and Tip-Adapter- [61] in natural images. Bag-level weakly supervised multi-instance learning methods, such as R2T(52,, generally perform worse than instance-level fully supervised methods due to the lack of fine-griined labels. Therefore, this paper does not directly compare with multi-instance learning methods. For implementation details of our method, please refer to the supplementary material A.2 To provide a comprehensive evaluation of these methods, we reported instance-level Area Under Curve and bag-level Area Under Curve (AUC) [5]. Since the TCGA-RENAL. dataset a multi-class classification task, we reported instance-level and bag-level AUCs separately for each category.
4.3 Experimental Results
CAMELYON16 The few-shot WSI classification results the CAMELYON16 dataset shown in Table 1. Firstly, in the zero-shot setting, the instance- AUC and bag-level AUC of zero-shot CLIP are very low, almost unable handle the classification task. In contrast, under extreme few-shot setting with 1 bag shot and 16 instance shots, FAST achieves 0.84 instance-level AUC, which shows significant improvement over zero-shot CLIP. Secondly, as the training samples increase, the performance of FAST shows a significant improvement trend. Meanwhile, our proposed method FAST consistently outperforms the comparison methods Tip-Adapter and Tip-Adapter- across different numbers of samples. Although Tip- Adppter and Tip-Adapter- have achieved significant success in natural images, their performance is poor on few-shot WSI classification task, especially with bagelevel AUC generally below 0.6. We think there are two main reasons for this: (1) there are domain differences between pathology images and natural images, making difficult for the text branch of CLIP to accurately classify instances and even more difficult to classify bags. (2) The limited WSIS make it more difficult to ensure the diversity of instances sampled from these WSIs, resulting in poor generalization of these methods. In contrast, while FAST also utilizes a small number of bags, fully leverages the labeled and unlabeled instances within these bags through a learnable label cache, enabling it to learn more comprehensive instance representations. Finally,
7

---

Table 1: Results on CAMELYON16 16 dataset
Bag Shot
Instance Shot
Methods
Instance-leve AUC
Bag-level AUC
0.6711
Zem-shot PLIP
0.6004
0.5444
Zero-shot CONCH
0,7113
FAST
0.8400-11.0030
0.6933-10.0846
1
16
Tip-Adapter-F
0.7162:+000443
0.5653=1:000600
Tip-Adapter
0.6275==0.0777
0,498=0.0187
FAST
0...54000000300
2
Tip- Tip-Adapter-
0.7200 0.7200110.0:99
0.5748 0.00337
Tip-Adapter
0.61981-0.0623
0..0111
FAST
0.8864110.68
0.7355 0.00855
Tip-Adapter-F
0.6990-1.00880
+400000
Tip-Adapter
0.5601 +000072
0.5321 +0000
FAST
0.0000
0.7742:10.024
lip-Adapter-F
0.7392:1:0.018
0.6045510000444
Tip Adapter
D.6782.10.00
0,4880 11880.1000 1.00097
FAST
0.9151=0.02000
0.8197:100047
16
Tip Sip-Adapeer-I
0.7227:100.008
0.5965.10.024
Tip-Adapter
0.6835-1:0.0133
0.4913-100014
Fully Supervised
FAST
FAST
0.95
Fully Supervised
0.85
Fully Supervised
Instunce-level AUC
Rag-level AUC
0.80
0.90
0.75
0.70
0.85
0.65
0.80
0.60
14x10-4
1.1x10-3
2.1x10 1..
1.4x10-4
1.1x10-3
2.1x10 ...
Ratio of Annotated Instance
Ratio of Annotated Instance
Figure 3: Results of FAST on CAMELYONI 16 dataset under different annotation ratio.
in the setting where only 16 bag shots and 16 instance shots available, proposed FAST method achieves 0.9151 instance-level and 0.8197 bag-level AUC, which is close to the performance of the fully supervised method using all instance annotations. Importantly, the annotation cost of FAST is only 0.22% of that of the fully supervised method, as detailed in Section 4.4 In summary, FAST achieves accuracy close to the fully supervised method with extremely low data collection and annotation costs. This significantly enhances the efficiency of establishing deep learning models in clinical settings and opens up possibilities for widespread clinical adoption of WSI classification algorithms based on deep learning.
TCGA-RENAL In the few-shot experiments on the TCGA-RENAL dataset, the classification results are shown in Table 2 Compared the binary classification task on the CAMELYON16 dataset, the three-class classification task is more challenging. Therefore, the zero-shot CLIP method only achieves around 0.5 instance-level AUC, almost unable to handle the classification task. In the setting with very few shots, such as one or two bags, the results of all methods are relatively poor, with instance-level AUC and bag-level AUC both less than 0.7000. However, our proposed FAST still achieves SOTA performance on most metrics. As the bag shot reaches 4 or more, the classification results begin to significantly improve, and FAST also outperforms other methods on all metrics. Especially in 16 bag shots, FAST significantly outperforms the comparison methods. The average bag-level AUC for chRCC reaches 0.9234, differing by only 0.0033 from the fully supervised method. Similarly, the bag-level AUC for ccRCC and pRCC also reach 0.9254 and 0.92 16 respectively, differing by only 0.0218 and 0.036 from the fully supervised method. This experimental result further demonstrates that, in complex multi-class classification tasks, FAST can approach the fully supervised method with very low annotation costs, achieving the state-of-ther ofttt--rttttttt performance in few-shot learning.

---

Table 2: Results on ICGA-RENAL dataset
Bag Shot
Instance
Methods
Instance-leve level AUC
Bag-level AUC
Shot
ccRCC
PRCC
chRCC
ccRCC
pRCC
chRCC
0
0
Sero-shot CLIP
0,5475
0,5521
0.3640
0.4959
0,5192
0.4811
0.4987
0
Zero-shot PLIP
0.5396
0.6006
0.4774
0.6036
0.6610
0.4905
0,5850
0
Zero-shot CONCH
0,8127
0,9122
0.9131
0,9039
0.89.3
0,9449
0.9141
FAST
1.5935-0.0088
0.6853:10.0445
0.65-88:
0.0066
0161667
0.6033-0000333
0.6921 000016
0.6-18:100009
0.6492
16
Tip-Adapter-F
0.5700=0.0387 0.6533=+000056
0.6230+=00000 0967
0.5981=0.039)
0.6523+:000844
0.6948-+0.111
0.6484
Tip-Adaater
0.5574=0.0503 0.6308=:-000655
0.6017-20.0734
0.5951=0.0080
0.6515=:0.0668
0.6624-t-0.10 1099
0.6363
FAST
0.6245=0.0733 0.6.98:20002229
0.6987=2000552
0.6745=0.0900
0.732/=:00079
0.7329
0.7133
16
'Tip-Adapeer+t
0.6084=006699 0.6820:1:000005
0.6985-20.0481
0.6359=0.1143
0.7391::0.0331
0.748111-000799
0.7077
Tip-Adapter
0.006810.0524 0.6413:100.0554
0.62238
0.6555510.0992
0.6682:1:010334
0.7405:10.0776
00.6877
FAST
0.1107=0.1056 0.7547::000044
0.7652:::000445
0..684=0.1681
0.8260::0.0816
0.8143++0.100
00.8029
4
16
Tip-Adapter-F
0.6587=0.0858 0,7266::::0775
0,7488:::0.0555
0.7220=00, |091
0.7621 0.85
0.7132:4:0.11 1631
0.7324
Tip-Adapter
0.631100.0737 0.6797:::0.0220
0.68355::00005
0.6671=0.1388
0.7274=:000391 0.7856=2:00064
0.7267
FAST
0.7940=0.0522 0.8228:1-000275
0.8098::0000194
0.8953
0.8978::0.0478
0.8964:000048
0.8966
8
16
Tip-Adapter-F
0.7249=0.0529
0.7832=4:000255
0.7918=+0:023
0.7854=0.138)
0.8239=:0.0595
0,7879+#0.005
0.7991
Tip-Adapter
0.6837=0.0567 0.7552:+:0.0275
0.7623+:0.0380
0.7735
0.7787::000026
0.7908::00007
0.7810
EAST
0.8252 0.0428
0.8420:10.01266
0.8609:1000165
0.9254=0.0220
0.9216:10,0233
0.9234:100184
0..9235
16
16
Tip-Adapler-F
0. 0.0709 0.8026.1:0.0151
0.8000011..0268
0.8712.100.776
0.8235.1:00054
0.8531 0.0059
0.8849
Tip-Adapter
0.0911=000484
0.7886:40.0220
0.7851=:00091
0.7624=0.0931
0.7503+:0:0400
0.8877400 .621
0.8031
All
All
Fully Supervised
0.8917
0,9055
0.8977
0.9472
0.9576
0.9267
0.9438
0.95
0.85
Instance- Restance-leve...e A AUC
0.80
0.90
Bug-level AUC
0.75
0.85
0.70
0.80
FAST
0.65
FAST
FAST (only Piror Branch)
FAST (only Piror Branch)
0.60
0.75
FAST (only Cache Branch)
FAST (only Cache Branch)
1
2
4
16
1
2
8
16
Bag Shot
Bag Shot
Figure 4: Comparison of cache branch and prior branch in FAST.
4.4 Annotation Efficiency
To illustrate the annotation efficiency of FAST, we compared the classification results of our method and the fully supervised method under different annotation ratios. For the CAMELYON16 dataset, the results are presented in Figure 3 It can be observed that the classification AUC of FAST rapidly increase with the growth of bag shots. When the bag shot reaches 16, the annotation only accounts for 0.22%. The average bag-level AUC reaches 96.22% of the fully supervised method. This result fully demonstrates the advantage of FAST in annotation efficiency. For the results on the TCAG-RENAL. dataset, please refer to the supplementary material A.3
4.5 Ablation Studies
To analyze the importance of each component in FAST, we conducted a serise of experiments under the conditions of 16 bag shots and 16 instance shots the CAMELYON16 dataset, including whether to use a cache branch, whether to set the feature cache as learnable, whether to use prior branch, and whether utilize unlabeled instances to construct learnable label cache. The results of the ablation experiments are shown in Table 3 The first three rows of the table correspond to zero-shot CLIP, Tip-Adapter, and Tip-Adapte--F respectively. The next three rows correspond FAST using only the prior branch, FAST using only the cache branch, and the complete FAST. Firstly, the first three rows of the table show that the instance-level AUC gradually increases from 0.67 11 0.7277 for the three methods. This indicates that the cache branch can leverage a small amount of supervised information, and making the cache feature learnable can further optimize the feature space distribution of the cache model. However, even the best model's 's instance- AUC is only 0.7227, and the pooled bag-level AUCs are all less than 0.6. Next, from the last three rows of the table, even with only the prior branch, using our proposed prompt-based method, the instance AUCs reach 0.8739 and 0.7931, respectively, which outperformed the results of instance-level and bag-level AUCs of only using the cache branch can reach 0.9165 and 0.8183, respectively, demonstrating that the cache branch is crucial for FAST. By further comparing the third and fifth rows of the table, we can infer the primary reason why FAST's s cache model surpasses that of
and bag-level classification The

---

Table Ablation study of FAST on CAMELYONI6 dataset
Cache Branch
Leamable Feature Cache
Prior Branch
Learnable Label Cache
Instanco-level AUC
Bag-level AUC
0,5409
0,6835=0000135
0.4913=0.0164
0.7227 1: 0.0009
0.5965
0.8739::00161
0:0037
0.9165 1. 0.0213
0.8183-10.0560
00.050
0.8197
Tip- Tip-Adapter-F natural images. Specifically, FAST utilizes a large number unannotated instances in small number of bags through the learnable label cache, thereby significantly improving the capacity and generalization ability of the cache model. Finally, comparing the last two rows of the table, we find that when both cache and prior branches utilized, FAST does not show significant improvement over using only the cache branch. This is because the number of samples is sufficient, and the prior branch primarily plays role when there are fewer samples.
To verify the role of two branchs, we compared the prior branch and cache branch of FAST under different bag shots. The results are shown in Figure When there are only I or 2 bags, the instance classification results of the prior branch are significantly higher than those of the cache branch. The instance and bag classification results combined with both the cache and prior branches also surpass those of using each branch separately, indicating that the prior branch performs better in extreme samples, and the information learned by the prior branch and the cache branch is complementary. As the number of bags increases to or more, the results the cache branch gradually surpass those of the prior branch. Therefore, in extreme few-shot scenarios, FAST is dominated by the prior branch, but as the sample size gradually increases, FAST is dominated by the image branch. This experimental analysis can provide effective guidance for the practical application of FAST. Additionally, we provide further analysis experiments on the number of annotated instance and the number of core sets in the supplementary material A.3
5
Conclusion
In this paper, we propose a dual-tier few-shot learning paradigm FAST for WSI classification, which achieves low-cost WSI annotation, high-accuracy WSI classification, and adaptation to various WSI classification tasks. To achieve these goals, we introduce two key technologies in FAST, including dual-level few-shot annotation strategy and a dual-branch classification framework. The dual-level few-shot annotation strategy effectively alleviates the problem of fine-grained annotation difficulty in WSI by annotating only a small number of patches in a limited number of WSIs. Experimental results demonstrate that the annotation cost of our method is only 0.22% of patch-level full annotation. In the dual-branch classification framework, we construct a cache branch where both features and labels are learnable, fully exploiting the partially annotated data obtained from the dual-level few-shot annotation strategy. Furthermore, combining a vision-language foundation model and prompt tuning technology, we build prior knowledge branch to assist the cache branch in improving classification performance. Through extensive experiments, we demonstrate that FAST can achieve state-of-tht the-arr performance both binary and multi-class classification tasks. However, limitations. For bag-level classification, we simply use pooled instance classification results, ignoring the relationships between instances. In the future, will further explore how instance-level classification results to obtain a better bag-level classification result.
method still has certain
Acknowledgments and Disclosure of Funding
This work was supported by the National Key R&D Program of China under Grant No. 2022ZD01 16800, and the National Natural Science Foundation of China under Grant 62471 149 and Taishan Scholars Program under Grant TSQN20221
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08874, 2023.
[2] Khaled Alsabti, Sanjay Ranka, and Vineet Singh. An efficient k-means clustering algorithm. 1997.
[3] Neha Baranwal, Preethi Doravari, and Renu Kachhoria. Classification of histopathology images of lung cancer using convolutional neural network (cnn). In Disruptive Developments in Biomedical Applications, pages 75 89. CRC Press, 2022.
[4] Babak Ehteshami Bejnordi, Mitko Veta, Paul Johannes Van Diest, Bram Van Ginneken, Nico Karssemeijer, Geert Litjens, Jeroen AWM Van Der Laak, Meyke Hermsen, Quirine F Manson, Maschenka Balkenhol, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. Jama. 318(22):2199-22 2210, 2017.
[5] Andrew P Bradley. The use of the area under the roc curve in the evaluation of machine learning algorithms. Puttern recognition, 30(7): 145-11999, 1997.
[6] Sherry Chao and David Belanger. Generalizing few-shot classification of whole-genome doubling across cancer types. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3382-3392, 2021.
[7] Nicolas Coudray, Paolo Santiago Ocampo, Theodore Sakellaropoulos, Navneet Narula, Matija Snuderl, David Fenyö, Andre I Moreira, Narges Razavian, and Aristotelis Tsirigos. Classifica- tion and mutation prediction from non-small cell lung cancer histopathology images using deep learning. Nature medicine, (5410)):559-157, 2018.
[8] Jessica Deuschel, Daniel Firmbach, Carol Geppent, Markus Eckstein, Arndt Hartmann, Volker Bruns, Petr Kuritcyn, Jakob Dexl, David Hartmann, Dominik Perrin, et al. Multi multi-prototype few-shot learning in histopathology. In Proceedings of the IEEE/CVF international conference on computer vision, pages 620 628, 2021.
[9] Jessica Deuschel, Daniel Firmbach, Carol Geppert, Markus Eckstein, Arndt Hartmann, Volker Bruns, Petr Kuritcyn, Jakob Dexi, David Hartmann, Dominik Perrin, al. Multi hulti-prototype few-shot learning in histopathology. In Proceedings of the IEEE/CVF international conference on computer vision, pages 620-628, 2021.
[10] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference on machine learning, pages 1126-1135. PMLR, 2017.
[11] Junyu Gao, Mengyuan Chen, and Changsheng Xu. Vectorized evidential learning for weakly- supervised temporal action localization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(12):15949 15963, 2023.
[12] Junyu Gao, Tianzhu Zhang, and Changsheng Xu. Learning to model relationships for zero- shot video classification. IEEE transactions on pattern analysis and machine intelligence. 43(10):3476-3444 2020.
[13] Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li. and Yu Qiao. Clip-adapter. Better vision-language models with feature adapters. International Journal of Computer Vision, 132(2):581 -595, 2024.
[14] Le Hou, Dimitris Samaras, Tahsin M Kure, Yi Gao, James E Davis, and Joel H Saltz. Patch- based convolutional neural network for whole slide tissue image classification. Proceedings of the IEEE conference computer vision and pattern recognition, pages 2424-2433, 2016.
[15] Yufan Hu, Junyu Gao, Jianfeng Dong, Bin Fan, and Hongmin Liu. Exploring rich semanties for open-set action recognition. IEEE Transactions on Multimedia, 2023.
11

---

[16] Zhi Huang, Federico Bianchi, Mert Yuksekgonul, Thomas J Montine, and James Zou. A visual language foundation model for pathology image analysis using medical twitter. Nature medicine, 29(9):2307-2316, 2023.
[17] Carolyn Hutter and Jean Claude Zenklusen. The cancer genome atlas: creating lasting value beyond its data. Cell, (73(2):283-285) 2018.
[18] Maximilian Ilse, Jakub Tomezak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127-2136. PMLR, 2018.
[19] Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learning. In International conference on machine learning, pages 2127-2136. PMLR, 2018.
[20] Muhammad Abdullah Jamal and Guo-Jun Qi. Task agnostic meta-learning for few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11719-11727, 2019.
[21] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, fin-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision- saiiunla representation learning with noisy text supervision. International conference on machine learning, pages 4904-4916. PMLR, 2021.
[22] Davood Karimi, Guy Nir, Ladan Fazli, Peter C Black, Larry Goldenberg, and Septimiu Salcudean. Deep learning-based gleason grading of prostate cancer from histopathology images-role of multiscale decision aggregation and data augmentation. IEEE journal of biomedical and health informatics, 24(5):1413-1425 1426, 2019.
[23] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[24] Zhengfeng Lai, Zhuoheng Li, Luca Cerny Oliveira, Joohi Chauhan, Brittany N Dugger, and Chen-Nee Chuah. Clipath: Fine-tune clip with visual feature fusion for pathology image analysis towards minimizing data collection efforts. In Proceedings of the IEEE/CVF International Conference Computer Vision Workshop, pages 2374 2380, 2023.
[25] Junghwan Lee, Cong Liu, Junyoung Kim, Zhehuan Chen, Yingcheng Sun, James R Rogers, Wendy K Chung, and Chunhua Weng. Deep learning for rare disease: A scoping review. Journal of Biomedical Informatics, 135: 104227, 2022.
[26] Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with self-supervised contrastive learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 14318-14328, 2021.
[27] Hang Li, Fan Yang. Yu Zhao, Xiaohan Xing, Jun Zhang, Mingxuan Gao, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Dt-mil: deformable transformer for multi-instance learning on histopathological image. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24, pages 206-216. Springer, 2021.
[28] Hang Li, Fan Yang, Yu Zhao, Xiaohan Xing, Jun Zhang, Mingxuan Gao, Junzhou Huang, Liansheng Wang, and Jianhua Yao. Dt-mil: deformable transformer for multi-instance learning on histopathological image. In Medical Image Computing and Computer Assisted Intervention- MICCAI 2021: 24th International Conference, Strasbourg. France. September 27-October 1, 2021, Proceedings, Part VIII 24, pages 206-216. Springer, 2021.
[29] Wenyuan Li, Jiayun Li, Karthik V Sarma, King Chung Ho, Shiwen Shen, Beatrice S Knudsen, Arkadiusz Gertych, and Corey W Arnold. Path r-cnn for prostate cancer diagnosis and gleason grading of histological images. IEEE transactions on medical imaging, 38(4):945-954 2018
[30] Sylwia Libard, Dijana Cerjan, and Irina Alafuzoff. Characteristics of the tissue section that influence the staining outcome in immunohistochemistry. Histochemistry and Cell Biology, 151:91-96, 2019.
12

---

[31] Min Liu, Lanlan Hu. Ying Tang, Chu Wang, Yu He, Chunyan Zeng, Kun Lin, Zhizi He, and Wujie Huo. A deep learning method for breast cancer classification in the pathology images. IEEE Journal of Biomedical and Health Informatics, 26 26(10):5025-5032 2022.
[32] Shaolei Liu, Xiaoyuan Luo, Kexue Fu, Manning Wang, and Zhijian Song. A learnable self- supervised task for unsupervised domain adaptation on point cloud classification and segmenta- tion. Frontiers of Computer Science, 17(6): 2023.
[33] Lawrence A Loeb, Keith R Loeb, and Jon Anderson. Multiple mutations and cancer. Pro- ceedings of the National Academy of Sciences, 100(3):77-771, 2003.
[34] Ming Y Lu, Bowen Chen, Drew FK Williamson, Richard J Chen, Ivy Liang, Tong Ding, Guil- laume Jaume, Igor Odintsov. Long Phi Le, Georg Gerber, et al. A visual-language foundation model for computational pathology. Nature Medicine, 30(3):863-874, 2024.
[35] Xiaoyuan Luo, Linhao Qu, Qinhao Guo, Zhijian Song, and Manning Wang. Negative instance guided self-distillation framework for whole slide image analysis. IEEE Journal of Biomedical and Health Informatics, 2023.
[36] Muhammad Khalid Khan Niazi, Anil V Parwani, and Metin N Gurean. Digital pathology and artificial intelligence. The lancet oncology, 2055)xe5,,, 2019.
[37] Siyuan Qiao, Chenxi Liu, Wei Shen, and Alan Yuille. Few-shot image recognition by predicting parameters from activations. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7229-7238, 2018.
[38] Linhao Qu, Kexue Fu, Manning Wang. Zhijian Song, et al. The rise of ai language patholo- gists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. Advances in Neural Information Processing Systems, 36, 2024.
[39] Linhao Qu, Xiaoyuan Luo, Shaolei I Liu, Manning Wang, and Zhijian Song. Dgmil: Distribution guided multiple instance learning for whole slide image classification. In International Confer- ence Medical Image Computing and Computer-Assisted Intervention, pages 24-34. Springer, 2022.
[40] Linhao Qu, Manning Wang. Zhijian Song, et al. Bi-directional weakly supervised knowledge distillation for whole slide image classification. Advances Neural Information Processing Systems, 35:15368--15333 2022.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning,
pages 8748-87666 8763. PMLR, 2021.
[42] Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances neural information processing systems, 32, 2019.
[43] Alexander Rakhlin, Alexey Shvets, Vladimir Iglovikov, and Alexandr A Kalinin. Deep con- volutional neural networks for breast cancer histology image analysis. In Image Analysis and Recognition: 15th International Conference, ICIAR 2018, Póvoa de Varzim, Portugal, June 27-29, 2018 Proceedings 15, pages 737-744. Springer, 2018
[44] Kaushiki Roy, Debapriya Banik, Debotosh Bhattacharjee, and Mita Nasipuri. Patch-based system for classification of breast histology images using deep learning. Computerized Medical Imaging and Graphics, 103, 2019.
[45] Nazim N Shaikh, Kamil Wasag, and Yao Nie. Artifact identification in digital histopathology images using few-shot learning. In 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI), pages 1-4, IEFF, 2022.
[46] Shuai Shao, Yu Bai, Yan Wang, Baodi Liu, and Bin Liu. Collaborative consortium of foundation models for ooen-world few-shot learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 4740-4747, 2024.
13

---

[47] Shuai Shao, Yu Bai, Yan Wang, Baodi Liu, and Yicong Zhou. Deil: for open-world few-shot learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 28505- 28514, 2024.
clip
[48] Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang. Jian Zhang, Xiangyang Ji, et al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Advances in neural information processing systems, 34:2136-2147, 2021.
[49] Rishav Singh, Vandana Bharti, Vishal Purohit, Abhinav Kumar, Amit Kumar Singh, and Sanjay Kumar Singh. Metamed: Few-shot medical image classification using gradient-based meta-learning. Pattern Recognition, 120: 108111, 11, 2021.
[50] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[51] Chetan L Srinidhi, Ozan Ciga, and Anne L Martel. Deep neural network models for computa- tional histopathology: A survey. Medical image analywis, 67:101813, 2021.
[52] Wenhao Tang, Fengtao Zhou, Sheng Huang, Xiang Zhu, Yi Zhang, and Bo Liu. Feature re-embedding: Towards foundation model-level performance in computational pathology. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11343-11352, 2024.
[53] Jeroen Van der Laak, Geert Litjens, and Francesco Ciompi. Deep learning in histopathology: the path to the clinic. Nature medicine, 27(5):775-784, 2021.
[54] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[55] Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Few-shot learning via embedding adaptation with set-to-set functions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8808-8888 18188881,,011 2020.
[56] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Learning Research.
[57] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chumyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint ar.
2021.
[58) Mingzhi Yuan, Xiaoshui Huang, Kexue Fu, Zhihao Li, and Manning Wang. Boosting 3d point cloud registration by transferring multi-modality knowledge. 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 11734-1111 11741. IEEE, 2023.
[59] Mingzhi Yuan, Ao Shen, Kexue Fu, Jiaming Guan, Yingfan Ma, Qin Qiao, and Manning Wang. Proteinmae: masked autoencoder for protein surface self- self-supervised learning. Bioinformatics, (9))bddad,724, 2023.
[60] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang, Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1521 155222,, 2023.
[61] Renrui Zhang, Wei Zhang, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, and Hongsheng Li. Tip-adapter: Training-free adaption of clip for few-shot classification. In European conference on computer vision, pages 493-510. Springer, 2022.
[62] Yunkun Zhang, Jin Gao, Mu Zhou, Xiaosong Wang. Yu Qiao, Shaoting Zhang, and Dequan Wang. Text-guided foundation model adaptation for pathological image classification. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 272-282. Springer, 2023.
14

---

[63] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 13009))3337-2348 2022.
[64] Xiangyang Zhu, Renrui Zhang, Bowei He, Aojun Zhou, Dong Wang, Bin Zhao, and Peng Gao. Not all features matter: Enhancing few-shot clip with adaptive prior refinement. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2605-2615, 2023.
15

---

A Supplemental Material
A.1 Datasets
CAMELYON16 CAMELYONI6 [4] contains 400 WSIs of lymph nodes, used to detect the presence of metastatic breast cancer. WSIs containing metastases are labeled positive, while others negative, with pixel-level annotations of the metastatic areas. We first cropped these WSIs at 10x magnification into 512x512 image patches, then removed image patches with entropy less than 15 as background, and marked patches with more than 30% cancer area as positive, resulting in 186,604 image patches, of which 8,117 were marked as positive (4.3%).
This dataset comes from The Cancer Genome Atlas (TCGA) project 17. covering high-resolution WSIs of the three main subtypes of renal cell carcinoma (RCC): clear cell renal cell carcinoma (ccRCC), papillary renal cell carcinoma (pRCC), and chromophobe renal cell carcinoma (chRCC), ccRCC is the most common subtype, characterized by clear cells containing abundant lipids and glycogen. pRCC follows, characterized by papillary structures formed on the surface of tumor cells. chRCC is relatively rare, with larger cells and granular cytoplasm. All WSIs are rigorously reviewed by professional pathologists to ensure the representativeness and quality of the data. To fully verify the effectiveness of our proposed method, organized experts to delineate all WSIs at the pixel level, including delineation of cancerous and non-cancerous areas. Subsequently, through preprocessing processes such as WSI segmentation and background removal similar to CAMELYONIC we obtained instance-level labels for the complete training and testing sets. Finally, all WSIs were divided into training and testing sets in ratio of 70% and 30%, respectively.
TCGA-RENAL
collected 910 WSIs from TCGA, and then
A.2 Implementation Details
FAST employs the image encoder and text encoder from the pre-trained CLIP-RN50 [41]. The cache capacity is typically determined by the number of annotated and unannotated instances in the training set after each sampling. For bag shot greater than 4, we used a core set selection strategy to prevent excessive caching. For the CAMELYONI6 and -REGL datasets, we set the number of core sets to 1000 and 2000, respectively. We set the number of learnable tokens to 10. During training, we utilized the Adam optimizer with learning rates set as follows: 0.001 for the feature cache, 0.01 for the label cache, and 0.001 for the tokens in the prior branch. We fine-tune our model with batch size of 4096 for 20,000 epochs. All models are trained and tested on an RTX 3090 GPU with 24GB memory.
For Aip-Ad We conducted comparative experiments according to the settings of the optimal model in the original Tip-Adapter paper. For aspects that cannot be adapted to the few-shot WSI classification task, we used the following approach. We designed a set of text prompts specifically for pathology images, which has been proven superior in the CONCH comparison experiments we conducted. We used all annotated patches to build the cache model.
A.3 Additional experiments
(1) Annotation Efficiency To provide a more intuitive illustration of the annotation efficiency of method, first calculated the annotation ratio under different bag shots (1, 2, 8, 16), with each bag containing 16 annotated instances. Then, we visualized the classification results of FAST under different annotation ratios and compared them with the fully supervised method. The results for TCGA-RENAS dataset are presented in Figure 5 It can be observed that the classification AUC of FAST rapidly increase with the growth of bag shots. When the bag shot reaches FAST achieves results comparable to fully supervised methods. At this point, the annotation ratio of FAST on the TCGA-RENAN dataset only 0.0067% of the total instance annotation. When the bag shot reaches 16, the annotation only accounts for 0.013% of all instances. Even under such extreme minimal annotation, FAST can still achieve results close to fully supervised methods, especially in bag classification results. For chRCC, pRCC, and ceRCC, FAST's average bag-level AUC reaches 99.64%, 96. 96.24% and 97.70% respectively compared the fully supervised method. This result fully demonstrates the advantage of FAST in annotation efficiency.
16

---

FAST Fully Supervised
0.90
Fally Supervised
Bag levelAUC
0.85
0.75
0.70
0.65
0.60
0.55
8.3x10-6
6.7x10-1
13x10-*...
6.3x10
6.7x10-1
1.3x10't...
Ratio of Amnotaedd Instance
Ratio of Annotated Instance
0.90
Fully Supervised
* Fully Supervised
FAST
0.80
level AUC
(b)
0.75
0.70
0.60
0.60
0.55
of A'Amootated
Ratio Annotated Instance
0.90
FAST
Fully Supervised
Fally Supervised
0.85
0.90
Dag AUC
AUC
ocRC
0.60
0.85
0.75
0.75
0.70
0.65
0.65
83x10
6,7x10
13x10 4,..
833110
6.7x
1.3x10 4..
Ratio of Amooaaed Instance
Ratio of Annotated Instance
Figure 5: Results of FAST TCGA-RENAL dataset under different annotation ratio.
(2) The Number of Annotated Instance The above experiments indicate that achieving good results only require annotating 16 instances per bag. To further investigate the influence of the number of annotated instances per bag, we conducted experiments on the CAMELYON dataset with the number of annotated instances per bag set to 4, 16, and 64, respectively. The results are shown in Figure 6 It can be observed that when the number of bags is small and there are only 1, 2, or bags per class, the instance-level and bag-level classification AUCs are significantly higher when 64 instances are annotated per bag compared to only annotating or instances. However, as number of bags increases to 8 and 16, the AUC of FAST gradually converge. Regardless of whether each bag is annotated with 4, 16, or 64 instances, FAST achieves similar results in instance-level AUC and bag-level AUC. These experimental results indicate that when the number of bags is extremely small, increasing the number annotated instances can effectively improve the performance of FAST. However, when the number of bags increases certain level, even if only small number of instances are annotated in each bag, FAST can achieve results similar to annotating a large number of instances. This indicates that FAST's ability achieve higher accuracy than other methods primarily relies on its learning from unlabeled instances. It also suggests that in practical applications of FAST, if there are large number of bags, the requirement for labeling instances can be appropriately reduced.
(3) Core Set Size Few-shot learning of WSI classification differs from conventional few-shot learning. Even with only several WSIs, they may produce tens of thousands even hundreds of thousands of patches. Therefore, to avoid optimization difficulties caused by excessively large caches,
17

---

Table 4: Results of FAST on the CAMELYONI6 dataset under different core set sizes
Bag Shot
Instance Shot
Core Set Size
Instance-level AUC
Bag-level AUC
100
0.8845=0:0525
0.7295H0.0043
500
0.8770=1000433
0.7210+0.1157
4
16
1000
0.8860=100056
0.7554
2000
0.8985
0.7595+10004
5000
0.9027=0.0342
0.7499
100
0.8827+000003
0.7179
500
0.8003+000162
0.7549+0.0445
8
1000
0.8957=000132
0.7079
2000
0.9075=000122
0.7848
5000
0.9101
0.8025
T00
0,9008 +0.0320
0.7353
500
0.9046-10.0300 10,0301
0.7924+0.0484
16
16
1000
0.9124
0.8078 +0.0626
2000
00.011
0.8240-10.0572
5000
0.9096=100221
0.8082::0.054
0.85
0.90
Bag sev--eeelll AUC
0.75
0.85
0.70
Instance shot:
Instance shot:
Instance shot: 16
0.65
Instance shot: 16
Instance shot:
Instance shot:
Insrance shot:
0.60
Instance shot:
0.75
Number Bag
Number Bag
Figure 6: Results of FAST on CAMELYON16 dataset under different instance shots.
FAST employs a K-means core set selection strategy to control the size of the learnable cache. To analyze the performance of FAST under different core set sizes, we conducted experiments on the CAMELYONI6 16 dataset with 4, 8, and 16 bags and 16 instances per bag. The results are shown in When the core size is only 100 or 500, FAST's bag-level AUC significantly lower, indicating that the core set size at this time is insufficient to cover all representative samples, resulting Joss of large amount of information. However, when the core size reaches 1000 or above, FAST's instance-level AUC and bagelevel AUC achieve good results, indicating that FAST has good robustness to core set size. However, when the core set size 5000 and the number of bags is 16, the results decrease instead, indicating that an overly large learnable cache is difficult optimize. Therefore, we set the core set size to 1000 or 2000 experiments where the number of bags is greater than 4.
Table 4
18

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
In the abstract and the section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: the section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer:
[NA]
Justification:
Not applicable.
Guidelines:
19

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In the abstract.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: In the abstract.
20

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
/ //nips wwtpcccppppcccccccccccc.cccccccc wtwp:l/ccggiideeeeeeeeeeeeeeeeee wwwoodeeumisssooooooioooo
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if' applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In the section A.2 of supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer:
[Yes]
Justification: In the section 4.1 and the section
4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
21

---

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the section A.2 of supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
Our research is beneficial for WSI classification.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: the CAMELYON16 paragraph the section 4.3
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymentt of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinfornation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
22

---

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In the section A.2 supplementary materials.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, haperswwthcode cwm/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
All codes and models will be made publicly accessible.
23

---

Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
24


Reviews:
[{"review_id": "WD9eIEittx", "reviewer": "Reviewer_Bx9c", "summary": "This paper investigates the issue of Whole Slide Images (WSI) classification, a study with practical value. It proposes a new working paradigm that is an improvement based on Tip-Adapter. Theoretically, this new paradigm can effectively address the problem and has strong scalability.", "questions": "see weakness", "limitations": "authors didn't present the limitation.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This study has practical significance, and the proposed method demonstrates strong scalability. The paper is clearly written and easy to understand, with comprehensive experiments.", "weaknesses": "1. This paper lacks some important related work. The proposed method is based on the Tip-Adapter. While, there are many improvements based on Tip-Adapter, such as [1-4].  I think the experiments should include comparisons with these related methods, or at the very least, mention and briefly analyze them.  \n[1] Collaborative Consortium of Foundation Models for Open-World Few-Shot Learning. AAAI, 2024.  \n[2] Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners. CVPR, 2023.  \n[3] DeIL: Direct-and-Inverse CLIP for Open-World Few-Shot Learning. CVPR, 2024.  \n[4] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement. ICCV, 2023.  \n\n2. In Figure 1, the blue and red boxes should be explained. Additionally, if space permits, I suggest that in future work, the authors could add more detailed descriptions in the caption. By this way, readers can understand the general idea of the method just by looking at the figure and caption, without having to spend effort finding the corresponding description in the main text.\n\n3. For instance-shot, only the results of 16-shot are shown, without the results of 1-shot, 2-shot, etc.  \n4. Line 160 seems to have a typo; it should be y_1L instead of y_1V.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1:}$ This paper lacks some important related work. The proposed method is based on the Tip-Adapter. While, there are many improvements based on Tip-Adapter, such as [1-4]. I think the experiments should include comparisons with these related methods, or at the very least, mention and briefly analyze them.\n\n$\\textbf{R1:}$ Thanks for your great suggestion on improving the quality of our manuscript. our method is orthogonal to these four studies, and they do not conflict with each other. Additionally, the four studies mentioned above mainly focus on natural images and would face similar challenges as Tip-Adapter when directly applied to WSIs. Combining our method with these four methods has great potential to further improve the WSIs classification accuracy. For these reasons, we did not compare them directly through experiments, but we analyzed and summarized these four important studies in the related work section. The specific content added in the related work section is as follows.\nIn the field of natural images, many subsequent works based on Tip-Adapter have also made significant contributions to the development of foundation model adaptation. For example, CaFo [2] effectively combines the different prior knowledge of various pre-trained models by cascading multiple foundation models. CO3 [1] goes a step further by considering both general and open-world scenarios, designing a text-guided fusion adapter to reduce the impact of noisy labels. Similarly, for open-world few-shot learning, DeIL [3] proposes filtering out less probable categories through inverse probability prediction, significantly improving performance. APE [4] proposes an adaptive prior refinement method that significantly enhances computational efficiency while ensuring high-precision classification performance. Due to the huge size and the lack of pixel-level annotations, these methods cannot effectively solve the classification problem of WSIs.\n\n\n$\\textbf{Q2:}$ In Figure 1, the blue and red boxes should be explained. Additionally, if space permits, I suggest that in future work, the authors could add more detailed descriptions in the caption. By this way, readers can understand the general idea of the method just by looking at the figure and caption, without having to spend effort finding the corresponding description in the main text.\n\n$\\textbf{R2:}$ Thanks very much for pointing out the problem. We have added the following description below Figure 1. Figure 1: Different few-shot learning paradigms for WSI classification. (a) The instance few-shot method divides all WSIs into a series of patches, then selects a few samples at the patch level and annotates them at the patch level. The red box represents positive samples, and the blue box represents negative samples. (b) The bag few-shot method directly selects a few WSIs at the slide level and annotates them weakly at the slide level. (c) Our method first selects a few WSIs at the slide level, then annotates a few patches for each selected WSI. Compared to (a) and (b), our method significantly reduces annotation costs while providing patch-level supervision information.\n\n\n$\\textbf{Q3:}$  For instance-shot, only the results of 16-shot are shown, without the results of 1-shot, 2-shot, etc.\n\n$\\textbf{R3:}$ We apologize for any inconvenience brought to you. Due to space limitations in the main text, we did not present the experimental results for different shot settings. Instead, we included these results on page 22 of the supplementary materials. In Figure 6, we show the results for 4-shot, 16-shot, and 64-shot settings. From Figure 6, it can be observed that the classification accuracy gradually increases with the number of shots. When the number of shots reaches 64, the accuracy increase nearly converges. Therefore, we recommend using 16-shot or 64-shot in practical applications. We have also added experimental results for the 1-shot and 2-shot settings. The results are shown in Figure A1, which we have uploaded in the PDF file. When the number of shots decreases to 1-shot or 2-shot, the accuracy decreases, with the lower limit being the result of Zero-shot CLIP.\n\n\n$\\textbf{Q4:}$ Line 160 seems to have a typo; it should be $y_{1,L}$ instead of $y_{1,V}$.\n\n$\\textbf{R4:}$ Thanks very much for pointing out the problem. The correct term here should indeed be $y_{1,L}$ . We have corrected this typo in the manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NAbRlQkDSE", "reviewer": "Reviewer_HxPT", "summary": "In this article, the authors propose a novel few-shot learning paradigm for WSI classification. This paradigm is based on two branches: the first is a learnable cache model that utilizes both labeled and unlabeled instance data, and the second, the Prior Branch, leverages the prior knowledge of a pre-trained CLIP model. By combining these two branches, efficient few-shot learning is achieved, and extensive experiments have been conducted on the CAMELYON16 dataset and the TCGA-RENAL dataset.", "questions": "What the difference between the proposed work and previous few-shot WSI classification methods like [4]?\n\n[4] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeurIPS 2023", "limitations": "The authors only tested up to 16 instances and bags, but there is still a significant performance improvement from 8 to 16. I am curious at what data ratio in few-shot learning the model will begin to overfit.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tFew-shot learning is inherently important in the field of WSI classification. The authors have proposed a new few-shot learning paradigm tailored for WSI classification and have achieved notable results.\n2.\tThe experiments on few-shot learning are quite comprehensive, thoroughly comparing the effects of different numbers of instances and bags.", "weaknesses": "1. The study lacks experiments with V-L models specific to the pathology field. Since CLIP is not originally based on pathology images, the authors should include comparisons using PLIP [1] and CONCH [2].\n2. The few-shot learning method proposed by the authors operates at both the instance-level and bag-level. Therefore, the comparative methods should include both instance-based methods and bag-level methods (multi-instance learning). However, the fully supervised methods chosen for comparison are only instance-based. The authors should supplement their comparisons with bag-level methods based on multi-instance learning, such as R2T [3].\n3. Comparing the third and fourth rows in Table 3 of the paper reveals that adding the Prior Branch on top of existing components brings almost no improvement. However, it requires first processing through GPT and then the Text-encoder, significantly increasing the cost without enhancing performance. \n\n[1] PLIP: A visual–language foundation model for pathology image analysis using medical Twitter. Nature Medicine 2023\n[2] CONCH：A Vision-Language Foundation Model for Computational Pathology. Nature Medicine 2024\n[3] Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology. CVPR 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1:}$ The study lacks experiments with V-L models specific to the pathology field. Since CLIP is not originally based on pathology images, the authors should include comparisons using PLIP [1] and CONCH [2].\n\n$\\textbf{R1:}$ Thanks for your great suggestion on improving the quality of our manuscript. Due to the huge size of WSIs, extracting features for the entire dataset using foundational models such as CLIP, PLIP, and CONCH requires a significant amount of time. We have not yet completed the extraction of features encoded using PLIP and CONCH. Additionally, many function wrappers in PLIP and CONCH are different from those in CLIP. Therefore, we need more time to build models using PLIP and CONCH. Once we obtain the latest experimental results, we will include these results in our subsequent replies. We have now completed the extraction of all features for the test set and have conducted some zero-shot classification experiments. Please refer to Table A1 and Table A2 in the PDF file for the experimental results.\n\n$\\textbf{Q2:}$ The few-shot learning method proposed by the authors operates at both the instance-level and bag-level. Therefore, the comparative methods should include both instance-based methods and bag-level methods (multi-instance learning). However, the fully supervised methods chosen for comparison are only instance-based. The authors should supplement their comparisons with bag-level methods based on multi-instance learning, such as R2T [3].\n\n$\\textbf{R2:}$ Thank you very much for your valuable suggestion. We did not compare bag-level methods based on multi-instance learning for the following reasons. First, instance-level fully supervised methods represent the upper bound of supervised learning classification results. Bag-level methods generally perform worse than instance-level fully supervised methods due to the lack of precise fine-grained labels. Second, we found it challenging to accurately reproduce the results of R2T within the limited time available. For these reasons, we did not include this comparison experiment. However, we have added an explanation in Section 4.2, “Comparing Methods and Evaluation Metrics,” about why we did not compare with “bag-level methods,” and we have also included references to relevant methods in the related work section. The specific content added is as follows. “Instance-level fully supervised methods represent the upper bound of supervised learning. Bag-level weakly supervised multi-instance learning methods, such as R2T, generally perform worse than instance-level fully supervised methods due to the lack of fine-grained labels. Therefore, this paper does not directly compare with multi-instance learning methods.”\n\n$\\textbf{Q3:}$ Comparing the third and fourth rows in Table 3 of the paper reveals that adding the Prior Branch on top of existing components brings almost no improvement. However, it requires first processing through GPT and then the Text-encoder, significantly increasing the cost without enhancing performance.\n\n$\\textbf{R3:}$ We apologize for any inconvenience brought to you. If we only compare the third and fourth rows of Table 3, it might indeed seem that way. However, we found that these results in Table 3 are due to the experiments being conducted under the 16-bag shot setting. To fully demonstrate the role of the prior branch, we conducted further experiments, and the related results and analysis can be found in the main text, Figure 4, and line 288. For convenience, we have included some key conclusions here: “When there are only 1 or 2 bags, the instance classification results of the prior branch are significantly higher than those of the cache branch. The instance and bag classification results that combine both the cache and prior branches also surpass those of using each branch separately, indicating that the prior branch performs better in extreme samples, and the information learned by the prior branch and the cache branch is complementary. Therefore, in extreme few-shot scenarios, FAST is dominated by the prior branch, but as the sample size gradually increases, FAST is dominated by the image branch.”\n\n$\\textbf{Q4:}$ What the difference between the proposed work and previous few-shot WSI classification methods like [4]? [4] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeurIPS 2023\n\n$\\textbf{R4:}$ Our method differs significantly from Top in the following ways.\n1. Different Scenarios: Top is a few-shot learning method under slide-level labels. While it also reduces annotation costs, it lacks precise patch-level label information. In contrast, we propose a dual few-shot learning scenario tailored for WSIs, which not only provides patch-level label information but also significantly reduces annotation costs. \n2. Different Technical Approaches: The focus of Top’s research is on designing better text prompt strategies to serve CLIP. In contrast, our method introduces new approaches from both the image cache branch and the text prior branch to fully utilize existing annotation information.\n\n$\\textbf{Q5:}$ The authors only tested up to 16 instances and bags, but there is still a significant performance improvement from 8 to 16. I am curious at what data ratio in few-shot learning the model will begin to overfit.\n\n$\\textbf{R5:}$ We conducted experiments with 64 instances in the supplementary materials, and the results are shown in Figure 6. From Figure 6, it can be observed that as the instance shot increases, the classification accuracy also gradually improves. When the number of shots reaches 64, the rate of accuracy increase slows down. Therefore, considering both training costs and performance benefits, we recommend using an instance shot of 16 or 64 in practical applications.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "13Mf932rYj", "reviewer": "Reviewer_hypM", "summary": "To address the challenges of expensive fine-grained annotation and data scarcity encountered in the clinical application of deep learning-based WSI classification methods, this paper proposes a novel and efficient dual-tier few-shot learning paradigm named FAST. Under this new paradigm, the authors introduce a dual-level annotation strategy that includes bag-level few-shot and instance-level few-shot, modeling the WSI classification problem as a new few-shot classification problem. Building on this, the authors further propose a classification framework composed of a learnable image cache branch and a CLIP prior knowledge branch, fully leveraging the value of the limited data and labels. Extensive experimental results show significant improvement over other few shot methods in both binary and multi-class classification tasks. Interestingly, the proposed method FAST achieves performance close to fully supervised methods with only 0.22% of the annotation cost. This showcases its efficiency and great potential for practical applications.", "questions": "Will the authors open source the relevant code and all model weights for this project? \nFor other issues, please refer to the weaknesses.", "limitations": "In the conclusion section, the authors acknowledge the limitations of their proposed method. I agree that such limitations exist and look forward to future work.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper is well written and easy to read. The authors intuitively demonstrate their methods and contributions through numerous figures and tables. Extensive comparative and ablation experiments illustrate the efficiency and generality of the proposed method. To ensure fairness and prevent randomness, the authors conducted multiple random experiments in their study. The results show significantly better performance in both bag-level and instance-level classification compared to other methods.\n2. The proposed dual-level WSI annotation strategy is a highly innovative and suitable method for WSI data annotation. It addresses the issues of single-level annotation and provides patch-level supervisory information at a cost close to slide-level annotation. Compared to fully supervised methods, the proposed method has significantly lower annotation costs, astonishingly reaching as low as one-thousandth or even one-ten-thousandth.\n3. This paper is inspired by Tip-adapter and proposes a learnable cache branch where both labels and image features are learnable. The final classification framework further integrates a CLIP prior knowledge branch incorporating GPT-4V. Comparative experiments show that this method achieves performance close to fully supervised methods with only 0.2% of the data annotation cost, which is an exciting advancement. Ablation experiments also demonstrate the importance of the proposed components.", "weaknesses": "1. The function ϕ(∙) in Figure 2 is not mentioned or explained in the paper, which may confuse readers. \n2. The authors conducted extensive comparisons in terms of accuracy and annotation cost but lacked analysis of time and memory consumption. \n3. In section 3.2, the authors mention obtaining the optimal fusion weight α through grid search but lack specific details in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1:}$ The function $\\phi(\\cdot)$ in Figure 2 is not mentioned or explained in the paper, which may confuse readers.\n\n$\\textbf{R1:}$ We apologize for any inconvenience brought to you. We have added a description of the function \\phi(\\cdot) in Section 3.2, revising “The retrieval result is $(\\dot{Q}\\dot{K}^T)\\dot{V}$ ” to “ The retrieval result is $\\phi(\\dot{Q} \\dot{K}^T)\\dot{V}$ , where $\\phi(\\cdot) = \\text{softmax}(\\cdot)$ . ”\n\n$\\textbf{Q2:}$ The authors conducted extensive comparisons in terms of accuracy and annotation cost but lacked analysis of time and memory consumption.\n\n$\\textbf{R2:}$ Thanks for your great suggestion on improving the quality of our manuscript. We conducted experiments on training time and memory consumption for the scenario with an instance shot of 16 and a bag shot of 16 using an NVIDIA RTX 3090. The results are shown in the Table A3. Our method achieves performance close to fully supervised methods with a training time of only 0.21 hours and a memory usage of just 5.12 GB. In comparison, training pathology large models like CONCH requires 8 NVIDIA A100 GPUs, highlighting a significant advantage of our method.\n\n$\\textbf{Table A3: analysis of time and memory consumption}$\n| Metric | Time (h) | Memory (GB) |\n| :---: | :---: | :---: |\n| FAST | 0.21 | 5.12 |\n\n$\\textbf{Q3:}$ In section 3.2, the authors mention obtaining the optimal fusion weight α through grid search but lack specific details in the paper.\n\n$\\textbf{R3:}$ Thanks for your great suggestion on improving the quality of our manuscript. We have added the following description in Section 3.2. We divide the fusion weight \\alpha into equal intervals with a step size of 100, then sequentially calculate the classification accuracy for each fusion ratio, and finally select the fusion weight that yields the highest classification accuracy as the fusion weight \\alpha for this task.\n\n\n$\\textbf{Q4:}$ Will the authors open source the relevant code and all model weights for this project? \n\n$\\textbf{R4:}$ Thank you very much for your recognition of our work. We will open-source all related code and model weights to promote further development in WSI classification research.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CSFHBESMia", "reviewer": "Reviewer_x5Ho", "summary": "This paper proposes a few-shot learning approach for WSI (Whole-Slide Image) classification. This approach, built upon Tip-Adapter, leverages a cache branch to memorize the knowledge from few-shot instances and then retrieve label information from the cached knowledge. In addition, a prior branch, which utilizes the knowledge from CLIP and GPT4-V, is built to boost the predictive performance. The experiments on two WSI datasets show the superiority of the proposed method over the original Tip-Adapter and Tip-Adapter-F.", "questions": "- Few-shot instances are randomly selected from the core set. Since only a few instances, e.g. 16, are selected from a very large instance pool and pathological patches (instances) often present heterogeneity, the final selected instance set could have a large variety and thus lead to unstable performances, calling into question the usability of the proposed method. This could also be observed from Fig 3 in the paper. So, the authors are encouraged to analyze and discuss the impact of randomly selected few-shot instances.\n- The original TCGA-RCC is not annotated at the pixel level. This work does a good job in terms of annotating the fine-grained region-of-interest of WSIs. However, if the annotation is not made public, this work would be difficult to follow and be very limited in research impact. The authors are encouraged to make their annotated dataset public.\n\nI would like to raise my score if the authors could resolve my concerns & questions above.\n\n---------------------------------------------After Rebuttal-------------------------------------------------\n\nMy main concerns have been addressed. I am happy to increase my score.", "limitations": "There is no explicit limitation that should be included in the paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "- Originality. This paper explores the few-shot setting in the context of WSI classification. It is under-studied in the field of computational pathology.\n- Significance. This work shows that the proposed approach could obtain a performance near that of fully supervised learning, only with a few labeled instances. Overall, it is an interesting work worthy of investigation in computational pathology, given that labeling WSIs at pixel level is extremely time-consuming and labor-expensive.  \n- Quality. This work presents a good experimental design. Its experiments are conducted from different angles to verify the effectiveness of the proposed algorithms.", "weaknesses": "In summary, my main concerns lie in (1) writing quality, (2) limited technical contribution, and (3) missing experimental comparisons with important vision-language-based models in computational pathology. Based on these critical weaknesses, it may be hard to recommend accepting this paper to NeurlPS. The details are given below:\n\n- This paper is overall rough and sub-par in writing, requiring substantial improvements in clarity. Some obvious flaws are as follows: i) slice-label in line 102, ii) a efficient annotation strategy in line 111, iii) undefined V in line 160, iv) some undefined notations in Section 3.2, and v) some citation errors such as [20]. The authors are encouraged to check these errors and improve their presentation for better academic communication.\n- The technical novelty is quite limited. Most key designs of the proposed approach have been proposed in Tip-Adapter (Zhang et al., ECCV 2022). Compared to Tip-Adapter, the proposed approach does not present valuable or substantial technical contributions, since its two crucial components, few-shot knowledge retrieval in the cache branch and the prior branch, seem borrowed from Tip-Adapter.\n- The authors claim that their work differs significantly from Tip-Adapter (line 142) because the key-value cache model built by Tip-Adapter only allows the key to be learnable; in contrast, their approach allows both the key and the value to be learnable. The authors are encouraged to rephrase this sentence, as i) also allowing the value to be learnable should not be called a significant modification from my humble understanding; ii) Tip-Adapter actually has proposed to use a learnable value in the key-value cache model but it leads to collapse during training. Moreover, given that the learnable value leads to collapsed training in Tip-Adapter, could the authors explain to readers why their methods can avoid collapsed training? \n- Some important vision-language-based models in computational pathology are not cited and compared to the proposed methods, such as **PLIP** (Huang et al., Nature Medicine, 2023) and **CONCH** (Lu et al., Nature Medicine, March 2024). These models show exciting zero-shot performance in WSI classification combined with MI-Zero. The proposed approach should at least show better performance than their zero-shot performance. It could be crucial for justifying the value and significance of this work.\n\nMinor issues:\n- It is suggested to rewrite Section 3.2 to make sure that all notations (those in texts and figures) and the implementation of components are clear and well-explained.\n- The original Tip-Adapter is proposed for traditional single-instance settings. Its implementation for few-shot WSI classification is not clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textbf{Q1:}$ This paper is overall rough and sub-par in writing, requiring substantial improvements in clarity. \n\n$\\textbf{R1: }$ Thanks very much for pointing out the problem. We have revised the above errors as follows: “slice-label” has been corrected to “slide-label,”, “a efficient annotation strategy” has been corrected to “an efficient annotation strategy,”, the symbol “V” has been corrected to “L,” and a “workshops” flag has been added to the citation information for reference [20]. We will conduct sentence-by-sentence proofreading of the paper before the official publication and will also have professionals polish the language.\n\n$\\textbf{Q2 and Q3:}$ The technical novelty is quite limited. The authors are encouraged to rephrase this sentence. Moreover, could the authors explain to readers why their methods can avoid collapsed training?\n\n$\\textbf{R2 and R3:}$ Thanks for your great suggestion on improving the quality of our manuscript. I believe that the second and third comments in the Weaknesses section are closely related, so we have combined them in our response. I consider the proposed method in this paper to be innovative for the following reasons.\n\n1. New Paradigm: For the problem of few-shot classification in pathology images, we propose a novel Dual-tier Few-Shot Learning Paradigm, which not only improves classification accuracy but also reduces annotation costs.\n\n2. Annotation Strategies and Classification Framework: We have introduced a dual-level WSI annotation strategy and a dual-branch classification framework. These two components must work together to achieve excellent WSI classification performance. \n\n3. Addressing Training Issues: As noted by the reviewer, using learnable keys and values in Tip-Adapter can lead to training instability. Therefore, directly applying a learnable Tip-Adapter to WSI classification is not feasible. Our method ensures that the cache model does not suffer from instability due to its overall design, including the classification paradigm, annotation strategy, and classification framework. First, we obtain precise labels for some patches through the new annotation strategy. Second, in the cache model we construct, the labels of annotated patches are not optimized; only the labels of unannotated patches are optimized. The labels of annotated patches provide correct guidance for learning the labels of other patches, ensuring that the proposed cache model remains stable.\n\nWe accept the reviewer’s suggestion “The authors are encouraged to rephrase this sentence.” To clearly state our contributions, we revised “To fully utilize all patches in WSIs, we build a cache model where key and value are learnable, enabling the CLIP to adapt well to the WSI classification task” to “To fully utilize all patches in WSIs, we built a cache model where keys and values are learnable. This effectively facilitates the learning of correct label information for a large number of unlabeled patches, significantly enhancing the performance of CLIP for WSI classification.” At the same time, we rephrased the extent of the differences between our method and Tip-Adapter, revising “Our work is inspired by Tip-Adapter, but it differs significantly from Tip-Adapter.” to “Our work is inspired by Tip-Adapter, but it differs from Tip-Adapter.”\n\n$\\textbf{Q4:}$Some important vision-language-based models in computational pathology are not cited and compared to the proposed methods, such as PLIP and CONCH.\n\n$\\textbf{R4:}$ Thanks for your great suggestion on improving the quality of our manuscript. We have added comparison experiments with PLIP and CONCH. We used the same experimental setting as Zero-shot CLIP, then employed the PLIP image encoder and text encoder to obtain Zero-shot PLIP, and used the CONCH image encoder and text encoder to obtain Zero-shot CONCH. The experimental results are shown in the following two tables: Table A1 presents the results of the comparison methods with FAST on the CAMELYON16 dataset, and Table A2 shows the results on the TCGA-RENAL dataset.\n\nFirst, from Table A2, we can see that the bag classification AUC of Zero-shot CONCH that we reproduced on the TCGA-RENAL dataset (referred to as TCGA RCC dataset in CONCH) is 91.94%, which significantly exceeds the 90.2% reported in the CONCH original paper. This indicates that our constructed prompt is very effective and that our Zero-shot CLIP serves as an excellent baseline model. This result also demonstrates that our comparative results across multiple datasets are very valid. From Table A1, it can be observed that our method outperforms PLIP and CONCH in all metrics on the CAMELYON16 dataset. From Table A2, it is evident that our method surpasses PLIP in all metrics on the TCGA-RENAL dataset. Compared to CONCH, FAST significantly outperforms CONCH in the average bag-level classification AUC. It is worth noting that our method uses a maximum of only 16 WSIs, while CONCH relies on 1.17 million pairs of pathology images based on CoCa, resulting in a significantly high training cost.\n\nLast but not least, our method is orthogonal to studies such as PLIP and CONCH, and they do not conflict with each other. For example, in the “Few-shot classification with task-specific supervised learning” section of CONCH, it is stated: “However, it may still be desirable to specialize the model with labeled training examples to maximize performance for a given task, ideally using as few labels as possible. In this section, we investigate the label efficiency when using the pretrained representation of the image encoder backbone of the visual-language foundation models for task-specific supervised classification.”. This indicates that PLIP, CONCH, and similar methods will also need to be combined with few-shot learning methods like FAST proposed in this paper in the future. Therefore, our method can be effectively combined with methods such as PLIP and CONCH to further enhance WSI classification performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for your valuable comments. These comments have greatly helped improve the quality of our manuscript. Next, we will reply to the questions raised by each reviewer individually. The figures and tables mentioned in our replies have all been uploaded in a single PDF file.

Meta-review: The paper studies few-shot learning for whole slide classification, and presents a method that can harness pre-trained language and vision-language models. After the rebuttal and discussion, the reviewers unanimously recommend acceptance of the paper. They recognize the significant practical implications of the proposed few-shot learning paradigm, and commend the strong empirical results. The proposed few-shot method demonstrates an impressive ability to match the performance of fully supervised learning while significantly reducing annotation costs, which all reviewers find particularly compelling.
Decision: Accept (poster)