FAST: A Dual-tier Few-Shot Learning Paradigm for Whole Slide Image Classification

Authors: kexue  fu
xiaoyuan  luo
  linhao qu
shuo  wang
ying  xiong
ilias  maglogiannis
longxiang  gao
manning  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-06 08:46:29

Abstract: The expensive fine-grained annotation and data scarcity 
  have become the primary obstacles for the widespread adoption of deep learning-based Whole Slide Images (WSI) classification algorithms in clinical practice. Unlike few-shot learning methods in natural images that can leverage the labels of each image, existing few-shot WSI classification methods only utilize a small number of fine-grained labels or weakly supervised slide labels for training in order to avoid expensive fine-grained annotation. They lack sufficient mining of available WSIs, severely limiting WSI classification performance. To address the above issues, we propose a novel and efficient dual-tier few-shot learning paradigm for WSI classification, named FAST. FAST consists of a dual-level annotation strategy and a dual-branch classification framework. Firstly, to avoid expensive fine-grained annotation, we collect a very small number of WSIs at the slide level, and annotate an extremely small number of patches. Then, to fully mining the available WSIs, we use all the patches and available patch labels to build a cache branch, which utilizes the labeled patches to learn the labels of unlabeled patches and through knowledge retrieval for patch classification. In addition to the cache branch, we also construct a prior branch that includes learnable prompt vectors, using the text encoder of visual-language models for patch classification. Finally, we integrate the results from both branches to achieve WSI classification. Extensive experiments on binary and multi-class datasets demonstrate that our proposed method significantly surpasses existing few-shot classification methods and approaches the accuracy of fully supervised methods with only 0.22% annotation costs. All codes and models will be publicly available on https://github.com/fukexue/FAST.

Main paper: 
 introduction: 
 With the advent of Whole Slide Images (WSI) scanners, automated diagnosis based on WSIs has become a critical problem in the field of computational pathology [53, 48, 26] . Due to the huge size of WSI [51] , deep learning-based methods typically divide it into a series of patches and apply classification models to each patch individually. Fully supervised methods annotate each patch *Equal Contribution ‡Corresponding Author The bag few-shot method directly selects a few WSIs at the slide level and annotates them weakly at the slide level. (c) Our method first selects a few WSIs at the slide level, then annotates a few patches for each selected WSI. Compared to (a) and (b), our method significantly reduces annotation costs while providing patch-level supervision information.and then train the classification model in an end-to-end manner [14, 7, 44] . However, fine-grained annotation of WSIs requires expert knowledge and is extremely expensive. To overcome these issue, weakly supervised methods formulate the WSI classification task as a multi-instance learning (MIL) problem [48, 18, 27] . In MIL, each WSI (or slide) is a bag containing thousands of instances (patches) cropped from the slide. They only use slide-level labels to train the classification model. These studies assume the availability of abundant WSIs in clinical settings. However, due to wide staining variations [30, 8] , multiple cancer types [33] , and rare diseases [25] , many clinical scenarios can only access a limited number of WSIs. The dual obstacles of fine-grained annotation difficulties and data scarcity severely limit the available supervised information for model training. Therefore, how to avoid expensive fine-grained annotations while fully utilizing limited WSIs has become a key issue in the field of WSI classification.Recent works [37, 61, 10, 42, 20, 54, 55, 59 ] in natural images have demonstrated the effectiveness of few-shot learning under limited data. Inspired by these studies, some methods [8, 6, 49, 45] gathered all patches obtained from dividing WSIs and randomly selected some patches for annotation, as shown in Figure 1 (a), termed instance few shot, and then used existing few-shot learning methods from natural images for classification. These methods provide strong supervision signals for network training, but discard a large number of unlabeled patches. Our comparative experiments indicate that the methods from natural images perform poorly in few-shot WSI classification. Another methods combine weakly supervised learning with few-shot learning, such as TOP [38] , which annotates only a few slide labels, as depicted in Figure 1 (b), termed bag few shot. Compared to instance few shot, bag few shot methods can utilize all patches. However, slide labels belong to weak supervision signals, and the few-shot scenario leads to even greater scarcity of supervision information for MIL-based weakly supervised learning methods. Therefore, the accuracy of TOP exhibits a significant gap compared to the fully supervised learning methods [35] . Overall, the key reason for the poor performance of existing few-shot WSI classification methods is that these methods cannot simultaneously utilize strong supervision from patch labels and the remaining unlabeled patches, lacking sufficient mining of available WSIs.In this paper, we propose a novel and efficient dual-tier Few-shot learning pAradigm for WSI classificaTion, named FAST. It consists of a annotation-efficient dual-level WSI annotation strategy and a parameter-efficient dual-branch WSI classification framework, which fully utilizes existing vision-language foundation models and can be rapidly applied to various WSI classification tasks.Specifically, we first design a dual-level WSI annotation strategy, as shown in Figure 1 (c). Under this strategy, a small number of WSIs are selected at the slide-level, followed by labeling a small number of patches within each selected WSI. Experts only need to annotate a very small number of patches without needing to perform fine-grained annotation on the entire WSI, significantly reducing the cost of fine-grained annotation while increasing the speed of annotation. Based on the proposed annotation strategy, we formulate the few-shot WSI classification task as a dual-tier few-shot learning problem. Unlike conventional few-shot learning [37] , which uses an "N-way K-shot" setting, FAST's "shots" consists of two levels: bag and instance. Subsequently, under the setting of dual-tier few-shot learning, we propose a dual-branch few-shot WSI classification framework that combines visionlanguage (V-L) models [41, 1] . To fully utilize the prior knowledge of V-L models and the limited WSIs, the classification framework includes a cache branch and a prior branch. For the cache branch, we use the image encoder of the V-L model CLIP [41] to extract features of all patches, then construct a cache model using the labeled instances, and finally classify each instance through knowledge retrieval. However, due to the very limited number of annotated instances, the cached model has only limited knowledge and lacks generalization capability. To improve the cache model's performance, we incorporate a large number of unlabeled instances from the WSIs into the cache model, and treat their labels as learnable parameters, which effectively increases the knowledge capacity of the cache model. During training, we use annotated instances as supervision to optimize these parameters. For the prior branch, we first use GPT4-V [1] to obtain task-related prompts, and then utilize CLIP's text-image matching prior and prompt-learning techniques to design a learnable visual-language classifier. Finally, we integrate the outputs of the cache and prior branches to obtain the final classification results. This framework is based on the foundational model, requires minimal optimization of parameters (cache model and prompt vectors), and can achieve parameter-efficient fine-tuning with a small amount of WSIs and labels. Additionally, by leveraging the prior knowledge of the foundational model, FAST can maintain good accuracy and generalization even with extremely limited annotated data. These characteristics make FAST suitable for rapid adaptation to various WSI classification tasks. In summary, our main contributions are as follows:• We propose a novel few-shot learning paradigm for WSI classification, which achieves high-accuracy WSI classification and rapid adaptation to various WSI classification tasks under low-cost annotation. • We propose an efficient dual-level WSI annotation strategy, which can provide patch-level supervisory information at a cost close to that of slide-level annotation. • We propose a learnable cache model based on the foundation model, which fully utilizes both annotated and unannotated patches. Furthermore, we utilize the prior knowledge of vision-language foundation models to construct a visual-language classifier, combining both to further enhance the performance of the classification framework. • Extensive experiments demonstrate that our method achieves state-of-the-art performance on the CAMELYON16 dataset and the TCGA-RENAL dataset. In addition, compared to fully supervised methods, the annotation cost is only 0.22% of that.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: In the abstract and the section 1.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 WSI Classification According to supervised information, WSI classification methods can be divided into two categories: fully supervised learning methods based on patch-label and weakly supervised learning methods based on slide-label. Fully supervised learning methods directly draw inspiration from supervised learning methods in natural images [23, 11, 12, 15, 58, 32] . Relevant studies on diseases such as breast cancer [44, 43] , lung cancer [7, 3, 31, 14] , and prostate cancer [22, 29] indicate that such methods have approached or even surpassed expert diagnostic accuracy [36] . However, expensive fine-grained annotation prevents their widespread adoption in clinical practice. Weakly supervised learning methods [48, 18, 27, 39, 40, 19, 48, 28, 35] formulate the WSI classification task as a multi-instance learning problem, avoiding expensive fine-grained annotation.Although great progress has been made, these studies rely on large amounts of training data and and cannot address the common issue of data scarcity encountered in practical clinical settings. In this paper, we propose a novel WSI classification paradigm composed of an efficient annotation strategy and a prior knowledge classification framework. The proposed method not only alleviates the poor performance issues of existing methods caused by data scarcity and difficulties in fine-grained annotation, but also enables rapid adaptation to various disease WSI classification tasks.Few-shot Learning for WSI Classification Inspired by the success of few-shot learning in natural images, similar research has emerged in pathology images. Some studies used meta-learning methods, such as MAML [10, 42] , prototypical networks [50, 9] , and matching networks [54] , for tasks like whole-genome doubling prediction [6] and cancer classification [49, 45] . Limited by the scale of pre-training data, these methods only achieve limited generalizability. Another group of studies borrowed the idea of fine-tuning V-L models, with related research still in the early stages. For example, CITE [62] applied visual prompt fine-tuning techniques to few-shot learning in pathology images. CLIPath [24] fine-tuned a learnable network layer on top of a frozen foundation model to transfer foundation model knowledge. However, these studies are only applicable to small-sized pathology image patches and cannot directly handle entire WSI. Qu et al. [38] leveraged the powerful generalization capabilities of CLIP, successfully achieving WSI classification tasks using only slide labels. Limited by the weakly supervised slide label, there still exists a significant gap in classification accuracy compared to fully supervised methods. PLIP [16] and CONCH [34] fine-tuned multimodal large models like CLIP [41] and CoCa [56] to perform pathology classification tasks. However, they still rely on large-scale pathology image-text pair datasets for effective performance. Different from previous works, we propose a dual-level few-shot annotation strategy and a dual-tier few-shot learning formulation approach for WSI classification, which balances annotation cost and granularity, achieving excellent classification accuracy close to fully supervised methods.Vision-Language Model Adaptation V-L foundation models such as CLIP [41] , ALIGN [21] , and Florence [57] have demonstrated remarkable generalization capabilities. How to fine-tune these V-L foundation models for adaptation to downstream tasks is crucial. The popular adaptation strategies can be divided into two groups: prompt tuning and feature adapter. CoOp [63] is a representative method of prompt tuning, which optimizes a set of learnable prompt tokens to enhance the performance of V-L models in downstream tasks. A representative method of feature adapter is CLIP-Adapter [13] , which fine-tunes the CLIP model by adding a lightweight residual module after the encoder. Furthermore, Tip-Adapter [61] constructs a key-value cache model to integrate the knowledge from a few-shot training set directly into the CLIP model, effectively speeding up model convergence during fine-tuning. In the field of natural images, many subsequent works based on Tip-Adapter have also made significant contributions to the development of foundation model adaptation. For example, CaFo [60] effectively combines the different prior knowledge of various pre-trained models by cascading multiple foundation models. CO3 [46] goes a step further by considering both general and open-world scenarios, designing a text-guided fusion adapter to reduce the impact of noisy labels.Similarly, for open-world few-shot learning, DeIL [47] proposes filtering out less probable categories through inverse probability prediction, significantly improving performance. APE [64] proposes an adaptive prior refinement method that significantly enhances computational efficiency while ensuring high-precision classification performance. Due to the huge size and the lack of pixel-level annotations, these methods cannot effectively solve the classification problem of WSIs. However, existing methods are only applicable to fully annotated data and cannot effectively utilize unannotated patches in pathology images. Our work is inspired by Tip-Adapter, but it differs from Tip-Adapter. The key-value cache model built by Tip-Adapter only allows the key to be learnable.To fully utilize all patches in WSIs, we built a cache model where keys and values are learnable. This effectively facilitates the learning of correct label information for a large number of unlabeled patches, significantly enhancing the performance of CLIP for WSI classification.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: In the section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
method: 
 In this chapter, we first explain how to efficiently annotate WSIs, then describe how to formulate the few-shot WSI classification problem under the novel annotation strategy, and finally introduce the classification framework of FAST.For a better understanding, we first present a fully labeled WSI dataset. Given a dataset X train = {X 1 , X 2 , . . . X M } consisting of M WSIs, where each WSI X i = {x i,1 , x i,2 , . . . , x i,Ui } consisting of U patches. Each patch x i,j is considered an instance, and all patches in X i form a bag.For Q= 𝐹 !&'! K = 𝐹 !"#$% ( V = 𝑌 !"#$% ) ** Q K V transpose 𝜙 "Patches𝑋 !"#$% * 𝑋 !&'! Test Slide 𝒑𝒊,𝟏 𝒑𝒊,𝟐 … 𝒑𝒊,𝒌 𝒑𝑳,𝟏 𝒑𝑳,𝟐 … 𝒑𝑳,𝒌 … … … … 𝑌 !"#$% ) * 𝑌 !"#$% ) * ! 𝑦 ()(*+ 𝑇 * 𝐹 !&+! × 1 -α ! 𝑦 /0120Cache Branch Piror BranchAs shown in Figure 2 , the classification framework consists of two branches: a learnable image cache branch and a CLIP prior knowledge branch. During training, only a very small number of instances need to be annotated to train the cache model and the learnable prompt tokens. During testing, the prediction results from both the image cache branch and the CLIP prior knowledge branch are integrated to obtain instance-level classification results. Finally, the instance-level classification results are pooled to yield a bag-level classification result.The cache model consists of a feature cache and a label cache, and its construction method is illustrated in Figure 2 . First, all patches from these WSIs are input into the image encoder for feature extraction, and the extracted features are stored in the feature cache.F train = VisualEncoder (X train ′ )The VisualEncoder represents the image encoder within CLIP. Simultaneously, the labels of the annotated instances are transformed into one-hot encoding and stored in the label cache. For the remaining instances without labels, we set their labels as learnable parameters P train and store them in the label cache. Y I train = Cat Y I train , ′ , P train P train = {{p 1,L+1 , . . . , p 1,L+K1 } , {p 2,L+1 , . . . , p 2,L+K2 } , . . . , {p i,L+1 , . . . , p i,L+Ki }} represents the pseudo-labels of all unannotated instances in X train ′ , where p is a learnable highdimensional vector. As illustrated in the few-shot knowledge retrieval module in Figure 2 , through knowledge retrieval, we can obtain the prediction of the cache branch. Specifically, we employ an attention mechanism to implement the knowledge retrieval module, and treat the features and labels as key-value pairs. The Features F train serve as keys, denoted by K. The labels Y I train serve as values, denoted by V . The feature of patch to be retrieved serve as query, denoted by Q. The retrieval result is ϕ( Q KT ) V , where ϕ(•) = sof tmax(•).We train the cache model using F train and Y train ′ . According to the idea of knowledge retrieval, given a query instance x train ∈ X train and its encoded feature f train ∈ F train , and the prediction result obtained from the few-shot knowledge retrieval is ỹcache = f train F T train Y I train ′′ . Subsequently, the cross-entropy between the predicted values ỹcache i,j from the cache model and the ground truth y i,j is calculated as the loss function.CacheLoss i,j = CE ỹcache i,j , y i,j = CE f i,j F T train Y I train ′′ , y i,jThe learnable labels in the cache model can be optimized by minimizing the loss function. Additionally, to further optimize the feature space of CLIP, we also set F train to be learnable as follow.P * train = min Ptrain i,j CacheLoss i,j , F * train = min Ftrain i,jCacheLoss i,jThe parameters of the VisualEncoder are frozen. It is important to note that in practical training, when there is an excess of unlabeled data, incorporating all the unlabeled instances into the cache model can exceed memory constraints. In such cases, we use K-means clustering algorithm to select representative instances to construct a core set. Only the core set is incorporated into the cache model for training and inference.2) CLIP Prior Knowledge Branch To further leverage the prior knowledge of the vision-language foundation model, we construct a prompt-learnable vision-language instance classifier based on CLIP's text encoder. Since the feature spaces of CLIP's text encoder and image encoder are aligned, image classification can be achieved by calculating the similarity between text and images. However, unlike natural image classification, pathological image classification requires more specialized and targeted prompts. As shown in the prior branch in Figure 2 , we input a small number of annotated instance images into GPT-4V, which generates descriptions of the images combining relevant pathological knowledge, forming prompts for detecting each type of WSI. Then, these category-specific prompts are feature-extracted by CLIP's text encoder, Furthermore, inspired by CoOp, we make the category-specific prompts learnable. Specifically, we add D learnable tokens to the prompts generated for each category, combining them into the final learnable prompts T ′ . The new text feature is as follows,f text c = TextEncoder (T c ) , T c = [W ] c,1 [W ] c,2 . . .f text ′ c = TextEncoder (T c ′ ) , T ′ i = [W ] i,1 [W ] i,2 . . . [V ] i,1 [V ] i,2 . . . [V ] i,DBased on the new text feature and the image feature f i,j , the classification result of prior branch is,ỹprior i,j = p (y i,j = c | f i,j ) = exp cos f text ′ c , f i,j /τ N j exp cos f text ′ j , f i,j /τSimilar to the image cache branch, we also use all annotated instances to train the prior knowledge branch. The optimization function and loss function of the prior branch are as follows,[V ] = min [V ] i,jT extLoss i,j , where T extLoss i,j = CE ỹprior i,j, y i,jThe parameters of the TextEncoder are also frozen. Finally, we combine the predictions and loss functions from the image cache branch and the prior knowledge branch to obtain the overall model's instance-level classification result and loss function,ỹ = α • ỹcache + (1 -α) • ỹprior Loss = i,j CacheLoss i,j + i,j T extLoss i,jwhere α is the fusion weight of the two branches, which can be considered a hyperparameter. We divide the fusion weight α into equal intervals with a step size of 100, then sequentially calculate the classification accuracy for each fusion ratio, and finally select the fusion weight that yields the highest classification accuracy as the fusion weight α for this task. Since the entire network has few parameters and fast inference speed, this parameter can be quickly optimized through grid search to obtain the optimal value. 
datasets and few-shot scenario simulation: 
 We evaluated our method on the public WSI datasets CAMELYON16 [4] and TCGA-RENAL [17] .CAMELYON16 is a binary dataset used to detect whether it contains breast cancer metastasis. TCGA-RENAL is a multi-class dataset for classifying renal cancer subtypes, covering clear cell renal cell carcinoma (ccRCC), papillary renal cell carcinoma (pRCC), and chromophobe renal cell carcinoma (chRCC). For more details about the datasets, please refer to the supplementary material A. First, we compared FAST with zero-shot learning method CLIP [36] and fully supervised method using all instance-level labels [35] , which can respectively be seen as the lower and upper bounds of deep learning method performance in few-shot WSI classification. Subsequently, due to lack of effective few-shot WSI learning methods, we compared the latest few-shot learning methods Tip-Adapter and Tip-Adapter-F [61] in natural images. Bag-level weakly supervised multi-instance learning methods, such as R2T [52] , generally perform worse than instance-level fully supervised methods due to the lack of fine-grained labels. Therefore, this paper does not directly compare with multi-instance learning methods. For implementation details of our method, please refer to the supplementary material A.2. To provide a comprehensive evaluation of these methods, we reported instance-level Area Under Curve and bag-level Area Under Curve (AUC) [5] . Since the TCGA-RENAL dataset is a multi-class classification task, we reported instance-level and bag-level AUCs separately for each category.The few-shot WSI classification results on the CAMELYON16 dataset are shown in Table 1 . Firstly, in the zero-shot setting, the instance-level AUC and bag-level AUC of zero-shot CLIP are very low, almost unable to handle the classification task. In contrast, under extreme few-shot setting with 1 bag shot and 16 instance shots, FAST achieves 0.84 instance-level AUC, which shows a significant improvement over zero-shot CLIP. Secondly, as the training samples increase, the performance of FAST shows a significant improvement trend. Meanwhile, our proposed method FAST consistently outperforms the comparison methods Tip-Adapter and Tip-Adapter-F across different numbers of samples. Although Tip-Adapter and Tip-Adapter-F have achieved significant success in natural images, their performance is poor on few-shot WSI classification task, especially with bag-level AUC generally below 0.6. We think there are two main reasons for this: (1) there are domain differences between pathology images and natural images, making it difficult for the text branch of CLIP to accurately classify instances and even more difficult to classify bags. (2) The limited WSIs make it more difficult to ensure the diversity of instances sampled from these WSIs, resulting in poor generalization of these methods. In contrast, while FAST also utilizes a small number of bags, it fully leverages the labeled and unlabeled instances within these bags through a learnable label cache, enabling it to learn more comprehensive instance representations. Finally, in the setting where only 16 bag shots and 16 instance shots are available, our proposed FAST method achieves 0.9151 instance-level and 0.8197 bag-level AUC, which is close to the performance of the fully supervised method using all instance annotations. Importantly, the annotation cost of FAST is only 0.22% of that of the fully supervised method, as detailed in Section 4.4. In summary, FAST achieves accuracy close to the fully supervised method with extremely low data collection and annotation costs. This significantly enhances the efficiency of establishing deep learning models in clinical settings and opens up possibilities for widespread clinical adoption of WSI classification algorithms based on deep learning.TCGA-RENAL In the few-shot experiments on the TCGA-RENAL dataset, the classification results are shown in Table 2 . Compared to the binary classification task on the CAMELYON16 dataset, the three-class classification task is more challenging. Therefore, the zero-shot CLIP method only achieves around 0.5 instance-level AUC, almost unable to handle the classification task. In the setting with very few shots, such as one or two bags, the results of all methods are relatively poor, with instance-level AUC and bag-level AUC both less than 0.7000. However, our proposed FAST still achieves SOTA performance on most metrics. As the bag shot reaches 4 or more, the classification results begin to significantly improve, and FAST also outperforms other methods on all metrics. Especially in 16 bag shots, FAST significantly outperforms the comparison methods. The average bag-level AUC for chRCC reaches 0.9234, differing by only 0.0033 from the fully supervised method. Similarly, the bag-level AUC for ccRCC and pRCC also reach 0.9254 and 0.9216 respectively, differing by only 0.0218 and 0.036 from the fully supervised method. This experimental result further demonstrates that, in complex multi-class classification tasks, FAST can approach the fully supervised method with very low annotation costs, achieving the state-of-the-art performance in few-shot learning. To illustrate the annotation efficiency of FAST, we compared the classification results of our method and the fully supervised method under different annotation ratios. For the CAMELYON16 dataset, the results are presented in Figure 3 . It can be observed that the classification AUC of FAST rapidly increase with the growth of bag shots. When the bag shot reaches 16, the annotation only accounts for 0.22%. The average bag-level AUC reaches 96.32% of the fully supervised method. This result fully demonstrates the advantage of FAST in annotation efficiency. For the results on the TCAG-RENAL dataset, please refer to the supplementary material A.3.To analyze the importance of each component in FAST, we conducted a serise of experiments under the conditions of 16 bag shots and 16 instance shots on the CAMELYON16 dataset, including whether to use a cache branch, whether to set the feature cache as learnable, whether to use a prior branch, and whether to utilize unlabeled instances to construct a learnable label cache. The results of the ablation experiments are shown in Table 3 . The first three rows of the table correspond to zero-shot CLIP, Tip-Adapter, and Tip-Adapter-F, respectively. The next three rows correspond to FAST using only the prior branch, FAST using only the cache branch, and the complete FAST. Firstly, the first three rows of the table show that the instance-level AUC gradually increases from 0.6711 to 0.7277 for the three methods. This indicates that the cache branch can leverage a small amount of supervised information, and making the cache feature learnable can further optimize the feature space distribution of the cache model. However, even the best model's instance-level AUC is only 0.7227, and the pooled bag-level AUCs are all less than 0.6. Next, from the last three rows of the table, even with only the prior branch, using our proposed prompt-based method, the instance-level and bag-level classification AUCs reach 0.8739 and 0.7931, respectively, which outperformed the results of Tip-Adapter-F. The instance-level and bag-level AUCs of only using the cache branch can reach 0.9165 and 0.8183, respectively, demonstrating that the cache branch is crucial for FAST. By further comparing the third and fifth rows of the table, we can infer the primary reason why FAST's cache model surpasses that of Tip-Adapter-F in natural images. Specifically, FAST utilizes a large number of unannotated instances in a small number of bags through the learnable label cache, thereby significantly improving the capacity and generalization ability of the cache model. Finally, comparing the last two rows of the table, we find that when both cache and prior branches are utilized, FAST does not show significant improvement over using only the cache branch. This is because the number of samples is sufficient, and the prior branch primarily plays a role when there are fewer samples.To verify the role of two branchs, we compared the prior branch and cache branch of FAST under different bag shots. The results are shown in Figure 4 . When there are only 1 or 2 bags, the instance classification results of the prior branch are significantly higher than those of the cache branch. The instance and bag classification results combined with both the cache and prior branches also surpass those of using each branch separately, indicating that the prior branch performs better in extreme samples, and the information learned by the prior branch and the cache branch is complementary.As the number of bags increases to 4 or more, the results of the cache branch gradually surpass those of the prior branch. Therefore, in extreme few-shot scenarios, FAST is dominated by the prior branch, but as the sample size gradually increases, FAST is dominated by the image branch. This experimental analysis can provide effective guidance for the practical application of FAST. Additionally, we provide further analysis experiments on the number of annotated instance and the number of core sets in the supplementary material A.3. 
conclusion: 
 In this paper, we propose a dual-tier few-shot learning paradigm FAST for WSI classification, which achieves low-cost WSI annotation, high-accuracy WSI classification, and adaptation to various WSI classification tasks. To achieve these goals, we introduce two key technologies in FAST, including a dual-level few-shot annotation strategy and a dual-branch classification framework. The dual-level few-shot annotation strategy effectively alleviates the problem of fine-grained annotation difficulty in WSI by annotating only a small number of patches in a limited number of WSIs. Experimental results demonstrate that the annotation cost of our method is only 0.22% of patch-level full annotation. In the dual-branch classification framework, we construct a cache branch where both features and labels are learnable, fully exploiting the partially annotated data obtained from the dual-level few-shot annotation strategy. Furthermore, combining a vision-language foundation model and prompt tuning technology, we build a prior knowledge branch to assist the cache branch in improving classification performance. Through extensive experiments, we demonstrate that FAST can achieve state-of-the-art performance on both binary and multi-class classification tasks. However, our method still has certain limitations. For bag-level classification, we simply use pooled instance classification results, ignoring the relationships between instances. In the future, we will further explore how to use instance-level classification results to obtain a better bag-level classification result. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: In the section A.2 of supplementary materials.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: In the CAMELYON16 paragraph of the section 4.3.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Datasets CAMELYON16 CAMELYON16 [4] contains 400 WSIs of lymph nodes, used to detect the presence of metastatic breast cancer. WSIs containing metastases are labeled positive, while others are negative, with pixel-level annotations of the metastatic areas. We first cropped these WSIs at 10x magnification into 512x512 image patches, then removed image patches with entropy less than 15 as background, and marked patches with more than 30% cancer area as positive, resulting in 186,604 image patches, of which 8,117 were marked as positive (4.3%).TCGA-RENAL This dataset comes from The Cancer Genome Atlas (TCGA) project [17] , covering high-resolution WSIs of the three main subtypes of renal cell carcinoma (RCC): clear cell renal cell carcinoma (ccRCC), papillary renal cell carcinoma (pRCC), and chromophobe renal cell carcinoma (chRCC). ccRCC is the most common subtype, characterized by clear cells containing abundant lipids and glycogen. pRCC follows, characterized by papillary structures formed on the surface of tumor cells. chRCC is relatively rare, with larger cells and granular cytoplasm. All WSIs are rigorously reviewed by professional pathologists to ensure the representativeness and quality of the data. To fully verify the effectiveness of our proposed method, we collected 910 WSIs from TCGA, and then organized experts to delineate all WSIs at the pixel level, including delineation of cancerous and non-cancerous areas. Subsequently, through preprocessing processes such as WSI segmentation and background removal similar to CAMELYON16, we obtained instance-level labels for the complete training and testing sets. Finally, all WSIs were divided into training and testing sets in a ratio of 70% and 30%, respectively.FAST employs the image encoder and text encoder from the pre-trained CLIP-RN50 [41] . The cache capacity is typically determined by the number of annotated and unannotated instances in the training set after each sampling. For bag shot greater than 4, we used a core set selection strategy to prevent excessive caching. For the CAMELYON16 and TCGA-RENAL datasets, we set the number of core sets to 1000 and 2000, respectively. We set the number of learnable tokens to 10. During training, we utilized the Adam optimizer with learning rates set as follows: 0.001 for the feature cache, 0.01 for the label cache, and 0.001 for the tokens in the prior branch. We fine-tune our model with batch size of 4096 for 20,000 epochs. All models are trained and tested on an RTX 3090 GPU with 24GB memory.For Tip-Adapter, We conducted comparative experiments according to the settings of the optimal model in the original Tip-Adapter paper. For aspects that cannot be adapted to the few-shot WSI classification task, we used the following approach. We designed a set of text prompts specifically for pathology images, which has been proven superior in the CONCH comparison experiments we conducted. We used all annotated patches to build the cache model.To provide a more intuitive illustration of the annotation efficiency of our method, we first calculated the annotation ratio under different bag shots (1, 2, 4, 8, 16), with each bag containing 16 annotated instances. Then, we visualized the classification results of FAST under different annotation ratios and compared them with the fully supervised method. The results for TCGA-RENAL dataset are presented in Figure 5 . It can be observed that the classification AUC of FAST rapidly increase with the growth of bag shots. When the bag shot reaches 8, FAST achieves results comparable to fully supervised methods. At this point, the annotation ratio of FAST on the TCGA-RENAL dataset is only 0.0067% of the total instance annotation. When the bag shot reaches 16, the annotation only accounts for 0.013% of all instances. Even under such extreme minimal annotation, FAST can still achieve results close to fully supervised methods, especially in bag classification results. For chRCC, pRCC, and ccRCC, FAST's average bag-level AUC reaches 99.64%, 96.24%, and 97.70% respectively compared to the fully supervised method. This result fully demonstrates the advantage of FAST in annotation efficiency. (2) The Number of Annotated Instance The above experiments indicate that achieving good results only require annotating 16 instances per bag. To further investigate the influence of the number of annotated instances per bag, we conducted experiments on the CAMELYON dataset with the number of annotated instances per bag set to 4, 16, and 64, respectively. The results are shown in Figure 6 . It can be observed that when the number of bags is small and there are only 1, 2, or 4 bags per class, the instance-level and bag-level classification AUCs are significantly higher when 64 instances are annotated per bag compared to only annotating 4 or 16 instances. However, as the number of bags increases to 8 and 16, the AUC of FAST gradually converge. Regardless of whether each bag is annotated with 4, 16, or 64 instances, FAST achieves similar results in instance-level AUC and bag-level AUC. These experimental results indicate that when the number of bags is extremely small, increasing the number of annotated instances can effectively improve the performance of FAST. However, when the number of bags increases to a certain level, even if only a small number of instances are annotated in each bag, FAST can achieve results similar to annotating a large number of instances. This indicates that FAST's ability to achieve higher accuracy than other methods primarily relies on its learning from unlabeled instances. It also suggests that in practical applications of FAST, if there are a large number of bags, the requirement for labeling instances can be appropriately reduced.(3) Core Set Size Few-shot learning of WSI classification differs from conventional few-shot learning. Even with only several WSIs, they may produce tens of thousands or even hundreds of thousands of patches. Therefore, to avoid optimization difficulties caused by excessively large caches, 4 . When the core set size is only 100 or 500, FAST's bag-level AUC is significantly lower, indicating that the core set size at this time is insufficient to cover all representative samples, resulting in a loss of a large amount of information. However, when the core set size reaches 1000 or above, FAST's instance-level AUC and bag-level AUC achieve good results, indicating that FAST has good robustness to core set size. However, when the core set size is 5000 and the number of bags is 16, the results decrease instead, indicating that an overly large learnable cache is difficult to optimize. Therefore, we set the core set size to 1000 or 2000 in experiments where the number of bags is greater than 4.Justification: Our research is beneficial for WSI classification.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The paper studies few-shot learning for whole slide classification, and presents a method that can harness pre-trained language and vision-language models. After the rebuttal and discussion, the reviewers unanimously recommend acceptance of the paper. They recognize the significant practical implications of the proposed few-shot learning paradigm, and commend the strong empirical results. The proposed few-shot method demonstrates an impressive ability to match the performance of fully supervised learning while significantly reducing annotation costs, which all reviewers find particularly compelling.

Reviews: review: paper_summary: paper_summary: This paper investigates the issue of Whole Slide Images (WSI) classification, a study with practical value. It proposes a new working paradigm that is an improvement based on Tip-Adapter. Theoretically, this new paradigm can effectively address the problem and has strong scalability.
strength_weakness: strength_weakness: 1. This paper lacks some important related work. The proposed method is based on the Tip-Adapter. While, there are many improvements based on Tip-Adapter, such as [1-4].  I think the experiments should include comparisons with these related methods, or at the very least, mention and briefly analyze them.  
[1] Collaborative Consortium of Foundation Models for Open-World Few-Shot Learning. AAAI, 2024.  
[2] Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners. CVPR, 2023.  
[3] DeIL: Direct-and-Inverse CLIP for Open-World Few-Shot Learning. CVPR, 2024.  
[4] Not All Features Matter: Enhancing Few-shot CLIP with Adaptive Prior Refinement. ICCV, 2023.  

2. In Figure 1, the blue and red boxes should be explained. Additionally, if space permits, I suggest that in future work, the authors could add more detailed descriptions in the caption. By this way, readers can understand the general idea of the method just by looking at the figure and caption, without having to spend effort finding the corresponding description in the main text.

3. For instance-shot, only the results of 16-shot are shown, without the results of 1-shot, 2-shot, etc.  
4. Line 160 seems to have a typo; it should be y_1L instead of y_1V.
questions: questions: see weakness
limitations: limitations: authors didn't present the limitation.
paper_summary: paper_summary: In this article, the authors propose a novel few-shot learning paradigm for WSI classification. This paradigm is based on two branches: the first is a learnable cache model that utilizes both labeled and unlabeled instance data, and the second, the Prior Branch, leverages the prior knowledge of a pre-trained CLIP model. By combining these two branches, efficient few-shot learning is achieved, and extensive experiments have been conducted on the CAMELYON16 dataset and the TCGA-RENAL dataset.
strength_weakness: strength_weakness: 1. The study lacks experiments with V-L models specific to the pathology field. Since CLIP is not originally based on pathology images, the authors should include comparisons using PLIP [1] and CONCH [2].
2. The few-shot learning method proposed by the authors operates at both the instance-level and bag-level. Therefore, the comparative methods should include both instance-based methods and bag-level methods (multi-instance learning). However, the fully supervised methods chosen for comparison are only instance-based. The authors should supplement their comparisons with bag-level methods based on multi-instance learning, such as R2T [3].
3. Comparing the third and fourth rows in Table 3 of the paper reveals that adding the Prior Branch on top of existing components brings almost no improvement. However, it requires first processing through GPT and then the Text-encoder, significantly increasing the cost without enhancing performance. 

[1] PLIP: A visual–language foundation model for pathology image analysis using medical Twitter. Nature Medicine 2023
[2] CONCH：A Vision-Language Foundation Model for Computational Pathology. Nature Medicine 2024
[3] Feature Re-Embedding: Towards Foundation Model-Level Performance in Computational Pathology. CVPR 2024
questions: questions: What the difference between the proposed work and previous few-shot WSI classification methods like [4]?

[4] The rise of ai language pathologists: Exploring two-level prompt learning for few-shot weakly-supervised whole slide image classification. NeurIPS 2023
limitations: limitations: The authors only tested up to 16 instances and bags, but there is still a significant performance improvement from 8 to 16. I am curious at what data ratio in few-shot learning the model will begin to overfit.
paper_summary: paper_summary: To address the challenges of expensive fine-grained annotation and data scarcity encountered in the clinical application of deep learning-based WSI classification methods, this paper proposes a novel and efficient dual-tier few-shot learning paradigm named FAST. Under this new paradigm, the authors introduce a dual-level annotation strategy that includes bag-level few-shot and instance-level few-shot, modeling the WSI classification problem as a new few-shot classification problem. Building on this, the authors further propose a classification framework composed of a learnable image cache branch and a CLIP prior knowledge branch, fully leveraging the value of the limited data and labels. Extensive experimental results show significant improvement over other few shot methods in both binary and multi-class classification tasks. Interestingly, the proposed method FAST achieves performance close to fully supervised methods with only 0.22% of the annotation cost. This showcases its efficiency and great potential for practical applications.
strength_weakness: strength_weakness: 1. The function ϕ(∙) in Figure 2 is not mentioned or explained in the paper, which may confuse readers. 
2. The authors conducted extensive comparisons in terms of accuracy and annotation cost but lacked analysis of time and memory consumption. 
3. In section 3.2, the authors mention obtaining the optimal fusion weight α through grid search but lack specific details in the paper.
questions: questions: Will the authors open source the relevant code and all model weights for this project? 
For other issues, please refer to the weaknesses.
limitations: limitations: In the conclusion section, the authors acknowledge the limitations of their proposed method. I agree that such limitations exist and look forward to future work.
paper_summary: paper_summary: This paper proposes a few-shot learning approach for WSI (Whole-Slide Image) classification. This approach, built upon Tip-Adapter, leverages a cache branch to memorize the knowledge from few-shot instances and then retrieve label information from the cached knowledge. In addition, a prior branch, which utilizes the knowledge from CLIP and GPT4-V, is built to boost the predictive performance. The experiments on two WSI datasets show the superiority of the proposed method over the original Tip-Adapter and Tip-Adapter-F.
strength_weakness: strength_weakness: In summary, my main concerns lie in (1) writing quality, (2) limited technical contribution, and (3) missing experimental comparisons with important vision-language-based models in computational pathology. Based on these critical weaknesses, it may be hard to recommend accepting this paper to NeurlPS. The details are given below:

- This paper is overall rough and sub-par in writing, requiring substantial improvements in clarity. Some obvious flaws are as follows: i) slice-label in line 102, ii) a efficient annotation strategy in line 111, iii) undefined V in line 160, iv) some undefined notations in Section 3.2, and v) some citation errors such as [20]. The authors are encouraged to check these errors and improve their presentation for better academic communication.
- The technical novelty is quite limited. Most key designs of the proposed approach have been proposed in Tip-Adapter (Zhang et al., ECCV 2022). Compared to Tip-Adapter, the proposed approach does not present valuable or substantial technical contributions, since its two crucial components, few-shot knowledge retrieval in the cache branch and the prior branch, seem borrowed from Tip-Adapter.
- The authors claim that their work differs significantly from Tip-Adapter (line 142) because the key-value cache model built by Tip-Adapter only allows the key to be learnable; in contrast, their approach allows both the key and the value to be learnable. The authors are encouraged to rephrase this sentence, as i) also allowing the value to be learnable should not be called a significant modification from my humble understanding; ii) Tip-Adapter actually has proposed to use a learnable value in the key-value cache model but it leads to collapse during training. Moreover, given that the learnable value leads to collapsed training in Tip-Adapter, could the authors explain to readers why their methods can avoid collapsed training? 
- Some important vision-language-based models in computational pathology are not cited and compared to the proposed methods, such as **PLIP** (Huang et al., Nature Medicine, 2023) and **CONCH** (Lu et al., Nature Medicine, March 2024). These models show exciting zero-shot performance in WSI classification combined with MI-Zero. The proposed approach should at least show better performance than their zero-shot performance. It could be crucial for justifying the value and significance of this work.

Minor issues:
- It is suggested to rewrite Section 3.2 to make sure that all notations (those in texts and figures) and the implementation of components are clear and well-explained.
- The original Tip-Adapter is proposed for traditional single-instance settings. Its implementation for few-shot WSI classification is not clear.
questions: questions: - Few-shot instances are randomly selected from the core set. Since only a few instances, e.g. 16, are selected from a very large instance pool and pathological patches (instances) often present heterogeneity, the final selected instance set could have a large variety and thus lead to unstable performances, calling into question the usability of the proposed method. This could also be observed from Fig 3 in the paper. So, the authors are encouraged to analyze and discuss the impact of randomly selected few-shot instances.
- The original TCGA-RCC is not annotated at the pixel level. This work does a good job in terms of annotating the fine-grained region-of-interest of WSIs. However, if the annotation is not made public, this work would be difficult to follow and be very limited in research impact. The authors are encouraged to make their annotated dataset public.

I would like to raise my score if the authors could resolve my concerns & questions above.

---------------------------------------------After Rebuttal-------------------------------------------------

My main concerns have been addressed. I am happy to increase my score.
limitations: limitations: There is no explicit limitation that should be included in the paper.

Comments: title: Response to the Authors
Thank you for your rebuttal, which clearly addressed my concerns.
Response to the Remaining Concerns from Reviewer x5Ho
Reply to the Authors' Rebuttal
Experiments with V-L models specific to the pathology field
Part2: Rebuttal by Authors
comment: Thank you very much for your suggestions, which have greatly helped improve the quality of our manuscript. We sincerely appreciate your recognition of our work and the higher score. We will include the aforementioned content in the camera-ready version of the paper.
Thanks for the authors responce and further experiments. My questions are addressed, and I will rise the score.
Thank you very much for your suggestions, which have been extremely helpful in improving the quality of our manuscript. We also sincerely appreciate your recognition of our work and the higher score. We will incorporate the aforementioned content in the camera-ready version of the paper.
Thank you very much for your suggestions, which have been extremely helpful in improving the quality of our manuscript. We also sincerely appreciate your recognition of our work and the higher score. We will incorporate the aforementioned content in the camera-ready version of the paper.
Thank you very much for your suggestions, which have been extremely helpful in improving the quality of our manuscript. We also sincerely appreciate your recognition of our work and the higher score. We will incorporate the aforementioned content in the camera-ready version of the paper.
I would like to thank the authors for their efforts. Most of the time during the rebuttal, I feel inefficient communication in reading the author's responses, as most of these responses often fail to capture the true meaning of my questions and lead to undesirable QA.


Yet, overall, most of my concerns have been resolved. Thanks for the authors' experiments provided in the rebuttal. I believe the proposed framework could facilitate the study of few-shot WSI analysis. In view of these, I am happy to increase my score. The authors are encouraged to include the important suggestions & questions into the final version of the paper.
Thanks for the authors' responses. The author has resolved my questions and I agree to accept this paper.
Thank you for your rebuttal, which clearly addressed my concerns. I have read other reviewers' comments and the author's rebuttal, particularly the comparison with vision-language-based models in computational pathology. The experimental results and the authors’ responses clearly and effectively demonstrate the contribution of this paper. Overall, the authors' rebuttal resolves my concerns, and their answers to other reviewers' questions also seem reasonable to me. I think this paper is highly valuable for advancing computational pathology. Thus, I increase my rating to strong accept.
Thank you very much for your rapid response, which is crucial for improving the quality of our manuscript.

$\textbf{Response to R2:}$ First, we sincerely appreciate the reviewer’s recognition of our innovation in the dual-tier few-shot learning paradigm, which is meaningful for WSI classification. Additionally, when Tip-adapter is applied to WSI classification, it faces challenges such as the inability to fully utilize WSI data and the issue of huge size. For the former, directly adopting the methods from Tip-adapter to fully utilize WSI data would lead to instability in training, as mentioned in response to an earlier question. To address this, we proposed a cache model where both labels and features are learnable, with the labels of annotated patches being frozen while those of unannotated patches remain learnable. This effectively alleviates the issue of Tip-adapter being unable to fully utilize WSI data. For the latter, we introduced a core set construction method that effectively addresses the challenge of training on entire WSIs caused by their huge size. Overall, our model design is inspired by Tip-adapter, but it is not entirely identical to Tip-adapter.

$\textbf{Response to R4:}$ We are very grateful for the problems pointed out by the reviewer, as they have been extremely helpful in improving the quality of our manuscript. The last three questions are closely related, so we will answer them together in response 2.
1. In the final version of the paper, we will revise “Compared to CONCH, FAST significantly outperforms CONCH in the average bag-level classification AUC” to “Compared to CONCH, FAST outperformed CONCH by 0.0094 in the average bag-level classification AUC.” 
2. First, I would like to provide a brief explanation of PLIP and CONCH. PLIP is a version of CLIP fine-tuned on large-scale pathological data. Similar to CONCH, which is trained based on the large-scale vision-language model CoCa. Secondly, we apologize for any inconvenience caused. Below, we present the experimental results of our method combined with CONCH. As shown in the table A5, when the bag shot is 1 and the instance shot is 16, FAST-CONCH improves the instance-level AUC by 0.1227 and the bag-level AUC by 0.1485 compared to FAST-CLIP. When the bag shot is 16 and the instance shot is 16, FAST-CONCH improves the instance-level AUC by 0.0615 and the bag-level AUC by 0.1373 compared to FAST-CLIP. CONCH achieved a bag-level classification AUC of 0.7113 on the CAMELYON16 dataset. In comparison, FAST-CONCH improved this by 0.2457.

$\textbf{Table A5: Results of using CONCH on CAMELYON16 dataset.}$
| Bag Shot | Instance Shot | Methods | Instance-level AUC | Bag-level AUC |
| :---: | :---: | :---: | :---: | :---: |
| 0 | 0 | CONCH | $0.8929$ | $0.7113$ |
| 1 | 16 | FAST-CLIP | $0.8400 \pm 0.0335$ | $0.6933 \pm 0.0846$ |
| 1 | 16 | FAST-CONCH | $0.9627 \pm 0.0132$ | $0.8418 \pm 0.0734$ |
| 2 | 16 | FAST-CLIP | $0.8584 \pm 0.0380$ | $0.7595 \pm 0.0391$ |
| 2 | 16 | FAST-CONCH | $0.9667 \pm 0.0115$ | $0.8399 \pm 0.0556$ |
| 4 | 16 | FAST-CLIP | $0.8864 \pm 0.0563$ | $0.7359 \pm 0.0853$ |
| 4 | 16 | FAST-CONCH | $0.9763 \pm 0.0036$ | $0.9326 \pm 0.0175$ |
| 8 | 16 | FAST-CLIP | $0.9060 \pm 0.0074$ | $0.7742 \pm 0.0249$ |  |
| 8 | 16 | FAST-CONCH | $0.9792 \pm 0.0024$ | $0.9507 \pm 0.0058$ |
| 16 | 16 | FAST-CLIP | $0.9151 \pm 0.0200$ | $0.8197 \pm 0.0474$ |
| 16 | 16 | FAST-CONCH | $0.9766 \pm 0.0036$ | $0.9570 \pm 0.0053$ |

$\textbf{Response to R6:}$ In WSI classification, we followed the settings from the original Tip-adapter paper. For aspects that were not directly applicable to WSI classification task, we made the following adjustments: 1. We designed a specific set of text prompts customized for WSI classification task. 2. We used all annotated patches to construct the cache model. Apart from these differences, the settings are consistent with those in the Tip-adapter paper.

$\textbf{Response to R7:}$
We sincerely apologize for any confusion we may have caused. We randomly selected a small number of instances from the core set for annotation. The remaining instances in the core set were not annotated but instead were all used in the training of the cached model. During training, the unannotated patches gradually learn the labels used for classification from the annotated patches. Since all patches in the core set are eventually used for model training, our model is relatively stable. The variance observed by the reviewer in Figure 3 comes from the random selection of bags. When the number of bags is 1 or 2, the variance is indeed large. We believe this is reasonable, as it is challenging to fit all the data with just one WSI. When the number of bags is greater than or equal to 4, the variance decreases considerably. Therefore, in practical applications, we recommend selecting 4 bags or more.
Thanks for the authors' efforts and responses. After carefully reading the replies, I still have the following concerns:
- **R2**: I am more concerned with the technical novelty since the proposed framework does incremental work to the existing Tip-Adapter. By the way, I do acknowledge the novelty of the two-level few-shot learning paradigm, as I mentioned in the Strengths.
- **R4**: Thanks for the experimental results. These results could be helpful to justify the value of this work. However, I cannot agree with the authors' claims made in R4.
  - "*Compared to CONCH, FAST significantly outperforms CONCH in the average bag-level classification AUC*". Is there any statistical test to verify the significance difference between the AUC of 0.9235 and 0.9141? Or, does this conclusion just come from a personal sense? I think in scientific research the conclusion given by the authors must be rigorous enough.
  - "*while CONCH relies on 1.17 million pairs of pathology images based on CoCa, resulting in a significantly high training cost.*". CONCH is a foundational model for pathology, just like CLIP. Here, discussing its efficacy and comparing it with the proposed FAST is not appropriate, because I) foundation models generally rely on massive data and large-scale pretraining, and ii) FAST also stands on the shoulder of such foundation models like GPT and CLIP, right?
  - Additionally, I mentioned a comparison with CONCH. It intends to encourage the authors to justify the significance and value of this work, not to question the validity of the experiments. Concretely, for example, if the foundational model, CONCH, could achieve an accuracy of 90% in zero-shot settings yet the FAST framework with CONCH only obtains 90.5% in few-shot settings, the improvement would be too marginal to demonstrate the value of the proposed few-shot FAST. 
  - "*Therefore, our method can be effectively combined with methods such as PLIP and CONCH to further enhance WSI classification performance.*". I failed to find the experiments on FAST + CONCH and see the improvements, so I think this claim, *i.e.*, it can further enhance WSI classification performance, may not be valid from my point of view. 
- **R6**: I cannot figure out the authors' implementation for Tip-Adapter in few-shot WSI classification, after carefully reading the authors' instructions. Could the authors please explain more? Thanks.
- **R7**: The authors mention that the model is relatively stable because the cache model includes a few labeled patches and a large number of unlabeled patches. It seems not obvious to me, since I just don't understand the logic behind the cause and consequence provided by the authors.
Thank you very much for your valuable suggestions. We have implemented FAST using the vision-language model CONCH from the pathology field and conducted experiments on the CAMELYON16 dataset. The experimental results are shown in Table A4.  We define the method using CLIP as the feature extractor as FAST-CLIP, and the method using CONCH as the feature extractor as FAST-CONCH. As shown in Table A4, compared to using CLIP as the feature extractor, using CONCH significantly improves the classification performance of FAST. Notably, the bag-level classification AUC can reach 0.957. This indicates that our method not only integrates well with V-L models like CLIP in natural images for WSI classification but also enhances the classification performance of V-L models in pathology.

$\textbf{Table A4: Results of using CONCH on CAMELYON16 dataset.}$
| Bag Shot | Instance Shot | Methods | Instance-level AUC | Bag-level AUC |
| :---: | :---: | :---: | :---: | :---: |
| 1 | 16 | FAST-CLIP | $0.8400 \pm 0.0335$ | $0.6933 \pm 0.0846$ |
| 1 | 16 | FAST-CONCH | $0.9627 \pm 0.0132$ | $0.8418 \pm 0.0734$ |
| 2 | 16 | FAST-CLIP | $0.8584 \pm 0.0380$ | $0.7595 \pm 0.0391$ |
| 2 | 16 | FAST-CONCH | $0.9667 \pm 0.0115$ | $0.8399 \pm 0.0556$ |
| 4 | 16 | FAST-CLIP | $0.8864 \pm 0.0563$ | $0.7359 \pm 0.0853$ |
| 4 | 16 | FAST-CONCH | $0.9763 \pm 0.0036$ | $0.9326 \pm 0.0175$ |
| 8 | 16 | FAST-CLIP | $0.9060 \pm 0.0074$ | $0.7742 \pm 0.0249$ |  |
| 8 | 16 | FAST-CONCH | $0.9792 \pm 0.0024$ | $0.9507 \pm 0.0058$ |
| 16 | 16 | FAST-CLIP | $0.9151 \pm 0.0200$ | $0.8197 \pm 0.0474$ |
| 16 | 16 | FAST-CONCH | $0.9766 \pm 0.0036$ | $0.9570 \pm 0.0053$ |
$\textbf{Q5:}$ It is suggested to rewrite Section 3.2 to make sure that all notations (those in texts and figures) and the implementation of components are clear and well-explained.

$\textbf{R5:}$ Thanks for your great suggestion on improving the quality of our manuscript. Firstly, we have revised the corresponding errors according to the suggestions in $\textbf{Q1}$. Secondly, we have corrected line 180 from “$\tilde{y}^{\text{cache}} = f_{\text{train}} F_{\text{train}}^T Y_{\text{train}}^I $” to “$\tilde{y}^{\text{cache}} = f_{\text{train}} F_{\text{train}}^T {Y_{\text{train}}^I}^ {{\prime}{\prime}}$”.  We have also added the definition of $P_{\text{train}}$ as follows. $P_{\text{train}} = [\{ p_{(1, L+1)}, p_{(1, L+2)}, \ldots, p_{(1, K_1)}\}, \{ p_{(2, L+1)}, p_{(2, L+2)}, \ldots, p_{(2, K_2)} \}, \ldots, \{ p_{(i, L+1)}, p_{(i, L+2)}, \ldots, p_{(i, K_i)} \} ]$ represents the pseudo-labels of all unannotated instances in ${{X}_{\text{train}}}^{\prime}$, where  $p$  is a learnable high-dimensional vector.


$\textbf{Q6:}$ The original Tip-Adapter is proposed for traditional single-instance settings. Its implementation for few-shot WSI classification is not clear.

$\textbf{R6:}$ We apologize for any inconvenience brought to you. We will add the implementation details of the Tip-adapter for few-shot WSI classification tasks to the supplementary materials. Additionally, we will open-source the code to facilitate further research by other researchers. The implementation details are as follows. We conducted experiments according to the settings of the optimal model in the original Tip-Adapter paper. For aspects that cannot be adapted to the few-shot WSI classification task, we used the following approach. 
1. We designed a set of text prompts specifically for pathology images, which has been proven superior in the CONCH comparison experiments we conducted. 
2. We used all annotated patches to build the cache model.


$\textbf{Q7:}$ Few-shot instances are randomly selected from the core set. Since only a few instances, e.g. 16, are selected from a very large instance pool and pathological patches (instances) often present heterogeneity, the final selected instance set could have a large variety and thus lead to unstable performances, calling into question the usability of the proposed method. This could also be observed from Fig 3 in the paper. So, the authors are encouraged to analyze and discuss the impact of randomly selected few-shot instances.

$\textbf{R7:}$ We apologize for any inconvenience brought to you. In our constructed cache model, not only does it include a few labeled patches, but it also contains a large number of unlabeled patches, making our model relatively stable. Additionally, as shown in Figure 3, when the instance shot reaches 16, the variance becomes relatively small.

$\textbf{Q8:}$ The original TCGA-RCC is not annotated at the pixel level. This work does a good job in terms of annotating the fine-grained region-of-interest of WSIs. However, if the annotation is not made public, this work would be difficult to follow and be very limited in research impact. The authors are encouraged to make their annotated dataset public.

$\textbf{R8:}$ Thank you very much for your suggestions on our work. To promote progress and development in the community, we will release the relevant datasets available for academic research.