SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We propose a prompt guided pooling strategy to aggregate instance features into bag features. The main idea is to first calculate similarity weights between the image features of each instance and the prototypes of multiple text descriptions, and then use the weighted average of all instance features in a bag as the bag feature.
Mathematically, assuming that  $X_{i}$  represents the current bag containing  $n_i$  instances, we use the Image Encoder  $f_{img}$  to extract instance features  $Z_{i} \in \mathbb{R}^{n_{i} \times m}$ , where  $m$  represents the dimension of the features.
$$
\boldsymbol {Z} _ {i} = f _ {\text {i m g}} \left(\boldsymbol {X} _ {i}\right) \tag {5}
$$
Then, we input multiple instance-level prompt groups  $T$  into the Text Encoder  $f_{text}$  to obtain multiple instance prototypes,  $P \in \mathbb{R}^{n_p \times m}$ , where  $n_p$  represents the number of prototypes, which corresponds to the number of instance-level prompt groups. It varies depending on the specific classification task.
$$
\boldsymbol {P} = f _ {\text {t e x t}} (\boldsymbol {T}) \tag {6}
$$
Next, we calculate the dot product of instance features  $Z_{i}$  and a set of prototypes  $P$ , and then perform softmax normalization by column (each column corresponds to a prototype), obtaining aggregation weights  $W_{i} \in \mathbb{R}^{n_{i} \times n_{p}}$  for each prototype with respect to the current bag. Next, we use the aggregation weights to obtain a set of weighted features  $W_{i}^{\top} \cdot Z_{i} \in \mathbb{R}^{n_{p} \times m}$ , and finally average the weights of all prototypes to obtain the bag feature  $F_{i} \in \mathbb{R}^{1 \times m}$ .
$$
\boldsymbol {W} _ {i} = \operatorname {S o f t m a x} \left(\boldsymbol {Z} _ {i} \cdot \boldsymbol {P} ^ {\top}\right) \tag {7}
$$
$$
\boldsymbol {F} _ {i} = \operatorname {m e a n} \left(\boldsymbol {W} _ {i} ^ {\top} \cdot \boldsymbol {Z} _ {i}\right) \tag {8}
$$
In addition, during training, we also constrain the minimum correlation between each instance prototype in  $P$  to avoid degeneration, which means that all prototypes are too similar to each other, with the following loss:
$$
\operatorname {L o s s} = \min  \left(\boldsymbol {W} ^ {\top} \times \boldsymbol {W}\right) \tag {9}
$$
This auxiliary loss aims to separate instance prototypes learned by each instance prompt, ensuring distinct phenotypes representing WSIs. Crucial instance prototypes for slide classification stand out during aggregation.
Table 1: Performance of bag-level classification on the Camelyon 16 Dataset.
Method	16-shot	8-shot	4-shot	2-shot	1-shot
Linear-Probe (Mean-pooling)	0.5816	0.5627	0.4930	0.3235	0.3357
Linear-Probe (Max-pooling)	0.5947	0.5814	0.5417	0.5401	0.5268
Linear-Probe (Attention-pooling)	0.7866	0.6738	0.6594	0.6187	0.5312
CoOp (Attention-pooling)	0.8012	0.6746	0.6639	0.6525	0.6462
Instance+Bag Prompt Learning (Ours)	0.8301	0.7287	0.7151	0.6958	0.6783
Table 2: Performance of instance-level classification on the Camelyon 16 Dataset.
Method	16-shot	8-shot	4-shot	2-shot	1-shot
Linear-Probe (Attention-pooling)	0.7508	0.6630	0.6316	0.6177	0.5934
CoOp (Attention-pooling)	0.8567	0.6648	0.6333	0.6221	0.5923
Instance+Bag Prompt Learning (Ours)	0.8896	0.7200	0.7142	0.7019	0.6938
5.1 Datasets, Evaluation Metrics and Comparison Methods
We comprehensively evaluated the instance classification and bag classification performance of TOP in FSWC tasks using three real-world datasets of different cancer types from different centers: the Camelyon 16 Dataset [2] for breast cancer, the TCGA-Lung Cancer Dataset² for lung cancer, and an in-house Cervical Cancer Dataset for cervical cancer. See Supplementary Material for detailed introductions to the datasets.
For both instance and bag classification, we use Area Under Curve (AUC) as the evaluation metric. However, it should be noted that only the Camelyon 16 Dataset has the true labels for each instance, while the other two datasets only have bag-level labels. Therefore, we evaluate the instance and bag classification performance of each method on the Camelyon 16 Dataset, and only evaluate the bag classification performance on the latter two datasets.
Because existing few-shot learning methods cannot be used in FSWC task, we constructed four baselines based on the current state-of-the-art few-shot learning methods CoOp [60] and Linear Probe [38]: (1) Linear-Probe (Mean pooling), (2) Linear-Probe (Max pooling), (3) Linear-Probe (Attention pooling), and (4) CoOp (Attention pooling). Specifically, we first used CLIP as the image feature extractor to extract all instance features within each bag. Then, we aggregated all instance features within a bag using simple Mean, Max pooling or learnable Attention Pooling [18] to obtain the bag feature. Linear-Probe indicates that we used a linear layer to perform bag-level classification on the aggregated bag feature. CoOp indicates that we used the bag-level label and learnable prompt to perform bag-level classification through prompt learning. We conducted few-shot classification experiments with 1, 2, 4, 8, and 16 labeled bags for each class.
We used the image encoder and text encoder of CLIP as the feature extractors for both images and text. The number of learnable parameters is empirically set to 10 tokens for both instance and bag prompt and other quantities can be used in practice. During training, we fixed all weights of CLIP and only trained the learnable parameters of bag prompt and instance prompt. For experiments with different shots, we randomly trained the network five times with fixed labeled bags and reported the average performance of each method. All comparative methods utilize the same labeled bags.
5.3 Results on the Camelyon 16 Dataset
The bag classification and instance classification performance on the Camelyon 16 dataset are shown in Tables 1 and 2, respectively. It can be seen that TOP achieved the best bag and instance classification performance in all few-shot settings, and significantly outperformed all comparison methods by a large margin. It can be observed that Linear-Probe with Mean/Max pooling can hardly work. Although using trainable attention pooling helps learn the importance of each instance and improves the performance of Linear-Probe, it still has limitations in performance. Prompt learning with fully trainable prompts in CoOp outperforms Linear-Probe. In contrast, our method used a two-level prompt learning paradigm, which achieved the best performance on both bag and instance
$^{2}$ http://www.cancer.gov/tcga
Table 3: Performance of bag-level classification on the TCGA-Lung Cancer Dataset.
Method	16-shot	8-shot	4-shot	2-shot	1-shot
Linear-Probe (Mean-pooling)	0.6022	0.5418	0.4934	0.4908	0.4646
Linear-Probe (Max-pooling)	0.6227	0.5547	0.5155	0.4985	0.4876
Linear-Probe (Attention-pooling)	0.7178	0.6539	0.6248	0.5832	0.5713
CoOp (Attention-pooling)	0.7840	0.6824	0.6811	0.6772	0.6801
Instance+Bag Prompt-Learning (Ours)	0.8235	0.8059	0.7531	0.7245	0.7123
Table 4: Performance of bag-level classification on the Cervical Cancer Dataset.
Method	16-shot	8-shot	4-shot	2-shot	1-shot
Linear-Probe (Mean-pooling)	0.6756	0.6684	0.6593	0.6246	0.6011
Linear-Probe (Max-pooling)	0.6322	0.6249	0.6038	0.5884	0.5869
Linear-Probe (Attention-pooling)	0.7345	0.7282	0.7155	0.6873	0.6137
CoOp (Attention-pooling)	0.7565	0.7349	0.7271	0.6927	0.6484
Instance+Bag Prompt-Learning (Ours)	0.8189	0.8007	0.7869	0.7618	0.7052
classification, with an average improvement of  $4.2\%$  and  $7.0\%$ , respectively, over the second-best method.
5.4 Results on the TCGA-Lung Cancer Dataset and the Cervical Cancer Dataset
The results on the TCGA-Lung Cancer Dataset are shown in Table 3. It can be seen that TOP still achieves the best bag classification performance in all few-shot settings, and significantly outperforms all competitors by a large margin, with an average improvement of  $6.3\%$  over the second-best method.
The results on the Cervical Cancer Dataset are shown in Table 4. It should be noted that this task is extremely challenging, and even pathologists cannot make direct judgments. TOP displays strongest performance, with an average improvement of  $6.3\%$  over the second-best method.
Table 5: Ablation results of bag-level classification on the Camelyon 16 Dataset.
Method	16-shot	8-shot	4-shot	2-shot	1-shot
Bag Prompt+Attention-pooling	0.8168	0.6980	0.6706	0.6673	0.6483
CoOp+Attention-pooling	0.8012	0.6746	0.6639	0.6525	0.6462
CoOp+Prompt guided pooling	0.8216	0.7079	0.6833	0.6732	0.6699
Bag Prompt+Prompt guided pooling (Ours)	0.8301	0.7287	0.7151	0.6958	0.6783
Table 6: Ablation results of instance-level classification on the Camelyon 16 Dataset.
Method	16-shot	8-shot	4-shot	2-shot	1-shot
Bag Prompt+Attention-pooling	0.8699	0.6753	0.6425	0.6399	0.5961
CoOp+Attention-pooling	0.8567	0.6648	0.6333	0.6221	0.5923
CoOp+Prompt guided pooling	0.8754	0.6912	0.6707	0.6515	0.6045
Bag Prompt+Prompt guided pooling (Ours)	0.8896	0.7200	0.7142	0.7019	0.6938
6 Ablation Study
We conducted ablation experiments on the two key components of TOP, the instance prompt guided pooling and bag-level prompt group. Experiments are conducted on the Camelyon 16 Dataset, and the bag classification and instance classification results are shown in Table 5 and Table 6, respectively. In the two tables, "Prompt guided pooling" represents the use of our proposed instance prompt guided aggregation method while "Attention-pooling" represents the use of attention-based method to aggregate instance features into bag features; "Bag Prompt" represents the use of our proposed Bag-level prompt group while "CoOp" represents using only the learnable part and bag labels as prompt at the bag-level prompt learning.
Effectiveness of instance prompt guided pooling. In both Table 5 and Table 6, no matter 'Bag Prompt' or 'CoOp' is used for the bag-level prompt learning, 'Prompt guided pooling' consistently results in significantly better performance than 'Attention pooling', which indicates the effectiveness of our proposed Prompt guided pooling strategy for instance feature aggregation.
Effectiveness of Bag-level prompt group. In Table 5 and Table 6, no matter what instance feature aggregation method is used, 'Bag-prompt' always outperforms 'CoOp', which indicates that our proposed Task descriptive prompt is better than current SOTA methods of fully learnable prompt.
Table 7: Ablation and sensitivity tests of the auxiliary loss on the Camelyon16 dataset.
Shot	Loss Weight	0	10	25 (ours)	50
1-shot	Bag AUC	0.6691	0.6759	0.6783	0.6771
Instance AUC	0.6837	0.6908	0.6938	0.6913
2-shot	Bag AUC	0.6867	0.6924	0.6958	0.6945
Instance AUC	0.6924	0.7008	0.7019	0.7011
4-shot	Bag AUC	0.7087	0.7123	0.7151	0.7154
Instance AUC	0.7033	0.7095	0.7142	0.7107
8-shot	Bag AUC	0.7137	0.7218	0.7287	0.7259
Instance AUC	0.7005	0.7188	0.7200	0.7243
16-shot	Bag AUC	0.8245	0.8281	0.8301	0.8299
Instance AUC	0.8834	0.8879	0.8896	0.8894
Effectiveness of Auxiliary Loss. Ablation and sensitivity tests on the Camelyon16 dataset (Table 7) demonstrate that our method is not highly sensitive to the loss weight, but its addition significantly improves performance compared to not using it.
In this paper, we introduce the novel problem of Few-shot Weakly-supervised WSI Classification (FSWC) for the first time. We proposed a Two-level Prompt Learning MIL framework named TOP to solve the FSWC problem effectively. TOP utilizes GPT-4 to generate both instance-level and bag-level visual descriptions to facilitate instance feature aggregation and bag-level prompt learning. Experiments on three WSI classification tasks show the high performance of TOP in FSWC tasks.
We carefully consider the validity, rationale, motivation and potential model updates' impact for using GPT-4's knowledge. For validity and rationale: We rigorously reviewed pathology knowledge descriptions from GPT-4 with three senior pathologists and found them accurate and detailed. Nori et al. [32] supports GPT-4's reliability in producing medical domain knowledge due to its vast medical expertise in training data. For motivation and importance: Leveraging GPT-4's knowledge as templates enhances efficiency versus manual design. This approach aligns with the few-shot learning goal, easing pathologist annotation. Manual templates might not cover all aspects; specialized doctors' templates could be needed for varied cancer types/tasks. Additionally, different doctors' descriptions vary, lacking a standardized manual description. By leveraging GPT-4's versatility, our aim is to attain knowledge descriptions for multiple cancer types and tasks while avoiding manual domain biases. For impact of model updates: GPT-4's language descriptions contributed to training pathology models in our research. We will publicly share all used descriptions, codes, and models. This disclosure ensures reproducibility in reported tasks without the need of invoking GPT-4 for inference or new training. GPT-4's upgrades won't influence current outcomes. We'll explore if GPT upgrades generate new descriptions and their effect on results.
As widely acknowledged, training a pathological foundation model demands a substantial corpus of correlated pathological images and textual data, in addition to significant computational resources. This poses considerable challenges in the field of computational pathology, characterized by high data privacy, scarce annotations, and extensive storage requirements. More recently, prominent research endeavors [25, 17] have entered the spotlight, aiming to develop a comprehensive vision-language model within the domain of digital pathology. Nevertheless, the focus of this study is to some extent orthogonal to these studies. The objective of our method is to develop new prompt-learning strategies based on an existing large vision-language model for few-shot learning. More importantly, our method can be combined with any large vision-language models (either in general natural image domain or pathology-specific domain) for few-shot WSI classification, not limited to the CLIP model used in the paper.
However, this paper has certain limitations, mainly due to the effectiveness of the proposed prompt depending on the capabilities of the large model. If the large model fails to provide effective descriptions, it can affect the model's performance. This study aims to inspire further research that combines foundational models with large-scale language models for the classification of pathology Whole Slide Images. Such research endeavors will herald a new era in AI pathology.