Enhancing Instance-Level Image Classification with Set-Level Labels
Authors: Renyu Zhang
Aly A Khan
Yuxin Chen
Robert L. Grossman
Venue: ICLR 2024 poster (2024)
Keywords: set-level labels
fast excess risk rate
representation learning
few-shot learning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: We present a novel approach to enhance instance-level image classification by leveraging set-level labels.

Abstract: Instance-level image classification tasks have traditionally relied on single-instance labels to train models, e.g., few-shot learning and transfer learning. However, set-level coarse-grained labels that capture relationships among instances can provide richer information in real-world scenarios. In this paper, we present a novel approach to enhance instance-level image classification by leveraging set-level labels. We provide a theoretical analysis of the proposed method, including recognition conditions for fast excess risk rate, shedding light on the theoretical foundations of our approach. We conducted experiments on two distinct categories of datasets: natural image datasets and histopathology image datasets. Our experimental results demonstrate the effectiveness of our approach, showcasing improved classification performance compared to traditional single-instance label-based methods. Notably, our algorithm achieves 13\% improvement in classification accuracy compared to the strongest baseline on the histopathology image classification benchmarks. Importantly, our experimental findings align with the theoretical analysis, reinforcing the robustness and reliability of our proposed method. This work bridges the gap between instance-level and set-level image classification, offering a promising avenue for advancing the capabilities of image classification models with set-level coarse-grained labels.

Published as a conference paper at ICLR 2024
ENHANCING INSTANCE-LEVEL IMAGE CLASSIFICA- TION WITH SET-LEVEL LABELS
Renyu Zhang
Aly A. Khan
Department of Computer Science University of Chicago zhangr@uch cago. edu
Department of Pathology and Family Medicine University of Chicago aa aakhan@ Schnnechhieee cago edu
Yuxin Chen
Robert L. Grossman
Department of Computer Science University of Chicago
Department of Computer Science and Medicine University of Chicago
chenyuxin@ueer cago. edu
rgrossman 1 @uch cago. edu
ABSTRACT
Instance-level image classification tasks, e.g., few-shot learning and transfer learn- ing, have traditionally relied on single-instance labels to train models. However, set-level coarse- -erriieg labels that capture relationships among instances also provide rich information real-world scenarios. In this paper, we present a novel approach to enhance instance-level image classification by leveraging set-level la- bels. We provide a theoretical analysis of the proposed method, including rec- ognizing conditions for fast excess risk rate, shedding light on the theoretical foundations of our approach. We conducted experiments on two distinct cat- egories of datasets: natural image datasets and histopathology image datasets. Our experimental results demonstrate the effectiveness of our approach, showcas- ing improved classification performance compared to traditional single-instance label-based methods. Notably, algorithm achieves 13% improvement in clas- sification accuracy compared to the strongest baseline on the histopathology im- age classification benchmarks. Importantly, the theoretical analysis, reinforcing the robustness and reliability of our proposed method. This work bridges the gap between instance-level and set-level image classification, offering a promising avenue for advancing the capabilities of image classification models with set-level boarse-grained labels.
experimental findings align with
INTRODUCTION
A large amount of labeled data is typically required in traditional machine learning approaches, e.g., few-shot learning (FSL) and transfer learning (TL), to learn a robust model. However, procuring sufficient labeled data for each task is often challenging or infeasible in real-world scenarios. In this paper, we consider novel problem setting where, similar to FSL, we have a limited number of fine-grained labels in the target domain. In the source domain, though, we have large amount of coarse-grained set-level labels, which are easier to obtain and relevant to fine-grained labels. For example, in digital library, there are coarse-grained set-level labels indicating the general content of photo albums, such as "beach vacation", "nature landscapes", or "picnic". However, within each of these albums, there are numerous individual images, each with its own unique details and characteristics that are not explicitly labeled. In the downstream task, for instance, we care about the object classification such as "tree". "beach", or "mountain". Similarly, in the medical domain, it is often useful to predict fine-grainee labels of tissues, while only set-level annotations of histopathology slides are available for training at scale. We seek to enhance the downstream classification tasks with the gaase-grained set-level labels.
An effective approach to addressing the overreliance on abundant training data is FSL a paradigm
that has gained significant attention in recent years (Vinyals al. 2016; Wang & Hebert, 2016
Triantaflllou et al. 2017; Finn et al. 20177] Snell et al. [2017] (Sung et al. 2018 Wang et al.
2018
Oreshkin al. 2018 Rusu et al. 2018 Ye al. 2018 Lee et al. 2019b al. 2019
FSL.

---

Published as a conference paper at ICLR 2024
vedia
bigenosuc sutder ooense
the guared kead
Parcher
TOGK ouarpii
XB
RACK
hamorta
iragu
Freg
TODA
(a)
Figure 1: (a) A collection of image sets sampled from CIFAR-100 are in the upper row. The coarse- grained label set is the most frequent superclass of images inside the set. WSI examples from TCGA and patches from NCT dataset are in the lower row. (b) Hierarchy of coarse- and fine-grain the-grained labels for histopathology images.
pretrains model that can quickly adapt new tasks using only a few labeled examples. Recent studies (Chen et al., 2019: Tian et al. 2020 Shakeri et al.) 2022; Yang et al. (0022) have shown that pretraining, coupled with fine-tuning on new task, outperforms more sophisticated episodic train- ing methods. This involves initially training base model diverse set of tasks using abundant labeled data from a source domain, and subsequently fine-tuning the model using only a small num- ber of labeled examples specific to the target task. Despite their promising performance, existing FSL models typically depend on finely labeled source data for predicting fine grained labels.
As illustrative example, consider histopathology image classification where acquiring sub- stantial number of fine-gratn ane-grained labels for individual patches (c.g.. tissue labels shown in the lower row of figure Tap is challenging. Conversely, wealth of coarse-grained labels (e.g. the site of origin of the tumors associated with whole slide images (WSIs) from TCGA shown on the left- hand side of the upper row of figure Tab are easily available. This motivates us leverage these abundant coarse-grained labels and hierarchical relationships, such as between organs and tissues (as depicted in figure lb), enhance representation learning. Tissues consist of cellular assemblies with shared functionalities, while organs are comprised of multiple tissues. This hierarchical rela- tionship serves as a conceptual foundation for our representation learning and provides significant contextual information for facilitating representation learning. By using coarse maarsegrrained information within this hierarchy, our goal is to learn efficiently fine-grained tissue representations within WSIs. Another example is shown in the upper row of figure Ia We emulate programmatic labeler that use hcuristics such as keywords, regular expression, or knowledge bases to solicit sets of images. The boarse-grainee labels, e.g., the most frequent superelass of images in the set, be used to facilitate representation learning for downstream tasks such as instance-level image classification.
Our contribution In this paper, introduce Fine-gr Fine-grAined representation learning from Coarse- grained LabEls (FACILE), a novel generic representation learning framework that uses easily ac- cessible coarse-grained annotations to quickly adapt new fine-grained tasks. Distinct from isting practices in FSL and TL, our approach utilizes coarse-grained labels in the source domain. This sets our methodology apart from conventional FSL and TL techniques, which typically rely on meticulously labeled source data to train models.
We provide initial theoretical analysis to motivate the empirical success of FACILE and examine the convergence rate for the excess risk of downstream tasks under novel Lipschitzness condition on the loss function concerning the fine-grained labels. Our study reveals that the availability of coarse-grained labels can lead to substantial acceleration in the excess risk rate for fine fine-grained label prediction tasks, achieving a fast rate of ((1/n), where n represents the number of fine- fine-grained data points. This analysis highlights the significant potential for leveraging coarse-grainel labels to enhance the learning process in fine-grained label prediction tasks.
In our experiments, we thoroughly investigate the effectiveness of FACILE through series of ex- tensive experiments on natural image datasets and histopathology image datasets. For natural image datasets, sample input sets from training data from 100 and use the unique superclass number and most frequent superclass as coarse- grained labels. The generated datasets are used to evaluate different models. We also evaluate models by fine-tuning the fully connected layer ap-
2

---

Published as a conference paper at ICLR 2024
pended to ViT-B/16 (Dosovitskiy et al. 2020) of CLIP (Radford et al. 2021) in an anomaly detec- tion dataset based on CUB200 (He& & Peng 2019). For histopathology applications, we leverage two large datasets with coarse- coarse-gramed labels pretrain models. Subsequently, evaluate the performance of these trained models on a diverse collection of histopathology datasets. Our algo- 4 downstream datasets. Notably, when tested LC25000 (Borkowski et al., 2022]. our model achieves roughly 90% average ACC with 1,000 randomly sam- pled tasks which only have 5 fine-grained labeled data points for each of the classes, outperforms the strongest baseline by roughly 13% with logistic regression fine- tne-grained classifier. We further evaluate various models by fine-tuning the fully connected layer appended to ViT-B/14 (Dosovit-) skiy et al., 2020) of DINO V2 (Oquab et al. 2023). These models can leverage the capability of "foundation" models and enhance the model performance on target tasks. Our experiments pro- vide compelling evidence of the efficacy and generalizability of FACILE across various datasets, highlighting its potential as a robust representation learning framework.
rithm achieves strong performance
2 FINE-GRAINED REPRESENTATION LEARNING FROM COARSE-GRAINED LABELS
Notation Our model pre-trains collection of samples, denoted by {(s, w;)) Each s; is an input set of instances - where a represents the variable input set size. {w;} are the set- level coarse-grained labels. The space of all instances is X' and the space of all instance labels, which we call fine-grained labels, is Y. The space of pre-training data is S x W, where S = { (xi...., }a 2j E X for Vjj E [a]} and W denotes the space of coarse- coarse-grained labels. We receive (X,Y) Y) from product space X' x ) and corresponding (S, W) from product space S x W. The goal predict the fine-grained labels y EY from the instance features x EX'
2.1 THE FACILE ALGORITHM
We study the model in a FSL setting where we have three datasets: (1) pre-training coarse-grained datasets DEg {(s,, w sampled i..d. from Ps, (2) fine-grained support dataset Dls { 34 } sampled i.i.d., from Px,,, and (3) query set Dquery, The support set Dis contains classes and k samples x in each class (i.e., n = kc). We assume a latent space 2 for embedding Z. We define instance feature maps & = fee X' - }]. set-input functions G = {g M - W) where
M = {f } 2j E 2 for j E [a]]. and fine-graun grained label predictors F = {f 2 y). The corresponding set-input feature map of instance feature map is defined as df S - M. We assume the class of f is parameterized and identify f with parameter vectors for theoretical analysis. We then learn feature map e, fine-grained label predictor f. and predict fine-graines label with f oe. The schema of our model is illustrated in figure 2
We assume two loss functions: d : y x y - R for fine-grained label prediction and fex : W x W + R for coarse- coarse-grained label prediction. eon measures the loss of the fine-grained label predictor.
{ Xx
X1
X.
8
T
tr
We assume this loss is dif- ferentiable in its first argument. for measures the loss of pre-training with coarse-grained labels. For theoretical anal- ysis, we are interested in two particular cases of ges i) (s (",( w') = [w + w'} where W is a categorical space; and ii) (w, w') = || - w'll (for some norm ||-I] on W) where W is continuous space. We can also measure the loss of a feature map e by Eor = 8o o &² (s), w), where ge E arg min, E.S. (g (S),, W). We assume there is an unknown "good" embedding M = (S) E M, by which a set-input function geg can determine W, i.e., Deg (M) =
Z
Za
Za
}
M
f
3/ 1/2
Y,
W
Figure 2: Schema of the FACILE model. The dotted lines represent the flow of fine-grained data, and the solid lines denote the flow of coarse-grained labels
Gro (S) W. The strict assumption of equality can be relaxed by incorporating an addi- tive error term into our risk bounds of 9en o d
Our primary goal is to learn an instance label predictor or fine-grained label predictor f a é that achieves low risk Epx,y [ o é(X), Y)] and we can bound the excess risk:
E E L (f (fo (fo((X)))) (s((S" o oe (X),
(1)
3

---

Published as a conference paper at ICLR 2024
Algorithm FACILE algorithm
Input: loss functions éع, 8 predictors & 9, datasets D and Dis
2: obtain feature map é <- A(e85, Do E)
"i==1 4: obtain fine-grained label predictor f é, where f + f A(els, Dig.aus F)
3: create dataset Dig.aug = {(z, 31) 24 = ē(zi), (Ii, vi) E Dig)n
5: Return: foe
Protraining FACILE w/ coarco grzined loools
Testing FACILE with grained lobels
Salling
label
support samples
labet vegutaables
sel Havel
FSP
neppoose Balkon
poor
g
arsage
SSppCon
mushrom
Figure 3: An overview of the FACILE algorithm. (a) Pre-training step of FACILE with coarse- grained labels. The input is a of images and the target set-level coarse coarse-grained label. is an instance feature map and of is the corresponding set-input feature map. is the set-input model. We can instantiate the A(ex, Dec E)) with any supervised learning algorithms, e.g., fully supervised pre- training (FSP) with cross-entropy loss and the SupCon model. (b) Fine-grained learning of FACILE with fine-grained labels. The learned instance feature map extracts instance-level features from patches of the support set and query set. f is the fine-grained label predictor.
where e" E arg min Eps. w (S, W) and f* E arg migminjee Epx.y ( eX),
The pseudocode for FACILE is provided in Algorithm ! and further illustrate the FACILE al- gorithm in figure Given an input set s; comprising instances Ti,... the feature map is employed extract instance-level features for all the instances within the input set. Subsequently, set-input model is utilized to generate set-level features based on the instance-level features. generic algorithm that compatible with any super- vised learning method in its pre-training stage. We chose SupCon (Supervised Contrastive Learn- ing) (Khosla et al. 2020) and FSP as they are representative the two main approaches within supervised learning: contrastive and non-contrastive (traditional supervised) learning, respectively. During testing, we extract the pre-trained feature map and fine-tune a classifier ff using the gen- erated embeddings from è and the fine-grained labels of the support The performance of the classifier f is then reported for the query set. Note that Algorithm is generic since the learning steps can use any supervised learning algorithm.
Our FACILE framework designed
2.2 THEORETICAL ANALYSIS
We denote the underlying distribution of D as Ps.w and the underlying distribution of Dfit as Px,y. We assume the joint distribution of Z and Y is Рz,.. After we learn the feature map can define new distribution Pz,y = P(Z, Y 1 The Dig.aug is i.i.d, samples from Pz,y. In order to include the underlying distribution of Deg and Dís into analysis, with a slight abuse of notation we use A.. (f Ps,w,E) )) to denote A(e*, Dog E) and use A, F)) to denote A(es Dig.aus, The two learning algorithms are described as follows.
é(X)), where 1 is the indicator function.
Definition Coarse-grained learning; pretraining) Let Ratem, (f% E; (abbreviated to Rate, (698, Ps,w,&) E)) be the rate of A. (e95, , E) which takes 6o5, E and m i.i.d. observations from W as input, and return a feature map E E such that
2o8 (S, W) < Rato, (e E; Eps.
with probability at least 1-6 8.
4

---

Published as a conference paper at ICLR 2024
Definition 2. (Fine-grainet learning: downstream task learning) Let Rate, Pz, F: (abbre- viated to Raten (C's, , be the excess risk rate of An (L ,z,y, which take ftz F, and i.idd observations from distribution P2,y as input, and returns a fine-grate fine-grained predictor F such that P2.
( (2, Y) f. (2, I Rate,, P2,x, F;; F;6) with probability at least 6.
Next, introduce our relative Lipschitz assumption and the central condition for quantifying task relatedness. The Lipschitz property requires that small perturbations to the feature map that do not harm the pre-training task, do not affect the loss of downstream task much either.
Definition 3. We say that f is L-Lipschitz relative if for all E E, s, y E V, and e, e' e'E E,
(f a e(f)) - (f o < (e o o o (0 (s))
The function class F is L-Lipschitz relative to &, if every f E F is L-Lipschitz relative to E.
Definition generalizes the definition of L-Lipschitzness in Robinson et al. 2020) to bound the downstream loss deviation through the loss of the set label predictions. In the special case where
8 = {z), and g is a classifier for the pretraining labels, our Lipschitz condition reduces to the Lipschitzness definition of Robinson al. (2020).
The central condition is well-known to yield fast rates for supervised learning (Van Erven et al., 2013). Please refer to definition 6 for the definition of central condition. We show that surrogate problem (6% F) satisfies a central condition in proposition
Theorem 4. Suppose that (Exx, P,,,, F) satisfies the central condition, F is L-Lipschitz rela- tive to E. fig bounded by B > O, F is L' L'psiphhitzz in its d-dimensional parameters in the norm, FF is contained in the Euclidean ball of radius R, and is compact. We also as- Then when An P), ERM we ob- (X,Y) - -foee (x)ooe-(XXY)) bound with probability at least 1 - 5 by
sume that Ratem (8 Pss,, =0 (1/m²²)
tain excess risk EPX,Y 1
for
O do log RT. n4l'g.
if = l(n and (w w/) = 1( + w'}. + B+2L
For a typical scenario where Rate,, Ps...)) =
we can obtain fast rates with m (nn2). Similarly, in the scenario where Am Ps.w. achieves fast rate, i.e., Ratem (e*x, E) (l1mm), one can obtains fast rates when m M(n). More generally, a. 2 1, we observe fast rates.
We prove our theorem by first showing that the excess risk of f can be bounded by 2LRate (8%5, (s,,, +Rae,, F2 in proposition
Then, we show that (elx, Fz) also satisfies the weak central condition in proposition Thus, Rate, (L* Pz,y, F) is also bounded by proposition g We refer interested readers to section H for full details the proof.
5
In the next section, we first aim to empirically study the relationship between generalization error, coarse-gres coarse-grained grained dataset size, and fine- fine-grained dataset size that our theoretical analysis predicts in section 3.3 and section 3.4 We also demonstrate the exceptional efficacy of the proposed algorithm compared to baseline models on natural image datasets and bistopathology image datasets.
3
EMPIRICAL STUDY
3.1
BASELINE MODELS AND ALGORITHM INSTANTIATION
We consider two sets of baseline models: self-supervised models (Bachman et al., 2019 He et al.)
2020 Chen al., 22200) Caron et al. 2020; Grill et al. 2020 Chen & He 2021) and weakly super- vised models (Donahue et al. 2014 Sun et al. 2017 Zeiler & Fergus 2014 Robinson et al. 2020
Self-supervieed models Given pre-training data (S, W). self-supervised learning models ignore the labels W and lear é from S. Then, we can test é with new task, which consists of support set and a query set. new model that leverages the learned è is fine-tuned on the support set and tested on the query set. We performed two self-supervised learning models in two categories, e.g., SimCLR (Chen et al. 2020) for contrastive learning and SimSiam (Chen & He 2021) for non-contrastive learning. Details of these self-supervised learning algorithms can be found in section G
5

---

Published as a conference paper at ICLR 2024
Weakly supervised models We assign each instance, from the pre-training dataset, label of the input set to which it belongs. We train feature map è appended with a linear classifier on the pre- training dataset. We call this model FSP-Patch, where FSP stands for fully supervised pre-training and the model is trained with the assigned instance-level labels. For a new task with a support set and a query set, use the extract features for both sets, train classifier the support set features, and test the classifier on the query set features.
Following previous work in FSL (Tian et al. 2020 Chen et al. 2019), we use Iz-normalized fea- tures for downstream tasks. Unless otherwise specified, we evaluate methods with 1,000 randomly sampled meta-tasks from each dataset. All meta-tasks use 15 samples per class as the query set. The average FI/ACC and 95% confidence interval (CI) are reported. We follow the test setting of Yang et vll. 2022) and use NearestCentroid (NC), LogisticRegression (LR), and RidgeClassifier (RC).
3.2 PRETRAIN WITH UNIQUE CLASS NUMBER OF INPUT SETS
In order show the advantages of using the coarse-grained labels, we introduce a new task of pre-training with the unique class number of input sets. Inspired by Lee et al. (2019a), we use the CIFAR-100 (Krizhevsky et al., 2009) dataset, which contains 100 classes grouped into 20 super- classes. We generate input sets by sampling between 6 and 10 images from CIFAR-100 100 training data. The targets of the input sets are the unique superclass number of the input sets. In our down- stream tasks, we perform few-shot classifications of fine- tne-grrined classes. Despite being distinct from the downstream fine-grained labels, the coarse granned labels offer useful information for learning useful representations for downstream tasks.
pretraining method
unique superclass number
most Irequent superelass
NC
L.R 75.88 T.I
RC
NC
I.R
RC
SimCLR
76.07 + 0.99
75.50 + 1.02
75.91 + 1.00
75.82 = 1.01
75.91 = 1.02
SimSiam
78.15 + 0.93
79.44 + 0.92
79.03 + 0.95
78.80 + 0.93
79.44 - 0.95
79.43 + 0.93
FSP-Patch
N/A
N/A
N/A
73.21
0.97
73.92
0.98
73.10 0.98
FACILE-SupCon
N/A
N/A
N/A
79.54 + 0.92
79.54
0.96
79.12 + 0.95
FACILE-FSP
86.25 = 0.79
85.42
+ 0.82
85.84 = 0.81
82.04 = 0.84
81.70
1.91
81.75 = 0.90
Table 1: Pretraining input sets from CIFAR-100. average Fl and CI are reported.
Testing with 5-shot 5-way meta-test sets;
The ResNet18 (He et al.) 2016) is used as feature maps é. For FACILE-FSP, we pre-train the fea- ture map é from these imput sets and targets with l loss. The features of CIFAR- 100 test images extracted with è. Training settings of SimSiam, SimCLR and FACILE-FSP can be found in section A.I We then test é in a few-shot manner. We random sample 5 classes, 5 examples from each class, for each meta-test dataset. The fine-grained label predictor f is trained on the support examples and tested on the query examples. The performance of these models is reported in table T We can see that FACILE-FSP outperforms self-supervised learning models by large margin.
3.3 PRETRAIN WITH MOST FREQUENT CLASS LABEL
We sample input sets randomly from training data of CIFAR-100. The targets are the most frequent superclass of the input sets. If there is tie in an input set, we choose a random top frequent superclass as the target of the input set. Training settings are similar to section 3.2 and can be found in section | The performances of all models are reported in table We can see that FACILE-FSP obtains better results compared to other models.
Note that the excess risk bound of the form C/n" implies a log-linear relationship log log C CC-y log n between the error and the number of tine-graieed labels. We can visually interpret the learning rate y. We study two cases: when the number of coarse-grand -rainedd labels m grows linearly with the number of fine-graines quadratically with the number of fine-grained labels. In order to show the generalization error rate of FACILE-FSP w.r.t. fine-graince label number 5 classes (i.c., 5-way testing) for each task. We then sample n/5 fine- greeggrriedd examples in cach class for the support set and sample 15 examples for each class for the query set. The curves shown in figure 4 The figure shows the log-linear relationship of FACILE-FSP's generalization error on downstream tasks w.r.t. fine-grained label number. This visualization effectively captures how coarse caasse-grained label number m impacts the model's generalization capabilities.
labels, and when the number of coarse-grained labels m grows
CIFAR-100 test dataset, we randomly sample
3.4 EVALUATION ON HISTOPATHOLOGY IMAGES

---

Published as a conference paper at ICLR 2024
Datasets and data extraction We pretrain our models using two independent sources of WSIs. First, we down- loaded data from The Cancer Genome Atlas (TCGA) from the NCI Genomic Data Commons (GDC) (Heath et al. (02T). Two collections of non-overlapping patches with different patch sizes, i.c., 224 x 224 and 1, 000 x 1, 000 at 20X magnification. Background patches with high or low intensity were removed. Because the number of patches generated with size 224 224 20X magni- fication is very large, at most 1, 000 randomly selected patches are kept for each slide. The names of the tu- mors/organs, from which slides are collected, are used as coarse-gras grasseed labels. Second, we downloaded all clini- cal slides from the Genotype-Tiss Tissue Expression (GTEx) project (Lonsdale et al.] 2013), which provides a resource for studying human gene expression and regulation in relation to genetic variation. We extracted non-overlapping patches with size 1, 000 x 1, 000 at 20X magnification and patches with intensity larger than 0. and smaller than 0.85 are kept. For these slides, we used the organs from which the tissues were extracted as coarse- grained labels. Examples and class distributions for the two datasets can be found in section C
22 219
Generalization error (with two growth rates) of FACILE-FSP on CIFAR-100 test dataset as function of the number of coarse-grained labels m.
Figure 4:
We test models on public datasets: LC (Borkowski et al., 202I), PAIP (Kim al. 202I). NCT (Kather et al. 2018) and private dataset PDAC. Details of these datasets are deferred to section C Note that the TCGA and GTEx have meticulously categorized an extensive array of cancer types and organs, covering a diverse range of tissues as outlined in the LC, PAIP, and NCT. The strategic use of WSI-level labels is rooted in their potential enrich tissue-level classification. While these labels may appear broad, they encapsulate a wealth of underlying heterogeneity inherent to different cancer regions and tissue types.
Pretrain on TCGA with patch size 224 x 224 We first train models on TCGA patches with size 224 x 224 at 20X magnification. After the models are trained, we test the feature map in these models on LC, PAIP, and NCT. Full details about FACILE-FSP. FACILE-SupCon, and baseline models' training settings can be found in section A.3 Latent augmentation (LA) has been shown to improve FSL performance for histopathology images (Yang al. (022). We use faiss (Johnson jet al. 2019) to perform k-means clustering. Following the setting of Yang et al. (2022), the number of prototypes in the base dictionary is 16. Each sample is augmented 100 times by LA. We refer readers to section[Efoor details of LA.
Main results The test result is shown in table 2 In or- der to show the performance improvement over models pre-trained on natural image datasets, we report the per- formance of the FSP model pre-trained on ImageNet. We can see from table 2 that our model FACILE-FSP per- forms the best, with large margin compared to other models. The contrastive learning model SimCLR per- forms worse than non-contrastive learning model Sim- Siam. A possible reason could be the small batch size we used for SimCLR. SimSiam maintains high performance even with small batch sizes. FSP-Patch achieves better performance compared to self-supervised learning mod- els and the ImageNet pretrained model, which shows the usefulness of the coarse-grained labels for downstream tasks. More experiment results about test ACC on LC. PAIP and NCT datasets can be found in section B.2.1 Test result with larger shot number is in section B.2.2
Figure 5: Generalization error on NCT dataset. The FACILE-FSP trains TCGA dataset with m coarse-grained labels. We show the error curve with two growth rates of m.
We further pre-train models on GTEx and TCGA with patch size 1, 000 x 1, 00000 and test the mod- els on our private dataset PDAC. We refer readers to section B.5 for experiment results on PDAC dataset.
We show the generalization error of FACILE-FSP w.r.t. fine-grained label number in figure 5 The figure reveals a pronounced log-linear relationship. A larger growth rate of coarse-grained labels implies faster rate of excess risk.
7

---

Published as a conference paper at ICLR 2024
pretraining method
NC
LR
RC
LR+LA
RC+LA
-shot 5-way test
LC dataset
ImageNet (FSP)
63.26 1 1.46
63.13 1 1.41
63.24 1 1,40
64.51
1.41
64.95 1,30
SimSiam
65.83 + 1.32
66.52 + 1.31
66.24 + 1.32
67.21
1.29
67.83
1.33
SimCLR
64.37 + 1.36
63.85 + 1.37
64.16
1.37
65.78
1.33
66.81
1.40
FSP-Patch
66.73 1.29
66.25
1.29
66.59
1.28
68.01
1.24
68.28
1.26
MACILE-SupCor
74.91
1.25
76.23
1.16
75.01
1.19
75.60
1.19
75.
1.18
FACILFF-FSS
1.21 77.39
76.14
1.25
75. 18 + 1.30
77.55 1.17
73.72
1.34
5-shot 5-way test
LC dataset
ImageNet (FSP)
82.82
80.13 0.82
80.23 = 0.83
84.70 0.70
84.42
0.74
SimSiam
85.12 0.68
82.69
82.80 + 0.76
87.45 - 0.63
87.50
0.66
SimCLR
0.77
81.93 0.83
81.40 + 0.89
85.69 0.73
84.93
0.79
FSP-Patch
84.96
$4.10
$4.45 0.68
86.31 0.65
86.29 0.68
FACILE-SupCOn
91.09 0.447
90.34 0.48
90.25 0.48
91.32 0.47
90.94
0.50
FACILF-FFS
91.67 + 0.45
90.64 0.50
90.52 0.52
92.07 0.48
0.61 89.81
I-shol 3-way Test
PATP datasel
ImageNet (FSP)
45.96 + 1.22
47.82 1.29
47.43 1.29
46.38
1.24
44.90
1.24
SimSiam
46.43 + 1.21
47.93 1193+1.2 1.24
47.74 1.23
47.20
1.21
46.31
1.22
SimCLR
44.51 1.16
46.44 1.14
1.15 45.59
1.14 45.40
45.04 1.16
FSP-Patch
48.85
49.44 1.26
50.27
1.22
1.20 49.76
48.
08.44=
1.21
FACILE-SupCOo
46.60 1.20
48.63 1.22
1.21 48.46
47.13
47.87
1.22
FACILE-FSP
1.24
46.71
1.20
46.60 + 1.21
1.22 46.36
1.20 45.49
S-shol 3-way Test
PAIP dataset
ImageNet (FSP)
60.73 1.02
61.21 11221112
61.04 +111
61.66 = 0.91
59.30 0.93
SimSiam
62.88 0 0.97
62.59 1.08
63.48 + 1.04
65.01 + 0.88
63.22
0.89
SinCLR
60,99 + 0.93
61.38 1.00
62.39 = 0.91
61.29 = 0.90
FSP-Patch
64.45
64.60 0.98
64.49 0.99
64.08 0.89
62.79 0.89
FACTLE-SupCOn
64.74 0.91
65.63 0.97
65.93
0.97
66.68 0.86
66.48 0.82
FACILE-FSP
0.94
64.59 0.96
65.43
0.96
66. 77 0.86
66.34 0.85
T-shot 9-way lest
NCT dataset
ImageNet (FSP)
57.35 1,68
56.39 + 1.64
56.08 = 1.64
57.78
1.66
56.85
1.64
SimSiam
63.60 +: 1.62
64.43 + 1.54
64.79 + 1.53
65.26
1.56
65.30 1.53
SinCLR
59.73 + 1.57
59.61 1.57
59.34 + 1.56
60.57 = 1.57
60.99
1.53
FSP-Patch
60.08 + 1.46
61.55 + 1.50
62.32 - 1.50
61.99 1.42
60.62 1.38
FACILE-SupCon
68.10 1.29
69.63 1.25
69.81 1.24
69.54 1.25
69.
1 1.22
FACILE-FSP
66.38 1.38
67.03
1.34
67.56
1.32
68.35 1.33
69.
1 30
5-shol 9-way Test
NCT dataset
ImageNet (FSP)
74.59 1 1.11
73.21 1.13
74.60 1 1.07
76.68 1.04
74.39 1.09
SimSiam
79.97 +: 1.05
79.81 + 1.03
80.84 0,98
83.45
0.92
83.61
0.90
SinCLR
76,80 4: 1.09
76.95 + 1.07
78.25 + 1.03
80.54 = 0.97
81.13 = 0.95
FSP-Patch
79.50 0.94
79.54 1 0.95
81.00 1 0.88
82.42 0.81
81.33 0.79
FACILE-SupCon
86.79 :: 0.61
87.89
0.58
89.10
0.52 87.44
89.53
0.52
88 0.54
FACILE-FSP
84.68 0.74
85.47
0.72
0.64
88.00 0.63
87,51 0.66
Table 2: Test result on LC, PAIP, and NCT dataset; average F1 and CI are reported.
Benefits of pre-training on Large Pathology Datasets In order to show the benefits of pre-training on large pathology datasets, we pre-train different models on the NCT training dataset and test the performance on the LCC dataset, following the setting of Yang et al. (2022). Instead of separating the
mixture-domaii and out-domain tasks, we directly report the average Fl and CI of LR models over
all 5 classes of the LC dataset. Training details of the models can be found in section B.4 The test result on the LC dataset is shown in table 3
We can see from table 3 the best model pre-trained on NCT, i.e., FSP with strong augmentation, performs worse than model FACILE-FSP in table Our method get roughly 13% improvement compared to Yang et al. (202)) on the LC dataset. The large margin between the two best models pre-trained on two different datasets shows the importance of pre-training with large number of
coarse-grained labels. More results on LCC and PAIP can be found in figure B Note that SimSiam
model, trained with a batch size of 55, maintains competitive performance MoCo v3 which needs a large batch size.
pretraining method
NC
I.
RC
LR+LA
RC+LA
SimSiaa MoCo v3 ( Yang et al. 2022
76.21 1 0.87
7100 2 1.10
74.59
1 1.10
77.87 0.87
76.03 = 0.94
72.82 + 1.25
70.29
1.43
71.31
+ 1.40
78.72
1.72:1.00 1.00
79.71 = 0.95
HSP (simple aug, Yang al 2022
56.44
1.50
52.27
1.81
55.62
1.74
63.47
1.37
63.47 1.46
FSP (strong sug) SupCon
83.53
0.79
80.81
+ 1.01
80.27
1.08
85.57
0.77
84.06
0.89
81.51 0.85
78.77
= 1.03
78.65
4 1.08
83.51
4 0.84
83.31 = 0.91
Pretraining on NCT and 5-shot 5- testing on LC dataset; average Fl and CI are reported. More experiments and ablation study We refer interested readers to section for ablation studies about set size, training procedures, and set-input models. We provide additional insights through
Table 3:
8

---

Published as a conference paper at ICLR 2024
fine-tuning experiments. In Appendix |.. we detail the fine-tuning of the ViT-B/16 (Dosovitskiy iet al., 2020) from CLIP (Radford et al.] using CUB200-basee anomaly detection data (He & Peng] (2019). Similarly, Appendix [B.3 discusses fine-tuning the ViT-B/14 model of DINO V2 (Oquab et al. (023) on TCGA dataset. These experiments extend our analysis to specialized tasks, showcasing the adaptability of FACILE foundation models.
RELATED WORK
Weakly supervised learning The concept of weakly supervised learning introduced as a means to alleviate the annotation bottleneck in the training of machine learning models. There has been a large body of existing work in learning with only weak labels. A comprehensive survey about weakly supervised learning is provided in Zhou (2018) Zhang et al. (2022). We study novel form of weak supervision which is provided by set-level coarse coarse-grained labels. Among weakly supervised learning methods, [Robinson al. (2020) studied the generalization properties of weakly supervised learning and proposed a generic learning algorithm that can learn from weak and strong labels and can be proved to achieve fast rate. The authors consider different setting where each instance has a weak label and a strong label, and the strong label predictor learns to predict the strong labels from the instances and their corresponding embeddings learned with weak labels. We consider the setting where we have some coarse-grainc labels of some sets, rather than instances and the downstream classifiers only use the learned embeddings to train and test on the downstream tasks.
Multiple-instance learning for WSIs WSI classification and regression are formulated based on multiple-instance learning (MIL) (Campanella et al., 2019] Xu al. 2022 Isse et al. [0188] Sharma et al. 2021 Hashimoto et al. 2020 Shao et 2021 Yao et al. 2020 Lu et 2021b a Chen et al.,
2021b et al. 2021 Chen et al. 2021a Myronenko et al. 2021 Xiang & Zhang 2022 laved et al,
These MIL models employ two procedures: i) feature extraction for patches cropped from a WSI and ii) aggregation of features from the same WSI. ImageNet pretrained backbones, self- supervised backbones pretrained on histopathology images, or backbones fine fine-tuned during training are used to extract features from patches. Deep attention pooling, graph neural networks, or se- quence models, adapted for WSIs, are used for feature aggregation. In this paper, we consider different problem setting where we enhance patch-level classification with related set-level labels. In the application of histopathology images, line 2 of generic algorithm can be instantiated with any MIL models that have the backbones with trainable modules to extract patch-level features, c.g., Isse et al. 2018 complete comparison of MIL models for WSIs out of the scope of this paper.
2022).
Learning from coarsely-labeled data Another related line of research is Phoo & Hariharan 2022 where the authors assume a taxonomy of classes with two levels, i.e., a set of fine aene-grined classes that are more challenging to annotate and set of coarse -rarieed classes that are easier to annotate. In our paper, we not assume taxonomy of classes for the boarse-grained and fine-graines labels. The coarse-grained and fine-grained labels are closely related via hierarchy. Also, the inputs that are fed to models to predict the coarse-graines or fine- fine-grained labels are different, i.e., set input for coarse-grasd conse-grained labels and instances for fine-grained labels.
5 CONCLUSION AND DISCUSSION
Summary We introduce FACILE, representation learning framework that leverages coarse-grained labels for model training and enhances model performance for downstream tasks. Our theoretical analysis highlights the significant potential of leveraging set-level labels benefit the learning pro- cess of fine-grad ine-grained label prediction tasks. To demonstrate the effectiveness of FACILE, we con- duct pre-training on CIFAR- 10sebased datasets and two large public histopathology datasets using coarse-grained labels and evaluate our model on diverse collection of datasets with fine-grained labels.
Limitation and future work In this paper, we consider a novel problem setting where we enhance downstream fine-grained label classification with easily available coarse-grained labels and propose generic algorithm that contains two supervised learning steps. It is important to note that the separate utilization of loosely related coarse- coarse-grained labels and fine-grained labels can be expensive. Specifically. the pre- traiirann of our proposed algorithm could be expensive given large amounts of coarse- coarse-grained grained data and the nature of the set-input data. For this reason, we are investigating methods of selecting a subset of the grained dataset accelerate pre-training.

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGEMENT
The research was funded in part by the Center for Translational Data Science at the University of Chicago, the Research Computing Center the University of Chicago, and the National Science Foundation under Grant No. 2313130. We extend our deepest gratitude to Christopher R. Weber for his generous provision of the PDAC dataset, which was invaluable the completion of this research.
REFERENCES
Brigham & Women's Hospital & Harvard Medical School Chin Lynda 9 11 Park Peter J. 12 Kucher- lapati Raju 13, Genome data analysis: Baylor College of Medicine Creighton Chad J. 22 23 Donehower Lawrence A. 22 23 24 25, Institute for Systems Biology Reynolds Sheila 31 Kreis- berg Richard B. 31 Bernard Brady 31 Bressler Ryan 31 Erkkila Timo 32 Lin Jake 31 Thorsson Vesteinn 31 Zhang Wei 33 Shmulevich Ilya 31, et al. Comprehensive molecular portraits of human breast tumours. Nature, 490(7418):61-22 2012.
Philip Bachman, R Devon Hjelm, and William Buchwalter. Learning representations by maximizing mutual information across views. Advances in neural information processing systems, 32, 2019.
Andrew A Borkowski, Marilyn M Bui, L Brannon Thomas, Catherine P Wilson, Lauren A DeLand, and Stephen M Mastorides. Lc25000 lung and colon histopathological image dataset, 2021.
Gabriele Campanella, Matthew G Hanna, Luke Geneslaw, Allen Miraforr, Vitor Werneck Krauss Silva, Klaus Busam, Edi Brogi, Victor E Reuter, David S Klimstra, and Thomas Fuchs. Clinical-grade computational pathology using weakly supervised deep learning whole slide images. Nature medicine, 25(8):13011109) 2019.
Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for un- supervised learning of visual features. In Proceedings of the European conference computer vision (ECCV), pp. 132-149, 2018.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
Richard Chen, Ming Lu, Muhammad Shaban, Chengkuan Chen, Tiffany Y Chen, Drew FK Williamson, and Faisal Mahmood. Whole slide images are 2d point clouds: Context-aware sur- vival prediction using patch-based graph convolutional networks. In Medical Image Computing and Computer Assisted Antervention-MICCA. 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part VIII 24, pp. 339-3499999333 Springer, 2021a.
Richard J Chen, Ming Lu, Wei-Hung Weng. Tiffany Y Chen, Drew FK Williamson, Trevor Manz, Maha Shady, and Faisal Mahmood. Multimodal co-attention transformer for survival prediction in gigapixel whole slide images. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 4015-4025, 2021b.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020.
Wei- Yu Chen, Yen-Cheng Liu, Zsolt Kira. Yu-Chiang Frank Wang, and Jia-Bin Huang. A closer look at few-shot classification. arXiv preprint arXiv: 1904.0422, 2019.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15750-15758, 2021.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021c.
Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hierarchical Image Database. In CVPR09, 2009.
10

---

Published as a conference paper at ICLR 2024
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In Inter- national conference machine learning, pp. 647-655. PMLR, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprini arXiv:2010.11929, 2020.
Simon S Du, Wei Hu, Sham M Kakade, Jason D Lee, and Qi Lei. Few-shot learning via learning the representation, provably. arXiv preprint arXiv:2002.00 09434, 2020.
Harrison Edwards and Amos Storkey. Towards
neural statistician.
arXiv preprint
1rXiv:1606.02188 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International conference on machine learning, pp. 1126-1135. PMLR, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing systems, 01-21271-21288 2020.
Noriaki Hashimoto, Daisuke Fukushima, Ryoichi Koga, Yusuke Takagi, Kaho Ko, Kei Kohno, Masato Nakaguro, Shigeo Nakamura, Hidekata Hontani, and Ichiro Takeuchi. Multi-scale for cancer subtype classification with unannotated
domain-adversarial multiple-instance histopathological images. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 3852-3861, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778. 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9729-9738, 2020.
Xiangteng He and Yuxin Peng. Fine-grained visual-textual representation learning. IEEE Transac- fions Circuits and Systems for Video Technology, 30(2):520-55) 531, 2019.
Allison P Heath, Vincent Ferretti, Stuti Agrawal, Maksim An, James C Angelakos, Renuka Arya, Rosita Bajari, Bilal Baqar, Justin HB Barnowski, Jeffrey Burt, et al. The nci genomic data com- mons. Nature genetics, (3(3):257-266) 2021.
Olivier Henaff. Data-efficient image recognition with contrastive predictive coding. In International conference on machine learning, pp. 41 4182-4192. PMLR, 2020.
Maximilian Ilse, Jakub Tomczak, and Max Welling. Attention-based deep multiple instance learn- ing. In International conference on machine learning. pp. 2127-2136 PMLR, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. pmlr, 2015.
Syed Ashar Javed, Dinkar Juyal, Harshith Padigela, Amaro Taylor-Weiner, Weiner, Limin Yu, and Aaditya Prakash. Additive mil: intrinsically interpretable multiple instance learning for pathology. Ad- vances Neural Information Processing Systems, 35:20689-20702. 2022.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with GPUs. JEEE Transactions Big Data, 7(3):535-541) 2019.
Jakob Nikolas Kather, Niels Halama, and Alexander Marx. 100,000 histological images of human colorectal cancer and healthy tissue. 10.528 1/zenodo. 124456.. 0 5281 12114....oo...145555
April 2018, doi: 1214456

---

Published as a conference paper at ICLR 2024
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishman. Supervised contrastive learning. Advances in neural information processing systems, 33:18611-16673, 2020.
Yoo Jung Kim, Hyungjoon Jang, Kyoungbun Lee, Seongkeun Park, Sung-Gyu Min, Choyeon Hong. Jeong Hwan Park, Kanggeun Lec, Jisoo Kim, Wonjae Hong, et al. Paip 2019; Liver cancer segmentation challenge. Medical Image Analysis, 67:101854, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. University of Toronto, 2009.
Juho Lee. Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: framework for attention-based permutation-invarian neural networks. In Interna- lional Conference on Machine Learning, pp. 3744-3753, PMLR, 2019a.
Kwonjoon Lee, Subhransu Maji, Avinash Ravichandran, and Stefano Soatto. Meta-learning with differentiable convex optimization. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10657-10665, 2019b.
Alexander C Li, Alexei A Efros, and Deepak Pathak. Understanding collapse in non-contrastive siamese representation learning. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXI, pp. 490-505. Springer, 2022.
Bin Li, Yin Li, and Kevin W Eliceiri. Dual-stream multiple instance learning network for whole slide image classification with suff-superviee contrastive learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14318-14328. 2021.
Hongyang Li, David Eigen, Samuel Dodge, Matthew Zeiler, and Xiaogang Wang. Finding task- relevant features for few-shot learning by category traversal. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1-10. 2019.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130, 2017.
John Lonsdale, Jeffrey Thomas, Mike Salvatore, Rebecca Phillips, Edmund Lo, Saboor Shad, Richard Hasz, Gary Walters, Fernando Garcia, Nancy Young, al. The genotype-tissue pression (gtex) project. Nature genetics, 45(6):580-585, 2013.
Ming Y Lu, Tiffany Y Chen, Drew FK Williamson, Melissa Zhao, Maha Shady, Jana Lipkova, and Faisal Mahmood. Ai-based pathology predicts origins for cancers unknown primary. Nature, 594(7861):106-111 10, 2021a.
Ming Y Lu, Drew FK Williamson, Tiffany Y Chen, Richard J Chen, Mattco Barbieri, and Faisal Mahmood. Data-efficient and weakly supervised computational pathology whole-slide images. Nature biomedical engineering, 5(6):555-570) 2021b.
Ishan Misra and Laurens van der Maaten. Self-supervised learning of pretext-invariant representa- tions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6707-6717, 2020.
Andriy Myronenko, Ziyue Xu, Dong Yang. Holger Roth, and Daguang Xu. Accounting for dependencies in deep learning based multiple instance learning for whole slide imaging. In In- ternational Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 329-338 Springer, 2021.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Weang-Wennl Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
12

---

Published as a conference paper at ICLR 2024
Boris Oreshkin, Pau Rodríguez López, and Alexandre Lacoste. Tadam: Task dependent adaptive metric for improved few-shot learning. Advances in neural information processing systems, 31, 2018.
Anabik Pal, Zhiyun Xue, Kanan Desai, Adekunbiola Aina F Banjo, Clement Akinfolarin Adepiti, L Rodney Long, Mark Schiffman, and Sameer Antani. Deep multiple-instumne learning for abnor- mal cell detection in cervical histopathology images. Computers in Biology and Medicine, 138: 104890, 2021.
Cheng Perng Phoo and Bharath Hariharan. Coarsely-labeled data for better few-shot transfer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 9052-9061, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp- 9748-8763. PMLR, 2021.
Colin Raffel and Daniel PW Ellis. Feed-forward networks with attention can solve some long-term memory problems. arXiv preprint arXiv: 2015.
Pierre H Richemond, Jean-Bastien Grill, Florent Altché, Corentin Tallec, Florian Strub, Andrew Brock, Samuel Smith, Soham De, Razvan Pascanu, Bilal Piot, et al. Byol works even without batch statistics. arXiv preprint arXiv:2010.10241 2020.
Joshua Robinson, Stefanie Jegelka, and Suvrit Sra. Strength from weakness: Fast learning using weak supervision. In International Conference on Machine Learning, pp. 8127-8136. PMLR, 2020.
Andrei A Rusu, Dushyant Rao, Jakub Sygnowski, Oriol Vinyals, Razvan Pascanu, Simon Osin- dero, and Raia Hadsell. Meta-learning with latent embedding optimization. arXiv preprint arXiv:/807.05961 2018.
Adam Santoro, David Raposo, David G Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A simple neural network module for relational reasoning. Ad- vances neural information processing systems, 30, 2017.
Fereshteh Shakeri, Malik Boudiaf, Sina Mohammadi, Ivaxi Sheth, Mohammad Havaei, Ismail Ben Ayed, and Samira Ebrahimi Kahou, Fhist: benchmark for few-shot classification of histological images. arXiv preprint arXiv:2206.000 00092, 2022.
Zhuchen Shao, Hao Bian, Yang Chen, Yifeng Wang, Jian Zhang, Xiangyang Ji, al. Transmil: Transformer based correlated multiple instance learning for whole slide image classification. Ad- vances neural information processing systems, 34:2136-2147, 2021.
Yash Sharma, Aman Shrivastava, Lubaina Ehsan, Christopher A Moskaluk, Sana Syed, and Donald Brown. Cluster-to-conquer: A framework for end-to-end multi-instance learning for whole slide image classification. In Medical Imaging with Deep Learning, pp. 682-698. PMLR, 2021.
Baoguang Shi, Song Bai, Zhichao Zhou, and Xiang Bai. Deeppano: Deep panoramic representation for 3-d shape recognition. IEEE Signal Processing Letters, 22(1)))3339 2343, 2015.
Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Ad- vances in neural information processing systems, 30, 2017.
Cancer Genome Atlas Research Network Tissue source sites: Duke University Medical School McLendon Roger Friedman Allan 2 Bigner Darrell 1, Emory University Van Meir Erwin G. 3 4 5 Brat Daniel J. 5 6 M. Mastrogianakis Gena 3 Olson Jeffrey J. J. 4 5, Henry Ford Hospital
Mikkelsen Tom 7 Lehman Norman 8, MD Anderson Cancer Center Aldape Ken Alfred Yung
WK 10 Bogler Oliver 11, University of California San Francisco VandenBerg Scott 12 Berger Mitchel 13 Prados Michael 13, et al. Comprehensive genomic characterization defines human
glioblastoma genes and core pathways. Nature, 455(72166:0061111666 2008
13

---

Published as a conference paper at ICLR 2024
Hang Su, Subhransu Maji, Evangelos Kalogerakis, and Erik Learned-Mille Multi-view convo- lutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pp. 945-953, 2015.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference computer vision, pp. 843-852, 2017.
Flood Sung, Yongxin Yang, Li Zhang, Tao Xiang, Philip HS Torr, and Timothy M Hospedales. Learning compare: Relation network for few-shot learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pp. 1199-1208, 2018.
Yonglong Tian, Yue Wang, Dilip Krishnan, Joshua B Tenenbaum, and Phillip Isola. Rethinking few-shot image classification: a good embedding is all you need? In Computer Vision-Eo ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 266-282. Springer, 2020.
Eleni Triantafillou, Richard Zemel, and Raquel Urtasun. Few-shot learning through an information retrieval lens. Advances neural information processing systems, 30, 2017.
Tim Van Erven, Peter Grunwald, Nishant Mehta, Mark Reid, Robert Williamson, et al. Fast rates in statistical and online learning. MIT Press, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Oriol Vinyals, Samy Bengio, and Manjunath Kudlur. Order matters: Sequence to sequence for sets. arXiv preprint arXiv: 1ttl:::511..500. 116391111.11.1111111111111111111 2015.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
Yu-Xiong Wang and Martial Hebert. Learning to learn: Model regression networks for easy small sample learning. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VI 14, pp. 616-634, Springer, 2016.
Yu-Xiong Wang, Ross Girshick, Martial Hebert, and Bharath Hariharan. Low-shot learning from imaginary data. In Proceedings of the IEEE conference on computer vision and pattern recogni- tion, pp. 7278-7286, 2018.
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018.
Jinxi Xiang and Jun Zhang. Exploring low-rank property in multiple instance learning for whole slide image classification. In The Eleventh International Conference on Learning Representations, 2022.
Zhixin Xu, Seohoon Lim, Hong-Kyu Shin, Kwang-Hyun Uhm, Yucheng Lu, Seung- Jung, and Sung-Jea Ko. Risk-aware survival time prediction from whole slide pathological images. Scientific Reports, 12(1):21948, 2022.
Jiawei Yang, Hanbo Chen, Jiangpeng Yan, Xiaoyu Chen, and Jianhua Yao. Towards better under- standing and better generalization of few-shot classification in histology images with contrastive learning. arXiv preprint
2022.
Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv:2101.066 06395, 2021.
Jiawen Yao, Xinliang Zhu, Jitendra Jonnagaddala, Nicholas Hawkins, and Junzhou Huang. Whole slide images based cancer survival prediction using attention guided deep multiple instance learn- ing networks. Medical Image Analysis, 65:101789 2020.
14

---

Published as a conference paper at ICLR 2024
Han-Jia Ye, Hexiang Hu, De-Chuan Zhan, and Fei Sha. Learning embedding adaptation for few-shot learning. arXiv preprint 1rX1v:1812.03664, 2018.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708. 1308000888, 2017.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part / 13, pp. 818-833. Springer, 2014.
Jieyu Zhang, Cheng-Yu Yu Hsieh, Yue Yu, Chao Zhang, and Alexander Ratner. A survey program- matic weak supervision. arXiv preprint arXiv:2202.05433. 2022.
Renyu Zhang, Aly A Khan, and Robert L Grossman. Evaluation of hyperbolic attention in histopathology images. In 2020 IEEE 20th International Conference on Bioinformatics and Bio- engineering (BIBE), pp. 773-776. IEFE, 2020a.
Renyu Zhang, Christopher Weber, Robert Grossman, and Aly A Khan. Evaluating and interpreting caption prediction for histopathology images. In Machine Learning for Healthcare Conference, pp. 418 435. PMLR, 2020b.
Zhi-Hua Zhou. brief introduction weakly supervised learning. National science review, 5(1): 44-53, 2018.
15

---

Published as a conference paper at ICLR 2024
A TRAINING DETAILS
A.1 PRETRAIN WITH UNIQUE CLASS NUMBER AND MOST FREQUENT CLASS OF INPUT SETS
In our study, an epoch refers to going through all the input sets in the dataset once. SimSiam is trained for 2,000 epochs using a batch size of 512. SGD is employed with a learning rate of 0.1, weight decay of le-4, and momentum of 0.9. The training process incorporates a cosine scheduler. Similarly, SimCLR is trained for 2,000 epochs with batch size of 256 and temperature of 0.07. SGD is used with learning rate of 0.05, weight decay of le-4, and momentum of 0.9. The training also utilizes cosine scheduler.
We train FSP-Patch for 800 epochs with a batch size of 256. The SGD is used with a weight decay of le-4, momentum of 0.9, and cosine scheduler.
FACILE-FSP is trained for 800 epochs with a batch size of 64, SGD is used with a learning rate of 0.0125, weight decay of lc-4, and momentum of 0.9. l loss is optimized for pretraining with unique class numbers of input sets. For FACILE-SupCon, we train the model with 2,000 epochs and batch size of 256. An additional temperature parameter is set to 0.07. The SGD is used with learning rate of 0.05, weight decay of le-4, and momentum of 0.9.
A.2 FINE-TUNE VIT-B/16 OF CLIP WITH CUB200
SimSiam is trained for 400 epochs using a batch size of 64. SGD is used with an initial learning rate of 0.0125, weight decay of le-4, and momentum of 0.9. The cosine scheduler is used for the optimizer. SimCLR is also trained for 400 epochs with a batch size of 64. An additional temperature parameter set to 0.07. SGD is used with learning rate of 0.0125, weight decay of le-4, and momentum of 0.9. The training also uses a cosine scheduler.
FACILE-FSP is trained for 200 epochs with a batch size 64. SGD is used with a learning rate of 0.0125, weight decay of 1e-4, and momentum of 0.9. For FACILE-SupCon, we train the model with 800 epochs and batch size of 64. An additional temperature parameter is set to 0.07. The SGD is used with initial learning rate of 0.0125, weight decay of le-4, and momentum of 0.9. Both models' training utilized a cosine annealing scheduler.
A.3 PRETRAIN RESNET18 18 WITH TCGA AND GTEX DATASET
In SimSiam, SimCLR, and FSP-Patch models, the data loader samples one patch for each slide. In FACILE-FSP and FACILE-SupCOn the data loader samples set of a patches for each slide.
SimSiam is trained for 5,000 epochs using batch size of 55. SGD is employed with a learning rate of 0.01, weight decay of le-4, and momentum of 0.9. The training process incorporates cosine scheduler. Similarly, SimCLR is trained for 5,000 epochs with a batch size of 32. An additional temperature parameter is set to 0.07. SGD is used with a learning rate of 0.006, weight decay of 1e-4, and momentum of 0.9. The training also utilizes cosine scheduler.
FSP-Patch is trained for 1,000 epochs with a batch size of 64. We employ SGD with learning rate of 0.05, weight decay of le-4, and momentum of 0.9. The training process includes the utilization of cosine scheduler.
FACILE-FSP is trained for 3,000 epochs with batch size 32. The input set size is 5 by default. We employ SGD with learning rate 0.0125, weight decay of 1e-4, and momentum of 0.9. The training process includes the utilization of cosine scheduler. Set Transformer with 3 inducing points and 4 attention heads is used for the set-input model g. Similarly, for our FACILE-SupCon model, we use the same input set size and set-input model. The training process is configured with a batch size of 32 and extends over 3,000 epochs. An additional temperature parameter is set to 0.07. We use SGD with a learning rate of 0.00625, weight decay of 1e,4, and momentum of 0.9. We use an MLP as projection head with two fo layers, a hidden dimension of 512, and an output dimension of 512.
16

---

Published as a conference paper at ICLR 2024
A.4 FINE-TUNE VIT-B/14 OF DINO V2 WITH TCGA
SimSiam is trained for 400 epochs with a batch size of 64, utilizing Stochastic Gradient Descent (SGD) with an initial learning rate of 0.0125, weight decay of le-4, and a momentum of 0.9. A cosine scheduler was employed. SimCLR underwent similar training regimen for 400 epochs and batch size of 64, with an additional temperature parameter set at 0.07 and identical SGD parameters, including the use cosine scheduler for learning rate adjustments.
FSP-Patch also completed 400 epochs of training with a batch size of 64. The model employed SGD with a learning rate of 0.0125, a weight decay of Ic-4, and a momentum of 0.9, along with a cosine scheduler to modulate the learning rate.
For FACILE-FSP, training spanned 200 epochs with a batch size of 64, using SGD with the same learning rate, weight decay, and momentum settings. FACILE-SupCon extended its training 800 epochs with a batch size of 64, including an additional temperature setting of 0.07 and the same SGD configuration. Both FACILE-FSP and FACILE-SupCon models utilized cosine annealing scheduler.
B ADDITIONAL RESULT
B. FINE-TUNE CLIP MODEL WITH ANOMALY DETECTION DATASET
In this experiment, we sought to enhance model performance with coarse-grained labels of the anomaly detection datasets (aaeeer al. 2017 Lee al.. 2019a). A total of 11.788 input sets of size 10 are constructed from the CUB200 (He & Peng 2019) training dataset by including one example that lacks an attribute common the other examples in the input set. The coarse-graines labels are the positions of the anomalies. This setup creates challenging scenario for models, as they must identify the outlier among otherwise similar instances. downstream tasks, we evaluate the fine- tuned feature encoder composed of the fixed CLIP (Radford et al. (021) image encoder ViT- B/16 and appended fully colle-connected layer on the classification of species of the CUB200 test dataset. The batch normalization (Loffe & Szegedy] 2015) and ReLU are applied to the fully- cullycconeected layer.
Following this experiment setup, the rationale behind utilizing gearse-grained labels is grounded in their potential to enhance model discernment in downstream tasks. By training the model to identify anomalies in sets where one item diverges from the rest, we essentially teach it to focus on subtle differences and critical attribute features. This enhanced focus is particularly beneficial for fine-grained classification tasks in the CUB200 test dataset, where distinguishing between closely related species requires the model to recogmize and prioritize minute, yet significant, differences.
The model training approach in this experiment centered around the CLIP image encoder, en- hanced with an additional fully-connected layer. FACILE-FSP and FACILE-SupCon incorporate this setup, utilizing the CLIP-based feature encoder and focusing on finetuning the fully-connected layer through the FACILE pretraining step. In contrast, the SimSiam approach leverages the CLIP image encoder as a backbone while finetuning the projector and predictor components. Similarly, the SimCLR method also uses the CLIP encoder as its foundation but focuses solely on finetuning the projector: These varied strategies reflect our efforts to optimize the feature encoder for accu- rately identifying anomalics and improving classification performance in related tasks. The training details can be found in section A.2
pretraining method
NC
LR
RC
CLIP (ViT-B/16)
83.84 + 1.10
81.01 # 1.23
82.75 + 1.17
SimCLR
84.03 + 1.08
83.49 1.14
86.30 + 1.03
SimSiam
84.02 1.10
4.10
85.68 + 1.07
FACILE-SupCAS
87.49 + 0.99
86.57 1.07
88.01 + 0.99
FACILE-FSP
88.74 + 0.94
88.45 + 0.96
88.36 + 0.95
Table 4: Pretraining on input sets from CUB200. Testing with 5-shot 20-way meta-test sets; average Fl and CI are reported.
Note that table 4 clearly demonstrates that all models tested benefit from incorporating data from the target domain. Notably, both FACILF-SSpCCS and FACILE-RSP exhibit superior performance
17

---

Published as a conference paper at ICLR 2024
compared to other baseline models. This observation underscores the effectiveness of our models in leveraging coarse- maassegraineddd labels to enhance their anomaly detection capabilities.
B.2 PRETRAIN RESNETIS WITH TCGA
B.2.1 ACC ON LC, PAIP, AND NCT DATASETS
We pretrain the models TCGA datasets with patches size 224 224 20X magnification. Then, these pretrained models are tested on LC, PAIP, and NCT datasets. The average ACC and CI on the
LC, PAIP, and NCT datasets are shown in 5
pretraining method
NC
LR
RC
LR+LA
RC+LA
shot 5-way test on LC dataset
ImageNet (FSP)
65.64 + 0.49
66.06 = 0.46
65.92 + 0.48
66.60 + 0.48
67.09 0.47
SimSiam
68.88 + 0.51
68.53 = 0.48
68.27 + 0.48
68.81 0.49
70.24 + 0.47
SimCI.
66.41 + 0.48
66.52 + 0.46
66.10 + 0.46
67.70
68.71 + 0.46
FSP-Patch
68.56 + 0.46
68.51 + 0.45
68.68 + 0.46
69.38 0.46
09.63 + 0.46
FACILE-SupCA ESSpCon
76.64 + 0.50
77.88 = 0.47
76.77
0.47
77.16 = 0.48
FACILE-FSP
79.01 = 0.49
78.16 + 0.48
77.43
0.50
79.15 = 0.47
75.81 0.48
5-shor 5-way test
LC dataset
ImageNet (FSP)
82.79 0.32
81.31 + 0.31
81.13 0.30
84.00
84.73 0.28
SimSiam
85.12 0.30
83.39 0.32
83.85
87.74
0.27
87.90 0.26
SimCLR
83.75 0.30
82.38 0.30
0.31 82.32
86.12
85.40 0.30
FSP-Patch
85.15 +:
84.38 0.31
85.01
86.71
86.24
FACILE-SupCon
91.16 0.24
90.48 0.24
90.40
0.24
91.39 0.24
91.03
0.22
FACILE-FSP
91.77 + 0.21
90.85 0.23
90.77
0.24
92.19 0.22
90.02 0.24
pretraining method
NC
LR+LA
RC+LA
T-shot 3-way test on PAIP dataset
ImagNNet (FSP)
48.44 0.65
50.34 = 0.65
50.21 + 0.62
48.90 + 0.62
47.51 0.59
SimSiam
49.42 :
50.25 = 0.65
49.76 + 0.65
49.51 + 0.62
49.09 0.63
SimCLR
47.39 + 0.59
48.35 0.59
47.97 + 0.58
47.77 + 0.59
17.63 - 0.60
FSP-Patch
51.61 + 0.68
51.61 1 0.67
52.06 - 0.67
51.74 0.66
51.38 1 0.66
EACILE-SupCon
49.65 + 0.61
51.32 0.66
51.16 + 0.63
50.00 +: 0.62
50.81 0.65
FACILE-FSP
48.91 + 0.61
49.57 = 0.63
49.68 + 0.63
49.42 + 0.65
48.60 + 0.64
S-shot 3-way test on PAIP dataset
ImageNet (FSP)
62.46 0.52
62,48 = 0.48
63.14 = 0.50
62.11 + 0.51 0.51
60.52 = 0.49
SimSiam
63.05 + 0.52
64.41 = 0.49
64.66 + 0.50
65.44 + 0.53
64.64 + 0.55
SimCLR
61.48 + 0.52
61.81 + 0.53
62.75 + 0.51
63.03 + 0.52
61.70 0.52
FSP-Patch
65.29 + 0.49
65.81 + 0.51
65.98 + 0.48
65.70 + 0.50
64.01 0.52
FACILE-SupCon
65.44 = 0.51
66.75 + 0.52
67.11 0.51
67.24 + 0.53
67.06 + 0.52
FACILE-FSP
64.68 + 0.53
65.75 = 0.49
66.58 + 0.51
67 42.42-25 0.53
67.06 0.53
pretraining method
NC
LR
RC
LR+LA
RC+LA
I-shot 9-way
NCT dataset
ImageNet (FSP)
58.75 + 0.35
58.66 0.36
58.48 + 0.34
58.83 0.36
57.32
SimSiam
64.76 0.40
66.09 0.39
66.09 0.39
66.54 0.40
67.05 0.41
SimCLR
60.47
61.17 + 0.38
61.43 0.39
61.65
FSP-Patch
61.03 1 0.42
63.53 0.40
63.26
62.75:00.45
61.57
FACILE-SUpCOn
68.99 + 0.45
70.76 0.40
70.89 0.41
70.45
0.45
70.63
FACILE-FSP
67.43 1 0.44
68.45 0.4
68.97 1 0.42
69.53 1: 0.43
0.42 70.89
5-shot 9-way test on NCT dataset
ImageNet (FSP)
74,82 1 0..66
74.35 + 0.26
75.20 + 0.26
77.1140.23
74.89 = 0.26
SimSiam
80.59 +: 0.23
80.51 - 0.23
81.54 + 0.21
83.68 1 0.22
83.85 0.22
SimCLR
77.30 1: 0.25
77.64 0.24
79.17 + 0.24
80.99 0.24
81.71
0.23
FSP-Patch
79.61 1: 0.25
79.89 0.24
81.71 + 0.23
82.92 1 0.24
81.67 1 0.24
FACILE-SupCon
86.89 + 0.22
88.06 - 0.20
89.26
- 0.19
89 62 0.19
88.67 67 0.21
FACILE-FSP
84.83 + 0.24
85.78 - 0.23
87.68 + 0.20
88. 18.16 0.20
87.67 0.20
Table 5: Models tested on LC, PAIP, and NCT dataset; average ACC and CI are reported.
B.2.2 TEST WITH LARGE SHOT NUMBER
We further test the trained models with larger shot number k. The result is shown in table 6
18

---

Published as a conference paper at ICLR 2024
pretraining method
NC
LR
RC
LR+LA
RC+LA
10-shot 5-way
LC
ImageNet (FSP)
78.76 0.94
78.92 0.92
S0.45 + 0.87
82.25
80.20
SimSiam
88.52 + 0.55
87.20
87.73 1 0.56
91.62 0.46
91.88 0.47
SimCLR
87.02 0.64
86.26
85.61 1 0.72
90.28
89.60 0.58
FSP-Patch
88,41 0.53
88.64 0.52
89.15 1 0.51
90.49 - 0.50
89.88 0.54
FACILE-SuPCor
92.84 + 0.39
0.38 92.87 = 0.38
93.21
- 0.37
94.25 - 0.36
93.72 0.39
FACILE-FSP
93.10 = 0.39
93.11 + 0.38
93.63
0.37
94.52 0.35
93.07 0.45
10-shot 3-way
on PAIP
ImageNet (FSP)
65.36 0.91
65.17 1.00
63.40 + 0.99
66.32 0.81
0.84
SimSiam
67.19 1: 0.88
0.98
68.55 + 0.94
70.88 0.77
70.62 0.77
SimCLR
65.77 +: 0.85
67.01 + 0.91
68.41 0.79
66.96 - 0.82
FSP-Patch
68,50 + 0.82
69.12 0.85
69.39 + 0.85
70.13 0.75
68.25 0.76
FACILE-SupCon
70.03 = 0.81
71.24
72.17 + 0.83
73.31 = 0.71
72.50 0.71
FACILE-FSP
69.19 # 0.82
0.82
71.78 + 0.81
73.22 + 0.73
72.78 + 0.71
10-shot 9-way
NCT 80.45 + 0.87
ImageNet (FSP)
78.76 + 0.94
78.92 0.92
82.25 0.883
80.20 0
SimSiam
82.92 +: 0.91
83.42 0.89
84.76 + 0.81
87.66 0.72
SimCLR
80.34 0.96
1L67
83.09 + 0.84
85.96 + 0.76
86,82 = 0.72
FSP-Patch
83.36 + 0.77
84.05
85.93 0.65
86.05 = 0.63
Suppon
89.57 = 0.49
01.11
92.20 = 0.41
92.88 0.39
0.41 92.02 + 0.41
FACILE-FSP
87.54 + 0.61
89.25 + 0.56
90.77 + 0.49
91.63 + 0.48
91.23 0.50
Table 6: Test result on LC, PAIP, and NCT dataset with shot number 10; average F1 and CI are reported.
B.3 FINE-TUNE VIT-B/14 OF DINO V2 ON TCGA DATASET
Similar to section B. fine-tune fully-connected layer that is appended after DINO V2 Oquab jet al. (2023) ViT-B/14. This methodology is applied across various models to assess their perfor- mance histopathology image datasets. By adopting the DINO V2 architecture, known for its robustness and effectiveness in visual representation learning, we aim to harness its potential for the specialized domain of histopathology. We refer interested readers section A.4 for details of pretraining.
Notably, methods, FACILE-SupCOS and FACILE-FSH demonstrated markedly superior results in comparison to other baseline models when applied to histopathology image datasets as shown in table This outcome highlights the effectiveness of these methods leveraging coarse- canse--rrained labels specific to histopathology, thereby greatly enhancing the model performance of downstream tasks. Another critical insight emerged from our research: the current foundation model, DINO V2, histopathology images. This suggests that
exhibits limitations in its generalization performance while DINO provides a strong starting point due to its robust visual representation capabilities, there is clear need for further finetuning prompt learning to optimize its performance for the unique challenges presented by histopathology datasets. This finding underscores the importance of specialized adaptation in the application of foundation models specific domains like medical imaging.
B.4 BENEFITS OF PRETRAINING LARGE PATHOLOGY DATASETS
In order to demonstrate the advantages of pretraining on large pathology datasets, compare the performance of models pretrained on TCGA datasets with those pretrained on NCT datast, which also studied in Yang et al. (2022).
The SimSiam model is trained for 100 epochs. SGD optimizer is used with learning rate of 0.01, weight decay of 0.0001, momentum of 0.9, and cosine learning rate decay. The batch size is 55.
For MoCo v3, similar to (Chen et al. 202 Ic Yang et al. 2002), LARS optimizer (You et al. 2017) was used with initial learning rate of 0.3. weight decay of 1.5e 6, the momentum of 0.9, and cosine decay schedule. MoCo v3 was trained with a batch size of 256 for 200 epochs.
The FSP model with simple augmentation follows the setting of Yang al. (2022). SGD optimizer with learning rate of 0.5, momentum of 0.9 and weight decay of are used. large batch size used 512. The model is trained for 100 epochs with "step decay" schedule. The learning rate
19

---

Published as a conference paper at ICLR 2024
pretraining method
NC
LR
RC
IR+LA
RC+LA
-hot 5-way test
LC dataset
DINO V2(1--B1)
44.82 112211111 1.41
47.51 1 1.39
47.6
1. 1.38
47 1 1.39
48.88 1 1.44
SimSiam
48.79
1.37
49.43 + 1.35
48.43 + 1.36
49.38
+ 1.34
49.50
+ 1.34
SimCLR
50.47
1.31
50.52
1.33
50.44
1.32
51.66
1.32
51.78
1.38
FSP-Patch
49.73
1.41
53.59
53.07
51.79
1.40
51.27
1.43
MACILE-SupCor
56.24
1.43
56.51
1.41
55.95
1.42
56.29
1.43
54.07
1.44
FACTLF--SS
1.40 55.67
56.26
1.36
55.83
1.35
56.01
1.38
55.35
1.40
5-shot 5-way test
LC dataset
DINO V2 (VIT-B/14)
64.71 = 1.12
66.30 = 1.10
72.95 = 0.93
75.11 0.91
SimSiam
67.51 +
64.99 + 1.05
65.39 + 1.05
70.30 + 0.93
71.19 + 0.93
SimCLR
0.92
69.28 0.96
69.18 + 0.97
72.99
0.92
72.91
0.94
FSP-Patch
0.96
71.11 1.04
71.19 1.03
73.96
0.94
73.20 0.96
FACILE-SupCon
75.58 0.88
74.26
73.20
75.81 0.90
74.34 0.96
FACTLF-FSP
75.86
0.86
74.64
0.89
74.12 + 0.93
76.17 + 0.88
75.08
0.95
T-shol 3-way Test
PXTP dataset
DINO V2 (VIT-B/14)
41.51 1.27
44.37 1.26
44,28 + 1.25
42.43 = 1.27
42.78 = 1.27
SimSiam
49.42 1.28
48.07 + 1.35
48.44 + 1.36
48.76 + 1.33
46.48 + 1.37
SimCLR
48.60 + 1.19
48.76 +: 1.25
47.98 +: 1.26
48.94 +: 1.23
47.20
1.26
FSP-Patch
46.09 1.17
47.44 1.18
48.09
1.19
46.76
1.18
43.68 = 1.22
FACILE-SupCon
51.97
1.18
52.25
1.22
51.80
1.22
51.36
1.22
50.24
1.23
FACILE-FSP
51.31 1.16
51.18
1.19
51.51
1.19
1.16 51.50
49.77 1.22
Seshol 3-way Test
PAIP dataset
DINO V2 (VIT-B/44
37.59 1.07
$8.19 1,10
59.37 = 1.07
61.84 = 0.85
60.81 = 0.80
SimSiam
0.97
62.52 1.01
62.81 + 1.01
64.40 + 0.86
62.44 + 0.93
SimCLR
62.20
61.78 0,99
63.20 0.97
63.38 = 0.86
63.03 = 0.88
FSP-Patch
63.77 0.88
63.85 0.94
63.85 0.93
63.61 0.85
60.91 = 0.87
FACTLE-SupCOn
67.16 0.84
67.29
66.88
67.61 0.85
66.34 0.84
FACILE-FSP
67.14 0.85
67.67 0.84
67.54 0.86
67 0.81
66.05 0.83
T-shot 9-way Test
NCT dataset
DINO (VIT-B/1))
36.03 + T.62
39.11 = 1.57
60.13 = 1.55
58.71 = 1.57
59.06 = 1.55
SimSiam
12.60+1..4
61.89 1.50
61.90 + 1.51
62.27 + 1.47
61.05 1 1.44
SimCLR
65.43 1.43
64.18 = 1.44
61.15 1.46
64.83 = 1.43
62.69 = 1.38
FSP-Patch
65,22 1.49
65.93 + 1.41
65.94
1.40
65.26
1.45
62.66
- 1.46
FACILE-SupCon
71.55 1.36
70.36
1.37
70.52 1.35
71.05
1.35
68.85
1 1.40
FACILE-FSP
72.05
1.34
70.70
1.35
70.77
1.34
71.14
1.34
68 1.40
5-shol 9-way Test
NCT dataset
DINO V2 (ViT-B/14)
0.998
76.51
1.02
78.67
0.94
82.20
0.82
82.75
0.83
SimSiam
0.88
80,06 + 0.87
81.55 + 0.85
83. 1 0.80
82.39 1 0.83
SimCLR
= 0.80
81.91 + 0.82
82.86 0.80
83.92 = 0.77
82.89 = 0.79
FSP-Patch
83.63 0.83
83.49 1 0.80
84.34 1 0.78
85.32 0.75
83.03 0.79
FACILE-SupCon
87.74 0.64
87.00 0.64
87
0.62
87 0.63
0.69
FACILE-FSP
87.93 0.65
87.52
0.65
87 0.62
88.01 0.64
86. 46 0.70
Table 7: Test result on LC, PAIP, and NCT dataset with ViT-B/14 from DINO V2; average F1 and CI are reported.
multiplied by 0.1 at 30, 60, and 90 epochs respectively. The FSP model with strong augmentation was trained for 50 epochs. The batch size is set to 64. The SGD is used with learning rate of 0.03, momentum of 0.9, weight decay of 0.0001, and the cosine schedule. The model is trained for 50 epochs.
The SupCon model is trained with trained for 100 epochs. The batch size is set to 64. The SGD optimizer is used with a learning rate of 0.01, momentum of 0.9, weight decay of 0.0001, and the cosine schedule.
table g shows the performance of the pretrained models on the LC and PAIP dataset with shot numbers or 5. Notably, the best best-performing models on the two test datasets exhibit a significant performance gap compared the best models pretrained on TCGA datasets as depicted in table
B.5 PRETRAIN ON TCGA AND GTEX WITH PATCH SIZE 1,000X1,000
We train the models using TCGA patches of size 1, 000 x 1, 000, which are extracted from 20X magnification and resized to 224 x 224. Subsequently, the pretrained models are evaluated on PDAC datasets, and the corresponding test performance is presented in Figure Notably, for shot number of and 5, our model significantly outperforms other models, demonstrating a substantial performance margin.
20

---

Published as a conference paper at ICLR 2024
pretraining method
NC
LR
RC
LR+LA
RC+LA
T-shor 5-way test
on LC dataset
SimSiuu
59.30 1.31
58.07
1.41
38.58
1.40
59.00 : 1.35
59.85 1:35
MoCo Yong al. 2022
59.38 1.62
59.39
1.68
39.46
1.68
60.15 1.59
60.54
+ 1.58
FSP (simple aug. Yang al. 2022)
51.42 1.59
46.06
1.88
46.33
+: 1.86
50.53
+: 1.65
51.00
= 1.65
FSP (strong aug) SupCon
68.00 = 1.29
66.17
1.41
66.18
1.46
68.39
1.34
68 02 1.40
64.48 + 1.33
63.52
1.42
63.84 + 1.40
65.43
+: 1.33
65.98
1.38
S-shor
5-way test
on L.C datasel
SimSium
76:21 0.87
74.05
1.10
74.59
1.10
77.87 +: 0.87
76.03 + 0.94
MoCo ( Yang al. 2022
1.25
70.29
1.43
71.31
1.40
78.72 1.00
79.71 0.95
FSP (simple aug; Yang el ar. 2022
56.44 + 1.50
52.27
1.81
55.62
1.74
63.47 + 1.37
63.47 1.46
FSP (strong sug)
83.53 = 0.79
80.81
1.01
80.27
1.08 78.65
85.37 0.77
84 0.89
SupCon
81.51
78.77
1.03
1.08
83.51 0.84
83.31 0.91
1-shot 3-way test
PAIP dataset
SimSium
37.13 1.14
38 = 1.13
37 3X = 1.15
35.00 + 1.12
38.67 = 1.12
MoCo ( Yang al. 2022
1.26
43.02
1.31
43.55
1.28
44.57
1.28
FSP (simple aug. Yang et 2022
37.15 + 1.07
36.09
1.13
37 39
+: 1.08
37.40
+ 1.07
35.28
= 1.09
FSP (strong aug) SupCon
17.67 + 1.18
18, 14 1.19
48. 16 1.21
18:27 + 1.17
49.38 38 + 1.19
48.45
1.20
49.47
48.53 1.20
5-shot 3-way test
PAIP dalaset
SimSiam
47.52 + 1.00
48.12 1.10
47.04 + 1.11
52.70 + 0.95
54.51 + 1.00
MoCo if Yang et 2022
55.43 :: 1.00
54.23 1.09
54.05 1.09
56.07 0.92
55.73 = 0.93
FSP (simple aug. Yang 2022)
44.98 0.95
45.13 =
45.
0.96
11.34 0.87
41.03 0.88
FSP (strong aug) SupCon
62.00 + 0.88
0.97 62.48
62.01 + 0.98
64.82 - 0.86
64.60 + 0.87
63.62 = 0.91
64.38 0.96
63.61 1.00
64.37 + 0.87
64.28 0.88
Table 8: pretraining on NCT dataset and testing on LC and PAIP dataset; average F1 and CI are reported.
pretraining method
NC
LR
RC
LR+LA
RC+LA
Ishot 5-way test
ImageNet (FSP)
29.37 1 1.07
31.32 1 1.09
31.16 1 1.07
30.88
1.08
30.14
1.08
SimSiam
30.48 + 1.08
30.18 + 1.12
30.19
+ 1.13
30.41
1.08
31.13
1.10
SimCLR
30.79 1: 1.08
30.93
1.13
30.78
1.12
31.33
1.08
31.22
1.07
FSP-Patch
34.04 1.16
33.99
1 1.20
33.69
1.20
34.29
1.15
34.99
1.16
MACILE-SupCon
35.44
1.17
34.94
1.20
34.58
1.16 35.68
35.27
FACILE-FSP
1.16 37.36
36.07
1.23
36.93
1.21
36.79
1.19
36.81
1.18
S-shot 5-way test
ImageNet (FSP)
41,83 0.996
41.30 1: 1.10
41.08 1: 1.08
42.38
0.94
41.29
0.93
SimSiam
40.15
1.03
37.29 + 1.21
37.43 + 1.21
41.87
1.00
42.70
1.01
SimCLR
40.30 1: 1.04
38.74
1.19
39.02
1.16
40.98
0.96
40.90
0.98
FSP-Patch
44.26
1.10
42.99
1.20
43.69
1.12
46.32
0.97
46.69
0.96
FACILE- E-Sppon
45.83
45.07
45.93
47.72
0.95
47.00
FACILE-FSP
48.21
1.04
47.62
1.12
47.94
1.08
48 84 0.95
48.37 37 0.95
Table 9: Models pretrained on TCGA and tested on PDAC dataset; average Fl and CI are reported.
Similarly, we train the models using GTEX patches with dimensions of 1, 000 1, 000. The patches are extracted from 20X magnification and resized to 224 224. The pretrained models are tested on PDAC datasets, revealing similar outcomes, as illustrated in Table TO
pretraining method
NC
LR
RC
LR+LA
RC+LA
1-shot 5-way test
SimStam
34.78 1.18
34.57
1.25
35.30
1.25
35.13 1.19
35.27
SimCLR
33.68
1.14
33.74
+ 1.18
33.69
1.17
34.28 1.14
33.84
1.12
FSP-Patch
31.87 + 1.09
32.90
+ 1.13
32.53
+ 1.11
32.55 = 1.09
32.10 1.07
MACIL.F-SuPCOO
34.36 + 1.06
34.35
+ 1.13
34.39
+ 1.14
34.70 1.07
34.35
1.07
FACILE-FSP
35.62 - 1.10
35.51 + 1.15
35.40
+ 1.13
35.87
1 1.10
36.16 + 1.09
5-shol 5-way test
SimSiam
46.00 1.10
43.26
1.30
44.19 1.26
47.24 1.00
47.85 # 1.00
SimCLR
44.44 1 1.08
43.40 1.19
43.58
1.15
44.60 0.98
44.17 0.96
FSP-Patch
42.09 0.99
10.15 1.15
40.69
1.09
42.71
0.92
42.66
0.90
FACILE-SupCon
44.85 + 1.02
43.65 + 1.15
44.01
1.13
46.37
0.93
45.10 0.92
FACILE-PSP
46.91 1: 0.97
46.32 1 1.07
47.10
1.02
48 01.01140
1 0.90
47.70 0.80
Table 10: Models pretrained on GTEx and tested on PDAC dataset; average F1 and CI are reported.
21

---

Published as a conference paper at ICLR 2024
C DATASETS
C.1 GTEX DATASET
The Genotype-Tissue Expression (GTEX) project is pioneering initiative aimed at constructing an extensive public repository to investigate sesuue-peccficc gene expression and regulation. The GTEx project collected samples from 54 non-diseased tissue sites across nearly 1000 individuals, with an emphasis on molecular assays such as Whole Genome Sequencing (WGS), Whole Exome Sequenc- ing (WES), and RNA-sequencing. Additionally, the GTEx Biohank contains plethora of unutilized samples. The GTEx portal thttps:: //ggexxpoottt org/home /) provides unrestricted access plethora of data, including gene expression levels, quantitative trait loci (QTLs), and histology images, to aid the research community in advancing our understanding of human gene expression and regulation.
We downloaded all the slides from the GTEx portal. The organs from which the slides are extracted are used for coarse-grained labels. We extract all the non-overlapping patches with size 000 x 1, 000 and only keep those with intensity in 0.85| to filter out backgrounds.
The number of slides from each organ for GTEx can be found figure Thumbnails of WSI examples from the GTEx dataset can be found in figure 7
1000
2110 number
600
400
uns
se
S sommipaaenngss saneers soun
17
exos
Figure 6: Slide number for each organ in GTEx
C.2 TCGA DATASET
The Cancer Genome Atlas (TCGA: https://www cwncer..cc....o gww www.acer.goo/ccg/reeeeeeeeeeeeee http:////rreeec/////////// is project that aims to comprehensively characterize genetic mutations responsible for cancer using genome sequeneing and bioinformatics. The TCGA dataset consists of 10,825 patient samples, including gene expression, DNA methylation, copy number variation, and mutation data, histopathology data, among others (source sites: Duke University Medical School McLendon Roger I Friedman Allan 2 Bigner Darrell et al.) [0088] 13 et al.) 2012). This large-scale dataset has enabled researchers to identify numerous genomic alterations associated with cancer and has contributed to the development of new diagnostic and therapeutic approaches.
genome sequencing/tccge
We downloaded all the diagnostic slides from GDC portal fittps://porte gdc cancer. gov/ The project names of the slides are used for coarse-graing labels. We extract patches at two different scales, i.e., 224 x 224 and 1, 000 1, 000 at 20X magnification, from all the slides.
The number of slides from each project for TCGA can be found figure g Thumbnails of WSI examples from TCGA dataset can be found in figure
22

---

Published as a conference paper at ICLR 2024
Colon
Adrenal
EsophaguaMuccsa
Adiposal Issuu lssuee
Ectepervix
Asta/Appendage
FallopianTube
SuprepubleSkin
Overy
SgnektDooon
TibiaWWtory
SallvaryGians
Endocervix
UnnaryBladser
Lung
Vagina
BrainCerebellum
BrainCortex
ThyroiGGland
EsophagusMusc.kris
Castroesshhagellunniionn
PhutaryGland
Spleen
Stomach
KidneyCortex
Sknn
KidneyMedulla
CoronanyArtery
Figure 7: Randomly deleted examples from GTEx dataset
23

---

Published as a conference paper at ICLR 2024
1000
510e number
600
400
200
TOGA-DLBC
TCGAUCEC
TEGA-SARC
TEGA-CHOL
TCGA- LNC
TEGA POro
TCGA CESC
TEGA UCS
TEGA-MII mee--e
TCDA-CH XXCH
TCGA rot
TEGA coo--o
TCGA LGG
TCBA 104--05
TCGE-ABS
TCGA TCG4
TCGA
TEGA-E0R COA-
TEG
TCGA-AAA
TCGA-E30
TZGATTICA
TEGA-570
TOOA-BL..
TCGA-4G2
TCG4-GV
TOGALUSO
TGGA-Ua LSAL
TCDA
TCGA-VV
Figure 8: Slide number for each tumor in TCGA
C.3 PDAC DATASET
To address the presence of multiple tissues within certain patches, employ a labeling strategy that involves identifying and labeling the centered tissues within these patches. To ensure annotation accuracy, each patch undergoes labeling by minimum of two pathologists, thereby maintaining the
quality of the annotations. For the specific patch numbers corresponding to each tissue in the PDAC dataset, please refer to Figure 10 Furthermore, examples of patches from the PDAC dataset are
provided in Figure 11, offering visual illustrations of the dataset.
Coarse-Grained Dataset
Data type and annotation
WSI number
Extracted patch number
GTEx
slides; organs
25,501
9,465,689
10,321,273
TCGA
slides; tumors
11,638
(11,588,226 w/ size 224)
Fine-Grained Dataset
Data type and annotation
WSI number
Extracted patch number
PDAC
patches; tissues
194
12,250
LC25000
patches; tissues
1,250
25,000
PAIP19
patches; tissues
60
75,000
NCT-CRC-HEE-ONE
patches; tissues
86
100,000
Table 11: Dataset statistics
In order validate model real-world dataset, generated WSIs of Panereatic Ductal Adenocarcinoma (PDAC PDAC, a particularly aggressive and lethal form of cancer originating in the pancreatic duct cells, presents various subtypes, each with distinct morphological characteris- tics. These variations underscore the need for advanced automated tools to accurately characterize and differentiate between these subtypes, thereby aiding disease studies and potentially informing treatment strategies. Examples of PDAC and class distribution are detailed in section C There are in total 12,250 annotated patches extracted from 194 slides. The patch size used for this analysis is 1, 000 x 1, 000 at a 20X magnification. Each patch was annotated into one of classes (i.e., Stroma, Normal Acini, Normal Duct, Tumor, and Islet) and confirmed by at least two pathologists.
C.4 NCT, PAIP, AND LC
We test models on 4 datasets with fine-grained labels. These datasets are from diverse body sites. Statistics of these datasets can be found in table I1
NCT-CRC-HE-1008 (NCT) is collected from colon Kather et al.) 2018). It consists of classes with 100K overllellgp patches. The patch size is 224 x 224. LC25000 (LC) is collected from lung and colon sites (Borkowski et al. 2021). It has 5 classes and each class has 5,000 patches. The patch size is 768 x 768. We resize the patches to 224 x 224. PAIP19 (PAIP) is collected from liver site (Kim et al. 2021. There in total 50 WSIs. The WSIs cropped into patches with size
We will make data publicly available upon acceptance of our paper
24

---

Published as a conference paper at ICLR 2024
TOGA-CHOL
TCBA-MESO
TCGAAACC
TOGA-KICH
TOGA-THMM
TOGA-TGCT
TEAD
TCBA-PCRG
TCGA-PAAD
TCGA-FBGA
TOGA-SARC
TOGA-KIRP
TCBA-CESC
TOGA-UHC
TOGA-BLCA
TOGA-STAD
TOGA-COAD
TOG4-SKOM
TOGA-LGG
TOBA-LUAD
TCGA-NNBC
TCGA-KIRC
TOBE-
TCBA-OV
TCBA-GBM
BRGA BRCA
Figure 9: Randomly selected examples from TCGA dataset
25

---

Published as a conference paper at ICLR 2024
3000
2000
1000
0
Stroma
NarmalAcini
NommaDuck
Tumar
Intet
Figure 10: Patch number for each tissue for PDAC
224 224. only keep those patches with masks and assign labels with majority voting similar to Yang et al. (2022) We downsample these patches to 75K patches, with 25K in each class.
D DATA AUGMENTATION
Two data augmentation strategies are used in this paper.
Simple augmentation Following Yang et (022), we also used simple augmentation policy which includes random resized cropping and horizontal flipping. In our paper, this simple augmen- tation policy only used for FSP-Patch model pretraining on the NCT dataset.
Strong augmentation Following previous work (Grill al. 2020; Chen et al. 2021c Yang al., 2022), for SimCLR and SupCon models, we used similar strong data augmentation which contains random resized cropping, horizontal flipping, horizontal flipping, color jittering (Wu al. 2018) with (brightness=0.8, contrast=0.8, saturation=0.8. hue=0.2, probability=0.8) grayscale conversion (Wu et al. 2018) with (probabilit=-0.2), Gaussian blurring (Chen al., (020) with (kernel size=5, 1, max=2.0, probability=0.5), and polarization (Grill et al. 2020 with (threshold=128, prob- ability=0.2).
In implementing the SimSiam model, we adopted a comparable augmentation strategy, utilizing bust data augmentation techniques. Specifically, we fine-tuned parameters for color jittering, setting brightness, contrast, and saturation adjustments to 0.4, and hue to 0.1. These modifications were applied with a probability of 0.8, as informed by Chen He (202T).
E LATENT AUGMENTATION
Latent augmentation (LA) was originally proposed in Yang et al. (2022) to improve the performance of the few-shot learning system in a simple unsupervised way. The pretrained feature extractor can only transfer parts of available knowledge in the pretraining datasets by the learned weights of the feature extractor. More transferable knowledge is inherent in the pretraining data representations.
In order to fully exploit the pretraining data, possible semantic shifts of clustered representations of the pretraining dataset are transferred to downstream tasks besides the pretrained feature extractor weights. The k-means clustering method is performed on the representations of pretraining datasets, which are generated by the pretrained feature extractor è. Assume we obtain clusters after clus- tering. The base dictionary B {(g, E is constructed, where ;; is the i-th cluster prototype,
26

---

Published as a conference paper at ICLR 2024
Islet
Normal acini
Normal duct
Stroma
Tumor
Figure 11: Randomly selected examples from each class of PDAC dataset.
27

---

Published as a conference paper at ICLR 2024
Le., representation of all samples in the cluster and Σ, is the covariance matrix of the cluster. During downstream task testing, LA uses the original representation 2 to select the closest prototype from B. We get additive augmentation غ the index closest prototype of z. The classifier of the downstream tasks then trained on both the original representations and the augmented representations.
6, where 5 is sampled from A/(0, Σ,-) and i"
F ABLATION STUDY
F.1 SET-INPUT MODELS
Pooling architectures have been used in various set-input problems, e.g, 3D shape recognition (Shi et al. 2015: Su et al., 2013), learning the statistics of a set (Edwards & Storkey 2016). Vinyals al. 2015); llse al. (2018) pool elements in set by weighted average with weights computed by the attention module. (Zaheer ct al.] 2017: Edwards & Storkey 2016) proposed to aggregate embeddings of instances, extracted using neural network, with pooling operations (e.g., mean, sum, max). This simple method satisfies the permutation invariant property and can work with any set size. Santoro al. (2017) used relational network to model all pairwise interactions of elements in given set. Lee et al. (2019a) proposed to use the Transformer (Vaswani et al. 2017) to explicitly model higher-order interactions among the instances in a set.
We evaluate three set-input models for the FACILE-FSP model: attention-base MIL pooling (Ise ctt al.] 2018), Deep Set (Zaheer et al. 2017), and Set Transformer (Lee et al.J 2019a). Attention- based MIL pooling uses weighted average of instance embeddings from set where weights
determined by a neural network. The attention based MIL. pooling corresponds to a version of attention (Lin al. 2017 Raffel & Ellis 2015). It has been adapted by Zhang al. (2020b a); Pal et al. (021) in the context of H&E images. uses a single fully connected layer and softmax predict the attention weights for instances. In the Deep Set model, each instance in a set is independently fed into a neural network that takes fixed-sized inputs. The extracted features are then aggregated using a pooling operation (i.e.. mean, sum, or max). The final output is obtained by further non-linear operations. The simple architecture satisfies the permutation invariant property and can work with any set size. Set Transformer adapted the Transformer model for set data. It leverages the attention mechanism (Vaswani et al. (017)) to capture interactions between instances of the input set. It applies the idea of inducing points from the sparse Gaussian process literature to reduce quadratic complexity to linear in the size of the input set.
with batch normalization and ReLU activation
We train FACILE-FSP with three set-input models. The set size is set to 5. In the attention- based MIL pooling model, we implemented the simple version, and use the single fe layer with softmax to predict attention weights from ResNet18 extracted features. For Deep Set model, we use two fo layers with ReLU activation functions in between to extract instance features before set pooling. In the Set Transformer,
4 attention heads and 3 inducing points.
From table 12 we conclude that none of the set-input models used in FACILE-FSP is consistently better than the other set-input models. The Deep Set model achieves the highest average F1 score with more tasks.
F.2 LEARNING CURVE
To validate the adequacy of training for all models, assess the intermediate checkpoints of each pretraining model on the LC dataset. The learning curves and confidence intervals (CI) of FACILE- FSP, FSP-Patch, and SimSiam displayed in figure 12 Upon careful examination of the learning curves in figure [2] we observe conclusive evidence of complete training for all models, as they have reached convergence.
F.3 INPUT SET SIZE
To examine the impact input set size on downstream tasks, we conduct pretraining experiments using FACILE-FSP on the TCGA dataset with varying input sizes. The resulting feature map
28

---

Published as a conference paper at ICLR 2024
set-input model
NC
LR
RC
IR+LA
RC+LA
I-shot 5-way test
LC
dataset
Allention-busee MIL pooling
70.53 1 1.32
69.86 1 L.39
69.75 1 1.37
71.15 1 1.31
70.31 1 1.34
Deep Set
77.84 + 1.16
77.56 + 1.16
77.56 + 1.17
79.16 + 1.09
77.38 + 1.18
Set Transformer
75.09
1.30
73.57
1.29
73.16 + 1.33
74.03
1.28
72.88
+ 1.34
5-shot
5-way test on LC
dataset
Attention-bused MIL pooling
88.12 1 0.59
81
1.04
82.51
0.97
89.18 0.57
88.15
Deep Set Transformer
90.35
90.91
0.47
91.54
0.46
0.50
90.97 0.54
90.67
0.54
89.18
0.61
89.02 1: 0.63
90.03 1: 0.59
88.71 +: 0.67
I-shot
test
PAIP dataset
Attention-han based MIL pooling
50.98
1.37
51.93
1.35
51.91 = 1.36
51.98 + 1.36
52.39 + 1.35
Deep Set Set Tramsformer
52.04 + 1.25
53.27 + 1.25
54.19 + 1.26
52.66 + 1.25
52.79 + 1.23
48.81
1.21
50.08
1.24
50.75 1 1.23
50.03
1.23
49.41 1.20
5-shol
3-way test on PAIP dataset
Attention caantoon-based MIL. pooling
67.04
1.00
66.06
1.17
66.61
1.10
70.19 + 0.87
70.54 + 0.81
Deep
69.42
0.85
69.93
0.92
70.52 0.87
69.96
0.84
68.39
0.84
Sct Transformer
66.61
67.57
0.95
67.78 0.95
68.24 0.85
67.20 0.86
T-shot
9-way lest
NCT dataset
Atlention-base MIL. pooling
60.04 + 1.40
64.53 + 1.29
64.81 + 1.31
64.00 + 1.34
06.66 + 1.32
Deep Set Set Transformer
68.21 +: 1.30
68.17 +: 1.31
68 68.69 +: 1.30
69. 24 1: 1.28
68. t: 1.33
67.76 = 1.31
68.52 + 1.30
68.55 = 1.28
08.33 + 1.28
67.72 # 1.28
5-shot
9-way test on NCT dataset
Attention-basse based MIL. pooling
81.94 0.75
82.40 + 0.72
84.46 + 0.65
86.49 + 0.62
87 0.59
Deep
85.18 0.60
85.87 0.60
87.11 0.56
87.06
0.61
85.81 + 0.66
Sct Transformer
86.45 0.62
87.74 0.59
87.97 0.58
88.00 0.59
86.92 0.61
Table 12: Performance of FACIEL-FSP with three different set-input models; average Fl and CI are reported.
05-
815
14-
775
AC
MC
MC
16
74
38
nurber stepp
rumber of spocha
runberal sparu
(a) FACILE-FSP model
(b) FSP-Patch model
(c) SimSiam model
Figure 12: Learning curves of FACILE-FSP model, FSP-Patch model, and SimSiam. The mean F1 score and CI of few-shot models tested the LC dataset with 5-shot are shown with curves.
from the trained FACILE-FSP then evaluated on LC, PAIP, and NCT datasets with shot numbers and The corresponding performances are reported in table 13
Observing table 13 we find that models with an input set size of 5 consistently demonstrate superior performance for LC and PAIP datasets. While slight improvements observed for larger input set sizes, they are not substantial. Conversely, for the NCT dataset, as presented in table 13 the best performance attained when the input set size 10.
G CONTRASTIVE AND NON-CONTRASTIVE LEARNING MODELS
Self-supervised learning achieves promising results on multiple visual tasks (Bachman et al. 2019 He et al., 2020 Chen et al. 2020 Grill et al.) 2020 Caron et al. 2020 Chen & He 2021 Con-
trastive learning method avoid collapse by encouraging the representations to be far apart for views from different images. Henaff (2020) He et al. (2000); Misra & Maaten 2020); Chen et al. (2020)
implemented instance discrimination, in which pair of augmented views from the same image are positive and others negative. Caron al. 2020
2018 contrasted different cluster of positives.
29

---

Published as a conference paper at ICLR 2024
set
5
10
15
T-shot 5-way
test on T.C. dataset
NC
75.29 + 1.33
= =7.8411.18 1.16
74.88
+ 1.36
75.25 = 1.29
LR
73.72 + 1.33
77.56 - 1.16
73.84
+ 1.29
74.00
1.27
RC
74.10 1.34
77.56 1.17
73.42
+ 1.31
73.42
1.29
LR+LA
1 75.27
79.16 1.09
74.41
+ 1.31
74.92
1.26
RC+LA
74.36 + 1.33
77.38
1.18
72.60
+ 1.34
73.16 = 1.32
5-shot 5-way
test on LC dataset
NC
90.62 1 0.56
90.35 0.50
90.62 1 0.57
90.83 0.55
89.41 1 0.63
90.91 0.47
89.80 + 0.59
89.63 0.60
RC
89.11 + 0.63 0.63
91.54 0.46
89.26 + 0.61
89.25 = 0.60
LR+LA
90.46 + 0.58
91.68 0.50
90.29 + 0.57
90.46 = 0.56
RC+LA
89.61 + 0.63
90.97 0.54
88.52 + 0.66
89.00 = 0.64
NC
48.95 + 1.24
52.04 + 1.25
51.72 1.22
52.46 + 1.20
LR
+ 1.22 1.22
53.27
52.33 + 1.25
53.38 1.23
RC
50.14 1.25
54.19 1.26
53.04 + 1.24
52.68 1.25
LR+LA
50.12 1.22
52.66 1.25
52.96 + 1.21
53.41 1.21
RC+LA
49.91 + 1.22
52.79 = 1.23
51.67 + 1.17
51.51 = 1.20
5-shot 3-way
test on PAIP dataset
NC
66.99 + 0.93
69.42 0.85
69.10
69.08 0.87
LR
68.11 + 0.94
69.93 0.92
70.30 + 0.90
69.28 = 0.90
RC
68.63 + 0.91
70.52 0.87
70.45 + 0.87
70.12 0.90
LR+LA
69.03 + 0.83
69.96 + 0.84
70.25 + 0.81 0.81
70.00 0.81
RC+LA
67.32 1 0.83
68 39 0.84
68.35 1 0.83
67.70 = 0.81
NC
66.31 + 1.36
68.21 + 1.30
72.44 = 1.25
72.05 = 1.27
LR
68.55 + 1.32
68.17 + 1.31
72.62 + 1.25
72.14 1.27
RC
68.58 + 1.32
68.69 1 1.30
72.60 = 1.25
72.04 = 1.27
LR+LA
67.42 1 1.33
69.24 1.28
72.18
1.26
71.92 1.27
RC+LA
65.87 + 1.36
68.18 + 1.33
69.98 + 1.31
69.88 = 1.28
5-shot 9-way
test on NCT dataset
NC
85.28 1 0.72
85.18 1 0.60
88.25 1 0.56
88.22 0.57
LR
86.39 0.69
85.87 0.60
88.80 - 0.55
88.55 0.55
RC
87.03 + 0.66
87.11 +
89 89.25 - 0.52
89.02 = 0.54
LR+LA
86.85 + 0.65
87.06 1 0.61
88.52 + 0.59
88.93 0.55
RC+LA
85.60 + 0.70
85.81 + 0.66
87.40 + 0.63
87.74 0.59
Table 13: Abation on set size; models tested on LC, PAIP, and NCT dataset; average F1 and CI are reported.
Non-contrastive models (Grill et al.) 2020: Richemond et al. 2020: Chen & He 2021 removed the
reliance on negatives. These non-contrastive models achieved strong results in the ImageNet (Deng et al. 2009) pretraining setting. SimSiam (Chen & He 2021) works with typical batches and does not rely on large-batch training, which makes preferable for academies and practitioners with low computation resources.
In this section, some contrastive learning and contrassttt learning models, e.g., SimCLR, Sup- Con, and SimSiam, that are used in this paper are explained. Details of implementation are provided. There are three main components SimCLR and SupCon framework. We follow the notation of Khosla al. (020) in this section to explain SimCLR and SupCon.
Data augmentation Aug(.). For each input sample I, the augmentation module generates two random augmented views, i.e., ã ~ Aug(z). The augmentation schedules used in this paper are explained in section
Encoder Enc(-)) The encoder extracts representation vector Enc(ã). The pair of augmented views are separately fed to the same encoder and generate a pair of representa- tions. The is normalized to the unit hypersphere.
Projection head Proj(-). It maps r to vector = (ro) We instantiate Proj(.) multi-layer perceptron (MLP) with single hidden layer of size 512 and output vector size of 512. We also normalize the output to the unit hypersphere.
30

---

Published as a conference paper at ICLR 2024
Backbone
Projection head
View
.
Contrastive loss
View 2
Backbone
Projection head
Figure 13: Abstraction of SimCLR structure
For set of N randomly sampled sample/label pairs, {(z,, ux]]k=1 The corresponding batch used for training consists of 2NV pairs, {( ūi)) where T26-1 and T2x are two random augmented views of F4 and U2k-1 U2k y/--
G.1 SIMCLR
Let i e / E 2N be the index of an arbitrary augmented sample and let j(() be the index of the other augmented sample originating from the same source sample. The abstraction of SimCLR structure can be found in figure 13 In SimCLR, the loss takes the following form.
exp (zi 2100TT)
Self
Colf
Σ iE!
log
(2)
Ene A(1) exp (z) Za/T)
IE7
where F is the temperature parameter. A(i) = TVt). The denominator has total of 2N 1 terms.
In this paper, the is always set to 0.07. The patches are augmented randomly by the augmentation module described in section We an MLP projection head with two fully-connected layers, a hidden dimension of 512, and an output dimension of 512.
G.2 SUPCON
For supervised learning. the contrastive loss in equation handle class discrimination (Khosla et al. 2020). Khosla et al. (2020) proposed two straightforward ways, as shown in equation and equation to generalize equation incorporate supervison.
exp (zi + Ep/T)
sup out
suu out
log pEr(s)
(3)
P(i)
EoeA(s) exp (zi za/T)
iE!
SET
exp (2+ Zp/T) PEP(1) Emer(o) exp (2: Za/T)
Suu
ii
log
Σ
(4)
P(i)
iel
NET
Here P(i)= = (p A(i) i/ 5z is the set of indices of all positives in the batch distinct from i. The authors showed that Chup Laup and Lou is the superior supervised loss function. Thus, use SupCon with equation as the default loss. The r is also set to 0.07.
In model FACILE-SuPCon, the input sample is set of randomly sampled patches and labels are slide properties, i.e., organs or TCGA projects. Each patch augmented randomly by the augmentation module described in section D The feature map and set function g work as the encoder Enc(-). We also use an MLP as projection head with two fully-connected layers, a hidden dimension of 512, and output dimension of 512.
31

---

Published as a conference paper at ICLR 2024
Backbone
Projection MLP
Prediction MLP
View
Simllartt
Stop-grad
Backbane
Projection MLP
Figure 14: Abstraction of SimSiam structure
When employing set-input data with the SupCon method, the standard practice of augmenting each instance within a set poses significant challenges for the training of SupCon models. These chal- lenges stem from two main aspects: 1) Complexity in maximizing agreement with set-input data: SupCon is traditionally trained maximize agreement between differently augmented views of the same data point using labeled data. In our application, using set-input data means that we apply con- ventional data augmentation methods each instance within set. This results in an independently augmented set of images, as opposed to augmenting a single instance. This complexity makes more challenging to achieve the desired maximization of agreement. 2) Constraints on batch sizes due to set inputs: Set-input models take batch of sets as input instead of a batch of instances. It requires us to use relatively smaller batch sizes when using the same hardware configuration because of the set input. It's important to emphasize that the batch size is critical factor for the effectiveness of the SupCon model.
We have observed that despite these challenges, the performance of FACILE-SupCon is commend- able in contexts involving smaller datasets or less complex models, i.e., CIFAR- in section 3.2 and section 3.3 or smaller trainable models discussed in Appendices B.1 and B.3. We believe that our approach, with its nuanced application of SupCon in a set-input context, offers a valuable contribution to the field and shows the versatility of the FACILE algorithm.
G.3 SIMSIAM
Non-contrastive models, e.g., SimSiam and BYOL, achieve strong results in typical ImageNet (Deng
al. 2009) pretraining setting (Chen & He 2021 Grill et al. 2020 Li et al. (022). Among the contrastive models, SimSiam removes the negatives and uses stop-grad to avoid collapse. Besides,
trains faster, requires less GPU memory, and works well with small batch size (Chen & He 2021
Li et al. 2022). which makes it extremely appealing to academics.
The abstraction of SimSiam structure shown in figure 14] Given two augmented views ã1 and 2o of the same image SimSiam learns to use ã to predict the representation of ø2. Specifically, 21 passed into the online backbone network on the upper. The ig is passed into the target backbone network on the lower. The outputs of the two backbone networks are passed to the projection MLPs and then a prediction MLP is used to predict the projected representation of 8e from the projected representation of ē1. SimSiam uses the same network for the online and target backbone and projection networks.
In our paper, the projection MLP has 3 fully-connected layers with a hidden dimension of 512 and an output dimension of 512. It has batch normalization (BN) applied to each fully-connected layer including its output fully-connected layer. The prediction MLP also has BN applied to its hidden fully- colneccon layer. Its output fully-com connecco layer does not have BN or ReLU, The prediction ML. has 2 layers.
32

---

Published as a conference paper at ICLR 2024
H EXCESS RISK BOUND OF FACILE
Our proof framework follows closely the work of Robinson et al. (2020). We consider the setting where we have some coarse-grained labels of some sets, rather than instances and the downstream classifiers only use the learned embeddings to train and test on the downstream tasks. [Robinson et al. (2020) considers different setting where each instance has weak label and strong label, and the strong label predictor learns to predict the strong labels from the instances and their corre- sponding embeddings learned with weak labels. The diagram of only using trained embeddings for downstream tasks is more often used in self- supervised learning and representation learning for FSL 2019 He al. 2020, Chen al., 2020 Grill et al. 2020: C Caron et al. 2020 Chen & He 2021). The coarse- garse--raaned data contains useful information, which is characterized by our defined Lipschitzness, to pretrain a instance feature map that can be leveraged for downstream FSL. We include the full proof of our key result as follows.
literature (Du al. 2020 Yang al. 2021; Bachman
In order to prove theorem 4 we first split the excess risk by the following proposition.
Proposition 5. Suppose that f is L-Lipschitz relative to E. The excess risk
plg Joe (x, Y) ff. (X, is bounded by,
2.Rate (8% Es + Rato, (P5,
Proof. We split the excess risk into three parts
Exxy (% (o (X,Y)
fo (X,
(as (9 (x,)) of (x,)) Exxx (x,) e.. (x,
-EPx
+ +Exx
1 (X,Y) ef. (x
For the second term and third term, relative Lipschitzness of f E delivers
Exx,Y (x,o) - f/ ono (x,r) Exx I (x) Foo (x,5)
< LEPx.....w EtH (ge o ((S), co(())
LEEs los (ga o Ico eo (S)),
E Pxx, (3 fooe (X) - de
00 (x,
= E.x.Y. 000 (x,Y) oo x, X,y)
< LEPx CoR (ge, o co((S) ge e (S))
= -LE 6os (geo o (S()), ge- oe oe (S))
Since e* attains minimal risk and W = Ien eo(S), the sum of the two terms can be bounded by,
LEPs 8 (ge o e(S), Sco o ee())) LEPs Ce8 (geo o oeo(S), ge- (S))
<2LEPs fo (ge ē(S), W) s 2LRato, ...
By combining the bounds on the three terms we can get the claim.
The central condition is well-known to yield fast rates for supervised learning (Van Erven et al. 2015). It directly implies that we could learn map Z + Y with ((1 /n) excess risk. The difficulty that test time we would need access to latent value Z = e(X). To circumnavigate this hurdle, we replace en with è and solve the supervised learning problem (et F).
It is not clear whether this surrogate problem satisfies the central condition. We show that (efH, F) indeed satisfies weak central condition and shows weak central condition still en- ables strong excess risk guarantees.
Following Robinson al. 2020); Van Erven al. (0055) define the central condition F.
33

---

Published as a conference paper at ICLR 2024
Definition 6.
(The central condition). A learning problem (etx, Pz,x FF) Z x is said to satisfy the e-weak y7-entral condition there exists an f E F such that
e
for all / E F. The O-weak central condition is known as the strong central condition.
Capturing relatedness of pretraining and downstream task with the central condition. Intu- itively, the strong central condition requires that the minimal risk model f attains a higher loss than E F on a set of Z, Y with an exponentially small probability. This likely to happen when Z is highly predictive of Y so that the probability of Z) concentrates in a single location for most the time.
most of Z. If f in F such that (Z) maps into this concentration, E7.. (Z, Y) will be close
We assume that the strong central condition holds for the learning problem , F)) where Z = (X)) Similar to Robinson et al. (2020), we split the learning procedure into two supervised tasks as depicted in Algorithm In the algorithm, we replace ( P F) with (e's, Pz,,
We will show that (C'5, F) satisfies the weak central condition.
Proposition 7. Assume that (=,w,, w') = 1 1 & w'} and that eo is bounded by B O, F is L-Lipschitz relative to &, and that (REH, ,)), satisfies c-weak central condition. Then (6%s P2,y,)) satisfies the + O eexp(nB) Ratem (E, (E, (s,P)))) -weak central condition with proba- bility at least 1-
7
Proof. Note that
1
log pay exp (nl, - -4) = log Exx, exp (nle7 - (
1
To prove that (Lts, F) satisfies the central condition we therefore need to bound log Exx,y exp (n(et. - ( by some constant.
log Exxy exp (nlef - (oal)
log EXxYS. exp (n(efs e
17
log Exx.v.. Texp(nle - ( ( = -w]]= - 10))) @(5) + wi]
log Exx.x.. eepp(nlffi -
n
1
exp(n(e es 1)1 w]]
log E Exx,s..
log E P.......
L o e(()) + w3
sccond
The third line follows from the fact that for any f in the event {ge o ē(S) = W} we have ds fog
As
fogo'
This is because 1 (X,Y) 1o00 (X, Y) < (8% E(S), eo())) Le W) 0.
We get 1 log loggEx.y.sw (exp(n(eff /ree - [) after we drop the 1(0s E(S) W}. It is bounded by with the weak central condition. The second term is bounded by
34

---

Combining bounds with proposition we conclude that Published as a conference paper at ICLR 2024
1 log Ep v.S.W Y.
exp(nle - As foo) o o((S) # wi]
7
1 log E EXX.Y...
(exp(nB)1 o ( + W]]
"
1 log E EP (exp(nB)1(0c o E(S) + W|]
7
1
EP (exp(nB)1 8(8) + W}]
")
exp(nB)
(a. E(S) # W)
"
exp(nB) Ratem (f' &))
"
The first inequality uses the fact that f is bounded by B. The forth line is because that log x. By combining this bound with bound on the first term get the claimed result of proposi- tion /
The proof of the main theorem further relies proposition provided by Robinson et al. (2020), as we show below:
Proposition 8. Robinson et al. ( 2020, Suppose 1 Qz,,, satisfies the e-weak central condition, d-dimensional parameters in the l2 norm, F contained in Euclidean ball of radius R. and y is compact. Then when An An(( (fis, Qz,y F) is ERM, fi (2, 2g [2,Y)]
gls is bounded by B 0, FF T -lipschtit
the excess risk Ez, QQu.
is bounded by.
V d d RL' + +og Vc
with probability at least 1 6, where V = B +
Proof of the main theorem: If m = 2((2)) the Ratem E) = O m proposition 7 concludes that (e F) satisfies the O N week central condi- tion with probability at least S. Thus by proposition we can get Rate, (E P2,y, = daß log nt' oogn
O noz
} + naa)
E
ds for
& for (x,1) (X,Y)
22LRatem (s,w, EE) Rate, (E P2,YFF)
day log RI'n + log 8
2L
00
nag +
na
35


Reviews:
[{"review_id": "r2Il4TF34p", "reviewer": "Reviewer_4JEd", "summary": "The paper propose to utilize set-level coarse-grained labels to improve fine-grained image classification. Essentially the paper is proposing a new pretraining method, key to the method is selecting a dataset with coarse label, and use the set prediction on coarse label as pretraining task. The paper provides theoretical analysis for the proposed approach, showing that using coarse-grained labels speed up the learning on the fine-grained classification task. The paper also demonstrates the effectiveness on several datasets.", "questions": "1. To what extent does the similarity between the pretraining dataset and its coarse labels and the target dataset with its fine labels affect the effectiveness of the method? For instance, can the method perform well when the pretraining dataset is CIFAR-100 while the downstream task involves a medical dataset? In such a scenario, which pretraining method is preferable: supervised pretraining on ImageNet, self-supervised pretraining (ignoring labels entirely), or the proposed method?\n\n2.  Given the same 'related' dataset, if you have both the fine-grained label and coarse-grained label, which pretraining strategy is preferable?\n(Let say your downstream task is classification on a medical image dataset, with fine-grained label A,B,C. The pretraining dataset you have is another medical image dataset (thus more related than ImageNet). You have both coarse label D,E,F and fine-grained label G,H,I,J,K,L. In this case, is fully supervised pretraining on G,H,I,J,K,L more beneficial on set level coarse pretraining on D,E,F more beneficial?)", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The idea of using set prediction on coarse label as pretraining task seems novel\n2. The performance seems strong compared to other baselines", "weaknesses": "1. More baselines for strong self-supervised pretraining methods (e.g., BYOL, DION  are needed to demonstrate the effectiveness. As proposed method is essentially a pretraining strategy, that bears a lot of similarity with exisiting self-supervised learning method\n\n2. More ablations and discussion on some key questions are needed (see below)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wti4tVwN2V", "reviewer": "Reviewer_jPxq", "summary": "The paper presents a new technique aimed at boosting instance-level image classification by utilizing set-level labels. Compared to conventional methods that rely on single-instance labels, the proposed approach achieves a 13% increase in classification accuracy when tested on histopathology image datasets. A theoretical examination of the method outlines conditions for rapid reduction of excess risk rate, adding credibility and robustness to the technique. This research serves to connect instance-level and set-level image classification, providing a noteworthy direction for enhancing image classification models that use coarse-grained set-level labels.", "questions": "- Do you think you could pictorially diagram the approach adding the relevant details? It is unclear to me if the method essentially pretrains using coarse-labels and then fine-tunes on the test set using the support set or is there more to the method\n\n- Why are the methods for pretraining SupCon and FSP chosen for pre-training? Adding rationale for this might help motivate the choice of pretraining method", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper presents a new technique for enhancing instance-level image classification by making use of set-level labels. This serves to fill the existing gap between instance-level and set-level image classification.\n\n- The robustness and reliability of the proposed method are underscored by a theoretical analysis, which outlines conditions for the rapid reduction of excess risk rate.\n\n- The paper clearly articulates the proposed method, shedding light on both its theoretical underpinnings and empirical results. These results are demonstrated on both natural and histopathology image datasets.\n\n- The method put forth in the paper holds promise for extending the capabilities of image classification models. By leveraging set-level coarse-grained labels, the approach achieves better classification performance compared to traditional methods reliant on single-instance labels. This is particularly relevant in real-world contexts where set-level labels may offer more comprehensive information.", "weaknesses": "- The use of coarse-grained labels like TCGA or NCT is an interesting choice. These are indeed umbrella terms for various subcollections, and traditionally they may not provide a strong learning signal. It could be beneficial to delve into why these particular labels were chosen and what advantages they offer in this context.\n\n- Your team's approach to pretraining with coarse labels and then fine-tuning on a support set is a solid and proven method. However, it would enrich the work to articulate what sets this particular application or implementation apart in terms of novelty.\n\n- The comparison with SimCLR and simSIAM provides useful insights, but considering the advancements in the field, benchmarking against more recent self-supervised learning methods like DINO or DINOv2 might offer a more comprehensive evaluation.\n\n- To further validate the generalizability of the method, it could be insightful to include results against standardized few-shot learning benchmarks, such as Mini-Imagenet 5-way (1-shot) or SST-2 Binary classification.\n\n- Adding ablation studies that feature additional pretrained models—or even models pretrained without the coarse labels—could help underscore the specific benefits of using coarse-grained label-based pretraining in your approach.\n\n- Your methodology would be even more robust if additional training details are shared. Information on image augmentations, learning schedules, and optimizer settings could offer valuable insights and help in the reproducibility of your results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6YFWeWAGpw", "reviewer": "Reviewer_xN2u", "summary": "This paper proposes a new setup for few-shot learning. The proposed model is pre-trained on coarse-grained set-level labels first and fine-tuned with fine-grained labels. Authors also provide theoretical analysis on the convergence rate for downstream tasks, which shows coarse-grained pre-training can enhance the learning process of fine-grained label tasks. The experiments are performed on both natural image datasets and medical histopathology datasets, where the baselines are mostly self-supervised learning methods.", "questions": "- There should be more training details about the framework in Sec.2.1. \n- Fig. 3 is referenced in text before Fig. 2\n- Most of the refernece hyperlinks other than page 1 is not working.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I think the idea of using coarse-grained label is reasonable. The conclusion of enhancing learning process of fine-trained labels is inspiring.", "weaknesses": "- I have some questions about the method part, Sec. 2.1. In Fig.2 (a), are input samples all belongs to the same set-level labels? I am confused by this figure and Fig.1(a) CIFAR images. What I believe is correct is that each batch contains samples belong to different set-level labels, and the coarse label is assigned to each sample for pre-training. \n- How is SupCon trained? It is superised that supervised contrastive learning perform a lot worse than basic CE approach in most setups. There is not much information about training details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)